AI Agents that Ship

Session Overview

Delivered on October 10, 2026 for the Malta Microsoft AI User Group (MMAUG) virtual meetup, hosted by Microsoft MVP Imoh Etuk.

📺 Watch Full Recording: The complete session recording is available on YouTube at youtube.com/watch?v=FzamICZphPA.

Most teams discover that building a functional prototype in a playground takes fifteen minutes, but taking that agent to production takes months. Gartner forecasts that over 40% of agentic AI projects will be cancelled by 2027 due to escalating costs, unclear business value, and inadequate risk controls. Notice what is missing from that list: “the model wasn’t smart enough.” All three failure modes stem from platform and architecture decisions.

“Production is a property of the system around the model, not of the model.”

This session and accompanying hands-on lab explore Microsoft Foundry as the end-to-end foundation for building, deploying, evaluating, and operating resilient agent fleets in enterprise environments.

---

Downloads & Companion Resources

---

The Five Production Pillars

To move from playground experiments to enterprise systems, teams must address five distinct architectural pillars:

Pillar	Focus	Foundry Capabilities
1. Ground	Shared, cited, permission-aware knowledge	Foundry IQ: Unified knowledge bases over Blob, SharePoint & SQL; agentic multi-step retrieval via MCP endpoints
2. Act	Sandboxed runtime and governed tools	Foundry Agent Service & Toolboxes: Managed execution sandboxes, versioned MCP tool endpoints, Model Router
3. Trust	Evaluated, guarded, red-teamed	Evaluations & Guardrails: Rubric, ASSERT test suites, prompt-shield and PII guardrails, automated red-teaming
4. Govern	Identity, inventory, cost, control	Microsoft Entra Agent ID & Agent 365: Standalone agent service principals, least-privilege scoping, fleet inventory
5. Operate	LLMOps to ship, AIOps to run	CI/CD Gating & Application Insights: GitHub Actions eval gates, OpenTelemetry tracing, quality canary probes

---

Live Demo: Build, Ship, Operate in Six Acts

1. Act 1: The Experiment (2 min) — Playground prompt agent with a policy PDF attached. When asked about hail damage on Policy C-123, it confidently answers incorrectly based on superseded policy wording, with no citations, running under personal user credentials.
2. Act 2: Foundry IQ (5 min) — Grounding with claims-kb over policy wordings, procedures, and claims data. Demonstrates minimal vs. medium reasoning effort and permission-aware retrieval over MCP.
3. Act 3: Productionize (3 min) — Unifying knowledge and claims APIs behind a secure Foundry Toolbox (claims-tools). Deploying a hosted agent with dedicated Microsoft Entra identity and Content Safety guardrails.
4. Act 4: LLMOps — Ship a Change (4 min) — Opening a PR with an instruction change. The GitHub Actions pipeline runs deterministic evaluation checks (evals/gate.py), catches an unauthorized $31,000 claim approval regression, and blocks deployment.
5. Act 5: AIOps — Operate It (4 min) — Service-level observability with OpenTelemetry and Application Insights. Continuous quality canaries detect silent knowledge drift, fire an alert, isolate the offending span in traces, and trigger a Git rollback.
6. Act 6: Govern — Scale the Workforce (3 min) — Reusing the knowledge base and Toolboxes for a nightly intake routine and a Microsoft Teams adjuster assistant, monitored in Microsoft Entra and Agent 365.

---

The Hands-on Lab: 7 Practical Modules

The companion 26-page runbook walks through building this architecture in your own Azure subscription around the Contoso Mutual synthetic insurance scenario:

• Module 1: Prepare the Sample Data — Configuring Azure Storage containers with 2026 wording (Form AUTO-26 with Endorsement E-7) vs. 2024 wording (AUTO-24 excluding outdoor hail).
• Module 2: Build a Prototype Agent and Find Its Limits — Reproducing silent prototype failures in the portal playground.
• Module 3: Ground Agents with a Foundry IQ Knowledge Base — Connecting Azure AI Search, setting retrieval instructions, and testing MCP endpoints with kb_query.py.
• Module 4: Make the Agent Production-Ready — Writing the FastMCP claims server, provisioning Foundry Toolboxes, scaffolding with azd ai agent, and attaching guardrails.
• Module 5: LLMOps: Test and Gate Every Change — Setting up Entra workload identities, GitHub Actions CI (agent-ci.yml), and automated PR evaluation gates.
• Module 6: AIOps: Monitor, Alert and Roll Back — Running canary.py, configuring Log Search alerts in Application Insights, simulating an incident, and executing a rollback from Git.
• Module 7: Scale to a Multi-Agent Workforce and Govern It — Creating scheduled intake routines, publishing the adjuster bot to Microsoft Teams, and monitoring fleet telemetry in Agent 365.

---

Three Things to Take into Monday

1. Separate the knowledge from the agent: Maintain one managed Foundry IQ knowledge base consumed by multiple agents, with permissions enforced at retrieval time.
2. Production is a checklist, not a feeling: Require identity, guardrails, an automated evaluation gate, SLO alerts, and a tested rollback plan before the first user logs on.
3. Build production on GA, pilot on preview: Architect production systems on generally available services, while exploring preview capabilities behind clean interfaces.

---

Speaker & Organization Links

• Speaker: Sarang Brahme — LinkedIn Profile
• Organized by: Malta Microsoft AI User Group
• Moderator: Imoh Etuk (Microsoft MVP, Malta Microsoft AI User Group)