Designing an AI Mesh: Lessons from Incident Investigation

articleOctober 14, 2026Updated October 28, 2026By Ali Haghighi

Building Enterprise AI Systems, part 2

In Part 1, I explored why AI Mesh comes up when independently owned agents need to work together. The idea sounds attractive: discovery, connectivity, security, and observability shared across an agent ecosystem.

But architecture becomes interesting when we have to choose what not to distribute.

Let's use a concrete scenario: a production API becomes slow after a deployment. An engineer wants to know what changed, what the metrics show, and whether a previous incident offers a useful explanation.

We could build a network of specialized agents to investigate it. We could also build one application with three tools. Which design would I start with, and what would make me change my mind?

Start with the investigation, not the agents

An incident investigation needs to produce something a human can evaluate:

A timeline of what happened.
Evidence from metrics, deployment records, and prior incidents.
Hypotheses, clearly distinguished from confirmed facts.
References back to the underlying sources.
Suggested next checks, not unsupported claims of root cause.

None of these requirements says we need multiple autonomous agents.

For an initial implementation, I'd use a single investigation coordinator and ordinary read-only tools:

TEXT
              Incident API / UI
                     |
           Investigation Coordinator
                     |
           Evidence collection tools
             /       |         \
         Metrics  Deployments  Runbooks
          API       API         Search
                     |
              Evidence store
                     |
             Human review / report

The coordinator owns the workflow. The tools return structured evidence. The evidence store keeps provenance separate from the model's interpretation.

This architecture is deliberately boring. It gives us one execution path, one place to enforce budgets, and fewer distributed failure modes.

Decide what deserves to be an agent

A common mistake is treating every integration as an agent. Reading a Prometheus metric or fetching a deployment record doesn't require autonomous reasoning. A deterministic tool is usually better.

I would apply a simple test before extracting a capability into an independent agent:

Does it own a meaningful workflow, not just an API call?
Does it need to make its own bounded decisions?
Is it independently owned, deployed, or reused?
Does it justify the extra cost of tracing, securing, evaluating, and operating it?

For example, a metrics query should normally remain an MCP tool or a normal API integration. An investigation service that correlates multiple signals, maintains a task lifecycle, and serves several consuming teams might justify an agent boundary.

Ownership and lifecycle are often better indicators than the number of LLM calls.

A gradual architecture for growing teams

Suppose the deployment team later builds its own deployment analysis agent. It can explain rollout stages and correlate changes across services. Another team operates a knowledge agent that understands incident histories.

Now the topology might look like this:

TEXT
                 Incident Investigation
                    Coordinator
                   /     |      \
          MCP tools   A2A task   A2A task
             |           |          |
        Metrics API   Deployment  Knowledge
                        Agent      Agent
                          |          |
                        CI/CD    Documents

The distinction matters. MCP is useful for exposing tools and context. A2A is designed for communication and task collaboration between independent agents. They are complementary, not competing layers.

Only after multiple independently operated capabilities create repeated connectivity and governance problems would I evaluate a mesh layer for discovery, identity, access policy, and cross-environment communication.

Neither A2A nor a mesh decides whether a deployment caused an incident. That remains application logic, tested against evidence.

Define evidence contracts before network protocols

For incident investigation, I care more about the evidence contract than the transport.

A minimal .NET representation might be:

C#
public sealed record EvidenceItem(
    string Id,
    string Source,
    string SourceReference,
    DateTimeOffset ObservedAt,
    string Summary,
    IReadOnlyDictionary<string, string> Attributes);

public sealed record InvestigationResult(
    string IncidentId,
    IReadOnlyList<EvidenceItem> Evidence,
    IReadOnlyList<string> Hypotheses,
    IReadOnlyList<string> RecommendedChecks);

These records are intentionally small examples, not a complete production schema. A real implementation would also define versioning, validation, source identity, correlation, and confidence or uncertainty semantics where appropriate.

One principle is non-negotiable: an agent-generated hypothesis must never silently become observed evidence.

If the deployment agent says, "This rollout may explain the latency increase," that is an interpretation. The deployment timestamp and measured latency curve are evidence. A reviewer must be able to inspect both separately.

A mesh can move these messages securely. It cannot make them true.

Put identity and authorization at every boundary

Read-only incident tooling can still expose sensitive information: service topology, internal documents, production metrics, and operational timelines.

I would separate three questions:

Who is calling? Authenticate the user or agent identity.
What may it access? Authorize the specific capability, environment, and data scope.
What may it do? Distinguish observation from mutation. Reading metrics and triggering a rollback must not share the same permissions.

A mesh or gateway may help enforce transport identity and centrally configured policies. Each downstream service still needs to validate its own authorization requirements. Trusting a request merely because it came from another agent creates a dangerous shortcut.

For an incident-investigation MVP, I'd begin with read-only tools and explicit allowlists. Mutating operations, if introduced later, should require a separate permission model and suitable human approval.

Avoid logging access tokens, secrets, or unredacted tool payloads just to make tracing easier.

Trace the full workflow, not only model calls

Consider an investigation that starts in the UI, retrieves metrics, contacts a deployment agent, searches earlier incidents, and asks a model to summarize the findings.

If these activities are traced independently, debugging is painful.

At minimum, I'd propagate a shared correlation context through the workflow and record:

Investigation and task IDs.
Tool or agent invoked, with start time and duration.
Outcome: success, timeout, denied, or invalid response.
Model usage and cost where relevant.
References to evidence used in the final report.

OpenTelemetry-style distributed tracing is a useful starting point, even if the implementation is initially a single .NET service.

For model calls, record useful metadata and usage without assuming full prompts and responses belong in telemetry. Production logs can become a source of data leakage.

Tracing tells us what ran. Evaluation tells us whether the result was useful and adequately supported. We need both.

Expect partial failures

Distributed investigations will fail in ordinary ways. A metrics endpoint times out. The deployment agent is unavailable. A knowledge search returns stale data. One downstream agent responds after the investigation deadline.

The wrong approach is to retry everything indefinitely or fabricate a complete answer.

I would establish a clear execution budget for each investigation:

TEXT
Investigation deadline
  |
  +-- Metrics: bounded timeout
  +-- Deployment analysis: bounded timeout
  +-- Knowledge search: bounded timeout
  |
  +-- Collect successful evidence
  +-- Record missing or stale sources
  +-- Produce a partial, qualified report

Retries should depend on the operation. A transient read request may be safe to retry within a deadline; a mutating operation requires stronger idempotency controls.

The final report should say that deployment evidence was unavailable rather than presenting "no deployment found" as a fact.

That difference can determine whether an engineer trusts the system.

Keep model cost under control

Adding more agents can add more model calls, larger context windows, and repeated analysis of the same evidence.

I would measure the total cost of an investigation, not just the cost of an individual prompt. Useful signals include model calls per investigation, input and output tokens, repeated retrieval work, wall-clock duration, and how often the result required human correction.

A practical cost-control sequence is:

Collect deterministic evidence before involving a model.
Reuse retrieved evidence where freshness and access rules allow.
Set per-investigation time and usage limits.
Use a model only where interpretation adds clear value.
Compare agent-based designs against simpler orchestration using the same evaluation cases.

For some tasks, a local model may reduce marginal API charges. It does not eliminate hosting, maintenance, quality, or latency costs. Those tradeoffs need measurement.

So when would I introduce AI Mesh?

Not when the first MCP server is added. Not when a second agent appears on a diagram.

I would introduce mesh infrastructure when we can point to recurring problems that it actually resolves: independently owned agents need reliable discovery, communication across security boundaries, consistent identity and policies, or shared operational visibility.

Even then, I'd evaluate the proposed mesh against simpler alternatives: API gateways, existing service-mesh infrastructure, a service registry, and explicit A2A connections.

A decision record could ask:

| Question | Evidence to collect | | --- | --- | | Why isn't direct integration enough? | Number and cost of integrations, change frequency | | What does mesh infrastructure centralize? | Identity, policies, discovery, routing, observability | | What remains application-owned? | Task orchestration, evaluation, evidence handling | | What new failures are introduced? | Control-plane availability, configuration, routing | | How will we know it helped? | Integration effort, incidents, latency, cost, operability |

A diagram is not evidence that the architecture is better.

A .NET implementation path

For a .NET engineering team, I'd use a staged approach rather than committing immediately to a particular agent framework.

First: build an ASP.NET Core investigation API, a bounded coordinator, typed tool adapters, and structured evidence storage. Use dependency injection, clear interfaces, and integration tests around each adapter.

Next: expose reusable read-only capabilities through MCP where a standard tool interface genuinely helps. Add OpenTelemetry tracing, permission checks, model-usage accounting, and replayable evaluation cases.

Then: if another team owns an independently operated agent, define an interoperable agent contract and assess A2A for task communication. Test failures, authorization, and trace propagation across that boundary before expanding.

Finally: evaluate an AI Mesh implementation when operating multiple agents exposes a clear infrastructure gap. Make the decision based on an experiment, not on the ambition to become "multi-agent."

This sequence leaves room to scale without paying for a distributed platform before the boundaries warrant it. The next two articles make the tradeoffs concrete: a bounded .NET investigation coordinator and a cross-team agent integration.

The decision I'd document

AI Mesh is an interesting answer to a real enterprise problem: connecting and governing AI capabilities that do not all belong to one application or team.

But it is an infrastructure idea, not a substitute for sound workflow design.

For incident investigation, I would prioritize trustworthy evidence, controlled tool execution, explicit permissions, and observable failures. Those requirements hold whether there is one agent or fifty.

If an organization eventually needs a mesh, it will benefit from those foundations. If it never needs one, it will still have built a reliable system.

Start with the investigation you need to complete. Let the architecture grow from the boundaries you can demonstrate.