Building a Bounded Incident Investigation Coordinator in .NET

articleOctober 18, 2026Updated October 28, 2026By Ali Haghighi

Building Enterprise AI Systems, part 3

In Part 2, I argued that an incident investigation should begin with one coordinator and a few predictable tools. Here is what that looks like in .NET.

The problem is small enough to describe clearly: an API alert comes in, and we want to collect deployment history, latency metrics, and relevant runbooks. A model may eventually summarize the findings, but it must not be the source of truth.

I would build the evidence collection path first. A system that cannot tell the difference between missing evidence and a negative finding should not be trusted to diagnose production incidents.

Make missing evidence explicit

Use immutable records at the boundary. Keep tool execution status separate from the observations returned by a tool.

C#
public sealed record Evidence(
    string Source,
    string Reference,
    DateTimeOffset ObservedAt,
    string Summary);

public enum ToolStatus { Succeeded, TimedOut, Failed }

public sealed record ToolResult(
    string Tool,
    ToolStatus Status,
    IReadOnlyList<Evidence> Items,
    string? ErrorCode = null);

public sealed record Investigation(
    string IncidentId,
    IReadOnlyList<ToolResult> ToolResults);

An empty Items collection with Succeeded means a successful query found no matching records. TimedOut means we do not know. This is a small distinction with significant consequences.

In production I would also add source timestamps, query parameters, structured provenance, and stable schema versions. The example stays intentionally focused on the execution boundary.

Separate collection from coordination

The coordinator should not know whether the implementation calls Prometheus, Grafana, a REST API, or an MCP server.

C#
public interface IIncidentTool
{
    string Name { get; }
    Task<IReadOnlyList<Evidence>> CollectAsync(
        string incidentId,
        CancellationToken cancellationToken);
}

In an ASP.NET Core application, the implementations can use typed HttpClient instances. Each adapter owns source-specific parsing and authorization requirements. It should return data, not a persuasive natural-language diagnosis.

For example, a metrics adapter might return: "P95 latency increased from the baseline during the alert window," with a link or query reference. It should not claim a deployment caused the increase simply because the timestamps overlap.

Parallelism needs a budget

We want parallel evidence collection, but we also need bounded execution. One slow dependency should not block the entire report.

The following coordinator uses a timeout per tool and preserves partial results. It assumes a small, trusted, statically configured set of tools. This sample is illustrative and has not been presented as a load-tested implementation.

C#
public sealed class InvestigationCoordinator(
    IEnumerable<IIncidentTool> tools,
    ILogger<InvestigationCoordinator> logger)
{
    public async Task<Investigation> RunAsync(
        string incidentId,
        CancellationToken cancellationToken)
    {
        var tasks = tools.Select(tool =>
            ExecuteAsync(tool, incidentId, cancellationToken));

        var results = await Task.WhenAll(tasks);
        return new Investigation(incidentId, results);
    }

    private async Task<ToolResult> ExecuteAsync(
        IIncidentTool tool,
        string incidentId,
        CancellationToken callerToken)
    {
        using var timeout = CancellationTokenSource
            .CreateLinkedTokenSource(callerToken);
        timeout.CancelAfter(TimeSpan.FromSeconds(8));

        try
        {
            var items = await tool.CollectAsync(
                incidentId, timeout.Token);

            return new ToolResult(
                tool.Name, ToolStatus.Succeeded, items);
        }
        catch (OperationCanceledException)
            when (!callerToken.IsCancellationRequested
                  && timeout.IsCancellationRequested)
        {
            return new ToolResult(
                tool.Name, ToolStatus.TimedOut, [],
                "TOOL_TIMEOUT");
        }
        catch (OperationCanceledException)
            when (callerToken.IsCancellationRequested)
        {
            throw;
        }
        catch (Exception ex)
        {
            logger.LogWarning(ex,
                "Evidence collection failed for {Tool}",
                tool.Name);

            return new ToolResult(
                tool.Name, ToolStatus.Failed, [],
                "TOOL_FAILED");
        }
    }
}

There are two important details here.

First, caller cancellation is not silently converted into a partial report. If the request was cancelled, we propagate cancellation.

Second, the timeout only works when tool implementations honor the cancellation token. A blocking driver or a dependency that ignores cancellation needs an additional isolation strategy. In a production system I would also enforce an overall investigation deadline, bound the number of concurrent calls, and make sure remote requests have suitable transport-level timeouts.

Expose one narrow endpoint

A minimal API registration could look like this:

C#
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddScoped<
    InvestigationCoordinator>();

// Register concrete, read-only IIncidentTool
// adapters and their typed HttpClient dependencies here.

var app = builder.Build();

app.MapPost("/incidents/{id}/investigate",
    async (string id,
           InvestigationCoordinator coordinator,
           CancellationToken ct) =>
    {
        var result = await coordinator.RunAsync(id, ct);
        return Results.Ok(result);
    });

app.Run();

This is a composition sketch, not a complete production API. Before deploying it, add authentication, scoped authorization, input validation, API-level rate limiting, and bounded resource use. Register actual tool implementations rather than resolving arbitrary tool names from caller input.

The tests that catch misleading results

Three tests matter more than a polished demo:

Successful empty query: The result is Succeeded with no evidence, not TimedOut.
One failed dependency: The other tool's evidence is still returned, and the failure is explicit.
Caller cancellation: Cancellation propagates instead of being reported as a normal investigation.

I'd also test a tool that ignores cancellation, because it exposes a limitation of a naive timeout design.

When you introduce a model, evaluate its report against this structured result. Require it to cite source references for factual claims and explicitly acknowledge unavailable sources. An answer that says "no recent deployment" after the deployment tool timed out is a failed evaluation.

Trace the boundary, not the whole payload

Record one span per tool execution, with a correlation ID linking it to the investigation. Track duration, outcome, and source, but avoid logging the raw incident payload or credentials.

For example:

C#
using System.Diagnostics;

public static class InvestigationTelemetry
{
    public static readonly ActivitySource Source =
        new("IncidentInvestigation");
}

Wrap each execution in an activity and set low-cardinality attributes such as tool name and outcome. Configure the OpenTelemetry pipeline separately; creating an ActivitySource alone does not export traces.

An MCP adapter can still be a tool

An MCP server can expose metrics or documentation tools to an AI application through a standard interface. It does not need to replace the typed adapter layer. The coordinator can call an adapter backed by MCP just as it can call one backed by HTTP.

That separation helps if the transport changes later. The evidence contract stays stable while the implementation evolves.

What stays out of this version

I would not introduce A2A, service discovery, a mesh control plane, or autonomous retries at this stage. Nothing in this scenario requires independently operated agents.

Before adopting any of those layers, I would want real measurements showing that direct integration has become a maintenance or governance problem.

The main engineering result is not the coordinator code itself. It is the contract that makes uncertainty explicit and keeps model interpretation away from source evidence.

In Part 4, we'll cross an actual distributed boundary and look at the security and failure modes that appear when one team calls an agent owned by another.

Companion code on GitHub

The short coordinator sketch accompanies this article. It illustrates typed evidence, parallel tool collection, and explicit outcomes when a dependency fails or times out. It is an educational snippet, not a tested or production-ready application. The interface and records are in the same file.

The complete article series is indexed in the repository README.