Logging & Observability

Core Principles

1. Structured logging with Serilog — Every log entry is a structured event with named properties, not a formatted string. This enables searching, filtering, and alerting. All setup (two-stage bootstrap, AddSerilog(), sinks, enrichers) lives in the serilog skill — that skill's AddSerilog()-over-UseSerilog() guidance is canonical.
2. OpenTelemetry for distributed tracing — Traces connect requests across services; metrics track system health over time. Full setup lives in the opentelemetry skill.
3. Health checks for operational readiness — Every service exposes /health endpoints for load balancers and orchestrators. Liveness and readiness are separate questions and separate endpoints.
4. Correlation IDs for request tracing — Every request gets a unique ID that flows through all log entries and downstream service calls, so one user complaint maps to one filtered log stream.

Patterns

How the Pieces Fit Together

Wire logging first (you need logs to debug the rest), then health checks, then tracing.

Correlation IDs

// Middleware to set correlation ID
public class CorrelationIdMiddleware(RequestDelegate next)
{
    private const string CorrelationIdHeader = "X-Correlation-Id";

    public async Task InvokeAsync(HttpContext context)
    {
        var correlationId = context.Request.Headers[CorrelationIdHeader].FirstOrDefault()
            ?? Guid.NewGuid().ToString();

        context.Items["CorrelationId"] = correlationId;
        context.Response.Headers[CorrelationIdHeader] = correlationId;

        using (LogContext.PushProperty("CorrelationId", correlationId))
        {
            await next(context);
        }
    }
}

// Program.cs — register early so every downstream log carries the ID
app.UseMiddleware<CorrelationIdMiddleware>();

Why middleware: pushing the property once at the pipeline edge attaches it to every log event in the request scope — no per-call-site plumbing. Propagate the same header on outgoing HttpClient calls via a DelegatingHandler (see the httpclient-factory skill).

Health Checks

// Program.cs
builder.Services.AddHealthChecks()
    .AddNpgSql(builder.Configuration.GetConnectionString("Default")!,
        name: "database", tags: ["ready"])
    .AddRedis(builder.Configuration.GetConnectionString("Redis")!,
        name: "redis", tags: ["ready"])
    .AddRabbitMQ(builder.Configuration.GetConnectionString("RabbitMq")!,
        name: "rabbitmq", tags: ["ready"]);

// Map endpoints
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = _ => false // No dependency checks — just "am I running?"
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready")
});

Why two endpoints: liveness failing means "restart me"; readiness failing means "stop sending traffic". Conflating them makes a slow database restart your app in a loop.

Log-Level Strategy

Why Warning as the production default: Information-level request noise at scale costs real money in log storage and drowns the signals. Keep Information for genuine business events via namespace overrides (see the serilog skill's MinimumLevel.Override pattern).

Anti-patterns

Don't Log Sensitive Data

// BAD — logging credentials
logger.LogInformation("User logged in: {Email} with password {Password}", email, password);

// GOOD — log identifiers, never secrets or PII at Information level
logger.LogInformation("User {UserId} logged in", userId);

Don't Skip Health Check Tags

// BAD — all checks run for liveness AND readiness
app.MapHealthChecks("/health");

// GOOD — separate liveness (am I running?) from readiness (can I serve traffic?)
app.MapHealthChecks("/health/live", new() { Predicate = _ => false });
app.MapHealthChecks("/health/ready", new() { Predicate = c => c.Tags.Contains("ready") });

Don't Re-Implement What the Owning Skill Provides

// BAD — hand-rolling Serilog bootstrap here from memory
builder.Host.UseSerilog(...);  // legacy API — the serilog skill forbids this

// GOOD — load the serilog skill and use its two-stage AddSerilog() bootstrap
builder.Services.AddSerilog((services, lc) => lc.ReadFrom.Configuration(builder.Configuration)...);

Decision Guide