L4Core4 min read

Observability: logs, metrics, traces

A trace tells you which of eleven services was slow; a log tells you why. Being fluent here turns "it failed sometimes" into a specific, fixable claim — and it is the same skill L7 needs for reading agent behaviour.

When a test fails against a system made of a dozen microservices, "it broke" is not an answer anyone can act on. Observability is the practice of instrumenting a system so that anyone, tester included, can ask an arbitrary question about its current behavior and get a real answer, rather than only the questions someone anticipated when they wrote the original logging statements.

The three pillars, and why none of them is enough alone

Logs are discrete, timestamped events: a specific request came in, a specific error was thrown. They are detailed but hard to correlate across services, since each service writes its own logs independently, with no shared thread connecting them.

Metrics are aggregated numbers over time: request rate, error rate, latency percentiles. They are cheap to store and great for dashboards and alerting, but they tell you something is wrong without telling you which specific request or user was affected.

Traces follow a single request across every service it touches, showing exactly where time was spent and where a failure occurred in a multi-service call chain. A trace is what turns "checkout is slow" into "checkout is slow because the inventory service is making three sequential calls to a cache that should be one call in parallel."

What OpenTelemetry actually standardizes

Before OpenTelemetry, every observability vendor had its own instrumentation library and its own data format, which meant switching vendors meant re-instrumenting your entire codebase. OpenTelemetry (often shortened to OTel) is a vendor-neutral standard for generating and exporting logs, metrics, and traces, so the instrumentation code stays the same regardless of which backend (Datadog, Honeycomb, Grafana, or others) receives the data.

For QA, this matters because it means test environments can use the same instrumentation as production. A trace captured during an automated test run looks structurally identical to one captured from a real user, which makes it possible to compare test behavior against production behavior directly instead of maintaining two separate mental models.

otel-span-example.ts

import { trace } from '@opentelemetry/api'

const tracer = trace.getTracer('checkout-service')

async function processCheckout(orderId: string) {
  return tracer.startActiveSpan('processCheckout', async (span) => {
    span.setAttribute('order.id', orderId)

    try {
      const result = await chargePayment(orderId)
      span.setStatus({ code: 1 })
      return result
    } catch (err) {
      span.recordException(err as Error)
      span.setStatus({ code: 2, message: 'checkout failed' })
      throw err
    } finally {
      span.end()
    }
  })
}

A worked example: chasing an intermittent failure with traces

A healthtech scheduling platform had a test that failed intermittently in CI, roughly one run in twenty, with no clear pattern. Logs alone showed a timeout but not why. Once the team added OpenTelemetry tracing to the test environment, the trace for a failing run showed the actual cause: a database connection pool exhaustion that only occurred when two specific test suites ran in parallel and both hit the same service.

The trace made the root cause visible in one view instead of requiring someone to manually correlate timestamps across three separate log files. The fix, increasing the connection pool size for the test environment and adding a synthetic check to catch pool exhaustion under load, took an afternoon once the actual cause was clear. It had resisted three prior debugging attempts based on logs alone.

Instrumenting for test observability, not just production observability

Most teams only think about observability for production, but test environments benefit from the same instrumentation, especially for flaky test triage. A flaky test with tracing attached tells you whether the flakiness comes from your code, a shared test database, or test ordering, instead of leaving you to guess. This connects directly to SLOs and error budgets, since the SLIs you track in production are usually measurable in a test environment the same way, using the same OpenTelemetry instrumentation.

Add trace context propagation to your test harness so a single test run's requests link into one trace.
Export test-environment telemetry to a separate destination from production, so dashboards do not mix real users and test traffic.
Alert on test-suite-level metrics (flake rate, average run time) the same way you would alert on a production SLI.

Getting started without boiling the ocean

Full OpenTelemetry adoption across a large codebase is a multi-quarter project if you try to instrument everything at once. Start with the one service most involved in your worst debugging sessions, add tracing there first, and expand outward as the value becomes obvious to the rest of the team.

FAQ

Questions people ask

Do we need all three of logs, metrics, and traces to call ourselves observable?

Not on day one. Most teams already have logs and metrics; traces are usually the missing piece, and adding them first closes the biggest gap for anyone debugging multi-service systems.

Is OpenTelemetry difficult to adopt in an existing codebase?

Many frameworks have auto-instrumentation libraries that capture common operations (HTTP calls, database queries) with minimal code changes. Custom spans for business logic, like the checkout example, take more deliberate work.

How does tracing help with flaky tests specifically?

A trace shows you the actual sequence and timing of operations during a failing test run, which turns "it fails sometimes" into a concrete, reproducible explanation, such as a race condition or a shared resource contention issue.

Does adding tracing slow down our test suite?

The overhead is usually small, in the low single-digit percent range, and most teams find the debugging time saved far outweighs the minor performance cost during test runs.