When a service misbehaves, the first thing people reach for is their telemetry: graphs, logs, traces. Far fewer people ask whether that telemetry system is itself in good order. This checklist is meant for a quarterly audit of exactly that. It treats ObservabilityOps as the discipline of operating the observability layer, a reading described in the Ops Atlas entry for ObservabilityOps, and turns it into plain yes-or-no questions. Print it, assign an owner to each section, and record the evidence next to every answer.
Why audit the watching layer at all
Telemetry infrastructure fails in quiet ways. A collector runs out of memory and restarts; a label sneaks in and doubles storage; a proxy drops a header and traces split in half. None of these trips a conventional alarm, because the alarms depend on the very data that has gone missing. A scheduled audit is the cheapest way to catch drift, and it also gives newcomers a map of how the pieces fit together. Teams that skip it usually discover the gaps in the middle of an outage, at which point the cost of finding out is highest.
Running the audit step by step
Take the sections one at a time, top to bottom. Against every line, mark one of three states: confirmed with evidence, partly in place, or absent. Anything absent or partial becomes a dated action. Resist the urge to score yourself generously; an item counts as confirmed only if you can point to a configuration file, a dashboard or a test result that proves it. A reasonable cadence is one section per week over a month, which keeps the effort small enough that it actually gets done.
Section A: ownership and scope
- A named person or group is accountable for the health of the collection and storage layer, and their name is visible to every engineering team.
- The scope is written down: instrumentation standards, pipelines, storage back ends, dashboard and alert conventions, retention and cost. Items such as reliability targets and incident roles are explicitly assigned elsewhere.
- An inventory lists every producer of telemetry and every place it lands.
- The owner sets rules but does not own other teams’ alerts; each service team decides what pages its own on-call.
The Atlas draws the boundary carefully. Setting reliability targets belongs to site reliability engineering, security telemetry belongs with a security data function, and the cloud bill belongs with financial operations. Your audit should confirm that nobody is quietly doing those jobs under the wrong banner.
Section B: instrumentation and context
- One instrumentation standard is documented, and new services use it by default. The Atlas points to OpenTelemetry as the usual choice.
- Trace context uses the W3C defaults, the traceparent header together with baggage, across every hop.
- Known break points are listed and tested: message queues (where span links are the usual remedy), proxies that strip headers, and any service still emitting an older propagation format alongside the newer one.
- Baggage keys are allow-listed, and values are stripped before requests leave your network so that internal details do not leak to outside APIs.
- A fresh trace starts at the public edge instead of trusting inbound headers from strangers.
- The share of telemetry arriving with an unknown service name is tracked and sits close to zero.
Section C: logs
- Logs are structured, with trace identifier, span identifier and severity carried as top-level fields instead of buried in a message string.
- Multi-line stack traces are recombined into single records before storage.
- Filters that depend on severity handle the value for unparsed or unknown records explicitly. In the OpenTelemetry data model an unset severity number is zero, so a rule such as “drop anything below warning” can silently discard records you never classified.
- Stream labels in the log store stay low in variety; the Atlas cites Loki guidance that they should ideally be limited to tens of values.
- Sensitive fields are redacted at the gateway, not at each application.
Section D: metrics and cardinality
- No unbounded values, such as user identifiers or session tokens, appear as metric labels.
- The Atlas rule of thumb is applied: keep the variety of any one metric’s labels under about ten, and investigate anything past one hundred.
- Scrape-time limits (sample and label limits) are configured, and the team understands that breaching them can fail the whole scrape and not just trim the excess.
- A pre-release check estimates series counts so a change cannot silently multiply storage.
- Span-derived metrics do not carry a per-instance identifier. The Atlas flags this trap: a collector instance label creates new series on every restart, and the fix is aggregation that removes the label, not dropping labels at scrape or raising the sample limit.
Section E: sampling and retention
- A sampling policy exists and states what is kept: errors, slow requests and a fraction of ordinary traffic.
- Rates are derived from all spans before any are discarded, so the statistics are not skewed.
- If decisions are made after a whole trace has been assembled, every span of a trace is routed to the same instance, for example by trace identifier, and someone has volunteered to operate that stateful component.
- Retention is tiered into hot, warm and archive, with the periods justified by who actually queries each tier.
- Disk is forecast with the simple product of retention seconds, samples per second and bytes per sample, and the forecast is compared with reality each quarter.
Section F: alerts and meta-monitoring
- Pages are tied to symptoms users can feel.
- A heartbeat alert fires continuously and an outside probe complains if it stops, proving the alert path itself is alive.
- One end-to-end test sends a synthetic event through the full path and measures how long it takes to appear. The Atlas suggests a freshness figure at the 95th percentile.
- Signals unused for ninety days are reviewed for removal, and the unused-signal ratio is reported.
- Page actionability is measured. In the Atlas worked example, only 15 of 42 pages deserved action, which is 35.7 percent; a figure like that is easier to improve than a vague sense of fatigue.
Section G: cost and governance
- A monthly unit-cost figure exists, for example total spend divided by requests served.
- Each team can see its own share of ingestion and storage, even if nobody is invoiced.
- Configuration lives in version control and is tested before it ships.
- Licence terms of self-hosted components are known; the Atlas notes that several popular back ends moved to AGPL terms, which can matter to legal reviewers.
- The claim “self-hosting is free” has been replaced with a real estimate of upgrade and on-call effort.
Scoring and next steps
Count confirmed items per section. Fewer than half confirmed in a section suggests that area needs a dedicated sprint. The Atlas sketches a three-stage plan over 30, 60 and 90 days that suits the findings of this audit. In the first month, name the owner, take the inventory, enable the heartbeat with an outside probe and fix the three worst labels; in the second, publish the standards, enforce limits and redaction at the gateway and run the first alert review; in the third, set platform-level targets, move configuration into version control, prune, and produce the first unit-cost report.
If you need a way to explain the effort to non-technical stakeholders, the guide on a services roadmap for new clients shows how a staged plan can be presented, and the piece on structured digital growth planning offers a template for tracking quarterly goals. Both help when you need to show leadership that a dull-sounding audit is tied to a business outcome.
Typical findings
First-time audits tend to surface the same handful of gaps: no heartbeat on the alert path, one or two high-variety labels that nobody remembers adding, retention left at whatever the installer chose, and an owner who exists on paper but has no time allocated. None of these is embarrassing. They are the normal residue of a system that grew faster than its governance, and each has a small, concrete fix.
Evidence to keep
For each confirmed item, keep a screenshot, a configuration excerpt or a test log with a date. Auditors, new hires and your future self will all benefit. When the next quarter arrives, repeat the audit, compare the counts and look for items that slipped back. Regression is common, particularly after a team reorganisation or a tooling migration, and catching it early is the point of the exercise.

