Observability · start here

How we see inside our apps

Every log, metric, span and profile uses a name from one hand-written list. Our own interfaces only accept those names. One tool, obs, checks the names, the queries, the alerts and the dashboards.

The design is settled and 3 real spikes back it: 18 questions, 12 yes, 4 partly, 2 no. TESTED · A193 main; A193 tab 6
3 picks are still open. This site is for you to understand and attack the design first. OPEN · A193 main
Nothing here is built yet. The libraries, obs and the templates come in 4 rounds after the picks. DECIDED · A193 tab 6 "The rounds"
Written from agenda item A193 by jaeda, your assistant. Tap any coloured tag to see its source. PROPOSED · this site

What a developer does: 6 steps

Write the name in contract.yaml, then run obs gen. Example: add checkout.charge.failed; you get Event.CheckoutChargeFailed. DECIDED · A193 tab 1 #9
Use only our ports. A lint rule stops direct OpenTelemetry or logger imports. Example: import from '@opentelemetry/api' fails Biome. DECIDED · A193 tab 2
Write a seed and incidents. Normal made-up traffic, plus bad windows. Example: stripe-outage.yaml. DECIDED · A193 tab 2
Queries work. Every dashboard and alert query uses real names and returns rows on the seed. DECIDED · A193 tab 2
Alerts fire in each incident and stay quiet in normal traffic. DECIDED · A193 tab 2
Dashboards look good. An AI scores screenshots and comments. It never blocks a merge. DECIDED · A193 tab 1 #11

An AI may also read steps 1 and 3 against real traffic and comment (obs review). DECIDED · A193 tab 1 #9

What flows where

Tap a signal to see its road. Every app talks only to Alloy; Alloy forwards each signal to its own store; Grafana reads all four.

Pick a signal

Every signal takes the same first steps: your code → a port → the adapter → Alloy. Then each goes to its own store, and Grafana reads all four. DECIDED · A193 tab 5

Where things stand

21DECIDED

your decisions in A193 tab 1

18TESTED

questions (Q1–Q18) run in 3 spikes: 12 yes, 4 partly, 2 no

13PROPOSED

calls jaeda made; 2 proven by a spike. Say if any is wrong

3OPEN

picks: metric ↔ trace, alert check, profiling share

Sources: DECIDED · A193 tab 1 TESTED · A193 tab 6 OPEN · A193 main

How to critique this

Push hardest on these. Each one has a full entry on the last page.

Sampled traces and "keep every error trace" clash. The app drops 9 in 10 traces at the start, so no collector can keep the errors among them. Log lines may point at traces that were never kept. PROPOSED · jaeda reading NOT TESTED · not tested
Metric → trace is weaker than exemplars. You land on many traces in a time window, not the one behind the spike. TESTED · Q13
Only some pods are profiled. A slow trace you click may have no flame graph. PROPOSED · A193 pick 3
The checks run on made-up data. A query can pass on the seed and still be wrong in production. DECIDED · A193 tab 2
Labels are free-form. A label name typo passes the cheap checks. DECIDED · A193 tab 1 #5
Flutter never ran. Faro was proven on the wire, but no Dart code ran here. TESTED · Q17
We own a lot of Bun glue code. Bun lacks auto spans and a profiler that loads. TESTED · Q3; Q6

Read in this order

All the words, with examples

contract: observability/contract.yaml: a hand-written list of every name a service may send. Example: checkout.charge.failed.
typed name: A constant made from the contract, so a wrong or made-up name does not compile. Example: Event.CheckoutChargeFailed in Bun.
port: Our own interface in each *-interfaces library: Logger, Meter, Tracer, Profiler. Example: logger.error(…).
adapter: The code behind a port that really sends the data. Example: the OpenTelemetry adapter in lib/bun/otel, or Faro for Flutter.
signal: One kind of data an app sends: a log line, a metric, a span or a profile.
span: One timed step of a request. Example: checkout.charge took 180 ms.
trace: All spans of one request, as a tree. They share one trace id.
trace id: A 32-character id every span of one request shares. Example: 4bf92f35…. It rides between services in the traceparent header.
exemplar: A trace id stored on one metric point, so a spike on a graph links to one real trace.
span profile: A flame graph cut down to one span. Example: click slow span checkout.charge and see only its CPU.
flame graph: A picture of "which code was running". A wide bar means more time spent there.
profile: Samples of "which code is running", taken many times a second. Example: 100 samples a second in Go.
LPSM: Landscape, platform, service, module: the 4 keys that say where a signal came from. Example: pichu / payments / checkout / api.
landscape: One running copy of the whole system. Example: pichu (test) and raichu (production-like).
semconv: OpenTelemetry's standard names. Example: http.server.request.duration, error.type.
OpenTelemetry: The open standard (and its libraries) for sending logs, metrics and traces. Short: OTel.
OTLP: OpenTelemetry's wire format. Apps send logs, metrics and spans to Alloy with it.
pprof: Go's standard profile file format. Pyroscope reads it.
Alloy: Grafana's collector: the one front door every app sends to. It forwards each signal to its store.
Loki: The store for logs.
Tempo: The store for traces (spans).
Pyroscope: The store for profiles (flame graphs).
VictoriaMetrics: The store for metrics. We keep it (decided). It drops exemplars.
Grafana: The screen: dashboards, search, alerts. It reads all 4 stores.
Faro: Grafana's SDK for apps on phones. Flutter sends through it.
seed: Made-up normal data, written by obs seed into a local stack. Example: 3 requests a second for 4 hours.
incident: Made-up bad data in a time window, plus the alerts that must fire. Example: Stripe fails 40% of charges for 30 minutes.
series: One distinct set of label values on a metric. Example: {provider="stripe", outcome="error"} is one series.
label budget: The most series one metric may have, per landscape. Example: budget: 30.
span metrics: Metrics that Tempo makes from spans: call counts, errors, durations. Example: traces_spanmetrics_calls_total.
pod share: The part of a service's pods that run the profiler. Example: 0.25 of 8 pods is about 2 pods.
Result: A return value that is either Ok(value) or Err(problem), instead of throwing. Our libraries use it everywhere.
Primordial: The platform that runs our clusters and installs shared tools, like the stores.
Garden: The shared test environment, with the same stores as a laptop.
obs: Our command-line tool that runs every check. It lives in AtomiCloud/nix-registry.
A193: The agenda item that holds this design's full record: 21 decisions, the spike results, the 3 open picks. The newest source; it wins when sources disagree.
jaeda: Your assistant. jaeda wrote the design record and its calls; everything tagged PROPOSED is jaeda's (or this site's) reading.
Q1–Q18: The 18 questions the 3 spikes (test runs on this machine: Bun, Go + .NET, stores) answered. Example: Q13 = "does VictoriaMetrics keep exemplars?" (no).
round: One batch of build work after the picks. There are 4. Example: round 1 = the new *-interfaces libraries and obs v1.
nix-registry: AtomiCloud/nix-registry: the repo that releases shared tools, like obs. You approve its pull requests.
template: A starter repo branch a new service is made from. Example: bun-base-w-observability.
head sampling: The app decides at the start of a request whether to keep its trace. Example: ratio 0.1 keeps about 1 in 10.
tail sampling: A collector keeps or drops a whole trace after seeing all of it. Example: keep every trace that has an error. It only works if the app sent the spans.
View: An OpenTelemetry setting per instrument. Here it sets histogram buckets and caps label sets.
normal mode: A store split into several parts with its data in object storage, as on Primordial. The opposite of one process on local disk.
metrics generator: The part of Tempo that turns spans into span metrics.
Gigapipe: The old plan's log and trace store on ClickHouse, imitating Loki and Tempo. Dropped.
eBPF: A Linux kernel feature. An eBPF profiler samples CPU for every process on a node without code changes; it needs root.
problem: The standard error shape (RFC 9457): type, title, status, detail, instance. Example: { status: 403, type: …/suspended }.
Biome: The linter for Bun and TypeScript code.
forbidigo: A Go lint rule that bans chosen code patterns. Example: `^signals\.Event<!--
=== page <id>: <file> | <nav label> | <nav sub-label> starts a page
contract: observability/contract.yaml: a hand-written list of every name a service may send. Example: checkout.charge.failed.
typed name: A constant made from the contract, so a wrong or made-up name does not compile. Example: Event.CheckoutChargeFailed in Bun.
port: Our own interface in each *-interfaces library: Logger, Meter, Tracer, Profiler. Example: logger.error(…).
adapter: The code behind a port that really sends the data. Example: the OpenTelemetry adapter in lib/bun/otel, or Faro for Flutter.
signal: One kind of data an app sends: a log line, a metric, a span or a profile.
span: One timed step of a request. Example: checkout.charge took 180 ms.
trace: All spans of one request, as a tree. They share one trace id.
trace id: A 32-character id every span of one request shares. Example: 4bf92f35…. It rides between services in the traceparent header.
exemplar: A trace id stored on one metric point, so a spike on a graph links to one real trace.
span profile: A flame graph cut down to one span. Example: click slow span checkout.charge and see only its CPU.
flame graph: A picture of "which code was running". A wide bar means more time spent there.
profile: Samples of "which code is running", taken many times a second. Example: 100 samples a second in Go.
LPSM: Landscape, platform, service, module: the 4 keys that say where a signal came from. Example: pichu / payments / checkout / api.
landscape: One running copy of the whole system. Example: pichu (test) and raichu (production-like).
semconv: OpenTelemetry's standard names. Example: http.server.request.duration, error.type.
OpenTelemetry: The open standard (and its libraries) for sending logs, metrics and traces. Short: OTel.
OTLP: OpenTelemetry's wire format. Apps send logs, metrics and spans to Alloy with it.
pprof: Go's standard profile file format. Pyroscope reads it.
Alloy: Grafana's collector: the one front door every app sends to. It forwards each signal to its store.
Loki: The store for logs.
Tempo: The store for traces (spans).
Pyroscope: The store for profiles (flame graphs).
VictoriaMetrics: The store for metrics. We keep it (decided). It drops exemplars.
Grafana: The screen: dashboards, search, alerts. It reads all 4 stores.
Faro: Grafana's SDK for apps on phones. Flutter sends through it.
seed: Made-up normal data, written by obs seed into a local stack. Example: 3 requests a second for 4 hours.
incident: Made-up bad data in a time window, plus the alerts that must fire. Example: Stripe fails 40% of charges for 30 minutes.
series: One distinct set of label values on a metric. Example: {provider="stripe", outcome="error"} is one series.
label budget: The most series one metric may have, per landscape. Example: budget: 30.
span metrics: Metrics that Tempo makes from spans: call counts, errors, durations. Example: traces_spanmetrics_calls_total.
pod share: The part of a service's pods that run the profiler. Example: 0.25 of 8 pods is about 2 pods.
Result: A return value that is either Ok(value) or Err(problem), instead of throwing. Our libraries use it everywhere.
Primordial: The platform that runs our clusters and installs shared tools, like the stores.
Garden: The shared test environment, with the same stores as a laptop.