Observability · how we see inside our apps

Observability · the obs tool

One tool, obs, runs every check

Cheap checks run on every pull request in seconds with no containers. Data checks run only when observability/ changes, on a small local stack. Two checks only comment: the dashboard look and the AI review.

  • obs is a tool in AtomiCloud/nix-registry, not a graph node. DECIDED · A193 tab 1 #17
  • None of obs is built yet. Every message below is the planned wording. DECIDED · A193 tab 2 NOT TESTED · not built

Plant a mistake

Pick a mistake. See which check catches it, when, and the exact message.

Your mistake

What catches it

blocks Step 1 · contract → code

obs gen --check

When: commit + every PR, ~1 s

signals.g.ts is stale vs contract.yaml (sha 9f2c… ≠ 41ab…); run obs gen

DECIDED · A193 tab 2

The steps at a glance

The live checks start the small local stack inside the pull request's CI job, only when observability/** changes. DECIDED · A193 tab 2

StepCommandWhenTime
1 contract → codeobs gen --check, obs config --checkcommit + every PR~1 s
2 code uses portslint rule per languagecommit + every PRinside lint
3 seed + incidentsobs seed --checkevery PR~1 s
4 queries workobs query --static, then --livestatic: every PR; live: observability/**~2 s; live ~46 s + ~30 s
5 alerts fireobs alertsobservability/**~15 s on the same stack
6 dashboards look goodobs look (comments only)dashboards change~90 s + 1 AI call each

All rows: DECIDED · A193 tab 2. Stack start time: TESTED · Q15.

Blocks, or only comments

  • Blocks the merge: steps 1–5. DECIDED · A193 tab 2
  • Only comments: obs look (step 6) and obs review (AI). DECIDED · A193 tab 1 #11
  • Runs outside the pull request: obs assert in the e2e job, obs audit daily against production, obs review weekly. DECIDED · A193 tab 2
  • Example comment: 3/5 incident visible: the spike is in panel 6 at the bottom; move "Charge errors by provider" to the top row. DECIDED · A193 tab 2

Extra checks around the 6 steps

CheckWhat it provesWhenTime
obs assert --run <id>the real app sent every name, linkede2e job~10 s
obs scrapean upstream chart really exposes its metricsPrimordial e2e~5 s
obs auditproduction still matches the contractdaily, read-only~20 s
obs reviewsteps 1 and 3 make sense (AI)e2e + weekly1–3 AI calls
obs sandbox upsee it yourself: seeds and opens Grafanaby hand~46 s

All rows: DECIDED · A193 tab 2.

What the cheap query check proves (no containers)

  • Every name in a query is in the contract or the auto: list. DECIDED · A193 tab 2
  • The stored spelling is right: _total, unit suffix, durian_cloud_event. DECIDED · A193 tab 2
  • No banned label in by (…) or in filters. Example: user_id. DECIDED · A193 tab 2
  • Alerts never select a landscape; every file has at least one query. DECIDED · A193 tab 2
  • It cannot check label values. A filtered value must appear in the seed or an incident; that is checked live. DECIDED · A193 tab 2

Alerts: how step 5 works

  • Each alert is a small typed file with a JSON schema, so your editor autocompletes it. Example: datasource: "metric" is flagged: not one of [metrics, logs]. DECIDED · A193 tab 1 #14
  • One query source, or an optional second. Example: metrics $A > 1 and logs $B > 10. DECIDED · A193 tab 1 #18
  • Recommended (pick 2): obs sends the rendered rule to real Grafana's POST /api/v1/eval at each minute of the seed window, and applies the for: timer itself. OPEN · A193 pick 2
  • The spike proved the eval API runs VictoriaMetrics + Loki + math + threshold at past times. It has no for:. TESTED · Q16
  • The spike ran one query over a past window, not "every minute for 4 hours". TESTED · Q16, backends/q4-grafana-eval NOT TESTED · full loop
  • Traces: alert on span metrics, not raw TraceQL (experimental for alerts). DECIDED · A193 tab 2
  • In production, the Primordial chart turns each alert file into one Grafana alert; obs uses the same renderer. DECIDED · A193 tab 2

Open pick 2 lives here: Grafana's eval API, our own evaluator, or real-time waits. Options and trade-offs. OPEN · A193 pick 2

Example alert file
# observability/alerts/charge-errors/critical.yaml
title: Charge errors high
queries:
  A: { datasource: metrics, expr: 'sum(rate(checkout_charges_total{outcome="error"}[5m]))' }
  B: { datasource: logs, expr: 'sum(count_over_time({service_name="checkout"} | durian_cloud_event="checkout.charge.failed" [5m]))' }
condition: $A > 1 && $B > 10
for: 10m
summary: Errors at {{ $values.A.Value }}/s

DECIDED · A193 tab 2

Example seed and incident
# observability/seed.yaml (shortened)
length: 4h
random_seed: 42            # same file → same data
traffic: { requests: 3/s, shape: daily_wave }
metrics:
  checkout.charges:
    - { where: { payment.provider: stripe, outcome: success }, share: 0.90 }
    - { where: { payment.provider: stripe, outcome: declined }, share: 0.07 }
incidents: [stripe-outage, checkout-slow]
# observability/incidents/stripe-outage.yaml (shortened)
story: Stripe returns 502 for 40% of charges for 30 minutes.
window: { start: 90m, duration: 30m }
change:
  - { metric: checkout.charges, where: { payment.provider: stripe, outcome: error }, rate: 1.2/s }
expect: { fire: [charge-errors/critical], quiet: [checkout-slow/warning], fire_within: 15m }
  • Example values are hand-written, never copied from queries, so the live check proves something. DECIDED · A193 tab 2

Services with no generator (Primordial charts)

  • Charts like CloudNativePG get a contract with generator: none: upstream names, units and budgets only. DECIDED · A193 tab 2
  • Step 2 is skipped. obs scrape in a local Kubernetes test proves the metrics exist instead. DECIDED · A193 tab 2
  • Example failure: cnpg_pg_replication_lag not exposed (renamed to cnpg_pg_replication_lag_seconds?). DECIDED · A193 tab 2

All pages

  1. 1Startthe whole thing on one screen
  2. 2How it worksone name, end to end
  3. 3Correlationhow the 4 signals link
  4. 4The checkswhat obs catches, and when
  5. 5Controls and costwhat you can turn, what it costs
  6. 6Tested or notwhat the spikes proved, and what they did not
  7. 7Open picks + weak spotswhat to decide, where to push