Observability · how we see inside our apps

Observability · evidence

What is proven, and what is only decided

3 spikes ran 18 questions for real on this machine: 12 yes, 4 partly, 2 no. Every "no" and "partly" became a pick or a fix. Much of the design is decided but has never run, starting with obs itself.

  • Evidence lives in spikes/js, spikes/go-dotnet and spikes/backends, each with a RESULT.md. TESTED · A193 tab 6
  • Flutter's app side was read from source, not run: there is no Flutter SDK on this machine. TESTED · Q17

The tags

  • DECIDED · A193 DECIDED: you ruled on it, or it is in the design record. Not proof it works.
  • TESTED · spike TESTED: a spike ran it for real. Tap a tag to see the evidence.
  • PROPOSED · jaeda PROPOSED: jaeda's call or this site's reading. Say if wrong.
  • OPEN · A193 OPEN: needs your pick.
  • NOT TESTED · — NOT TESTED: decided or written down, never run.

The 18 spike questions

3 spikes (test runs) asked 18 questions, Q1–Q18. Tags like "Q13" point here. TESTED · A193 tab 6

  1. 1 Bun: exemplars on metrics? NO

    OpenTelemetry JS has no exemplar code; no trace id bytes in the payload. TESTED · spikes/js/q1/check-bytes.ts

  2. 2 Bun: trace ids reach logs? YES

    Context survives await, timers and setImmediate; logs carry the ids. TESTED · spikes/js/q2/collector-evidence.txt

  3. 3 Bun: CPU profile → flame graph? YES

    Via node:inspector + an ~80-line converter. @pyroscope/nodejs fails under Bun. TESTED · spikes/js/q3/flame-query-evidence.txt

  4. 4 Bun: sampled or span-linked profiles? PARTLY

    Manual 1-in-N only; the profiler is process-global; you can't search by profile id. TESTED · spikes/js/q4/sampled-requests.ts

  5. 5 Bun: Result → span error? YES

    A 19-line helper: Error status, problem event, error.type. TESTED · spikes/js/q5/collector-evidence.txt

  6. 6 Bun: database spans? PARTLY

    pg yes; ioredis 0 spans; Bun SQL none. TESTED · spikes/js/q6/pg-span-evidence.txt

  7. 7 Go: closed names? YES

    Compile errors outside the generated file; forbidigo or a short analyzer catch signals.Event{}. TESTED · spikes/go-dotnet/q1-closed-names/evidence.txt

  8. 8 Go: span profiles? YES

    A span's flame graph has only that span's frames; a bogus id gives an empty one. TESTED · spikes/go-dotnet/q2-go-span-profiles/evidence.txt

  9. 9 .NET: span profiles? YES

    Stock sdk:8.0 container; span-filtered flame graph; CPU + alloc + gc:cpu. TESTED · spikes/go-dotnet/q3-dotnet-span-profiles/evidence-dotnet.txt

  10. 10 Go/.NET: profiling rate + cost? PARTLY

    No supported rate knob (Go 100 Hz, .NET ~9 ms). Cost: .NET under 1%; Go under ~2% inside noise. TESTED · spikes/go-dotnet/q4-sampling/EVIDENCE.md

  11. 11 Go/.NET: Result → span error? YES

    Small helpers, same shape as Bun. TESTED · spikes/go-dotnet/q5-result-problems/

  12. 12 Go/.NET: database spans? YES

    With caveats: otelpgx 2 spans per query; redisotel old names; .NET Redis errors unmarked. TESTED · spikes/go-dotnet/q6-db/evidence-pgx-error-span.txt

  13. 13 VictoriaMetrics keeps exemplars? NO

    Accepted, then dropped; the maintainer confirms. TESTED · spikes/backends/q1-vm-exemplars/evidence.md

  14. 14 Pyroscope takes OpenTelemetry profiles? YES

    Directly at /v1development/profiles. Alloy can't carry them; pprof works both ways. TESTED · spikes/backends/q2-pyro-otlp/evidence.md

  15. 15 Local stack size + cross-links? YES

    ~630 MiB idle, ready in 46 s; trace ↔ log ↔ profile links work. TESTED · spikes/backends/q3-compose/evidence.md

  16. 16 Grafana tests alert rules at a past time? YES

    POST /api/v1/eval: VictoriaMetrics + Loki + math + threshold, past windows, no for:. TESTED · spikes/backends/q4-grafana-eval/evidence.md

  17. 17 Faro for Flutter? YES

    faro 0.18.0 sends 5 signal kinds with trace context through Alloy → Loki + Tempo. Wire only: no Dart ran. TESTED · spikes/backends/q5-faro-flutter/evidence.md

  18. 18 Profile a % of traffic? PARTLY

    No per-request knob; pod targeting, eBPF sample rate, per-type on/off, server-side probability. TESTED · spikes/backends/q6-profiling-controls/evidence.md

Decided, but never run

  • obs itself. No command exists yet: not gen, not the checks, not look. DECIDED · A193 tab 6 round 1b, 2c NOT TESTED · not built
  • The 4 ports and their in-memory doubles. Designed per language; not written. DECIDED · A193 tab 3 NOT TESTED · not built
  • The for: timer in obs alerts. The eval API was proven; obs's own timer on top was not. TESTED · Q16 NOT TESTED · timer not built
  • Pod share picking. How a pod decides it is "the 1 in 4" is not designed or tested. OPEN · A193 pick 3 NOT TESTED · not tested
  • Metric → traces by shared labels. No spike clicked from a metric to traces. OPEN · A193 pick 1 NOT TESTED · not tested
  • Bun Redis and Bun SQL wrappers. Decided; the spike only showed they are needed. DECIDED · A193 tab 1 #20 NOT TESTED · not built
  • The budget × 4 runtime cap. A View per instrument; not run. DECIDED · A193 tab 4 NOT TESTED · not tested
  • Bun's instrumentServe and instrumentedFetch. Needed because Bun is not Node; not written. DECIDED · A193 tab 4 NOT TESTED · not built
  • The stores on Primordial, in normal mode with object storage. Only single binaries on a laptop ran. DECIDED · A193 tab 1 #13 NOT TESTED · not tested
  • Tempo's span metrics into VictoriaMetrics on Primordial. DECIDED · A193 tab 5 NOT TESTED · not tested
  • A real Flutter app through Faro. Only the receiver side ran. TESTED · Q17, wire NOT TESTED · app side
  • eBPF profiling (pyroscope.ebpf): needs root and the host process list; read from docs. NOT TESTED · A193 tab 5
  • obs look and obs review quality: whether the AI scores and comments help. DECIDED · A193 tab 2 NOT TESTED · not tested
  • LPSM keys as labels in each store. The record says "check". NOT TESTED · A193 tab 5
  • Server-side Dart: same ports, no adapter yet. DECIDED · A193 tab 3 NOT TESTED · later round

jaeda's 13 calls

Decisions jaeda made inside the design. Say if any is wrong. All from PROPOSED · A193 tab 1, "Calls I made".

  1. Names are unique across all kinds. Example: checkout.charges can't be both a counter and an event.
  2. Units are UCUM and durations are seconds. Example: unit: s, not ms.
  3. Go ports take ctx first on every call.
  4. C# ports are named ISignal*, so they don't clash with ILogger or Meter.
  5. tracer.run returns your function's own Result; telemetry errors never change it.
  6. Budget = label sets per metric, per landscape, all pods. The runtime cap is budget × 4 per process.
  7. The base config has profiling off; each landscape turns it on.
  8. Go keeps names closed with generated structs with a hidden field, plus a forbidigo rule. TESTED · Q7
  9. Bun profiling is CPU only, via node:inspector → pprof → Pyroscope. TESTED · Q3
  10. obs maps stored names. Example: checkout.charges → checkout_charges_total.
  11. Span events reuse the Event names. Example: span.event(Event.CheckoutChargeRetried).
  12. The event name is sent twice: as durian.cloud/event and in the OTLP EventName field.
  13. Tail sampling (keep every error trace) stays the platform team's collector job. See weak spot 1: it clashes with app-side sampling. PROPOSED · jaeda

Questions the record does not answer

Good places to push; none of these is decided anywhere. PROPOSED · jaeda, this site

  • How a landscape's config reaches the pod (mounted file or env vars). OPEN · not in A193
  • How long each store keeps data, and what storage costs. OPEN · not in A193
  • How the seed builds "one linked set per request" (trace + logs + metrics + profile) from a file that mostly lists metrics. OPEN · not in A193
  • What happens to today's TraceEmitter code, atomi.* keys and dashboards. The record says nothing breaks because no template uses the libraries yet. DECIDED · A193 tab 3 OPEN · dashboards
  • How errors are kept when the app samples traces (see weak spot 1). OPEN · not in A193

What the spikes changed

  • No exemplars. Metrics link to traces by shared labels → pick 1. TESTED · Q1; Q13
  • Real Grafana can test alerts at past times, so obs need not write its own evaluator → pick 2. TESTED · Q16
  • No profiling rate knob anywhere. The old cpu_hz control is gone → pick 3. TESTED · Q10; Q18
  • Bun's Redis client and Bun SQL have no auto spans → thin wrappers (decided 09 Nov). TESTED · Q6 DECIDED · A193 tab 1 #20
  • Alloy cannot carry OpenTelemetry profiles → pprof through Alloy now (decided 09 Nov). TESTED · Q14 DECIDED · A193 tab 1 #21
  • The dashboard check never blocks, so its fixed checks only report now. Older text said they blocked. DECIDED · A193 tab 1 #11
Versions the spikes used

Bun 1.3.13; OTel JS api 1.10, sdk 3.1, exporters 0.301; Go 1.26, OTel Go 1.47, pyroscope-go 1.4.3, otel-profiling-go 0.7.0; .NET 8.0.427, OTel .NET 1.20.1, Pyroscope .NET 1.6.0; VictoriaMetrics 1.155, Pyroscope 2.4.3, Alloy 1.21.3, Grafana 13.3, Loki 3.7.9, Tempo 3.1.1; faro 0.18.0. TESTED · A193 tab 6

All pages

  1. 1Startthe whole thing on one screen
  2. 2How it worksone name, end to end
  3. 3Correlationhow the 4 signals link
  4. 4The checkswhat obs catches, and when
  5. 5Controls and costwhat you can turn, what it costs
  6. 6Tested or notwhat the spikes proved, and what they did not
  7. 7Open picks + weak spotswhat to decide, where to push