Observability · open picks and weak spots
3 picks left, and where to push
Three picks are open: how metrics link to traces, how alerts are checked, and how to profile only part of the traffic. jaeda recommends the first option in each. Below them are the places a sharp critic should push.
- Picks 4, 5 and 6 were decided on 09 Nov (key spelling, Bun wrappers, profile transport). DECIDED · A193 tab 1 #19–21
- No form here. Reply in chat or on agenda item A193 in this shape:
1A 2A 3A, plus any weak spot you want fixed first. PROPOSED · this site
Pick 1 · Metric ↔ trace links
Why it is open: VictoriaMetrics drops exemplars, and Bun has none. TESTED · Q1; Q13
How should a metric link to its traces?
A. Keep VictoriaMetrics; link by shared labels jaeda recommends
You get
- Works the same in every language, Bun included.
- No store change. VictoriaMetrics answered a 10k-series query in ~28 ms vs Mimir's ~65 ms. TESTED · A193 tab 5, Mimir test
- Tempo's trace → metrics link still works through span metrics. DECIDED · A193 tab 5
You lose
- No "click one spike → the exact trace". You get a list of likely traces.
- Only works if metrics and spans use the same label names. obs checks this on the seed only.
OPEN · A193 pick 1
B. Move metrics to Mimir to get exemplars
You get
- Click a graph spike → the exact trace, for Go and .NET.
- Mimir stored 50,000 exemplars in the test. TESTED · A193 tab 5, Mimir test
You lose
- Bun gets nothing: OpenTelemetry JS sends no exemplars. TESTED · Q1
- One more store change; ~2× slower query and ~20 s vs ~4 s start in the small test.
- Exemplars only on counters, not gauges, in that test.
OPEN · A193 pick 1
C. Keep VictoriaMetrics + a small Mimir for span-metric exemplars
You get
- Exemplar links on span metrics (the request rate and errors per span).
- Tempo makes span metrics from spans, so this may cover Bun too: jaeda's reading, not tested. PROPOSED · jaeda
You lose
- Two metric stores to run and keep in step.
- Not tested at all.
OPEN · A193 pick 1 NOT TESTED · not tested
| What waits on it | The local stack in the templates (round 3b) and the stores Primordial installs (round 4). PROPOSED · jaeda reading of A193 tab 6 |
| Can it change later? | A and C can move to B later: it is a store swap plus re-pointing Grafana; app code does not change. Money and ops time for Mimir: not measured. PROPOSED · jaeda |
jaeda recommends A. Exemplars would help only Go and .NET, and the link they give is to one example trace, which a filtered trace list usually also shows. PROPOSED · jaeda, A193 pick 1
Pick 2 · How obs alerts checks that alerts fire
Why it is open: real Grafana can test a rule at a past time, but has no for: timer in that API. TESTED · Q16
How does obs alerts prove an alert fires in an incident and stays quiet otherwise?
A. Real Grafana's eval API; obs adds the for: timer jaeda recommends
You get
- The exact engine production uses. TESTED · Q16
- Seconds, not minutes: obs asks "would it fire at 10:00, 10:01 …?" DECIDED · a193-qa2
- Little code to own: only the for: timer.
You lose
- +~270 MiB for Grafana in step 5.
- Grafana's own UI uses this API; it may change, so Grafana's version must be pinned.
- Our for: timer is a small copy of Grafana logic and could differ (pending state, no-data).
OPEN · A193 pick 2
B. obs's own evaluator, checked once against Grafana (old plan)
You get
- No Grafana in step 5; fast and small.
You lose
- A large copy of Grafana's rule logic to write and keep up.
- High risk of a false green when our copy drifts.
OPEN · A193 pick 2
C. Real Grafana rules, waiting in real time
You get
- Simplest; no code of our own.
You lose
- A rule with for: 10m makes the check wait 10 minutes.
- Slow and flaky on timing.
OPEN · A193 pick 2
| What waits on it | obs v2 obs alerts (round 2c: "Grafana eval if pick 2 holds"). DECIDED · A193 tab 6 |
| Can it change later? | Inside obs only: switching later changes one obs command, not any service. PROPOSED · jaeda |
jaeda recommends A. It is the only option that uses production's engine without real-time waits. PROPOSED · jaeda, a193-qa2
Pick 3 · Profiling only part of the traffic
Why it is open: no runtime lets you set a profiling rate or pick requests (except Bun, by hand). TESTED · Q10; Q18
How do we profile only part of the traffic?
A. Profile 1 in N pods, plus Pyroscope server-side sampling jaeda recommends
You get
- Works in Bun, Go and .NET. Span profiles keep working on the profiled pods.
- Config
pod_share(e.g. 0.25 → ~2 of 8 pods), capped by the contract. DECIDED · A193 tab 2 - Pyroscope can drop more on the server if volume grows. TESTED · Q18, config check
You lose
- Requests on other pods have no flame graph: at 0.25, about 3 in 4 slow traces have none.
- A service with 1 or 2 pods is all or nothing.
- How a pod knows it is picked is not designed yet.
OPEN · A193 pick 3
B. Profile every pod all the time
You get
- Every slow trace has a flame graph (Go, .NET).
- Simplest config.
You lose
- Cost on every pod: under 1–2% CPU measured, under 50 MB memory (Grafana's numbers). TESTED · Q10
- Storage and network at scale: not measured.
OPEN · A193 pick 3
C. Per-request sampling inside the app
You get
- True "% of traffic".
You lose
- Only possible in Bun, by hand: its profiler is process-global, one at a time. TESTED · Q4
- Go and .NET cannot do it.
OPEN · A193 pick 3
D. A node eBPF profiler instead (not offered in A193)
You get
- No code change; one agent per node profiles every process. Its sample rate is settable (default 19 stacks a second). DECIDED · A193 tab 5 TESTED · Q18, docs and config
You lose
- CPU only, and weak span links. DECIDED · A193 tab 5
- Needs root and the host process list on every node. DECIDED · A193 tab 5
- Never run here. This is probably why A193 did not offer it. PROPOSED · jaeda NOT TESTED · not tested
PROPOSED · this site
| What waits on it | The otel. config schema in obs v1 (round 1b) and the profiler in each otel library (round 2a). PROPOSED · jaeda reading of A193 tab 6 |
| Can it change later? | A and B are the same code with a different pod_share (1.0 = every pod), so moving between them is a config change. C needs code. PROPOSED · jaeda |
jaeda recommends A. It is the only option that works in all three languages. Since measured CPU cost is small, B is a fair choice if you care more about always having a flame graph. PROPOSED · jaeda; this site
Suggested 4th question: sampling and error traces
Not on the agenda yet. jaeda's reading from the record, not tested. PROPOSED · jaeda NOT TESTED · not tested
- The record has the app sample traces (raichu 0.1) and says a collector keeps every error trace. DECIDED · A193 tab 2; tab 1, jaeda call 13
- A collector only sees spans the app sent, so it cannot keep errors from traces the app dropped.
- Options: (a) apps send every span and the collector tail-samples, keeping errors (more network to the collector); (b) keep head sampling and accept lost error traces and dead log links; (c) raise the ratio where errors matter.
- Settle before round 2 (the otel libraries set the sampler). PROPOSED · jaeda
Weak spots: where to push
jaeda's honest list, most serious first. Each says what we claim, why it might be wrong, what would settle it, and by when. PROPOSED · jaeda, this site
1 Sampled traces: "keep every error trace" cannot work, and log links go dead high
Settle before: round 2a (the otel libraries set the sampler)
We claim: Every log line carries a trace id. DECIDED · A193 tab 2
Why it might be wrong: jaeda's reading: with raichu's ratio 0.1, a log line can carry the id of a trace that was never kept, so about 9 in 10 log → trace clicks may find nothing. Keeping error traces is left to a collector, which never sees spans the app dropped. PROPOSED · A193 tab 1, jaeda call 13
What would settle it: Check in the Bun/Go/.NET SDKs whether logs carry unsampled trace ids; decide whether
obs assertshould measure the dead-link rate.PROPOSED · jaeda NOT TESTED · not tested
2 Metric → trace is only a filtered list high
Settle before: you answer pick 1
We claim: Shared labels replace exemplars. OPEN · A193 pick 1
Why it might be wrong: When many requests look alike, the list does not point at the request behind the spike. No spike ever clicked from a metric to traces.
What would settle it: Run the click on the local stack with the seed: does the list contain the incident's traces near the top?
NOT TESTED · not tested
3 Checks prove the seed, not production high
Settle before: round 2c (obs v2)
We claim: Queries and alerts are proven on made-up data. DECIDED · A193 tab 2
Why it might be wrong: A query can return rows on hand-written seed values and nothing in production. The only production check that comments is the AI review, and it never blocks.
What would settle it: Make
obs audit(daily) fail loudly when a dashboard panel has had no data for 7 days in a landscape.PROPOSED · jaeda
4 The record has loose ends high
Settle before: round 1 starts (the generator needs one spelling)
We claim: Tab 1 is the full record. DECIDED · A193 context
Why it might be wrong: Key spelling differs (dots vs slash); the count says 18 decisions but lists 21; rollout still says "6 picks"; "under 50 MB" is listed as proven but is Grafana's number. Older corpus goals still say Gigapipe and
atomi..* What would settle it: Fix A193 and the corpus goal files before round 1 starts, so the generator has one spelling.
OPEN · fix the record
5 Some pods profiled means some traces have no flame graph medium
Settle before: you answer pick 3
We claim: Profile 1 in N pods. OPEN · A193 pick 3
Why it might be wrong: At
pod_share0.25 most slow traces you click have no profile. Small services are all or nothing. How a pod is chosen is not designed.What would settle it: Decide the picking rule (hash of pod name?) and show "not profiled" on the span instead of an empty graph.
NOT TESTED · not designed
6 The label budget is not enforced where it is defined medium
Settle before: round 2c (obs v2)
We claim: Budget = label sets per metric, per landscape, across all pods. PROPOSED · A193 tab 1, jaeda call 6
Why it might be wrong: Each process only caps itself at budget × 4, so 10 pods can make 40 × budget series together.
obs assertchecks the budget on e2e traffic, which is small. A histogram also multiplies series by its buckets.What would settle it: Have
obs auditcount real series per metric per landscape daily and fail over budget.PROPOSED · jaeda
7 Alert test uses an API Grafana's UI owns medium
Settle before: you answer pick 2
We claim:
POST /+ our ownapi/ v1/ eval for:timer. OPEN · A193 pick 2Why it might be wrong: It is the API behind Grafana's UI, not a promised public contract; an upgrade may change it. Our
for:timer is a small evaluator after all, and could differ on pending, no-data or error states. The spike ran one query, not a 4-hour step loop.What would settle it: Pin Grafana; add one integration test where real Grafana fires a known rule in real time and compare with obs.
TESTED · Q16 NOT TESTED · loop
8 Flutter never ran medium
Settle before: round 2b (lib/dart/faro)
We claim: Faro sends logs, measurements, errors and traces. TESTED · Q17
Why it might be wrong: Only the receiving side ran; the app side was read from source (no Flutter SDK here). Grafana calls the SDK "private preview". Measurements land in Loki, not as metrics, and field names differ from the record.
What would settle it: Run one real Flutter app through Faro → Alloy → Loki + Tempo before round 2b.
NOT TESTED · app side
9 Bun is the weakest language medium
Settle before: round 2a (lib/bun/otel)
We claim: Bun gets the same 4 signals. DECIDED · A193 tab 1 #8
Why it might be wrong: Bun needs our own HTTP, Redis and SQL span wrappers, our own pprof converter, and hand-built providers (NodeSDK sent 0 spans). It gets CPU profiles only, no span profiles, no exemplars, and its profiling cost is not measured.
What would settle it: Measure Bun profiling cost; budget the upkeep of the glue code; re-check each Bun release.
TESTED · Q3; Q4; Q6
10 Free-form labels hide typos medium
Settle before: round 1b (obs v1)
We claim: Labels are free-form with guards. DECIDED · A193 tab 1 #5
Why it might be wrong:
provder="stripe"passes the static check. The live check only catches it if the seed uses the right name; nothing ties seed label names to code.What would settle it: Have
obs assertcompare label names seen in the e2e run with the label names used in queries.PROPOSED · this site
11 "The same locally and in production" is the software, not the setup medium
Settle before: round 4 ends (Primordial)
We claim: Same stores on a laptop, in Garden and on Primordial. DECIDED · A193 tab 1 #13
Why it might be wrong: Locally each store is one process on disk; on Primordial they run in normal mode on object storage. Only the laptop setup ran. Tempo's span-metrics generator was not in the spike config.
What would settle it: One Primordial test cluster with all four stores and the Grafana links before round 4 ends.
NOT TESTED · not tested
12 Keeping VictoriaMetrics rests on a small test low
Settle before: pick 1, only if speed decides it
We claim: VictoriaMetrics is faster than Mimir. TESTED · A193 tab 5
Why it might be wrong: 10,000 series, about 15 pushes, an older VictoriaMetrics (1.102.1). The results exist only in the write-up, not in a results file.
What would settle it: If pick 1 hinges on speed, rerun with the current version and keep the output.
TESTED · a193-mimir.md
13 A big rollout with one approver low
Settle before: round 1 starts
We claim: 4 rounds: interfaces, otel, templates, Primordial. DECIDED · A193 tab 6
Why it might be wrong: 4 interfaces majors, 3 otel majors, a new Faro library, obs v1 and v2, 8 template nodes and Primordial. Every
nix-registrypull request needs you; agents never merge them.What would settle it: Agree up front which obs releases you review closely and which you skim.
DECIDED · A193 tab 6
What happens after you answer
- Round 1: 4 new
*-interfacesmajors and obs v1, side by side. DECIDED · A193 tab 6 - Round 2: 3 new
*-otelmajors, the newlib/, and obs v2 (seed, live, alerts, look). DECIDED · A193 tab 6dart/ faro - Round 3: the base templates get the lint rule; the 4
-w-observabilitytemplates get the wiring. DECIDED · A193 tab 6 - Round 4: Primordial installs Loki, Tempo, Pyroscope and the Grafana links. DECIDED · A193 tab 6
You approve every nix-registry pull request (agents never merge them), each library major and each template round. DECIDED · A193 tab 6