Observability · correlation
How one click gets you to the next signal
Logs, spans and profiles share the trace id, so Grafana jumps between them exactly. Metrics carry no trace id here, so a metric links to traces only by shared labels and time: you get a list of likely traces, not the exact one.
- Trace ↔ log ↔ profile links were proven on a real local stack. TESTED · Q15
- VictoriaMetrics drops exemplars, and Bun's OpenTelemetry has none. TESTED · Q1; Q13
- How to link metrics to traces is open pick 1. OPEN · A193 pick 1
What links to what
Tap a starting signal. The diagram shows each jump, what it rides on, and how strong it is.
- exact one id, one target
- good same service and name
- rough labels + time window
Metric → Trace
You click: a spike on a metric graph
You get: a list of traces with the same labels in that time window, not the one trace
OPEN · A193 pick 1 NOT TESTED · not tested
Trace → Log
You click: "Logs for this span" on a span
You get: only the log lines with that trace id
TESTED · Q15
Trace → Profile
You click: "Profile" on a span (Go, .NET)
You get: a flame graph of only that span's samples
TESTED · Q8; Q9
Trace → Metric
You click: "Metrics" on a span
You get: span metrics for that service and span name
DECIDED · A193 tab 5 NOT TESTED · not in the spike config
Log → Trace
You click: the trace_id on a log line
You get: that request's whole trace in Tempo, if the trace was kept by sampling (see the warning below)
TESTED · Q15
Profile → Trace
You click: a flame graph
You get: no jump back is designed; you go by service and time
NOT TESTED · not in the design
Flutter → Trace
You click: a Flutter span
You get: the backend span under it, via traceparent
TESTED · Q17, wire only
The trace context: one id, carried by itself
- Each request gets a trace id. Every span, log line and (in Go and .NET) profile sample in that request carries it. DECIDED · A193 tab 1 #2
- You never pass it by hand. Bun keeps it in AsyncLocalStorage, Go in
ctx, .NET inActivity., Dart in aCurrent Zone. DECIDED · A193 tab 3 - Bun proved it survives
await, timers andsetImmediate. TESTED · Q2 - Between services it travels in the W3C
traceparentheader. Example:00-4bf92f35…-00f067aa…-01. DECIDED · A193 tab 4 - Go sets no propagator today, so incoming traces are ignored. The new
Register()fixes it. DECIDED · A193 tab 4 - Flutter: Faro puts
trace_idandspan_idon every signal and sendstraceparenton HTTP calls. TESTED · Q17 - Faro's fields land in Loki under other names, and obs must map them. The record says
event_data_*; the spike shows log context arrives ascontext_*. TESTED · Q17, loki-faro-response.json OPEN · fix the mapping
Each jump, honestly
| Jump | Rides on | Strength | Status |
|---|---|---|---|
| log → trace | trace_id on the log line | exact, if the trace was kept | TESTED · Q15 PROPOSED · sampling caveat |
| trace → logs | trace id search in Loki | exact | TESTED · Q15 |
| trace → profile | span id on each sample (Go, .NET) | exact, one span | TESTED · Q8; Q9; Q15 |
| trace → profile, Bun | service name + time window | rough | TESTED · Q4 |
| trace → metrics | span metrics made by Tempo | good: same service and span | DECIDED · A193 tab 5 NOT TESTED · jaeda: not shown in a spike table |
| metric → traces | same LPSM and route labels + time | rough: many traces | OPEN · A193 pick 1 NOT TESTED · not tested |
| Flutter span → backend span | traceparent header | exact | TESTED · Q17, wire only |
Sampling can break the exact links. With raichu's ratio 0.1, the app keeps about 1 in 10 traces. Log lines still carry the trace id, so up to 9 in 10 log → trace clicks may find nothing. The record leaves "keep every error trace" to a collector, but a collector never sees spans the app dropped. See weak spot 1. DECIDED · A193 tab 2; tab 1, jaeda call 13 PROPOSED · jaeda reading NOT TESTED · not tested
Why no exemplars
- An exemplar is a trace id stored on a metric point. Click a spike on a graph → open that one trace. DECIDED · A193 terms
- VictoriaMetrics accepts exemplars and silently drops them.
query_exemplarsalways returns[]. Its maintainer says it does not support them. TESTED · Q13 - OpenTelemetry for JavaScript has no exemplar code at all. No trace id bytes in the payload. So Bun would have none, even with a store that keeps them. TESTED · Q1
- Mimir keeps exemplars (50,000 stored in a test), but only for Go and .NET, and its query was about 2× slower. TESTED · A193 tab 5, Mimir test 09 Nov; numbers only in a193-mimir.md, no results file
What "link by shared attributes" means
- A metric point and its spans carry the same labels: the 4 LPSM keys plus the route or provider. OPEN · A193 pick 1, recommended
- Example: a spike in
checkout_charges_total{durian_cloud_service="checkout", payment_provider="stripe", outcome="error"}at 14:07. PROPOSED · this site - You click it → Grafana searches Tempo for spans of
checkoutwithpayment.and status error, between 14:02 and 14:12. PROPOSED · this siteprovider=stripe - You get a list of traces, not the one trace. Usually good enough; worse when many requests look alike. PROPOSED · this site
- Nothing forces the metric and span to use the same label names. If a metric says
providerand the span sayspayment., the link finds nothing.provider obs query --liveruns each link on the seed and fails a broken one. DECIDED · A193 tab 5
The 4 LPSM keys: one query finds any signal
| Part | Standard key | Our key | Stored as |
|---|---|---|---|
| Landscape | deployment. | durian. | durian_cloud_landscape |
| Platform | service. | durian. | durian_cloud_platform |
| Service | service. | durian. | durian_cloud_service |
| Module | — | durian. | durian_cloud_module |
- Both keys are sent, so standard tools work and one query on our keys alone finds any signal. DECIDED · A193 tab 1 #4
- Our keys use dots (your call on 09 Nov). DECIDED · A193 tab 1 #19
- Example LogQL:
{durian_cloud_landscape="pichu", durian_cloud_service="checkout"}. DECIDED · A193 tab 5 - Old
atomi.keys are dropped. Never* atomi.. DECIDED · A193 tab 1 #4* - Not yet checked: how each store turns these keys into labels. NOT TESTED · A193 tab 5 "check how each store exposes"
The record disagrees with itself here. Decision 19 says dots (durian.). The main view and tab 5 still say a slash (durian.). This site follows decision 19, the newer ruling. The event key is still written durian. everywhere. Both store as underscores, so queries do not change. DECIDED · A193 tab 1 #19 OPEN · fix the record
How obs tests the links
- The seed writes one linked set per request: a trace, its log lines, metric points with the same LPSM keys, and a span profile. DECIDED · A193 tab 2
obs query --liveruns every jump. A broken link fails step 4. DECIDED · A193 tab 5- Example failure:
link trace→logs: 0 log lines for trace 4bf92f35… (datasource derived field uses "traceID", logs carry "trace_id"). DECIDED · A193 tab 5 obs assert(e2e) checks 100% of log events carry trace ids. DECIDED · A193 tab 2
Open pick 1 lives here: keep VictoriaMetrics and link by shared labels, or move to Mimir for exemplars. Options and trade-offs. OPEN · A193 pick 1