Observability · controls and cost
What you can turn, and what it costs
Each landscape can turn each signal on or off, sample traces, cap labels and profile only some pods. The contract sets the limits; a landscape can never go past them. Costs are measured only on one machine; real-scale cost is not measured.
- Order: env > landscape config > base config > contract default. Nothing may pass a contract ceiling. DECIDED · A193 tab 2
- There is no supported profiling rate knob in any runtime (Go can be forced lower only through an unsupported race), so "profile a % of traffic" means "profile some pods". TESTED · Q10; Q18
Try it: what gets collected
Move the sliders. Numbers come from the design and the spikes; anything not measured says so.
Sources: DECIDED · A193 tab 2 TESTED · spike 10; spike 15 OPEN · A193 pick 3 PROPOSED · jaeda: dead-link estimate
The controls
| Control | Contract sets | Landscape config | Env override |
|---|---|---|---|
| Signal on/off | which signals exist | otel. | OTEL_SDK_DISABLED |
| Trace sampling | min_ratio | otel. | OTEL_TRACES_SAMPLER_ARG |
| Log level | — | otel. | DURIAN_OTEL_LOG_LEVEL |
| Profiled pods | max_pod_share | otel. | DURIAN_OTEL_PROFILES_ENABLED |
| Label budget | budget (default 100) | cardinality_limit ≤ budget × 4 | never raised at runtime |
All rows: DECIDED · A193 tab 2. Pod share row: OPEN · A193 pick 3.
- Example: raichu sets
ratio: 0.,1 profiles.,enabled: true pod_share: 0.. Pichu keeps the base. DECIDED · A193 tab 225 - The base config has profiling off; each landscape turns it on. PROPOSED · A193 tab 1, jaeda call 7
- Going past a ceiling fails:
raichu.. DECIDED · A193 tab 2yaml: otel. profiles. pod_share 0. 8 > contract max_pod_share 0. 5 OTEL_SDK_DISABLED=truekeeps the ports, sends nothing and turns profiling off. DECIDED · A193 tab 2
The label budget
Free-form labels are allowed, with two guards: banned names, and a budget of series per metric. DECIDED · A193 tab 1 #5
checkout.chargespayment.card_bin as a labelSources: DECIDED · A193 tab 2; tab 3; tab 4
- Banned names fail at once. Example:
user.on a metric →id Err(invalid-input, "user.. DECIDED · A193 tab 3id is banned on metrics; put it on the span or log") - The budget counts label sets per metric, per landscape, across all pods. PROPOSED · A193 tab 1, jaeda call 6
- Each process caps itself at budget × 4. Extra label sets go into one overflow series,
otel.. Example: budget 30 → cap 120. DECIDED · A193 tab 4metric. overflow=true obs assertfails the e2e run when a metric goes over budget. Example:checkout.. DECIDED · A193 tab 2charges: 212 series > budget 30 (new label: payment. card_bin) - A budget over 1000 needs a written
reason:. DECIDED · A193 tab 3 - Profile labels count as series in Pyroscope too, so the same rules apply. DECIDED · A193 tab 4
Profiling: what each runtime allows
| Bun | Go | .NET | |
|---|---|---|---|
| How | node:inspector → our ~80-line pprof converter | pyroscope-go | Pyroscope .NET profiler |
| Types | CPU only | CPU, memory, goroutines, locks | CPU, wall, memory, locks, exceptions |
| Span profiles | no (link by time only) | yes | yes |
| Rate knob | none (one profile at a time) | none (fixed 100 Hz) | none (fixed ~9 ms) |
Rows: TESTED · Q3; Q4; Q8; Q9; Q10.
@pyroscope/fails under Bun (nodejs ERR_DLOPEN_FAILED). Hence our own converter. TESTED · Q3- Flutter has no profiling. Faro sends CPU and memory measurements, not profiles. DECIDED · A193 tab 5
- Pyroscope can also drop profiles on the server. Example: probability
0.keeps half of5 checkout's profiles. TESTED · Q18, docs and config check
Open pick 3 lives here: profile 1 in N pods, every pod, or per request. Options and trade-offs. OPEN · A193 pick 3
Measured costs
| What | Number | Measured where | Status |
|---|---|---|---|
| Go profiling CPU | under ~2%, but inside ±5% run noise: not a firm bound | spike, one machine | TESTED · Q10 |
| .NET profiling CPU | under 1% (19.3 vs 19.3 req/s) | spike, one machine | TESTED · Q10 |
| Bun profiling CPU | not measured | — | NOT TESTED · A193 tab 4 |
| Profiling memory | under 50 MB per pod | Grafana's published numbers, not ours | NOT TESTED · Q18 cites docs |
| Local stack memory | ~630 MiB idle, under 800 MiB busy (busy = only 15 pushes) | 6 containers, one machine | TESTED · Q15 |
| Local stack start | 46 s cold, images cached (evidence: up in 18 s, ready 46 s later) | one machine | TESTED · Q15 |
| Local stack CPU | ~2.7% of a 32-core box | one machine | TESTED · Q15 |
| Pyroscope delay | 1–2 min from send to query | one machine | TESTED · Q15 |
| Grafana in step 5 | +~270 MiB | one machine | TESTED · A193 pick 2 |
Not measured anywhere: storage and network cost at production scale, Loki/Tempo/Pyroscope in normal mode on object storage, and Bun's profiling cost. The Mimir vs VictoriaMetrics test used 10,000 series for a short time; "production scale wasn't tested". NOT TESTED · A193 tab 5