Observability
From an observation to a defensible conclusion
Logs, metrics and traces explain runtime behavior; business records establish committed outcomes. Useful charts preserve where observations reached, what they cover and what an empty result means.
Separate runtime diagnosis from business results
“Why was the request slow?” and “Was the charge posted?” need different evidence. Traces, durations and dependency calls help explain stages. The charge requires the original request ledger. A log describes what a producer reported; it does not replace committed business state.
Telemetry supplies shared context propagation, HTTP observations and standard signal export. OLAP assembles collection, storage and native queries. Projects select business events, recordable fields and analytic meaning. Receiver acceptance, exporter completion and backend query visibility are separate observations along that path.
Response headers do not complete the exchange
A streaming request can return headers quickly and continue producing a body before EOF, failure or abandonment. Telemetry distinguishes header availability from complete exchange duration. After headers arrive, the returned body retains the observation until its actual terminal event.
A fast initial response does not establish fast full completion or delivery of every byte. Instrumentation observes one service call without buffering the body, adding a business retry or sending another request. More detailed stage analysis needs explicit observations within that same original invocation.
| Observation | Clock ends at | Meaning |
|---|---|---|
| Response-header duration | Response headers become available | Waiting before response availability |
| Complete HTTP duration | Body EOF, error or abandonment | The end of this exchange |
| Business commitment | The business authority confirms the write | The original business outcome is stored |
Give zero, empty results and query failure different meanings
The observation pipeline has explicit capacity and loss semantics. Producer queues can drop observations when full. Collector queues, retry horizons and memory limits are resource choices, not lossless-delivery guarantees. Business completion does not wait for every signal to be stored, and exporter shutdown is not proof of receiver persistence.
An empty query may mean no matching records in that window. A failed query does not establish even that observation. Retained readings keep their original observation time; rerendering does not make them fresh. A meaningful zero requires a successful observation with an established source, scope and unit.
| State | May express | Do not substitute |
|---|---|---|
| Observed zero | Zero within the observed scope | No activity in every larger scope |
| Successful empty query | No returned records for the window and selectors | A current value of zero |
| Unavailable query | The required observation is unavailable | An empty set or healthy state |
| Retained older reading | A known result at its original time | A newly collected sample |
Keep units, subjects and denominators with the chart
OLAP’s host overview selects CPU non-idle ratio, available and total memory, and filesystem usage. CPU uses the complement of the idle ratio. Memory and filesystem bytes convert to GiB with explicit units, while device, mountpoint and used/free/reserved remain separate series.
Adding mountpoints indiscriminately can duplicate aliases or combine unrelated capacities. Native enumeration does not guarantee every mount is included. Query evaluation time is not a new sample time, and a continuous line alone establishes neither freshness nor current host health.
Compare the same subjects and units
Retain device, mountpoint, state and time window so each series has a defined subject.
Do not count metric rows as requests
Gauge samples and business requests belong to different populations.
Observation policy belongs to the business boundary
Standard instrumentation does not authorize recording every input. Path capture and query redaction depend on the business’s authority and content. Telemetry distinguishes complete, redacted and omitted path observations. Exceeding an observation budget does not retain a misleading prefix or rewrite the original business request.
Trace context correlates runtime work without elevating remote trust or business authority. Authentication headers, cookies and bodies are not ordinary HTTP observation payloads. Projects select safe business fields and destinations, keeping diagnostic clues separate from secrets, identity proof and financial authority.
Verify chart meaning as well as successful import
An importable dashboard can still use the wrong units, population or selectors. The rules call for independently distinguishable records against the actual queries, including meaningful empty results, gaps and failures. Valid collection evidence can be reused rather than repeating the whole pipeline for every panel.
| Meaning to verify | Observation to arrange | Compare |
|---|---|---|
| Full exchange duration | Immediate headers with a delayed body | Header and body completion times remain distinct |
| Memory and filesystem units | Distinct byte values, devices and mounts | Correct conversion and separate series |
| Data visibility | Trigger a signal and query expected records | Do not replace backend reads with export ACKs |
| No-data meaning | Empty results, old samples and query failure | Zero, missing and unavailable stay distinct |