Skip to article

Observability

From an observation to a defensible conclusion

Logs, metrics and traces explain runtime behavior; business records establish committed outcomes. Useful charts preserve where observations reached, what they cover and what an empty result means.

Separate runtime diagnosis from business results

“Why was the request slow?” and “Was the charge posted?” need different evidence. Traces, durations and dependency calls help explain stages. The charge requires the original request ledger. A log describes what a producer reported; it does not replace committed business state.

Telemetry supplies shared context propagation, HTTP observations and standard signal export. OLAP assembles collection, storage and native queries. Projects select business events, recordable fields and analytic meaning. Receiver acceptance, exporter completion and backend query visibility are separate observations along that path.

Logs, metrics and traces enter Collector processing and are exported to OpenObserve. Application and host signals share the collection and query foundation.
Bring logs, metrics and traces to one query platformThe three signal types retain their sources through Collector intake, bounded processing and export to OpenObserve storage and queries. Check where a record has reached: reception, export and queryability are separate outcomes.

Response headers do not complete the exchange

A streaming request can return headers quickly and continue producing a body before EOF, failure or abandonment. Telemetry distinguishes header availability from complete exchange duration. After headers arrive, the returned body retains the observation until its actual terminal event.

A fast initial response does not establish fast full completion or delivery of every byte. Instrumentation observes one service call without buffering the body, adding a business retry or sending another request. More detailed stage analysis needs explicit observations within that same original invocation.

State the actual endpoint on a timing axis
ObservationClock ends atMeaning
Response-header durationResponse headers become availableWaiting before response availability
Complete HTTP durationBody EOF, error or abandonmentThe end of this exchange
Business commitmentThe business authority confirms the writeThe original business outcome is stored

Give zero, empty results and query failure different meanings

The observation pipeline has explicit capacity and loss semantics. Producer queues can drop observations when full. Collector queues, retry horizons and memory limits are resource choices, not lossless-delivery guarantees. Business completion does not wait for every signal to be stored, and exporter shutdown is not proof of receiver persistence.

An empty query may mean no matching records in that window. A failed query does not establish even that observation. Retained readings keep their original observation time; rerendering does not make them fresh. A meaningful zero requires a successful observation with an established source, scope and unit.

Do not flatten distinct states into one curve
StateMay expressDo not substitute
Observed zeroZero within the observed scopeNo activity in every larger scope
Successful empty queryNo returned records for the window and selectorsA current value of zero
Unavailable queryThe required observation is unavailableAn empty set or healthy state
Retained older readingA known result at its original timeA newly collected sample

Keep units, subjects and denominators with the chart

OLAP’s host overview selects CPU non-idle ratio, available and total memory, and filesystem usage. CPU uses the complement of the idle ratio. Memory and filesystem bytes convert to GiB with explicit units, while device, mountpoint and used/free/reserved remain separate series.

Adding mountpoints indiscriminately can duplicate aliases or combine unrelated capacities. Native enumeration does not guarantee every mount is included. Query evaluation time is not a new sample time, and a continuous line alone establishes neither freshness nor current host health.

  • Compare the same subjects and units

    Retain device, mountpoint, state and time window so each series has a defined subject.

  • Do not count metric rows as requests

    Gauge samples and business requests belong to different populations.

Observation policy belongs to the business boundary

Standard instrumentation does not authorize recording every input. Path capture and query redaction depend on the business’s authority and content. Telemetry distinguishes complete, redacted and omitted path observations. Exceeding an observation budget does not retain a misleading prefix or rewrite the original business request.

Trace context correlates runtime work without elevating remote trust or business authority. Authentication headers, cookies and bodies are not ordinary HTTP observation payloads. Projects select safe business fields and destinations, keeping diagnostic clues separate from secrets, identity proof and financial authority.

Verify chart meaning as well as successful import

An importable dashboard can still use the wrong units, population or selectors. The rules call for independently distinguishable records against the actual queries, including meaningful empty results, gaps and failures. Valid collection evidence can be reused rather than repeating the whole pipeline for every panel.

Checks matched to the question
Meaning to verifyObservation to arrangeCompare
Full exchange durationImmediate headers with a delayed bodyHeader and body completion times remain distinct
Memory and filesystem unitsDistinct byte values, devices and mountsCorrect conversion and separate series
Data visibilityTrigger a signal and query expected recordsDo not replace backend reads with export ACKs
No-data meaningEmpty results, old samples and query failureZero, missing and unavailable stay distinct