Observability Engineering: Notes
When a software is mentioned to have observability? When we can understand: * Inner workings * System state * inner workings and system state from external observation * internal state without any custom code
Shortfalls of traditional monitoring ppaorach: * Many services * polyglot persistence * dynamic * variety of correlation
Debugging with Metrics Versus Observability
Common ones: six high-cardinality dimensions per event: time, app, host, user, endpoint, and status
Why Intuitive Troubleshooting Has Stopped Working for You * sense-analyze-respond needed due to complexities * core analysis loop eg: Most of the errors are coming from a specific endpoint –> there’s a subset of requests that seem to have a much higher latency than others.–> one of the slow requests seems to be hanging up in a caching subsystem –> all the slow caching calls seem to be referring to the same object ID. –> a specific cached object has become corrupted somehow –> test that hypothesis by looking at the payload of those objects. –> execute a command to flush that cached object and watch for the effects
- Observability allows us to probe deep into our systems to understand behavior, down to the level of individual requests between services.
Structured Events * Numerical values generated by a metric reflect an aggregated report of system state over a predefined period of time
- High cardinality and high dimensionality not feasible. eg: “all Canadian users, running iOS11 version 11.0.4, using the French language pack, who installed the app last Tuesday, running firmware version 1.4.101, who are storing photos on shard3 in region us-west-1.”
- Structured logs are machine parsable and can be useful for the goals of observability, presuming they are redesigned to resemble a structured event.
Stitching Events into Traces Distributed traces are an interrelated series of events.
Components of trace: UUID Trace ID Span ID Parent ID Timestamp Duration