Understand system state via external signals. Collect metrics, logs, and traces; drive action with SLOs, error budgets, and symptom-focused alerts.
1. The Three Pillars of Observability
A. Metrics
- Definition: Numeric time-series (counters, gauges, histograms) revealing what is happening.
- Tooling: Prometheus (collect/query) + Grafana (dashboards).
- Importance: Great for monitoring, alerting, trend analysis.
B. Logs
- Definition: Timestamped event records showing why it happened.
- Tooling: ELK Stack or cloud-native log solutions.
- Importance: Deep debugging and forensics; prefer structured JSON logs.
C. Traces
- Definition: End-to-end request journey across services, showing where time is spent.
- Tooling: Jaeger/Zipkin; instrument via OpenTelemetry.
- Importance: Crucial for microservices bottlenecks and failed calls.
2. Actionable Monitoring and Alerting
A. Service Level Objectives (SLOs)
- Definition: Targets for SLIs (e.g., 99.9% of requests < 300ms).
- SLI: The measured indicator (latency, error rate).
B. Error Budget
- Definition:
100% - SLO; e.g., 99.9% SLO → 0.1% budget. - Importance: Throttle features if burn rate is high; focus on reliability.
C. Alerting Strategy
- Actionable: Alert on SLO/SLI symptoms (99th latency > 300ms), not raw infra metrics.
- User-Centric: Prioritize alerts when users experience errors/latency.
Error Budget Visualization
Example SLO 99.9% with remaining budget across weeks; bars show budget consumed.
3. Tooling and Implementation Strategy
A. Prometheus and Grafana
- Prometheus: Scrapes
/metrics; query with PromQL. - Grafana: Dashboards for SLIs and operations.
B. Implementation: Telemetry Collection
- Instrumentation: Use OpenTelemetry to emit metrics/logs/traces at business points.
- Standardization: Naming conventions and consistent trace IDs enable correlation.