Servernah SLA Monitoring Framework
The company had been operating on a 14-year-old manual SLA process with no real-time visibility into service performance and no consistent methodology for measuring the metrics that determine whether a service is actually meeting its commitments.
The work started with mapping: understanding how every service, asset, and stakeholder across Main engineering departments (DevOps and EBU) interacted before writing a single formula. From that came two reusable templates:
- Service Recovery Objectives workbook — automated RTO, RPO, MTTR, and MTBF calculations built on quantifiable inputs (TTD, TTIR, TTCR, backup frequency, incident rate) and qualitative scoring (resource availability, staff expertise, service criticality), with dependency tracking, impact analysis, and improvement planning across eight sheets
- Service Dependency Mapping template — full infrastructure dependency chains per service covering physical, virtual, network, software, external, and regulatory layers, with cascade failure scenarios, SPOF identification, and ordered recovery sequences
Both are available as downloads, stripped of company-specific context. The dependency mapping template applies to any organisation with cascading service relationships. The recovery objectives workbook gives any team a structured baseline for measuring whether their SLA commitments reflect operational reality.
Parameter key
| Parameter | Definition |
|---|---|
| TTD | Time to Detect |
| TTIR | Time to Initiate Recovery |
| TTCR | Time to Complete Recovery |
| IC | Infrastructure Complexity (1–5 scale) |
| GD | Geographic Distribution (1–3 scale) |
| RA | Redundancy Availability (1–3 scale) |
| SEA | System Evolution Age (1–3 scale) |
The core metric formulas account for infrastructure complexity and geographic distribution as inflation factors on raw recovery times, and apply redundancy availability and system maturity as deflation factors on failure frequency:
The automated monitoring layer replaced the manual process by correlating three data sources - infrastructure alerts from Zabbix, ticket lifecycle timestamps from MySupport, and OpenStack service states from OpenSearch - to compute SLA metrics in real time.
Incident to metric: event flow
- Detection — Zabbix raises an alert. MySupport creates a ticket capturing TTD. Airflow sensor DAGs pick up the event and query OpenSearch to snapshot the exact infrastructure service states at the moment of impact.
- Recording — Airflow triggers an isolated Spark job to process affected service boundaries. Metadata and state snapshots are committed to PostgreSQL and InfluxDB.
- Recovery initiation — Engineers interact with the ticket. MySupport records TTIR. Airflow detects the lifecycle change, updates the active metrics stream, and syncs data to live dashboards.
- Resolution — Zabbix confirms recovery and clears the alert, capturing TTCR. Spark joins all collected timestamps across the three telemetry sources, computes final RTO, RPO, and MTTR records, and closes the transaction in PostgreSQL.
- Reporting — Scheduled Airflow DAGs compute MTBF across resolved incidents, refresh compliance indices, and update executive dashboards.
Key decisions
| Decision | Chosen | Why |
|---|---|---|
| Orchestration | Airflow with dedicated SLA DAGs | Isolated from existing tenant production loops. |
| Data ingestion | Tri-factor correlation | Zabbix, MySupport, and OpenSearch. Eliminates single-point telemetry failures in incident boundary detection. |
| Time-series storage | InfluxDB | Optimised for per-incident metric storage and live dashboard queries. |
| Structured metadata | PostgreSQL | SLA definitions, thresholds, and compliance records need relational structure. |
| Service state source | OpenSearch | Targeted index lookups at alert timestamps without continuous compute node polling. |