← Work

Servernah SLA Monitoring Framework

Atlancis Technologies · April 2025 – present · servernah-sla-pipeline repo

The company had been operating on a 14-year-old manual SLA process with no real-time visibility into service performance and no consistent methodology for measuring the metrics that determine whether a service is actually meeting its commitments.

The work started with mapping: understanding how every service, asset, and stakeholder across Main engineering departments (DevOps and EBU) interacted before writing a single formula. From that came two reusable templates:

Both are available as downloads, stripped of company-specific context. The dependency mapping template applies to any organisation with cascading service relationships. The recovery objectives workbook gives any team a structured baseline for measuring whether their SLA commitments reflect operational reality.

Parameter key

Parameter Definition
TTD Time to Detect
TTIR Time to Initiate Recovery
TTCR Time to Complete Recovery
IC Infrastructure Complexity (1–5 scale)
GD Geographic Distribution (1–3 scale)
RA Redundancy Availability (1–3 scale)
SEA System Evolution Age (1–3 scale)

The core metric formulas account for infrastructure complexity and geographic distribution as inflation factors on raw recovery times, and apply redundancy availability and system maturity as deflation factors on failure frequency:

$$\text{Adjusted MTTR} = \text{Base MTTR} \times \left(1 + \frac{\text{IC} - 1}{4}\right) \times \left(1 + \frac{\text{GD} - 1}{4}\right)$$
$$\text{Adjusted MTBF} = \text{Base MTBF} \times \left(\frac{\text{RA}}{3}\right) \times \left(\frac{\text{SEA}}{3}\right)$$

The automated monitoring layer replaced the manual process by correlating three data sources - infrastructure alerts from Zabbix, ticket lifecycle timestamps from MySupport, and OpenStack service states from OpenSearch - to compute SLA metrics in real time.

Incident to metric: event flow

  1. Detection — Zabbix raises an alert. MySupport creates a ticket capturing TTD. Airflow sensor DAGs pick up the event and query OpenSearch to snapshot the exact infrastructure service states at the moment of impact.
  2. Recording — Airflow triggers an isolated Spark job to process affected service boundaries. Metadata and state snapshots are committed to PostgreSQL and InfluxDB.
  3. Recovery initiation — Engineers interact with the ticket. MySupport records TTIR. Airflow detects the lifecycle change, updates the active metrics stream, and syncs data to live dashboards.
  4. Resolution — Zabbix confirms recovery and clears the alert, capturing TTCR. Spark joins all collected timestamps across the three telemetry sources, computes final RTO, RPO, and MTTR records, and closes the transaction in PostgreSQL.
  5. Reporting — Scheduled Airflow DAGs compute MTBF across resolved incidents, refresh compliance indices, and update executive dashboards.

Key decisions

Decision Chosen Why
Orchestration Airflow with dedicated SLA DAGs Isolated from existing tenant production loops.
Data ingestion Tri-factor correlation Zabbix, MySupport, and OpenSearch. Eliminates single-point telemetry failures in incident boundary detection.
Time-series storage InfluxDB Optimised for per-incident metric storage and live dashboard queries.
Structured metadata PostgreSQL SLA definitions, thresholds, and compliance records need relational structure.
Service state source OpenSearch Targeted index lookups at alert timestamps without continuous compute node polling.