2022-11-05: Christie RAAS Outage
Summary
| Field | Value |
|---|---|
| Date | 5 November 2022 |
| Start time | 06:28 NZDT |
| End time | 07:02 NZDT |
| Duration | ~33 minutes |
| Severity | P1 |
| Customer | Christie NHS Foundation Trust |
| Impact | Full loss of Rhapsody message processing |
Root Cause
Termination node 1 experienced excessive CPU usage, which caused StrongSwan to stop responding. This brought down the HSCN VPN tunnel on termination node 1.
HA failover to termination node 2 did not occur because termination node 2 was also under-resourced — it did not have the capacity to handle the full BGP routing load, so the expected BGP routes were missing after the failover attempt.
Result: both termination nodes were effectively offline, cutting off Christie's HSCN connectivity.
Timeline
| Time (NZDT) | Event |
|---|---|
| 06:28 | HSCN VPN tunnel detected as down; alerts fire |
| 06:35 | On-call engineer paged |
| 06:42 | Root cause identified — termination node 1 CPU at 100% |
| 06:50 | Termination node 1 terminated via AWS console |
| 07:02 | Replacement instance bootstrapped; BGP routes restored; connectivity confirmed |
Resolution
Terminate and restart termination node 1. The ASG launched a fresh instance from the launch template, which bootstrapped correctly and re-established the BGP routes.
Action Items
| Jira | Action |
|---|---|
| RAAS-1886 | Resize HSCN termination nodes to a larger instance type |
| RAAS-2187 | Add BGP route count alerting to detect missing routes before they cause an outage |
Lessons Learned
- Under-resourced nodes are a single point of failure even in an HA configuration — if node 2 cannot handle a full failover, HA provides no protection
- BGP route monitoring is essential — the absence of routes was a leading indicator that was not alerting; RAAS-2187 addresses this
- Termination node CPU should be monitored with aggressive thresholds; a sustained CPU spike on a VPN node should trigger a P1 alert before connectivity impacts customer