2022-04-07: West Hertfordshire HSCN VPN Down
Source: 2022-04-07 WestHerts HSCN VPN Down (TRIB 19324698652)
Summary
| Field | Value |
|---|---|
| Date | 7 April 2022 |
| Start time | 13:39 NZT |
| End time | 16:35 NZT |
| Duration | ~2 hours 56 minutes |
| Severity | P1 |
| Customer | West Hertfordshire NHS Trust |
| Impact | Full loss of HSCN VPN connectivity; Rhapsody unable to reach hospital systems |
Root Cause
An engineer ran the config strongswan Ansible playbook WITHOUT the vpn_role parameter.
This caused the playbook to configure the standard VPN correctly but removed the HSCN tunnel and routes in the process. The HSCN VPN configuration was overwritten/cleared because the HSCN-specific branch was not selected.
A second attempt to fix by running the playbook WITH the vpn_role=hscn parameter also failed — likely because the earlier strongswan restart had already removed the Transit Gateway VPN static routes from the routing table, leaving the network in an inconsistent state that the playbook could not fully recover.
The only reliable resolution was to terminate and rebuild the prod instance from scratch.
Timeline
| Time (NZT) | Event |
|---|---|
| 13:39 | config strongswan run without vpn_role parameter |
| 13:42 | HSCN VPN tunnel down alert fires |
| 13:50 | Engineer attempts re-run with vpn_role=hscn — does not resolve |
| 14:15 | Escalated to Senior Engineer |
| 15:30 | Decision made to terminate prod instance |
| 15:32 | Prod instance terminated via AWS console |
| 16:35 | Replacement instance bootstrapped; HSCN VPN restored |
Resolution
Terminate the production Rhapsody instance. The ASG launched a fresh replacement, which bootstrapped from scratch including correct StrongSwan/HSCN configuration.
Action Items
| Action | Description |
|---|---|
| Playbook parameter validation | Add a check to config strongswan to error if vpn_role is not specified for HSCN-connected accounts |
| Consider removing Westherts prod VPN monitor | The per-host VPN monitor may have contributed to the instability during the failed recovery attempt |
Lessons Learned
Critical: Always use vpn_role parameter when running config strongswan
Running config strongswan without explicitly specifying vpn_role on an HSCN-connected instance will destroy the HSCN tunnel. This is a destructive and non-obvious default. Always confirm the correct vpn_role before running this playbook.
config strongswanwithoutvpn_roleis dangerous — it silently overwrites HSCN configuration with the standard VPN config without any warning or confirmation- A failed Ansible fix can leave routing in a worse state — once Transit Gateway routes were removed, the playbook could not fully recover without an instance rebuild
- Instance rebuild is sometimes the fastest path to recovery — don't spend too long trying to fix a broken VPN config in place; a termination-and-rebuild took 63 minutes but was reliable
- Document parameter requirements explicitly — any playbook that has destructive defaults for HSCN customers should clearly state the required parameters in both the playbook README and in the Ansible Tower template description