Skip to content

HSCN VPN Operations

Sources: Making a New HSCN VPN (COE 20102774821) · Update HSCN VPN (TRIB 19243270180) · VPN Routes HSCN (TRIB 19397181499)

Consult Lee before any HSCN VPN changes

All HSCN VPN work requires senior engineer oversight. HSCN carries live patient data.

Runbook: Create a New HSCN VPN

When to use: Onboarding a new UK HSCN customer.

Prerequisites:

  • Access to hscn-connection-stack GitLab repo
  • Access to raas-management AWS account (S3 + SSM)
  • Customer has supplied their VPN endpoint IP and PSK
  • Authorisation from Franck Dakpogan or Brendon McAlevey

Phase 1 — Config (Terraform)

  1. Clone the hscn-connection-stack repo and create a branch named OPS-XXX/description (where OPS-XXX is the Jira ticket).

  2. Add VPN endpoint route in AWS Console:

  3. Go to raas-management AWS Console → VPC → Subnets
  4. Find hscn-subnet-1 and hscn-subnet-2

  5. For each: Route table → Routes → Add route → destination = customer VPN endpoint IP → target = eni-x (the termination node ENI)

  6. Add new stanza to config/raas-management.tfvars: 

    {
name = "customer-name"
vpn_endpoint = "x.x.x.x" # Customer firewall IP
rightid = "@customer-identifier"

# Link-local IPs for the tunnel
ll_interface = "169.254.1.6/30" # Node 1 (termination01)
lr_interface = "169.254.1.5/30" # Node 2 (termination02)

ike = "aes256-sha256-modp2048"
esp = "aes256-sha256-modp2048,aes128-sha256-modp2048"
keyexchange = "ikev1"
reauth = "no"
ikelifetime = "86400s"
keylife = "3600s"

mark = 100 # Must be UNIQUE across all VPNs
leftsubnet = "10.104.29.0/24"
rightsubnet = ["x.x.x.x/32"] # Customer's subnet(s) to advertise
routes = ["x.x.x.x/32"]
}

!!! important "mark must be unique" The mark value is a unique identifier for this VPN (used for routing). A lower mark value is preferred when the same route exists in multiple tunnels. Check all existing entries before assigning a new mark.

  1. Commit and push. Create a Merge Request in GitLab with Lee as reviewer.

Phase 2 — PSKs / Pipelines / Ansible

  1. Go to AWS raas-management S3 bucket: hscn-vpn-bootstrap-bucket-20200308213701382700000002

  2. Navigate to: strongswan/hscn-vpn-termination01/ and download ipsec.secrets

  3. Add the new PSK entry in this format: 

    10.104.29.4 x.x.x.x : PSK: "customer-supplied-PSK"
    Where 10.104.29.4 is the termination node IP and x.x.x.x is the customer VPN endpoint IP.

  4. Upload the updated ipsec.secrets back to S3 for hscn-vpn-termination01.

  5. Repeat steps 2–4 for hscn-vpn-termination02 (using 10.104.29.20 as the left IP).

!!! warning "Do not mix up the two files" Each node has its own ipsec.secrets. Termination01 uses 10.104.29.4, termination02 uses 10.104.29.20. Do not upload the wrong file to the wrong node.

  1. After the MR is merged, trigger the HSCN VPN Deploy workflow in GitHub Actions (with ACTION = apply) to apply the new config.

Phase 3 — Test

  1. Go to EC2 in raas-management and connect to hscn-vpn-termination01 via Session Manager.

  2. Escalate to root: 

    sudo -i

  3. Check the new VPN tunnel status: 

    strongswan statusall | grep '<customer-vpn-name>'

  4. Review the logs for any errors: 

    tail -f /var/log/messages | grep <customer-vpn-name>

  5. Repeat on hscn-vpn-termination02.

Phase 4 — Grafana / DataDog

After the VPN is confirmed up, ensure monitoring is configured:

  • Add the new tunnel to the Grafana/DataDog HSCN VPN monitoring dashboard
  • Verify BGP routes are appearing correctly in the TGW route table

Runbook: Update VPN Routes

When to use: A customer needs an additional subnet routed through their HSCN VPN, or existing routes need to change.

Sources: VPN Routes HSCN (TRIB 19397181499)

Primary Method (Pipeline)

  1. Clone hscn-connection-stack and create a branch for the change.

  2. Update the routes and/or rightsubnet arrays in the relevant VPN stanza in config/raas-management.tfvars.

  3. Commit, push, create MR → pipeline deploys the updated config to S3.

  4. Trigger the HSCN VPN Deploy workflow (with ACTION = apply) to apply the new routes to the termination nodes.

  5. Verify in TGW:

  6. AWS Console → Transit Gateways → tgw-039e174fce8f69655
  7. Route Tables → tgw-rtb-09351fe404a1a0891
  8. Check that the new routes appear in the route table with the correct attachment

Manual Update (if pipeline not available)

  1. SSH to the termination nodes (see Troubleshooting): 

    ssh -i .ssh/raas-uk.pem centos@10.104.29.4
ssh -i .ssh/raas-uk.pem centos@10.104.29.20

  2. Sync the bootstrap bucket: 

    aws s3 sync s3://hscn-vpn-bootstrap-bucket-20200308213701382700000002 /etc/strongswan/

  3. Run the Ansible playbook with the full remote_vpns list: 

    ansible-playbook -i inventory strongswan.yml -e remote_vpns="<full list>"

  4. Restart strongswan: 

    strongswan restart

Runbook: Update Existing VPN Config

When to use: Cipher suite upgrade, PSK rotation, or modifying VPN parameters.

Sources: Update HSCN VPN (TRIB 19243270180) Cipher suite reference: strongSwan IKEv2 Cipher Suites

Key tfvars fields for a new tunnel:

Field Example Notes
ikelifetime 86400s IKE SA lifetime
keylife 3600s IPsec SA lifetime
keyexchange ikev1 Usually IKEv1 for older customer hardware
rightsubnet ["0.0.0.0/0"] Use 0.0.0.0/0 to accept all destinations
mark 100 Lower = preferred route if destination has multiple tunnels

After updating tfvars:

  1. Via SSH to 10.104.29.4 and 10.104.29.20
  2. Sync from S3
  3. Run the ansible-playbook with the updated remote_vpns list
  4. strongswan restart