Known Issues & Troubleshooting

Source: HSCN VPN Troubleshooting (TRIB 19612991626)

SSH to Termination Nodes

You can SSH directly to the HSCN VPN termination nodes from the AWS Workspace (UK region):

# Primary node
ssh -i .ssh/raas-uk.pem centos@10.104.29.4

# Failover node
ssh -i .ssh/raas-uk.pem centos@10.104.29.20

Note

These nodes are not accessible from outside the management network. You must be connected to the AWS Workspace (UK) or within the management VPC.

StrongSwan Diagnostic Commands

# Show status of all VPN tunnels
strongswan status

# Show status of a specific tunnel
strongswan status | grep <VPN_NAME>

# Detailed output including all SAs and connections
strongswan statusall

# Bring a specific tunnel down
strongswan stroke down <VPN_NAME>

# Bring a specific tunnel up
strongswan stroke up <VPN_NAME>

# Restart the strongswan service
service strongswan restart

Log Inspection

# Live log tail
tail -f /var/log/messages

# Search for a specific VPN in logs
grep <VPN_NAME> /var/log/messages

# View recent log entries
cat /var/log/messages | tail -100

When troubleshooting VPN bring-up issues, look for: - CHILD_SA <name> established — tunnel came up successfully - IKE_AUTH request / IKE_AUTH response — IKE handshake in progress - authentication failed — PSK mismatch - no proposal chosen — cipher suite mismatch

Packet Capture

tcpdump can be run directly on the termination nodes for deep packet inspection:

# Capture on the VPN interface
tcpdump -i <interface> -w /tmp/capture.pcap

# Upload pcap to S3 for analysis
aws s3 cp /tmp/capture.pcap s3://raas-cert-monitoring-uk/pcap/

Viewing History

history | grep <COMMAND>

Useful for checking what commands have been run on the termination node recently.

Common Issues

MTU / DF-bit Issues

Symptom: Partial connectivity — some traffic works, larger packets drop silently.

Root cause: Packets with the Don't Fragment (DF) bit set exceed the VPN MSS (1320–1330 bytes).

Diagnosis: - Ask the customer: what packet sizes are they using? Is the DF bit set on their firewall? - Run tcpdump on the termination node to inspect the DF bit in captured packets - Check if the MSS is configured correctly: ip link show <interface>

Fix: - Configure the customer firewall to reduce MTU or clear the DF bit before entering the VPN - In the Christie case: Cisco device with jumbo frames — the fix was to prevent the VPN from clearing the DF bit, allowing the firewall to manage fragmentation itself

See HSCN Overview → MTU / MSS for full details.

Tunnel Established but No Traffic

Symptom: strongswan status shows ESTABLISHED but traffic is not flowing.

Check: 1. Verify BGP routes: ip route show | grep zebra — routes must be present for traffic to flow 2. Verify the TGW route table includes the customer's CIDR: AWS Console → Transit Gateway → tgw-039e174fce8f69655 → Route Tables 3. Check the proxy is reachable: nc -z -v proxy.service 3128 4. Check security groups in the customer VPC allow traffic from 10.104.29.0/24

PSK Authentication Failure

Symptom: authentication failed in /var/log/messages.

Fix: - Verify the PSK in ipsec.secrets matches exactly what the customer configured on their firewall - Confirm the left/right IP pair in ipsec.secrets is correct (left = termination node IP, right = customer VPN endpoint IP) - After updating ipsec.secrets, trigger the HSCN VPN Deploy workflow (with ACTION = apply) or run strongswan restart

No Proposal Chosen (Cipher Mismatch)

Symptom: no proposal chosen in logs during IKE negotiation.

Fix: - Check the ike and esp fields in the customer's VPN stanza in raas-management.tfvars - Ask the customer what cipher suites their firewall supports - Reference: strongSwan IKEv2 Cipher Suites - Common working values: ike = "aes256-sha256-modp2048", esp = "aes256-sha256-modp2048,aes128-sha256-modp2048"