Connectivity Checks

Sources: Checks for HSCN RaaS instance (TRIB 19805143041) · HSCN Connectivity IP table (LS 6908903520)

Post-Maintenance Checklist — HSCN RaaS Instance

Run these checks after any maintenance that involves stopping or replacing an HSCN-connected Rhapsody instance (e.g., AMI updates, instance termination/replacement).

Take snapshots before maintenance

Always take snapshots of data volumes before starting maintenance. Do not proceed without recent S3 backups.

1. Wait for EIP and Volume Attachment

After an instance launches, wait until: - The Elastic IP (EIP) is attached (check EC2 console → Instances → Public IPv4) - EBS data volumes are attached (check EC2 → Volumes — status should be in-use)

2. Verify SSM Access

Confirm the instance is accessible via Session Manager: - EC2 → Instances → select instance → Connect → Session Manager - If Session Manager is not available after ~5 minutes, check that the SSM agent is running and the instance role has SSM permissions

3. Strong­Swan Status

sudo systemctl status strongswan

If not running:

sudo systemctl start strongswan

4. Rhapsody Service Status

sudo systemctl status rhapsody.service

Warning

Do not start or restart rhapsody.service without explicit authorisation from a senior engineer.

5. Kernel Version

uname -r

Verify this matches the expected kernel version for the current AMI release.

6. Check Bootstrap Log for Errors

less /var/log/rhapsody-bootstrap/playbookError.log

Any errors here indicate the bootstrap Ansible playbook encountered issues during provisioning.

7. Proxy Connectivity

nc -z -v proxy.service 3128
nc -z -v smtp.service 465

Both should return Connection to ... succeeded!. If proxy.service is unreachable, check: - Route via VPC routing table - Management account proxy instances are healthy - Security group rules allow port 3128

8. Instance Metadata

curl http://169.254.169.254/latest/meta-data/

Should return instance metadata fields. Failure indicates a network issue with the instance or missing IMDSv2 permissions.

9. BGP Routes

ip route show | grep zebra

Quagga/zebra injects BGP-learned routes. You should see routes to HSCN customer subnets via the zebra daemon. If missing, strongswan may not have established the tunnel yet — check strongswan status.

10. DataDog BGP Monitor

Check DataDog monitor ID 143660477 for BGP route status. This monitor alerts if BGP routes are missing from the HSCN VPN.

11. Verify HSCN VPN Tunnels in Management Account

For HSCN-connected customers, verify the customer-side VPN tunnels are up in the management account:

1. Log into AWS management account (640815619001, eu-west-2)
2. VPC → Site-to-Site VPN connections
3. Locate the tunnel for this customer and confirm both tunnels show UP status

12. Rhapsody Custom Metrics in DataDog

Check DataDog monitor ID 12604900 for Rhapsody-specific health metrics. This confirms the Rhapsody engine is running and emitting metrics.

13. Ansible Tower — Run Smoke Test

In Ansible Tower (UK region), run the RaaS smoke test playbook for this customer. This runs a set of connectivity and health checks against the provisioned environment.

---

HSCN Customer IP Reference

For the full IP address table of HSCN customers, see Customers.

Quick reference for VPN termination nodes:

Node	IP	SSH Command
termination01 (primary)	10.104.29.4	ssh -i .ssh/raas-uk.pem centos@10.104.29.4
termination02 (failover)	10.104.29.20	ssh -i .ssh/raas-uk.pem centos@10.104.29.20

Note: These nodes are only reachable from the AWS Workspace (UK region) or from within the management VPC.