Skip to content

Maintenance

Source: Maintenance Standard RaaS (TRIB 19634880696)

Snapshots FIRST — always

Always take snapshots of data volumes before starting any maintenance. Check S3 for recent backups and abort if none are present.

Prerequisites

Before starting a maintenance window:

  1. Copy the AMI from the release/centos7 branch into the destination account.
  2. In Ansible Tower, run:
  3. Deploy VPC — updates the Proxy instance AMI
  4. Deploy Rhapsody — updates IAM and the RaaS instance AMI
  5. Execute all pending CloudFormation changesets in the customer account.

Runbook: Proxy Instance Maintenance

When to use: Replacing or updating a proxy instance (e.g., AMI update, security patch).

Steps

  1. Identify the current Launch Template version:
  2. EC2 → Auto Scaling Groups → select the proxy ASG

  3. Note the Launch Template name and version

  4. Verify the AMI:

  5. From the Launch Template, copy the AMI ID
  6. EC2 → Images → AMIs → paste the AMI ID

  7. Confirm it matches the expected AMI from the AMI build pipeline

  8. Terminate the proxy instance:

  9. EC2 → Instances → select the proxy instance
  10. Instance State → Terminate

The Auto Scaling Group will automatically launch a replacement from the new Launch Template.

Post-Termination Checks

After the replacement instance launches:

  • EIP is attached (check EC2 → Instances → Public IPv4 address)
  • EBS volumes are attached (in-use status)
  • Instance has correct tags (check for Patch: US/EMEA/APAC tag)
  • Instance is accessible via Session Manager (EC2 → Connect → Session Manager)
  • Squid proxy is running: sudo systemctl status squid
  • Kernel version correct: uname -r
  • SSM agent version current: check SSM → Fleet Manager

Runbook: RaaS Instance Maintenance

When to use: Replacing or updating a Rhapsody instance (e.g., AMI update, kernel patch, hardware issue).

HSCN customers

For HSCN-connected customers, extra checks are required post-replacement (VPN tunnel verification in both the customer account and the management account).

Steps

  1. Check for recent S3 backups:
  2. Go to the customer's S3 backup bucket
  3. Verify there is a backup from within the last 24 hours

  4. Abort if no recent backup exists — escalate to Lee

  5. Take snapshots of data volumes:

  6. EC2 → Volumes → select the Rhapsody data volume(s)
  7. Actions → Create Snapshot
  8. Add a description: Pre-maintenance YYYY-MM-DD

  9. Wait for the snapshot to complete before proceeding

  10. Identify the current Launch Template version: (same as proxy steps above)

  11. Terminate the desired Rhapsody instance:

  12. EC2 → Instances → select the Rhapsody instance
  13. Instance State → Terminate
  14. The ASG will launch a replacement from the updated Launch Template

Post-Termination Checks

  • EIP attached
  • EBS volumes attached
  • Correct tags present
  • Instance accessible via Session Manager
  • StrongSwan running (HSCN customers): sudo systemctl status strongswan
  • Rhapsody service running: sudo systemctl status rhapsody.service
  • Kernel version correct: uname -r
  • SSM agent version current
  • Bootstrap log clean: less /var/log/rhapsody-bootstrap/playbookError.log
  • HSCN: VPN tunnels up in AWS Console (customer account)
  • HSCN: VPN tunnels up in management account (640815619001, eu-west-2)
  • Rhapsody custom metrics visible in DataDog (monitor ID 12604900)
  • Ansible Tower: run the RaaS smoke test playbook for this customer

See Connectivity Checks for the full HSCN-specific post-maintenance checklist.

HSCN Stack Upgrade Procedures

Source: HSCN Stack Upgrade (TRIB 19552370847)

This section documents the upgrade strategy for the 4 HSCN VPN nodes during major platform upgrades (e.g., monitoring agent replacement, OS packages).

VPN Nodes in Scope

Node Role
hscn-vpn01 Primary VPN (fallback node)
hscn-vpn02 VPN (upgrade first)
hscn-vpn-termination01 Primary termination (upgrade after vpn02 verified)
hscn-vpn-termination02 Termination (upgrade second)

Upgrade Order

Always upgrade in this order to minimise impact — upgrade the non-primary/backup node first:

  1. vpn02 (non-primary VPN)
  2. termination02 (secondary termination)
  3. termination01 (primary termination)
  4. vpn01 (primary VPN — last)

Example Upgrade: DataDog v6 → v7 / AV Sophos → SentinelOne + Prometheus

For each node (in order above):

  1. Run the upgrade Ansible playbook targeting the single node
  2. Verify service is running: tailf /var/log/messages (or journalctl -f)
  3. Confirm the node is healthy in Coralogix and VPN tunnels are up in AWS console
  4. Proceed to next node only after verification

Rollback Strategy

If a node fails to upgrade or service becomes unstable:

  1. Identify the failed HSCN node in AWS console
  2. Navigate to the Auto Scaling Group for that node
  3. Revert the Launch Template to the previous version
  4. Terminate the failed instance — ASG will launch a new instance from the reverted template
  5. Verify the replacement node comes up correctly before continuing

Example Maintenance Schedule

The following is a historical reference for how maintenance windows have been scheduled:

Node Time (BST) Engineer
hscn-vpn01 29 June 2024, 23:00 Lee Jelley
hscn-vpn02 29 June 2024, 23:00 Lee Jelley
hscn-vpn-termination01 29 June 2024, 23:30 Mark Lehmann
hscn-vpn-termination02 29 June 2024, 23:30 Mark Lehmann