Skip to content

Resilience Patterns

Purpose — This page compares two resilience patterns and is suitable for internal reference and customer-facing conversations.

At a glance: Self-healing focuses on cost-efficient resilience via rebuild; Active-passive HA delivers continuity via rapid failover to a pre-provisioned standby.

Executive takeaway

  • Single data disk self-healing — Restores service by rebuilding the only node; think of it as a rebuild event measured in double-digit minutes.

  • Active-passive HA — Restores service by failing over to a ready passive node; customer impact is roughly a couple of minutes while the failed node recovers off-path.

Comparison at a glance

Category Single Data Disk Self-Healing Active-Passive HA
Primary design goal Cost-efficient resilience Business continuity / high availability
Production node count One production node Two production nodes (active/passive)
Storage model One managed data disk attached to the service node Shared Azure NetApp Files (ANF) storage
Traffic during failure Unavailable while the only node is rebuilt Fails over to the passive node
Recovery model Rebuild-based recovery Failover first; rebuild in parallel
Estimated recovery / failover time 10+ minutes Approximately 1–2 minutes
Standby capacity None Passive node already provisioned
Shared customer endpoint Not a defining mechanism Shared VIP and shared HA DNS endpoint
Control plane complexity Lower Higher
Cost profile Lower cost Higher cost
Best fit Smaller or cost-sensitive environments Environments with strong uptime/continuity requirements

How each solution works

1) Single data disk self-healing with VMSS + user data automation

Architecture summary

This path uses a single VM with a local managed disk as the data backend. Terraform provisions a single-instance flexible VMSS and standalone managed disks from block_device_mappings. The Ansible default storage backend is managed_disk.

  • VMSS replaces or recreates the unhealthy VM instance.

  • User data / cloud-init / Ansible runs on the rebuilt node.

  • Ansible discovers the managed disk by tags; if not attached, executes az vm disk attach.

  • Detects NVMe vs SCSI, waits for device path, validates readiness, mounts /data/orionhealth and persists in /etc/fstab.

  • If /data/orionhealth/rhapsody is absent, bootstrap + install + configure + start services.

This is resilient automation, but still a rebuild-based recovery model. Service resumes only after the VM is healthy, the disk is attached/mounted, and the app is ready.

Terraform notes that Ansible attaches disks to VMSS instances post-provisioning, which is why destroy-time logic accounts for disk locks.

Operational interpretation

Improves resilience and reduces manual toil, but does not provide rapid continuity: recovery is gated by rebuilding the only serving node.

2) Active-passive HA with shared storage and automated failover

Architecture summary

Two production VMs in an active/passive pair. Shared Azure NetApp Files volume replaces local data disks. HA lock storage/blobs coordinate ownership. Terraform derives two prod instances from the prod ASG and tags roles (active/passive).

  • Shared HA load balancer with both nodes in one backend pool and a shared VIP.

  • Shared HA DNS record (rhap-ha) targets the VIP; health probes on HTTPS 8444 gate traffic to the healthy node.

  • Monitoring alerts trigger an Azure Function that performs failover logic with fencing and role swaps.

  • Blob-lease locking ensures single-writer semantics to shared storage.

Failover flow: identify active/passive; validate alert targets the active; fence the active (deallocate and tag failed); break the blob lease; promote passive via tag; lock script performs mount/ownership; split-brain detection and cooldowns prevent thrash.

On each node, raas-ha-lock.sh owns the ANF mount lifecycle. Ansible installs NFS tooling and prepares mountpoints; the active mounts /data/orionhealth via the lock service, while the passive remains unmounted until promotion.

Operational interpretation

Designed for continuity: customer recovery is a traffic handoff to a warm standby, not a wait for infrastructure rebuild.

Side-by-side operational comparison

Single data disk self-healing

Continuity depends on rebuilding the same node. VMSS restores infrastructure; automation reattaches and mounts storage. Great when you want automated recovery without paying for a hot standby, but any node failure interrupts service until rebuild completes.

Active-passive HA

Continuity depends on traffic handoff. The passive is pre-provisioned; the shared VIP and health probes steer traffic; monitoring and leases enforce safe promotion. Repair of the failed node is decoupled from customer recovery.

Estimated recovery / failover times

Single data disk self-healing model

Recovery expectation

Plan for a practical recovery window in the double-digit minutes when self-healing is triggered.

  • VM recreation and user-data execution

  • Ansible run, disk discovery by tag, attach, device detection

  • Mount verification and fstab persistence; potential bootstrap and service start

Active-passive HA model

Failover expectation

Expect a brief customer-facing cutover on the order of minutes once failover is triggered.

  • Standby node already running and reachable via shared VIP/DNS

  • Automated promotion; explicit lease break for immediate ownership transfer

  • Fencing and cooldowns protect against split-brain and flapping

Cost considerations and tradeoffs

Single data disk self-healing: lower cost, higher outage window

Cost profile

Cheaper: one production VM, one data disk, and no HA control-plane or shared storage stack.

Tradeoff: materially longer, customer-visible recovery because every infra fault becomes a rebuild of the only serving node.

Active-passive HA: higher cost, much better continuity

Cost profile

  • Second production VM

  • Shared ANF storage (replacing local managed disk)

  • HA lock storage/blobs and shared HA VIP + DNS

  • Monitoring/action groups + failover function

  • More coordination logic and moving parts

Benefit: significantly lower RTO, graceful failover, and far less customer-visible downtime during node failures or maintenance.

What customer / operational problems we are solving

  • Unplanned VM failure should not require manual rebuild — VMSS automation handles infra recovery; disk attach/mount is automated.

  • Production data must survive node loss — persistence on managed disk (single) or shared ANF (HA).

  • Endpoints should remain stable during failover — shared VIP and rhap-ha DNS preserve entry points.

  • Failover must be safe and controlled — fencing, leases, alert validation, and split-brain controls.

  • Passive nodes should be warm yet safe — lock service ensures single-writer ownership of shared state.

  • Operations should not hand-reattach disks post-replacement — Ansible automates metadata lookup, attach, discovery, mount, and fstab updates.

  • Clear packaging for budgets vs. uptime — distinct single vs. HA deployment tracks.

Single data disk self-healing is a strong baseline resiliency offer that removes manual recovery toil at low cost; downtime remains tied to rebuild.

Active-passive HA is a premium availability offer: shifts experience from “wait for rebuild” to “fail over to standby,” materially improving continuity.

Customer-facing comparison language

Single-node self-healing provides automated infrastructure recovery at lower cost. If the VM fails, the platform rebuilds the node and reattaches the data disk; recovery still depends on that rebuild, typically taking double-digit minutes.

Active-passive HA provides continuity, not just recovery. A warm standby is in place, traffic stays on the same endpoint, and failover completes in a couple of minutes while the failed node repairs in the background.

Sales-ready value proposition

Sales pitch

We offer two resilience tiers. Self-healing keeps cost down by rebuilding a failed node and automatically reattaching its data disk. High availability adds a live standby and a shared service endpoint, enabling recovery in minutes rather than waiting for a full rebuild. In short: self-healing reduces manual effort; HA protects continuity.

Short version

Self-healing = cost-optimized recovery. Active-passive HA = continuity-grade resilience.

Suggested usage guidance

  • Use the single data disk framing for cost efficiency, baseline resiliency, and reduced operational toil.

  • Use the active-passive HA framing for uptime expectations, continuity, RTO targets, and availability commitments.

  • Keep the core distinction consistent: self-healing is rebuild-based; HA is fast failover plus independent repair.