Fenced active–standby

Enterprise HA is one active node and one standby serving the same workload. CitusIO does not ship a cluster daemon. An external orchestrator owns fencing, route movement, and membership. nginz-iotd exposes readiness, writer term, replay frontier, ring generation, and drain state.

A routable node is the whole local stack: nginx with CitusIO modules, nginz-iotd, optional nginz-iot-bridge, node-local rings, runtime files, and the configured PostgreSQL provider.

Route switch signal

route_to_node =
  node_ready &&
  shard_ready &&
  owns_current_writer_term &&
  replay_frontier_valid &&
  ring_generation_valid &&
  required_gates_valid &&
  !controlled_drain &&
  !stale_active_rejected

Heartbeat, process liveness, nginx reachability, or an open TCP port is not sufficient to move traffic.

Fencing

Production fencing requires stale-writer exclusion. Acceptable providers include:

• an external cluster manager with STONITH or equivalent isolation
• managed volume or storage reservation that excludes a stale writer
• Ceph RBD lock or equivalent storage-backed lock
• a validated NFSv4 profile only where the vendor guarantees stale-client exclusion

Local flock() is a development sequencing fixture. It is not a production fencing provider.

Takeover is blocked unless the orchestrator proves the old active is excluded. Prefer unavailability to split-brain.

The configured nginx iot_cluster_writer_term is a proposal. PostgreSQL lease ownership is the authority.

Failover outline

1. Confirm the active is serving and the standby is not in the data-plane backend set.
2. Capture redacted support bundles from both nodes.
3. Simulate active loss or withdraw the active from traffic.
4. Fence the active through the selected provider.
5. Acquire a newer writer term for the standby.
6. Start nginz-iotd and replay broker/store state to the required frontier.
7. Create fresh local SHM rings using the new term and generation.
8. Start configured bridge workers. Replay only term-valid durable outbox work.
9. Attach nginx modules to the fresh rings.
10. Wait for route_to_node=true.
11. Move the VIP, load balancer, DNS, or mesh route.
12. Reconnect MQTT clients and verify persistent session, retain, Will, and QoS 1 behavior.
13. Keep the old active out of service until it returns as standby under a later orchestrator decision.

Controlled handoff

Put the source shard into controlled drain before a planned move. Drain stops new assignment; it does not by itself fence the old writer or move the route.

Validate destination binary, config, schema, store compatibility, and capacity first. Capture support bundles from both nodes. After the destination reports route_to_node=true, move traffic externally and confirm the former active has lost readiness.

Current operating limits

Enterprise supports externally fenced active–standby operation. Active-active writes and transparent TCP migration are unavailable. Promotion requires fencing proof; writer ownership is granted by the external authority. Kubernetes can restart pods. It cannot synthesize replay frontier or own the writer lease.