How two small HAProxy health endpoints let a plain Layer 4 load balancer send writes to the PostgreSQL primary and reads to the replicas, with no floating IP and no single point of failure in the routing layer.
Key Takeaways
- On AWS, a keepalived floating IP does not carry over cleanly because VPCs do not support VRRP multicast, so a Patroni cluster needs another way to tell the AWS NLB which node is the primary.
- An NLB works at Layer 4 and does not know which PostgreSQL node is the primary.
- HAProxy runs on every PostgreSQL node and routes writes to the Patroni leader and reads to replicas, so any node forwards connections correctly.
- Two small HAProxy health endpoints, port 9000 for the primary and port 9001 for reads, tell the NLB each node’s role, so it sends writes straight to the primary.
- Applications use two connection strings: port 5000 for writes and reads that must see the latest data, and port 5001 for reads that can tolerate a small lag.
- pgBackRest copies backups from a standby and finds the current primary itself, so backups keep working after a failover.
- Each node’s
haproxy.cfgmust name that node in its health checks. Copying one file to every node silently breaks the NLB’s view of roles.
The Starting Point
Building a highly available PostgreSQL cluster with Patroni is a well-documented exercise. Patroni manages failover and leader election, etcd acts as the consensus store that decides which node holds the leader lock, streaming replication keeps the standbys current, and pgBackRest takes care of backups. There are plenty of guides for all of that.
The part that is far less documented is the question every application eventually asks: where do I connect? After a failover, the primary is on a different machine. The application should not need to know, and it certainly should not need a config change at 3 a.m.
On bare metal, the classic answer is a floating virtual IP managed by keepalived: whichever node is primary holds the IP, and when the primary moves, the IP moves with it. On AWS, that approach does not carry over cleanly. Keepalived relies on VRRP (Virtual Router Redundancy Protocol) multicast to coordinate the move, which AWS VPCs do not support. Making it work means switching to unicast VRRP and calling the EC2 API to reassign the address, which adds another moving part that has to work correctly in the middle of a failure.
I built this cluster for a production AWS environment, with the goal of a routing design that used only managed AWS components and the tools already in the stack. This post walks through that design. The full configuration is open source on my GitHub at Patroni-Postgres-Cluster.
The Architecture in One Picture
The cluster runs on six EC2 instances across two private subnets: three PostgreSQL 17 nodes, each running Patroni and HAProxy, and three dedicated etcd nodes. An internal AWS Network Load Balancer (NLB) is the single entry point for applications.
Client Application
|
AWS Network Load Balancer
|-- Port 5000 -> WRITES
|-- Port 5001 -> READS
|
HAProxy (on every PostgreSQL node)
|-- 5000 -> current Patroni leader only
|-- 5001 -> replicas, round-robin (primary if none)
|-- 9000 -> NLB health: 200 only if THIS node is primary
|-- 9001 -> NLB health: 200 if THIS node serves reads
|
PostgreSQL 17 on port 5432Each port in the design at a glance:
| Port | Runs on | What it does |
|---|---|---|
| 5000 | NLB and HAProxy | Writes, sent only to the current Patroni leader |
| 5001 | NLB and HAProxy | Reads, sent round-robin to replicas, or to the primary if no replica is up |
| 9000 | HAProxy | NLB health check for writes: returns 200 only on the primary |
| 9001 | HAProxy | NLB health check for reads: returns 200 on nodes that serve reads |
| 8008 | Patroni REST API | Answers /primary and /replica, which HAProxy uses to find each node’s role |
| 5432 | PostgreSQL 17 | The database itself |
Applications connect to the NLB on port 5000 for writes and port 5001 for reads. They never see an individual node address. Using two ports is deliberate, and it is the common pattern for Patroni clusters.
Replication here is asynchronous: a replica usually trails the primary by milliseconds, but that is not guaranteed, so a user who writes and immediately reads from a replica may not see their own change. Only the application knows which queries can tolerate that, so the application uses two connection strings: port 5000 for writes and any read that must see the latest data, such as a page shown right after a save, and port 5001 for reads that can tolerate a small lag, such as reports, dashboards and search. An application that does not need read scaling can simply send everything to port 5000.
For guaranteed read-your-writes on replicas, Patroni’s synchronous mode with synchronous_commit = remote_apply, plus routing reads to the /synchronous endpoint, closes the gap at the cost of higher write latency.
The etcd cluster runs on its own nodes and subnet on purpose. etcd is very sensitive to disk latency, and sharing a disk with PostgreSQL WAL writes is a known cause of false leader elections. Quorum comes from those three etcd nodes, not from the number of PostgreSQL nodes, so the cluster can lose any one etcd node and still elect a leader.
The Real Problem: The NLB Does Not Know Who the Primary Is
An NLB works at Layer 4. It forwards TCP connections to healthy targets, and it decides “healthy” with a health check. It knows nothing about PostgreSQL roles. If all three nodes are in the write target group and all three pass a simple port check, the NLB will happily send writes to a replica, which rejects them.
Patroni already knows the answer. Each node exposes a REST API on port 8008, and two endpoints are exactly what a router needs:
GET /primaryreturns 200 only on the current leader and 503 everywhere else.GET /replicareturns 200 on a healthy, running replica and 503 everywhere else.
The challenge is to get that role information to the NLB, and to route correctly even when the NLB gets it wrong.
Layer One: HAProxy Routes by Role
HAProxy runs on every PostgreSQL node, not on a separate tier, so there is no routing box that can fail on its own. Every HAProxy has the same three backends and checks every node’s Patroni API every three seconds:
Writes go only to the node whose Patroni API answers 200 on /primary:
backend primary_node
mode tcp
option httpchk GET /primary
http-check expect status 200
default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
server pg-node-1 PG_NODE_1_IP:5432 maxconn 300 check port 8008
server pg-node-2 PG_NODE_2_IP:5432 maxconn 300 check port 8008
server pg-node-3 PG_NODE_3_IP:5432 maxconn 300 check port 8008Only one server in primary_node can ever be up: the current leader. With inter 3s fall 3, HAProxy marks a failed node down within about nine seconds, and on-marked-down shutdown-sessions closes existing connections to it straight away instead of leaving applications hanging on a dead socket.
Reads prefer replicas and fall back to the primary only when no replica is up:
frontend pg_reads
bind *:5001
mode tcp
acl has_replica nbsrv(replica_nodes) gt 0
use_backend replica_nodes if has_replica
default_backend primary_reads_fallbackThe important property is that any HAProxy, on any node, routes correctly. If a write connection lands on a replica’s HAProxy, it is still forwarded to the primary. That property matters later.
Layer Two: Teaching the NLB the Node’s Role
HAProxy alone would work, but the NLB would still spread connections across all three nodes, and two out of three write connections would take an extra network hop through a replica. The better result is for the NLB itself to send writes only to the primary’s node.
NLB target groups support HTTP health checks on a custom port and path. So each HAProxy exposes two tiny HTTP endpoints that do nothing but answer “what is my role?” with a status code:
Port 9000: healthy only on the node that is currently primary:
frontend nlb_writes_healthcheck
bind *:9000
mode http
monitor-uri /health
monitor fail unless { srv_is_up(primary_node/THIS_NODE) }Port 9001: healthy on replicas, or on the primary only if no replica is up:
frontend nlb_reads_healthcheck
bind *:9001
mode http
monitor-uri /health
acl is_replica srv_is_up(replica_nodes/THIS_NODE)
acl is_primary srv_is_up(primary_node/THIS_NODE)
acl no_replicas nbsrv(replica_nodes) eq 0
monitor fail if !is_replica !is_primary
monitor fail if !is_replica is_primary !no_replicasThe trick is THIS_NODE. On pg-node-1 it is pg-node-1, on pg-node-2 it is pg-node-2, and so on. Each node asks HAProxy about its own server entry, which HAProxy already knows from the Patroni checks above. The result is that port 9000 returns 200 on exactly one machine, the primary, and 503 on the others. The write target group health-checks port 9000, the read target group health-checks port 9001, and the NLB now routes by role without ever talking to Patroni directly.
You can see what the NLB sees from any node:
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9000/health # 200 only on the primary
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9001/health # 200 on replicasSo there are two layers of protection. The NLB sends traffic only to the right node, and if it ever sends a connection somewhere else, HAProxy on that node still forwards it correctly.


Planning a similar setup on AWS? Our PostgreSQL Forward Deployed Engineers work alongside your team to design, build and test high availability that holds up in production.
What a Failover Looks Like
When the primary fails, the sequence is:
- The primary stops renewing its leader lock in etcd.
- Once the lock’s TTL expires (30 seconds by default), the surviving Patroni nodes hold an election, and the healthiest candidate with the least replication lag wins.
- The winner promotes PostgreSQL and starts answering 200 on
/primary. - Every HAProxy sees the change within a few seconds and sends writes to the new leader.
- Port 9000 on the new primary’s node turns 200, and the NLB marks it healthy once its health-check threshold is met.
- When the old primary comes back, Patroni demotes it, rewinds it with
pg_rewind, and it rejoins as a replica.
The design budget for an unplanned failover is the etcd TTL, plus HAProxy detection, plus promotion, plus the NLB’s health-check interval multiplied by its healthy threshold. A planned switchover with patronictl switchover is much faster, because Patroni hands over the lock directly instead of waiting for it to expire.
You can watch roles change during a test with:
patronictl -c /etc/patroni/patroni.yml listHas your failover ever been tested end to end? Our PostgreSQL Health Check reviews your architecture, replication and backups, and shows where your setup could break before it happens in production.
Backups That Stay Off the Primary
Routing is only half of availability; the other half is being able to recover. pgBackRest runs with backup-standby=y, so the heavy file copy is read from a standby rather than from the primary. Even so, pgBackRest still runs pg_backup_start() and pg_backup_stop() on the primary. The configuration lists all three PostgreSQL nodes, so pgBackRest finds the current primary at backup time and keeps working after a failover without config changes.
WAL is archived asynchronously (archive-async=y) through a local spool, so a slow S3 response does not hold up commits. Backups go to S3, with an optional MinIO repository as an encrypted, off-site second copy.
An Honest Section: What I Would Tell You Before You Copy This
Production ran two PostgreSQL nodes, not three. The environment this design came from had a primary and one replica, because a third EC2 instance was not available. No separate witness node was needed: Patroni runs the election, but the leader lock lives in the three dedicated etcd nodes, and their quorum decides who holds it. In effect, etcd is the witness. If the primary failed, Patroni on the replica could still take the leader lock from a healthy etcd quorum and promote itself. The cost was resilience after that point. Once a failover happened, or while either node was down for maintenance, the cluster had no standby at all, so a second failure would have meant an outage, and reads fell back to the primary. Three PostgreSQL nodes is the layout I recommend, and it is what the repository describes.
The per-node config is easy to get wrong. The backends in haproxy.cfg are identical on every node, which makes it tempting to copy one file everywhere. Do that, and every node’s port 9000 will report on pg-node-1, so the NLB’s view of roles is wrong. The routing still works, because of layer one, but you lose the benefit of layer two without any error to tell you.
The NLB “fails open”: if every target in a target group is unhealthy at once, an AWS NLB stops filtering and sends traffic to all targets. During a leader election, there is a short window when no node passes the port 9000 check. In that window, layer one is what keeps writes going to the right place. This is the strongest argument for running HAProxy on every node instead of letting the NLB route on its own.
Use IAM roles instead of static keys. As a general rule, applications running on AWS should get their AWS access from IAM roles wherever possible, rather than from access keys stored in config files. pgBackRest is a good example: instead of putting S3 keys in pgbackrest.conf, it can use the EC2 instance’s IAM role (repo1-s3-key-type=auto) and fetch short-lived credentials automatically. This means there are no credentials in a config file to leak, rotate or accidentally commit.
Closing Thought
None of the individual pieces described here (Patroni leader election, HAProxy role-based routing, the node-aware health endpoints, the NLB, and pgBackRest configured on every node) would have delivered high availability alone. What mattered was treating availability as a system, not a single component to install: asking “what happens to this when the primary moves?” of every part, including the jobs only the primary does, like WAL archiving and backup scheduling. Just as important was being honest about where the design was tested and where it still needs measuring. That combination, layered design plus honesty about what has and has not been proven, is what moves a cluster from “it fails over in the demo” to a design you can trust at 3 a.m.
If your team is running PostgreSQL on AWS and facing similar questions, such as connection routing after failover, backups that may not survive a node change, or an HA setup that has never been properly tested, this is exactly the kind of architecture work Stormatics does.
FAQ
Why not use keepalived and a floating IP for Patroni on AWS?
Keepalived relies on VRRP multicast to move the IP, which AWS VPCs do not support. Making it work means switching to unicast VRRP and calling the EC2 API to reassign the address, which adds another moving part that has to work correctly in the middle of a failure.
Can an AWS NLB send writes to the PostgreSQL primary on its own?
No. An NLB works at Layer 4 and knows nothing about PostgreSQL roles. In this design, an HTTP health check on HAProxy port 9000 returns 200 only on the primary, so the write target group sends traffic only to that node.
What happens if no node passes the NLB health check during a failover?
The NLB fails open: if every target in a target group is unhealthy at once, it sends traffic to all targets. HAProxy runs on every node and still forwards write connections to the primary, which is why it is the main safeguard during a leader election.
Do I need three PostgreSQL nodes for this design?
Two nodes can work, because the leader lock lives in the three dedicated etcd nodes and their quorum acts as the witness. But after a failover, or while a node is down for maintenance, a two-node cluster has no standby at all. Three PostgreSQL nodes is the recommended layout.
How do applications avoid reading stale data from a replica?
Send writes and any read that must see the latest data to port 5000, and only reads that can tolerate a small lag to port 5001. For guaranteed read-your-writes on replicas, Patroni’s synchronous mode with synchronous_commit = remote_apply closes the gap at the cost of higher write latency.
Do pgBackRest backups keep working after a failover?
Yes. The configuration lists all three PostgreSQL nodes, so pgBackRest finds the current primary at backup time. With backup-standby=y, the heavy file copy is read from a standby rather than from the primary.

