High Availability (HA) refers to a system design approach that ensures a service remains accessible even in the event of hardware or software failures. In PostgreSQL, HA is typically implemented through replication, failover mechanisms, and clustering solutions to minimize downtime and ensure data consistency. Hence, HA is very important for your mission-critical applications. 

In this blog post, we will try to explore a critical failure condition known as a split-brain scenario that can occur in PostgreSQL HA clusters. We will first see what split-brain means, and then how it can impact PostgreSQL clusters, and finally discuss how to prevent it through architectural choices and tools available in the PostgreSQL ecosystem

What is a Split-Brain Scenario?

A split-brain scenario occurs when two or more nodes in a cluster lose communication with each other but continue operating as if they are the primary (or leader) node.

We know that the standby node receives data from the primary in real time. However, if a network glitch occurs between the primary and the standby, and the standby is unable to receive data for a certain timeout period, it may assume that the primary has failed. 

As a result, the standby might promote itself to a new primary. But since the issue was only a temporary network problem, the original primary is still active. This leads to a situation where two primary nodes are accepting writes at the same time, which is a highly dangerous state for the cluster.

Split-brain is particularly dangerous because it breaks the fundamental guarantees of consistency in an HA cluster. Writes can happen independently on multiple primary nodes, and reconciling those changes later is often impossible without data loss or manual intervention.

Common Scenarios That Lead to Split-Brain

As we have seen earlier, the most common cause of split-brain is a network failure between the primary and standby nodes. However, there are several other situations where a cluster might also encounter a split-brain scenario.

• Someone manually promotes a standby without ensuring the primary is actually down.
• Systems without quorum-based consensus (like etcd) are also vulnerable to split-brain.
• When using custom automation scripts for handling failovers, it’s equally important to implement error handling. Any errors or misconfigurations in these high availability management scripts, which are responsible for detecting failures and initiating failovers, can result in premature or incorrect failover decisions, potentially leading to a split-brain scenario.
• Two-node clusters are inherently more prone to split-brain. If communication breaks, each node may assume it’s safe to become/stay primary.

How to Avoid Split-Brain in PostgreSQL HA Setups

Here are several techniques that can help protect your cluster from split-brain scenarios.

Use a Consensus-Based Leader Election Mechanism

Use tools like Patroni and integrate with etcd to ensure a majority of nodes agree before promoting a new leader. These tools implement Raft or consensus algorithms, which drastically reduce the risk of split-brain.

To gain a clearer understanding, let’s examine the following example.

Consider a HA PostgreSQL cluster managed by Patroni with a backend ETCD key-value store used for consensus and leader election. The cluster consists of three nodes:

• Node A – Current Primary
• Node B – Standby1
• Node C – Standby2

All nodes are registered with a shared ETCD cluster used by Patroni for storing the leader key and coordinating failover decisions.

Let’s say a network partition causes Node A to lose connectivity to Nodes B and C, as well as to the ETCD cluster. The situation now looks like this:

• Node A: Isolated, no access to ETCD.
• Nodes B and C: Can communicate with each other and with the ETCD cluster.

This is how the cluster elects a new primary while avoiding a split-brain scenario.

• Nodes B and C, still having quorum (2 out of 3), detect that Node A’s leader key in ETCD has expired.
• They perform a Raft-based consensus via ETCD and decide to promote Node B to the new primary.
• Node B updates the leader key in ETCD to reflect its new role.
• Node A, being isolated and without ETCD access, cannot verify the current cluster state, and most importantly, cannot write to or renew the leader key.
• Patroni on Node A, recognizing the loss of quorum and ETCD connectivity, enters a pause or demotion state, preventing accidental promotion.
• Only Node B holds the leader key in ETCD and acts as the valid primary.
• Because Node B wins the quorum and successfully writes its new status to the ETCD leader key, Node C remains a standby node and continues replicating from the newly elected primary, Node B.
• Node A has been removed from the cluster or identified as faulty.

Hence, we have saved our cluster from a split-brain scenario successfully

Avoid Two-Node Architectures When Possible

Two-node setups cannot form a quorum and often rely on manual or semi-automatic failover logic, which is more prone to errors. Adding a lightweight witness node (which doesn’t need to store data) can help form a quorum without much overhead. Can reduce the risk of a split brain

Use a Cluster with an Odd Number of Nodes 

Clusters with three or more nodes (ideally an odd number) use quorum-based decision-making, where a majority (n/2 + 1) of nodes must agree to promote a new primary. 

• n is the total number of nodes in the cluster.
• n / 2 gives you half of the nodes.
• + 1 ensures you have a majority, not just half, meaning more than half the nodes must agree.

This avoids conflicting decisions and prevents multiple primaries from being elected

Use a Stable and Resilient Network for Cluster Communication

To avoid a split-brain scenario, it is also very important to maintain a stable and resilient network between all cluster nodes. Reliable network connectivity ensures that nodes can consistently communicate heartbeat signals and replication data without interruption.

Conclusion

Split-brain is a serious concern for the consistency and reliability of PostgreSQL HA clusters. It results from poor coordination between nodes and can lead to irreversible data loss. While PostgreSQL doesn’t inherently protect against split-brain scenarios, HA orchestration tools and quorum-based architectures can mitigate the risk effectively.