Highly available PostgreSQL clusters are an essential component of modern database infrastructures. These clusters provide critical services to organizations that must ensure that their applications have reliable and continuous access to their databases. In such clusters, auto-failover is a crucial feature that ensures that the cluster continues to operate even when one of the nodes fails. However, network latency can pose significant challenges in auto-failover situations. This blog will discuss network latency’s challenges in auto failover situations in highly available PostgreSQL clusters, their causes, prevention, and resolution.

What is Auto Failover?

Auto failover is a feature of PostgreSQL that allows a standby node to take over the role of the primary node in the event of a failure. This ensures that the service remains available even if the primary node goes down. A highly available PostgreSQL cluster has at least two nodes: a primary node and a standby node. The primary node serves read and write requests from clients, while the standby node is kept in sync with the primary node and takes over in case of a failure.

What is Network Latency?

Network latency refers to the time delay that occurs when data is transmitted from one point to another over a network. It is measured as the time it takes for a packet of data to travel from the source to the destination and back. Network latency is influenced by many factors, such as the distance between the source and destination, the quality of the network infrastructure, and the number of hops between the nodes.

Causes

Network latency can be caused by various factors, including:

1. Network Congestion: When multiple users or applications try to access the database simultaneously, it can result in network congestion, causing delays in data transmission.
2. Distance: When the distance between the primary and standby nodes in a PostgreSQL cluster is significant, it can result in latency issues due to the physical distance between the nodes.
3. Network Infrastructure: The network infrastructure used in a PostgreSQL cluster can also impact network latency. Poor network hardware, configuration, and routing can cause delays in data transmission.
4. Database Configuration: The configuration of the PostgreSQL database can also impact network latency. Poorly configured databases can cause significant delays in data transmission, which can affect the performance of auto-failover.

Challenges of Network Latency in Auto Failover Situations

In a highly available PostgreSQL cluster, auto-failover is triggered when the primary node becomes unavailable. The standby node takes over the role of the primary node and begins serving read and write requests. However, network latency can cause a delay in the failover process, leading to service disruptions and data loss. Here are some of the challenges of network latency in auto-failover situations:

1. Split-brain: Network latency can cause the nodes to lose communication with each other. This can result in a split-brain scenario where each node believes it is the master and begins to accept writes. This can lead to data inconsistency, as both nodes are writing to different sets of data. You can read more about split-brain in this blog: Split-Brain in PostgreSQL Clusters – Causes, Prevention, and Resolution. 
2. Delayed replication: Network latency can cause delays in data replication, which can lead to data inconsistency. If a node fails, the data on the failed node needs to be replicated to the other nodes. However, if the replication is delayed due to network latency, the data on the failed node may not be up-to-date, resulting in data loss.
3. False positives: Network latency can cause the nodes to believe that a node has failed when it has not. This can trigger an unnecessary failover, which can result in downtime and data loss.

Prevention of Network Latency in Auto Failover Situations

To prevent network latency from causing service disruptions and data loss in auto failover situations, you can take the following measures:

1. Deploy Nodes in Proximity: Deploy the primary and standby nodes in proximity to reduce network latency. This will enable faster detection of primary node failure, quicker promotion of standby node, and faster data syncing.
2. Use a high-speed and low-latency network: Using a high-speed network can reduce network latency and prevent auto failover challenges.
3. Implement an alerting system: A monitoring system can detect and alert administrators of network latency issues. This can help prevent auto failover challenges.
4. Implement a quorum-based system: A quorum-based system ensures that a majority of the nodes need to agree on a failover before it is triggered. This can prevent false positives from triggering an unnecessary failover.
5. Optimize Network Configuration: Optimize the network configuration to reduce network latency. This includes configuring the network for low latency, reducing the number of hops between nodes, and minimizing network congestion.

Resolution of Network Latency in Auto Failover Situations

In the event of network latency causing service disruptions and data loss in auto failover situations, you can take the following measures to resolve the issue:

1. Restart the failed node: If the node failed due to network latency, restarting the node can resolve the issue.
2. Promote a standby node: If the failed node cannot be restarted, promoting a standby node can ensure that the cluster continues to operate.
3. Manual intervention: In some cases, manual intervention may be required to resolve the auto-failover challenges caused by network latency.

Conclusion

In conclusion, network latency is a significant challenge in auto-failover situations in highly available PostgreSQL clusters. It can cause delays in the detection of primary node failure, promotion of standby node, and syncing of data, leading to service disruptions and data loss. To prevent and resolve network latency issues, you should deploy nodes in proximity, use high-speed network infrastructure, optimize network configuration, increase network capacity, implement network monitoring, implement load balancing, and implement connection pooling. By taking these measures, you can ensure high availability and uninterrupted service in your PostgreSQL cluster.

A more abstract view of highly available PostgreSQL and its challenges is available in my previous blog: High availability made easy: A 100,000 ft view of auto failover in PostgreSQL.