“The database is locked up.” We heard some version of that sentence more than once this past week, from the same team, about what looked like the same problem. It was not the same problem. Once, Postgres had genuinely stopped responding. Every other time, Postgres was fine, and a connection sitting in the application’s pool had quietly died somewhere between the app and the database.

Both failures produce the same page at 2am: the app cannot reach the database. Only one of them means the database is actually in trouble. Mixing the two up costs you the most expensive resource in an incident, which is the first ten minutes, when you are still deciding what kind of problem you have.

A true PostgreSQL lockup means the server itself has stopped responding at the operating system level. You cannot open a new session against it, from any client, from anywhere. A stale connection problem means Postgres is healthy and reachable. The trouble is a specific connection already sitting in your application’s pool, pointed at a socket that died somewhere along the way, usually without either side getting a clean signal that it happened.

What a true lockup actually looks like

This one is rare, and when it happens, it is unambiguous. Every attempt to open a fresh session hangs or refuses, including a brand new psql connection run directly on the box. It does not matter which application, which pool, or which network path you try. Nothing gets in.

The usual causes sit below Postgres itself: the OS is out of memory and the kernel is reclaiming pages, a storage volume has stopped responding to I/O, or a runaway process has pinned every CPU core hard enough that even accepting a new connection cannot get scheduled. Whatever the trigger, the signature is consistent. Existing sessions may still be limping along, but nothing new can start.

What a stale connection actually looks like

This one is far more common, and it hides well because it looks identical from the app’s point of view. Somewhere between your application and the database, a connection dies quietly. A load balancer decides the connection has been idle long enough and drops it. A NAT gateway ages an entry out of its connection table. A Kubernetes pod restarts and takes its network namespace with it. In every case, the TCP session ends without a clean FIN or RST reaching both sides.

Postgres, on its end, still believes that backend is alive and simply idle. Your application, on its end, still believes the pooled connection is good. The failure only shows up the next time someone tries to use it, and it shows up as an error, not a hang: connection reset by peer, or a driver-level message that the connection was already closed. New connections work fine the entire time. It is specifically the connections that were already established, sitting quietly in the pool, that fail.

Why Postgres cannot see this happening on its own

This is the detail that trips people up, and it is worth stating plainly because it is not a Postgres shortcoming. TCP has no built-in way for either side to be notified the instant a connection dies silently in the middle. Without some form of active probing, both sides can sit there believing the connection is fine indefinitely, right up until one of them tries to send data.

That is exactly what TCP keepalives exist to fix, and it is exactly why a connection pool needs its own health checking on top of whatever the database provides. Detecting a quietly dead connection is not the server’s job by default. It has to be built in deliberately, at more than one layer.

The five minute checklist

When the same kind of report comes in twice in a week, we stopped guessing and started running through this in order:

• Try a brand new connection directly on the database host. If that also hangs or refuses, you have a true lockup. Stop here and look at OS-level resources: memory, disk I/O, and CPU.
• If a new direct connection succeeds immediately, Postgres is not the problem. Move to the next check.
• Check whether the failure is limited to connections that were already established before the incident. If new connections through the same application path succeed while old pooled ones fail, that is the signature of a stale connection.
• Check the Postgres log and the application’s driver logs for reset by peer, broken pipe, or a closed-connection error at the exact time of the failure. A true lockup produces silence or timeouts. A stale connection produces a clean, immediate error once something finally tries to use the dead socket.
• Check pg_stat_activity for the affected backend. If the backend is still listed as idle with a normal state_change timestamp, the server never saw anything go wrong. That confirms the problem happened outside Postgres entirely.

That sequence usually gets you to the right answer in under five minutes, which matters far more than it sounds like it should when a page has just gone off.

Closing the gap at every layer

Once you know stale connections are your actual problem, the fix is about making sure something, somewhere, notices before the application does.

On the Postgres server. tcp_keepalives_idle, tcp_keepalives_interval, and tcp_keepalives_count all default to 0 in postgresql.conf, which tells Postgres to defer to the operating system’s defaults. On most Linux systems, that default is two hours before the first keepalive probe even goes out. For a production database sitting behind a load balancer with a much shorter idle timeout, two hours is not a keepalive, it is a formality. Set these explicitly. An idle time in the range of 60 to 120 seconds, with a handful of probes a short interval apart, keeps the connection genuinely alive from the server’s side rather than only on paper.

Postgres 14 also added client_connection_check_interval, which is worth knowing about and rarely configured. It has the server itself periodically check whether the client socket is still connected while a query is running, and free the backend early if it is not. It defaults to 0, meaning off. Turning it on gives the server a way to notice a dead client without waiting for a keepalive cycle to catch up, which matters most for long-running queries on connections that may not survive the wait.

In the connection pool. If you are on HikariCP, maxLifetime defaults to 30 minutes, and keepaliveTime, added in HikariCP 4.0, defaults to off. Set keepaliveTime to something comfortably shorter than maxLifetime, and shorter than the tightest idle timeout anywhere in the network path between the app and the database. That keeps a lightweight validation query flowing often enough that no middlebox in between ever considers the connection worth reaping.

If you are on PgBouncer, server_idle_timeout defaults to 600 seconds and governs how long an idle connection between PgBouncer and Postgres survives. client_idle_timeout, on the other hand, defaults to 0, disabled, which means PgBouncer will not proactively close an idle client-facing connection on its own unless you turn that on. A lot of teams assume PgBouncer handles this end to end by default. It does not, until you tell it to.

On the network path. Whatever sits between your application and the database, a load balancer, a NAT gateway, a service mesh, almost certainly has its own idle timeout, and it is usually shorter than people expect. AWS Application Load Balancers default to 60 seconds. AWS Network Load Balancers default to 350 seconds. The specifics vary by platform, but the pattern is the same everywhere: something in the middle will eventually decide a quiet connection is not worth keeping open. Your keepalive interval needs to be shorter than that number, not the other way around.

One clarification worth making here. idle_in_transaction_session_timeout and statement_timeout solve a different problem entirely. They control how long Postgres will tolerate a client that is connected and behaving badly, holding a transaction open or running a slow query. Neither one has anything to do with a connection that has already gone silently dead at the network layer. It is easy to reach for these settings because they sound related. They will not touch a stale connection problem.

Why the five minutes are worth protecting

For any team running Postgres under a real uptime commitment, whether that is a regulatory RTO in a regulated financial services environment or a customer-facing SLA at a growth-stage SaaS platform, the cost of an incident is rarely the fix itself. It is the time spent deciding what you are actually looking at. A true lockup and a stale connection both start the same way, with an application that suddenly cannot reach its database. Only one of them means the database needs attention. The other means a connection needs to be replaced, and the fix is a few milliseconds of reconnection, not an emergency.

The distinction takes five minutes to check once you know what to look for. Skipping it costs a lot more than five minutes, usually in the direction of paging the wrong person, restarting the wrong service, or escalating a problem that was never in the database at all.

We keep writing these up for the same reason we keep running the checklist ourselves. The failure mode is not rare, and the diagnostic is not obvious the first time you meet it. It is worth knowing cold before the page comes in, not while you are staring at it.