Patroni, Ubuntu 24.04, and two unplanned PostgreSQL failovers
I usually run PostgreSQL on managed services, like Azure Database for PostgreSQL Flexible Server but for once I was running PostgreSQL/Patroni on VMs. An overnight data-loading operation had been running for eight and a half hours when its PostgreSQL connections were lost. The client-side symptoms were:
SSL connection has been closed unexpectedly
An SSL error is a symptom, not a diagnosis. PostgreSQL had not run out of memory or disk. The server was being stopped by service management. Neither PostgreSQL nor Patroni needed a package upgrade, but an unattended zlib shared-library update caused the issue.
Ubuntu 24.04's needrestart automatically restarted affected services, including my custom Patroni service. The primary stopped, another node was promoted, and the load failed. Then the new primary received its own unattended updates and stopped too, causing a second failover.
What was running
This was a self-managed PostgreSQL 17/Patroni lab on Azure:
- Three PostgreSQL nodes: pg1, pg2 and pg3, across three availability zones.
- Three separate etcd nodes for Patroni's distributed configuration store.
- One application client VM.
- Canonical Ubuntu 24.04 LTS guests.
Hostnames, service prefixes, and local policy filenames below are consistently anonymized. Timestamps, package versions, and database messages are unchanged.
The deployment record identifies the image as Canonical:ubuntu-24_04-lts:server:latest, resolved to 24.04.202610210, with Azure patchMode=ImageDefault and assessmentMode=ImageDefault. Those settings did not disable maintenance already configured inside Ubuntu.
Root cause analysis
I correlated PostgreSQL, Patroni, and systemd messages from the same journal. PostgreSQL's stderr was captured under the Patroni unit here. A patroni[PID] journal prefix does not mean every message was generated by Patroni itself: the PostgreSQL LOG and FATAL messages identify the database side.
On pg1:
sudo journalctl --utc --no-pager -o short-iso \
--since '2026-11-04 06:18:50 UTC' \
--until '2026-11-04 06:20:20 UTC' \
-u franckpachot-lab-patroni.service |
grep -E 'Stopping |Stopped |Started |shutdown request|administrator command'
Selected captured lines, with repeated connection-termination messages omitted:
2026-11-04T06:19:04+00:00 franckpachot-lab-pg1-kc-20261103 patroni[46600]: 2026-11-04 06:19:04 UTC-6aea534b.b608-LOG: received smart shutdown request
2026-11-04T06:19:04+00:00 franckpachot-lab-pg1-kc-20261103 patroni[46623]: 2026-11-04 06:19:04 UTC-6aea534c.b61f-FATAL: terminating connection due to administrator command
2026-11-04T06:19:04+00:00 franckpachot-lab-pg1-kc-20261103 systemd[1]: Stopping franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab...
2026-11-04T06:19:06+00:00 franckpachot-lab-pg1-kc-20261103 patroni[46600]: 2026-11-04 06:19:06 UTC-6aea534b.b608-LOG: received fast shutdown request
2026-11-04T06:20:14+00:00 franckpachot-lab-pg1-kc-20261103 systemd[1]: Stopped franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab.
2026-11-04T06:20:14+00:00 franckpachot-lab-pg1-kc-20261103 systemd[1]: Started franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab.
This is positive evidence of an administered shutdown and a service restart, not merely an inference from lost connections. "Administrator command" does not identify a human: automated service management can produce it.
There is a service-unit detail worth checking as well:
systemctl show franckpachot-lab-patroni.service \
-p KillMode -p KillSignal
The follow-up inspection returned:
KillMode=control-group
KillSignal=15
My custom unit had no explicit KillMode setting. With control-group, systemd's stop signal is not restricted to the Patroni parent process. Service-unit shutdown semantics matter too. I should not assume that restarting the supervisor leaves PostgreSQL untouched.
Prove that unattended upgrades actually ran
The familiar status command was a useful starting point:
systemctl status unattended-upgrades.service --no-pager
Before remediation, it showed:
Loaded: loaded (/usr/lib/systemd/system/unattended-upgrades.service; enabled; preset: enabled)
Active: active (running) since Tue 2026-11-03 16:25:04 UTC; 15h ago
Main PID: 1641 (unattended-upgr)
/usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
This does not prove packages were being installed at that moment. That particular process waits to coordinate shutdown with unattended upgrades. The decisive evidence comes from the scheduled service and transaction logs.
On pg1:
sudo journalctl --utc --no-pager -o short-iso \
--since '2026-11-04 06:10:00 UTC' \
--until '2026-11-04 06:25:00 UTC' \
-u apt-daily-upgrade.service
Selected captured lines:
2026-11-04T06:18:58+00:00 franckpachot-lab-pg1-kc-20261103 systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-04T06:20:31+00:00 franckpachot-lab-pg1-kc-20261103 systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
The unattended-upgrades log supplies the package list:
sudo grep -E '^2026-11-04 06:.*(Starting unattended|Packages that will be upgraded|All upgrades installed)' \
/var/log/unattended-upgrades/unattended-upgrades.log
The first transaction's relevant lines:
2026-11-04 06:18:58,776 INFO Starting unattended upgrades script
2026-11-04 06:19:02,465 INFO Packages that will be upgraded: bind9-dnsutils bind9-host bind9-libs libbrotli1 libmspack0t64 libnss3 libpcre2-8-0 zlib1g
2026-11-04 06:20:30,481 INFO All upgrades installed
This was automated package maintenance in the exact incident window.
What package changed, and when?
The ordinary dpkg database log gives timestamped package actions:
sudo grep -E '^2026-11-04 06:19:.*(upgrade zlib1g|configure zlib1g)' \
/var/log/dpkg.log
Captured output:
2026-11-04 06:19:03 upgrade zlib1g:amd64 1:1.3.dfsg-3.1ubuntu2.2 1:1.3.dfsg-3.1ubuntu2.3
2026-11-04 06:19:03 configure zlib1g:amd64 1:1.3.dfsg-3.1ubuntu2.3 <none>
This proves dpkg performed the package upgrade/configuration. It is not a process-audit record of dpkg's full command-line arguments.
Where did the systemctl restart command come from?
Not from /var/log/dpkg.log. The literal command appears in the captured package-manager output:
sudo sed -n \
'/^Log started: 2026-11-04 *06:19:02/,/^Log ended:/p' \
/var/log/unattended-upgrades/unattended-upgrades-dpkg.log
Selected lines from that transaction:
Log started: 2026-11-04 06:19:02
Preparing to unpack .../zlib1g_1%3a1.3.dfsg-3.1ubuntu2.3_amd64.deb ...
Setting up zlib1g:amd64 (1:1.3.dfsg-3.1ubuntu2.3) ...
Restarting services...
systemctl restart azuremonitor-coreagent.service franckpachot-lab-host-metrics.service franckpachot-lab-patroni.service fwupd.service gcd.service polkit.service rsyslog.service ssh.service udisks2.service walinuxagent.service
Log ended: 2026-11-04 06:20:15
The custom Patroni service is explicitly named.
These logs also print commands under a deferred-restarts heading elsewhere. A bare grep for systemctl restart is not sufficient: inspect the heading and surrounding transaction. Here the Patroni command is under Restarting services..., and systemd independently records that exact
service stopping and starting.
The output does not timestamp each printed command. I assign 06:19:04 to the observed Patroni stop/PostgreSQL shutdown from the journal.
Together, the transaction output and systemd journal establish the restart path. I did not have an auditd/execve trace recording every parent PID and argument, so this is log correlation, not a reconstructed process audit.
Why would a zlib upgrade restart Patroni?
The installed APT hook answers that question:
cat /etc/apt/apt.conf.d/99needrestart
Its active configuration line was:
DPkg::Post-Invoke {"test -x /usr/lib/needrestart/apt-pinvoke && /usr/lib/needrestart/apt-pinvoke -m u || true"; };
The invoked script ultimately executes:
exec /usr/sbin/needrestart "$@"
needrestart checks running processes for outdated shared libraries. A service can therefore be selected without its own package being upgraded. It is not simply walking package dependencies.
The explanation was already installed on the VM:
cat /usr/share/doc/needrestart/README.Ubuntu
The first paragraph:
Starting from Ubuntu 24.04, needrestart in the default configuration will by default not show the Debconf prompt, and restart automatically the affected services. This includes when run as part of `unattended-upgrades` to ensure that the security updates it brings are actually applied to running services
rather than leaving the system vulnerable.
This is a documented Ubuntu 24.04 behavior, not a claim about every Ubuntu release or every customized Azure image. Canonical describes the change and its security rationale in
needrestart changes in Ubuntu 24.04: service restarts. While this may be great for services that are resilient to restarts, it's probably not a good idea for a monolithic database such as PostgreSQL, where restarting the primary interrupts all client sessions.
Updating a library on disk does not update the copy already mapped into a running process. Restarting affected services applies the fix. The missing piece in my setup was coordinating that legitimate maintenance with HA and a long-running load.
But why would Patroni have zlib loaded?
A direct import zlib in Patroni's own source is not required. Patroni runs inside Python, and imported modules or native dependencies can load shared libraries transitively. Python's zlib extension is one possible route: modules such as gzip and zipfile can use it. A native dependency of another
component in the service is another possibility.
However, being a Python process does not automatically mean that zlib is loaded at startup. On typical distribution builds, the Python zlib module is a dynamically loaded extension, not evidence that the interpreter itself unconditionally links and loads libz. The actual imports and native dependencies matter.
needrestart examines running processes, including their memory mappings, to identify obsolete library files and associate affected processes with services. It does not need to prove that application code is actively calling the library. A process can keep an old library mapped after an upgrade replaces the on-disk file.
There is another distinction: the Patroni systemd service also contains PostgreSQL processes. Selecting that unit does not, by itself, prove that the Patroni Python parent, rather than a process it manages, held the obsolete mapping.
For a running service, investigate with:
pid=$(systemctl show franckpachot-lab-patroni.service -p MainPID --value)
sudo grep -E 'libz\.so|deleted' "/proc/$pid/maps"
systemctl status franckpachot-lab-patroni.service --no-pager
Inspect the mappings of the other processes in the service as well. Use ldd on the relevant executable or native extension to investigate library dependencies, remembering that ldd alone does not list every library loaded later at runtime. These are diagnostic commands, not captured historical output: after a restart, the old mappings are gone.
The retained evidence establishes: zlib upgrade -> immediate restart list containing Patroni -> observed service shutdown -> failover. It does not include a pre-restart memory-map snapshot identifying the precise importing module or process, so I would not present a particular Python dependency chain as proven.
Nor would I explain it as zlib enabling compressed PostgreSQL connections: PostgreSQL's standard client/server protocol does not provide general-purpose compression of that kind. Any proposed libpq dependency path needs evidence from the installed build, not that assumption.
What about other releases and distributions?
There are three separate policy questions:
- Are updates checked or downloaded automatically?
- Are packages actually installed automatically?
- Which running services are restarted afterward, and by whom?
A timer being enabled does not answer all three. Neither does an
Automatic-Reboot setting.
Ubuntu before and after 24.04
Canonical's description of the 24.04 change explicitly contrasts it with the previous needrestart behavior:
- Before 24.04, the APT-integrated tool prompted for service restarts in interactive use. In noninteractive use, it listed affected services without restarting them. This is the pre-Noble behavior relevant when comparing older releases such as 20.04 and 22.04, where the tool is installed and configured that way.
- Ubuntu 24.04 changed that APT-hook behavior to restart affected services directly in both contexts, including unattended upgrades. That is the behavior documented and observed on these VMs.
- Other or later releases and customized images need their own checks. An upgraded host can retain configuration overrides, and a cloud image can add automation. Do not infer the effective policy from the release name alone.
The older behavior was not a promise that package upgrades could never restart a service: package maintainer scripts and other automation are separate paths. An older release also doesn't mean unattended security updates were disabled. The relevant change here is what happens to affected running processes after an APT transaction.
On an Ubuntu or Debian host, inspect rather than guess:
cat /etc/os-release
dpkg-query -W unattended-upgrades needrestart
apt-config dump | grep -E 'APT::Periodic|Unattended-Upgrade::Automatic-Reboot'
systemctl list-timers --all 'apt-*' --no-pager
cat /etc/apt/apt.conf.d/99needrestart
grep -R -n 'nrconf.*restart' /etc/needrestart
Debian, RHEL and Fedora
These are documented policy differences:
| System | Automatic package installation | What to check for service disruption |
|---|---|---|
| Debian, including 12/13 deployments |
unattended-upgrades is controlled by APT periodic configuration and installer/image choices. Do not assume every installation enables it. |
Inspect the installed needrestart version, mode, hooks, and package scripts. Sharing APT does not establish that Ubuntu's specific -m u policy applies. |
| RHEL 9 | Red Hat documents installing dnf-automatic and enabling a timer. By default, it downloads and reports updates without installing them. |
The selected timer matters: dnf-automatic-install.timer explicitly installs updates and overrides the corresponding configuration options. Inspect package scripts and any external maintenance tooling too. |
| Fedora 41 and newer, traditional DNF-based systems | Fedora documents that the automatic-update component is not installed by default; after installation, its default policy downloads without installing. The documented setup uses dnf5-plugin-automatic and an explicitly enabled timer. |
Check /etc/dnf/automatic.conf, the timer, and the restart/reboot policy. Desktop offline updates and image-based/Atomic systems are different update paths. |
Sources:
For the documented DNF-based setups, useful inspection commands include:
systemctl list-timers --all 'dnf*' --no-pager
grep -E '^[[:space:]]*(apply_updates|download_updates|reboot)[[:space:]]*=' \
/etc/dnf/automatic.conf
Defaults are often bad for critical systems such as monolithic databases. None of these host-local tools automatically knows which PostgreSQL node is the current leader or whether taking down another replica is safe. The HA maintenance policy still has to be designed for the full stack.
Every VM had its own maintenance window
The effective configuration enabled daily unattended upgrades:
cat /etc/apt/apt.conf.d/20auto-upgrades
apt-config dump | grep -E 'APT::Periodic::(Update-Package-Lists|Unattended-Upgrade)'
The configuration file contained:
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
The packaged timer definition, captured on all seven hosts, contained:
cat /usr/lib/systemd/system/apt-daily-upgrade.timer
[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer
[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true
[Install]
WantedBy=timers.target
The guests used UTC. Each machine independently scheduled maintenance around 06:00, with up to a 60-minute randomized delay. Persistent=true also permits catch-up after a missed timer activation. This is not a cluster-aware rolling
maintenance policy.
I checked the unattended-upgrades logs and dpkg logs on all seven VMs:
| VM | Unattended upgrades started, UTC | zlib upgrade, UTC | Observed effect on the owned workload service |
|---|---|---|---|
| etcd2 | 06:09:38.423 | 06:09:48 | No etcd restart recorded; process start remained November 3 |
| pg1 | 06:18:58.776 | 06:19:03 | Patroni stop at 06:19:04; first primary lost |
| pg3 | 06:19:28.899 | 06:19:44 | Patroni stop at 06:19:45; standby also taken down |
| etcd1 | 06:24:02.614 | 06:24:09 | No etcd restart recorded; process start remained November 3 |
| client | 06:26:39.054 | 06:26:44 | Host-metrics service in restart list; load had already failed |
| etcd3 | 06:35:02.180 | 06:35:20 | No etcd restart recorded; process start remained November 3 |
| pg2 | 06:38:01.956 | 06:38:16 | Patroni stop at 06:38:17; replacement primary lost |
All seven listed the same set of packages for upgrade. Every PostgreSQL node's dpkg-output log explicitly included franckpachot-lab-patroni.service in the immediate service-restart list.
"Around the same time" means the same maintenance hour, not simultaneous restarts. pg1 and pg3 stopped only 41 seconds apart. pg2 stopped 19 minutes and 13 seconds after pg1, by which point it had become the primary.
Random jitter spread the jobs out. It did not prevent maintenance from overlapping recovery or targeting the new leader.
Patroni and PostgreSQL logs show two failovers
Run this on each PostgreSQL VM:
sudo journalctl --utc --no-pager -o short-iso \
--since '2026-11-04 06:18:50 UTC' \
--until '2026-11-04 06:45:00 UTC' \
-u franckpachot-lab-patroni.service |
grep -E 'Stopping |Stopped |Started |promoted self|selected new timeline'
On pg2, the first promotion is explicit:
2026-11-04T06:19:29+00:00 franckpachot-lab-pg2-kc-20261103 patroni[47953]: 2026-11-04 06:19:29,047 INFO: promoted self to leader by acquiring session lock
2026-11-04T06:19:29+00:00 franckpachot-lab-pg2-kc-20261103 patroni[48105]: 2026-11-04 06:19:29 UTC-6aea538b.bbe9-LOG: selected new timeline ID: 23
pg3 was then stopped by its own maintenance:
2026-11-04T06:19:45+00:00 franckpachot-lab-pg3-kc-20261103 systemd[1]: Stopping franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab...
2026-11-04T06:21:15+00:00 franckpachot-lab-pg3-kc-20261103 systemd[1]: Started franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab.
Later, maintenance caught up with pg2, now the primary:
2026-11-04T06:38:17+00:00 franckpachot-lab-pg2-kc-20261103 systemd[1]: Stopping franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab...
Two seconds later, pg3 promoted:
2026-11-04T06:38:19+00:00 franckpachot-lab-pg3-kc-20261103 patroni[101738]: 2026-11-04 06:38:19,889 INFO: promoted self to leader by acquiring session lock
2026-11-04T06:38:19+00:00 franckpachot-lab-pg3-kc-20261103 patroni[101843]: 2026-11-04 06:38:19 UTC-6aeacedd.18dd3-LOG: selected new timeline ID: 24
The resulting leadership sequence was:
pg1 / timeline 22
-> pg2 / timeline 23 at 06:19:29
-> pg3 / timeline 24 at 06:38:19
Patroni uses the DCS leader lock to coordinate leadership. Stopping a leader can release the lock or expire the lease, allowing an eligible replica to acquire leadership. These logs prove the promotions and new timelines. They do not by themselves establish whether a particular lock was explicitly deleted or expired after ttl, as that would require additional
lease/key history.
For planned PostgreSQL restarts, Patroni provides patronictl restart and the REST /restart endpoint. Restarting the Patroni systemd service is not the same as a restart. A controlled maintenance plan may also require a deliberate switchover, replica checks, and application reconnection handling. HA does not make existing client sessions survive a primary shutdown.
Why "automatic reboot disabled" did not protect me
I initially looked at:
grep Reboot /etc/apt/apt.conf.d/50unattended-upgrades
It showed:
//Unattended-Upgrade::Automatic-Reboot "false";
// when Unattended-Upgrade::Automatic-Reboot is set to true
//Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
//Unattended-Upgrade::Automatic-Reboot-Time "02:00";
Those are commented examples/default documentation, not active assignments. Use apt-config dump to inspect effective overrides rather than treating a comment as the final configuration.
More importantly, a service restart is not a VM reboot. That option controls unattended-upgrades' automatic reboot behavior, not needrestart's service-restart behavior.
To check the distinction:
uptime -s
journalctl --list-boots --no-pager
last -x reboot | head
The captured pg1 boot time remained:
2026-11-03 16:24:54
The PostgreSQL VMs' boot times all preceded the incident. The observed 06:19 event was a service shutdown/restart, not a fresh guest boot.
Durability is not availability
Even with healthy replicas, sufficient retained/archived WAL, and successful node rejoin, these unplanned failovers would still be an availability incident. Recovering the data does not recover the application sessions that were connected to the stopped primary.
It helps to separate three questions:
- Durability / RPO: which acknowledged transactions survive?
- Availability / RTO: when can the application successfully execute work again?
- Session and transaction continuity: what happens to connections, in-flight transactions, and long-running operations?
WAL archiving is important for recoverability, but it does not preserve TCP connections or, by itself, guarantee zero data loss. The failover RPO depends on replication, commit settings, and which replica can safely be promoted. The point here is not a claim about lost committed data: even a zero-data-loss
failover can interrupt every session connected to the old primary.
A stable endpoint, virtual IP, or load balancer can direct new connections to the new leader. It cannot move an existing PostgreSQL session, its transaction, and its session state to another server. A proxy may keep a frontend socket open, but that is not a general guarantee that the backend transaction survives.
The logs show about 25 seconds between pg1's shutdown request and pg2's promotion, and about two seconds between pg2's shutdown and pg3's promotion. Those are server event intervals, not measured application downtime. Endpoint health checks, connection timeouts, pool recovery, retries, and transaction replay can extend the interruption.
For an application whose database sessions are terminated, that interruption is downtime, no matter how quickly a dashboard turns green again.
Resilience includes controlling avoidable failures
Three availability zones protect against some infrastructure failures. They do not isolate nodes from the same package update and the same host-local restart policy.
This was a correlated operational failure: the same maintenance mechanism ran independently on every VM, with no cluster-level gate. Random delay is not coordination. The second restart did not know that its target had just become the replacement primary.
Automatic failover remains valuable for actual failures. But it is a recovery mechanism, not permission to turn routine maintenance into repeated unplanned outages. The objective should be to avoid unnecessary failovers, not merely celebrate that promotion worked twice.
For resilience, I want both sides of the contract:
- Cluster-aware maintenance. Patch replicas in a controlled sequence, verify health and catch-up before proceeding, preserve enough capacity for the configured synchronous-replication policy, and coordinate any leader transition. Apply equivalent quorum-aware sequencing to etcd. One host at a time is a starting point, not proof that every quorum configuration can tolerate that host being unavailable.
- Application-aware recovery. Exercise connection-pool recovery and bounded retries with backoff. Retry the right transaction or resumable unit of work, not an arbitrary fragment of a transaction. Handle uncertain commit outcomes with idempotency or reconciliation rather than assuming that a lost connection means nothing committed.
- Application-level measurement. Test successful requests, error rates, latency, and time to resume useful work, not just leader election. Include maintenance scenarios where a replica is already unavailable.
- An explicit disruption budget. Define how many nodes may be unavailable and which checks must pass before continuing maintenance. Abort the rollout if recovery is incomplete rather than letting another timer fire.
A planned switchover can still disconnect clients. "Planned" does not mean "invisible." It makes the timing, sequencing, application handling, and expected impact deliberate. That is materially different from two independent package jobs unexpectedly stopping successive primaries.
What I changed to avoid the issue happening again
For this lab, I chose explicit maintenance windows rather
than independent automatic updates on seven machines.
First, I checked for active package transactions. I did not kill a running dpkg operation. Then I stopped, disabled, and persistently masked the named automatic-update units on all seven VMs:
sudo systemctl disable --now \
apt-daily.timer apt-daily-upgrade.timer unattended-upgrades.service
sudo systemctl mask \
apt-daily.timer apt-daily-upgrade.timer unattended-upgrades.service
I also set needrestart to list-only mode, so a later normal APT transaction would report affected services rather than automatically restarting them:
printf '%s\n' '$nrconf{restart} = "l";' |
sudo tee /etc/needrestart/conf.d/zz-franckpachot-lab-list-only.conf
The earlier targeted exclusions for the owned database, DCS, and supporting services remain in place. The broader list-only setting was an explicit subsequent decision, not the initial policy.
The rollout verified each host's effective configuration, not just the existence of a new file. Every host reported:
VERIFIED: effective needrestart restart mode=l (list only)
apt-daily.timer enabled=masked active=inactive
apt-daily-upgrade.timer enabled=masked active=inactive
unattended-upgrades.service enabled=masked active=inactive
To inspect unit state:
systemctl show apt-daily.timer apt-daily-upgrade.timer unattended-upgrades.service \
-p Id -p UnitFileState -p ActiveState
The effective needrestart configuration was verified by loading its main Perl configuration, which includes the sorted drop-ins, and checking that $nrconf{restart} resolves to l. On the installed version, an explicit restart-mode setting is honored by the APT hook's -m u mode.
I also persisted the policy in provisioning and added a pre-operation check across all seven VMs. It refuses to proceed if maintenance protection cannot be verified. Fixing running machines without fixing their replacement images would only postpone the same surprise.
This is not a recommendation to abandon security updates. It moves the responsibility to an explicit maintenance procedure:
- Agree on a maintenance window and the acceptable application interruption.
- Verify replication, DCS health, and recoverability.
- Patch in a controlled sequence, coordinating service restarts and any necessary switchover through Patroni.
- Verify the cluster and apply the restarts needed to use updated libraries.
- Reverify maintenance protection and application readiness afterward.
The masks block these named systemd activation paths. List-only mode blocks needrestart's default automatic restart path. Neither is a universal barrier against explicit mode overrides, package maintainer scripts, manual restarts, VM reboots, or separately configured Azure patch orchestration.
The lesson
The failure domain was not just a VM, a disk, or an availability zone. It was also the same maintenance policy installed independently on every node.
Ubuntu was applying security fixes as designed. Patroni performed failovers when primaries stopped. The missing piece was coordination: routine maintenance should have been a planned operation, not a sequence of independently triggered primary failures.
Durable data is essential, but it is not the whole HA contract. If every connection to the primary is dropped and applications must recover their work, availability has been affected. Resilience means reducing that impact, preventing avoidable disruptions, and electing a new leader.
A database service spans layers that make decisions independently. In this case, Ubuntu restarted a service after a package upgrade, and Patroni responded to the resulting primary shutdown with a failover. Each did what it was configured to do, but the disruption stemmed from the lack of coordination between them. Kubernetes-based deployments can encounter similar interactions between controllers, as I explored in an earlier CloudNativePG transient-failure test. A managed service may use some of the same components, but the key difference is whether maintenance, failover, and client recovery are designed and tested together—not whether one vendor manages the components.