It was quite challenging to perform postgres major upgrade, e.g., v17 to v18, in a Percona operator/K8s environment. There was no convenient approach to accomplish this task. The only viable solutions left were a Manual Dump/Restore to a new environment or using Logical replication.  But these options were not free from challenges. Procuring a new … Continued

The post Beyond the Version Bump: Running PostgreSQL Major Upgrades in Percona Operator for PostgreSQL appeared first on Percona.

I’ve been a self-employed contractor for about a decade now, and I’m bummed about Harvest.

Harvest was a time-tracking and invoicing service. You’d track how many hours or weeks you spent on a project, put together an invoice, email it to the client, and close it out when they paid. There was a little Stripe integration for accepting credit and ACH payments on the invoice web view. A historical chart of total billed/paid amounts over time, and a table of invoices. Harvest was exactly what you want from small-business software: it solved a well-defined problem with a reliable, simple interface, and did so at a reasonable monthly price. They managed to avoid scope creep and never tried to become an accounting platform, or a blockchain for washing machines.¹ I loved Harvest and raved about it to every self-employed person I met.

Back in 2025 Harvest got bought out by Bending Spoons, the company that gutted Vimeo. I was worried they might ruin it, but was busy with other things and didn’t have time to look for new invoicing software.

Long story short, they raised my price from $13.32 to $133.98 per month by moving everyone to a new “enterprise” plan and “unlimited” usage billing model, which is AFAICT, exactly what they offered before for a tenth of the price. There’s also a “flex” plan which costs $12.65/mo… but that plan now skims an additional 2% on top of the usual ~3% card transaction fees. To be clear, that 3% pays for the whole credit card network, plus all the people working at Stripe and Mastercard, plus the financial risk of chargebacks and fraud, etc. Harvest isn’t doing any of that—it’s just a web form handing off data to Stripe.

I wrote to Harvest about this, and discovered that they don’t read customer emails any more. Instead they have a Large Language Model repeatedly vomit thousands of words at you. I’m no stranger to enshittification, but man, I did not anticipate the speed at which Bending Spoons would drive a perfectly good company into the ground.

Time to find a new invoicing service, I guess. Maybe bookkeeping too—Intuit’s Quickbooks Online seems to have completely jumped the shark. Any of y’all freelancers have a system you like?

In my recent blog post I remarked that database benchmarking is in a crisis because AI agents start to overfit on individual benchmarks with questionable techniques, which drives execution times towards zero. This has happened with TPC-D in the past, and broke the benchmark. TPC-H tries to fight that with rules, but most people who publish (unofficial) TPC-H results simply ignore the rules. In combination with overly eager AI agents, this leads to brutal overfitting.

We are seeing the next level of that with the GenDB-V system. The corresponding paper commands “Stop Handcrafting Your Engine! GenDB-V: Towards AI-Generated and Formally Verified Database Systems”, and reports speedups of 22× to 61× over Umbra and DuckDB, and all that in a fully verified system. These are indeed impressive claims, and would suggest that traditional database engines are in trouble.

Taking a step back, though, it should be clear that a performance improvement of a factor of 22 over Umbra is infeasible, at least if we are talking about traditional query processing. At SF10, the 22 TPC-H queries read roughly 15GB of (compressed) base data, and on a 9950X3D Umbra runs all queries in a total of roughly 540ms. On that machine the memory touched for data divided by the memory bandwidth (54GB/s) alone is about 280ms; anything below that is infeasible. Better compression might bring that down a bit, but not by an order of magnitude.

Unsurprisingly, the most dramatic gains of GenDB-V come from precomputation. The agents have carefully handcrafted auxiliary data structures for each query, which are computed at load time, and which avoid most of the work. The physical design is heavily tuned to the queries, too, for example, there is an extra structure that lists the supplier rows in s_acctbal DESC order, which makes Q2 trivial. Similar per-query data structures are built for most queries. And we see that in the performance numbers: Several queries already run in near-constant time at SF100: Q2 in 0.7 ms, Q17 in about 15 ms and Q4 in about 17 ms on one thread. That is the TPC-D symptom, without stored aggregates.

There is also a “plain-array” mode, which, at first glance, is more fair, as it avoids these excessive preparations and also reports performance improvements of just roughly a factor of 3, which seems achievable. But when looking closely, we see that here the agents did something even more insane: They hard-coded their knowledge about the data generator. They “know” that each customer k will be at row k-1. And they know that dbgen numbers the i-th order 32·(i/8) + i%8, so key k should be at row 8·(k/32) + k%32 − 1. Which makes the join in query Q18 super cheap.

How does all this fit together with the claim of a fully verified database engine? Well, the code does check if the hard-coded knowledge about the data generator holds in practice, e.g., by checking if the guessed join partner row is correct. If not, it falls back to a super slow generic nested loop implementation, but as that never happens with the official TPC-H data generator, that code path is irrelevant in the reported experiments.

Which brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless. They will do their best to find insane tricks to speed up this particular benchmark, but none of that would work in a real system. The high tuning and synthesis time aside, deleting a single tuple (or inserting one in the middle) can most likely cause the whole approach to fall off a cliff, as it forces the slow fallback path. Which is not something you want in a production system.

In middle school I could not get through long passages no matter how hard I tried. I would read an entire page and find that nothing had registered, so I started taking notes out of desperation. This was the late 80s, and Tony Buzan's mind maps had just reached Turkey. This led me to produce pages and pages of notes that looked like spiderwebs. Creating those notes have helped me tremendously. I made sense of what I was reading by figuring out which branch they would hang off in my mind maps. I realized my pages were not for storing the information as much as they were for processing it.

It took me another ten years to understand why this works so well. During a stretch of PhD procrastination, I read Manuel Blum's advice to graduate students:

You are all computer scientists.

You know what FINITE AUTOMATA can do.

You know what TURING MACHINES can do.

For example, Finite Automata can add but not multiply.

Turing Machines can compute any computable function.

Turing machines are incredibly more powerful than Finite Automata.

Yet the only difference between a FA and a TM is that

the TM, unlike the FA, has paper and pencil.

Think about it.

It tells you something about the power of writing.

Without writing, you are reduced to a finite automaton.

With writing you have the extraordinary power of a Turing machine.

This is a crisp technical claim about which problems you can solve with writing. If you take the paper away, you downgrade yourself by an entire complexity class!

I think this explains a lot about human history as well. Language let us externalize some of our thinking, and our species got smarter. Writing let us externalize even more of it, and we got smarter again. I suspect this is also why LLMs are as capable as they are: so much of our intelligence is already deposited in language that a model trained on it inherits a good deal of intelligence for free.

Out-of-core execution

When we write, we are externalizing our thinking onto paper, and that is what raises our capability to think bigger thoughts. I liken this to out-of-core execution, or external memory algorithms.

A good example of this for me is reviewing a paper. When I read a paper, I think I understand it, and then I sit down to write the review and find out I understood maybe 60% of it. While writing the review, I end up contextualizing the paper better, and get to understand it fully. And sometimes I even realize the ideas are bigger than the authors noticed, and they connect to many other things. 

Sometimes the unearthing goes further than understanding one paper. In 2018, I wrote a quick post about the anatomical similarities between Paxos and Nakamoto consensus. I had not started with that thesis originally. I started writing, and the connections got clearer as I went, and that post became the seed for one of our research papers. This is not uncommon, this is called freewriting. This is what happens when you give the machine a tape.

Writing is also the reason I remember anything. I refer back to my own blog constantly. Because I strained my brain writing those posts, rereading them re-fires the same neurons, and I reconstruct the understanding quickly. In this sense, the blog became my external memory. The thoughts I never wrote down do not stick well, because they were never fully thought.

If you think without writing, you only think you're thinking.

-- Leslie Lamport

Writing in the age of LLMs

Is writing for thinking getting obsolete now since you can now argue anything with a model that has read everything?

Well, the model has read everything except what is in your brain! It knows the literature, the standard framing, and the consensus. But it does not know your half-formed intuition about why the standard framing may be wrong for this particular case. AI made a lot of progress, but I don't trust an LLM to take the creative leap and make a new connection. I think that would require another AI breakthrough. So, if you open the chat immediately on anything, you lose the opportunity to write/think for yourself.

This is a real problem, and it seems to be spreading quickly. We have been celebrating Amazon's six-pager as a great example of writing-for-thinking. Bezos had said: "There is no way to write a six-page, narratively structured memo and not have clear thinking." Unfortunately, with LLMs, that sentence has stopped being true. You can now produce six pages with no thinking at all, and the six pages still look like evidence that the thinking happened. 

A model optimizing next-token probability cannot do your thinking. If you trust it to hold the pen, no thinking would get done. As everyone turns their thinking down and becomes a meat-proxy for generated text, if you still keep writing yourself, you would still be thinking for yourself. So writing is now a comparative advantage, and a widening one!

Anyways, enough about why you should write. Here is my advice on how you should write.

Rule 1: Mess up

The first rule of writing club is to produce in bulk with no restraints at all. Don't criticize, don't fix the typos, don't write in full sentences if you don't feel like it. The goal in this phase is to just accumulate text and get to a zeroth draft you can work with.

McPhee argues best for why getting anything down matters so much. Even if you had a bad writing session, it gave your mind something to work with. After you get something down, you can spot problems with it and think of better ways of phrasing things. But none of that happens without the drafted version to react against. 

This is also how research papers get written. Open a notes.org file and treat it as a lab notebook: write while you do the work, explore every tangent, produce without any pressure to be presentable. In the Köroğlu post I compared this to the foal fattened in a pitch-black stable before any training begins. The key part of that frame is the stable. During this phase you guard the idea from critics, including and especially yourself. Ideas start out fragile and easily squished. So suppress judgment for a while and see what the thing becomes.

Rule 2: Tidy up

Now you must become the critic, the editor. Forget that you wrote this. Imagine some other guy wrote it, and do a harsh review. Cut ruthlessly. Take delight in deleting text the way you would in deleting code. If you cannot bear to delete a good sentence, move it to a junk buffer at the bottom of the file so you can tell yourself you might bring it back. Spoiler: you won't.

Don't be discouraged by how bad the first draft was. McPhee says the way to do a piece of writing is three or four times over, never once. This is good news. Thinking in multiple drafts takes the pressure off: you don't have to be spontaneous, brilliant on the first pass, or finished today.

I think the reason writing works so well for thinking is that it slows you down while keeping you on the problem. In that extra time, the insights and connections get room to surface. Chunking/abstracting happens in the background. Editing is then the magic trick where you pretend you had an organized thought all along. It is the packaging of the product.

Rule 3: Keep it simple, plain, and succinct

I am an Elements of Style guy. Strunk and White got it right, and their writing still reads fresh decades later. I adore Vonnegut too. His writing is simple as fuck and completely eloquent. It took a lot of balls to write that plainly. Complexity in prose is usually a place to hide sloppy thinking, or a way to weasel out of taking a stance. If you can't say it plainly, you probably haven't thought it through yet. And the way you think it through is by writing/editing.

Rule 4: Write without expectations

Write for yourself. The audience I have in mind is me. If others read it, that is a bonus, but the motivation has to be intrinsic, because the external rewards are far too delayed and too random to steer on. Writing like no one is watching also frees you up, and you can decide afterwards how much to share. Get your dopamine hit from the writing itself, not from people reading or sharing what you wrote, as that is very unreliable.

I have been writing and sharing on this blog since 2011, and around 2016 I started running into people at conferences who read it. I assumed it was a fluke, but it kept happening. Writing with a human voice turns out to get a lot of attention, and I am still surprised by how much. Paradoxically, you seem to get an audience only if you weren't doing it for the audience.

Rule 5: Put it on the schedule

You should stop trying to get inspiration to write and start scheduling it. Waiting for inspiration doesn't work, instead, putting your butt on seat and writing breeds good ideas for writing. 

I find that keeping a posting schedule for the blog keeps me in "a writing frame of mind", which is really "a noticing frame of mind". Many weeks I tell myself a topic isn't interesting, but I start writing about it anyway, and then, what do you know, I am startled by the fount of interesting things that surfaces.

I also feel relieved after writing something as the thought is out of my system, I have closure on it, and I have room to think new thoughts. If a week goes by without a post, I feel clogged. So put writing on the calendar and keep it there until it no longer needs to be on the calendar.

In my previous post I said that MongoDB published some impressive numbers for 9.0, but I couldn’t find any details about how the tests were done. I ended with “always run your own benchmarks”. So I figured I should follow my own advice. Let me be clear up front: this is a first, quick test, … Continued

The post MongoDB 9.0 vs. 8.0 vs. 8.3: A Quick First Look at Performance appeared first on Percona.

MySQL and DuckDB Unlike row-oriented databases, which store data for a single record sequentially on disk, favoring single-row lookups, columnar databases store all the values from a column contiguously in storage, allowing only the selected column to be read from disk. The default MySQL engine is InnoDB, which is ACID-compliant and great for OLTP (Online … Continued

The post WooCommerce Analytics with DuckDB appeared first on Percona.

Welcome back to Village News, our curated roundup of MySQL and database news. This week, AWS contributed a fix for large-transaction binlog stalls to community MySQL, and MariaDB launched Enterprise Platform 2027.

If you want to get these updates, just subscribe to the blog.

Enjoy!

MySQL News

Note: Aggregated

I usually run PostgreSQL on managed services, like Azure Database for PostgreSQL Flexible Server but for once I was running PostgreSQL/Patroni on VMs. An overnight data-loading operation had been running for eight and a half hours when its PostgreSQL connections were lost. The client-side symptoms were:

SSL connection has been closed unexpectedly

An SSL error is a symptom, not a diagnosis. PostgreSQL had not run out of memory or disk. The server was being stopped by service management. Neither PostgreSQL nor Patroni needed a package upgrade, but an unattended zlib shared-library update caused the issue.

Ubuntu 24.04's needrestart automatically restarted affected services, including my custom Patroni service. The primary stopped, another node was promoted, and the load failed. Then the new primary received its own unattended updates and stopped too, causing a second failover.

What was running

This was a self-managed PostgreSQL 17/Patroni lab on Azure:

• Three PostgreSQL nodes: pg1, pg2 and pg3, across three availability zones.
• Three separate etcd nodes for Patroni's distributed configuration store.
• One application client VM.
• Canonical Ubuntu 24.04 LTS guests.

Hostnames, service prefixes, and local policy filenames below are consistently anonymized. Timestamps, package versions, and database messages are unchanged.

The deployment record identifies the image as Canonical:ubuntu-24_04-lts:server:latest, resolved to 24.04.202610210, with Azure patchMode=ImageDefault and assessmentMode=ImageDefault. Those settings did not disable maintenance already configured inside Ubuntu.

Root cause analysis

I correlated PostgreSQL, Patroni, and systemd messages from the same journal. PostgreSQL's stderr was captured under the Patroni unit here. A patroni[PID] journal prefix does not mean every message was generated by Patroni itself: the PostgreSQL LOG and FATAL messages identify the database side.

On pg1:

sudo journalctl --utc --no-pager -o short-iso \
  --since '2026-11-04 06:18:50 UTC' \
  --until '2026-11-04 06:20:20 UTC' \
  -u franckpachot-lab-patroni.service |
  grep -E 'Stopping |Stopped |Started |shutdown request|administrator command'

Selected captured lines, with repeated connection-termination messages omitted:

2026-11-04T06:19:04+00:00 franckpachot-lab-pg1-kc-20261103 patroni[46600]: 2026-11-04 06:19:04 UTC-6aea534b.b608-LOG:  received smart shutdown request
2026-11-04T06:19:04+00:00 franckpachot-lab-pg1-kc-20261103 patroni[46623]: 2026-11-04 06:19:04 UTC-6aea534c.b61f-FATAL:  terminating connection due to administrator command
2026-11-04T06:19:04+00:00 franckpachot-lab-pg1-kc-20261103 systemd[1]: Stopping franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab...
2026-11-04T06:19:06+00:00 franckpachot-lab-pg1-kc-20261103 patroni[46600]: 2026-11-04 06:19:06 UTC-6aea534b.b608-LOG:  received fast shutdown request
2026-11-04T06:20:14+00:00 franckpachot-lab-pg1-kc-20261103 systemd[1]: Stopped franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab.
2026-11-04T06:20:14+00:00 franckpachot-lab-pg1-kc-20261103 systemd[1]: Started franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab.

This is positive evidence of an administered shutdown and a service restart, not merely an inference from lost connections. "Administrator command" does not identify a human: automated service management can produce it.

There is a service-unit detail worth checking as well:

systemctl show franckpachot-lab-patroni.service \
  -p KillMode -p KillSignal

The follow-up inspection returned:

KillMode=control-group
KillSignal=15

My custom unit had no explicit KillMode setting. With control-group, systemd's stop signal is not restricted to the Patroni parent process. Service-unit shutdown semantics matter too. I should not assume that restarting the supervisor leaves PostgreSQL untouched.

Prove that unattended upgrades actually ran

The familiar status command was a useful starting point:

systemctl status unattended-upgrades.service --no-pager

Before remediation, it showed:

Loaded: loaded (/usr/lib/systemd/system/unattended-upgrades.service; enabled; preset: enabled)
Active: active (running) since Tue 2026-11-03 16:25:04 UTC; 15h ago
Main PID: 1641 (unattended-upgr)
/usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal

This does not prove packages were being installed at that moment. That particular process waits to coordinate shutdown with unattended upgrades. The decisive evidence comes from the scheduled service and transaction logs.

On pg1:

sudo journalctl --utc --no-pager -o short-iso \
  --since '2026-11-04 06:10:00 UTC' \
  --until '2026-11-04 06:25:00 UTC' \
  -u apt-daily-upgrade.service

Selected captured lines:

2026-11-04T06:18:58+00:00 franckpachot-lab-pg1-kc-20261103 systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-04T06:20:31+00:00 franckpachot-lab-pg1-kc-20261103 systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.

The unattended-upgrades log supplies the package list:

sudo grep -E '^2026-11-04 06:.*(Starting unattended|Packages that will be upgraded|All upgrades installed)' \
  /var/log/unattended-upgrades/unattended-upgrades.log

The first transaction's relevant lines:

2026-11-04 06:18:58,776 INFO Starting unattended upgrades script
2026-11-04 06:19:02,465 INFO Packages that will be upgraded: bind9-dnsutils bind9-host bind9-libs libbrotli1 libmspack0t64 libnss3 libpcre2-8-0 zlib1g
2026-11-04 06:20:30,481 INFO All upgrades installed

This was automated package maintenance in the exact incident window.

What package changed, and when?

The ordinary dpkg database log gives timestamped package actions:

sudo grep -E '^2026-11-04 06:19:.*(upgrade zlib1g|configure zlib1g)' \
  /var/log/dpkg.log

Captured output:

2026-11-04 06:19:03 upgrade zlib1g:amd64 1:1.3.dfsg-3.1ubuntu2.2 1:1.3.dfsg-3.1ubuntu2.3
2026-11-04 06:19:03 configure zlib1g:amd64 1:1.3.dfsg-3.1ubuntu2.3 <none>

This proves dpkg performed the package upgrade/configuration. It is not a process-audit record of dpkg's full command-line arguments.

Where did the systemctl restart command come from?

Not from /var/log/dpkg.log. The literal command appears in the captured package-manager output:

sudo sed -n \
  '/^Log started: 2026-11-04  *06:19:02/,/^Log ended:/p' \
  /var/log/unattended-upgrades/unattended-upgrades-dpkg.log

Selected lines from that transaction:

Log started: 2026-11-04  06:19:02
Preparing to unpack .../zlib1g_1%3a1.3.dfsg-3.1ubuntu2.3_amd64.deb ...
Setting up zlib1g:amd64 (1:1.3.dfsg-3.1ubuntu2.3) ...
Restarting services...
 systemctl restart azuremonitor-coreagent.service franckpachot-lab-host-metrics.service franckpachot-lab-patroni.service fwupd.service gcd.service polkit.service rsyslog.service ssh.service udisks2.service walinuxagent.service
Log ended: 2026-11-04  06:20:15

The custom Patroni service is explicitly named.

These logs also print commands under a deferred-restarts heading elsewhere. A bare grep for systemctl restart is not sufficient: inspect the heading and surrounding transaction. Here the Patroni command is under Restarting services..., and systemd independently records that exact
service stopping and starting.

The output does not timestamp each printed command. I assign 06:19:04 to the observed Patroni stop/PostgreSQL shutdown from the journal.

Together, the transaction output and systemd journal establish the restart path. I did not have an auditd/execve trace recording every parent PID and argument, so this is log correlation, not a reconstructed process audit.

Why would a zlib upgrade restart Patroni?

The installed APT hook answers that question:

cat /etc/apt/apt.conf.d/99needrestart

Its active configuration line was:

DPkg::Post-Invoke {"test -x /usr/lib/needrestart/apt-pinvoke && /usr/lib/needrestart/apt-pinvoke -m u || true"; };

The invoked script ultimately executes:

    exec /usr/sbin/needrestart "$@"

needrestart checks running processes for outdated shared libraries. A service can therefore be selected without its own package being upgraded. It is not simply walking package dependencies.

The explanation was already installed on the VM:

cat /usr/share/doc/needrestart/README.Ubuntu

The first paragraph:

Starting from Ubuntu 24.04, needrestart in the default configuration will by default not show the Debconf prompt, and restart automatically the affected services. This includes when run as part of `unattended-upgrades` to ensure that the security updates it brings are actually applied to running services
rather than leaving the system vulnerable.

This is a documented Ubuntu 24.04 behavior, not a claim about every Ubuntu release or every customized Azure image. Canonical describes the change and its security rationale in
needrestart changes in Ubuntu 24.04: service restarts. While this may be great for services that are resilient to restarts, it's probably not a good idea for a monolithic database such as PostgreSQL, where restarting the primary interrupts all client sessions.

Updating a library on disk does not update the copy already mapped into a running process. Restarting affected services applies the fix. The missing piece in my setup was coordinating that legitimate maintenance with HA and a long-running load.

But why would Patroni have zlib loaded?

A direct import zlib in Patroni's own source is not required. Patroni runs inside Python, and imported modules or native dependencies can load shared libraries transitively. Python's zlib extension is one possible route: modules such as gzip and zipfile can use it. A native dependency of another
component in the service is another possibility.

However, being a Python process does not automatically mean that zlib is loaded at startup. On typical distribution builds, the Python zlib module is a dynamically loaded extension, not evidence that the interpreter itself unconditionally links and loads libz. The actual imports and native dependencies matter.

needrestart examines running processes, including their memory mappings, to identify obsolete library files and associate affected processes with services. It does not need to prove that application code is actively calling the library. A process can keep an old library mapped after an upgrade replaces the on-disk file.

There is another distinction: the Patroni systemd service also contains PostgreSQL processes. Selecting that unit does not, by itself, prove that the Patroni Python parent, rather than a process it manages, held the obsolete mapping.

For a running service, investigate with:

pid=$(systemctl show franckpachot-lab-patroni.service -p MainPID --value)
sudo grep -E 'libz\.so|deleted' "/proc/$pid/maps"
systemctl status franckpachot-lab-patroni.service --no-pager

Inspect the mappings of the other processes in the service as well. Use ldd on the relevant executable or native extension to investigate library dependencies, remembering that ldd alone does not list every library loaded later at runtime. These are diagnostic commands, not captured historical output: after a restart, the old mappings are gone.

The retained evidence establishes: zlib upgrade -> immediate restart list containing Patroni -> observed service shutdown -> failover. It does not include a pre-restart memory-map snapshot identifying the precise importing module or process, so I would not present a particular Python dependency chain as proven.

Nor would I explain it as zlib enabling compressed PostgreSQL connections: PostgreSQL's standard client/server protocol does not provide general-purpose compression of that kind. Any proposed libpq dependency path needs evidence from the installed build, not that assumption.

What about other releases and distributions?

There are three separate policy questions:

1. Are updates checked or downloaded automatically?
2. Are packages actually installed automatically?
3. Which running services are restarted afterward, and by whom?

A timer being enabled does not answer all three. Neither does an
Automatic-Reboot setting.

Ubuntu before and after 24.04

Canonical's description of the 24.04 change explicitly contrasts it with the previous needrestart behavior:

• Before 24.04, the APT-integrated tool prompted for service restarts in interactive use. In noninteractive use, it listed affected services without restarting them. This is the pre-Noble behavior relevant when comparing older releases such as 20.04 and 22.04, where the tool is installed and configured that way.
• Ubuntu 24.04 changed that APT-hook behavior to restart affected services directly in both contexts, including unattended upgrades. That is the behavior documented and observed on these VMs.
• Other or later releases and customized images need their own checks. An upgraded host can retain configuration overrides, and a cloud image can add automation. Do not infer the effective policy from the release name alone.

The older behavior was not a promise that package upgrades could never restart a service: package maintainer scripts and other automation are separate paths. An older release also doesn't mean unattended security updates were disabled. The relevant change here is what happens to affected running processes after an APT transaction.

On an Ubuntu or Debian host, inspect rather than guess:

cat /etc/os-release
dpkg-query -W unattended-upgrades needrestart
apt-config dump | grep -E 'APT::Periodic|Unattended-Upgrade::Automatic-Reboot'
systemctl list-timers --all 'apt-*' --no-pager
cat /etc/apt/apt.conf.d/99needrestart
grep -R -n 'nrconf.*restart' /etc/needrestart

Debian, RHEL and Fedora

These are documented policy differences:

Sources:

• Debian's upgrade documentation,
• RHEL 9 automatic updates
• Fedora automatic updates.

For the documented DNF-based setups, useful inspection commands include:

systemctl list-timers --all 'dnf*' --no-pager
grep -E '^[[:space:]]*(apply_updates|download_updates|reboot)[[:space:]]*=' \
  /etc/dnf/automatic.conf

Defaults are often bad for critical systems such as monolithic databases. None of these host-local tools automatically knows which PostgreSQL node is the current leader or whether taking down another replica is safe. The HA maintenance policy still has to be designed for the full stack.

Every VM had its own maintenance window

The effective configuration enabled daily unattended upgrades:

cat /etc/apt/apt.conf.d/20auto-upgrades
apt-config dump | grep -E 'APT::Periodic::(Update-Package-Lists|Unattended-Upgrade)'

The configuration file contained:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

The packaged timer definition, captured on all seven hosts, contained:

cat /usr/lib/systemd/system/apt-daily-upgrade.timer

[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer

[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true

[Install]
WantedBy=timers.target

The guests used UTC. Each machine independently scheduled maintenance around 06:00, with up to a 60-minute randomized delay. Persistent=true also permits catch-up after a missed timer activation. This is not a cluster-aware rolling
maintenance policy.

I checked the unattended-upgrades logs and dpkg logs on all seven VMs:

All seven listed the same set of packages for upgrade. Every PostgreSQL node's dpkg-output log explicitly included franckpachot-lab-patroni.service in the immediate service-restart list.

"Around the same time" means the same maintenance hour, not simultaneous restarts. pg1 and pg3 stopped only 41 seconds apart. pg2 stopped 19 minutes and 13 seconds after pg1, by which point it had become the primary.

Random jitter spread the jobs out. It did not prevent maintenance from overlapping recovery or targeting the new leader.

Patroni and PostgreSQL logs show two failovers

Run this on each PostgreSQL VM:

sudo journalctl --utc --no-pager -o short-iso \
  --since '2026-11-04 06:18:50 UTC' \
  --until '2026-11-04 06:45:00 UTC' \
  -u franckpachot-lab-patroni.service |
  grep -E 'Stopping |Stopped |Started |promoted self|selected new timeline'

On pg2, the first promotion is explicit:

2026-11-04T06:19:29+00:00 franckpachot-lab-pg2-kc-20261103 patroni[47953]: 2026-11-04 06:19:29,047 INFO: promoted self to leader by acquiring session lock
2026-11-04T06:19:29+00:00 franckpachot-lab-pg2-kc-20261103 patroni[48105]: 2026-11-04 06:19:29 UTC-6aea538b.bbe9-LOG:  selected new timeline ID: 23

pg3 was then stopped by its own maintenance:

2026-11-04T06:19:45+00:00 franckpachot-lab-pg3-kc-20261103 systemd[1]: Stopping franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab...
2026-11-04T06:21:15+00:00 franckpachot-lab-pg3-kc-20261103 systemd[1]: Started franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab.

Later, maintenance caught up with pg2, now the primary:

2026-11-04T06:38:17+00:00 franckpachot-lab-pg2-kc-20261103 systemd[1]: Stopping franckpachot-lab-patroni.service - franckpachot-lab PostgreSQL Patroni quorum lab...

Two seconds later, pg3 promoted:

2026-11-04T06:38:19+00:00 franckpachot-lab-pg3-kc-20261103 patroni[101738]: 2026-11-04 06:38:19,889 INFO: promoted self to leader by acquiring session lock
2026-11-04T06:38:19+00:00 franckpachot-lab-pg3-kc-20261103 patroni[101843]: 2026-11-04 06:38:19 UTC-6aeacedd.18dd3-LOG:  selected new timeline ID: 24

The resulting leadership sequence was:

pg1 / timeline 22
  -> pg2 / timeline 23 at 06:19:29
  -> pg3 / timeline 24 at 06:38:19

Patroni uses the DCS leader lock to coordinate leadership. Stopping a leader can release the lock or expire the lease, allowing an eligible replica to acquire leadership. These logs prove the promotions and new timelines. They do not by themselves establish whether a particular lock was explicitly deleted or expired after ttl, as that would require additional
lease/key history.

For planned PostgreSQL restarts, Patroni provides patronictl restart and the REST /restart endpoint. Restarting the Patroni systemd service is not the same as a restart. A controlled maintenance plan may also require a deliberate switchover, replica checks, and application reconnection handling. HA does not make existing client sessions survive a primary shutdown.

Why "automatic reboot disabled" did not protect me

I initially looked at:

grep Reboot /etc/apt/apt.conf.d/50unattended-upgrades

It showed:

//Unattended-Upgrade::Automatic-Reboot "false";
// when Unattended-Upgrade::Automatic-Reboot is set to true
//Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
//Unattended-Upgrade::Automatic-Reboot-Time "02:00";

Those are commented examples/default documentation, not active assignments. Use apt-config dump to inspect effective overrides rather than treating a comment as the final configuration.

More importantly, a service restart is not a VM reboot. That option controls unattended-upgrades' automatic reboot behavior, not needrestart's service-restart behavior.

To check the distinction:

uptime -s
journalctl --list-boots --no-pager
last -x reboot | head

The captured pg1 boot time remained:

2026-11-03 16:24:54

The PostgreSQL VMs' boot times all preceded the incident. The observed 06:19 event was a service shutdown/restart, not a fresh guest boot.

Durability is not availability

Even with healthy replicas, sufficient retained/archived WAL, and successful node rejoin, these unplanned failovers would still be an availability incident. Recovering the data does not recover the application sessions that were connected to the stopped primary.

It helps to separate three questions:

• Durability / RPO: which acknowledged transactions survive?
• Availability / RTO: when can the application successfully execute work again?
• Session and transaction continuity: what happens to connections, in-flight transactions, and long-running operations?

WAL archiving is important for recoverability, but it does not preserve TCP connections or, by itself, guarantee zero data loss. The failover RPO depends on replication, commit settings, and which replica can safely be promoted. The point here is not a claim about lost committed data: even a zero-data-loss
failover can interrupt every session connected to the old primary.

A stable endpoint, virtual IP, or load balancer can direct new connections to the new leader. It cannot move an existing PostgreSQL session, its transaction, and its session state to another server. A proxy may keep a frontend socket open, but that is not a general guarantee that the backend transaction survives.

The logs show about 25 seconds between pg1's shutdown request and pg2's promotion, and about two seconds between pg2's shutdown and pg3's promotion. Those are server event intervals, not measured application downtime. Endpoint health checks, connection timeouts, pool recovery, retries, and transaction replay can extend the interruption.

For an application whose database sessions are terminated, that interruption is downtime, no matter how quickly a dashboard turns green again.

Resilience includes controlling avoidable failures

Three availability zones protect against some infrastructure failures. They do not isolate nodes from the same package update and the same host-local restart policy.

This was a correlated operational failure: the same maintenance mechanism ran independently on every VM, with no cluster-level gate. Random delay is not coordination. The second restart did not know that its target had just become the replacement primary.

Automatic failover remains valuable for actual failures. But it is a recovery mechanism, not permission to turn routine maintenance into repeated unplanned outages. The objective should be to avoid unnecessary failovers, not merely celebrate that promotion worked twice.

For resilience, I want both sides of the contract:

1. Cluster-aware maintenance. Patch replicas in a controlled sequence, verify health and catch-up before proceeding, preserve enough capacity for the configured synchronous-replication policy, and coordinate any leader transition. Apply equivalent quorum-aware sequencing to etcd. One host at a time is a starting point, not proof that every quorum configuration can tolerate that host being unavailable.
2. Application-aware recovery. Exercise connection-pool recovery and bounded retries with backoff. Retry the right transaction or resumable unit of work, not an arbitrary fragment of a transaction. Handle uncertain commit outcomes with idempotency or reconciliation rather than assuming that a lost connection means nothing committed.
3. Application-level measurement. Test successful requests, error rates, latency, and time to resume useful work, not just leader election. Include maintenance scenarios where a replica is already unavailable.
4. An explicit disruption budget. Define how many nodes may be unavailable and which checks must pass before continuing maintenance. Abort the rollout if recovery is incomplete rather than letting another timer fire.

A planned switchover can still disconnect clients. "Planned" does not mean "invisible." It makes the timing, sequencing, application handling, and expected impact deliberate. That is materially different from two independent package jobs unexpectedly stopping successive primaries.

What I changed to avoid the issue happening again

For this lab, I chose explicit maintenance windows rather
than independent automatic updates on seven machines.

First, I checked for active package transactions. I did not kill a running dpkg operation. Then I stopped, disabled, and persistently masked the named automatic-update units on all seven VMs:

sudo systemctl disable --now \
  apt-daily.timer apt-daily-upgrade.timer unattended-upgrades.service
sudo systemctl mask \
  apt-daily.timer apt-daily-upgrade.timer unattended-upgrades.service

I also set needrestart to list-only mode, so a later normal APT transaction would report affected services rather than automatically restarting them:

printf '%s\n' '$nrconf{restart} = "l";' |
  sudo tee /etc/needrestart/conf.d/zz-franckpachot-lab-list-only.conf

The earlier targeted exclusions for the owned database, DCS, and supporting services remain in place. The broader list-only setting was an explicit subsequent decision, not the initial policy.

The rollout verified each host's effective configuration, not just the existence of a new file. Every host reported:

VERIFIED: effective needrestart restart mode=l (list only)
apt-daily.timer enabled=masked active=inactive
apt-daily-upgrade.timer enabled=masked active=inactive
unattended-upgrades.service enabled=masked active=inactive

To inspect unit state:

systemctl show apt-daily.timer apt-daily-upgrade.timer unattended-upgrades.service \
  -p Id -p UnitFileState -p ActiveState

The effective needrestart configuration was verified by loading its main Perl configuration, which includes the sorted drop-ins, and checking that $nrconf{restart} resolves to l. On the installed version, an explicit restart-mode setting is honored by the APT hook's -m u mode.

I also persisted the policy in provisioning and added a pre-operation check across all seven VMs. It refuses to proceed if maintenance protection cannot be verified. Fixing running machines without fixing their replacement images would only postpone the same surprise.

This is not a recommendation to abandon security updates. It moves the responsibility to an explicit maintenance procedure:

1. Agree on a maintenance window and the acceptable application interruption.
2. Verify replication, DCS health, and recoverability.
3. Patch in a controlled sequence, coordinating service restarts and any necessary switchover through Patroni.
4. Verify the cluster and apply the restarts needed to use updated libraries.
5. Reverify maintenance protection and application readiness afterward.

The masks block these named systemd activation paths. List-only mode blocks needrestart's default automatic restart path. Neither is a universal barrier against explicit mode overrides, package maintainer scripts, manual restarts, VM reboots, or separately configured Azure patch orchestration.

The lesson

The failure domain was not just a VM, a disk, or an availability zone. It was also the same maintenance policy installed independently on every node.

Ubuntu was applying security fixes as designed. Patroni performed failovers when primaries stopped. The missing piece was coordination: routine maintenance should have been a planned operation, not a sequence of independently triggered primary failures.

Durable data is essential, but it is not the whole HA contract. If every connection to the primary is dropped and applications must recover their work, availability has been affected. Resilience means reducing that impact, preventing avoidable disruptions, and electing a new leader.

A database service spans layers that make decisions independently. In this case, Ubuntu restarted a service after a package upgrade, and Patroni responded to the resulting primary shutdown with a failover. Each did what it was configured to do, but the disruption stemmed from the lack of coordination between them. Kubernetes-based deployments can encounter similar interactions between controllers, as I explored in an earlier CloudNativePG transient-failure test. A managed service may use some of the same components, but the key difference is whether maintenance, failover, and client recovery are designed and tested together—not whether one vendor manages the components.

Percona Search for MongoDB 1.70.5-3 is now Generally Available and supported for production use. You can now easily run full-text, vector, and hybrid searches with automatic embedding on production Percona Server for MongoDB deployments. The release also adds the Hugging Face Inference API as a new automatic embedding provider. In this blog post, I cover why … Continued

The post General Availability of Percona Search for MongoDB appeared first on Percona.