In Part 1 we ran analytical reports in DuckDB on a DBTrail copy instead of on MySQL. This time I want to look at a different question: what if the application does not need to know the copy exists? Most MySQL servers have a few queries that make someone wait. The monthly report that aggregates … Continued

The post Heavy MySQL Reports in Seconds: Open Source HTAP with DBTrail and Percona Server for MySQL appeared first on Percona.

It was quite challenging to perform postgres major upgrade, e.g., v17 to v18, in a Percona operator/K8s environment. There was no convenient approach to accomplish this task. The only viable solutions left were a Manual Dump/Restore to a new environment or using Logical replication.  But these options were not free from challenges. Procuring a new … Continued

The post Beyond the Version Bump: Running PostgreSQL Major Upgrades in Percona Operator for PostgreSQL appeared first on Percona.

I’ve been a self-employed contractor for about a decade now, and I’m bummed about Harvest.

Harvest was a time-tracking and invoicing service. You’d track how many hours or weeks you spent on a project, put together an invoice, email it to the client, and close it out when they paid. There was a little Stripe integration for accepting credit and ACH payments on the invoice web view. A historical chart of total billed/paid amounts over time, and a table of invoices. Harvest was exactly what you want from small-business software: it solved a well-defined problem with a reliable, simple interface, and did so at a reasonable monthly price. They managed to avoid scope creep and never tried to become an accounting platform, or a blockchain for washing machines.¹ I loved Harvest and raved about it to every self-employed person I met.

Back in 2025 Harvest got bought out by Bending Spoons, the company that gutted Vimeo. I was worried they might ruin it, but was busy with other things and didn’t have time to look for new invoicing software.

Long story short, they raised my price from $13.32 to $133.98 per month by moving everyone to a new “enterprise” plan and “unlimited” usage billing model, which is AFAICT, exactly what they offered before for a tenth of the price. There’s also a “flex” plan which costs $12.65/mo… but that plan now skims an additional 2% on top of the usual ~3% card transaction fees. To be clear, that 3% pays for the whole credit card network, plus all the people working at Stripe and Mastercard, plus the financial risk of chargebacks and fraud, etc. Harvest isn’t doing any of that—it’s just a web form handing off data to Stripe.

I wrote to Harvest about this, and discovered that they don’t read customer emails any more. Instead they have a Large Language Model repeatedly vomit thousands of words at you. I’m no stranger to enshittification, but man, I did not anticipate the speed at which Bending Spoons would drive a perfectly good company into the ground.

Time to find a new invoicing service, I guess. Maybe bookkeeping too—Intuit’s Quickbooks Online seems to have completely jumped the shark. Any of y’all freelancers have a system you like?

In my recent blog post I remarked that database benchmarking is in a crisis because AI agents start to overfit on individual benchmarks with questionable techniques, which drives execution times towards zero. This has happened with TPC-D in the past, and broke the benchmark. TPC-H tries to fight that with rules, but most people who publish (unofficial) TPC-H results simply ignore the rules. In combination with overly eager AI agents, this leads to brutal overfitting.

We are seeing the next level of that with the GenDB-V system. The corresponding paper commands “Stop Handcrafting Your Engine! GenDB-V: Towards AI-Generated and Formally Verified Database Systems”, and reports speedups of 22× to 61× over Umbra and DuckDB, and all that in a fully verified system. These are indeed impressive claims, and would suggest that traditional database engines are in trouble.

Taking a step back, though, it should be clear that a performance improvement of a factor of 22 over Umbra is infeasible, at least if we are talking about traditional query processing. At SF10, the 22 TPC-H queries read roughly 15GB of (compressed) base data, and on a 9950X3D Umbra runs all queries in a total of roughly 540ms. On that machine the memory touched for data divided by the memory bandwidth (54GB/s) alone is about 280ms; anything below that is infeasible. Better compression might bring that down a bit, but not by an order of magnitude.

Unsurprisingly, the most dramatic gains of GenDB-V come from precomputation. The agents have carefully handcrafted auxiliary data structures for each query, which are computed at load time, and which avoid most of the work. The physical design is heavily tuned to the queries, too, for example, there is an extra structure that lists the supplier rows in s_acctbal DESC order, which makes Q2 trivial. Similar per-query data structures are built for most queries. And we see that in the performance numbers: Several queries already run in near-constant time at SF100: Q2 in 0.7 ms, Q17 in about 15 ms and Q4 in about 17 ms on one thread. That is the TPC-D symptom, without stored aggregates.

There is also a “plain-array” mode, which, at first glance, is more fair, as it avoids these excessive preparations and also reports performance improvements of just roughly a factor of 3, which seems achievable. But when looking closely, we see that here the agents did something even more insane: They hard-coded their knowledge about the data generator. They “know” that each customer k will be at row k-1. And they know that dbgen numbers the i-th order 32·(i/8) + i%8, so key k should be at row 8·(k/32) + k%32 − 1. Which makes the join in query Q18 super cheap.

How does all this fit together with the claim of a fully verified database engine? Well, the code does check if the hard-coded knowledge about the data generator holds in practice, e.g., by checking if the guessed join partner row is correct. If not, it falls back to a super slow generic nested loop implementation, but as that never happens with the official TPC-H data generator, that code path is irrelevant in the reported experiments.

Which brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless. They will do their best to find insane tricks to speed up this particular benchmark, but none of that would work in a real system. The high tuning and synthesis time aside, deleting a single tuple (or inserting one in the middle) can most likely cause the whole approach to fall off a cliff, as it forces the slow fallback path. Which is not something you want in a production system.

In middle school I could not get through long passages no matter how hard I tried. I would read an entire page and find that nothing had registered, so I started taking notes out of desperation. This was the late 80s, and Tony Buzan's mind maps had just reached Turkey. This led me to produce pages and pages of notes that looked like spiderwebs. Creating those notes have helped me tremendously. I made sense of what I was reading by figuring out which branch they would hang off in my mind maps. I realized my pages were not for storing the information as much as they were for processing it.

It took me another ten years to understand why this works so well. During a stretch of PhD procrastination, I read Manuel Blum's advice to graduate students:

You are all computer scientists.

You know what FINITE AUTOMATA can do.

You know what TURING MACHINES can do.

For example, Finite Automata can add but not multiply.

Turing Machines can compute any computable function.

Turing machines are incredibly more powerful than Finite Automata.

Yet the only difference between a FA and a TM is that

the TM, unlike the FA, has paper and pencil.

Think about it.

It tells you something about the power of writing.

Without writing, you are reduced to a finite automaton.

With writing you have the extraordinary power of a Turing machine.

This is a crisp technical claim about which problems you can solve with writing. If you take the paper away, you downgrade yourself by an entire complexity class!

I think this explains a lot about human history as well. Language let us externalize some of our thinking, and our species got smarter. Writing let us externalize even more of it, and we got smarter again. I suspect this is also why LLMs are as capable as they are: so much of our intelligence is already deposited in language that a model trained on it inherits a good deal of intelligence for free.

Out-of-core execution

When we write, we are externalizing our thinking onto paper, and that is what raises our capability to think bigger thoughts. I liken this to out-of-core execution, or external memory algorithms.

A good example of this for me is reviewing a paper. When I read a paper, I think I understand it, and then I sit down to write the review and find out I understood maybe 60% of it. While writing the review, I end up contextualizing the paper better, and get to understand it fully. And sometimes I even realize the ideas are bigger than the authors noticed, and they connect to many other things. 

Sometimes the unearthing goes further than understanding one paper. In 2018, I wrote a quick post about the anatomical similarities between Paxos and Nakamoto consensus. I had not started with that thesis originally. I started writing, and the connections got clearer as I went, and that post became the seed for one of our research papers. This is not uncommon, this is called freewriting. This is what happens when you give the machine a tape.

Writing is also the reason I remember anything. I refer back to my own blog constantly. Because I strained my brain writing those posts, rereading them re-fires the same neurons, and I reconstruct the understanding quickly. In this sense, the blog became my external memory. The thoughts I never wrote down do not stick well, because they were never fully thought.

If you think without writing, you only think you're thinking.

-- Leslie Lamport

Writing in the age of LLMs

Is writing for thinking getting obsolete now since you can now argue anything with a model that has read everything?

Well, the model has read everything except what is in your brain! It knows the literature, the standard framing, and the consensus. But it does not know your half-formed intuition about why the standard framing may be wrong for this particular case. AI made a lot of progress, but I don't trust an LLM to take the creative leap and make a new connection. I think that would require another AI breakthrough. So, if you open the chat immediately on anything, you lose the opportunity to write/think for yourself.

This is a real problem, and it seems to be spreading quickly. We have been celebrating Amazon's six-pager as a great example of writing-for-thinking. Bezos had said: "There is no way to write a six-page, narratively structured memo and not have clear thinking." Unfortunately, with LLMs, that sentence has stopped being true. You can now produce six pages with no thinking at all, and the six pages still look like evidence that the thinking happened. 

A model optimizing next-token probability cannot do your thinking. If you trust it to hold the pen, no thinking would get done. As everyone turns their thinking down and becomes a meat-proxy for generated text, if you still keep writing yourself, you would still be thinking for yourself. So writing is now a comparative advantage, and a widening one!

Anyways, enough about why you should write. Here is my advice on how you should write.

Rule 1: Mess up

The first rule of writing club is to produce in bulk with no restraints at all. Don't criticize, don't fix the typos, don't write in full sentences if you don't feel like it. The goal in this phase is to just accumulate text and get to a zeroth draft you can work with.

McPhee argues best for why getting anything down matters so much. Even if you had a bad writing session, it gave your mind something to work with. After you get something down, you can spot problems with it and think of better ways of phrasing things. But none of that happens without the drafted version to react against. 

This is also how research papers get written. Open a notes.org file and treat it as a lab notebook: write while you do the work, explore every tangent, produce without any pressure to be presentable. In the Köroğlu post I compared this to the foal fattened in a pitch-black stable before any training begins. The key part of that frame is the stable. During this phase you guard the idea from critics, including and especially yourself. Ideas start out fragile and easily squished. So suppress judgment for a while and see what the thing becomes.

Rule 2: Tidy up

Now you must become the critic, the editor. Forget that you wrote this. Imagine some other guy wrote it, and do a harsh review. Cut ruthlessly. Take delight in deleting text the way you would in deleting code. If you cannot bear to delete a good sentence, move it to a junk buffer at the bottom of the file so you can tell yourself you might bring it back. Spoiler: you won't.

Don't be discouraged by how bad the first draft was. McPhee says the way to do a piece of writing is three or four times over, never once. This is good news. Thinking in multiple drafts takes the pressure off: you don't have to be spontaneous, brilliant on the first pass, or finished today.

I think the reason writing works so well for thinking is that it slows you down while keeping you on the problem. In that extra time, the insights and connections get room to surface. Chunking/abstracting happens in the background. Editing is then the magic trick where you pretend you had an organized thought all along. It is the packaging of the product.

Rule 3: Keep it simple, plain, and succinct

I am an Elements of Style guy. Strunk and White got it right, and their writing still reads fresh decades later. I adore Vonnegut too. His writing is simple as fuck and completely eloquent. It took a lot of balls to write that plainly. Complexity in prose is usually a place to hide sloppy thinking, or a way to weasel out of taking a stance. If you can't say it plainly, you probably haven't thought it through yet. And the way you think it through is by writing/editing.

Rule 4: Write without expectations

Write for yourself. The audience I have in mind is me. If others read it, that is a bonus, but the motivation has to be intrinsic, because the external rewards are far too delayed and too random to steer on. Writing like no one is watching also frees you up, and you can decide afterwards how much to share. Get your dopamine hit from the writing itself, not from people reading or sharing what you wrote, as that is very unreliable.

I have been writing and sharing on this blog since 2011, and around 2016 I started running into people at conferences who read it. I assumed it was a fluke, but it kept happening. Writing with a human voice turns out to get a lot of attention, and I am still surprised by how much. Paradoxically, you seem to get an audience only if you weren't doing it for the audience.

Rule 5: Put it on the schedule

You should stop trying to get inspiration to write and start scheduling it. Waiting for inspiration doesn't work, instead, putting your butt on seat and writing breeds good ideas for writing. 

I find that keeping a posting schedule for the blog keeps me in "a writing frame of mind", which is really "a noticing frame of mind". Many weeks I tell myself a topic isn't interesting, but I start writing about it anyway, and then, what do you know, I am startled by the fount of interesting things that surfaces.

I also feel relieved after writing something as the thought is out of my system, I have closure on it, and I have room to think new thoughts. If a week goes by without a post, I feel clogged. So put writing on the calendar and keep it there until it no longer needs to be on the calendar.

In my previous post I said that MongoDB published some impressive numbers for 9.0, but I couldn’t find any details about how the tests were done. I ended with “always run your own benchmarks”. So I figured I should follow my own advice. Let me be clear up front: this is a first, quick test, … Continued

The post MongoDB 9.0 vs. 8.0 vs. 8.3: A Quick First Look at Performance appeared first on Percona.