This has results for the sysbench benchmark on a 2-socket, 24-core server. A post with results from 8-core and 32-core servers is here.
tl;dr
(QPS for some version) / (QPS for base version)
This has results for MySQL versions 5.6 through 9.5 with the Insert Benchmark on a small server. Results for Postgres on the same hardware are here.
tl;dr
| dbms | l.i0 | l.x | l.i1 | l.i2 | qr100 | qp100 | qr500 | qp500 | qr1000 | qp1000 |
|---|---|---|---|---|---|---|---|---|---|---|
| 5.6.51 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 5.7.44 | 0.91 | 1.53 | 1.16 | 1.09 | 0.83 | 0.83 | 0.83 | 0.84 | 0.83 | 0.83 |
| 8.0.44 | 0.60 | 2.42 | 1.05 | 0.87 | 0.69 | 0.62 | 0.70 | 0.62 | 0.70 | 0.62 |
| 8.4.7 | 0.59 | 2.54 | 1.04 | 0.86 | 0.68 | 0.61 | 0.68 | 0.61 | 0.67 | 0.60 |
| 9.4.0 | 0.59 | 2.57 | 1.03 | 0.86 | 0.69 | 0.62 | 0.69 | 0.62 | 0.70 | 0.61 |
| 9.5.0 | 0.59 | 2.61 | 1.05 | 0.85 | 0.69 | 0.62 | 0.69 | 0.62 | 0.69 | 0.62 |
| dbms | l.i0 | l.x | l.i1 | l.i2 | qr100 | qp100 | qr500 | qp500 | qr1000 | qp1000 |
|---|---|---|---|---|---|---|---|---|---|---|
| 5.6.51 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 5.7.44 | 0.91 | 1.42 | 1.52 | 1.78 | 0.84 | 0.92 | 0.87 | 0.97 | 0.93 | 1.17 |
| 8.0.44 | 0.62 | 2.58 | 1.56 | 1.81 | 0.76 | 0.88 | 0.79 | 0.99 | 0.85 | 1.18 |
| 8.4.7 | 0.60 | 2.65 | 1.54 | 1.82 | 0.74 | 0.87 | 0.77 | 0.98 | 0.82 | 1.17 |
| 9.4.0 | 0.61 | 2.68 | 1.52 | 1.76 | 0.75 | 0.86 | 0.80 | 0.97 | 0.85 | 1.16 |
| 9.5.0 | 0.60 | 2.75 | 1.53 | 1.73 | 0.75 | 0.87 | 0.79 | 0.97 | 0.84 | 1.17 |
This has results for Postgres versions 12.22 through 18.1 with the Insert Benchmark on a small server.
Postgres continues to be boring in a good way. It is hard to find performance regressions.
tl;dr for a cached workload
This post has results for RocksDB on an Arm server. I previously shared results for RocksDB performance using gcc and clang. Here I share results using clang with LTO.
RocksDB is boring, there are few performance regressions.
tl;dr
Software
I used RocksDB versions 6.29, 7.0, 7.10, 8.0, 8.4, 8.8, 8.11, 9.0, 9.4, 9.8, 9.11 and 10.0 through 10.8.
I compiled each version clang version 18.3.1 with link-time optimization enabled (LTO). The build command line was:
flags=( DISABLE_WARNING_AS_ERROR=1 DEBUG_LEVEL=0 V=1 VERBOSE=1 )# for clang+LTOAR=llvm-ar-18 RANLIB=llvm-ranlib-18 CC=clang CXX=clang++ \make USE_LTO=1 "${flags[@]}" static_lib db_bench
I used a small Arm server from the Google cloud running Ubuntu 22.04. The server type was c4a-standard-8-lssd with 8 cores and 32G of RAM. Storage was 2 local SSDs with RAID 0 and ext-4.
Benchmark
Overviews on how I use db_bench are here and here.
The benchmark was run with 1 thread and used the LRU block cache.
Tests were run for three workloads:
Relative QPS
Many of the tables below (inlined and via URL) show the relative QPS which is:
(QPS for my version / QPS for RocksDB 6.29)
The base version varies and is listed below. When the relative QPS is > 1.0 then my version is faster than RocksDB 6.29. When it is < 1.0 then there might be a performance regression or there might just be noise.
The spreadsheet with numbers and charts is here. Performance summaries are here.
Results: byrx
This has results for by byrx workload where the database is cached by RocksDB.
RocksDB 10.x is faster than 6.29 for all tests.
Results: iobuf
This has results for by iobuf workload where the database is larger than RAM and RocksDB used buffered IO.
Performance in RocksDB 10.x is about the same as 6.29 except for overwrite. I think the performance decreases in overwrite that arrived in versions 7.x and 8.x are from new correctness checks and throughput in 10.8 is 7% less than in 6.29. The big drop for fillseq in 10.6.2 was from bug 13996.
Results: iodir
This has results for by iodir workload where the database is larger than RAM and RocksDB used O_DIRECT.
Performance in RocksDB 10.x is about the same as 6.29 except for overwrite. I think the performance decreases in overwrite that arrived in versions 7.x and 8.x are from new correctness checks and throughput in 10.8 is 7% less than in 6.29. The big drop for fillseq in 10.6.2 was from bug 13996.
I drive my daughter to school as part of a car pool. Along the way, I am learning a new language, Brainrot.
So what is brainrot? It is what you get when you marinate your brain with silly TikTok, YouTube Shorts, and Reddit memes. It is slang for "my attention span is fried and I like it". Brainrot is a self-deprecating language. Teens are basically saying: I know this is dumb, but I am choosing to speak it anyway.
What makes brainrot different from old-school slang is its speed and scale. When we were teenagers, slang spread by word of mouth. It mostly stayed local in our school hallways or neighborhood. Now memes go global in hours. A meme is born in Seoul at breakfast and widespread in Ohio by six seven pm. The language mutates at escape velocity and gets weird fast.
Someone even built a brainrot programming language. The joke runs deep, and is getting some infrastructure.
Here are a few basic brainrot terms you will hear right away.
I can anticipate the reactions to this post. Teenagers will shrugg: "Obviously. How is this news?" Parents of teens will laugh in recognition. Everyone else will be lost and move away.
I have seen things. I am not 50 yet, but I am getting there. I usually write about distributed systems and databases. I did not plan to write this post. This post insisted on being written through me. We will return to our regularly scheduled programming.
But here is my real point. I think the kids are alright.
It is not uncommon for a generation to get declared doomed by the one before it, yet Gen Z and Gen Alpha may have taken the heaviest hit, and written off as lost causes. But what I see is a generation with sharp self-mocking humor. They have short attention spans for things they do not care about. I think they do this out of sincerity. They don't see the purpose in mundane things, or and for many things theyfeel they lack enough agency. But what is new is how hard they can lock in (focus) on what they care about, and how fast they can form real bonds around shared interests. They are more open with each other. They are more inclusive by default. They are a feeling bunch. They waste no patience on things they find pointless. But when it matters, they show up fully.
From the outside, their culture may looks absurd and chaotic. But, under the memes, I see a group that feels deeply, adapts quickly, and learns in public. They are improvising in real time. And despite all predictions to the contrary, they might actually know what they are doing.
From a letter to Valve Corporation’s CEO Gabe Newell, lightly edited for formatting and links.
Dear Mr. Newell,
Steam has been my main source for games for over twenty years. I am disheartened that you chose not to publish Santa Ragione’s recently released game, Horses.
I’ve read some substantive critique; Polygon and Rock Paper Shotgun’s among them. I have also read Santa Regione’s discussion of the ban. I bought Horses on Itch and played it through; I found it enjoyable and thought provoking, though not transformative. I was surprised to find a much tamer experience than I had been led to believe. I am particularly concerned that Steam found this title dangerous enough to ban it.
Is Horses unsettling? Yes, though you would see far worse in popular horror films. I find Hostel or Saw gut-wrenching to near-unwatchable; Horses felt almost cute by comparison. It is nowhere near Argento’s classic Suspiria or Aster’s harrowing Midsommar, which deals with similar themes. The game’s pixelated censorship, silly animations, and cutting away from its worst moments comes across as coy, even camp. I suspect this is intentional: the game is in large part concerned with authoritarianism and the reproductive dynamics (in all senses) of cinema and games themselves. It is also concerned with complicity: the character’s choices to voice disgust or approval have apparently no impact on the story. Its four explicit themes—laid out in the embedded narrative of a VHS tape you must watch and decode to progress—are the repression of patriarchy, religion, chastity, and silence.
Steam has long been willing to publish works engaging with brutal dehumanization and authoritarian violence. Games like Amnesia: A Machine for Pigs or SOMA depict humans involuntarily warped, both physically and mentally, beyond recognition. Like Horses, Amnesia uses horror as a lens for the industrial revolution and the power of the wealthy. Valve’s own work has not shied away from horror; Half Life 2 is entirely about the visceral subjugation of political and bodily autonomy. Valve gave us the headcrab and the stalker—both instances of forcible objectification—and the game’s camera shies away from neither.
What, then, makes Horses unpublishable? Surely not violence, or you’d have pulled half the Steam catalogue by now. It can’t be body horror: I flinched far more at Dead Space 2’s eyeball scene. Could it be nudity? Half Life 2’s stalkers are fully nude and completely uncensored; I find the image of their mutilated bodies far more visually disturbing than the titular horses. Is it sex? Steam publishes the wildly popular Cyberpunk 2077, which has no shortage of breasts, vaginas, penises, and first-person sex scenes. It also depicts rape, torture, murder, and young boys hooded, restrained, and tortured as livestock on a farm. Could Horses be at fault for flagellation? Surely not; Steam published Robert Yang’s charming Hurt Me Plenty, where players directly spank a simulated consensusal partner. Is it complicity in authoriarian abuse? Lucas Pope’s highly-regarded Papers Please, also on Steam, puts players in the increasingly disturbing role of enforcing a fascist state’s border. It too contains pixelated nudity and violence.
I love cute, safe, and friendly games; Astroneer is my jam! And as an adult, I also enjoy challenging, even harrowing narratives. For me, Horses falls somewhere in the middle—one might call it the Animal Farm of fascist horror parables. I think Steam ought to publish it, and more transgressive works as well.
Yours truly,
Kyle Kingsbury
At AWS re:Invent 2025, I had the opportunity to give a brief demonstration at the New Relic booth that opened the stage to some AWS Heroes. In Linux environments, administrators often rely on CPU usage and load average metrics to determine whether an instance is appropriately sized. An oversized instance that sits idle wastes resources and drives up the cloud bill, whereas an undersized instance pushed to its limits can degrade application performance—ultimately impacting a company’s revenue.
To set the stage for my demo, I began with the Summary tab’s first graph, which displays CPU usage as a percentage. I started a simple CPU-bound task using the yes command, which continuously outputs the character "y". Then, I moved on to a more complex workload: fio with 32 threads performing synchronous, direct I/O disk writes:
fio --rw=write --ioengine=psync --direct=1 --bs=1M --numjobs=32 --name=test --filename=/tmp/x --size=10G --thread
One could expect the yes command to saturate the CPU, and this fio workload to be I/O-bound, but surprisingly, both tasks show 100% CPU usage:
The load average graph doesn't help and shows the 32 threads on the CPU:
We don't have more details on the load average. The Process tab shows yes, but with only low CPU usage, and not fio:
What could help detail the load average is the process state, not gathered here. The R state is real CPU usage, for running tasks, like the 12.5% of yes. The fio processes are mostly doing disk I/O in the D state, which doesn't utilize CPU, despite what is displayed here.
Back to the CPU usage graph, I can get more details. I open the query and instead of cpuPercent I display the details:
SELECT --average(cpuPercent),
average(cpuUserPercent), average(cpuSystemPercent), -- running
average(cpuStealPercent), -- hypervisor
average(cpuIdlePercent), average(cpuIOWaitPercent) -- idle (incl. wait I/O)
FROM SystemSample WHERE (entityGuid = 'NzM2NzA3MXxJTkZSQXxOQXw0MDAyNTY2MTYyODgwNDkyMzM0')
TIMESERIES AUTO SINCE 5 minutes ago UNTIL now
The 100% CPU usage was in "Steal" for the running yes command because I've run that on an overprovisioned virtual machine where the hypervisor gives only 1/4th of the CPU cycles, and was in "IO Wait" for fio when it was waiting for IO completion rather than running in CPU:
To explain this "IO Wait" and that it is just like "Idle", I've started yes again while fio was running and the "IO Wait" disappeared:
The reason is that "IO Wait" is accounted when a CPU is idle, because the process that was running waits on an IO call, and no other process had to run on CPU. Then, the process stays scheduled on the CPU with this state. But if another process comes to run in CPU then the "IO Wait" is not accounted anymore. The CPU usage (%) is not the state of the processes, but the state of the processor:
Back to the load average, the reason why the D state ("IO Wait") is accounted in addition to the R state ("User", "Sys" and "Steal") is visible in the loadavg.c code:
* The global load average is an exponentially decaying average of nr_running +
* nr_uninterruptible.
*
* Once every LOAD_FREQ:
*
* nr_active = 0;
* for_each_possible_cpu(cpu)
* nr_active += cpu_of(cpu)->nr_running + cpu_of(cpu)->nr_uninterruptible;
*
* avenrun[n] = avenrun[0] * exp_n + nr_active * (1 - exp_n)
However, the most important comment is the one that explains that it is a silly number:
/*
* kernel/sched/loadavg.c
*
* This file contains the magic bits required to compute the global loadavg
* figure. Its a silly number but people think its important. We go through
* great pains to make it work on big machines and tickless kernels.
*/
The load average metric traces back to early Unix systems (1970s), where it reflected how busy a system was by counting processes in the run queue. At that time, computers used periodic scheduling ticks—even while idle—to keep track of time and processes. Sysadmins would see a load average of 1 on a single CPU as one process running or waiting, thus equating it to CPU usage. Today, with multi-core processors, sophisticated scheduling, and tickless kernels, the load average is a far less reliable indicator of real‑time CPU usage and is often misunderstood without considering the real state of the processes.
The uninterruptible state D is not necessarily linked to disk activity. For example, asynchronous disk I/O operations that collect completed IO do not enter the D state. I demonstrated this by changing the fio job from psync to async, observing identical I/O throughput and rate but less "IO Wait" and a lower load average. Furthermore, some system calls can appear as "IO Wait" and increase the load average, even when they are idle and harmless. I also showed this by launching a thousand processes with {vfork();sleep(300);}.
The first step when using the New Relic dashboard for Linux is to replace 'cpuPercent' in the CPU usage charts with more detailed metrics:
cpuUserPercent and cpuSystemPercent for tasks running in CPU, respectively, for user (application) and system (kernel) code.cpuStealPercent, cpuIOWaitPercent, cpuIdlePercent for idle CPU, because the hypervisor didn't allow tasks to run (and steals CPU cycles), or no tasks has something to run (with or without a task waiting on uninterruptible call)Remember, the load average is not a reliable metric because it does not exclusively reflect actively running processes. It may also include various types of waits, but not all. The 'IO wait' percentage does not indicate high IO activity. Instead, it shows the CPU is idle while many processes are waiting for IO operations. In cloud environments where minimizing costs by reducing idle CPU time is crucial, you should focus on CPU usage for user and system processes, excluding waiting tasks, when sizing an instance. An inexperienced user might misinterpret the 100% usage shown in my fio demo as a sign the instance is too small, while in fact, it's the opposite.
It is that time of year again to look back on a year of posts. I average about sixty posts annually. I don't explicitly plan for the number, and I sometimes skip weeks for travel or work, yet I somehow hit the number by December. Looking back, I always feel a bit proud. The posts make past Murat look sharp and sensible, and I will not argue with that. Here are some of the more interesting pieces from the roughly sixty posts of 2025.
Looks like I wrote several advice posts this year. I must be getting old.
The Invisible Curriculum of Research
What I'd do as a College Freshman in 2025
What makes entrepreneurs entrepreneurial?
Publish and Perish: Why Ponder Stibbons Left the Ivory Tower
Use of Time in Distributed Databases (part 5): Lessons learned (Link to the index)
Morty: Scaling Concurrency Control with Re-Execution
Serializable Isolation for Snapshot Databases
We started a live paper reading series with Aleksey Charapko. It has been a lot of fun and provably a better way to read papers. My summaries/reviews of the papers we read together are more insightful than the papers I read solo. The first seven paper reviews below are from our live reading sessions.
Asymmetric Linearizable Local Reads
Cabinet: Dynamically Weighted Consensus Made Fast
Can a Client–Server Cache Tango Accelerate Disaggregated Storage?
Real Life Is Uncertain. Consensus Should Be Too!
Vive la Difference: Practical Diff Testing of Stateful Applications
Mitigating Application Resource Overload with Targeted Task Cancellation
Tiga: Accelerating Geo-Distributed Transactions with Synchronized Clocks
Analyzing Metastable Failures in Distributed Systems
Disaggregation: A New Architecture for Cloud Databases
Disaggregated Database Management Systems
Taurus Database: How to be Fast, Available, and Frugal in the Cloud
ATC/OSDI’25 Technical Sessions
Of course AI!
Barbarians at the Gate: How AI is Upending Systems Research
Supporting our AI overlords: Redesigning data systems to be Agent-first
Neurosymbolic AI: Why, What, and How
And of course formal methods! Well mostly TLA+ in my case.
Modular verification of MongoDB Transactions using TLA+
Multi-Grained Specifications for Distributed System Model Checking and Verification
Notes from the TLA+ Community Event
Smart Casual Verification of the Confidential Consortium Framework
TLA+ Modeling of AWS outage DNS race condition
Best of metadata in 2021
Best of metadata in 2020
Best of metadata in 2019
Best of metadata in 2018
Research, writing, and career advice
