What I describe here works for me given my goal, which is to find performance regressions. A benchmark run at low concurrency is used to find regressions from CPU overhead. A benchmark run at high concurrency is used to find regressions from mutex contention. A benchmark run at medium concurrency might help find both.

My informal way for classifying servers by size is:

• small - has less than 10 cores
• medium - has between 10 and 20 cores
• large - has more than 20 cores

This week I was running benchmarks to understand how fast RocksDB could do IO, and then compared that to fio to understand the CPU overhead added by RocksDB. While looking at flamegraphs taken during the benchmark I was confused that about 20% of the samples were from page fault handling. This confused me at first.

The lesson here is to run your benchmark long enough to reach a steady state before you measure things or there will be confusion. And I was definitely confused when I first saw this. Perhaps my post saves time for the next person who spots this.

The workload is db_bench with a database size that is much larger than memory and read-only microbenchmarks for point lookups and range scans.

Then I wondered if this was a transient issue that occurs while RocksDB is warming up the block cache and growing process RSS until the block cache has been fully allocated.

While b-trees as used by Postgres and MySQL will do a large allocation at process start, RocksDB does an allocation per block read, and when the block is evicted then the allocation is free'd. This can be a stress test for a memory allocator which is why jemalloc and tcmalloc work better than glibc malloc for RocksDB. I revisit the mallocator topic every few years and my most recent post is here.

In this case I use RocksDB with jemalloc. Even though per-block allocations are transient, the memory used by jemalloc is mostly not transient. While there are cases where jemalloc an return memory to the OS, with my usage that is unlikely to happen.

Were I to let the benchmark run for a long enough time, then eventually jemalloc would finish getting memory from the OS. However, my tests were running for about 10 minutes and doing about 10,000 block reads per second while I had configured RocksDB to use a block cache that was at least 36G and the block size was 8kb. So my tests weren't running long enough for the block cache to fill, which means that during the measurement period:

• jemalloc was still asking for memory
• block cache eviction wasn't needed and after each block read a new entry was added to the block cache

If you want to store vectors in your database then what you store as a row, KV pair or document is likely to be larger than the fixed-page size (when your DBMS uses fixed-page sizes) and you will soon care about efficient and performant support for large objects. I assume this support hasn't been the top priority for many DBMS implementations and there will be some performance bugs.

In a SQL DBMS, support for large objects will use the plumbing created to handle LOB (Large OBject) datatypes. We should define what the L in LOB means here and I will wave my hands and claim larger than a fixed-page in your favorite DBMS but smaller than 512kb because I limit my focus to online workloads.

Perhaps now is the time for industry standard benchmarks for workloads with large objects. Should it be TPC-LOB or TPC-BLOB?

Most popular DBMS use fixed-size pages whether that storage is index-organized via an update-in-place b-tree (InnoDB) or heap-organized (Postgres, Oracle). For rows that are larger than the page size, which is usually between 4kb and 16kb, the entire row or largest columns will be stored out of line and likely split across several pages in the out of line storage. When the row is read, additional reads will be done to gather all of the too-large parts from the out of line locations.

This approach is far from optimal as there will be more CPU overhead, more random IO and might be more wasted space. But this was good enough because support for LOBs wasn't a priority for these DBMS as their focus was on OLTP where rows were likely to be smaller than a fixed-size page.

Perhaps by luck, perhaps it was fate, but WiredTiger is a great fit for MongoDB because it is more flexible about page sizes. And it is more flexible because it isn't an update-in-place b-tree, instead it is a copy-on-write random (CoW-R) b-tree that doesn't need or use out-of-line storage, although for extra large documents there might be a benefit from out-of-line.

MyRocks, and other LSM-based DBMS, also don't require out-of-line storage but they can benefit from it as shown by WiscKey and other engines that do key-value separation. Even the mighty RocksDB has an implementation of key-value separation via BlobDB.

This is yet another great result for Postgres 18.0 vs sysbench. This time I used a 32-core server. Results for a 24-core server are here. The goal for this benchmark is to check for regressions from new CPU overhead and mutex contention.

I repeated the benchmark twice because I had some uncertainty about platform variance (HW and SW) on the first run.

tl;dr, from Postgres 17.6 to 18.0

• There might be regressions from 17.6 to 18.0 but they are small (usually <= 3%)

tl;dr, from Postgres 12.22 through 18.0

• the hot-points test is almost 2X faster starting in 17.6
• scan is ~1.2X faster starting in 14.19
• all write tests are much faster staring in 17.6

A few months back I wound up concluding, based on conversations with Ofcom, that aphyr.com might be illegal in the UK due to the UK Online Safety Act. I wrote a short tutorial on geoblocking a single country using Nginx on Debian.

Now Mississippi’s 2024 HB 1126 has made it illegal for essentially any web site to know a user’s e-mail address, or other “personal identifying information”, unless that site also takes steps to "verify the age of the person creating an account”. Bluesky wound up geoblocking Mississippi. Over on a small forum I help run, we paid our lawyers to look into HB 1126, and the conclusion was that we were likely in the same boat. Collecting email addresses put us in scope of the bill, and it wasn’t clear whether the LLC would shield officers (hi) from personal liability.

This blog has the same problem: people use email addresses to post and confirm their comments. I think my personal blog is probably at low risk, but a.) I’d like to draw attention to this legislation, and b.) my risk is elevated by being gay online, and having written and called a whole bunch of Mississippi legislators about HB 1126. Long story short, I’d like to block both a country and an individual state. Here’s how:

First, set up geoipupdate as before. Then, in /etc/nginx/conf.d.geoblock.conf, pull in the country and city databases, and map the countries and states you’d like to block to short strings explaining the applicable law. This creates variables $geoblock_country_law and $geoblock_state_law.

geoip2 /var/lib/GeoIP/GeoLite2-Country.mmdb {
  $geoip2_data_country_iso_code country iso_code;
}

geoip2 /var/lib/GeoIP/GeoLite2-City.mmdb {
  $geoip2_data_state_name subdivisions 0 names en;
}

map $geoip2_data_country_iso_code $geoblock_country_law {
  GB      "the UK Online Safety Act";
  default "";
}

map $geoip2_data_state_name $geoblock_state_law {
  Mississippi "Mississippi HB 1126";
  default     "";
}

Create an HTML page to show to geoblocked IPs. I’ve put mine in /var/www/custom_errors/451.html. The special comments here are Server-Side Include (SSI) directives; they’ll insert the contents of the $geoblock_law variable from nginx, which we’ll set shortly.

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>Unavailable Due to
      <!--# echo var="geoblock_law" default=""-->
    </title>
  </head>
  <body>
    <h1>Unavailable Due to
      <!--# echo var="geoblock_law" default=""-->
    </h1>
</body>
</html>

Then, in /etc/nginx/sites-enabled/whatever.conf, add an error page for status code 451 (unavailable for legal reasons). In the main location block, check the $geoblock_country_law and $geoblock_state_law variables, and use them to return status 451, and set the $geoblock_law variable for the SSI template:

server {
  ...
  # Status 451 renders this page
  error_page 451 /451.html;
  location /451.html {
    ssi on;
    internal;
    root /var/www/custom_errors/;
  }

  location / {
    # If either geoblock variable is set, return status 451
    if ($geoblock_state_law != "") {
      set $geoblock_law $geoblock_state_law;
      return 451;
    }
    if ($geoblock_country_law != "") {
      set $geoblock_law $geoblock_country_law;
      return 451;
    }
  }
}

Test with nginx -t, and reload with service nginx reload, as usual.

Geoblocking is a bad experience in general. In Amsterdam and Frankfurt, I’ve seen my cell phone’s 5G connection and hotel WiFi improperly identified as being in the UK. I’m certain this is going to block people who aren’t in Mississippi either. If you don’t want to live in this world either, start calling your representatives to demand better legislation.

This week, Aleksey and I met not to dissect a research paper, but to chat about "the process of PhD". I had recently wrote a post titled "The Invisible Curriculum of Research", where I framed research as an iceberg, with the small visible parts (papers, conferences) resting on the hidden 5 Cs:

• Curiosity/Taste: what problems are worth solving.
• Clarity: how to ask precise and abstracting questions.
• Craft: writing, experimentation, presentation.
• Community: collaboration and contribution.
• Courage: resilience through setbacks.

Above is the video of our chat, with a lot of personal anecdotes and a few rants. But if you want to cut to the chase, the highlight reel is below.

What a PhD Really Produces

The real product of a PhD is not the thesis, but you, the researcher! The thesis is just the residue of this long internal transformation. Like martial arts, the training breaks you and rebuilds you into someone who sees and thinks differently. This transformation cannot be faked and you should take your time to grow your wings. But you can be effective about it.

Curiosity and taste

Taste blends curiosity, creativity, and judgment of what's important. Curiosity alone can lure you into many bottomless technical rabbit holes. Taste filters what matters and channels curiosity into focus. And you definitely need passion to sustain yourself through the ups and downs of the arduous PhD journey.

The serendipitous path to research

Many researchers stumble into research through chance encounters, unexpected opportunities, and detours. So it is worth keeping an open mind, noticing what sparks your curiosity and suits you best, and following it. Aleksey shares his own unlikely path to a PhD, which is well worth watching. I have written before about how I started, but here I go deeper into where those interests first took root.

Growing through friction and mentorship

Taste, curiosity, and confidence grow through friction. The best labs are loud where debates spill into hallways. When Aleksey, Aili, and I worked together, neighboring faculty sometimes complained about the noise, wondering why we were always arguing. But intellectual sparring sharpens your ideas. Research maturity comes from questioning, defending, refining. In this type of hands-on, messy mentorship, taste, passion, craft all rub off.

Asking good questions and abstracting well

Abstraction is the art of asking the right questions. The best questions cut away accidental complexity and get to the essence of the problem. Leslie Lamport's genius was exactly knowing what to ignore/abstract-away. "Craft is knowing how to work, and art is knowing when to stop."

By finding the right question/framing/abstraction, you can pivot a project that is not getting any lift into an impactful hit! (This I believe.)

The Craft of Research

Most research is unglamorous: debugging, writing, revising, rejections. But you gotta  do the craft, and do it well, for your ideas to lift off. You need routines and ritual to keep you steady and improving. Aleksey's productivity routine involves daily 90-minute walks as his "thinking time". Thinking, for him, is a physical process. We used to walk a lot when we worked together, but somehow I have fallen off that wagon. My thinking time now comes through freewriting on Emacs or on my tablet, and arguing with myself on the page. We both agree, though, that talking/arguing with collaborators forces clarity and generates ideas.

On Courage and Resilience

Every researcher fails as much as (if not more than) they succeed. The researcher needs to endure through failures and rejections. You need to keep showing up to write the next draft, rerun the next experiment, submit again. Passion helps, without it, survival in research is unlikely. But you also need to make a habit of endurance. Courage also means questioning norms and pursuing ideas that may not yet be fashionable but feel true.

But, sometimes (ah Retroscope) you have to take the loss, cut your losses, and move on.  Maybe you can return later at a more opportune time.

Top skills/qualities for a PhD

We discussed our picks for top three skills needed for a successful Phd. For me, it is writing/communication, asking the right questions, and metacognition (knowing when to stop, reframe, or abstract; seeing the essence rather than surface detail). Reading skills came up very high in our discussion too. You can't outsource that to ChatGPT. People skills also matter: work well with your collaborators. Conferences and brutal rankings in academia can feel like SquidGames at times, but what truly matters is people, mentorship, and the craft itself.

What makes a bad researcher

Bad research habits are easy to spot: over-competition, turf-guarding, incremental work, rigidity, and a lack of intellectual flexibility. Bad science follows bad incentives such as benchmarks over ideas, and performance over understanding. These days the pressure to run endless evaluations has distorted the research and publishing process. Too many papers now stage elaborate experiments to impress reviewers instead of illuminating them with insights. Historically, the best work always stood on its own, by its simplicity and clarity. 

Onboarding and Departmental Support

Advisor fit is crucial, and students should be free to explore before committing. Early rotations and cohort boot camps, which Aleksey mentioned is common in biomedical programs, help build both skills and faculty connections. Unfortunately, computer science still lacks this scaffolding. Industry treats onboarding as an investment, with structured mentorship, regular check-ins, and clear expectations. Academia, by contrast, seems to treat the absence of onboarding as a filtering mechanism. New PhD students are frequently left on their own for months, without direction, feedback, or a sense of belonging. Even small rituals (weekly meetings, mentorship pairings, consistent feedback) could change and catch struggling/blocked students early rather than years later.