MongoDB indexes enable quick equality and range filtering (with $match) and ordering (with $sort). For aggregation operations like $group with $first or $last accumulators, indexes can dramatically improve performance through the DISTINCT_SCAN optimization.

However, for hash-based aggregations such as $group with $sum or $sortByCount, there's no such optimization. While indexes can provide significant performance benefits by creating covered query plans (avoiding document fetches), the query planner won't automatically select them. You must explicitly specify the index using a hint.

Setup a collection for the test

I created a test collection with 50,000 documents. Each document has a "groupme" field with 1000 distinct values, and a filler field (a random 50KB base64 string).

base64 -w 51200 /dev/urandom |
 awk 'NR>50000{exit}{print int(1000*rand())"\t"$0}' |
 mongoimport -c demo -f groupme,filler --type=tsv /dev/stdin --drop

I continue with mongosh and use an aggregation pipeline to group by "groupme" and count the documents:

db.demo.countDocuments()

50000

db.demo.aggregate([  { $sortByCount: "$groupme" } ])

[
  { _id: 937, count: 78 }, { _id: 517, count: 71 },
  { _id: 798, count: 70 }, { _id: 182, count: 68 },
  { _id: 812, count: 68 }, { _id: 158, count: 67 },
  { _id: 60, count: 67 },  { _id: 157, count: 66 },
  { _id: 653, count: 66 }, { _id: 901, count: 66 },
  { _id: 450, count: 66 }, { _id: 587, count: 66 },
  { _id: 701, count: 66 }, { _id: 403, count: 66 },
  { _id: 110, count: 65 }, { _id: 757, count: 65 },
  { _id: 461, count: 65 }, { _id: 593, count: 65 },
  { _id: 418, count: 65 }, { _id: 39, count: 64 }
]
Type "it" for more

Collection scan

I executed it again to get the execution plan with execution statistics:

x=db.demo.aggregate([  
 { $sortByCount: "$groupme" } 
]).explain("executionStats")

In my small lab, the aggregation ran for 580 milliseconds:

print(x.stages[0].executionTimeMillisEstimate)

Long('590')

The execution plan shows a collection scan.

print(x.stages[0].$cursor.queryPlanner.winningPlan.queryPlan)

{
  stage: 'GROUP',
  planNodeId: 3,
  inputStage: {
    stage: 'COLLSCAN',
    planNodeId: 1,
    filter: {},
    direction: 'forward'
  }
}

According to execution statistics, the elapsed time is primarily caused by the scan operation, which took 549 milliseconds, and the aggregation step:

print(x.stages[0].$cursor.executionStats.executionStages.inputStage.inputStage)

{
  stage: 'group',
  planNodeId: 3,
  nReturned: 1000,
  executionTimeMillisEstimate: 579,
  opens: 1,
  closes: 1,
  saveState: 37,
  restoreState: 36,
  isEOF: 1,
  groupBySlots: [ Long('5') ],
  expressions: { '6': 'count() ', initExprs: { '6': null } },
  mergingExprs: { '4': 'sum(s4) ' },
  usedDisk: false,
  spills: 0,
  spilledBytes: 0,
  spilledRecords: 0,
  spilledDataStorageSize: 0,
  inputStage: {
    stage: 'project',
    planNodeId: 3,
    nReturned: 50000,
    executionTimeMillisEstimate: 569,
    opens: 1,
    closes: 1,
    saveState: 37,
    restoreState: 36,
    isEOF: 1,
    projections: { '5': '(s3 ?: null) ' },
    inputStage: {
      stage: 'scan',
      planNodeId: 1,
      nReturned: 50000,
      executionTimeMillisEstimate: 549,
      opens: 1,
      closes: 1,
      saveState: 37,
      restoreState: 36,
      isEOF: 1,
      numReads: 50000,
      recordSlot: 1,
      recordIdSlot: 2,
      scanFieldNames: [ 'groupme' ],
      scanFieldSlots: [ Long('3') ]
    }
  }
}

Index on the grouping key

To demonstrate the benefit of a covering index, I create an index on the grouping key:

db.demo.createIndex(
 { groupme: 1 }
);

The query planner doesn't use the index and the query still uses a collection scan:

x=db.demo.aggregate([  
 { $sortByCount: "$groupme" } 
]).explain("executionStats")

print(x.stages[0].$cursor.queryPlanner.winningPlan.queryPlan)

{
  stage: 'GROUP',
  planNodeId: 3,
  inputStage: {
    stage: 'COLLSCAN',
    planNodeId: 1,
    filter: {},
    direction: 'forward'
  }
}

MongoDB's query planner doesn't automatically choose covering indexes, but I can force it with a hint:

x=db.demo.aggregate([  
 { $sortByCount: "$groupme" } 
],
 { hint: { groupme: 1 } } ).explain("executionStats")

print(x.stages[0].$cursor.queryPlanner.winningPlan.queryPlan)

{
  stage: 'GROUP',
  planNodeId: 3,
  inputStage: {
    stage: 'PROJECTION_COVERED',
    planNodeId: 2,
    transformBy: { groupme: true, _id: false },
    inputStage: {
      stage: 'IXSCAN',
      planNodeId: 1,
      keyPattern: { groupme: 1 },
      indexName: 'groupme_1',
      isMultiKey: false,
      multiKeyPaths: { groupme: [] },
      isUnique: false,
      isSparse: false,
      isPartial: false,
      indexVersion: 2,
      direction: 'forward',
      indexBounds: { groupme: [ '[MinKey, MaxKey]' ] }
    }
  }
}

The execution plan uses an index scan with a projection that covers the grouping columns. Because the query accesses only the small index entries, not the large documents, execution time dropped to 19 milliseconds:

x=db.demo.aggregate([  
 { $sortByCount: "$groupme" } 
],
 { hint: { groupme: 1 } } ).explain("executionStats")

print(x.stages[0].executionTimeMillisEstimate)

Long('22')

print(x.stages[0].$cursor.executionStats.executionStages.inputStage.inputStage)

{
  stage: 'group',
  planNodeId: 3,
  nReturned: 1000,
  executionTimeMillisEstimate: 20,
  opens: 1,
  closes: 1,
  saveState: 8,
  restoreState: 7,
  isEOF: 1,
  groupBySlots: [ Long('4') ],
  expressions: { '5': 'count() ', initExprs: { '5': null } },
  mergingExprs: { '3': 'sum(s3) ' },
  usedDisk: false,
  spills: 0,
  spilledBytes: 0,
  spilledRecords: 0,
  spilledDataStorageSize: 0,
  inputStage: {
    stage: 'project',
    planNodeId: 3,
    nReturned: 50000,
    executionTimeMillisEstimate: 10,
    opens: 1,
    closes: 1,
    saveState: 8,
    restoreState: 7,
    isEOF: 1,
    projections: { '4': '(s2 ?: null) ' },
    inputStage: {
      stage: 'ixseek',
      planNodeId: 1,
      nReturned: 50000,
      executionTimeMillisEstimate: 10,
      opens: 1,
      closes: 1,
      saveState: 8,
      restoreState: 7,
      isEOF: 1,
      indexName: 'groupme_1',
      keysExamined: 50000,
      seeks: 1,
      numReads: 50001,
      recordIdSlot: 1,
      outputSlots: [ Long('2') ],
      indexKeysToInclude: '00000000000000000000000000000001',
      seekKeyLow: 'KS(0A01) ',
      seekKeyHigh: 'KS(F0FE) '
    }
  }
}

While some databases use different aggregation algorithms when the input data is ordered, MongoDB always relies on a hash-based algorithm. I plan to test two indexes to determine if one with a prefix matching the grouping key is more efficient.

Index not starting with the grouping key

I create an index that is not ordered on the grouping key but only covers it:

db.demo.createIndex(
 { _id:1, groupme: 1 }
);

The index is slightly larger, and the scan takes a few additional milliseconds:

x=db.demo.aggregate([  
 { $sortByCount: "$groupme" } 
],
 { hint: { _id:1, groupme: 1 } } ).explain("executionStats")

print(x.stages[0].executionTimeMillisEstimate)

Long('24')

print(x.stages[0].$cursor.executionStats.executionStages.inputStage.inputStage)

{
  stage: 'group',
  planNodeId: 3,
  nReturned: 1000,
  executionTimeMillisEstimate: 19,
  opens: 1,
  closes: 1,
  saveState: 8,
  restoreState: 7,
  isEOF: 1,
  groupBySlots: [ Long('4') ],
  expressions: { '5': 'count() ', initExprs: { '5': null } },
  mergingExprs: { '3': 'sum(s3) ' },
  usedDisk: false,
  spills: 0,
  spilledBytes: 0,
  spilledRecords: 0,
  spilledDataStorageSize: 0,
  inputStage: {
    stage: 'project',
    planNodeId: 3,
    nReturned: 50000,
    executionTimeMillisEstimate: 19,
    opens: 1,
    closes: 1,
    saveState: 8,
    restoreState: 7,
    isEOF: 1,
    projections: { '4': '(s2 ?: null) ' },
    inputStage: {
      stage: 'ixseek',
      planNodeId: 1,
      nReturned: 50000,
      executionTimeMillisEstimate: 19,
      opens: 1,
      closes: 1,
      saveState: 8,
      restoreState: 7,
      isEOF: 1,
      indexName: '_id_1_groupme_1',
      keysExamined: 50000,
      seeks: 1,
      numReads: 50001,
      recordIdSlot: 1,
      outputSlots: [ Long('2') ],
      indexKeysToInclude: '00000000000000000000000000000010',
      seekKeyLow: 'KS(0A0A01) ',
      seekKeyHigh: 'KS(F0F0FE) '
    }
  }
}

For comparison, I created another index with the same fields in the key, but arranged in a different order, to match the size and start with the grouping key:

db.demo.createIndex(
 { groupme: 1, _id:1 }
);

x=db.demo.aggregate([  
 { $sortByCount: "$groupme" } 
],
 { hint: { groupme: 1, _id:1 } } ).explain("executionStats")

print(x.stages[0].executionTimeMillisEstimate)

Long('22')

print(x.stages[0].$cursor.executionStats.executionStages.inputStage.inputStage)

{
  stage: 'group',
  planNodeId: 3,
  nReturned: 1000,
  executionTimeMillisEstimate: 20,
  opens: 1,
  closes: 1,
  saveState: 8,
  restoreState: 7,
  isEOF: 1,
  groupBySlots: [ Long('4') ],
  expressions: { '5': 'count() ', initExprs: { '5': null } },
  mergingExprs: { '3': 'sum(s3) ' },
  usedDisk: false,
  spills: 0,
  spilledBytes: 0,
  spilledRecords: 0,
  spilledDataStorageSize: 0,
  inputStage: {
    stage: 'project',
    planNodeId: 3,
    nReturned: 50000,
    executionTimeMillisEstimate: 20,
    opens: 1,
    closes: 1,
    saveState: 8,
    restoreState: 7,
    isEOF: 1,
    projections: { '4': '(s2 ?: null) ' },
    inputStage: {
      stage: 'ixseek',
      planNodeId: 1,
      nReturned: 50000,
      executionTimeMillisEstimate: 0,
      opens: 1,
      closes: 1,
      saveState: 8,
      restoreState: 7,
      isEOF: 1,
      indexName: 'groupme_1__id_1',
      keysExamined: 50000,
      seeks: 1,
      numReads: 50001,
      recordIdSlot: 1,
      outputSlots: [ Long('2') ],
      indexKeysToInclude: '00000000000000000000000000000001',
      seekKeyLow: 'KS(0A0A01) ',
      seekKeyHigh: 'KS(F0F0FE) '
    }
  }
}

The execution time remains similar, and the main benefit of the index is realized when it includes the fields used in the $group stage, no matter their order.

Forcing disk spill

To compare the work done with the two indexes, I force a disk spill by setting the hash aggregation to a value smaller than the aggregate:

db.adminCommand({
 setParameter: 1,
 internalQuerySlotBasedExecutionHashAggApproxMemoryUseInBytesBeforeSpill: 100
}).was

Long('104857600')

Both indexes show 50,000 disk spill when grouping 50,000 entries:

x=db.demo.aggregate([  
 { $sortByCount: "$groupme" } 
],
 { hint: { groupme: 1, _id:1 } } ).explain("executionStats")

print(x.stages[0].executionTimeMillisEstimate)

Long('59')

print(x.stages[0].$cursor.executionStats.executionStages.inputStage.inputStage.spills)

996

x=db.demo.aggregate([  
 { $sortByCount: "$groupme" } 
],
 { hint: {  _id:1, groupme: 1 } } ).explain("executionStats")

print(x.stages[0].executionTimeMillisEstimate)

Long('617')

print(x.stages[0].$cursor.executionStats.executionStages.inputStage.inputStage.spills)

16667

When the aggregation needs to spill to disk due to many distinct values for the grouping key or reduced hash aggregation memory, the index starting with the grouping key offers an advantage.

I reset the configuration to the default of 100MB:

db.adminCommand({
 setParameter: 1,
 internalQuerySlotBasedExecutionHashAggApproxMemoryUseInBytesBeforeSpill: 104857600 
}).was

Long('1')

Conclusion

For aggregations, especially when documents are large, using a covering index can greatly improve performance by allowing MongoDB to read compact index entries instead of full documents. Because MongoDB’s query planner does not automatically select covering indexes for these work... (truncated)

Down with template (or not)!

template <typename T, typename F>
struct JoinFilterHasher {
 template <typename S>
 static uint64_t hashDict(const byte* data, unsigned index, const byte* arg) {
 const T* dict = unalignedLoad<const T*>(arg);
 const S pos = *reinterpret_cast<const S*>(data + index * sizeof(S));
 return F::template hash<type>(dict + pos, nullptr);
 // ^^^^^^^^ what is this???
 }
};

The template Keyword

struct Foo {
 template <int I>
 static int myfunc(int a) { return a + I; }
};

template <typename T>
int genericFoo(int a) {
 return T::myfunc<42>(a);
}

int test() {
 return genericFoo<Foo>(1);
}

Down with template?

<source>: In function 'int genericFoo(int)':
<source>:8:15: warning: expected 'template' keyword before dependent template name [-Wmissing-template-keyword]
 8 | return T::myfunc<42>(a);
 | ^~~~~~
 | template

<source>:8:21: error: invalid operands of types '<unresolved overloaded function type>' and 'int' to binary 'operator<'
 8 | return T::myfunc<42>(a);
 | ~~~~~~^~~

return // return keyword
T // name referring to a type
:: // scope resolution operator

// !!! Here's the interesting part !!!
myfunc // name referring to a variable
< // operator<
42 // integer literal
> // operator>
// !!! end of interesting part !!!

( // opening parenthesis (not a function call!)
a // name referring to a variable
) // closing parenthesis
; // semicolon

return (T::myfunc < 42) > a;

Template Madness

Operator Overloading

struct DownWithTemplate {
 /* ... */
 struct MyFunc {
 OtherHalf operator<(int i) const {
 return OtherHalf{i};
 }
 };
 static constexpr MyFunc myfunc{};
};

struct DownWithTemplate {
 struct OtherHalf {
 int value;
 int operator>(int i) const {
 return /* ??? */;
 }
 };
 /* ... */
};

<source>:8:16: note: template argument deduction/substitution failed:
<source>:27:32: error: '*(const DownWithTemplate::OtherHalf*)this' is not a constant expression
 27 | return Foo::myfunc<value>(i);
 | ^~~~~

Runtime Template Arguments (or: Inner Circle of C++ Template Hell)

switch (value) {
 case 0: return Foo::myfunc<0>(i);
 case 1: return Foo::myfunc<1>(i);
 /* ... */
 case 2147483647: return Foo::myfunc<2147483647>(i);
 case -1: return Foo::myfunc<-1>(i);
 case -2: return Foo::myfunc<-2>(i);
 /* ... */
 case -2147483648: return Foo::myfunc<-2147483648>(i);
}

struct DownWithTemplate {
 template <int... Is>
 static int callMyFunc(std::integer_sequence<int, Is...>, int a, int b) {
 return (((Is == a) ? Foo::myfunc<Is>(b) : 0) + ...) +
 (((-Is-1 == a) ? Foo::myfunc<-Is-1>(b) : 0) + ...);
 }
 struct OtherHalf {
 int value;
 int operator>(int i) const {
 return callMyFunc(std::make_integer_sequence<int, std::numeric_limits<int>::max()>{}, value, i);
 }
 };
 /* ... */
};

((value == 0) ? Foo::myfunc<0>(i) : 0) +
((value == 1) ? Foo::myfunc<1>(i) : 0) +
/* ... */

The Fallout

clang++ -c -std=c++23 -O0 ./down_with_template.cpp

Conclusion

Appendix

Benchmark Code: Vectorized Sums

down_with_template.cpp

#include <limits>
#include <utility>

// DANGER: changing this to any larger type will make the compiler eat all of your RAM!
using Int = signed char;

struct Foo {
 template <Int I>
 static Int myfunc(Int a) { return a + I; }
};

template <typename T>
Int genericFoo(Int a) {
 return T::myfunc<42>(a);
}

struct DownWithTemplate {
 template <Int... Is>
 static Int callMyFunc(std::integer_sequence<Int, Is...>, Int a, Int b) {
 return (((Is == a) ? Foo::myfunc<Is>(b) : 0) + ...) +
 (((-Is-1 == a) ? Foo::myfunc<-Is-1>(b) : 0) + ...);
 }

 struct OtherHalf {
 Int value;
 Int operator>(Int i) const {
 return callMyFunc(std::make_integer_sequence<Int, std::numeric_limits<Int>::max()>{}, value, i);
 }
 };
 struct MyFunc {
 OtherHalf operator<(Int i) const {
 return OtherHalf{i};
 }
 };
 static constexpr MyFunc myfunc{};
};

Int test() {
 return genericFoo<DownWithTemplate>(1);
}

How efficient is RocksDB for workloads that are IO-bound and read-only? One way to answer this is to measure the CPU overhead from RocksDB as this is extra overhead beyond what libc and the kernel require to perform an IO. Here my focus is on KV pairs that are smaller than the typical RocksDB block size that I use -- 8kb.

By IO efficiency I mean: (storage read IOPs from RocksDB benchmark / storage read IOPs from fio)

And I measure this in a setup where RocksDB doesn't get much benefit from RocksDB block cache hits (database size > 400G, block cache size was 16G).

This value will be less than 1.0 in such a setup. But how much less than 1.0 will it be? On my hardware the IO efficiency was ~0.85 at 1 client and 0.88 at 6 clients. Were I to use storage that had a 2X larger storage latency then the IO efficiency would be closer to 0.95.

 Note that:

• IO efficiency increases (decreases) when SSD read latency increases (decreases)
• IO efficiency increases (decreases) when the RocksDB CPU overhead decreases (increases)
• RocksDB QPS increases by ~8% for IO-bound workloads when --block_align is enabled

Q and A

I ran tests on a Beelink SER7 with a Ryzen 7 7840HS CPU that has 8 cores and 32G of RAM. The storage device a Crucial is CT1000P3PSSD8 (Crucial P3, 1TB) using ext-4 with discard enabled. The OS is Ubuntu 24.04.

From fio, the average read latency for the SSD is 102 microseconds using O_DIRECT with io_depth=1 and the sync engine.

CPU frequency management makes it harder to claim that the CPU runs at X GHz, but the details are:

$ cpupower frequency-info

Results from fio

I started with fio using a command-line like the following for NJ=1 and NJ=6 to measure average IOPs and the CPU overhead per IO.

Results are:

Results from RocksDB

I used an edited version of my benchmark helper scripts that run db_bench. In this case the sequence of tests was:

1. fillseq - loads the LSM tree in key order
2. revrange - I ignore the results from this
3. overwritesome - overwrites 10% of the KV pairs
4. flush_mt_l0 - flushes the memtable, waits, compacts L0 to L1, waits
5. readrandom - does random point queries when LSM tree has many levels
6. compact - compacts LSM tree into one level
7. readrandom2 - does random point queries when LSM tree has one level, bloom filters enabled
8. readrandom3 - does random point queries when LSM tree has one level, bloom filters disabled

This recent paper from the Berkeley Sky Computing Lab has been making waves in systems community. Of course, Aleksey and I did our live blind read of it, which you can watch below. My annotated copy of the paper is also available here.

This is a fascinating and timely paper. It raises deep questions about how LLMs will shape the research process, and how that could look like. Below, I start with a short technical review, then move to the broader discussion topics.

Technical review

The paper introduces AI-Driven Research for Systems (ADRS) framework. By leveraging the OpenEvolve framework,  ADRS integrates LLMs directly into the systems research workflow to automate much of the solution-tweaking and evaluation process. As shown in Figure 3, ADRS operates as a closed feedback loop in which the LLM ensemble iteratively proposes, tests, and refines solutions to a given systems problem. This automation targets the two most labor-intensive stages of the research cycle, solution tweaking and evaluation, leaving the creative areas (problem formulation, interpreting results, and coming up with insights) untouched.

Within the inner loop, four key components work together. The Prompt Generator creates context-rich prompts that seed the LLM ensemble (Solution Generator), which outputs candidate designs or algorithms. These are then assessed by the Evaluator, a simulator or benchmark written by humans, for gathering quantitative feedback. The Solution Selector identifies the most promising variants, which are stored along with their scores in the Storage module to inform subsequent iterations. This automated loop runs rapidly and at scale, and enables exploration of large design spaces within hours rather than weeks! They applied ADRS to several systems problems, including cloud job scheduling, load balancing, transaction scheduling, and LLM inference optimization. In each case, the AI improved on prior human-designed algorithms, often within a few hours of automated search. Reported gains include up to 5x faster performance or 30–50% cost reductions compared to published baselines, which are achieved in a fraction of the time and cost of traditional research cycles.

Outside the optimization loop, the creative and difficult work happens. The scientist identifies the research problem, directs the search, and decides which hills are worth climbing. Machines handle the iterative grunt work of tweaking and testing solutions, while humans deal with abstraction, framing, and insight.

There are several other important limitations for the framework's effectiveness as well. The paper's examples mostly involve trivial correctness, and also no concurrency, security, or fault-tolerance concerns. These domains require reasoning beyond performance tuning. Another limitation is that these LLMs focus/update one component only, and can't handle system-wide interactions yet.

Simulator-based evaluation makes this approach feasible, but the systems field undervalues simulation work and this leads to limited infrastructure for automated testing. Similarly, evaluators also pose risks: poorly designed ones invite reward hacking, where LLMs exploit loopholes rather than learn real improvements. If AI-driven research is to scale, we need richer evaluators, stronger specifications, and broader respect for simulation as a first-class research tool.

Discussion topics

Here I wax philosophical on many interesting questions this work raises.

LLMs provide breadth, but research demands depth

LLMs excel at high-throughput mediocrity. By design, they replicate what has already been done, and optimize across the surface of knowledge. Research, however, advances through novelty, depth, and high-value insight.

"Research is to see what everybody else has seen, and to think what nobody else has thought."

-- Albert Szent-Györgyi (Nobel laureate)

In this sense, LLMs are not as dangerous as the "Barbarians" at the gates. They are more like "Barbies" at the gates, with gloss, confidence, and some hollowness. They may dazzle with presentation but they will lack the inner substance/insights/value that mastery, curiosity, and struggle bring.

LLMs address only the tip of the iceberg

LLMs operate on the visible tip of the research iceberg I described earlier. They cannot handle the deep layers that matter: Curiosity, Clarity, Craft, Community, Courage.

Worse, they may even erode those qualities. The danger in the short-term is not invasion, but imitation: the replacement of thought with performance, and depth with polish. We risk mistaking synthetic polish with genuine understanding.

In the long term though, I am not worried. In the long term, we are all dead.

I'm kidding, ok. In the long term, we may be screwed as well. The 2004 movie "Idiocracy" rings more true every day. I am worried that due to the inherent laziness of our nature, we may end up leaning more and more on AI to navigate literature, frame questions, or spin hypotheses, that we may not get enough chances to exercise our curiosity or improve our clarity of understanding.

LLMs are bad researchers, but can they still make good collaborators?

In our academic chat follow-up to the iceberg post, I wrote about what makes a bad researcher:

Bad research habits are easy to spot: over-competition, turf-guarding, incremental work, rigidity, and a lack of intellectual flexibility. Bad science follows bad incentives such as benchmarks over ideas, and performance over understanding. These days the pressure to run endless evaluations has distorted the research and publishing process. Too many papers now stage elaborate experiments to impress reviewers instead of illuminating them with insights. Historically, the best work always stood on its own, by its simplicity and clarity. 

LLMs are bad researchers. The shoe fits. 

But can they still be good collaborators? Is it still worth working with them? The hierarchy is simple:

Good collaborators  >  No  collaborators  >  Bad collaborators

Used wisely, LLMs can climb high enough to reach the lowest range of the good collaborator category. If you give them bite-sized well defined work, they can reduce friction, preserve your momentum, and speed up parts of your work significantly. In a sense, they can make you technically fearless. I believe that when used for rapid prototyping, LLMs can help improve the design. And, through faster iteration, you may uncover some high-value insights.

But speed cuts both ways, because premature optimization is the root of all evil. If doing evaluations and optimizations becomes very cheap and effortless, we will more readily jump to this step, without nothing forcing us to think harder. Human brains are lazy by design. They don't want to think hard, and they will take the quick superficial route out, and we don't get to go deep. 

So, we need to tread carefully here as well.

Can we scale human oversight?

The worst time I ever had as an advisor was when I had to manage 6-7 (six-seveeeen!) PhD students at once. I would much rather work with 2 sharp creative students I support myself than 50 mediocre ones handed to me for free. The former process of working is more productive and it results in deep work and valuable research. Focus is the key, and it does not scale. 

The same holds for LLM-augmented research. Validation (via human focus) remains as the bottleneck. They can generate endless results, but without distilling those results into insight or wisdom, they all remain as AI slop in abundance.

Can clear insights distill without dust, tear, and sweat?

One may argue that with machines handling the grunt work, the researchers would finally get more time for thinking. Our brains are --what?-- yes, lazy. Left idle, they will scroll Reddit/Twitter rather than solve concurrency bugs.

I suspect we need some friction/irritation to nudge us to think in the background. And I suspect this is what happens when we are doing the boring work and working in the trenches. While writing a similar code snippet for the fifth time in our codebase, an optimization opportunity or an abstraction would occur to us. Very hard problems are impossible to tackle head on. Doing the legwork, I suspect we approach the problem sideways, and have a chance to make some leeway.

Yes, doing evaluation work sucks. But it is often necessary to generate the friction and space to get you think about the performance, and more importantly the logic/point of your system.  Through that suffering, you gradually get transformed and enlightened. Working in the trenches, you may even realize your entire setup is flawed, and your measurements are garbage due to using closed loop clients instead of open loop ones.

What happens when we stop getting our hands dirty? We risk distilling nothing at all. Insights don't bubble up while we are sitting in comfort and scrolling cat videos. In an earlier post, Looming Liability Machines (LLMs), I argued that offloading root-cause analysis to AI misses the point. RCA isn't about assigning blame to a component. It is an opportunity to think about the system holistically, and understand it better, and improve. Outsourcing this to LLMs strike me as a very stupid thing to do. We need to keep exercising those muscles, otherwise they would atrophy alongside our understanding of the system.

What will happen to the publication process?

In his insightful blog post on this paper, Brooker concludes:

Which leads systems to a tough spot. More bottlenecked than ever on the most difficult things to do. In some sense, this is a great problem to have, because it opens the doors for higher quality with less effort. But it also opens the doors for higher volumes of meaningless hill climbing and less insight (much of which we’re already seeing play out in more directly AI-related research). Conference organizers, program committees, funding bodies, and lab leaders will all be part of setting the tone for the next decade. If that goes well, we could be in for the best decade of systems research ever. If it goes badly, we could be in for 100x more papers and 10x less insight.

Given my firm belief in human laziness, I would bet on the latter. I have been predicting the collapse of the publishing system for a decade, and the flood of LLM-aided research may finally finish the job. That might not be a bad outcome either. We are due for a better model/process anyways.