This is an external post of mine. Click here if you are not redirected.

I had written earlier that the first step of my paper reading process is actually printing the paper. I like to physically touch the paper and handwrite and doodle in the margins. For years, a Pilot Metropolitan fountain pen loaded with Waterman blue-black ink was my weapon of choice for wrestling with the papers. For thinking hard and for getting things out of my chest (exploring how I feel about something), that I also relied on that fountain pen.

This past year, I served as a Program Committee member for SOSP, OSDI, NSDI, and ATC. This meant reviewing about 15 heavy-duty papers for each conference. My HP laser printer is 20 years old and has finally gotten glitchy (the paper feed is broken, requiring me to feed pages carefully lest it jams). Facing the review workload, I looked for a solution that could save my sanity, and some trees as well.

So I bought a reMarkable Paper Pro (RMPP). This is an e-ink reader/writer with color and 11.8 inch screen. I knew what I was getting into:  muted colors, screen flashing, and UI slowness. There are no software apps on the tablet; it is strictly a digital paper substitute.

The claim is that this forced minimalism is the appeal of RMPP compared to an iPad, which comes feature-maxxed and distraction-heavy. After a year of using the RMPP, I agree.

Beyond focus, the writing feel is a selling point. The RMPP does well here, whereas the iPad still feels like writing on glass (though I hear that PaperLike screen protector for iPad improves the experience by providing paper-like friction). Does the RMPP fully replace the soul of a Pilot Metropolitan? No, it doesn't. But it gets close enough for doing some longhand thinking, and it is editable and searchable (unlike my fountain pen).  

Battery life is another interesting trade-off. E-ink is incredibly power efficient, but since the RMPP screen is somewhat dark (far from a bright paper-white background), I have to use it with the backlight. This reduces the battery duration from weeks to a couple days, depending on use. Well, a standard iPad would also last through a full day of work as well, which makes it also acceptable. While e-ink may be better on the eyes compared to the iPad's LCD screen, and works well under sunlight, let's be honest, how many times are you actually going to use the RMPP outside?

Overall, I can't complain about the RMPP. It solved my paper reviewing use case, and serves as a scratchpad during Zoom meetings. It has also taken over as my notebook for longhand writing. But at the end of the day, it remains a niche product. Considering the cost of RMPP versus iPad, an iPadAir with 13 inch screen is likely a better deal (if you can de-claw its distractions). The RMPP gels really well with some people, but its limitations will rub many the wrong way.

So, do your own research before you get a tablet. But as I wrote in my previous post, having a plan to escape the computer screen is a smart move. It is getting harder to do deep work in front of a glowing monitor, and securing an "analog" escape (be it a digital tablet, or a physical notebook) is important for doing more focused and centered thinking.

I quit my job at EnterpriseDB hacking on PostgreSQL products last month to start a company researching and writing about software infrastructure. I believe there is space for analysis that is more focused on code than TechCrunch or The Register, more open to covering corporate software development than LWN.net, and (as much as I love some of these folks) less biased than VCs writing about their own investments.

I believe that more than ever there is a need for authentic and trustworthy analysis and coverage of the software we depend on.

This company, The Consensus, will talk about databases and programming languages and web servers and everything else that is important for experienced developers to understand and think about. It is independent of any software vendor and independent of any particular technology.

Some people were surprised (in a positive way) to see me cover MySQL already, for example. But that is exactly the point.

I don't want The Consensus to be just "Phil's thoughts". I have already started working with a number of experienced developers who will be writing, and paid to write, for The Consensus.

I also hope that this is another way, beyond the many communities I already run, to give back to the community such as in highlighting the work of open-source developers (the first interview with a DataFusion developer is coming soon), and highlighting compelling events and jobs in the software infrastructure world.

The Consensus is entirely bootstrapped and will depend on the support of subscribers and, potentially, sponsors. The first few subscribers signed up just this past week.

You can read more about the background and goals here, you can read about how contributors will work with The Consensus here, and you can get a sense for where this is going by browsing the homepage of The Consensus already.

Thank you for your support in advance! Thank you to the folks who have subscribed already despite very little fanfare. Feedback is very welcome. I'm very excited and having quite a bit of fun already. We're all going to learn a lot.

Last year I shared many posts documenting MariaDB performance for vector search using ann-benchmarks. Performance was great in MariaDB 11 and this blog post explains that it is even better in MariaDB 12. This work was done by Small Datum LLC and sponsored by the MariaDB Foundation. My previous posts were published in January and February 2025.

tl;dr

• Vector search recall vs precision in MariaDB 12.3 is better than in MariaDB 11.8
• Vector search recall vs precision in Maria 11.8 is better than in Postgres 18.2 with pgvector 0.8.1
• The improvements in MariaDB 12.3 are more significant for larger datasets
• MariaDB 12.3 has the best results because it use less CPU per query

A few years ago I started getting issues of Modern Luxury in the mail. I had no idea why they started coming, and I tried to get them to stop. This should have been easy, and was instead hard. Here’s my process, in case anyone else is in the same boat.

First, if you use it, try to unsubscribe via PaperKarma. This is convenient and works for a decent number of companies. PaperKarma kept reporting they’d successfully unsubscribed me, but Modern Luxury kept coming.

Second, write to subscriptions@modernluxury.com. I got no response.

Third, call any numbers you can find associated with the company. Leave voicemails on anything that claims to be Modern Luxury related. Along this path I wound up discovering a Borgesian labyrinth of sketchy offers for life-alert style emergency devices and other things that felt vaguely like elder abuse; long story short, this did not work.

Fourth, Modern Luxury’s email format is [first initial][last name]@modernluxury.com. Start writing emails to a few names from your local edition that seem relevant, like the local publisher and editor. When they don’t respond, expand your emails to include everyone listed in the magazine. Start digging through corporate filings of their parent company, Cumulus Media, and emailing people there. Start short and simple; when that doesn’t work, try humor. This didn’t work either, but it was fun to write:

I love me some esoteric rich people nonsense. Fabergé eggs! Ominous lawn obelisks! Having oneself taxidermied and wheeled out for council meetings of University College London! Unfortunately, Modern Luxury contains nothing like this; perhaps rich people have forgotten how to be interesting. In any event, I would like you to stop. If you can figure out how to stop sending me magazines, I promise to stop sending you emails about it, and we can all go on to live happy lives.

Contraluxuriantly,

Kyle Kingsbury

Finally, cut out a suitable article from an issue of the magazine. Look up up the home address of the regional group publisher in city records. Mail the article back to the publisher, along with a letter asking them to stop.

Dear Mr. Uslan,

As the regional group publisher of Modern Luxury magazine, I would like you to stop publishing Modern Luxury to my home each month. I never asked for it, and I have been trying to unsubscribe for years. E-mails, phone calls, Paper Karma: nothing works. I appreciate your most recent column, entitled “Spirit of Generosity”, but please: it is possible to be too generous. Kindly stop sending these magazines.

Exhaustedly,

Kyle Kingsbury

This actually seems to have worked.

I think a lot about this idea of the Annoyance Economy—that modern life places ordinary people in contact with a dizzying array of opaque, nonresponsive bureaucracies, and that those bureaucracies have financial incentives to ignore you. This is why it’s so hard to replace a CPAP or get paid back when movers break things. This is why Redplum (one of those advertising/coupon mailers) ignored my unsubscribe requests for years, and only stopped when I started e-mailing the entire C-suite about it. I try to pick and choose these battles, but sometimes it’s hard to let it go. And goshdarnit, if nobody pushes back then bureaucratic indifference works, and we all have to live with it.

I don’t want to bother people like this; I think it’s unreasonably rude. I still start with the official support channels and escalate gradually. I like Patrick McKenzie’s strategy of presenting oneself as a boring, dangerous professional. However, I have also found that in the Annoyance Economy, one of the ways to get things done is to find specific people with power, and annoy them right back.

I hope this whole misadventure convinced Modern Luxury to build and document an easy unsubscribe process. If not, you know what to do.

This is an external post of mine. Click here if you are not redirected.

In databases designed for high availability and scalability, secondary nodes can fall behind the primary. Typically, a quorum of nodes is updated synchronously to guarantee durability while maintaining availability, while remaining standby instances are eventually consistent to handle partial failures. To balance availability with performance, synchronous replicas acknowledge a write only when it is durable and recoverable, even if it is not yet readable.

As a result, if your application writes data and then immediately queries another node, it may still see stale data.

Here’s a common anomaly: you commit an order on the primary and then try to retrieve it from a reporting system. The order is missing because the read replica has not yet applied the write.

PostgreSQL and MongoDB tackle this problem in different ways:

• PostgreSQL 19 introduces a WAIT FOR LSN command, allowing applications to explicitly coordinate reads after writes.
• MongoDB provides causal consistency within sessions using the afterClusterTime read concern.

Both approaches track when your write occurred and ensure subsequent reads observe at least that point. Let’s look at how each database does this.

PostgreSQL: WAIT FOR LSN (PG19)

PostgreSQL records every change in the Write‑Ahead Log (WAL). Each WAL record has a Log Sequence Number (LSN): a 64‑bit position, typically displayed as two hexadecimal halves such as 0/40002A0 (high/low 32 bits).

Streaming replication ships WAL records from the primary to standbys, which then:

1. Write WAL records to disk
2. Flush them to durable storage
3. Replay them, applying changes to data files

The write position determines what can be recovered after a database crash. The flush position defines the recovery point for a compute instance failure. The replay position determines what queries can see on a standby.

WAIT FOR LSN allows a session to block until one of these points reaches a target LSN:

• standby_write → WAL written to disk on the standby (not yet flushed)
• standby_flush → WAL flushed to durable storage on the standby
• standby_replay (default) → WAL replayed into data files and visible to readers
• primary_flush → WAL flushed on the primary (useful when synchronous_commit = off and a durability barrier is needed)

A typical flow is to write on the primary, commit, and then fetch the current WAL insert LSN:

pg19rw=*# BEGIN;

BEGIN

pg19rw=*# INSERT INTO orders VALUES (123, 'widget');

INSERT 0 1

pg19rw=*# COMMIT;

COMMIT

pg19rw=# SELECT pg_current_wal_insert_lsn();

 pg_current_wal_insert_lsn
---------------------------
 0/18724C0

(1 row)

That LSN is then used to block reads on a replica until it has caught up:

pg19ro=# WAIT FOR LSN '0/18724C0'
  WITH (MODE 'standby_replay', TIMEOUT '2s');

This LSN‑based read‑your‑writes pattern in PostgreSQL requires extra round‑trips: capturing the LSN on the primary and explicitly waiting on the standby. For many workloads, reading from the primary is simpler and faster.

The pattern becomes valuable when expensive reads must be offloaded to replicas while still preserving read‑your‑writes semantics, or in event‑driven and CQRS designs where the LSN itself serves as a change marker for downstream consumers.

MongoDB: Causal Consistency

While PostgreSQL reasons in WAL positions, MongoDB tracks causality using oplog timestamps and a hybrid logical clock.

In a replica set, each write on the primary produces an entry in local.oplog.rs, a capped collection. These entries are rewritten to be idempotent (for example, $inc becomes $set) so they can be safely reapplied. Each entry carries a Hybrid Logical Clock (HLC) timestamp that combines physical time with a logical counter, producing a monotonically increasing cluster time. Replica set members apply oplog entries in timestamp order.

Because MongoDB allows concurrent writes, temporary “oplog holes” can appear: a write with a later timestamp may commit before another write with an earlier timestamp. A naïve reader scanning the oplog could skip the earlier operation.

MongoDB prevents this by tracking an oplogReadTimestamp, the highest hole‑free point in the oplog. Secondaries are prevented from reading past this point until all prior operations are visible, ensuring causal consistency even in the presence of concurrent commits.

Causal consistency in MongoDB is enforced by attaching an afterClusterTime to reads:

• Drivers track the operationTime of the last operation in a session.
• When a session is created with causalConsistency: true, the driver automatically includes an afterClusterTime equal to the highest known cluster time on subsequent reads.
• The server blocks the read until its cluster time has advanced beyond afterClusterTime.

With any read preference that allows reading from secondaries as well as the primary, this guarantees read‑your‑writes behavior:

// Start a causally consistent session
const session = client.startSession({ causalConsistency: true });

const coll = db.collection("orders");

// Write in this session
await coll.insertOne({ id: 123, product: "widget" }, { session });

// The driver automatically injects afterClusterTime into the read concern
const order = await coll.findOne({ id: 123 }, { session });

Causal consistency is not limited to snapshot reads. It applies across read concern levels. The key point is that the session ensures later reads observe at least the effects of earlier writes, regardless of which replica serves the read.

Conclusion

Here is a simplified comparison:

Both PostgreSQL’s WAIT FOR LSN and MongoDB’s causal consistency ensure reads can observe prior writes, but at different layers:

• PostgreSQL offers manual, WAL‑level precision.
• MongoDB provides automatic, session‑level guarantees.

If you want read‑your‑writes semantics to “just work” without additional coordination calls, MongoDB’s session‑based model is a strong fit. Despite persistent myths about consistency, MongoDB delivers strong consistency in a horizontally scalable system with a simple developer experience.

I remember the early 2010s as the golden age of productivity hacking. Lifehacker, 37signals, and their ilk were everywhere, and it felt like everyone was working on jury-rigging color-coded Moleskine task-trackers and web apps into the perfect Getting Things Done system.

So recently I found myself wondering: what happened to all that excitement? Did I just outgrow the productivity movement, or did the movement itself lose stream?

After poking around a bit, I think it's both. We collectively grew out of that phase, and productivity itself fundamentally changed.

The Trap of Micro-Optimizations

Back then, the underlying promise of productivity culture was about outputmaxxing (as we would now call it). We obsessed over efficiency at the margins: how to auto-sync this app with that one, or how to shave 5 seconds off an email reply. We accumulated systems, hacks, and integrations like collectors.

Eventually, the whole thing got exhausting. I think we all realized that tweaking task managers wasn't helping the bottom line. We were doing a lot of organizing, but that organizing wasn't reflecting in actually getting the work done.

The reason is simple: not all tasks matter equally. Making some tasks faster does not move the bottomline if the core task remains the serial bottleneck. Amdahl’s Law says that speeding up one part of a system improves overall performance only in proportion to the time that part consumes. If the hard, irreducible core is untouched, optimizations elsewhere are just noise. 

Painting the deck of a sinking ship faster doesn't help anyone. Productivity should be about making sure we are working on the right things in the first place. The main thing is to keep the main thing the main thing. 

Away From the Glowing Rectangle

For more than 15 years, I've relied on Emacs org-mode to run my life. It's the ultimate organization system, that has survived every software trend of the past decade and a half. But despite having this powerful writing system at my fingertips, my best ideas never arrive while I'm staring at a screen. Almost without exception, my hard thinking happens away from the screen. That's where the ideas come from.

If I'm being rational about it: I should be paid for the time I spend thinking hard, not for the time I spend managing my inbox, or doing trivial office work, or wrangling text on a screen.

So that's how I try to work now. I do my deep thinking, messy brainstorming, and wrestling-with-ideas completely away from the screen. Then I plan my next 45 minutes or so (what I'm going to do, in what order, and why) and only then do I go to my laptop to execute it. In other words, I arrive at the screen with a plan.

(OK, let's first take a moment to appreciate my self-restraint for not mentioning AI until this late in to the post. But here it comes.)

What does productivity even mean in the age of AI? What are we actually here to contribute? Are we supposed to be architects or butlers to LLMs?

If AI absorbs all the shallow work, the only things left that genuinely require a human are the core parts that demands genuine creativity, judgment, taste, and the type of thinking that can't be prompted away. This raises the stakes considerably, and changes what "a productive day" even means.

That kind of deep creative work is best done away from the glowing rectangle.


I recently launched a free email newsletter for the blog. Subscribe here to get these essays delivered to your inbox, along with behind-the-scenes commentary. 

A document database is more than a JSON datastore. It must also support efficient storage and advanced search: equality and range predicates, fuzzy text search, ranking, pagination, and limited sorted results (top‑k). BM25 indexes, which combine an inverted index and columnar doc values, are ideal for this, with mature open‑source implementations like Lucene (used by MongoDB) and Tantivy (used by ParadeDB).

ParadeDB brings Tantivy indexing to PostgreSQL via the pg_search extension and recently published an excellent article showing where GIN indexes fall short and how BM25 bridges the gap. Here, I’ll present the MongoDB equivalent using its Lucene‑based search indexes. I suggest reading ParadeDB’s post first, as it clearly explains the problem and the solution:

I'll be lazy and use the same dataset, index and query.

MongoDB with search indexes

You can use BM25 indexes on MongoDB in several environments: the cloud-managed service (MongoDB Atlas), its local deployment (Atlas Local), on-premises MongoDB Enterprise Server, and the open-source MongoDB Community edition. The mongot engine that powers MongoDB Search is in public preview, with its source available at github.com/mongodb/mongot.

I started a local Atlas deployment on my laptop with Atlas CLI and connected automatically:

atlas deployments setup  mongo --type local --connectWith mongosh --force 

Dataset generation

I generated 100,000,000 documents similar to ParadeDB's benchmark:

const batchSize = 10000;
const batches   = 10000;

const rows      = batches * batchSize;
print(`Generating ${rows.toLocaleString()} documents`);

db.benchmark_logs.drop();

const messages = [ 'The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.', 'The research facility analyzed samples from ancient artifacts, revealing breakthrough findings about civilizations lost to the depths of time.', 'The research station monitored weather patterns across mountain peaks, collecting data about atmospheric changes in the remote depths below.', 'The research observatory captured images of stellar phenomena, peering into the cosmic depths to understand the mysteries of distant galaxies.', 'The research laboratory processed vast amounts of genetic data, exploring the molecular depths of DNA to unlock biological secrets.', 'The research center studied rare organisms found in ocean depths, documenting new species thriving in extreme underwater environments.', 'The research institute developed quantum systems to probe subatomic depths, advancing our understanding of fundamental particle physics.', 'The research expedition explored underwater depths near volcanic vents, discovering unique ecosystems adapted to extreme conditions.', 'The research facility conducted experiments in the depths of space, testing how different materials behave in zero gravity environments.', 'The research team engineered crops that could grow in the depths of drought conditions, helping communities facing climate challenges.' ];

const countries = [ 'United States', 'Canada', 'United Kingdom', 'France', 'Germany', 'Japan', 'Australia', 'Brazil', 'India', 'China' ];

const labels = [ 'critical system alert', 'routine maintenance', 'security notification', 'performance metric', 'user activity', 'system status', 'network event', 'application log', 'database operation', 'authentication event' ];

let batch = [];
const startDate = new Date("2020-01-01T00:00:00Z");
for (let i = 0; i < rows; i++) {

  batch.push({
    message: messages[i % 10],
    country: countries[i % 10],
    severity: (i % 5) + 1,
    timestamp: new Date(startDate.getTime() + (i % 731) * 24 * 60 * 60 * 1000),
    metadata: {
      value: (i % 1000) + 1,
      label: labels[i % 10]
    }
  });

  if (batch.length === batchSize) {
    db.benchmark_logs.insertMany(batch);
    batch = [];
  }

}

I checked the document schema and counts:

print(`Done!
 \nSample: ${EJSON.stringify(  db.benchmark_logs.find().limit(1).toArray(),  null,  2  )}
 \nDocument count: ${db.benchmark_logs.countDocuments().toLocaleString()}
`);

Sample: [
  {
    "_id": {
      "$oid": "6997580679ab8450f81ff93c"
    },
    "message": "The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.",
    "country": "United States",
    "severity": 1,
    "timestamp": {
      "$date": "2020-01-01T00:00:00Z"
    },
    "metadata": {
      "value": 1,
      "label": "critical system alert"
    }
  }
]

Document count: 100,000,000

With 100 million documents, this is a large dataset. Because many fields can be queried, we can’t create every compound index combination. A single search index will make queries on this collection efficient.

Search index creation

I created the search index similar to the one used on ParadeDB (here):

const mapping = {
  mappings: {
    // Equivalent to: USING bm25 Atlas Search uses Lucene BM25 by default
    dynamic: false,
    fields: {
      // Equivalent to: bm25(id, message, ...) Standard full-text field scored by BM25
      message: { type: "string" },
      // Equivalent to: text_fields = {  "country": {  fast: true, tokenizer: { type: "raw", lowercase: true } } } // fast = true → implicit in Atlas Search; docValues optional in cloud
      country: { type: "string", analyzer: "keywordLowercase" },
      // Equivalent to:numeric field indexed for filtering
      severity: { type: "number", representation: "int64" },
      // Equivalent to:timestamp field included in the BM25 index
      timestamp: { type: "date" },
      // Equivalent to: json_fields = { "metadata": { fast: true, tokenizer: raw } }
      metadata: {
        type: "document",
        fields: {
          value: {
            type: "number",
            representation: "int64"
          },
          // Equivalent to: metadata tokenizer = raw + lowercase
          label: {
            type: "string",
            analyzer: "keywordLowercase"
          }
        }
      }
    }
  },
  analyzers: [
    {
      // Equivalent to: tokenizer = raw, lowercase = true
      name: "keywordLowercase",
      tokenizer: { type: "keyword" },
      tokenFilters: [{ type: "lowercase" }]
    }
  ]
};

db.benchmark_logs.createSearchIndex(
  "benchmark_logs_idx",
  mapping
);

The index is created asynchronously and updated via change stream operations.

Query and result

The query combines text search, range filter, sort by score, and limit for Top-K:

query = [
  {
    $search: {
      index: "benchmark_logs_idx",
      compound: {
        must: [{ text: { query: "research team", path: "message" } }],
        filter: [{ range: { path: "severity", lt: 3 } }]
      },
      sort: { score: { $meta: "searchScore" } }
    }
  },
  { $limit: 10 },
  {
    $project: {
      message: 1,
      country: 1,
      severity: 1,
      timestamp: 1,
      metadata: 1,
      rank: { $meta: "searchScore" }
    }
  }
]

const start = Date.now();

print(EJSON.stringify(db.benchmark_logs.aggregate(query).toArray(),null,2));

const end = Date.now();
print(`\nExecution time: ${end - start} ms`);

It is important that the sort is part of $search because an additional $sort stage would not be pushed down. This allows Atlas Search to run the query in Lucene’s Top‑K mode, enabling block‑max WAND (BMW) pruning via competitive score feedback during collection.

Here is the result and timing:

[{"_id":{"$oid":"699757049ce6a7c42c65d105"},"message":"The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.","country":"United States","severity":1,"timestamp":{"$date":"2020-01-11T00:00:00Z"},"metadata":{"value":11,"label":"critical system alert"},"rank":0.6839379072189331},{"_id":{"$oid":"699757049ce6a7c42c65d10f"},"message":"The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.","country":"United States","severity":1,"timestamp":{"$date":"2020-01-21T00:00:00Z"},"metadata":{"value":21,"label":"critical system alert"},"rank":0.6839379072189331},{"_id":{"$oid":"699757049ce6a7c42c65d119"},"message":"The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.","country":"United States","severity":1,"timestamp":{"$date":"2020-01-31T00:00:00Z"},"metadata":{"value":31,"label":"critical system alert"},"rank":0.6839379072189331},{"_id":{"$oid":"699757049ce6a7c42c65d123"},"message":"The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.","country":"United States","severity":1,"timestamp":{"$date":"2020-02-10T00:00:00Z"},"metadata":{"value":41,"label":"critical system alert"},"rank":0.6839379072189331},{"_id":{"$oid":"699757049ce6a7c42c65d12d"},"message":"The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.","country":"United States","severity":1,"timestamp":{"$date":"2020-02-20T00:00:00Z"},"metadata":{"value":51,"label":"critical system alert"},"rank":0.6839379072189331},{"_id":{"$oid":"699757049ce6a7c42c65d137"},"message":"The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.","country":"United States","severity":1,"timestamp":{"$date":"2020-03-01T00:00:00Z"},"metadata":{"value":61,"label":"critical system alert"},"rank":0.6839379072189331},{"_id":{"$oid":"699757049ce6a7c42c65d141"},"message":"The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.","country":"United States","severity":1,"timestamp":{"$date":"2020-03-11T00:00:00Z"},"metadata":{"value":71,"label":"critical system alert"},"rank":0.6839379072189331},{"_id":{"$oid":"699757049ce6a7c42c65d14b"},"message":"The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.","country":"United States","severity":1,"timestamp":{"$date":"2020-03-21T00:00:00Z"},"metadata":{"value":81,"label":"critical system alert"},"rank":0.6839379072189331},{"_id":{"$oid":"699757049ce6a7c42c65d155"},"message":"The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.","country":"United States","severity":1,"timestamp":{"$date":"2020-03-31T00:00:00Z"},"metadata":{"value":91,"label":"critical system alert"},"rank":0.6839379072189331},{"_id":{"$oid":"699757049ce6a7c42c65d15f"},"message":"The research team discovered a new species of deep-sea creature while conducting experiments near hydrothermal vents in the dark ocean depths.","country":"United States","severity":1,"timestamp":{"$date":"2020-04-10T00:00:00Z"},"metadata":{"value":101,"label":"critical system alert"},"rank":0.6839379072189331}]

Execution time: 1850 ms

On my laptop, this search over 100 million documents returns results in under two seconds, with no tuning. It performs a broad text match, and the high‑frequency terms "research" and "team" generate tens of millions of candidate documents. The additional severity filter and scoring require comparing tens of millions of scores, which has been heavily parallelized to stay within the two‑second budget.

Performance breakdown (explain)

Because the execution plan is long, I’ve packed it into a short string that you can easily copy and paste into your preferred AI chatbot:

EJSON.stringify(
db.benchmark_logs.aggregate(query).explain("executionStats")
);

{"explainVersion":"1","stages":[{"$_internalSearchMongotRemote":{"mongotQuery":{"index":"benchmark_logs_idx","compound":{"must":[{"text":{"query":"research team","path":"message"}}],"filter":[{"range":{"path":"severity","lt":3}}]},"sort":{"score":{"$meta":"searchScore"}}},"explain":{"query":{"type":"BooleanQuery","args":{"must":[{"path":"compound.must","type":"BooleanQuery","args":{"must":[],"mustNot":[],"should":[{"type":"TermQuery","args":{"path":"message","value":"research"},"stats":{"context":{"millisElapsed":1.273251,"invocationCounts":{"createWeight":2,"createScorer":87}},"match":{"millisElapsed":0},"score":{"millisElapsed":1292.607756,"invocationCounts":{"score":40000011}}}},{"type":"TermQuery","args":{"path":"message","value":"team"},"stats":{"context":{"millisElapsed":0.292666,"invocationCounts":{"createWeight":2,"createScorer":87}},"match":{"millisElapsed":0},"score":{"millisElapsed":379.190071,"invocationCounts":{"score":10000011}}}}],"filter":[],"minimumShouldMatch":0},"stats":{"context":{"millisElapsed":2.268162,"invocationCounts":{"createWeight":2,"createScorer":87}},"match":{"millisElapsed":0},"score":{"millisElapsed":3838.859709,"invocationCounts":{"score":40000011}}}}],"mustNot
                                    
                                    
                                    

                                        by Franck Pachot