Generating vector embeddings for semantic search locally
This is an external post of mine. Click here if you are not redirected.
This is an external post of mine. Click here if you are not redirected.
This has results for HammerDB tproc-c on a small server using MySQL and Postgres. I am new to HammerDB and still figuring out how to explain and present results so I will keep this simple and just share graphs without explaining the results.
tl;dr
(NOPM for some-version / NOPM for base-version)
I provide three charts below:
Results: MySQL 5.6 to 8.4
Legend:
Summary
Results: Postgres 12 to 18
Legend:
Summary
Results: MySQL vs Postgres
Legend:
Summary
Relational database joins are, conceptually, a cartesian product followed by a filter (the join condition). Without that condition, you get a cross join that returns every possible combination. In MongoDB, you can model the same behavior at read time using $lookup, or at write time by embedding documents.
Define two collections: one for clothing sizes and one for gender-specific fits:
db.sizes.insertMany([
{ code: "XS", neckCm: { min: 31, max: 33 } },
{ code: "S", neckCm: { min: 34, max: 36 } },
{ code: "M", neckCm: { min: 37, max: 39 } },
{ code: "L", neckCm: { min: 40, max: 42 } },
{ code: "XL", neckCm: { min: 43, max: 46 } }
]);
db.fits.insertMany([
{
code: "MEN",
description: "Straight cut, broader shoulders, narrower hips"
},
{
code: "WOMEN",
description: "Tapered waist, narrower shoulders, wider hips"
}
]);
Each collection stores independent characteristics, and every size applies to every fit. The goal is to generate all valid product variants.
In order to add all sizes to each body shape, use a $lookup without filter condition and, as it adds them as an embedded array, use $unwind to get one document per combination:
db.sizes.aggregate([
{
$lookup: {
from: "fits",
pipeline: [],
as: "fit"
}
},
{ $unwind: "$fit" },
{ $sort: { "fit.code": 1, code: 1 } },
{
$project: {
_id: 0,
code: { $concat: ["$fit.code", "-", "$code"] }
}
}
]);
For such small static reference collections, the application may simply read both and join with loops:
const sizes = db.sizes.find({}, { code: 1, _id: 0 }).sort({ code: 1 }).toArray();
const fits = db.fits.find({}, { code: 1, _id: 0 }).sort({ code: 1 }).toArray();
for (const fit of fits) {
for (const size of sizes) {
print(`${fit.code}-${size.code}`);
}
}
While it's good to keep the reference in a database, such static data can stay in cache in the application.
Because sizes are inherently tied to body shapes (no size exists without a body shape), embedding them in the fits documents is often a better model:
db.fits.aggregate([
{
$lookup: {
from: "sizes",
pipeline: [
{ $project: { _id: 0, code: 1, neckCm:1 } },
{ $sort: { code: 1 } }
],
as: "sizes"
}
},
{
$merge: {
into: "fits",
on: "_id",
whenMatched: "merge",
whenNotMatched: "discard"
}
}
]);
Here is the new shape of the single collection:
Once embedded, the query becomes straightforward, simply unwind the embedded array:
db.fits.aggregate([
{ $unwind: "$sizes" },
{
$project: {
_id: 0,
code: {
$concat: ["$code", "-", "$sizes.code"]
}
}
}
]);
You may embed only the fields required, like the size code, or all fields like I did here with the neck size, and then remove the size collection:
db.sizes.drop()
Although this may duplicate the values for each body shape, it only requires using updateMany() instead of updateOne() when updating it. For example, the following updates one size:
db.fits.updateMany(
{},
{ $set: { "sizes.$[s].neckCm": { min: 38, max: 40 } } },
{
arrayFilters: [
{ "s.code": "M" }
]
}
);
Duplication has the advantage of returning all required information in a single read, without joins or multiple queries, and it is not problematic for updates since it can be handled with a single bulk update operation. Unlike relational databases—where data can be modified through ad‑hoc SQL and business rules must therefore be enforced at the database level—MongoDB applications are typically domain‑driven, with clear ownership of data and a single responsibility for performing updates.
In that context, consistency is maintained by the application's service rather than by cross‑table constraints. This approach also lets business rules evolve, such as defining different sizes for men and women, without changing the data model.
In a fully normalized relational model, all relationships use the same pattern: a one-to-many relationship between two tables, enforced by a primary (or unique) key on one side and a foreign key on the other. This holds regardless of cardinality (many can be three or one million), lifecycle rules (cascade deletes or updates), ownership (shared or exclusive parent), navigation direction (and access patterns). Even many-to-many relationships are just two one-to-many relationships via a junction table.
MongoDB exposes these same concepts as modeling choices—handled at read time with $lookup, at write time through embedding, or in the application—instead of enforcing a single normalized representation. The choice depends on the domain data and access patterns.
Academic writing has long been criticized for its formulaic nature. As I wrote about earlier, research papers are unfortunately written to please 3 specific expert reviewers who are overwhelmingly from academia. Given this twisted incentive structure (looking impressive for peer-review), the papers end up becoming formulaic, defensive, and often inpenetrable.
Ironically, this very uniformity makes it trivially easy for LLMs to replicate academic writing. It is easy to spot LLM use in personal essays, but I dare you to do it successfully in academic writing.
Aside: Ok, I baited myself with my own dare. In general, it is very hard to detect LLM usage at the paragraph level in a research paper. But LLM usage in research papers becomes obvious when you see the same definition repeated 3-4 times across consecutive pages. The memoryless nature of LLMs causes them to recycle the same terms and phrases, and I find myself thinking "you already explained this to me four times, do you think I am a goldfish?" I have been reviewing a lot of papers recently, and this is the number one tell-tale sign. A careful read by the authors would clean this up easily, making LLM usage nearly undetectable. To be clear, I am talking about LLM assistance in polishing writing, not wholesale generation. A paper with no original ideas is a different beast entirely. They are vacuous and easy to spot.
Anyway, as LLM use become ubiquitous, conference/journal reviewing is facing a big crisis. There are simply too many articles being submitted, as it is easy to generate text and rush half-baked ideas into the presses. I am, of course, unhappy about this. Writing that feels effortless because an LLM smooths every step deprives you of the strain that produces "actual understanding". That strain in writing is not a defect; it creates the very impetus for discovering what you actually think, rather than faking/imitating thought.
But here we are. We are at an inflection point in academic publishing. I recently came across this post, which documents an experiment where an LLM replicated and extended a published empirical political science paper with near-human fidelity, at a fraction of the time and cost.
I have been predicting the collapse of the publishing system for a decade. The flood of LLM-aided research might finally break its back. And here is where I want to take you in this post. I want to imagine how academic writing may change in this new publishing regime. Call it a 5-10 year outlook, because at this day and age, who can predict anything beyond that.
I claim that costly signals of genuine intelligence will become the currency of survival in this new environment.
Costly signals work because they are expensive to fake, like a peacock’s tail or an elk’s antlers. And I claim academic writing will increasingly demand features that are expensive to fake. Therefore, a distinctive voice becomes more valuable precisely because it cannot be generated without genuine intellectual engagement. Personal narratives, peculiar perspectives, unexpected conceptual leaps, and field-specific cultural fluency are things that require deep immersion and creative investment that LLMs lack. These are the costly signals that will make a paper worth publishing.
Literature reviews are cheap to automate, so they will shrink --as we are already seeing. But reviews with distinctive voice and genuine insight, ones that reflect on the author's own learning and thought process, will survive. Work that builds creative frameworks and surprising connections, which are expensive to produce, will flourish. When anyone can generate competent prose, only writing that screams "a specific human spent serious time thinking about this" will cut through.
So, LLMs may accidentally force academia toward what it always claimed to value: original thinking and clear communication. The costliest signal of all is having something genuinely new to say, and saying well. I am an optimist, as you can easily tell, if you are a long time reader of this blog.
“Simplicity and elegance are unpopular because they require hard work and discipline to achieve and education to be appreciated.”
-- Edsger W. Dijkstra
Prisma is an ORM (Object-Relational Mapper). With MongoDB, it acts as an Object Document Mapper, mapping collections to TypeScript models and providing a consistent, type-safe query API.
MongoDB is a document database with a flexible schema. Prisma does not provide schema migrations for MongoDB, but it supports nested documents and embedded types to take advantage of MongoDB’s data locality.
This article walks through a minimal “Hello World” setup on a Docker environment:
Prisma requires MongoDB to run as a replica set. While MongoDB supports many operations without transactions, Prisma relies on MongoDB sessions and transactional behavior internally, which are only available on replica sets.
Start MongoDB in a Docker container with replica set support enabled:
docker run --name mg -d mongo --replSet rs0
Initialize the replica set (a single‑node replica set is sufficient for local development and testing):
docker exec -it mg mongosh --eval "rs.initiate()"
Start a Node.js container that can access MongoDB using the hostname mongo:
docker run --rm -it --link mg:mongo node bash
Update the package manager, install an editor, update npm, disable funding messages, and move to the working directory:
apt-get update
apt-get install -y vim
npm install -g npm@11.9.0
npm config set fund false
cd /home
Install Prisma Client and enable ES modules by adding "type": "module" to package.json:
npm install @prisma/client@6.19.0
sed -i '1s/{/{\n "type": "module",/' package.json
Using ES modules enables standard import syntax and aligns the project with modern Node.js tooling.
Install the Prisma CLI and supporting tooling, and generate the initial Prisma configuration:
npm install -D prisma@6.19.0 @types/node
npm install -D tsx
npx prisma init
Edit prisma/schema.prisma, change the provider from postgresql to mongodb, and define a minimal Message model:
generator client {
provider = "prisma-client"
output = "../generated/prisma"
}
datasource db {
provider = "mongodb"
url = env("DATABASE_URL")
}
model Message {
id String @id @default(auto()) @map("_id") @db.ObjectId
content String
createdAt DateTime @default(now())
}
Prisma maps MongoDB’s _id field to a String backed by an ObjectId.
The prisma-client generator produces TypeScript output in a custom directory to avoid using @prisma/client.
Define the MongoDB connection string in .env:
DATABASE_URL="mongodb://mongo:27017/test"
Prisma reads DATABASE_URL at generation time, while the application reads it at runtime. Importing dotenv/config ensures both environments are consistent.
Generate the Prisma client from the schema:
npx prisma generate
This produces TypeScript client files in generated/prisma.
Create prisma/index.ts:
import 'dotenv/config'
import { PrismaClient } from '../generated/prisma/client.ts'
const prisma = new PrismaClient()
async function main() {
await prisma.$connect()
console.log('Connected to MongoDB')
await prisma.message.create({
data: {
content: 'Hello World',
},
})
const messages = await prisma.message.findMany()
console.log('Messages in database:')
for (const message of messages) {
console.log(`- ${message.content} at ${message.createdAt}`)
}
}
main()
.catch(console.error)
.finally(() => prisma.$disconnect())
This program connects to MongoDB, inserts a “Hello World” document, and prints all stored messages.
For running TypeScript directly in modern Node.js projects, tsx is generally preferred over ts-node due to better ESM support and faster startup.
Execute the TypeScript file:
npx tsx prisma/index.ts
Output:
Connected to MongoDB
Messages in database:
- Hello World at Wed Feb 11 2026 17:36:08 GMT+0000 (Coordinated Universal Time)
This example shows a minimal Prisma + MongoDB setup:
From here, you can add schema evolution, indexes, and more complex queries while keeping the same core configuration.
MongoDB is often called schemaless, but that’s misleading in practice, as we started to declare the database schema in schema.prisma and generate the client for it. Real‑world MongoDB applications are schema‑driven, with structure defined in the application layer through models, validation rules, and access patterns.
Unlike relational databases—where the schema is enforced in the database and then mapped into the application—MongoDB uses the same document structure across all layers: in‑memory cache, on‑disk storage, and application models. This preserves data locality, avoids ORM overhead and migration scripts, and simplifies the development.
Prisma makes this explicit by defining the schema in code, providing type safety and consistency while keeping MongoDB’s document model flexible as your application evolves.
The crux of this chapter is how to schedule tasks without perfect knowledge. If you remember from the previous chapter, the core tension in CPU scheduling is these two conflicting goals:
Unfortunately, the OS does not have a crystal ball. It doesn't know if a process is a short interactive job or a massive number-crunching batch job. The Multi-Level Feedback Queue (MLFQ) solves this by encoding/capturing information from history of the job, and assumes that if a job has been CPU-intensive in the past, it likely will be in the future. As we'll see below, it also gives a chance for jobs to redeem themselves through the boosting process.
I really enjoyed this chapter. MLFQ, invented by Corbato in 1962, is a brilliant scheduling algorithm. This elegant solution served as the base scheduler for many systems, including BSD UNIX derivatives, Solaris, and Windows NT and subsequent Windows operating systems.
(This is part of our series going through OSTEP book chapters.)
The chapter constructs the MLFQ algorithm iteratively, starting with a basic structure involving distinct queues, each with a different priority level.
But how does a job get its priority?
This setup cleverly approximates Shortest Job First. Because the scheduler assumes every new job is short (giving it high priority), true short jobs finish quickly. Long jobs eventually exhaust their time slices and sink to the bottom queues, where they run only when the system isn't busy with interactive tasks.
However, this basic version has fatal flaws.
To fix these issues, the chapter introduces two crucial modifications.
The Priority Boost: To prevent low-priority jobs from starving, the scheduler employs Rule 5: After a set time period (S), all jobs are moved back to the topmost queue. This "boost" ensures that CPU-bound jobs get at least some processing time and allows jobs that have become interactive to return to a high-priority state.
Better Accounting: To stop users from gaming the system, the scheduler rewrites Rule 4 regarding how it tracks time. Rule 4: Instead of resetting the allotment every time a job yields the CPU, the scheduler tracks the total time a job uses at a given priority level. Once the allotment is used up (regardless of how many times the job yielded the CPU) it is demoted.
The remaining piece of the puzzle is parameterization. An MLFQ requires choosing the number of queues, the time slice length for each, and the frequency of the priority boost. There are no easy answers to these questions, and finding a satisfactory balance often requires deep experience with specific workloads. For example, most implementations employ varying time-slice lengths, assigning short slices (e.g., 10 ms) to high-priority queues for responsiveness and longer slices (e.g., 100s of ms) to low-priority queues for efficiency. Furthermore, the priority boost interval is often referred to as a "voodoo constant" because it requires magic to set correctly; if the value is too high, jobs starve, but if it is too low, interactive performance suffers.
MLFQ is a milestone in operating systems design. It delivers strong performance for interactive jobs without prior knowledge of job length, while remaining fair to long-running tasks. As noted earlier, it became the base scheduler for many operating systems, with several variants refining the core idea. One notable variant is the decay-usage approach used in FreeBSD 4.3. Instead of using fixed priority tables (as in Solaris), it computes priority using a mathematical function of recent CPU usage. Running increases a job’s usage counter and lowers its priority, while the passage of time decays this counter. Decay plays the same role as periodic priority boosts. As usage fades, priority rises, ensuring long-running jobs eventually run and allowing jobs that shift from CPU-bound to interactive to regain high priority.
I used Gemini to write a TLA+ model of the MLFQ algorithm here. To run this MLFQ TLA+ model at Spectacle for visualization, click this link and it will open the model on your browser, no installation or plugin required. What you will see is the initial state. Click on any enabled action to take it, you can go back and forward on the right pane to explore the execution. And you can share a URL back with anyone to point to an interesting state or trace, just like I did here.
My recent LinkedIn post (below, with minor changes) provides the motivation for the more detailed deep dive presented in this article:
All this snow has me thinking about skiing (again). And data. And databases. I was inspired by the “Analyzing Stack Overflow data with ClickHouse” article I read, thinking how amazing it is they just raised $400M. After working through their examples, it occurred to me that, as fun as that was, the SQL queries there were very simple. In skiing jargon, I might liken them to the green ski runs, for beginners.
As skiers progress, they move to the blue intermediate runs, then may get bold and go try the black diamond runs. That’s the direction I headed with the Stack Overflow data set. ClickHouse provides a convenient Parquet export function, which I was able to use in conjunction with CedarDB’s Parquet import functionality to load my CedarDB instance with the data set I’d used in ClickHouse. I deployed each of the DB’s on my m7a.8xlarge (32 vCPU, 128 GB RAM, $1.85/hour) EC2 node, in Docker, running them one at a time. A little pair programming with ChatGPT yielded a set of seven interesting queries, which I ran on each of these DBs, discarding the initial run time and recording the average timing for five runs.
The Stack Overflow data set appeals to us for several reasons, the first of which is that many of us can relate to it, having used the Web site over the years. Some other aspects are:
SELECT
arrayJoin(arrayFilter(t -> (t != ''), splitByChar('|', Tags))) AS Tags,
count() AS c
FROM stackoverflow.posts
GROUP BY Tags
ORDER BY c DESC
LIMIT 10
SELECT
q."Id" AS question_id,
q."Title" AS question_title,
qu."DisplayName" AS question_owner,
a."Id" AS answer_id,
au."DisplayName" AS answer_owner,
b."Name" AS answerer_badge
FROM posts q
JOIN posts a
ON a."Id" = q."AcceptedAnswerId"
JOIN users qu
ON qu."Id" = q."OwnerUserId"
JOIN users au
ON au."Id" = a."OwnerUserId"
LEFT JOIN badges b
ON b."UserId" = au."Id"
WHERE q."AcceptedAnswerId" IS NOT NULL
AND q."Title" <> ''
AND q."CreationDate" >= TIMESTAMP '2019-01-01'
ORDER BY q."CreationDate" DESC
LIMIT 10;
-- Clickhouse:
ip-10-0-1-175.us-east-2.compute.internal :) SELECT *
:-] FROM votes
:-] INTO OUTFILE 'votes.parquet'
:-] FORMAT Parquet;
-- CedarDB:
postgres=# CREATE TABLE votes AS SELECT * FROM '/var/lib/cedardb/data/ext/votes.parquet';
SELECT 238984011
Time: 46287.163 ms (00:46.287)
Refer to Appendix: Après-Ski Details, below, for the step-by-step procedure.
Here are the questions that our SQL queries will ask, numbered to correspond with the numbered files used in the experiments and also with the entries in the results table (below). To view a SQL query, click the right arrow symbol next to its number.
SELECT
b."Name" AS badge_name,
COUNT(*) AS badge_awards
FROM badges b
JOIN users u
ON u."Id" = b."UserId"
JOIN posts p
ON p."OwnerUserId" = u."Id"
JOIN comments c
ON c."PostId" = p."Id"
WHERE u."Reputation"::int >= 100000
AND p."CreationDate" >= TIMESTAMP '2018-01-01'
GROUP BY b."Name"
ORDER BY badge_awards DESC
LIMIT 10;
WITH vote_counts AS (
SELECT
v."PostId",
SUM(CASE WHEN v."VoteTypeId" = 2 THEN 1 ELSE 0 END) AS upvotes,
SUM(CASE WHEN v."VoteTypeId" = 3 THEN 1 ELSE 0 END) AS downvotes
FROM votes v
GROUP BY v."PostId"
)
SELECT
p."Id",
p."Title",
u."DisplayName" AS owner,
vc.upvotes,
vc.downvotes,
(CASE WHEN vc.upvotes > vc.downvotes THEN vc.upvotes - vc.downvotes ELSE vc.downvotes - vc.upvotes END) AS abs_diff,
COUNT(c."Id") AS comment_cnt
FROM vote_counts vc
JOIN posts p
ON p."Id" = vc."PostId"
JOIN users u
ON u."Id" = p."OwnerUserId"
LEFT JOIN comments c
ON c."PostId" = p."Id"
WHERE p."Title" <> ''
AND vc.upvotes >= 50
AND vc.downvotes >= 50
GROUP BY p."Id", p."Title", u."DisplayName", vc.upvotes, vc.downvotes
ORDER BY abs_diff ASC, comment_cnt DESC
LIMIT 10;
SELECT
q."Id" AS question_id,
q."Title" AS question_title,
qu."DisplayName" AS question_owner,
a."Id" AS answer_id,
au."DisplayName" AS answer_owner,
b."Name" AS answerer_badge
FROM posts q
JOIN posts a
ON a."Id" = q."AcceptedAnswerId"
JOIN users qu
ON qu."Id" = q."OwnerUserId"
JOIN users au
ON au."Id" = a."OwnerUserId"
LEFT JOIN badges b
ON b."UserId" = au."Id"
WHERE q."AcceptedAnswerId" IS NOT NULL
AND q."Title" <> ''
AND q."CreationDate" >= TIMESTAMP '2019-01-01'
ORDER BY q."CreationDate" DESC
LIMIT 10;
WITH vote_counts AS (
SELECT
v."PostId",
COUNT(*) AS votes_total
FROM votes v
GROUP BY v."PostId"
)
SELECT
p."Id",
p."Title",
owner."DisplayName" AS owner_name,
editor."DisplayName" AS editor_name,
editor."Reputation" AS editor_rep,
COALESCE(vc.votes_total, 0) AS votes_total,
COUNT(c."Id") AS comment_cnt
FROM posts p
JOIN users owner
ON owner."Id" = p."OwnerUserId"
JOIN users editor
ON editor."Id" = p."LastEditorUserId"
LEFT JOIN vote_counts vc
ON vc."PostId" = p."Id"
LEFT JOIN comments c
ON c."PostId" = p."Id"
WHERE p."LastEditorUserId" IS NOT NULL
AND p."Title" <> ''
AND p."OwnerUserId" <> p."LastEditorUserId"
GROUP BY
p."Id", p."Title", owner."DisplayName", editor."DisplayName", editor."Reputation", vc.votes_total
ORDER BY votes_total DESC, comment_cnt DESC
LIMIT 10;
SELECT
pl."PostId" AS src_post_id,
src."Title" AS src_title,
su."DisplayName" AS src_owner,
pl."RelatedPostId" AS dst_post_id,
dst."Title" AS dst_title,
du."DisplayName" AS dst_owner,
pl."CreationDate" AS link_time
FROM postlinks pl
JOIN posts src
ON src."Id" = pl."PostId"
JOIN posts dst
ON dst."Id" = pl."RelatedPostId"
LEFT JOIN users su
ON su."Id" = src."OwnerUserId"
LEFT JOIN users du
ON du."Id" = dst."OwnerUserId"
WHERE src."Title" <> ''
MongoDB guarantees durability—the D in ACID—over the network with strong consistency—the C in the CAP theorem—by default. It still maintains high availability: in the event of a network partition, the majority of nodes continue to serve consistent reads and writes transparently, without raising errors to the application.
A consensus protocol based on Raft is used to achieve this at two levels:
It's important to distinguish the two types of consensus involved: one for controlling replica roles and one for the replication of data itself. By comparison, failover automation around monolithic databases like PostgreSQL can use a consensus protocol to elect a primary (as Patroni does), but replication itself is built into PostgreSQL and does not rely on a consensus protocol—a failure in the middle may leave inconsistency between replicas.
Consensus on writes increases latency, especially in multi-region deployments, because it requires synchronous replication and waiting on the network, but it guarantees no data loss in disaster recovery scenarios (RPO = 0). Some workloads may prefer lower latency and accept limited data loss (for example, a couple of seconds of RPO after a datacenter burns). If you ingest data from IoT devices, you may favor fast ingestion at the risk of losing some data in such a disaster. Similarly, when migrating from another database, you might prefer fast synchronization and, in case of infrastructure failure, simply restart the migration from before the failure point. In such cases, you can use {w:1} write concern in MongoDB instead of the default {w:"majority"}.
Most failures are not full-scale disasters where an entire data center is lost, but transient issues with short network disconnections. With {w:1}, the primary risk is not data loss—because writes can be synchronized eventually—but split brain, where both sides of a network partition continue to accept writes. This is where the two levels of consensus matter:
{w:"majority"}, writes that cannot reach a majority are not acknowledged on the side of the partition without a quorum. This prevents split brain. However, with {w:1}, those writes are acknowledged until the old primary steps down.Because the failure is transient, when the old primary rejoins, no data is physically lost: writes from both sides still exist. However, these writes may conflict, resulting in a diverging database state with two branches. As with any asynchronous replication, this requires conflict resolution. MongoDB handles this as follows:
Thus, when you use {w:1}, you accept the possibility of limited data loss in the event of a failure. Once the node is back, these writes are not entirely lost, but they cannot be merged automatically. MongoDB stores them as BSON files in a rollback directory so you can inspect them and perform manual conflict resolution if needed.
This conflict resolution is a Recover To a Timestamp (RTT).
Let's try it. I start 3 containers as a replica set:
docker network create lab
docker run --network lab --name m1 --hostname m1 -d mongo --replSet rs0
docker run --network lab --name m2 --hostname m2 -d mongo --replSet rs0
docker run --network lab --name m3 --hostname m3 -d mongo --replSet rs0
docker exec -it m1 mongosh --eval '
rs.initiate({
_id: "rs0",
members: [
{ _id: 0, host: "m1:27017", priority: 3 },
{ _id: 1, host: "m2:27017", priority: 2 },
{ _id: 2, host: "m3:27017", priority: 1 }
]
})
'
until
docker exec -it m1 mongosh --eval "rs.status().members.forEach(m => print(m.name, m.stateStr))" |
grep -C3 "m1:27017 PRIMARY"
do sleep 1 ; done
The last command waits until m1 is the primary, as set by its priority. I do that to make the demo reproducible with simple copy-paste.
I insert "XXX-10" when connected to m1:
docker exec -it m1 mongosh --eval '
db.demo.insertOne(
{ _id:"XXX-10" , date:new Date() },
{ writeConcern: {w: "1"} }
)
'
{ acknowledged: true, insertedId: 'XXX-10' }
I disconnect the secondary m2:
docker network disconnect lab m2
With a replication factor of 3, the cluster is resilient to one failure and I insert "XXX-11", when connected to the primary:
docker exec -it m1 mongosh --eval '
db.demo.insertOne(
{ _id:"XXX-11" , date:new Date() },
{ writeConcern: {w: "1"} }
)
'
{ acknowledged: true, insertedId: 'XXX-11' }
I disconnect m1, the current primary, and reconnect m2, and immediately insert "XXX-12", still connected to m1:
docker network disconnect lab m1
docker network connect lab m2
docker exec -it m1 mongosh --eval '
db.demo.insertOne(
{ _id:"XXX-12" , date:new Date() },
{ writeConcern: {w: "1"} }
)
'
{ acknowledged: true, insertedId: 'XXX-12' }
Here, m1 is still a primary for a short period before it detects it cannot reach the majority of replicas and steps down. If the write concern was {w: "majority"} it would have waited and failed, not able to sync to the quorum, but with {w: "1"} the replication is asynchronous and the write is acknowledged when written to local disks.
Two seconds later, a similar write fails because the primary stepped down:
sleep 2
docker exec -it m1 mongosh --eval '
db.demo.insertOne(
{ _id:"XXX-13" , date:new Date() },
{ writeConcern: {w: "1"} }
)
'
MongoServerError: not primary
I wait that m2 is the new primary, as set by priority, and connect to it to insert "XXX-20":
until
docker exec -it m2 mongosh --eval "rs.status().members.forEach(m => print(m.name, m.stateStr))" |
grep -C3 "m2:27017 PRIMARY"
do sleep 1 ; done
docker exec -it m2 mongosh --eval '
db.demo.insertOne(
{ _id:"XXX-20" , date:new Date() },
{ writeConcern: {w: "1"} }
)
'
{ acknowledged: true, insertedId: 'XXX-20' }
No nodes are down, it's only a network partition, and I can read from all nodes as long as I don't connect through the network. I query the collection on each side:
docker exec -it m1 mongosh --eval 'db.demo.find()'
docker exec -it m2 mongosh --eval 'db.demo.find()'
docker exec -it m3 mongosh --eval 'db.demo.find()'
The inconsistency is visible, "XXX-12" is only in m1 and "XXX-20" only in m2 and m3:
I reconnect m1 so that all nodes can communicate and synchronize their state:
docker network connect lab m1
I query again and all nodes show the same values:
"XXX-12" has disappeared and all nodes are now synchronized to the current state. When it rejoined, m1 rolled back the operations that occurred during the split-brain window. This is expected and acceptable, since the write used a { w: 1 } write concern, which explicitly allows limited data loss in case of failure in order to avoid cross-network latency on each write.
The rolled back operations are not lost, MongoDB logged them in a rollback directory in the BSON format, with the rolled back document as well as the related oplog.
I read and decode all BSON in the rollback directory:
docker exec -i m1 bash -c '
for f in /data/db/rollback/*/removed.*.bson
do
echo "$f"
bsondump $f --pretty
done
' | egrep --color=auto '^|^/.*|.*("op":|"XXX-..").*'
The deleted document is in /data/db/rollback/0ae03154-0a51-4276-ac62-50d73ad31fe0/removed.2026-02-10T10-40-58.1.bson:
{
"_id": "XXX-12",
"date": {
"$date": {
"$numberLong": "1770719868965"
}
}
}
The deleted oplog for the related insert is in /data/db/rollback/local.oplog.rs/removed.2026-02-10T10-40-58.0.bson:
{
"lsid": {
"id": {
"$binary": {
"base64": "erR2AoFXS3mbcX4BJSiWjw==",
"subType": "04"
}
},
"uid": {
"$binary": {
"base64": "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=",
"subType": "00"
}
}
},
"txnNumber": {
"$numberLong": "1"
},
"op": "i",
"ns": "test.demo",
"ui": {
"$binary": {
"base64": "CuAxVApRQnasYlDXOtMf4A==",
"subType": "04"
}
},
"o": {
"_id": "XXX-12",
"date": {
"$date": {
"$numberLong": "1770719868965"
}
}
},
"o2": {
"_id": "XXX-12"
},
"stmtId": {
"$numberInt": "0"
},
"ts": {
"$timestamp": {
"t": 1770719868,
"i": 1
}
},
"t": {
"$numberLong": "1"
},
"v": {
"$numberLong": "2"
},
"wall": {
"$date": {
"$numberLong": "1770719868983"
}
},
"prevOpTime": {
"ts": {
"$timestamp": {
"t": 0,
"i": 0
}
},
"t": {
"$numberLong": "-1"
}
}
}
By default, MongoDB favors strong consistency and durability: writes use { w: "majority" }, are majority-committed, never rolled back, and reads with readConcern: "majority" never observe rolled-back data. In this mode, MongoDB behaves like a classic Raft system: once an operation is committed, it is final.
MongoDB also lets you explicitly relax that guarantee by choosing a weaker write concern such as { w: 1 }. In doing so, you tell the system: "Prioritize availability and latency over immediate global consistency." The demo shows what that implies:
This rollback behavior is where MongoDB intentionally diverges from vanilla Raft.
In classic Raft, the replicated log is the source of truth, and committed log entries are never rolled back. Raft assumes a linearizable, strongly consistent state machine where the application does not expect divergence. MongoDB, by contrast, comes from a NoSQL and event-driven background, where asynchronous replication, eventual consistency, and application-level reconciliation are sometimes acceptable trade-offs.
As a result:
In short, MongoDB replication is based on Raft, but adds rollback semantics to support real-world distributed application patterns. Rollbacks happen only when you explicitly allow them, never with majority writes, and they are fully auditable and recoverable.
Despite the ubiquity of the MongoDB aggregation framework, it has been lacking a formal mathematical framework/specification. This paper aims to fix this gap by providing a theoretical foundation, and proposes MQuery. The formalization in MQuery is largely based on the paper published at ICDT 2018 (for which the first author is involved), extending it to include more pipeline operators, relax the assumption that the JSON documents stored in the database comply to a predefined schema, and allow objects that are either ordered or unordered sets of key-value pairs.
For decades, SQL proponents have flaunted the rigorous mathematical foundation of relational algebra (courtesy of Edgar Codd). The world of JSON document databases, however, has remained a bit of a Wild West in comparison. The analogy is apt because, like the frontier, there is immense opportunity here. JSON is the undisputed king of data exchange, and the MongoDB aggregation framework has emerged as the widely adopted query language for JSON collections. Thanks to its expressive pipeline model, massive developer base, and popularity, MongoDB aggregation framework has effectively become the de facto standard for querying JSON. The fact that major vendors (including Amazon, Microsoft, Oracle, and Google) seek to provide compatibility with the MongoDB API further underscores its recognition as a common lingua franca. (The authors' words, not mine, so don't think I'm bragging on behalf of MongoDB.)
To further motivate the need for a rigorous mathematical framework, the authors highlight current challenges. They argue that MongoDB's semantics are procedural rather than declarative. While the aggregation pipeline is pragmatic and powerful, its documentation often overlooks edge cases, leading to ambiguity.
The paper illustrates this with an example about query predicates. In MongoDB, the query origin: "UK" matches a document where origin is the string "UK". However, it also matches a document where origin is the array ["UK", "Japan"]. While this loose equality is convenient for developers, it is bad for mathematical logic, as it violates the property of transitivity: ["UK"] matches "UK", and "UK" matches "UK", yet [["UK"]] does not match "UK".
Furthermore, the paper argues MongoDB suffers from path polysemy. A path like origin.country is inherently ambiguous. Does it refer to a nested field in a single object? Or, if origin is an array, does it refer to the country field of every object inside that array? This leads to data-dependent behavior, where a valid query might throw a runtime error simply because a new document with a different structure was inserted into the collection.
MQuery (which, admittedly, looks a lot like McQuery, and now that I've said this, you won't be able to read it any other way) serves as a formalized abstraction of the MongoDB language. MQuery formalizes the data model using "d-values" (document values), which encompass literals, arrays, and objects. It also defines 7 core pipeline stages that mirror the MongoDB aggregation framework: $match, $unwind, $project, $group, $lookup, $graphLookup, and $union.
By formalizing these stages, the authors confirm in Section 4 that "the MongoDB aggregation framework is very expressive: at least as expressive as full relational algebra (RA)". They mention:
They demonstrate that $match corresponds to selection, $project to projection, and $lookup (or a combination of $unwind and $group) to joins. This confirms that document databases can theoretically perform every operation relational databases can, including complex joins and set operations. They also note the MongoDB aggregation framework goes beyond RA by handling Nested Relational Algebra and linear recursion via $graphLookup.
Why does all this math matter? The formal definition can help us safely optimize queries. The final section of the paper demonstrates algebraic rewriting rules. Thanks to the formal definitions, the authors can prove when it is safe to reorder pipeline stages without altering the result. They provide rules for filter anticipation (moving $match earlier to reduce data volume), unnesting postponement (moving $unwind later to save memory), and join optimization.