Jim Gray started HPTS 42 years ago, and it still runs the same way at Asilomar with 0 proceedings, 1 track, 100 invited people, and ample discussion breaks. It's one of my favorite events, and I look forward to attending it every two years. This is my...umm...fifth HPTS. I have been attending all of them since 2015, but I seem to have missed it in 2019. Yep, my life is a blur, even for me.

Here are my notes from day 1. I will put day 2 and day 3 in a second post.

Keynote: Reality is the Final Verifier. Two Key Gaps in Agentic Software Engineering (Ion Stoica, UC Berkeley)

Ion started with the promising parts. LLMs are able to solve difficult math problems, and provide proofs for conjectures that have been outstanding for many decades (Ok, maybe this is not as promising to mathematicians). LLMs also play scientist with AlphaEvolve, and OpenEvolve. In summer 2025 Berkeley ran a seminar where students applied agentic tools to research problems. They took a published state-of-the-art result, used it as the baseline, kept the objective fixed, and let the loop run. Ten of twelve students got better results than the published baseline. The ADRS blog series and papers come out of this work.

But then comes the reality check. Yes, AI seems to improve researcher productivity, but it did not disrupt how we do research. It cannot come up with new research ideas. It cannot build new production systems. The rest of the talk was his attempt to explain this gap.

Ion led on with an anecdote from the Jitski project, where they asked an agent to synthesize a key-value store from scratch to maximize throughput. They applied the standard AI development loop. And surprisingly, one solution came back 6x faster than anything before it. How? The benchmark was YCSB, and YCSB generates its values with a hash function. The agent noticed this and stopped storing values altogether and computed them on the fly with get(key) = hash(key). This is the very definition of "reward hacking".

Ion frames this in terms of two gaps:

• Requirement gap: between the user's intentions and what the prompt and environment capture. Intentions are always broader than the prompt.
• Model gap: between the model of the world and the world. The real world is always broader than the model.

Ion elaborated more on why closing the requirement gap is hard. He asked the room: would you think to specify that "a key-value store must actually store values", "a load balancer should actually process the requests", and "a vacuum shouldn't spill garbage on the floor to pick back up as a reward hack"?

As a formal methods person: yes! Yes, I totally would. These would be about the first things I'd write to stop a vacuous specification satisfaction. That is exactly what writing a specification is. You state the thing that is so obvious which most people would not bother to say. Ion raised these examples and then waved them off by mocking how silly they sound, which to me (and other formal methods people) is backwards.

For illustrating the model gap, he reached for Brian Cantwell Smith's "The Limits of Correctness." On October 5, 1960, the Ballistic Missile Early Warning System station at Thule reported a massive incoming attack, which was about to trigger a nuclear retaliation. But the reality was different. The moon had risen, and the radar was getting its own signal back off the lunar surface. The system was recently deployed and nobody had accounted for the fricking moon. The model was fine, but the world was bigger than the model.

Note that the agentic success stories are all in domains with narrow gaps. In mathematics, the model gap is closed by construction, because the axiomatic system is the world, so the model can match it exactly. The requirement gap stays narrow, since translating intent into a formal theorem statement has already been done by a mathematician. Hardware design is similar. And optimizing an existing system is much easier than building one, because the existing system pins things down and narrows the specification.

The two gaps give birth to two bottlenecks. The requirement gap bottlenecks on human judgment and the model gap bottlenecks on evaluation. Ion's prescription to alleviate these gaps is an outer loop assurance revision loop with (again) humans involved to improve the process. He listed some future directions: selective escalation (ask only the highest-value human question), iterative disambiguation, cascading evaluations, impact-bounded control, continual system learning.

In disguise, this was a talk about how badly we need specifications. It looks like people are rediscovering the importance of specifications, as LLMs outsmart and bedevil them. Look at day 1! Three of the four sessions were about AI, and they all mention the importance of specification! When you can't trust the generator, you need to trust something you can check. And that is the formal methods argument. This was the vibe for HPTS'26. Low key, I felt like a rockstar in HPTS. Whoa, I know TLA+!

Student intros

One of the best traditions at HPTS is that students get dedicated time to introduce themselves. Among this year's cohort, there were an unusual number of piano players, a healthy number of odd hobbies, and someone who does Byzantine music (which I presume requires 3F+1 singers).

A note for the record. We will have AGI before we solve projecting a laptop onto a screen. Twenty-six years I have been going to conferences, and it is the same dance every time: the connector is incompatible, the resolution is awry, the mirrored display doesn't work, and people hunch over the laptop to fix it. This scenario played out at least 5 times to sabotage/delay presentations over the two and a half days. The people in this room keep trillions of blobs available across the world, but somehow getting the projector playing well with the laptop is still unreliable.

Session 2: Putting database researchers out of a job 

The Future Is Bespoke: Synthesizing One-Size-Fits-One DBMSs with LLM Coding Agents (Carsten Binnig, TU Darmstadt)

Carsten opened by playing the room, saying "relational databases are the foundation of civilization" (think clay tablets Sumerian traders used for recording transactions). He then argued that bespoke databases already happened. TigerBeetle is 3000x faster than Postgres for what it does. TigerBeetle only exists because it targets a market big enough to pay for it. But bespoke for everybody has never worked, because the engineering cost per customer is absurd. LLMs change the cost side of course. Feed in the schema, the data, and the workload queries, and synthesize an OLAP DBMS for exactly that workload. Correctness has an oracle here, because DuckDB serves as a trusted reference implementation to check the results against. The synthesis proceeds in layers: start general purpose, then specialize storage, then execution, then the concurrency control scheme. Ten hours and $140 per synthesized system. Future work is to extend the idea to bespoke compute stacks, where the application is codesigned with the OS and the hardware.

Towards AI-Generated Database Management Systems (Jiale Lao, Cornell)

Jiale started from the same premise as Carsten, fast = f(hardware, workload), but went a different direction on correctness. He pointed out that even the best databases, DuckDB included, return wrong results. No database is formally verified; they are all tested, and testing shows the presence of bugs, not their absence. So where Carsten uses DuckDB as the verifier, Jiale wants to not need a verifier at all. The goal is generating and proving, rather than generating and testing. The tools are Verus for verifying Rust, and SQLSpec, a mathematical specification of SQL, as the thing you prove against. They are at about 75% proved, with 3x amplification (rather than 20x with CompCert). His conclusion is one I keep hearing this year: our job is now writing specs and checkers.

Specification-Driven Synthesis of Query Execution Logic for Novel Storage Systems (Puneet Mehrotra, UBC)

The talk introduces FlexoGraph, a graph database built on WiredTiger. As far as I can understand, the synthesis problem here is to produce the query execution logic from a specification, for a storage system nobody has written an engine for yet. This seems to be a challenging test of specification-driven synthesis, since there is no reference implementation to lean on.

Learned Systems Are Dead; Long Live Learned Systems (Todd Lipcon, Google)

On paper, the opportunity for learned systems is real. Databases are full of heuristics (query plans, index optimization, cache eviction), and every one of them is a knob somebody guessed at. But the challenges turn out to be just as real: a low-latency budget that an inference call blows immediately, interpretability, maintenance, and the fact that these systems are hard to simulate. Todd then put up the list of projects that didn't make it: B-tree learned indices (never actually launched despite the NeurIPS'20 paper), ML cache eviction, Vizier-tuned cache eviction, and Kepler query optimization. The RL merge compaction work, the Ziggurat algorithm, was hard to train, complex, and got 2% improvement, so it didn't ship. Todd's diagnosis is the tax of the embedded model. You pay inference latency on every decision, and you pay opacity and risk forever, and that tax is bigger than the win for most database decisions. The takeaway is this: Don't ship the model, ship what the model wrote. Instead of embedding an LLM in the hot path, have it generate the C++ algorithm, read the C++, and ship the C++. You get the search power of the model and you pay none of the inference tax. You also get an artifact a human can review, which handles the interpretability and maintenance problems at the same time. The cache eviction work pivoted this way in 2024.

Session 3: What data stack do our agents need?

Commits, Transactions, Durable Execution: Git-for-everything and the self-driving data stack (Jacopo Tagliabue, Bauplan)

Bauplan is OLAP infrastructure for agents. Jacopo's mental model for agent infrastructure is git: branch, try, throw away, merge. The systems work is getting from files to Iceberg tables. Iceberg brings database-like tables to object storage, with table atomicity and optimistic locking. With those two primitives you can give every agent its own branch, and make them merge transactions.

Execution LogAct: Enabling Agentic Reliability via Shared Logs (Mahesh Balakrishnan, Meta)

Agents corrupt state, crash mid-change, and overwrite each other. You cannot review an agent ahead of time because an agent is a self-writing program: it goes to the LLM and asks for the next lambda to run. So how do you provide guarantees for an execution whose code is being written as it runs? Mahesh's answer is LogAct, a transactional framework for agents built on a shared log. Before execution, the agents vote through an AgentBus that carries intention, votes, and commit. During execution, a WAL handles failure recovery. After execution, the log serves as the audit trail. The insight is this: Do not try to make the agent trustworthy, instead make the record of what it did trustworthy, and build the guarantees there. This also helps with introspecting agentic activity and auditability. When your agents cause an outage at agentic speed, the log is your friend.

The Postgres Trap: Database Incrementalism in the Age of Agents (Ajay Rayasam, Osage University Partners)

I took no notes in this one. I'm listing this for completeness.

AI for Production (Mayank Agarwal, Resolve AI)

Mayank is CTO of Resolve AI, and the talk was about self-driving production. Again, no notes from this presentation. Maybe check the HPTS agenda. Some authors will provide their presentations to be listed there.

Session 4: Architecture Lessons 

Ten Years of Magic Pocket: Lessons From Scaling an On-Prem Exabyte-Scale Blob Store (Facundo Agriel, Dropbox)

Dropbox left S3 ten years ago, and this talk reflects on the reliability lessons learned from managing ten exabytes, trillions of blobs, and billions of deletes every day. A durability model takes three inputs: AFR (annual failure rate), repair rate (how quickly failed fragments get repaired), and redundancy (erasure coding or replication). The model is only as good as the AFR you plug into it, and the lesson Facundo drew is that failures don't care about your durability model. He gave two examples. Drives hate the heat: high temperature affected 30% of one specific drive population. The AFR shot up, but only for that revision, in those conditions. They built a system to drain those drives early and got out from under it. The second example is that SSDs wear out together. Their metadata tier was being used harder than they knew, and because the drives were installed together and wore at the same rate, they were heading toward a correlated failure.

He also talked about the importance of checksums and verifiers, but I found the changing tradeoffs of HDD storage more interesting. Drives are now 40 TB. For a decade, rising density reliably lowered unit cost, but it is ending. Cost is going up, density is still going up, and I/O is flat. Over the period he measured, drive capacity grew 19x while bandwidth grew only 1.5x. It now takes 36 hours to read a single drive end to end. So everyone will have to optimize for I/O rather than capacity. When you buy a drive twice the size at the same bandwidth, each stored byte comes with half the bandwidth to reach it. Buying capacity now means buying progressively less access to it. The market numbers are bad too. Western Digital is essentially sold out. Memory contract forecasts are up 3x. Seagate's 2028 nearline exabytes are already allocated.

How We Built Spanner Omni (Sebastian Kanthak, Google)

Omni is externally-facing Spanner: a downloadable Spanner that runs from a single laptop up to a multi-regional cluster of thousands of VMs. Version 1 shipped September 30, 2026, but don't get too excited. It's not opensource, and it requires buying a license to use after trial period expires.

The engineering problem is the fun part. Spanner sits on Colossus for the filesystem, TrueTime for the clock, and Chubby for leader election and membership. You cannot ship any of those. So for each dependency they had to decide what the smallest replacement is. Colossus makes file movement transparent, so for Omni they built the part of Colossus that Spanner actually depends on. Chubby became Spanner Omni-Chubby, providing leader election and the membership service.

The clock was the question everyone was interested in. Sebastian effectively said Spanner does not need GPS and atomic clocks in every datacenter after all. The commit wait only has to cover the clock uncertainty, and the cost is hidden as long as the wait is comparable to the 1-RTT you were already paying to replicate the commit. So you don't need the uncertainty to be tiny, you only need it to be smaller than 1-RTT. And to get there, a single site doing clock sync is enough: have everyone discipline against one logically centralized reference, with a fallback reference in each region in case the central site becomes unavailable.

Preventing and Mitigating Logical Corruptions in Salesforce Database (Vaibhav Arora & Jameison Martin, Salesforce)

I don't have notes from this presentation either.

Gong Show

These are lightning talks that are five minutes each. I provide some notes with little or no context.

What Postgres would look like if we started today (Manish Jain)

Own your resources; watch your egress. An LSM forest, bitmaps, batched execution. Physical replication rebuilt for an LSM. High availability with read-your-writes, where the WAL is Raft plus hybrid logical clocks. And instant branching with paying for what you change, not for what you touch.

Your database is all you need (Qian Li, DBOS)

DBOS is about providing a durable execution as a library on top of the database you already have: workflows, queues, and events, with DBOS tables living in your existing database. No new system to operate was a big part of the "pitch". Qian sang her presentation with perfect pitch to the tune of "my favorite things" and she was a co-winner of the Gong Show.

From Vitess to Multigres: Lessons learned (Sugu Sougoumarane, Supabase)

The talk was mostly about trying to build consensus guarantees on native Postgres replication following from an informal statement of consensus.

Generative AI; Generative Testing (Kyle Kingsbury, Jepsen)

The talk is delivered at Kyle speed and was very entertaining, making him the co-winner of the Gong Show. Kyle is not a fan of LLMs (to say the least) and does not find them useful for his checking/testing work.

Resurrecting Semantic Concurrency Control (William Schultz, MongoDB) 

Will gave a great presentation arguing that traditional database concurrency control operates at a too low-level, at the key-value storage layer, which discards useful application semantics. By shifting concurrency control to a higher semantic layer using a unified intermediate representation, databases can reason directly about rich operation logic rather than simple reads and writes. He argued preserving this context enables optimizations like operation merging and smarter scheduling, which significantly reduce transaction aborts under high contention.

Open Storage Format for OLTP (Sunkanmi Adewumi, UC Berkeley)

Scalable Image AI via Self-Designing Storage (Utku Sirin, Harvard)

Mind the Bandwidth Gap: Scaling Analytics with CXL Memory (Georgiy Lebedev, EPFL)

A Little Little's Law Keeps the Metastable Failure Away (Kiran Kumar Muniswamy Reddy, MongoDB)

Badly configured queue sizes are a common factor for metastability failures. Kiran proposed using Little's Law to compute the queue size instead of guessing it.

Failure resilience in large-scale LLM training on TPUs (Alexander Shraer, Google)

LLMs are compute. Context is memory. Inference speed is king. (Mehul Shah, Glean)

And remember, all of these were only day 1.

Welcome back to Village News, our curated roundup of MySQL and database news. This issue covers two weeks, with MySQL 26.10 landing at the end of them.

If you want to get these updates, just subscribe to the blog.

Enjoy!

MySQL News

Note: Aggregated MySQL news can be found

A data lake is rarely one file. Once you archive or publish data by period, you get a tree of files — and the trick is querying the whole thing as one table while only reading the parts a query needs. pg_duckdb does both: globs read many files as one, and Hive-style partitioning turns the folder names into columns and lets DuckDB skip files it doesn't need. With read_parquet(), the query itself chooses the file set and reader options, rather than requiring a foreign table for each file. Here are some examples from Azure Database for PostgreSQL.

Setup

Only the Blob secret is needed for this post — everything else reads from the lake:

CREATE EXTENSION IF NOT EXISTS pg_duckdb;
SELECT duckdb.create_azure_secret('<blob connection string>');

A partitioned layout

The convention: a folder per partition, named column=value, each holding one or more Parquet files. Here, four yearly files of sales data:

az://lake/sales_part/
   year=2022/data.parquet
   year=2023/data.parquet
   year=2024/data.parquet
   year=2025/data.parquet

You produce these one file per year. The only new thing here is the folder naming (year=YYYY/). The write from PostgreSQL is covered in the archive post. With the sales table from that post, one year's export looks like this:

SELECT duckdb.raw_query($$
  COPY (
    SELECT * FROM pgduckdb.public.sales
    WHERE sale_date >= DATE '2022-01-01'
      AND sale_date < DATE '2023-01-01'
  ) TO 'az://lake/sales_part/year=2022/data.parquet'
  (FORMAT parquet)
$$);

Note the SELECT before duckdb.raw_query and the pgduckdb.public. catalog prefix: inside raw_query you're in DuckDB's catalog, not PostgreSQL's. Keep this wrapper on the tested Flexible Server: plain PostgreSQL COPY ... TO 'az://...' is rejected by the service's file-COPY guard, even though self-managed pg_duckdb can intercept it. The export also requires permission to execute duckdb.raw_query(). A client-side export and upload is the alternative described in the archive post.

The column=value convention lets the reader interpret the folder names as partition columns, and a wildcard selects the files across those folders.

Read them all with one glob

A * in the path reads every matching file as a single table:

postgres=> SELECT count(*) FROM read_parquet(
 'az://lake/sales_part/year=*/data.parquet'
)
;
  count
---------
 8001032
(1 row)

A specific path reads one partition:

postgres=> SELECT count(*) FROM read_parquet(
 'az://lake/sales_part/year=2024/data.parquet'
)
;
  count
---------
 2003093
(1 row)

And an explicit list reads an exact set — note the ARRAY[...] syntax (a bare [...] is DuckDB dialect that the PostgreSQL parser rejects):

postgres=> SELECT count(*) FROM read_parquet(ARRAY[
 'az://lake/sales_part/year=2024/data.parquet',
 'az://lake/sales_part/year=2025/data.parquet'
           ])
;
  count
---------
 4003149
(1 row)

Want to know which files a glob actually touched? Add the filename column:

postgres=> SELECT DISTINCT r['filename']
           FROM read_parquet(
 'az://lake/sales_part/year=*/data.parquet',
 filename := true
) r
           ORDER BY 1
;
                  filename
---------------------------------------------
 az://lake/sales_part/year=2022/data.parquet
 az://lake/sales_part/year=2023/data.parquet
 az://lake/sales_part/year=2024/data.parquet
 az://lake/sales_part/year=2025/data.parquet
(4 rows)

(pg_duckdb passes DuckDB's optional arguments with PostgreSQL's named-argument syntax, name := value — so filename := true, hive_partitioning := true.)

Turn the folder name into a column

The partition value lives only in the path, not inside the files. Turn on hive_partitioning and year becomes a real column you can group and filter on:

postgres=> SELECT r['year'] AS year, count(*) AS n
           FROM read_parquet(
 'az://lake/sales_part/year=*/data.parquet',
  hive_partitioning := true
) r
           GROUP BY r['year']
           ORDER BY 1
;
 year |    n
------+---------
 2022 | 1998570
 2023 | 1999313
 2024 | 2003093
 2025 | 2000056
(4 rows)

Because the year is encoded in the layout, you can compute per-year totals even without the files carrying a date column at all.

Partition values are not necessarily text. On this server (PostgreSQL 18.6, pg_duckdb 1.1.0), SELECT pg_typeof(r['year']) over this reader returns bigint: DuckDB infers the type from the folder values. Both r['year'] = 2024 and the quoted literal below work and prune to one file.

When file schemas evolve

If a later year's file adds a column, use union_by_name := true to align columns by name rather than position:

SELECT count(*)
FROM read_parquet(
  'az://lake/sales_part/year=*/data.parquet',
  hive_partitioning := true,
  union_by_name := true
);

Columns absent from a file become NULL for that file's rows. This handles added or missing columns, not arbitrary incompatible type changes.

The payoff: partition pruning, visible in the plan

This is why you partition. Filter on the partition column and DuckDB reads only the matching files. EXPLAIN shows it — and here is the full verbatim plan:

postgres=> EXPLAIN (ANALYZE, COSTS OFF)
           SELECT count(*)
           FROM read_parquet('az://lake/sales_part/year=*/data.parquet',
                             hive_partitioning := true) r
           WHERE r['year'] = '2024'
;
                   QUERY PLAN
-------------------------------------------------
 Custom Scan (DuckDBScan) (actual time=0.001..0.002 rows=0.00 loops=1)
   DuckDB Execution Plan:

 ┌─────────────────────────────────────┐
 │┌───────────────────────────────────┐│
 ││    Query Profiling Information    ││
 │└───────────────────────────────────┘│
 └─────────────────────────────────────┘
 ┌────────────────────────────────────────────────┐
 │┌──────────────────────────────────────────────┐│
 ││              Total Time: 0.78s               ││
 │└──────────────────────────────────────────────┘│
 └────────────────────────────────────────────────┘
 ┌───────────────────────────┐
 │           QUERY           │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │      EXPLAIN_ANALYZE      │
 │    ────────────────────   │
 │           0 rows          │
 │          (0.00s)          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │    UNGROUPED_AGGREGATE    │
 │    ────────────────────   │
 │        Aggregates:        │
 │        count_star()       │
 │                           │
 │           1 row           │
 │          (0.00s)          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │         TABLE_SCAN        │
 │    ────────────────────   │
 │         Function:         │
 │        READ_PARQUET       │
 │                           │
 │       File Filters:       │
 │       (year = 2024)       │
 │                           │
 │    Scanning Files: 1/4    │
 │    Total Files Read: 1    │
 │                           │
 │       2,003,093 rows      │
 │          (0.11s)          │
 └───────────────────────────┘
 Planning:
   Buffers: shared hit=1
 Planning Time: 329.131 ms
 Execution Time: 330.574 ms

The lines that matter are Scanning Files: 1/4 and Total Files Read: 1. The glob matched four files, but the year = 2024 filter let DuckDB skip scanning data from three of them — it decided that from the folder names alone (File Filters: (year = 2024)). This does not mean zero listing or metadata requests: resolving the glob and binding the reader can still require them. The 2,003,093 rows at the scan is the one file's actual row count; a count(*) can use Parquet metadata rather than decode every row. With hundreds of monthly partitions, a filter selecting one month can restrict the data scan to that month's files, whether there is one file or several. That's why this layout pays off, and pg_duckdb gives it to you over Azure Blob with the filter sitting in ordinary SQL.

Does abfss:// work on Azure Database for PostgreSQL?

DuckDB's Azure extension supports both Blob (az://) and ADLS Gen2 (abfss://) paths, but the managed server's allow-list comes first. I retested on this Flexible Server with PostgreSQL 18.6 and pg_duckdb 1.1.0:

postgres=> SHOW duckdb.enable_external_access;
 duckdb.enable_external_access
-------------------------------
 off
(1 row)

postgres=> SHOW duckdb.allowed_directories;
 duckdb.allowed_directories
----------------------------
 az://
(1 row)

postgres=> SELECT count(*)
FROM read_parquet('abfss://lake/sales_part/year=2024/data.parquet');
ERROR:  (PGDuckDB/CreatePlan) Prepared query returned an error: Permission Error: Cannot access file "abfss://lake/sales_part/year=2024/data.parquet" - file system operations are disabled by configuration

LINE 1: SELECT count(*) AS count FROM system.main.read_parquet('abfss://lake/sales_part/year=2024...
                                      ^
postgres=>

Both a single-file read and a glob fail, including with the fully qualified abfss://<account>.dfs.core.windows.net/lake/... form. The same reads through az:// succeed. These settings require superuser privileges to change, which this login does not have.

So the two schemes are not interchangeable on this server. The rejection happens in DuckDB's filesystem permission check, before we can test the ADLS endpoint or compare listing performance. Use az:// here; there is no measured basis in this test for recommending abfss:// as faster.

Takeaways

• A glob (year=*/data.parquet) reads a whole partitioned lake as one table. A path reads one partition. ARRAY[...] reads an explicit set.
• hive_partitioning := true turns column=value folder names into queryable columns — group and filter on them like any column.
• Filtering the partition column prunes files before reading: the plan shows Scanning Files: 1/4. This is the main reason partitioning pays off.
• Pass DuckDB's reader options with PostgreSQL named-arg syntax (:=); use ARRAY[...] not [...] for file lists.
• Use union_by_name := true when files add or omit columns across partitions.
• On the tested Flexible Server, az:// is allowed but abfss:// is blocked by configuration, for both single files and globs.

Everything so far has read Parquet. Blob also collects the messier arrivals — vendor CSVs, newline-delimited JSON event streams. Next I'll point read_csv and read_json at those and load them into Postgres with a single statement.

Monday, 9:00. The app team writes: “MongoDB was slow on Thursday afternoon. What happened?” No alert fired, nobody looked at the time, and now it is your problem. The answer is usually in the FTDC. Every mongod writes a sample of its own metrics once per second into diagnostic.data, and MongoDB calls this Full Time … Continued

The post “MongoDB was slow on Thursday”: answering the app team with mongo-health appeared first on Percona.

We’ve recently been dealing with an interesting customer case where a PXC version upgrade from 5.7 to 8.0 seemed to trigger a completely unexpected problem. Even though the workload and MySQL settings did not change, all PXC members began exhibiting sparse but regular bursts of massive disk writes. Yet the MySQL side workload statistics that … Continued

The post When a Small Change Writes Too Much – How A Linux Kernel Improvement Can Strike You in PXC/Galera Workloads appeared first on Percona.

At Percona Live Europe 2026, Dmitry Lenev and I presented JavaScript in My Database? Over My Dead Query, the story of component_js_lang, the V8-based JavaScript engine for Percona Server for MySQL 8.4. That experience told us few things: The presentation was seen as “for developers” What most DBA or DevOps heard was “JavaScript in the … Continued

The post JavaScript in MySQL: Extending SQL, Not Moving the Application appeared first on Percona.

After the archive post, I got a good question: what happens when an old row must change after its year has been written to Parquet? Rewriting a large immutable file for every correction or deletion would defeat much of the purpose of the archive.

My first answer is to archive a fact only once the business considers it immutable. Historical sales shouldn’t change silently after reports are published. Personal data, however, follows a different lifecycle: a customer's name, email, phone number, or address may need to be corrected or erased even if the financial fact must remain unchanged.

That suggests a useful split:

The immutable sales facts go to Parquet. The identifying data remains in PostgreSQL, within the operational system's existing security and governance boundary: database roles and row-level rules, audited changes, constraints, transactional corrections and deletions, retention procedures, and controlled backup and restore processes. pg_duckdb joins both sources only when a query requires them.

The point is not that object storage cannot be secured. It is to avoid creating immutable analytical files from personal data, each of which would require its own access controls, audit trail, retention policy, backup handling, and erasure procedure. Under GDPR's data-minimization principle, keeping direct identifiers out of the archive also reduces the amount of personal data copied into the analytical tier.

This is where pg_duckdb adds more than just convenient file syntax. The PostgreSQL table and the Parquet file become two scan inputs for a single DuckDB relational plan. DuckDB chooses the join order and build side, pushes filters toward both sources, performs the hash join, and runs the aggregation in a single vectorized pipeline.

The query uses a LEFT JOIN with the archived facts on the preserved side, so deleting the matching identity leaves the sale in the result with NULL values in the personal columns.

Lab setup

The sales data is the same deterministic 10-million-row dataset used in the previous posts. There is no random(), so the counts below are reproducible.

SET duckdb.force_execution = false;

DROP TABLE IF EXISTS sales;
CREATE TABLE sales AS
SELECT g AS sale_id,
       (DATE '2021-01-01' + ((g * 7) % 1826))                        AS sale_date,
       1 + (g % 50000)                                               AS cust_id,
       1 + (g % 2000)                                                AS prod_id,
       1 + (g % 5)                                                   AS channel_id,
       (ARRAY['US','CH','FR','DE','UK','JP','BR','IN'])[1 + (g % 8)] AS country,
       (ARRAY['Consumer','SMB','Enterprise'])[1 + (g % 3)]           AS segment,
       1 + (g % 10)                                                  AS qty,
       ((5 + (g % 500))::numeric(10,2))                              AS amount
FROM generate_series(1, 10000000) g;
ANALYZE sales;

CREATE EXTENSION IF NOT EXISTS pg_duckdb;
SELECT duckdb.create_azure_secret('<blob connection string>');

Build the source table on the PostgreSQL executor (force_execution = false). If DuckDB is forced on, it warns about +(DATE, BIGINT) in this generate_series DDL and falls back to PostgreSQL anyway. You don't need to force DuckDB for this example.

Now create the table that owns the erasable identity. The values are synthetic:

DROP TABLE IF EXISTS customer_identity;
CREATE TABLE customer_identity (
  customer_ref int PRIMARY KEY,
  full_name    text NOT NULL,
  email        text NOT NULL,
  phone        text NOT NULL,
  address      text NOT NULL
);

INSERT INTO customer_identity
SELECT g,
       format('Customer %s', lpad(g::text, 5, '0')),
       format('customer%s@example.test', lpad(g::text, 5, '0')),
       format('+41 22 555 %s', lpad(g::text, 4, '0')),
       format('%s Example Street, Test City', g)
FROM generate_series(1, 50000) g;

CREATE UNIQUE INDEX customer_identity_email_idx
  ON customer_identity (email);

ANALYZE customer_identity;

For readability, this lab reuses the sequential cust_id as customer_ref. In a real design, use a stable, opaque token with no business meaning, and keep the only mapping to an identity in the protected operational database.

Write a minimized fact file

When exporting the 2021 sales data, select only the immutable fact columns. Do not copy the name, email, phone, address, or other identifying free text into the file:

postgres=> SELECT duckdb.raw_query($$
             COPY (
               SELECT sale_id,
                      sale_date,
                      cust_id AS customer_ref,
                      prod_id,
                      channel_id,
                      qty,
                      amount
               FROM pgduckdb.public.sales
               WHERE sale_date < DATE '2022-01-01'
             ) TO 'az://lake/archive/sales_2021_minimized.parquet'
               (FORMAT parquet)
           $$)
;
 raw_query
-----------

(1 row)

As in the archive post, raw_query sends the COPY command to the embedded DuckDB engine, and pgduckdb.public.sales is the DuckDB catalog name for the PostgreSQL table.

Check both sides:

postgres=> SELECT count(*) FROM customer_identity
;
 count
-------
 50000
(1 row)

postgres=> SELECT count(*) AS n_sales,
                  count(DISTINCT (s['customer_ref'])::int) AS n_customers
           FROM read_parquet(
             'az://lake/archive/sales_2021_minimized.parquet'
           ) s
;
 n_sales | n_customers
---------+-------------
 1998938 | 50000
(1 row)

The file has 1,998,938 immutable sales and only an opaque bridge to the 50,000 identity rows kept in PostgreSQL.

The join before erasure

Customer 42 has 39 sales in the archived year. The query resolves the bridge to the live identity using a LEFT OUTER JOIN:

postgres=> SELECT
             COALESCE(c.full_name, '[erased customer]') AS customer_name,
             COALESCE(c.email, '[removed]') AS email,
             count(*) AS n_sales,
             sum((s['amount'])::numeric(10,2)) AS revenue
           FROM read_parquet(
             'az://lake/archive/sales_2021_minimized.parquet'
           ) s
           LEFT OUTER JOIN customer_identity c
             ON c.customer_ref = (s['customer_ref'])::int
           WHERE (s['customer_ref'])::int = 42
           GROUP BY c.full_name, c.email
;
 customer_name  | email                      | n_sales | revenue
----------------+----------------------------+---------+---------
 Customer 00042 | customer00042@example.test | 39      | 1794.00
(1 row)

No SET duckdb.force_execution = true is required. DuckDB automatically executes any statement that references read_parquet, including the scan of the PostgreSQL table and the join.

Delete the identity, not the history

Now process the erasure in the operational database:

postgres=> DELETE FROM customer_identity WHERE customer_ref = 42
;
DELETE 1

Run exactly the same join again:

postgres=> SELECT
             COALESCE(c.full_name, '[erased customer]') AS customer_name,
             COALESCE(c.email, '[removed]') AS email,
             count(*) AS n_sales,
             sum((s['amount'])::numeric(10,2)) AS revenue
           FROM read_parquet(
             'az://lake/archive/sales_2021_minimized.parquet'
           ) s
           LEFT OUTER JOIN customer_identity c
             ON c.customer_ref = (s['customer_ref'])::int
           WHERE (s['customer_ref'])::int = 42
           GROUP BY c.full_name, c.email
;
 customer_name     | email     | n_sales | revenue
-------------------+-----------+---------+---------
 [erased customer] | [removed] | 39      | 1794.00
(1 row)

The 39 sales and their 1,794.00 in revenue remain. The name and email are gone. I used COALESCE to show a value for nulls, but NULL is semantically correct: the customer exists, but their personal data is unknown.

After the deletion, all 1,998,938 archived facts are still counted:

postgres=> SELECT
             CASE WHEN c.customer_ref IS NULL THEN 'erased'
                  ELSE 'identified'
             END AS identity_status,
             count(*) AS n_sales,
             sum((s['amount'])::numeric(10,2)) AS revenue
           FROM read_parquet(
             'az://lake/archive/sales_2021_minimized.parquet'
           ) s
           LEFT JOIN customer_identity c
             ON c.customer_ref = (s['customer_ref'])::int
           GROUP BY CASE WHEN c.customer_ref IS NULL THEN 'erased'
                         ELSE 'identified'
                    END
           ORDER BY identity_status
;
 identity_status | n_sales | revenue
-----------------+---------+--------------
 erased          | 39      | 1794.00
 identified      | 1998899 | 508736609.00
(2 rows)

It's always important to review the execution plan because SQL code is an abstraction that doesn't show how it's executed or how performant and scalable it is.

The full plan: which side is hashed?

Here is the full EXPLAIN (ANALYZE, COSTS OFF) from Azure Database for PostgreSQL:

postgres=> EXPLAIN (ANALYZE, COSTS OFF)
           SELECT
             CASE WHEN c.customer_ref IS NULL THEN 'erased'
                  ELSE 'identified'
             END AS identity_status,
             count(*) AS n_sales,
             sum((s['amount'])::numeric(10,2)) AS revenue
           FROM read_parquet(
             'az://lake/archive/sales_2021_minimized.parquet'
           ) s
           LEFT OUTER JOIN customer_identity c
             ON c.customer_ref = (s['customer_ref'])::int
           GROUP BY CASE WHEN c.customer_ref IS NULL THEN 'erased'
                         ELSE 'identified'
                    END
           ORDER BY identity_status
;
                   QUERY PLAN
-------------------------------------------------
Custom Scan (DuckDBScan) (actual time=0.001..0.003 rows=0.00 loops=1)
  DuckDB Execution Plan:

┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││    Query Profiling Information    ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
EXPLAIN ANALYZE  SELECT CASE WHEN (c.customer_ref IS NULL) THEN 'erased'::text ELSE 'identified'::text END AS identity_status, count(*) AS n_sales, sum((s.amount)::numeric(10,2)) AS revenue FROM (system.main.read_parquet('az://lake/archive/sales_2021_minimized.parquet'::text) s LEFT JOIN pgduckdb.public.customer_identity c ON ((c.customer_ref = (s.customer_ref)::integer))) GROUP BY CASE WHEN (c.customer_ref IS NULL) THEN 'erased'::text ELSE 'identified'::text END ORDER BY CASE WHEN (c.customer_ref IS NULL) THEN 'erased'::text ELSE 'identified'::text END
┌────────────────────────────────────────────────┐
│┌──────────────────────────────────────────────┐│
││              Total Time: 0.515s              ││
│└──────────────────────────────────────────────┘│
└────────────────────────────────────────────────┘
┌───────────────────────────┐
│           QUERY           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│      EXPLAIN_ANALYZE      │
│    ────────────────────   │
│           0 rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│__internal_decompress_strin│
│           g(#0)           │
│             #1            │
│             #2            │
│                           │
│           2 rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│          ORDER_BY         │
│    ────────────────────   │
│       CASE  WHEN ((c      │
│  .customer_ref IS NULL))  │
│   THEN (CAST('erased' AS  │
│    VARCHAR)) ELSE CAST(   │
│  'identified' AS VARCHAR) │
│           END ASC         │
│                           │
│           2 rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│__internal_compress_string_│
│        uhugeint(#0)       │
│             #1            │
│             #2            │
│                           │
│           2 rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│__internal_decompress_strin│
│           g(#0)           │
│             #1            │
│             #2            │
│                           │
│           2 rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│       HASH_GROUP_BY       │
│    ────────────────────   │
│         Groups: #0        │
│                           │
│        Aggregates:        │
│        count_star()       │
│          sum(#1)          │
│                           │
│           2 rows          │
│          (0.02s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│__internal_compress_string_│
│   uhugeint(CASE  WHEN (   │
│  (customer_ref IS NULL))  │
│    THEN ('erased') ELSE   │
│     'identified' END)     │
│           amount          │
│                           │
│       1,998,938 rows      │
│          (0.01s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         HASH_JOIN         │
│    ────────────────────   │
│      Join Type: LEFT      │
│                           │
│        Conditions:        ├──────────────┐
│customer_ref = customer_ref│              │
│                           │              │
│       1,998,938 rows      │              │
│          (0.08s)          │              │
└─────────────┬─────────────┘              │
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│         TABLE_SCAN        ││         TABLE_SCAN        │
│    ────────────────────   ││    ────────────────────   │
│         Function:         ││           Table:          │
│        READ_PARQUET       ││     customer_identity     │
│                           ││                           │
│        Projections:       ││        Projections:       │
│        customer_ref       ││        customer_ref       │
│           amount          ││                           │
│                           ││                           │
│    Total Files Read: 1    ││                           │
│                           ││                           │
│       1,998,938 rows      ││        49,999 rows        │
│          (0.11s)          ││          (0.01s)          │
└───────────────────────────┘└───────────────────────────┘


Planning:
  Buffers: shared hit=1
Planning Time: 223.697 ms
Execution Time: 223.914 ms
(126 rows)

One top-level engine over heterogeneous sources

The outer PostgreSQL plan is a single Custom Scan (DuckDBScan). Inside it, the DuckDB plan includes both leaves—the Blob READ_PARQUET scan and the PostgreSQL customer_identity scan—and the HASH_JOIN and HASH_GROUP_BY above them. DuckDB, not PostgreSQL, owns the cross-source join and aggregation.

That is different from the read-only foreign-data-wrapper solution. With a conventional FDW, PostgreSQL remains the top-level relational engine: the foreign side returns rows or a pushed-down fragment, and PostgreSQL typically joins the result with local tables. Here, DuckDB sees both sides together and can optimize the pipeline across the storage boundary, including the runtime join filter. Everything between the two scans and the result is one vectorized pipeline: the Parquet scan streams ~2M rows, probes the hash table, and HASH_GROUP_BY reduces them to 2 rows without materializing the join output.

DuckDB, a single engine, executes the query with all its analytics optimizations, while PostgreSQL continues to read its own storage. The PGDUCKDB_POSTGRES_SCAN leaf delegates to SQL, where the PostgreSQL optimizer chooses the heap or index access path. PostgreSQL handles only this scan leaf. DuckDB controls the join order, hash table, filtering between inputs, grouping, and final result.

Read a DuckDB plan bottom-up. A physical HASH_JOIN builds its in-memory hash table from the right child, then streams the left child through it:

• The right child is customer_identity: 49,999 PostgreSQL rows, with only customer_ref projected. This is the side that is hashed.
• The left child is READ_PARQUET: 1,998,938 Blob rows, with only customer_ref and amount projected. This is the probe side.
• The join emits all 1,998,938 facts, then HASH_GROUP_BY reduces them to two rows.

Hashing 49,999 integers once is the natural shape. A nested loop would prevent DuckDB from performing a B-tree lookup in Parquet. It would replace one small hash build with repeated file probes. Even when the customer filter below leaves one row, DuckDB keeps a hash join because hashing one key is negligible.

What statistics does DuckDB see?

An EXPLAIN without ANALYZE (the default, or explicitly EXPLAIN (ANALYZE OFF, COSTS ON)) displays the estimated cardinalities (DuckDB prefixes them with ~):

 Custom Scan (DuckDBScan)  (cost=0.00..0.00 rows=0 width=0)
   DuckDB Execution Plan:

 ┌───────────────────────────┐
 │         PROJECTION        │
 │    ────────────────────   │
 │__internal_decompress_strin│
 │           g(#0)           │
 │             #1            │
 │             #2            │
 │                           │
 │          ~0 rows          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │          ORDER_BY         │
 │    ────────────────────   │
 │       CASE  WHEN ((c      │
 │  .customer_ref IS NULL))  │
 │   THEN (CAST('erased' AS  │
 │    VARCHAR)) ELSE CAST(   │
 │  'identified' AS VARCHAR) │
 │           END ASC         │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │         PROJECTION        │
 │    ────────────────────   │
 │__internal_compress_string_│
 │        uhugeint(#0)       │
 │             #1            │
 │             #2            │
 │                           │
 │       ~999,469 rows       │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │         PROJECTION        │
 │    ────────────────────   │
 │__internal_decompress_strin│
 │           g(#0)           │
 │             #1            │
 │             #2            │
 │                           │
 │       ~999,469 rows       │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │       HASH_GROUP_BY       │
 │    ────────────────────   │
 │         Groups: #0        │
 │                           │
 │        Aggregates:        │
 │        count_star()       │
 │          sum(#
                                    
                                    
                                    

                                        by Franck Pachot