This is the second half of my HPTS 2026 notes. Day 1 was mostly about AI and the importance of specifications. Day 2 and day 3 cover hardware, object storage, failures, consensus, and lakehouses.

Session 5: Hardware will save us all (chair: Natassa Ailamaki)

The Fastest Way to be the Fastest (Bobbi Yogatama, NVIDIA)

GPUs and databases have a love-hate relationship, and Bobbi listed the three reasons: limited GPU memory capacity, the interconnect bottleneck, and engineering complexity. The first two are being fixed by the hardware roadmap. HBM is getting larger and faster. Interconnects and network fabrics are getting faster. But engineering complexity requires some work. No one is eager to rewrite their database to get it work with a GPU.

Bobbi introduced Sirius, a GPU-native composable analytics engine. It offers a drop-in accelerator for your existing database, with GPU-native execution underneath. MICE is the design principle: modular, interoperable, composable, extensible. Bobbi showed how Sirius plugs into DuckDB with no modification to DuckDB.

The other design lesson from Bobby is to spend compute to save memory. On a GPU, compute is the abundant resource and memory capacity and bandwidth are the scarce ones, so the classic tradeoff needs to flip here. It is best to compress the data, compress everywhere, and operate on the compressed data. 

The Next Watt Saved Won't Come from a CPU or a GPU (Devashish Purandare, Microsoft)

Datacenter power consumption is the bottleneck in building datacenters right now. Where is the power consumed? Tax cycles are 60-80% of all execution. There is datacenter tax: compression, crypto, data movement, memory allocation, protocol and RPC handling. Then there is also system tax: error handling, checksums, filesystems, kernel, STL, networking, multithreading. Unfortunately, no single category dominates, so there is no single high value thing to fix.

So how do we fix this? Dev titled his next section: how I learned to stop worrying about Amdahl's law and love the accelerator. Again, unfortunately, both on-chip accelerators and off-chip accelerators have big drawbacks. On-chip you are stuck with whatever the CPU vendor gave you. Off-chip you pay to get data there and back. Dev's answer to this is yes. Cluster smaller accelerators together. That brings us to the fungible DPU, from the company Fungible, which Microsoft acquired and which is now the Azure Boost DPU. Six cores times four threads, and every task gets its own dedicated accelerator. If the tax is spread across a dozen categories, you need a dozen accelerators, not one big one.

Hardware alone won't save us, but hardware/software codesign can. The software side is a data path programming model with work-unit based development. Offload the repetitive, avoid the coordination, optimize the rest.

Using DPU-based SmartNICs for Database Management (Phil Bernstein, Microsoft Research)

Phil also covered the same topic, but more broadly. SmartNICs offload tasks the CPU would normally do. There are three kinds: ASIC-based, FPGA-based, and SoC-based, which are the DPUs. A DPU has two halves with opposite characteristics: a multicore CPU (low latency, lower cost, but weaker throughput) and an ASIC for data path tasks such as DMA, compression, encryption, regex (low cost, high throughput, but high latency).

Everyone is shipping them: Fungible, AMD Pensando, NVIDIA BlueField, AWS Nitro. They are deployed in volume in hyperscaler datacenters, maybe millions of units. The natural database use is the pipeline moving data from storage to compute: decrypt, decompress, regex. All data path, all repetitive, all exactly what the ASIC half is for.

Challenge 1: how do you program a DPU? Mostly they run Linux, so you program it like a computer. Their work on DPU-optimized disaggregated storage, DDS at VLDB'24, uses a page-oriented interface: get page from the server, get page at LSN.

Challenge 2: how do we achieve portability across DPUs? This is their VLDB'26 paper. 

What Changes When Memory is Disaggregated? Building a Database to Find Out (Chiradip Mandal, Space-RF.ORG)

I have no notes here.

Session 6: Query Processing (chair: Peter Boncz)

CoddSpeed! Hardware Accelerated Analytics in Microsoft Fabric (Matteo Interlandi, Microsoft)

Privacy Policy of Cloud Data Services is Harmful (Surajit Chaudhuri, Microsoft)

No notes on these two as well.

No more promises: ROI-Guided Resource-Centric System Design (Natassa Ailamaki, EPFL)

Natassa opened with "procrastination is great." Her proposal is to turn everything into a dynamic decision you make at the last possible second, when you have the most data. In her framing the database sits in between the hardware and the applications as the resource broker. Nothing else in the stack has this much visibility into both sides. The application knows what it wants but not what the machine looks like. The OS knows the machine but not the query. Only the database knows both, and so only the database can make the call.

But then there are challenges. Processors compute faster than memory feeds them, and the situation has not improved. We are memory bandwidth bound, and data movement dominates memory efficiency. Memory accesses are energy hungry, and data movement dominates the energy budget. Bandwidth is king: the CXL/DRAM bandwidth gap predicts the slowdown.

This fits well with Dev's talk. He measured 60-80% of cycles going to tax, much of it data movement. Natassa is also finding the same thing: data movement dominates both the time budget and the energy budget.

Her proposal is ROI-guided runtime interleaving. The scan bottleneck cannot be fixed, and it is impossible to make the data movement decision a priori, because the fastest strategy (pushdown, staging, and the rest) depends on the query and the hardware, and too many factors interact to model it well. Here is the key move to address this: the query plan stays fixed, the data movement strategy changes. Don't try to solve this in the optimizer, where you have to guess. Decide at runtime, based on how much the downstream operators actually consume. Procrastinate until you know. A system is only as efficient as its least adaptive component. The static piece in your pipeline sets your ceiling, no matter how clever everything around it is.

Session 7: Object Storage and Open Formats (chair: Rebecca Taft)

Beyond Disaggregated Storage: The Case for Open Storage in OLTP Systems (Sergey Melnik, Databricks)

Sergey's claim: OLTP needs a diff API. The question an external consumer actually asks is "what changed, and how, since this LSN," and there is no good way to ask it. CDC is the current answer, but it is in the wrong shape. His proposal is Differential Change Discovery, which is a pun on CDC. DCD provides net changes with replacement semantics, partitionable and parallelizable, with a system-defined unit of change that could be a row, a block, or a partition. If a row was updated forty times, CDC hands you forty events and you reconstruct the answer, but in contrast DCD hands you the final net change as answer. Sergey argued that a diff API plus open table formats enables a rich third-party ecosystem.

Ex Falso SQL Quodlibet: Excluding the Middle (Chris Douglas, UC Berkeley)

Today's open table formats struggle with concurrent writers because repeatedly rewriting metadata trees creates massive performance bottlenecks, prompting vendors to build centralized commit catalogs that quietly regress lakehouses back into proprietary warehouses where single engines own tables. To fix this, Chris proposes replacing heavy metadata tree rewrites with an append-only shared log where engines simply post compact "intention records". Crucially, catalog state, invariants, and governance policies are stored directly in this same log as declarative logic rules, allowing each engine to deterministically self-police and reject invalid transactions locally. This design achieves shared ownership and asymmetric provisioning, enabling heterogeneous engines to concurrently read and write structured data without needing an active centralized coordinator.   

Operational Benefits of Databases Built on Object Storage (Colin Breck, Tesla)

Colin works on software for the energy products. Megapack battery storage, which matters for renewables and now for datacenters. Solar. Tesla charging, which is 7 MW of solar generation. Routing a car to a charger is a dynamic decision against traffic and outages. All of it is time series: history, recent, series.

But how fast is fast enough? For them, real-time means one second response, and they cannot tolerate ingestion delay. Colin listed six concrete operational challenges with their clustered time series database. Scaling requires weeks of repartitioning. Replication can fail catastrophically. Upgrades are in place. Workload isolation requires running multiple databases. Deletes require expensive recompaction. Data migration requires custom tools.

To address these challenges, they built a time series database on S3. It is log structured. L0 and L1 are Parquet. DuckDB is used only for querying, and it queries the Parquet on S3 directly, without downloading datasets, using projection and filter pushdown. It reads the Parquet footer for schema and min/max statistics and then fetches only the byte ranges it needs. There is no traditional external index anywhere, because Parquet carries micro-indexes in the file format and DuckDB evaluates them at query time. No B-trees to maintain, no secondary index to keep in sync. This helps resolve the six challenges above. S3 handles redundancy and durability, and the rest of the operations were artifacts of owning the cluster. They also run on on-prem object storage, using the S3 API for portability. Nice finishing touch.

S3 is the Future, S3 is the Past (Viktor Leis, TU Munich)

Victor said Snowflake pioneered S3 for analytics. Redshift and Microsoft Fabric followed. Parquet plus Iceberg, datalakes and lakehouses, made S3 the standard for everybody. But what is S3? (What are frogs?) Viktor argues S3 is essentially a million magnetic disks. Files hash-partitioned across them, with erasure coding for durability. It provides 10 ms latency per request, 80 MB/s throughput per request. You get high aggregate throughput only by issuing many large parallel requests from one node; in their experiments, 100 Gbps. Viktor claims that every architectural rule we derived from S3 is actually a rule derived from spinning disks. HDD seek times, therefore caches are necessary. Wide erasure coding, therefore no in-place writes, therefore log structuring is necessary. And, and, therefore large objects and caching. These are disk facts dictating cloud design.

Now, Viktor shows us the promised land. SSDs are getting ridiculously fast. Datacenter networks are amazing (Steinert et al. CIDR'26). So what would an SSD-based S3 change? No need for tiering, no need for caching, no need for large sequential reads, no need to batch writes. Most of the complexity in the modern cloud data stack is there to hide HDD latency, and it would simply be deleted. But there is a catch. SSD versus hard disk cost ration is 5x, up from 3x last year. Though keep in mind that the long-term trend favors flash, since NAND capacity scales faster than HDDs. So SSD-based S3 is the future... (Viktor dismisses S3 Express: "it is so bad, I didn't put a slide about it. With 3 ms latency, it is not close to what an SSD can do.")

In Q&A, Viktor kept strengthening his case for SSD based design. The OLTP consequence of HDD based S3 should alarm people: Iceberg on S3 achieves about 1 transaction per second (Ginter and Leis, VLDB'26). So every system that wants OLTP on object storage ends up implementing its own flash-based storage or caching service. Everybody is building the same missing layer, separately. However, an SSD local cache is dumb, you destroy the drives like crazy! The right answer is of course an SSD-based object store, not an SSD cache in front of an HDD-based one.

Session 8: Failures Considered Harmful (chair: Atul Adya)

Real Life is Uncertain. Consensus Should Be Too! (Reggie Frank, UC Berkeley)

I wrote about this work earlier here: https://muratbuffalo.blogspot.com/2025/07/real-life-is-uncertain-consensus-should.html

Advancing Storage Efficiency in Replicated Databases (Scott Andreas, Apple)

Replication directly multiplies the cost of your system. This is very problematic for Apache Cassandra deployments with dozens of petabytes. Scott's goal is to aggressively cut this storage cost, while providing the same level of durability.

Let's start with the metric: physical storage provisioned divided by logical bytes stored. Where does that extra storage go? It goes to the replication factor, compression, LSM overhead, commitlog/WAL, and scratch space or headroom. A default deployment is 3x replication, 50% compression, and 30% reserved for maintenance and scratch. That's a lot, but through the rest of the talk, Scott kept cutting that cost.

Witness replicas. Journal-only replicas that participate in the quorum without storing the full dataset. The idea goes back to a 1986 Paris paper, and it gives you availability and durability without the storage. Implementing it in Cassandra required changing repair: conventional repair scans all data, builds Merkle trees, and exchanges deltas, which a witness cannot do because it has no data. The replacement is mutation tracking and reconciliation, analogous to read repair, with background reconciliation feeding hints. That takes the ratio from 6.2x to 4.2x.

LSM efficiency, by reducing the buffers. Tablets: splittable, reassignable ranges, which improve elasticity. The efficiency win comes from making moves and compactions cheaper, which shrinks the headroom you must reserve. 4.2x to 3.6x. Compression advances: 3.6x to 2.4x. Satellite datacenter: two durable replicas across two regions plus a satellite. 1.6x!

Retry is an open problem (Peter Alvaro, UC Santa Cruz and AWS)

Retry causes a storm in a jar: metastability! The industry view is "thank you for the papers, but these retry storms keep happening." The postmortem party goes something like this: the timeout was too aggressive (queue_bound times service time), the queue bound was too high, the token bucket was misconfigured, we should just over-provision, or eliminate the lines so there is no head-of-line blocking (LIFO versus FIFO). Each explanation may have some local truth to it, but none of them resolves things. 

Simulation helps by helping you measure: success rate, latency, queue depth, offered load, effective load, amplification. But how do you calibrate the simulation? That is hard.

Peter argued that we need to prove things about actual code, and pointed at Rupak Majumdar's Descartes, which runs actual code at high speed. He is also chasing other directions : a single-program view (Hydro and Cress) which enables global reasoning about configuration.

MESSI: Reasoning about Performance and Metastability (Aleksey Charapko, UNH)

Queueing theory is the standard tool for perfromance modeling. You can model MultiPaxos and Raft this way. But you get a spherical cow in a vacuum. Real systems are dynamic, have cross-component interactions, and run concrete algorithms with concrete details that matter.

So Aleksey built MESSI, a metastability simulator. Nodes, processors, and a qitem as the basic unit of work, with execution control to express the dynamics the closed-form models cannot. He showed a case study from applying MESSI to MongoDB 9 Atlas intelligent workload management to avoid the convoy effect with two queues separating short and long tasks.

Tuesday Night Panel - AI and Systems

The panel consisted of Anastasia Ailamaki (EPFL), Qian Li (DBOS), Carsten Binnig (TU Darmstadt), Ippokratis Pandis (Databricks), and Sailesh Krishnamurthy (Google). The panel stirred a controversy around code review. A lot of people lined up to profess their views and it was getting very heated. I choose to sit down and listen. This is a very complex topic. I had came to HPTS to see how my peers have processed the AI shock. But the panel discussions showed that people have not processed the AI trauma yet. We are all going through it together.

Session 9: Distributed Systems Are Hard! (chair: Justin Levandoski)

Please Implement A Quorum Database (Alex Miller, Oracle)

This was a fun talk  with a running evaluation question: can it CashDB? Meaning, can this algorithm actually run the workload a real database has? The talk considered a quorum database, like Cassandra: leaderless replication, no downtime from leader failure, majority quorums. The wishlist for anything that wants to replace what we have are survive contention, consistent local snapshot, and incremental catch up. The last two are the ones papers never mention but operators cannot live without.

Then began the tour of candidates. 20 protocols in 20 minutes. Megastore, single-synod Paxos. EPaxos, consensus over graphs: can it CashDB? No.

QuePaxa, including QuePaxa over logs. 7 RTT is painful, and the open question is how to pipeline it. Cloudflare is trying to push this into production, which makes it more interesting than a paper.

RIOT, consensus over graphs: can it CashDB? Probably. Slow path latency equal to Raft, but 10x the network bandwidth.

CEP-15, fast general purpose transactions, with efficient replication, Tempo, and Accord: can it CashDB? Yes.

TAPIR: SurrealDB is putting it into production.

Exploiting commutativity in consensus, Ousterhout's CURP. Jetpack and nil externality. And a surprise appearance from the escrow transactional method from the 1980s, now as high throughput escrow for replicated databases. It can CashDB.

Among the audience, a very small group of distributed systems nerds knew all these protocols. A lot of preparation must have gone into this bespoke talk. It was a lot of fun. I can watch this talk again.

Til Depth Do Us Part: Commitments and Determination (Joseph Hellerstein, UC Berkeley and AWS)

Joe started all the way back with Jim Gray's definition/formalization of transaction. The concept derives from law, specifically marriage. A commitment, made publicly, that cannot be unilaterally undone, witnessed and sealed by an authority. But ambiguity and nondeterminism are inherent in distributed systems, which makes implementing transactions challenging.

Joe argued that semantic specifications are too dumb, just input to output. They cannot express what matters. There is a moment at which the inputs stop being negotiable, and the standard i/o spec has no vocabulary for that moment.  (I need to think more about this. A high level spec may leave that out, but I don't see why a more refined spec would not be able to include this.) Joe argued a spec may have to commit a seal, in the sense of the minister sealing the input.

He wants three things in this framework: decision, complexity, and provenance. He calls it a commitment-centric formulation. He focused only on the decision question for this presentation. 

The decision question is: is coordination avoidable? And we already have a good lineup here. CALM: coordination-free consistency iff the spec is monotone Datalog.  CRDTs: coordination-free consistency if a join semilattice. Functional CALM: coordination-free consistency iff the spec is any monotone function, with sealing the input as a Hoare preorder. Complete CAP: CAP holds iff the spec is monotone.

Sealing is what lets you say "this input set is final," which is what turns a non-monotone problem into a monotone one. Joe argues that this is why it belongs in the spec and not in the implementation. He also showed a hierarchy at the end: consensus is a single commitment, transactions are linear commitments. The depth of that commitment structure is what the title is about. I will wait for their paper to appear to understand this better. This was a great talk.

Autoscaling Is Not just a Resource Allocation Problem (Sacheendra Talluri, ClickHouse)

No notes. Listing it for completeness.

Durable Execution: An End Run on the End-to-End Principle? (Johann Schleier-Smith, Temporal Technologies)

Durable execution is the latest attempt at an old goal (Phoenix and David Lomet's "Application Recovery: Advances Toward an Elusive Goal" at MSR), and the benefits are all about the programmer rather than the system: readable control flow, local reasoning, scoped error handling, fewer bugs, less code. The end-to-end argument says reliability belongs at the endpoints, because the middle cannot know what correctness means. But a question in Q&A observed that durable execution puts recovery in the middle, in the runtime. This is an interesting open question.

Session 10: Lakehouses vs HTAP, the showdown (chair: Natacha Crooks)

Lakehouses don't cut it (yet) (Stefano Stefani, Goldman Sachs)

Stefano works on digital transformation at Goldman, post Sybase liberation day, and he framed it as a Goldilocks problem with no right option: OLTP is isolated from the analytical platforms, the lakehouse is great for analytics but not fresh enough, federation is too slow. Let's say a client is on the phone, so the trader needs position and risk data now, and that means a join, over fresh data, fast. Any one of those is easy but all three together is the problem. Stefano says they need a single source of truth with zero copy reads, transactional isolation, and schema evolution.

Lakebases & Lakehouses => LTAP (Ippokratis Pandis, Databricks)

Ippo started with what actually changed: intra-datacenter bandwidth went 10 Gbps in March 2015, 3200 Gbps in July 2023, 6.4 Tbps now, and every assumption about keeping compute near data was priced against the old number. Then the barbarians came, and AI's impact on OLAP is a new access pattern. Rethinking analytics, they hit a wall at around one second and could not beat it. The accepted wisdom is to copy your data into a separate real-time layer for sub-100ms, ClickHouse or Pinot, but then the data is not fresh, complexity grows, and governance breaks, see earlier talk(s). Ippo then talked about Lakehouse/RT, powered by Reyden: tens of thousands of concurrent queries at sub-100ms directly on live Delta and Iceberg tables, Unity Catalog governance intact, no sync pipelines, granular autoscaling down to zero. He also talked about OLTP side and Neon and argued that being able read at an LSN straight out of storage gives a way for OLTP and OLAP workloads to stop fighting over the same machine.

No, HTAP in the Cloud Is Not Solved. The Industry Traded Freshness for Scalability and Called It Innovation (Daniel Ritter, SAP)

No notes, but the title was the argument.

Becca Taft, who joined MongoDB Research to become our colleague, is co-chairing HPTS'28 with Prof. Natacha Crooks. I will look forward to it, if we are still around by then. 

About PHP Remote Code Execution Vulnerability (PHP2Shell CVE-2026-109618 & CVE-2026-109619) Percona Security has been actively monitoring and responding to the recently disclosed PHP2Shell vulnerability, assessing its potential impact on our infrastructure and services.Protecting our customers and their data remains our highest priority. Following the disclosure, our Security and IT teams took immediate steps to ensure … Continued

The post PHP2Shell Vulnerability: Percona Security Update appeared first on Percona.

Data arrives as files. A vendor drops a CSV in a container, an upstream system exports newline-delimited JSON, a daily extract lands in Blob. The usual way into Postgres is a client-side load tool and a hand-written COPY with a column list. With pg_duckdb you can skip most of that: point read_csv() / read_json() at the file in Blob, let DuckDB infer the schema, and CREATE TABLE AS straight into a normal PostgreSQL table — or just query the file in place and never import it.

Setup

Only the Blob secret is needed:

CREATE EXTENSION IF NOT EXISTS pg_duckdb;
SELECT duckdb.create_azure_secret('<blob connection string>');

The files in Blob

A vendor list as CSV and an event stream as newline-delimited JSON:

az://lake/incoming/vendors.csv
   vendor_id,name,country,active,rating
   1,Acme Corp,US,true,4.5
   2,Globex,CH,false,3.9
   3,Initech,FR,true,4.1
   4,Umbrella,DE,true,2.8
   5,Soylent,JP,false,4.9

az://lake/incoming/events.json
   {"event_id":101,"ts":"2026-01-05T10:00:00","user":"alice","action":"login","value":1}
   {"event_id":102,"ts":"2026-01-05T10:02:13","user":"bob","action":"purchase","value":42}
   {"event_id":103,"ts":"2026-01-05T10:05:41","user":"alice","action":"logout","value":1}

Read a CSV with inferred types

read_csv() parses and infers types from the file — no column list, no cast boilerplate to look at the data:

postgres=> SELECT r['vendor_id'], r['name'], r['country'], r['active'], r['rating']
           FROM read_csv('az://lake/incoming/vendors.csv') r
;
 vendor_id |   name    | country | active | rating
-----------+-----------+---------+--------+--------
 1         | Acme Corp | US      | t      | 4.5
 2         | Globex    | CH      | f      | 3.9
 3         | Initech   | FR      | t      | 4.1
 4         | Umbrella  | DE      | t      | 2.8
 5         | Soylent   | JP      | f      | 4.9
(5 rows)

Note active came back as a boolean and rating as a number — DuckDB inferred them from the file; you didn't declare anything.

Materialize into a real PostgreSQL table

To bring it in, wrap it in CREATE TABLE AS and cast each column to the Postgres type you want. (Casting on the way in is good practice — it pins the schema rather than trusting inference, and it gives money/number columns an explicit precision.)

postgres=> CREATE TABLE vendors AS
           SELECT r['vendor_id']::int       AS vendor_id,
                  r['name']::text           AS name,
                  r['country']::text        AS country,
                  r['active']::boolean      AS active,
                  r['rating']::numeric(3,1) AS rating
           FROM read_csv('az://lake/incoming/vendors.csv') r
;
SELECT 5

"vendors" is now an ordinary PostgreSQL table — index it, join it, update it, EXPLAIN it like any other. Two plans make the point.

With duckdb.force_execution = true, DuckDB will still scan the table (because it can), through a TABLE_SCAN on Table: vendors — which already proves "vendors" is a real Postgres relation, not a file handle:

postgres=> SET duckdb.force_execution = true;
SET
postgres=> EXPLAIN (ANALYZE, COSTS OFF) SELECT count(*) FROM vendors WHERE active
;
                   QUERY PLAN
-------------------------------------------------
 Custom Scan (DuckDBScan) (actual time=0.001..0.021 rows=0.00 loops=1)
   DuckDB Execution Plan:

 ┌─────────────────────────────────────┐
 │┌───────────────────────────────────┐│
 ││    Query Profiling Information    ││
 │└───────────────────────────────────┘│
 └─────────────────────────────────────┘
 ┌────────────────────────────────────────────────┐
 │┌──────────────────────────────────────────────┐│
 ││             Total Time: 0.0012s              ││
 │└──────────────────────────────────────────────┘│
 └────────────────────────────────────────────────┘
 ┌───────────────────────────┐
 │           QUERY           │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │      EXPLAIN_ANALYZE      │
 │    ────────────────────   │
 │           0 rows          │
 │          (0.00s)          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │    UNGROUPED_AGGREGATE    │
 │    ────────────────────   │
 │        Aggregates:        │
 │        count_star()       │
 │                           │
 │           1 row           │
 │          (0.00s)          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │         TABLE_SCAN        │
 │    ────────────────────   │
 │       Table: vendors      │
 │      Filters: active      │
 │                           │
 │           3 rows          │
 │          (0.00s)          │
 └───────────────────────────┘
 Planning:
   Buffers: shared hit=1
 Planning Time: 1.196 ms
 Execution Time: 1.034 ms

The TABLE_SCAN with Table: vendors and 3 rows (the active vendors) proves "vendors" is a real Postgres relation DuckDB reads, not a file handle. Turn forcing back off (the default) and the data is served entirely by the PostgreSQL executor — no DuckDB in the picture at all:

postgres=> SET duckdb.force_execution = false;
SET
postgres=> EXPLAIN (ANALYZE, COSTS OFF) SELECT count(*) FROM vendors WHERE active
;
                           QUERY PLAN
-----------------------------------------------------------------
 Aggregate (actual time=0.024..0.025 rows=1.00 loops=1)
   Buffers: shared hit=1
   ->  Seq Scan on vendors (actual time=0.018..0.019 rows=3.00 loops=1)
         Filter: active
         Rows Removed by Filter: 2
         Buffers: shared hit=1
 Planning Time: 0.071 ms
 Execution Time: 0.062 ms

That Seq Scan on vendors is the proof the load worked: the rows are physically in Postgres, queryable by the native engine like any table you created by hand.

Same thing for JSON

read_json() handles newline-delimited JSON (and arrays), inferring fields:

postgres=> SELECT r['event_id'], r['user'], r['action'], r['value']
           FROM read_json('az://lake/incoming/events.json') r
;
 event_id | user  |  action  | value
----------+-------+----------+-------
 101      | alice | login    | 1
 102      | bob   | purchase | 42
 103      | alice | logout   | 1
(3 rows)

And into a table, parsing the timestamp as you go:

postgres=> CREATE TABLE events_log AS
           SELECT r['event_id']::int   AS event_id,
                  (r['ts'])::timestamp AS ts,
                  r['user']::text      AS usr,
                  r['action']::text    AS action,
                  r['value']::int      AS value
           FROM read_json('az://lake/incoming/events.json') r
;
SELECT 3

postgres=> SELECT * FROM events_log ORDER BY event_id
;
 event_id |         ts          |  usr  |  action  | value
----------+---------------------+-------+----------+-------
 101      | 2026-01-05 10:00:00 | alice | login    | 1
 102      | 2026-01-05 10:02:13 | bob   | purchase | 42
 103      | 2026-01-05 10:05:41 | alice | logout   | 1
(3 rows)

For a recurring load, use INSERT INTO events_log SELECT … FROM read_json(…) instead of CREATE TABLE AS, and point the path at the new file (or a glob over a folder of drops — see the partitioned-lakes post).

Or don't import at all

Sometimes you don't need the data in Postgres — you just need an answer. You can aggregate the Blob file directly:

postgres=> SELECT r['country'] AS country,
                  count(*)     AS n,
                  avg((r['rating'])::numeric(3,1)) AS avg_rating
           FROM read_csv('az://lake/incoming/vendors.csv') r
           GROUP BY r['country']
           ORDER BY 1
;
 country | n | avg_rating
---------+---+------------
 CH      | 1 | 3.9
 DE      | 1 | 2.8
 FR      | 1 | 4.1
 JP      | 1 | 4.9
 US      | 1 | 4.5
(5 rows)

That's the nice property of this approach: the same read_csv() / read_json() is either a one-off query or the source of an import, depending on whether you wrap it in CREATE TABLE AS. Query in place for exploration and ad-hoc checks. Materialize when you need indexes, constraints, or OLTP access.

The reverse direction: writing a result back out

Loading isn't one-way. You can also write a Postgres query result back to Blob as Parquet — handy for publishing an aggregate or an extract. On Azure Database for PostgreSQL the plain COPY … TO 'az://…' syntax is blocked for a normal login (it needs superuser), so you drive the write through the DuckDB engine with duckdb.raw_query, which authorizes it with the secret and the az:// allow-list — the object-storage controls PostgreSQL's server-file gate has no say over:

postgres=> SELECT duckdb.raw_query($$
             COPY (SELECT country, count(*) AS n
                   FROM pgduckdb.public.vendors GROUP BY country)
             TO 'az://lake/exports/vendors_by_country.parquet' (FORMAT parquet)
           $$)
;
 raw_query
-----------

(1 row)

Note the pgduckdb.public. prefix on the table: inside raw_query you're in DuckDB's catalog, not PostgreSQL's search path, so a bare vendors won't resolve. The archive post covers this write path in full.

A few practical notes

• Cast on import. Inference is convenient for a look, but pin the types in your CREATE TABLE AS so the table schema is deterministic — and give decimals an explicit precision (numeric(p,s)), which also keeps later DuckDB queries on the table from falling back (see the numeric gotcha side-note).
• CSV options mirror DuckDB's, passed with PostgreSQL named-arg syntax: read_csv('az://…/f.csv', delim := ';', header := true) and so on.
• Globs work here too. read_csv('az://lake/incoming/*.csv') loads a whole drop folder in one statement.
• Big loads stream. DuckDB reads the file in chunks, so you're not limited to what fits in memory for the parse.

Takeaways

• read_csv / read_json over az:// read files straight from Blob with type inference — no pre-defined table, no external load tool.
• CREATE TABLE AS SELECT … FROM read_csv(…) materializes into a normal Postgres table (cast the columns to pin the schema); the native Seq Scan plan proves the rows really landed in Postgres. INSERT … SELECT appends for recurring loads.
• Or skip the import and aggregate the file in place — same function, your choice.
• Globs and DuckDB's CSV options (delim :=, header :=, …) are available via PostgreSQL named-argument syntax.

The archive post wrote cold data out to Blob by hand, step by step. In the last post of the series I turn that export-verify-delete cycle into a single durable, scheduled workflow with pg_durable — so a crash mid-way resumes instead of re-exporting.

In this series, I encountered a pg_duckdb gotcha with unconstrained numeric column. With duckdb.force_execution = true, I expected DuckDB to run my aggregate. But it ran on the PostgreSQL engine instead, with a warning. Here's how to see it and fix it.

Reproduce

I created a table with four columns of the same number, differing only in declared type:

DROP TABLE IF EXISTS nf;
CREATE TABLE nf (
 id bigint
 , n_bare  numeric
 , n_10_2  numeric(10,2)
 , d       double precision
);
INSERT INTO nf SELECT g, g+0.5, g+0.5, g+0.5 
 FROM generate_series(1,100000) g
;
ANALYZE nf
;

Here are their Postgres type modifiers (-1 means "no declared precision"):

postgres=> SELECT attname, atttypmod FROM pg_attribute
           WHERE attrelid='nf'::regclass AND attnum>0 AND NOT attisdropped ORDER BY attnum
;
 attname | atttypmod
---------+-----------
 id      |        -1
 n_bare  |        -1
 n_10_2  |    655366
 d       |        -1
(4 rows)

• n_bare and d show -1 (no declared precision)
• n_10_2 shows 655366, the encoded form of numeric(10,2)

The symptom

I forced DuckDB on, then sum the bare numeric:

postgres=> SET duckdb.force_execution = true;
SET
postgres=> EXPLAIN SELECT sum(n_bare) FROM nf
;
WARNING:  (PGDuckDB/CreatePlan) Prepared query returned an error: Binder Error: No function matches the given name and argument types 'sum(UnsupportedPostgresType(DuckDB requires the precision of a NUMERIC to be set))'
                      QUERY PLAN
-----------------------------------------------------
 Aggregate  (cost=2074.00..2074.01 rows=1 width=32)
   ->  Seq Scan on nf  (cost=0.00..1824.00 rows=100000 width=8)
(2 rows)

That's a plain PostgreSQL plan. I've set force_execution = true, yet the top node is a plain Aggregate, not a Custom Scan (DuckDBScan). DuckDB declined and PostgreSQL ran it. A WARNING at plan time is emitted when pg_duckdb tries to build the DuckDB plan and bails on the unconstrained numeric. But it's a WARNING, not an error, easy to miss from an application, and the query still returns the right answer on the native executor. If you were only watching the response time you'd think pg_duckdb didn't help, when it never ran the query in DuckDB at all.

I've run the same sum over the numeric(10,2) column:

postgres=> EXPLAIN SELECT sum(n_10_2) FROM nf
;
                   QUERY PLAN
-------------------------------------------------
 Custom Scan (DuckDBScan)  (cost=0.00..0.00 rows=0 width=0)
   DuckDB Execution Plan:

 ┌───────────────────────────┐
 │    UNGROUPED_AGGREGATE    │
 │    ────────────────────   │
 │    Aggregates: sum(#0)    │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │         PROJECTION        │
 │    ────────────────────   │
 │           n_10_2          │
 │                           │
 │       ~100,000 rows       │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │  PGDUCKDB_POSTGRES_SCAN   │
 │    ────────────────────   │
 │         Table: nf         │
 │    Projections: n_10_2    │
 │                           │
 │       ~100,000 rows       │
 └───────────────────────────┘
(25 rows)

DuckDB runs it. The only difference between the two queries is that one column has a declared precision and the other doesn't.

Why

DuckDB's DECIMAL is fixed precision, up to 38 digits. A Postgres numeric with a declared (p,s) maps straight onto it. A bare numeric has no bound — there's nothing for DuckDB to map it to — so pg_duckdb leaves the whole query to PostgreSQL. (A numeric(38,10) works fine, only the unconstrained one falls back.) The fallback itself is a feature: forcing DuckDB can never break a query, it just might not accelerate it. The problem is only that it's silent.

The fix

One session setting makes the bare numeric run in DuckDB, read as double:

postgres=> SET duckdb.convert_unsupported_numeric_to_double = true;
SET
postgres=> EXPLAIN SELECT sum(n_bare) FROM nf
;
                   QUERY PLAN
-------------------------------------------------
 Custom Scan (DuckDBScan)  (cost=0.00..0.00 rows=0 width=0)
   DuckDB Execution Plan:

 ┌───────────────────────────┐
 │    UNGROUPED_AGGREGATE    │
 │    ────────────────────   │
 │    Aggregates: sum(#0)    │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │         PROJECTION        │
 │    ────────────────────   │
 │           n_bare          │
 │                           │
 │       ~100,000 rows       │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │  PGDUCKDB_POSTGRES_SCAN   │
 │    ────────────────────   │
 │         Table: nf         │
 │    Projections: n_bare    │
 │                           │
 │       ~100,000 rows       │
 └───────────────────────────┘
(25 rows)

No more warning, and the top node is now Custom Scan (DuckDBScan) — DuckDB runs it, reading the bare numeric as double.

The caveat is in the name: double is floating point, so you can lose the exact decimal precision numeric guarantees — fine for analytical sums and averages, think twice for exact financial rounding. The cleaner fix is to declare the precision in the schema (numeric(12,2)), so DuckDB runs it exactly, no flag.

Takeaway

With force_execution = true, always EXPLAIN and check the top node is Custom Scan (DuckDBScan). If it's a plain PostgreSQL node, DuckDB declined — and a bare numeric (typmod -1) is the usual reason. Declare the precision, or set duckdb.convert_unsupported_numeric_to_double to true if your can accept that decimal fractions that cannot be stored with exact base-10 precision are rounded.

Even with duckdb.force_execution = true, you should always run EXPLAIN to verify that the query utilizes the Custom Scan (DuckDBScan) node. If the planner falls back to native PostgreSQL nodes, it indicates that DuckDB declined the query—most commonly due to an unconstrained PostgreSQL NUMERIC column (with a type modifier of -1). To resolve this, explicitly declare the column's precision and scale, or set duckdb.convert_unsupported_numeric_to_double = true if your workload can tolerate the inherent precision loss and rounding behavior of binary floating-point (DOUBLE) conversions.

Jim Gray started HPTS 42 years ago, and it still runs the same way at Asilomar with 0 proceedings, 1 track, 100 invited people, and ample discussion breaks. It's one of my favorite events, and I look forward to attending it every two years. This is my...umm...fifth HPTS. I have been attending all of them since 2015, but I seem to have missed it in 2019. Yep, my life is a blur, even for me.

Here are my notes from day 1. I will put day 2 and day 3 in a second post.

Keynote: Reality is the Final Verifier. Two Key Gaps in Agentic Software Engineering (Ion Stoica, UC Berkeley)

Ion started with the promising parts. LLMs are able to solve difficult math problems, and provide proofs for conjectures that have been outstanding for many decades (Ok, maybe this is not as promising to mathematicians). LLMs also play scientist with AlphaEvolve, and OpenEvolve. In summer 2025 Berkeley ran a seminar where students applied agentic tools to research problems. They took a published state-of-the-art result, used it as the baseline, kept the objective fixed, and let the loop run. Ten of twelve students got better results than the published baseline. The ADRS blog series and papers come out of this work.

But then comes the reality check. Yes, AI seems to improve researcher productivity, but it did not disrupt how we do research. It cannot come up with new research ideas. It cannot build new production systems. The rest of the talk was his attempt to explain this gap.

Ion led on with an anecdote from the Jitski project, where they asked an agent to synthesize a key-value store from scratch to maximize throughput. They applied the standard AI development loop. And surprisingly, one solution came back 6x faster than anything before it. How? The benchmark was YCSB, and YCSB generates its values with a hash function. The agent noticed this and stopped storing values altogether and computed them on the fly with get(key) = hash(key). This is the very definition of "reward hacking".

Ion frames this in terms of two gaps:

• Requirement gap: between the user's intentions and what the prompt and environment capture. Intentions are always broader than the prompt.
• Model gap: between the model of the world and the world. The real world is always broader than the model.

Ion elaborated more on why closing the requirement gap is hard. He asked the room: would you think to specify that "a key-value store must actually store values", "a load balancer should actually process the requests", and "a vacuum shouldn't spill garbage on the floor to pick back up as a reward hack"?

As a formal methods person: yes! Yes, I totally would. These would be about the first things I'd write to stop a vacuous specification satisfaction. That is exactly what writing a specification is. You state the thing that is so obvious which most people would not bother to say. Ion raised these examples and then waved them off by mocking how silly they sound, which to me (and other formal methods people) is backwards.

For illustrating the model gap, he reached for Brian Cantwell Smith's "The Limits of Correctness." On October 5, 1960, the Ballistic Missile Early Warning System station at Thule reported a massive incoming attack, which was about to trigger a nuclear retaliation. But the reality was different. The moon had risen, and the radar was getting its own signal back off the lunar surface. The system was recently deployed and nobody had accounted for the fricking moon. The model was fine, but the world was bigger than the model.

Note that the agentic success stories are all in domains with narrow gaps. In mathematics, the model gap is closed by construction, because the axiomatic system is the world, so the model can match it exactly. The requirement gap stays narrow, since translating intent into a formal theorem statement has already been done by a mathematician. Hardware design is similar. And optimizing an existing system is much easier than building one, because the existing system pins things down and narrows the specification.

The two gaps give birth to two bottlenecks. The requirement gap bottlenecks on human judgment and the model gap bottlenecks on evaluation. Ion's prescription to alleviate these gaps is an outer loop assurance revision loop with (again) humans involved to improve the process. He listed some future directions: selective escalation (ask only the highest-value human question), iterative disambiguation, cascading evaluations, impact-bounded control, continual system learning.

In disguise, this was a talk about how badly we need specifications. It looks like people are rediscovering the importance of specifications, as LLMs outsmart and bedevil them. Look at day 1! Three of the four sessions were about AI, and they all mention the importance of specification! When you can't trust the generator, you need to trust something you can check. And that is the formal methods argument. This was the vibe for HPTS'26. Low key, I felt like a rockstar in HPTS. Whoa, I know TLA+!

Student intros

One of the best traditions at HPTS is that students get dedicated time to introduce themselves. Among this year's cohort, there were an unusual number of piano players, a healthy number of odd hobbies, and someone who does Byzantine music (which I presume requires 3F+1 singers).

A note for the record. We will have AGI before we solve projecting a laptop onto a screen. Twenty-six years I have been going to conferences, and it is the same dance every time: the connector is incompatible, the resolution is awry, the mirrored display doesn't work, and people hunch over the laptop to fix it. This scenario played out at least 5 times to sabotage/delay presentations over the two and a half days. The people in this room keep trillions of blobs available across the world, but somehow getting the projector playing well with the laptop is still unreliable.

Session 2: Putting database researchers out of a job 

The Future Is Bespoke: Synthesizing One-Size-Fits-One DBMSs with LLM Coding Agents (Carsten Binnig, TU Darmstadt)

Carsten opened by playing the room, saying "relational databases are the foundation of civilization" (think clay tablets Sumerian traders used for recording transactions). He then argued that bespoke databases already happened. TigerBeetle is 3000x faster than Postgres for what it does. TigerBeetle only exists because it targets a market big enough to pay for it. But bespoke for everybody has never worked, because the engineering cost per customer is absurd. LLMs change the cost side of course. Feed in the schema, the data, and the workload queries, and synthesize an OLAP DBMS for exactly that workload. Correctness has an oracle here, because DuckDB serves as a trusted reference implementation to check the results against. The synthesis proceeds in layers: start general purpose, then specialize storage, then execution, then the concurrency control scheme. Ten hours and $140 per synthesized system. Future work is to extend the idea to bespoke compute stacks, where the application is codesigned with the OS and the hardware.

Towards AI-Generated Database Management Systems (Jiale Lao, Cornell)

Jiale started from the same premise as Carsten, fast = f(hardware, workload), but went a different direction on correctness. He pointed out that even the best databases, DuckDB included, return wrong results. No database is formally verified; they are all tested, and testing shows the presence of bugs, not their absence. So where Carsten uses DuckDB as the verifier, Jiale wants to not need a verifier at all. The goal is generating and proving, rather than generating and testing. The tools are Verus for verifying Rust, and SQLSpec, a mathematical specification of SQL, as the thing you prove against. They are at about 75% proved, with 3x amplification (rather than 20x with CompCert). His conclusion is one I keep hearing this year: our job is now writing specs and checkers.

Specification-Driven Synthesis of Query Execution Logic for Novel Storage Systems (Puneet Mehrotra, UBC)

The talk introduces FlexoGraph, a graph database built on WiredTiger. As far as I can understand, the synthesis problem here is to produce the query execution logic from a specification, for a storage system nobody has written an engine for yet. This seems to be a challenging test of specification-driven synthesis, since there is no reference implementation to lean on.

Learned Systems Are Dead; Long Live Learned Systems (Todd Lipcon, Google)

On paper, the opportunity for learned systems is real. Databases are full of heuristics (query plans, index optimization, cache eviction), and every one of them is a knob somebody guessed at. But the challenges turn out to be just as real: a low-latency budget that an inference call blows immediately, interpretability, maintenance, and the fact that these systems are hard to simulate. Todd then put up the list of projects that didn't make it: B-tree learned indices (never actually launched despite the NeurIPS'20 paper), ML cache eviction, Vizier-tuned cache eviction, and Kepler query optimization. The RL merge compaction work, the Ziggurat algorithm, was hard to train, complex, and got 2% improvement, so it didn't ship. Todd's diagnosis is the tax of the embedded model. You pay inference latency on every decision, and you pay opacity and risk forever, and that tax is bigger than the win for most database decisions. The takeaway is this: Don't ship the model, ship what the model wrote. Instead of embedding an LLM in the hot path, have it generate the C++ algorithm, read the C++, and ship the C++. You get the search power of the model and you pay none of the inference tax. You also get an artifact a human can review, which handles the interpretability and maintenance problems at the same time. The cache eviction work pivoted this way in 2024.

Session 3: What data stack do our agents need?

Commits, Transactions, Durable Execution: Git-for-everything and the self-driving data stack (Jacopo Tagliabue, Bauplan)

Bauplan is OLAP infrastructure for agents. Jacopo's mental model for agent infrastructure is git: branch, try, throw away, merge. The systems work is getting from files to Iceberg tables. Iceberg brings database-like tables to object storage, with table atomicity and optimistic locking. With those two primitives you can give every agent its own branch, and make them merge transactions.

Execution LogAct: Enabling Agentic Reliability via Shared Logs (Mahesh Balakrishnan, Meta)

Agents corrupt state, crash mid-change, and overwrite each other. You cannot review an agent ahead of time because an agent is a self-writing program: it goes to the LLM and asks for the next lambda to run. So how do you provide guarantees for an execution whose code is being written as it runs? Mahesh's answer is LogAct, a transactional framework for agents built on a shared log. Before execution, the agents vote through an AgentBus that carries intention, votes, and commit. During execution, a WAL handles failure recovery. After execution, the log serves as the audit trail. The insight is this: Do not try to make the agent trustworthy, instead make the record of what it did trustworthy, and build the guarantees there. This also helps with introspecting agentic activity and auditability. When your agents cause an outage at agentic speed, the log is your friend.

The Postgres Trap: Database Incrementalism in the Age of Agents (Ajay Rayasam, Osage University Partners)

I took no notes in this one. I'm listing this for completeness.

AI for Production (Mayank Agarwal, Resolve AI)

Mayank is CTO of Resolve AI, and the talk was about self-driving production. Again, no notes from this presentation. Maybe check the HPTS agenda. Some authors will provide their presentations to be listed there.

Session 4: Architecture Lessons 

Ten Years of Magic Pocket: Lessons From Scaling an On-Prem Exabyte-Scale Blob Store (Facundo Agriel, Dropbox)

Dropbox left S3 ten years ago, and this talk reflects on the reliability lessons learned from managing ten exabytes, trillions of blobs, and billions of deletes every day. A durability model takes three inputs: AFR (annual failure rate), repair rate (how quickly failed fragments get repaired), and redundancy (erasure coding or replication). The model is only as good as the AFR you plug into it, and the lesson Facundo drew is that failures don't care about your durability model. He gave two examples. Drives hate the heat: high temperature affected 30% of one specific drive population. The AFR shot up, but only for that revision, in those conditions. They built a system to drain those drives early and got out from under it. The second example is that SSDs wear out together. Their metadata tier was being used harder than they knew, and because the drives were installed together and wore at the same rate, they were heading toward a correlated failure.

He also talked about the importance of checksums and verifiers, but I found the changing tradeoffs of HDD storage more interesting. Drives are now 40 TB. For a decade, rising density reliably lowered unit cost, but it is ending. Cost is going up, density is still going up, and I/O is flat. Over the period he measured, drive capacity grew 19x while bandwidth grew only 1.5x. It now takes 36 hours to read a single drive end to end. So everyone will have to optimize for I/O rather than capacity. When you buy a drive twice the size at the same bandwidth, each stored byte comes with half the bandwidth to reach it. Buying capacity now means buying progressively less access to it. The market numbers are bad too. Western Digital is essentially sold out. Memory contract forecasts are up 3x. Seagate's 2028 nearline exabytes are already allocated.

How We Built Spanner Omni (Sebastian Kanthak, Google)

Omni is externally-facing Spanner: a downloadable Spanner that runs from a single laptop up to a multi-regional cluster of thousands of VMs. Version 1 shipped September 30, 2026, but don't get too excited. It's not opensource, and it requires buying a license to use after trial period expires.

The engineering problem is the fun part. Spanner sits on Colossus for the filesystem, TrueTime for the clock, and Chubby for leader election and membership. You cannot ship any of those. So for each dependency they had to decide what the smallest replacement is. Colossus makes file movement transparent, so for Omni they built the part of Colossus that Spanner actually depends on. Chubby became Spanner Omni-Chubby, providing leader election and the membership service.

The clock was the question everyone was interested in. Sebastian effectively said Spanner does not need GPS and atomic clocks in every datacenter after all. The commit wait only has to cover the clock uncertainty, and the cost is hidden as long as the wait is comparable to the 1-RTT you were already paying to replicate the commit. So you don't need the uncertainty to be tiny, you only need it to be smaller than 1-RTT. And to get there, a single site doing clock sync is enough: have everyone discipline against one logically centralized reference, with a fallback reference in each region in case the central site becomes unavailable.

Preventing and Mitigating Logical Corruptions in Salesforce Database (Vaibhav Arora & Jameison Martin, Salesforce)

I don't have notes from this presentation either.

Gong Show

These are lightning talks that are five minutes each. I provide some notes with little or no context.

What Postgres would look like if we started today (Manish Jain)

Own your resources; watch your egress. An LSM forest, bitmaps, batched execution. Physical replication rebuilt for an LSM. High availability with read-your-writes, where the WAL is Raft plus hybrid logical clocks. And instant branching with paying for what you change, not for what you touch.

Your database is all you need (Qian Li, DBOS)

DBOS is about providing a durable execution as a library on top of the database you already have: workflows, queues, and events, with DBOS tables living in your existing database. No new system to operate was a big part of the "pitch". Qian sang her presentation with perfect pitch to the tune of "my favorite things" and she was a co-winner of the Gong Show.

From Vitess to Multigres: Lessons learned (Sugu Sougoumarane, Supabase)

The talk was mostly about trying to build consensus guarantees on native Postgres replication following from an informal statement of consensus.

Generative AI; Generative Testing (Kyle Kingsbury, Jepsen)

The talk is delivered at Kyle speed and was very entertaining, making him the co-winner of the Gong Show. Kyle is not a fan of LLMs (to say the least) and does not find them useful for his checking/testing work.

Resurrecting Semantic Concurrency Control (William Schultz, MongoDB) 

Will gave a great presentation arguing that traditional database concurrency control operates at a too low-level, at the key-value storage layer, which discards useful application semantics. By shifting concurrency control to a higher semantic layer using a unified intermediate representation, databases can reason directly about rich operation logic rather than simple reads and writes. He argued preserving this context enables optimizations like operation merging and smarter scheduling, which significantly reduce transaction aborts under high contention.

Open Storage Format for OLTP (Sunkanmi Adewumi, UC Berkeley)

Scalable Image AI via Self-Designing Storage (Utku Sirin, Harvard)

Mind the Bandwidth Gap: Scaling Analytics with CXL Memory (Georgiy Lebedev, EPFL)

A Little Little's Law Keeps the Metastable Failure Away (Kiran Kumar Muniswamy Reddy, MongoDB)

Badly configured queue sizes are a common factor for metastability failures. Kiran proposed using Little's Law to compute the queue size instead of guessing it.

Failure resilience in large-scale LLM training on TPUs (Alexander Shraer, Google)

LLMs are compute. Context is memory. Inference speed is king. (Mehul Shah, Glean)

And remember, all of these were only day 1.

Welcome back to Village News, our curated roundup of MySQL and database news. This issue covers two weeks, with MySQL 26.10 landing at the end of them.

If you want to get these updates, just subscribe to the blog.

Enjoy!

MySQL News

Note: Aggregated MySQL news can be found

A data lake is rarely one file. Once you archive or publish data by period, you get a tree of files — and the trick is querying the whole thing as one table while only reading the parts a query needs. pg_duckdb does both: globs read many files as one, and Hive-style partitioning turns the folder names into columns and lets DuckDB skip files it doesn't need. With read_parquet(), the query itself chooses the file set and reader options, rather than requiring a foreign table for each file. Here are some examples from Azure Database for PostgreSQL.

Setup

Only the Blob secret is needed for this post — everything else reads from the lake:

CREATE EXTENSION IF NOT EXISTS pg_duckdb;
SELECT duckdb.create_azure_secret('<blob connection string>');

A partitioned layout

The convention: a folder per partition, named column=value, each holding one or more Parquet files. Here, four yearly files of sales data:

az://lake/sales_part/
   year=2022/data.parquet
   year=2023/data.parquet
   year=2024/data.parquet
   year=2025/data.parquet

You produce these one file per year. The only new thing here is the folder naming (year=YYYY/). The write from PostgreSQL is covered in the archive post. With the sales table from that post, one year's export looks like this:

SELECT duckdb.raw_query($$
  COPY (
    SELECT * FROM pgduckdb.public.sales
    WHERE sale_date >= DATE '2022-01-01'
      AND sale_date < DATE '2023-01-01'
  ) TO 'az://lake/sales_part/year=2022/data.parquet'
  (FORMAT parquet)
$$);

Note the SELECT before duckdb.raw_query and the pgduckdb.public. catalog prefix: inside raw_query you're in DuckDB's catalog, not PostgreSQL's. Keep this wrapper on the tested Flexible Server: plain PostgreSQL COPY ... TO 'az://...' is rejected by the service's file-COPY guard, even though self-managed pg_duckdb can intercept it. The export also requires permission to execute duckdb.raw_query(). A client-side export and upload is the alternative described in the archive post.

The column=value convention lets the reader interpret the folder names as partition columns, and a wildcard selects the files across those folders.

Read them all with one glob

A * in the path reads every matching file as a single table:

postgres=> SELECT count(*) FROM read_parquet(
 'az://lake/sales_part/year=*/data.parquet'
)
;
  count
---------
 8001032
(1 row)

A specific path reads one partition:

postgres=> SELECT count(*) FROM read_parquet(
 'az://lake/sales_part/year=2024/data.parquet'
)
;
  count
---------
 2003093
(1 row)

And an explicit list reads an exact set — note the ARRAY[...] syntax (a bare [...] is DuckDB dialect that the PostgreSQL parser rejects):

postgres=> SELECT count(*) FROM read_parquet(ARRAY[
 'az://lake/sales_part/year=2024/data.parquet',
 'az://lake/sales_part/year=2025/data.parquet'
           ])
;
  count
---------
 4003149
(1 row)

Want to know which files a glob actually touched? Add the filename column:

postgres=> SELECT DISTINCT r['filename']
           FROM read_parquet(
 'az://lake/sales_part/year=*/data.parquet',
 filename := true
) r
           ORDER BY 1
;
                  filename
---------------------------------------------
 az://lake/sales_part/year=2022/data.parquet
 az://lake/sales_part/year=2023/data.parquet
 az://lake/sales_part/year=2024/data.parquet
 az://lake/sales_part/year=2025/data.parquet
(4 rows)

(pg_duckdb passes DuckDB's optional arguments with PostgreSQL's named-argument syntax, name := value — so filename := true, hive_partitioning := true.)

Turn the folder name into a column

The partition value lives only in the path, not inside the files. Turn on hive_partitioning and year becomes a real column you can group and filter on:

postgres=> SELECT r['year'] AS year, count(*) AS n
           FROM read_parquet(
 'az://lake/sales_part/year=*/data.parquet',
  hive_partitioning := true
) r
           GROUP BY r['year']
           ORDER BY 1
;
 year |    n
------+---------
 2022 | 1998570
 2023 | 1999313
 2024 | 2003093
 2025 | 2000056
(4 rows)

Because the year is encoded in the layout, you can compute per-year totals even without the files carrying a date column at all.

Partition values are not necessarily text. On this server (PostgreSQL 18.6, pg_duckdb 1.1.0), SELECT pg_typeof(r['year']) over this reader returns bigint: DuckDB infers the type from the folder values. Both r['year'] = 2024 and the quoted literal below work and prune to one file.

When file schemas evolve

If a later year's file adds a column, use union_by_name := true to align columns by name rather than position:

SELECT count(*)
FROM read_parquet(
  'az://lake/sales_part/year=*/data.parquet',
  hive_partitioning := true,
  union_by_name := true
);

Columns absent from a file become NULL for that file's rows. This handles added or missing columns, not arbitrary incompatible type changes.

The payoff: partition pruning, visible in the plan

This is why you partition. Filter on the partition column and DuckDB reads only the matching files. EXPLAIN shows it — and here is the full verbatim plan:

postgres=> EXPLAIN (ANALYZE, COSTS OFF)
           SELECT count(*)
           FROM read_parquet('az://lake/sales_part/year=*/data.parquet',
                             hive_partitioning := true) r
           WHERE r['year'] = '2024'
;
                   QUERY PLAN
-------------------------------------------------
 Custom Scan (DuckDBScan) (actual time=0.001..0.002 rows=0.00 loops=1)
   DuckDB Execution Plan:

 ┌─────────────────────────────────────┐
 │┌───────────────────────────────────┐│
 ││    Query Profiling Information    ││
 │└───────────────────────────────────┘│
 └─────────────────────────────────────┘
 ┌────────────────────────────────────────────────┐
 │┌──────────────────────────────────────────────┐│
 ││              Total Time: 0.78s               ││
 │└──────────────────────────────────────────────┘│
 └────────────────────────────────────────────────┘
 ┌───────────────────────────┐
 │           QUERY           │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │      EXPLAIN_ANALYZE      │
 │    ────────────────────   │
 │           0 rows          │
 │          (0.00s)          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │    UNGROUPED_AGGREGATE    │
 │    ────────────────────   │
 │        Aggregates:        │
 │        count_star()       │
 │                           │
 │           1 row           │
 │          (0.00s)          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │         TABLE_SCAN        │
 │    ────────────────────   │
 │         Function:         │
 │        READ_PARQUET       │
 │                           │
 │       File Filters:       │
 │       (year = 2024)       │
 │                           │
 │    Scanning Files: 1/4    │
 │    Total Files Read: 1    │
 │                           │
 │       2,003,093 rows      │
 │          (0.11s)          │
 └───────────────────────────┘
 Planning:
   Buffers: shared hit=1
 Planning Time: 329.131 ms
 Execution Time: 330.574 ms

The lines that matter are Scanning Files: 1/4 and Total Files Read: 1. The glob matched four files, but the year = 2024 filter let DuckDB skip scanning data from three of them — it decided that from the folder names alone (File Filters: (year = 2024)). This does not mean zero listing or metadata requests: resolving the glob and binding the reader can still require them. The 2,003,093 rows at the scan is the one file's actual row count; a count(*) can use Parquet metadata rather than decode every row. With hundreds of monthly partitions, a filter selecting one month can restrict the data scan to that month's files, whether there is one file or several. That's why this layout pays off, and pg_duckdb gives it to you over Azure Blob with the filter sitting in ordinary SQL.

Does abfss:// work on Azure Database for PostgreSQL?

DuckDB's Azure extension supports both Blob (az://) and ADLS Gen2 (abfss://) paths, but the managed server's allow-list comes first. I retested on this Flexible Server with PostgreSQL 18.6 and pg_duckdb 1.1.0:

postgres=> SHOW duckdb.enable_external_access;
 duckdb.enable_external_access
-------------------------------
 off
(1 row)

postgres=> SHOW duckdb.allowed_directories;
 duckdb.allowed_directories
----------------------------
 az://
(1 row)

postgres=> SELECT count(*)
FROM read_parquet('abfss://lake/sales_part/year=2024/data.parquet');
ERROR:  (PGDuckDB/CreatePlan) Prepared query returned an error: Permission Error: Cannot access file "abfss://lake/sales_part/year=2024/data.parquet" - file system operations are disabled by configuration

LINE 1: SELECT count(*) AS count FROM system.main.read_parquet('abfss://lake/sales_part/year=2024...
                                      ^
postgres=>

Both a single-file read and a glob fail, including with the fully qualified abfss://<account>.dfs.core.windows.net/lake/... form. The same reads through az:// succeed. These settings require superuser privileges to change, which this login does not have.

So the two schemes are not interchangeable on this server. The rejection happens in DuckDB's filesystem permission check, before we can test the ADLS endpoint or compare listing performance. Use az:// here; there is no measured basis in this test for recommending abfss:// as faster.

Takeaways

• A glob (year=*/data.parquet) reads a whole partitioned lake as one table. A path reads one partition. ARRAY[...] reads an explicit set.
• hive_partitioning := true turns column=value folder names into queryable columns — group and filter on them like any column.
• Filtering the partition column prunes files before reading: the plan shows Scanning Files: 1/4. This is the main reason partitioning pays off.
• Pass DuckDB's reader options with PostgreSQL named-arg syntax (:=); use ARRAY[...] not [...] for file lists.
• Use union_by_name := true when files add or omit columns across partitions.
• On the tested Flexible Server, az:// is allowed but abfss:// is blocked by configuration, for both single files and globs.

Everything so far has read Parquet. Blob also collects the messier arrivals — vendor CSVs, newline-delimited JSON event streams. Next I'll point read_csv and read_json at those and load them into Postgres with a single statement.