a curated list of database news from authoritative sources

November 17, 2026

Beyond Disk and WAL: What Limits PostgreSQL Scaling?

The earlier tests improved read I/O and removed standby acknowledgment waiting, but the primary still waited for local WAL synchronization. Separate, later durable-regime traces located substantial delay in guest-OS write admission. This post returns to the diagnostic where I stopped waiting for commit durability to see what remained.

I used session-level synchronous_commit=off for the same 512-worker, 15+45-minute workload. This allows a successful commit response before WAL is durable: a crash can lose recently acknowledged transactions. Keep in mind that I do this in a lab to investigate PostgreSQL bottlenecks. It deviates from the initial comparison with HorizonDB's built-in high-availability and from the original durable-HA configuration. Still, let's see how far we can increase the throughput in this lab.

WAL, fsync and full_page_writes remained enabled. Both streaming replicas were retained. Automatic Patroni failover was separately paused for the diagnostic.

Verify the mode, then measure

The override was supplied through the workload's PGOPTIONS. Before launch, a test Pgtcl connection using that environment executed:

SHOW synchronous_commit;

Its recorded check was:

2026-11-08T08:08:47.692827+00:00 BENCH_COMMIT_MODE_CHECK adapter=Pgtcl effective=off

This records the adapter test connection's setting. The workload was launched with the same connection environment, while the server default remained on.

The result was:

2026-11-08T09:08:53.198898+00:00 Vuser 1:TEST RESULT : System achieved 66096 NOPM from 152007 PostgreSQL TPM

93.92% of sampled active-client observations were at LWLock:BufferContent. The question had moved from when WAL becomes durable to which backends needed conflicting access to an in-memory page. I wanted to identify that page before changing shared_buffers.

The page identification below comes from a later synchronous_commit=on follow-up.

What BufferContent protects

BufferContent is a wait for a lightweight lock protecting the contents of a cached relation page, normally 8 KiB. Shared holders can coexist. Exclusive access conflicts with other holders, even when the page is already in memory.

BufferMapping instead concerns associating a block with a buffer slot. BufferIO concerns buffer I/O completion.

For Oracle readers, LWLocks are closer to latches and mutexes than transactional enqueues: they protect shared-memory structures, not business rows until commit. PostgreSQL's heavyweight lock manager is the closer comparison for transaction and object locks. pg_locks and pg_blocking_pids() do not expose a content-LWLock holder.

A buffer pin is different again: it prevents recycling the slot, but does not grant exclusive access to its contents.

Oracle's buffer busy waits includes P1/P2/P3 parameters identifying file, block, and block class. PostgreSQL's pg_stat_activity exposes neither that page identity nor per-wait elapsed time for BufferContent. query_start times the statement, not its current wait. Oracle active-session history is sampled too. Samples and timed waits are different measurements.

Observe active backends

The collector grouped pg_stat_activity by backend type, state, and wait event. This query shows the active-client part:

SELECT wait_event_type,
       wait_event,
       count(*) AS sessions
FROM pg_stat_activity
WHERE backend_type = 'client backend'
  AND state = 'active'
GROUP BY wait_event_type, wait_event
ORDER BY sessions DESC;

active describes statement execution status and includes statements currently waiting.

These are retained samples from the synchronous_commit=off measured window, November 8, 2026, 08:23:51-09:08:53 UTC:

Observer timestamp, UTC Phase Active clients At BufferContent Percentage
08:24:43.424849 measurement 513 476 92.79
08:34:43.913026 measurement 512 483 94.34
08:44:44.413830 measurement 512 491 95.90
08:54:44.923233 measurement 513 484 94.35
09:08:45.608882 measurement 513 470 91.62

The collector included all active client backends, including its own connection, without database/user filtering. These counts therefore differ from the 512 workload workers.

Across all 45 samples taken one minute apart inside the measured interval, there were 23,063 active-client observations:

Wait Observations Share
LWLock:BufferContent 21,660 93.92%
IO:DataFileRead 981 4.25%
No reported wait 376 1.63%
Client:ClientRead 25 0.11%
IO:DataFileFlush 10 0.04%
IO:DataFileWrite 7 0.03%
IO:DataFileExtend 2 0.01%
IPC:BufferIo 2 0.01%

None of the 45 snapshots contained LWLock:WALWrite or IPC:SyncRep. The 93.92% is 21660 / 23063, the share of sampled active-client observations.

Background persistence continued. In-window WAL counters showed approximately 194.46 MiB/s generated and 220.76 syncs/s. WAL must still precede durable data-page writes.

A buffer is not necessarily a table's data page

Shared buffers can hold more than application table rows:

Page contents Examples
Table heap Row versions, headers and item identifiers
Index B-tree, GIN, GiST, SP-GiST, hash or BRIN pages, including metadata
TOAST relation Out-of-line values and their index
System catalogue Catalogue heap and index pages
Materialized view Stored rows and indexes
Sequence The sequence relation's state page
Relation metadata Free-space-map and visibility-map pages
Unlogged relation Working pages and initialization-fork pages

Partitions store their own pages. The partitioned parent does not store heap rows. Temporary relations use backend-local buffers. WAL buffers and SLRU caches, such as transaction-status caches, are separate structures with different synchronization.

Relation forks and page identity

A fork is a physical part of a relation with a specific purpose:

Number Fork File suffix Contents
0 main none Heap, index or other primary relation storage
1 fsm _fsm Free-space map
2 vm _vm All-visible and all-frozen information for heap pages
3 init _init Unlogged relation's initialization state after an unclean shutdown

Not every relation has every fork. Indexes have no visibility map. See the file-layout documentation.

PostgreSQL's BufferTag identifies a page by tablespace, database, filenode, fork, and block number. All five matter: main-fork block 0 and VM-fork block 0 are different pages. Catalog OIDs and physical filenodes are distinct identifiers. A rewrite can change the filenode while preserving the OID.

Separate follow-up: identifying pages with synchronous_commit=on

For page-level diagnosis, I used a separate 512-worker pilot with synchronous_commit=on and normal HA. BufferContent was still visible in the capture-trigger samples at 5.47-18.75%, enough to trace page contention under ordinary durable commits.

1️⃣ Capture the page tag at the lock event. I traced selected PostgreSQL processes by PID with Linux perf. Uprobes are tracing points in userspace machine code. I checked their instruction locations and registers against the installed PostgreSQL executable.

At acquisition, semaphore-wait and release points, the probes recorded the lock address and the full page tag from its buffer descriptor. Pairing required the same page identity at the beginning and end of an interval.

Eight nominal three-second recordings sampled selected backends and background processes. This is a selected-field excerpt of the decoded, aggregated wait output:

{
  "tag": [1663, 57465, 57491, 2, 10215],
  "waited_loop_count": 78,
  "loop_wait_ns": 2357679631
}

In order, that tag means tablespace 1663, database 57465 (tpcc), filenode 57491, fork 2, block 10215. Fork 2 was recorded by the probe.

2️⃣ Resolve the filenode through the catalogs. Connected to tpcc, the collector mapped captured filenodes through pg_class. This reduced lookup resolves the same identity:

SELECT n.nspname AS schema,
       c.relname AS relation,
       c.relkind,
       am.amname,
       pg_relation_filenode(c.oid) AS filenode
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
LEFT JOIN pg_am am ON am.oid = c.relam
WHERE c.oid = pg_filenode_relation(1663::oid, 57491::oid);

The retained catalog row, reformatted from the collector's JSON:

 schema | relation | relkind | amname | filenode
--------+----------+---------+--------+---------
 public | stock    | r       | heap   |    57491

The relation is public.stock. Fork 2 identifies its visibility map, so this is stock VM block 10215, not stock heap block 10215.

3️⃣ Keep the block identity in the results. These were the leading measured pages by cumulative completed semaphore-loop elapsed time. All have the same tablespace/database/filenode and VM fork:

Stock VM block Completed loops Cumulative loop ms
10215 78 2,357.680
10031 66 2,206.913
10032 63 2,076.327
10033 55 1,984.848

Stock's VM fork was 94,912,512 bytes, or 11,586 8-KiB pages, consistent with those block numbers.

Selected processes contributed these completed loops. The elapsed intervals include semaphore work, rescheduling, and probe overhead. Several loops can belong to one acquisition.

Captured holder operations on these pages included heap_update. The longer waiter intervals remain only partially attributed.

The identification chain in this normal-on pilot is event-time tag -> catalogue lookup -> stock VM block identity. pg_buffercache shows current cache contents, where slots can be reused and the view is not one atomic snapshot.

Why visibility-map pages can contend

The visibility map stores two bits per heap page: all-visible and all-frozen, used by index-only scans and vacuum. With an 8-KiB block and 24-byte page header:

(8192 - 24) bytes * 8 bits / 2 bits per heap page
= 32672 heap pages per VM page
= 255.25 MiB of heap address space per VM page

Vacuum sets these bits. Data modifications can clear them. Sessions updating unrelated rows on different heap pages can therefore need exclusive access to the same VM buffer. One metadata page can serialize updates across many heap pages.

A related PostgreSQL change

A July 2026 correctness fix, authored by Melanie Plageman and Andres Freund and included in PostgreSQL 17.11, registers modified VM blocks in heap WAL records. Previously, incremental backups could omit those changes, and clearing VM bits did not produce the full-page images needed for torn-page protection.

Correct logging requires holding the VM lock through heap/VM modification and WAL insertion, extending the critical section.

In the performance discussion, Melanie tested 32 clients clearing bits for different heap pages covered by one VM page, while a vacuum loop set bits again. She observed no degradation with mostly default settings, but 10-15% lower TPS and more BufferContent waits with synchronous_commit=off. Vacuum timing varied between runs.

Her test illustrates how a longer VM critical section becomes visible as BufferContent contention when commit waits are removed.

Where this test ends

Here is the full-run record, linked back to the decisions behind it:

Configuration NOPM Leading active-client observation
Premium SSD v2, normal HA 50,648 DataFileRead, 79.6%
Ultra primary, normal HA 55,784 WALWrite, 83.18%
Ultra primary, commit local 54,556 WALWrite, 87.29%
Ultra primary, commit off 66,096 BufferContent, 93.92%
Full normal-HA repeat 54,163 WALWrite, 83.17%

The off result is 1.305x the original baseline, a 30.5% increase. The primary used the highest nonbursting Ultra budget documented for that tested VM configuration, while commits no longer waited for local or remote durability.

That result combined higher storage provisioning, weaker commit durability, and temporarily paused automatic failover. The runs reused the dataset, with cache contents, dirty pages, checkpoint phase, and vacuum state evolving between tests.

Mean primary CPU utilization in the off window was 18.1%. Low aggregate CPU utilization can coexist with serialization around a few lock holders. Page-level attribution was collected in the normal-on follow-up and identified stock visibility-map pages.

The progression is from a device-read constraint, through a check of remote commit waiting, into OS writeback admission, and finally to shared-page coordination. Removing replica acknowledgment waiting left WALWrite dominant. Each step narrowed where to investigate next.

At the last boundary, more disk IOPS does not make a content lock grant conflicting access simultaneously or split one shared metadata page into independently locked pieces. More workers can instead add contention. The remaining question concerns the amount of shared work, the duration of critical sections, and how access is coordinated.

Keep durability, move the work

The storage and commit-mode experiments took this workload from 50,648 to 66,096 NOPM, a 30.5% increase. The highest result allowed recent transaction loss after a crash. The PostgreSQL primary still had to coordinate access to shared pages.

Bypassing commit durability was a diagnostic step. The production objective is to preserve durable commits while reducing the work and coordination concentrated on the writer.

One approach is disaggregated PostgreSQL storage. Dedicated services handle WAL durability, replication, and data-page materialization. Moving that persistence work can reduce compute-side writeback and buffer maintenance. That's the HorizonDB architecture, which achieved 8x higher throughput for this workload, with better resilience to failures.

SQL execution, transaction semantics, and page-content synchronization remain in PostgreSQL compute. The writer has more headroom for execution and scale-up, while storage services can scale independently across machines. The important change is where the work runs, with the durability guarantee preserved.

HorizonDB retains PostgreSQL’s foreground page synchronization and visibility-map maintenance, while offloading page persistence, checkpoint I/O, full-page-write protection, and WAL distribution. This can ease pressure on those critical sections and give the writer more capacity for transactions. That may explain why the workload achieves higher throughput than this PostgreSQL configuration can support.

If you have resolved similar WAL or BufferContent contention on PostgreSQL, I'd be interested in the configuration, schema, or implementation change, and the measurements that showed improvement, while retaining durable commits.

Fast Disks, Slow WAL Sync: Finding Linux Writeback Throttling

As described in the previous post, the primary now uses a high-I/O Ultra disk. Reads and writes took fractions of a millisecond, yet PostgreSQL WAL synchronization averaged 25-30 ms. Removing synchronous-standby waits didn't help: synchronous_commit=local still requires local WAL durability. With device-read constraints reduced and remote acknowledgments excluded, I traced the primary's synchronization call across the guest-OS boundary.

To investigate those durable-commit timings, I returned to synchronous_commit=on with normal HA for a full repeat, then used a separate pilot prepared for controller tracing.

Measure the synchronization, not only the disk

The relevant cumulative PostgreSQL counters are:

SELECT wal_write, wal_sync, wal_write_time, wal_sync_time, stats_reset
FROM pg_stat_wal;

With track_wal_io_timing=on, the interval mean is:

mean WAL sync ms = delta(wal_sync_time) / delta(wal_sync)

The full runs showed:

Run NOPM Mean WAL sync ms WALWrite observation share
Ultra, normal synchronous commit 55,784 25.443 83.18%
Ultra, local commit 54,556 29.550 87.29%
Full normal-HA repeat 54,163 24.995 83.17%

For example, the repeat's in-window counter endpoints contained 45,266 additional syncs and 1,131,416.247 additional milliseconds of synchronization time. Their ratio is 24.995 ms.

The repeat was measured on November 8, 2026, 10:26:56-11:11:58 UTC with the same 512-worker, 15+45-minute protocol. These durable-regime traces and the later pilot were collected after the asynchronous-commit experiment described in part 4.

Why request completion and fdatasync disagree

fdatasync must synchronize a file's data. With buffered I/O, that can involve preparing dirty pages, submitting writeback, waiting for previously submitted work, and completing the necessary persistence operations. The entire call can therefore include:

PostgreSQL calls fdatasync
    dirty-page writeback and submission
    admission waits before block dispatch
    issued requests and their completion
    remaining file-synchronization work
fdatasync returns

The application call includes work before device dispatch. PostgreSQL can queue other backends behind its WAL write lock while the holder is delayed in that earlier work. I traced the caller to locate the delay.

The captured delay was before dispatch

A 30-second capture during the full normal-on repeat, 10:48:02-10:48:33 UTC, observed 536 completed fdatasync calls, averaging 24.504 ms, in PostgreSQL processes other than the separately classified WAL writer and checkpointer.

Captured sleep/rescheduling time inside those calls was classified as:

Captured stack location Share
Pre-dispatch WBT admission 96.43%
File-writeback completion 3.36%
Block-flush completion 0.20%

An abbreviated stack, from the sleeping function towards its callers, was:

io_schedule
rq_qos_wait
wbt_wait
__rq_qos_throttle
blk_mq_submit_bio
...
ext4_do_writepages

WBT is Linux block-layer writeback throttling. Its admission rules can make a submitting process sleep before work reaches the device. Fast completion of admitted requests does not measure that earlier wait.

These percentages describe captured sleep/rescheduling inside PostgreSQL fdatasync calls, including the delay until a process runs again. The syscall trace covers PostgreSQL synchronization calls broadly, while pg_stat_wal measures WAL synchronization specifically.

A healthy latency controller can still have admission waits

WBT controls write admission using read-latency feedback and write-depth limits. A separate normal-on pilot prepared the instrumentation before launch. Its five short measured captures showed:

Observation Value
PostgreSQL fdatasync calls 867
Mean syscall elapsed time 26.123 ms
WBT share of captured sleep/rescheduling 96.404%
Controller timer decisions 482, all LAT_OK
Observed read minima / target 0.100-0.213 ms / 2 ms
Maximum / normal / background admission limits 191 / 96 / 48

Read latency was healthy in the captured windows while callers still slept in the admission machinery. The captures recorded controller limits and aggregate wait paths. Slow synchronization was also observed before and after the probes ran.

The running kernel was 7.0.0-1017-azure. The upstream WBT implementation and request-QoS implementation explain the policy.

What this establishes

In the captured durable regime, much of the synchronization delay was in Linux's write-admission path, before device dispatch.

Provisioning more disk IOPS leaves the guest-OS admission rule unchanged. The relevant question has changed from the disk's advertised capacity to the caller's progress through writeback. All captures used the original WBT policy.

HorizonDB delegates durable WAL handling and data-page persistence to dedicated storage services. This replaces the compute-local data/WAL writeback arrangement, where we observed Linux throttling, while retaining durable commits. This explains why the workload achieves higher throughput than this PostgreSQL configuration can support.

The lab also tested asynchronous commit, where background persistence continues while commits can return before WAL is durable. Part 4 returns to that result and follows BufferContent into page identification.

Faster SSDs for PostgreSQL: From Read I/O to WAL Waits

The baseline in the previous post reached 50,648 NOPM, with most active-client observations at IO:DataFileRead. To understand the next bottleneck, I first increased the storage budget, then checked the next boundary in the commit path: acknowledgment from another availability zone. The objective was to distinguish device capacity from waiting higher in the path.

The 95,000-warehouse dataset, 512 workers, 15-minute ramp-up, and 45-minute measurement remained. The primary still had 64 vCPUs, 512 GiB RAM, and 128 GiB of shared buffers.

What changed on the primary

I changed storage only on the primary because this step targeted the benchmark's foreground read I/O (IO:DataFileRead). In this HammerDB TPROC-C run, all transactions, including their SELECT statements, used the primary endpoint. The workload did not offload reads to replicas. Doing so would require changing the tested execution and checking its consistency assumptions.

I migrated the existing data to an Ultra disk, which supports up to 400K IOPS and 10 GB/s throughput at low latency. The IO limits are then determined by the non-bursting E64bds_v5 with NVMe: 294,800 IOPS and 7,800 MB/s throughput.

Original primary (previous post) Higher-I/O primary (this post)
VM E64ds_v5 E64bds_v5, NVMe
CPU / memory 64 vCPUs / 512 GiB 64 vCPUs / 512 GiB
Data disk Premium SSD v2 Ultra Disk
Capacity 20,480 GiB, or 20 TiB 20,480 GiB, or 20 TiB
Provisioned IOPS 80,000 294,800
Provisioned bandwidth 1,200 MB/s 7,800 MB/s
Filesystem ext4 ext4

Both data and WAL were on that remote disk.

The two replicas retained their Premium SSD v2 disks at 80,000 IOPS / 1,200 MB/s, as my goal was only to accelerate the reads on the primary. Each benchmark used the same-zone D64ds_v5 client with 64 vCPUs / 256 GiB. The standbys still performed I/O to receive, flush, and replay WAL. Their storage could affect synchronous commit. The local-commit diagnostic below separates their acknowledgments from the primary's own synchronization.

Keep in mind that the usable budget depends on both the disk and the VM. Ultra Disks support up to 400,000 IOPS / 10,000 MB/s, but the E64bds_v5 NVMe configuration lists 294,800 IOPS / 7,800 MB/s for Ultra Disk and Premium SSD v2 without bursting. I provisioned that budget. Ultra Disk has been generally available since August 2019 and can be used for self-managed PostgreSQL on Azure VMs. They are now generally available also for Azure Database for PostgreSQL.

Premium SSD v2: waiting for reads

The first result covered in the previous post in this series was 50,648 NOPM (new orders per minute). During measurement, the primary data disk delivered approximately 51,130 read + 30,469 write IOPS, with 429 MiB/s reads + 412 MiB/s writes. Its guest busy-time counter was approximately 100%, and mean read/write completion times were 7.98 / 10.95 ms. The active-backend sample observations were:

Wait Share
IO:DataFileRead 79.6%
LWLock:WALWrite 10.8%
IPC:SyncRep 5.2%

That is the read-I/O-heavy baseline from part 1, with guest IOPS close to the provisioned 80,000-IOPS budget. The guest counters describe device activity. The wait percentages are shares of sampled active-client observations.

Ultra Disk: faster reads, then WALWrite

After moving the primary to Ultra, I got 55,784 NOPM, a 10.1% increase. Read/write completion times were about 0.31 / 0.23 ms. DataFileRead dropped to 2.56% of active-backend observations. But the wait mix had changed:

Wait Share
LWLock:WALWrite 83.18%
IPC:SyncRep 7.53%
IO:DataFileRead 2.56%

WALWrite was now the dominant wait, and that's why it doesn't fully benefit from the faster disks.

The Ultra primary delivered about 81,221 combined read/write IOPS and 976 MB/s, below its provisioned 294,800 IOPS / 7,800 MB/s. Guest busy time was 99.95%: I/O was almost continuous while the totals remained below the provisioned limits.

WALWrite means a backend is waiting for PostgreSQL's WAL write lock. The holder may be performing writes or waiting for synchronization. A long wait can therefore originate above device execution, even when admitted requests complete quickly.

Faster standby storage could reduce acknowledgment time, but that was a separate part of the path. Before attributing the remaining limit to cross-zone synchronization, I removed that wait from the diagnostic.

Remove the standby wait, for the investigation

The replicas were still on Premium SSD v2. I used session-level synchronous_commit set to local to remove their required acknowledgments from the commit path while retaining local durability and streaming replication. In the benchmark, the override was supplied through the workload's PGOPTIONS, so the worker connections inherited the setting.

Patroni automatic failover was separately paused for this bounded diagnostic. It's a lab to investigate what happens, not a production setup.

The result was that the throughput didn't increase, with 54,556 NOPM, versus 55,784 in the preceding run:

Active-client observation Ultra, commit on Ultra, commit local
LWLock:WALWrite 83.18% 87.29%
IPC:SyncRep 7.53% 0 observed
IO:DataFileRead 2.56% 3.93%

This change tested whether waiting for standby acknowledgments was holding back the workload. SyncRep disappeared from the samples, but throughput stayed near 55,000 NOPM and WALWrite remained dominant. Both standbys continued streaming, with a lag.

The name local explains why. It still waits for local durable WAL flushing. The WAL synchronization counter averaged 25.44 ms in the normal-HA run and 29.55 ms in the local-commit run.

The PostgreSQL commit path flushes local WAL before waiting for synchronous replication. WALWriteLock is released before the standby-acknowledgment wait reported as IPC:SyncRep. SyncRepLock protects the shared queue and state updates and is released while the backend waits on its latch. Group commit can make several transactions durable with a single synchronization.

What faster storage changed

The initial read waits fell substantially. WALWrite remained prominent after acknowledgment waits were removed. The investigation had moved from the device-read budget to what happened inside the primary's write/flush path.

These runs reused the dataset. Cache contents, dirty pages, checkpoint phase, and vacuum state evolved between tests. What we have proved is that even with the fastest disk and the lowest replication latency, we gained only 10%, far from the 8x throughput observed in HorizonDB.

HorizonDB maintains cross-zone durability using its shared WAL service, rather than waiting for a PostgreSQL standby to flush its private WAL disk. WAL replication happens outside the SQL writer, which explains why the workload achieves higher throughput than this PostgreSQL configuration can support.

I next traced the lock holder's path to understand why local synchronization averaged tens of milliseconds while device requests completed in fractions of a millisecond. Part 3 follows that delay into the Linux writeback path.

A 9-TiB OLTP Benchmark on PostgreSQL

At Microsoft Ignite, HorizonDB was announced to be 8x faster than traditional PostgreSQL, based on an I/O-bound HammerDB TPROC-C workload that showed 427,770 NOPM without specific tuning, 8x the 52,167 NOPM measured with self-managed PostgreSQL on the same compute.

I observed the same throughput on HorizonDB and reproduced the PostgreSQL side with a tuned HA configuration: 128 GB of shared buffers on huge pages, two standbys, and synchronous commit to at least one. The result was 50,648 NOPM, very close to the published number. But the interesting question is not the number itself. What is PostgreSQL waiting for at that throughput, and what happens when that constraint is removed? This post identifies the first bottleneck. The following three remove the bottlenecks one by one until the limit moves inside PostgreSQL itself.

The workload

Here is the definition of the benchmark workload I've run:

Setting Tested value
HammerDB 6.1.1, direct CLI
Workload PostgreSQL TPROC-C, function-based driver
Dataset 95,000 warehouses, nonpartitioned
Reported size after loading About 8.92 TiB
Workload concurrency 512 virtual users, plus the monitor
Ramp-up / measurement 15 / 45 minutes
Warehouse selection All warehouses enabled
Keying and thinking time Disabled
Connection pooling / time profiling Disabled
Driver-requested vacuum Disabled, while server autovacuum remained enabled
Transaction counter Timestamped samples every 10 seconds

HammerDB reports new orders per minute (NOPM) and PostgreSQL transactions per minute (TPM). These are the relevant driver choices, apart from connection details:

dbset db pg
dbset bm TPC-C
diset tpcc pg_count_ware 95000
diset tpcc pg_storedprocs false
diset tpcc pg_partition false
diset tpcc pg_driver timed
diset tpcc pg_rampup 15
diset tpcc pg_duration 45
diset tpcc pg_allwarehouse true
diset tpcc pg_keyandthink false
diset tpcc pg_connect_pool false
diset tpcc pg_timeprofile false
diset tpcc pg_vacuum false
vuset vu 512

The database was loaded before the timed test (with some pain, as described in a previous blog post, because loading nine terabytes takes a long time and HammerDB can't resume a failed load).

A dataset larger than the caches

The primary had 512 GiB of RAM, including 128 GiB of shared buffers. Even the machine's total memory was a small fraction of the database size. This is how we define an I/O bound workload: the working set is larger than any cache. All warehouses were enabled, and the measured read waits and sustained physical I/O showed frequent reads beyond the caches.

The baseline machines and durability

The initial deployment used three PostgreSQL VMs across three Azure availability zones:

  • Each database VM: E64ds_v5, 64 vCPUs / 512 GiB RAM.
  • Each data disk: 20,480 GiB Premium SSD v2, provisioned at 80,000 IOPS / 1,200 MB/s.
  • Data and WAL: the same ext4 filesystem.
  • Client in the primary's zone: D64ds_v5, 64 vCPUs / 256 GiB RAM.
  • PostgreSQL 17.11, Patroni 4.1.5, Ubuntu 24.04.
  • Three separate etcd members for the distributed configuration store.

This configuration used synchronous_commit=on, fsync=on and full_page_writes=on. A commit required local WAL synchronization and durable WAL acknowledgment from at least one of the two cross-zone standbys. Azure's torn-write prevention requires a supported NVMe controller, kernel and filesystem, with the application issuing aligned atomic direct writes. This run used PostgreSQL 17's ordinary buffered writes, so full-page logging remained enabled for torn-page recovery.

The standby acknowledgment confirms a flush, not replay. Network transport, standby storage, and acknowledgment processing can all contribute to the wait.

Continuous WAL archiving was disabled for this lab.

The PostgreSQL profile

I used the following PostgreSQL profile for the baseline and the later tests. The commit-mode overrides are described in the posts where they apply:

Area Important settings
Memory shared_buffers=128GB, work_mem=16MB, effective_cache_size=352GB
Shared-memory backing Explicit 2-MiB huge pages, huge_pages=on
Connections max_connections=640
WAL wal_compression=zstd, wal_buffers=64MB, max_wal_size=64GB
Synchronization wal_sync_method=fdatasync, fsync=on, full_page_writes=on
Checkpoints checkpoint_timeout=30min, checkpoint_completion_target=0.9
Planner / I/O random_page_cost=1.5, effective_io_concurrency=64, jit=off
Query parallelism max_parallel_workers_per_gather=0
Autovacuum 8 workers, 10-second naptime, vacuum scale factor 0.01
Autovacuum cost control Limit 3000, delay 2 ms
Monitoring track_io_timing=on, track_wal_io_timing=on

effective_cache_size is a planner estimate. WAL volume can trigger checkpoints before the 30-minute timeout.

Result: 50,648 NOPM (New Orders per Minute)

The measured interval ran from November 6, 2026, 23:21:13 UTC to November 7, 00:06:15 UTC. The result message was:

TEST RESULT : System achieved 50648 NOPM from 116618 PostgreSQL TPM

To observe active backends, the collector used grouped pg_stat_activity data:

SELECT count(*) AS sessions,
       wait_event_type, wait_event
FROM pg_stat_activity
WHERE backend_type = 'client backend'
  AND state = 'active'
GROUP BY wait_event_type, wait_event
ORDER BY sessions DESC;

Repeated measured-window samples gave:

Wait Active-client observation share
IO:DataFileRead 79.6%
LWLock:WALWrite 10.8%
IPC:SyncRep 5.2%

DataFileRead means PostgreSQL is waiting for a relation-file read, which may be served by the operating-system cache or the device. The guest disk counters confirmed substantial physical read activity during this run:

Primary data-disk observation Value
Read / write IOPS Approximately 51,130 / 30,469
Read / write bandwidth Approximately 429 / 412 MiB/s
Mean read / write request completion 7.98 / 10.95 ms
Guest busy time Approximately 100%

Delivered IOPS were near the provisioned disk budget. Together, the read waits, I/O rate, and request latency put the device-read path first in the investigation.

Next: change the storage constraint

HorizonDB combines shared buffers and local NVMe caching with a distributed page-storage fleet. Cache misses are served by this fleet, rather than the single provisioned data disk used in this test. This explains why the workload achieves higher throughput than this PostgreSQL configuration can support.

On PostgreSQL, increasing the storage budget would let me observe how the remaining waits changed. LWLock:WALWrite already accounted for 10.8% of active-client observations and could become the next bottleneck as read waits decreased.

In part 2, Faster SSDs for PostgreSQL: From Read I/O to WAL Waits
, I move the primary to Ultra SSD and then remove standby acknowledgment waiting to examine the primary's own path. In this series, the goal is to identify what limits PostgreSQL throughput as these constraints are reduced or bypassed. The investigation starts with disk reads, checks remote commit acknowledgments, follows local synchronization into the OS, and ends with shared-page coordination inside PostgreSQL.

Build an AI-powered DBA assistant for Amazon Aurora and Amazon RDS with Kiro

Learn how to build a DBA assistant with Kiro that investigates and resolves Amazon Aurora PostgreSQL performance incidents through natural language. Using two MCP servers, the agent reads Amazon CloudWatch metrics and Database Insights, runs diagnostic queries, and surfaces fixes you apply through your change management process.

How VillageSQL does documentation

How VillageSQL writes its docs: a public repo, an AI agent that runs every example it writes on a live server, and a person who approves.

Connect Your AI Agents to CedarDB Cloud

When we launched CedarDB Cloud, we teased a dedicated space to give your agents safe access to your data.

Today we’re launching CedarDB Cloud’s integrated MCP server. Connect Claude, Claude Code, or ChatGPT once, and ask about your data, build dashboards, or write data-driven applications in plain language directly from your web browser.

Your agents are most impactful when they get access and can react to your real data. But that’s also where a wrong query, prompt injection, or hallucination does the most damage. So we built the integration around a simple rule:

Your agents get their own database user

… and we don’t hide that user from you.

Every request your agent sends is evaluated as if it was sent by that user. By default, this user has access to nothing. On your project’s Agents tab, you can decide per table what it may do: No access at all, read, or write.

CedarDB checks these permissions on every single query. No prompt, injected or not, can talk an agent to do something you don’t want it to do.

Need finer control? The agent user is a regular database role called cedar_mcp, so you can use GRANT to set exactly what it may do, and row-level security to limit which rows it sees. For example, you can let agents file support tickets without reading anyone else’s, or only show them orders from one region:

-- Agents can add tickets, but not read existing ones
REVOKE ALL ON support_tickets FROM cedar_mcp;
GRANT INSERT ON support_tickets TO cedar_mcp;

-- Agents only see orders from the EU
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;
CREATE POLICY agents_eu_only ON orders TO cedar_mcp USING (region = 'EU');

And if you want agents gone, switch them off for the project or disconnect a single client.

What Your Agents Can Do

Answer questions. Agents find their way around an unfamiliar schema and answer questions that would take you several queries.

Build dashboards. Claude turns query results into charts and interactive dashboards you can share, and they stay connected to your live data.

Write code against your schema. Claude Code writes application code, migrations, and tests that match your actual tables, and checks its queries against your data while it works.

Get started

  1. On your project’s Agents tab, select Switch on and choose the tables agents may access.
  2. Add https://api.app.cedardb.cloud/mcp as a connector in Claude, or as an MCP server in Claude Code or ChatGPT.
  3. Sign in with your CedarDB Cloud account, allow the connection, and start asking.

Our docs walk you through each client step by step.

New to CedarDB Cloud? Your first project is free for 30 days.

Introducing Insights Agent

Insights Agent connect to your codebase to turn slow queries and schema recommendations into pull requests.

November 16, 2026

PostgreSQL large objects: Best practices for Amazon Aurora and RDS

PostgreSQL large objects live outside your tables and are referenced only by an OID, so autovacuum never reclaims the orphans. This post shows how to monitor orphaned large objects on Amazon Aurora and Amazon RDS for PostgreSQL, run vacuumlo safely, set up lo_manage triggers, prevent OID exhaustion, and prepare for major version upgrades.

Inside Kvrocks

How Valkey/Redis data lives on RocksDB Based on my Percona Live 2026 talk “On-disk and Redis Compatible: Evaluating Kvrocks.” I will cover benchmarking Kvrocks in a separate post.   We all run Valkey or Redis for well-known reasons: all the data lives in RAM, and it’s fast. Storing data in RAM is of course expensive. … Continued

The post Inside Kvrocks appeared first on Percona.

November 15, 2026

November 13, 2026

Fix Amazon Aurora MySQL error 63955: Virtual index ID overflow

An Amazon Aurora MySQL cluster can exhaust its virtual index ID limit, after which any CREATE INDEX that includes a virtual column fails with error 63955 until you rebuild the databases. This post shows how to recognize the error, why backups cannot clear it, and how to run a logical dump-and-restore migration with monitoring to catch it next time.

Improve RDS Proxy multiplexing for PostgreSQL, Part 1: Common causes

RDS Proxy multiplexes many client connections across a small pool of database connections, but connection pinning, pool fragmentation, and uncommitted transactions can silently disable that benefit for PostgreSQL workloads. Part 1 shows how to recognize each condition with Amazon CloudWatch metrics and the fix that restores multiplexing.

Improve RDS Proxy multiplexing for PostgreSQL, Part 2: Diagnose and monitor

Metrics alone can't always pinpoint what is reducing RDS Proxy multiplexing for PostgreSQL. Part 2 shows how to use Amazon CloudWatch Logs Insights with enhanced proxy logs to find the root cause, what healthy multiplexing looks like in metrics and logs, and how to set up CloudWatch alarms that catch regression.

Will your MongoDB upgrade break? Check it with UCFM

You were assigned one of the most common, yet feared, tasks in the life of a database administrator: Upgrades are needed for many reasons, whether to fix an existing bug or to keep the environment under support and eligible to receive security fixes. Either way, the task is yours. The binary swap is the easy … Continued

The post Will your MongoDB upgrade break? Check it with UCFM appeared first on Percona.

Village News: MySQL News + Events (13 November 2026)

Welcome back to Village News, our curated roundup of MySQL and database news. This week, PostgreSQL shipped updates fixing 52 security vulnerabilities and retired PostgreSQL 14, and the FOSDEM M-Ecosystem devroom opened its call for participation.

If you want to get these updates, just subscribe to the blog.

Enjoy!

Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights

We all know that when using a cluster based on Galera (Codership) such as Percona XtraDB Cluster (PXC), it is recommended to deploy an odd number of nodes. The generic answer to this recommendation is that otherwise the risk of having an inconsistent cluster (like split-brain) is higher.  (see Get started with Percona XtraDB Cluster … Continued

The post Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights appeared first on Percona.

November 12, 2026

Introducing Percona Database Service (PDS): What Happens When an Engineer Just Wants to Try Something

It’s Thursday afternoon, and a developer has an idea. Nothing complicated. They are working on a new application feature and want to test an idea before committing to a larger implementation. A quick prototype is already running locally. So far, so good. Next, they want to deploy the application to a shared environment so their … Continued

The post Introducing Percona Database Service (PDS): What Happens When an Engineer Just Wants to Try Something appeared first on Percona.

Build a serverless recipe-sharing API with Go and Amazon Aurora DSQL

Build a fully serverless recipe-sharing REST API in Go on Amazon Aurora DSQL, AWS Lambda, and Amazon API Gateway. Learn UUID primary keys for even write distribution, short-lived IAM token authentication, application-layer referential integrity, and automatic optimistic-concurrency retries, with a one-command CloudFormation deployment.

I asked Convex's CTO 34 spicy questions

I sat down with James Cowling, co-founder and CTO of Convex, and fired 34 spicy questions at him. These are the answers I keep coming back to, on learning to code, junior devs, vibe coding a production database, code review, hiring and build vs buy.

Deploying OpenEverest on a 3-Node K3s Cluster

Part 1 of 2: From bare Ubuntu servers to a running MySQL cluster OpenEverest is an open source platform for running databases such as MySQL, PostgreSQL and MongoDB on Kubernetes. You get a web console and an API, and Kubernetes operators handle provisioning, scaling, backups and failover behind the scenes. This post builds a small … Continued

The post Deploying OpenEverest on a 3-Node K3s Cluster appeared first on Percona.

Inside an OpenEverest MySQL Cluster: Galera Health and a Live Failover Test

Part 2 of 2: Reading wsrep status, killing the writer, and proving zero data loss In Part 1 we built a 3-node K3s cluster, installed OpenEverest, and created a 3-node Percona XtraDB Cluster (PXC) named mysql-vjg. In this post we look inside that cluster. Section A explains how to read Galera’s wsrep_* status variables, so … Continued

The post Inside an OpenEverest MySQL Cluster: Galera Health and a Live Failover Test appeared first on Percona.

Announcing Hegel-clj

I’ve just released hegel-clj, a Clojure interface for the Hegel generative testing library. You can jump right to the docs, but I thought it might be interesting to explain why I want another testing library.

I do a lot of generative testing in Clojure with the excellent test.check. However, using test.check can be frustrating! In particular, complex generators can get stuck during shrinking, leaving me trying to prune down a complex data structure by hand. The culprit is usually bind. If you’re not familiar with test.check, Quickcheck, or other libraries in this family, bind is a generator which draws some random value, then uses that to construct a new generator, which in turn is used to draw some other value.

(require '[clojure.test.check :as tc]
         '[clojure.test.check.generators :as tcg]
         '[clojure.test.check.properties :as tcp])
(tcg/bind ; Take a number from `nat`, a generator of small positive
          ; integers.
          tcg/nat
          ; Now with that number, construct a generator of vectors, each
          ; with that many elements.
          (fn [size]
            (tcg/vector tcg/large-integer size)))

We can also write this using test.check’s let macro, which expands to a series of nested binds under the hood.

(tcg/let [size tcg/nat]
  (tcg/vector tcg/large-integer size))

This leads to the canonical example of poor shrinking in test.check. Let’s try to find a vector containing 42:

(-> (tc/quick-check 1000
      (tcp/for-all [xs (tcg/let [size tcg/nat]
                         (tcg/vector tcg/large-integer size))]
                   (not-any? #{42} xs))
      {:seed 1777986545686})
  :shrunk :smallest first)

[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42 0]

That does contain 42, but this vector is enormous. Why are there all these zeroes? Once test.check found this vector, surely it could have shrunk it to just [42], right?

The answer has to do with shrink trees. Test.check generators don’t actually return values like 64. Instead they return a lazily evaluated tree of values: the root might be 64, and its children might be [32 48 ... 63], and each of those children has further children: 32 has children [16, 24, ... 31], and so on. Simplifying a bit, here’s how the tree for large integers is defined:

(defn- shrink-long
  "Given a number x, returns a coll of (x-x, x-x/2, x-x/4, x-x/8, ..., x-1)"
  [^long x]
  (core/let [size (- 64 (Long/numberOfLeadingZeros (Math/abs x)))
             ^longs vals (long-array size)]
    (loop [i 0 n x]
      (if (zero? n)
        vals
        (do
          (aset vals i (- x n))
          (recur (inc i) (quot n 2))))))))

So, if test.check finds that 64 violates some property we’re testing, it tries the first child of 64, 32. Let’s say that passes the test, so test.check tries the next child of 64: 48. If 48 fails, test.check tries its first child 24, and so on. By performing this depth-first search of the tree, and choosing the branches cleverly, test.check can often quickly shrink an integer to a smaller one that still fails the test.

A similar strategy applies to fixed-size vectors. The tree for a vector like [2 2] has children like [1 2] and [2 1], based on the trees for each individual element of the vector. This explains why test.check produced a vector like [0 0 0 ... 42 0]: it found something like [91 23 1034543 ... 42 15], then shrank the first element to zero, then the second, and so on. The only element it couldn’t shrink was the one that caused our property to fail: 42. Because our generator (tcg/vector tcg/large-integer size) always used a fixed size, the tree for that vector always contained size elements—in this case, 39.

Hang on! We didn’t just generate fixed-size vectors. We used bind to draw a size, then built a vector of that size:

(tcg/bind tcg/nat
          (fn [size]
            (tcg/vector tcg/large-integer size)))

What kind of trees does bind produce? The code involves some gnarly higher-order functions, but in essence bind’s trees can shrink in two ways. They can shrink the outer value (the size returned by tcg/nat), or they can shrink the inner value (the vector returned by (tcg/vector ...).

Now imagine bind draws a size of 100, then generates a vector of a hundred elements like [15 32 42 ...]. This contains 42, so test.check begins hunting through the shrink tree. One of the branches it can take shrinks the size parameter, starting with [50, 75, ... 99]. But if bind picks any of those sizes, the vector [15 32 42 ...] is useless—it’s the wrong size! So bind calls our function with 50 to construct a generator of 50-element vectors, and generates a single vector from that generator. This vector will have different random elements, like [69, 45, 1, ...].

If there’s no 42 in the vector of 50 elements, then test.check abandons that branch and moves on to size 75. It draws a new vector of 75 elements like [8 7 42 ...], and since that contains 42, it can continue shrinking. But since 42 is an uncommon integer, and test.check draws new elements for each size, it is very likely that most sizes will not produce a vector with 42, and test.check will get stuck with, say, 39 elements. From there it can shrink each element, producing the comically oversized…

[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42 0]

For this vector it’s fairly obvious what’s wrong. But for nontrivial test.check generators, there might be a dozen nested levels of bind with complex downstream consequences. Every one of them offers a place for test.check to get stuck, and the smallest failing case can be devilishly tricky to understand. Experienced test.check users try to avoid bind and let—but they’re fundamentally inescapable for many problems.

Over in the Python world, the Hypothesis team took a very different approach to shrinking. Test.check is fundamentally a two-phase system: first it asks a generator to produce a shrink tree. Then, if the root of that tree looks “bad”, it explores the rest of that tree. Instead, Hypothesis generators produce a single value, and if that value looks bad, they re-run the generator and ask it to produce something similar, but smaller. This approach, called internal test-case-reduction, is the foundation of Hegel.

Hegel-clj

Hegel is a new cross-platform library for generative testing. It provides deterministic generators which draw random values from an imperative, structured RNG, as well as a rich family of special values and shrinking tactics. Writing these generators and shrinkers is hard. The idea is that Hegel’s core library will do this hard work once, and simpler wrapper libraries will extend the benefits of that hard work across many languages.

I’ve been working for a few months on hegel-clj, which offers Clojure bindings for Hegel, and I think it’s ready for the folks to try it out. To test something, you call hegel-clj/test!, draw some values from generators, and throw an exception if the test fails. Hegel searches for values that make that body throw, then shrinks them.

(require '[hegel-clj [core :as h] [generator :as g]])
(h/test! {:seed 4}
  (assert (even? (h/draw! (g/integer)))))

{:passed? false
 :status :failed
 :statistics {:total 34
              :valid 5
              :invalid 0
              :overrun 18
              :interesting 11}
 :health-check-failed? false
 :failures ({:draws {"draw_1" 1}
             :exception
             #error {:cause "Assert failed: (even? (h/draw! (g/integer)))"
                     ...}})
 :reproduce-blob "AAEAAAAACgEAAAAB", :caveat nil}

The smallest case Hegel found involved a single draw of the number 1, which caused the assertion (even? 1) to throw.

Generators are pure, composable structures, just like in test.check. Here’s a vector of booleans:

(h/sample 5 (g/boolean))

[false false true true false]

Or a map representing a person, with a name generated by a regular expression and an integer age:

(h/sample 5 (g/hmap {:name (g/regex-str #"[A-Z][a-z]+")
                     :age  (g/integer 0 101)}))

[{:name "Aa", :age 0}
 {:name "Iyjdklppzxhqdnwmd", :age 89}
 {:name "Kslvqorm", :age 64}
 {:name "Fvyybe", :age 45}
 {:name "Puwppxqag", :age 4}]

The higher-order combinators are all there too. You can transform generated values with fmap:

(h/sample 5 (g/fmap keyword (g/string)))

[: :R :👍🏻 :True :çç¦ç¦򜝉ÏÏ򜝉¦¦Ï¦¦ç򜝉Ϧç򜝉çÏ򜝉çç¦]

Or build one generator out of another using bind:

(h/sample 2 (g/bind (fn [size]
                      (g/tuple (g/vector size (g/integer))
                               (g/vector size (g/boolean))))
                    (g/integer 0 3)))

[[[] []]
 [[144229 531621 -364 -1569883 8194954 -913 16 -745]
 [false true false true]]]

Like test.check, we offer a let macro which lets you bind both regular values or generators to variables; if the right-hand side is a generator, it draws a random value from it. And unlike test.check, these generators shrink well. Instead of thirty-nine elements, Hegel can find the much-shorter “bad” vector, [42].

(-> (h/test! {:test-cases 1000
              :seed       1777986545686}
      (h/let [size (g/integer {:min 0, :max 128})
              xs   (g/vector {:size size} (g/integer))]
        (assert (not-any? #{42} xs))))
    :failures first :draws (get "xs"))

[42]

With test.check, if you want to thread multiple values through bind or fmap, you have to build and destructure maps or tuples. Hegel-clj lets you lift arbitrary code into a generator using composite, which makes it easier to build complex generators:

(h/sample 2
  (g/composite
    (h/let [name      (g/string)
            cat-count (g/integer 0 2)
            cats      (g/vector {:size cat-count, :unique? true} (g/string))]
      {:name name     
       :cats cats})))

[{:name "",
  :cats []}
 {:name "𝐓𝐡𝐞 𝐪𝐮𝐢𝐜𝐤 𝐛𝐫𝐨𝐰𝐧 𝐟𝐨𝐱 𝐣𝐮𝐦𝐩𝐬 𝐨𝐯𝐞𝐫 𝐭𝐡𝐞 𝐥𝐚𝐳𝐲 𝐝𝐨𝐠",
  :cats ["🇺🇸" "👍🏻"]}]

You can build recursive generators with a promise-style API. The deferred generator provides a blank space for a generator, which can be filled in later:

(let [leaf (g/integer 0 9)
      tree (g/deferred)]
  (g/set-deferred! tree (g/one-of [leaf (g/vector tree)]))
  (mapv prn (h/sample 5 tree)))

0
[[1] 8 [2 []] 5 [] [[4 6 []]]]
[[1] 2]
3
0

And best of all, hegel-clj ships with easy-to-use integration for clojure.test, making generative tests as easy as example-based ones:

(require '[clojure.test :refer :all] '[hegel-clj.clojure-test :refer [with]])

(defn bad-add
  "Adds two integers via floating-point"
  [x y]
  (long (+ (float x) (float y))))

(deftest bad-add-test
  (with {:test-cases 100}
        [x (g/integer)
         y (g/integer)]
    (h/fprn :x x, :y y)
    (is (= (+ x y) (bad-add x y)))))
    
(bad-add-test)

:x 0 :y 16777217

FAIL in (bad-add-test) (form-init7301103820420620476.clj:6)
expected: (= (+ x y) (bad-add x y))
  actual: (not (= 16777217 16777216))

It’s hard to log debugging output with test.check—you’ll have to sift through thousands of test runs with complex inputs, and many of those runs won’t actually trigger the bug you’re searching for. Hegel has the notion of a “final” test case: after it’s found a counterexample, and shrunk it to a smaller version, it comes back and re-runs that case as a part of the final phase. We can use (when (h/final?) ...) to execute some debugging code only during these final cases, or fprn and finfo, which wrap clojure.core/prn and clojure.tools.logging/info respectively. I think there’s a bug in Hegel that causes it to run more final cases than it should, but it’s still a big step up.

Finally, test.check has been around for a while, and like many Clojure projects it has an exceptionally stable API. That’s great for compatibility, but it means some early choices are now fossilized. It’s (fmap gen f), but (bind f gen). If you want an integer you have to remember the difference between nat, int, pos-int, s-pos-int, small-integer, large-integer, large-integer*, etc. Common generators require keyword arguments, like (large-integer* {:min 0, :max 10}). Some generators like hash-map are only callable with varargs, which makes composition tricky. The properties API feels a tad fussy. And many of the ergonomic features you want, like generating strings from a regular expression, or integration with clojure.test, require an extra library like test.chuck. Test.check is really good, but with the benefit of hindsight and a fresh canvas, I’ve tried to design a more streamlined, consistent, and approachable API.

I’ve used test.check for over a decade now and it’s one of my favorite things about Clojure. I want to thank Reid Draper, Gary Fredericks, Alex Miller, and everyone else who’s contributed to the library. I’m also indebted to David MacIver, Liam DeVoe, and Esau Rodriguez from the Hegel team at Antithesis. They took time to talk me through Hegel’s design, answered endless questions about the protocol, and helped fix a whole slew of issues that we found. Y’all rock.

I’m pleased to announce hegel-clj 0.1.1, and I’m starting to use it to test my own projects. I hope it will be useful for you too. ❤️

November 11, 2026

Similarity Is Not Truth: Comparing vector distance, is_true(), and extract() in Azure AI for PostgreSQL

Vector search queries often use top-k similarity, assuming the highest-scoring results are true and can serve as factual validation. These scores represent distances between vectors (e.g., <=> for cosine distance in PostgreSQL with pgvector), so they should be used to retrieve relevant results—not as a final ranking of truth. Azure AI on HorizonDB can call an AI model to generate embeddings for comparison and provides more specific functions, such as is_true() or extract(), for structured responses.

Objective

In this blog post, I'll use the three of them on a small dataset, a table of countries, with the same goal: tell me if the country is an island or not.

  • <=> should return a very high score for countries that are islands and a very low score for those that are not, so I can try to determine a threshold that tells me the truth
  • is_true() should return true for countries that are islands and false for those that are not
  • extract() should extract the "is an island?" information as a boolean.

Create the table

I import a list of countries from a public resource


DROP TABLE IF EXISTS countries;
CREATE TABLE countries (
  country text primary key,
  region  text
);

\copy countries(country,region)
FROM PROGRAM 'curl -L "https://raw.githubusercontent.com/cs109/2014_data/master/countries.csv"' WITH (FORMAT csv, HEADER true);

SELECT * FROM countries WHERE region = 'OCEANIA' LIMIT 10;

Here is a sample:

     country      | region
------------------+---------
 Australia        | OCEANIA
 Fiji             | OCEANIA
 Kiribati         | OCEANIA
 Marshall Islands | OCEANIA
 Micronesia       | OCEANIA
 Nauru            | OCEANIA
 New Zealand      | OCEANIA
 Palau            | OCEANIA
 Papua New Guinea | OCEANIA
 Samoa            | OCEANIA
(10 rows)

I generate embeddings from the country name, assuming that the model has enough knowledge about countries:


CREATE EXTENSION vector;
CREATE EXTENSION azure_ai;

ALTER TABLE countries
  ADD COLUMN embedding vector(1536);

UPDATE countries
SET embedding = azure_openai.create_embeddings(
  input => country,
  dimensions => 1536
)::vector(1536);

For a real application, you may provide more information, such as a geographical description of the country, rather than using the AI model as a source of knowledge, but I keep this simple for the demo.

Naive query

Here is a query that uses Azure AI functions to generate embeddings for 'is an island' to compare the vector distance with the country, return a true or false for the question 'is an island' with the country name, and extract an 'is an island' boolean:

SELECT country
    -- cosine distance to 'is an island'
  , azure_openai.create_embeddings(
    input => 'is an island',
    dimensions => 1536
  )::vector(1536) <=> embedding AS distance
    -- boolean answer to 'is an island'
  , azure_ai.is_true(
    statement => country ||' is an island'
) AS is_true_result
    -- extract 'is an island?' as a boolean, in addition to other questions
  , azure_ai.extract(
    country,
    ARRAY[
      'land: boolean - is an area of land',
      'water: boolean - is surrounded by water',
      'continent: boolean - is a continent',
      'island: boolean - is an island?',
      'neighbours: string - list the countries with common border'
    ]
  ) AS extracted
FROM countries
ORDER BY distance
;

I get this result:

             country              |      distance      | is_true_result |                                                                                                          extracted
----------------------------------+--------------------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Iceland                          | 0.5288325821583937 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Dominica                         |  0.583984959074203 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Marshall Islands                 | 0.5923968276529713 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Seychelles                       | 0.5933297352264011 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Solomon Islands                  | 0.5992036897764201 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Barbados                         | 0.6026798894629409 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Bahamas                          | 0.6037901963467236 | f              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Mauritius                        | 0.6124306023780004 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Jamaica                          | 0.6180866747206792 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Ireland                          | 0.6208333769699201 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": "United Kingdom (Northern Ireland)"}
 Grenada                          | 0.6314871259473416 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Palau                            | 0.6331963890581136 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Kiribati                         | 0.6340927146328686 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Cyprus                           | 0.6354914993816243 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Maldives                         | 0.6368475975326672 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Malta                            | 0.6370369487669678 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 New Zealand                      | 0.6520422783711599 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Antigua and Barbuda              | 0.6560949198973425 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Fiji                             | 0.6563765953128096 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Saint Lucia                      | 0.6565374364556272 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Saint Kitts and Nevis            | 0.6569025217334334 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Madagascar                       | 0.6579842393491677 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Nauru                            | 0.6588623575239401 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Belize                           | 0.6617206536439253 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Mexico, Guatemala"}
 Cape Verde                       | 0.6656987305382893 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Guyana                           | 0.6690943980832992 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Brazil, Suriname, Venezuela"}
 Micronesia                       | 0.6694775662174064 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Sri Lanka                        | 0.6755248175409809 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Andorra                          | 0.6756446018762446 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "France, Spain"}
 Israel                           | 0.6763307658990072 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Lebanon, Syria, Jordan, Egypt, Palestine"}
 Cuba                             | 0.6780035821480603 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Tonga                            | 0.6854439801282708 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Vanuatu                          | 0.6898198008050471 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Dominican Republic               | 0.6930591990187845 | f              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": "Haiti"}
 Saint Vincent and the Grenadines | 0.6936146574365081 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Singapore                        | 0.6939824470932374 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Samoa                            | 0.6942054226212125 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Haiti                            | 0.6970099016123408 | f              | {"land": true, "water": false, "island": true, "continent": false, "neighbours": "Dominican Republic"}
 Indonesia                        | 0.6982757826983539 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Malaysia, Papua New Guinea, Timor-Leste"}
 Bahrain                          | 0.7010005561641666 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Comoros                          | 0.7067622419634335 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Estonia                          | 0.7085149756741925 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Latvia, Russia"}
 Liechtenstein                    | 0.7098232671398083 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Austria, Switzerland"}
 Tuvalu                           | 0.7100676352558253 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Brunei                           |  0.710790435672632 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Malaysia"}
 San Marino                       | 0.7188084142495059 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Italy"}
 Vatican City                     | 0.7195959897300721 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Italy"}
 Suriname                         | 0.7227729682040444 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Brazil, Guyana, French Guiana"}
 Sao Tome and Principe            | 0.7228708359839435 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Guinea-Bissau                    | 0.7250435443354935 | f              | {"land": true, "water": true, "island": false, "continent": false, "neighbours": "Senegal, Guinea"}
 Trinidad and Tobago              | 0.7255906511994212 | t              | {"land": true, "water": true, "island": true, "continent": false, "neighbours": ""}
 Greece                           | 0.7294355944328467 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Albania, North Macedonia, Bulgaria, Turkey"}
 Qatar                            | 0.7311974837740045 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Saudi Arabia"}
 Bhutan                           | 0.7334687504057431 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "India, China"}
 Monaco                           | 0.7340602079110607 | f              | {"land": true, "water": true, "island": false, "continent": false, "neighbours": "France"}
 Iraq                             |  0.734913671407935 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Turkey, Iran, Kuwait, Saudi Arabia, Jordan, Syria"}
 Kuwait                           | 0.7369097472864523 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Iraq, Saudi Arabia"}
 Denmark                          | 0.7379886532695916 | f              | {"land": true, "water": true, "island": false, "continent": false, "neighbours": "Germany"}
 Slovenia                         | 0.7381978525807786 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Italy, Austria, Hungary, Croatia"}
 Liberia                          | 0.7392671571539429 | f              | {"land": true, "water": true, "island": false, "continent": false, "neighbours": "Sierra Leone, Guinea, Côte d'Ivoire"}
 Djibouti                         |  0.740412556074715 | f              | {"land": true, "water": true, "island": false, "continent": false, "neighbours": "Eritrea, Ethiopia, Somalia"}
 Oman                             | 0.7413224461946565 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Saudi Arabia, United Arab Emirates, Yemen"}
 Somalia                          | 0.7449227200702471 | f              | {"land": true, "water": false, "island": false, "continent": false, "neighbours": "Djibouti, Ethiopia, Kenya"}
 Gabon                            | 0.7456118851623608 | f              | ... (truncated)
                                    

Refresh a single tenant across accounts with Aurora MySQL and Amazon S3

Move a single tenant's data between Amazon Aurora MySQL clusters in two separate AWS accounts with no network route between them, using Aurora cloning, SELECT INTO OUTFILE S3, and LOAD DATA FROM S3. The clusters never connect, and one encrypted Amazon S3 staging bucket is their only shared touchpoint.

Resilient IAM database authentication with Amazon Aurora Global Database writer endpoint

Learn how to combine the Amazon Aurora Global Database writer endpoint with IAM database authentication so that passwordless access survives a cross-Region switchover or failover with no configuration changes. This walkthrough signs the token for the dynamically discovered primary Region and scopes the policy to every Regional cluster resource ID.

Building a multi-tenant CRM with Amazon Aurora DSQL

Learn how to design and implement a multi-tenant CRM on Amazon Aurora DSQL using the pooled tenancy model. Tenant isolation is enforced in a centralized data-access layer, with Aurora DSQL-specific design considerations, and a working proof of concept you can deploy on Amazon ECS Express Mode.

Ten ways a glibc change can break your PostgreSQL cluster

Between any two glibc versions, locales can change, and those changes alter the order of character string comparison (text/varchar/char/citext). It may also happen that locales are removed, such as en_US@ampm, present in RHEL8 and removed in RHEL9, and aa_ER@saaho, present in RHEL9 and then renamed to ssy_ER in RHEL10. Since PostgreSQL relies on locales to … Continued

The post Ten ways a glibc change can break your PostgreSQL cluster appeared first on Percona.

Build an Oracle AWR-style report for RDS/Aurora PostgreSQL

PostgreSQL has no built-in Oracle AWR equivalent. This post shows how to build an AWR-style, snapshot-based performance report for Amazon RDS for PostgreSQL and Amazon Aurora PostgreSQL using pg_stat_statements and pg_cron, producing top-SQL, database-activity, wait-event, and load-profile reports between any two snapshot IDs, entirely in SQL.

Help prevent out-of-memory restarts on Aurora MySQL 8.4

Out-of-memory restarts drop every connection and force an Aurora MySQL cluster restart. Learn how Aurora MySQL 8.4's automatic memory management prevents them: confirm protection is active, read the signals during a memory-pressure event, monitor OOM avoidance with CloudWatch, and decide when to take manual control.

Diagnose Amazon Aurora MySQL performance using CloudWatch Database Insights

When an Aurora MySQL or Amazon RDS for MySQL database slows down, it can be hard to tell why. This post shows how to read DB load in Amazon CloudWatch Database Insights, interpret the most common MySQL wait events, and turn each into a concrete action, so you can fix the real bottleneck instead of scaling up to a larger instance.

Automate Aurora zero-ETL recovery after global failover

After an Amazon Aurora Global Database switchover or failover, the Aurora zero-ETL integration with Amazon Redshift keeps referencing the former primary and stops ingesting. Learn how to automate integration recreation with Amazon EventBridge and AWS Lambda so your analytics pipeline recovers without manual intervention.

How we migrate TBs from Heroku without logical replication

We've equipped our customers with an open source tool that not only quickly transfers your data but also provides a replication-based experience, so that any user can migrate from Heroku without extended downtime or the need to purchase a third-party product.

November 10, 2026

Heavy MySQL Reports in Seconds: Open Source HTAP with DBTrail and Percona Server for MySQL

In Part 1 we ran analytical reports in DuckDB on a DBTrail copy instead of on MySQL. This time I want to look at a different question: what if the application does not need to know the copy exists? Most MySQL servers have a few queries that make someone wait. The monthly report that aggregates … Continued

The post Heavy MySQL Reports in Seconds: Open Source HTAP with DBTrail and Percona Server for MySQL appeared first on Percona.