Today we’re launching CedarDB Cloud, the fully managed way to run CedarDB, our high-performance, PostgreSQL-compatible database for both transactional and analytical workloads. Keep the drivers, tools, and SQL you already know, with all the versatility of a full data stack and none of the trade-offs.

Why CedarDB?

Why CedarDB? Because your application and your analytics shouldn’t need two databases. CedarDB handles all your transactions, dashboards, reports, and AI agents in a single engine, so every query always sees the freshest data. There is no reason to tolerate brittle ETL pipelines or stale copies of your data.

Built from scratch for modern hardware, it puts every core to work on every query, running mixed workloads orders of magnitude faster than PostgreSQL and beats purpose-built analytical systems at their own game.

What’s Included?

CedarDB Cloud delivers all of the functionality of CedarDB with the infrastructure managed for you. Let us take care of provisioning servers, tuning them, and managing upgrades and backups for you. Sign up, create a database, load your data, and connect your favorite tools with just a few clicks.

Fully Managed, Ready in Seconds. We handle provisioning, infrastructure, and day-to-day operations for you. Go from sign-up to a connection string in seconds, then manage and query your data in our focused UI.

Sized for Your Workload. Pick from nine instance types, starting at 2 vCPUs and scaling all the way up to 192 vCPUs and 768 GiB of memory to support even the most demanding production workloads. Storage scales independently of compute, so you always get the right fit.

PostgreSQL-Compatible. Keep your favorite drivers and tools, and migrate your existing PostgreSQL deployment with minimal downtime. Our cookbooks cover a wide range of migration paths, from fully managed transitions via AWS DMS and other change-data-capture tools to classic PostgreSQL tools.

Point-and-Click Management. Handle backups, user access, and more from our UI with just a few clicks, and see your instance’s current configuration at a glance.

Easy Monitoring. Keep an eye on your instance’s health and activity in our always up-to-date interactive charts and drill down on any activity using the interactive shell.

Get Started and See It in Action

1. Start an Instance

After signing in, start a new instance from the overview page by picking an instance size and name. CedarDB Cloud then takes over, and moments later you already have access to your own instance with a PostgreSQL connection string that you can point your existing PostgreSQL tools at right away.

2. Monitor Your Database

Once your instance is up, switch over to the Monitoring tab to see what is happening on it in real time. Track disk, network, and memory pressure to catch bottlenecks before they affect your applications, or dig into the raw logs for detailed output. To drill down further, the Sessions tab shows you who is running what on your database at any time.

3. Manage Your Deployment

Hop over to the Backup & Restore tab to save the current state of your database for a later restore. Then open the Users & Permissions tab to add users and manage their access with role-based access control, making sure each person and application can only access what it needs.

Pricing and Free Trial

CedarDB Cloud pricing is based on three things:

1. The instance size you choose
2. The storage you provision for your instance
3. Additional backups (three backups are included with every project)

You see the estimated monthly cost for any combination up front, before you create anything. To get an estimate right now, hop over to our pricing calculator. And the first month is free for selected instance sizes, so you can try CedarDB Cloud with your own workload before committing.

Looking for More?

Don’t worry, we’ve got plenty more lined up for CedarDB Cloud in the coming weeks and months. Watch this space for announcements on new management features, such as finer-grained connection controls, scheduled backups, schema management, and more. Rumor has it there might even be a dedicated space to give your agents safe and efficient access to your data.

In the archive post I moved cold rows out of a Postgres table into Parquet on Azure Blob and kept querying them. The steps were manual: export to Blob, verify the file, delete the rows, record what happened. If the process dies after the export but before the delete, you're stuck — re-run and you re-export (wasteful), skip ahead and you might delete rows that were never safely written (dangerous). That partial-failure gap is exactly what a durable workflow closes.

Azure Database for PostgreSQL ships another open-source Microsoft extension that fits here: pg_durable. It runs checkpointed, crash-safe workflows inside Postgres — define a graph of SQL steps, and the engine persists progress between them. If the database restarts or a step fails, execution resumes from the last completed step instead of starting over. No cron-plus-status-table-plus-worker scaffolding. This post turns the archive into one durable, schedulable workflow, with pg_duckdb doing the write to Blob.

Enabling pg_durable

Like pg_duckdb, pg_durable loads through shared_preload_libraries, so it needs allow-listing and a restart:

1. Add pg_durable to azure.extensions (the allow-list).
2. Add pg_durable to shared_preload_libraries (append to the current value, next to pg_duckdb) and restart the server.

Then create both extensions in the database:

CREATE EXTENSION IF NOT EXISTS pg_duckdb;
CREATE EXTENSION IF NOT EXISTS pg_durable;
SELECT duckdb.create_azure_secret('<blob connection string>');

pg_durable runs its steps in a background worker. Right after a restart that worker takes a moment to come up. The first df.start() can return pg_durable background worker not yet initialized — try again in a moment. Wait a few seconds and retry. A normal (non-superuser) login also needs a one-time grant:

SELECT df.grant_usage(current_user);

A quick smoke test confirms the engine runs:

postgres=> SELECT df.start('SELECT 1 AS ok')
;

 start
----------
 50a27f7d

(1 row)

postgres=> SELECT df.status('50a27f7d')
;

  status
-----------
 completed

(1 row)

postgres=> SELECT df.result('50a27f7d')
;

             result
--------------------------------
 {"rows": [{"ok": 1}], "row_count": 1}

(1 row)

The workflow status, errors, and results are persisted for each step and you can query them with pg_durable functions and tables.

The data to archive

A small orders table spanning three years — the 2023 rows are the cold slice we'll move out:

DROP TABLE IF EXISTS orders;
CREATE TABLE orders AS
SELECT g AS order_id,
       (DATE '2023-01-01' + ((g * 7) % 1095))::date AS order_date,
       ((g % 500) + 1)::numeric(10,2)              AS amount
FROM generate_series(1, 1000) g;

CREATE TABLE IF NOT EXISTS archive_log (
  id            serial,
  archived_year int,
  n_rows        bigint,
  blob          text,
  at            timestamptz DEFAULT now()
);

Here is the current data in this table:

postgres=> -- total number of rows
postgres=> SELECT count(*) FROM orders;
 count
-------
  1000

postgres=> -- the cold slice
postgres=> SELECT count(*) FROM orders 
           WHERE order_date < DATE '2024-01-01'
;
 count
-------
   364

I have a reproducible test case where I want to archive orders from before 2024. I'll create a one-time workflow for that and let it run.

The durable workflow

The whole archive is one df.start():

• steps are chained with ~>
• step's result is captured into a variable with |=> 'name'
• next step can reuse the result as $name:

SELECT df.start(
  -- 1. count the cold slice, capture it as $n
     $Step1$
        SELECT count(*) AS n FROM orders WHERE order_date < DATE '2024-01-01'
     $Step1$ |=> 'n'
  -- 2. write those rows to Blob as Parquet, through the DuckDB engine
  ~> $Step2$ 
        SELECT duckdb.raw_query(
        $DuckDB$
           COPY (
             SELECT order_id, order_date, amount
             FROM pgduckdb.public.orders
             WHERE order_date < DATE '2024-01-01'
           ) TO 'az://lake/archive/orders_2023.parquet' (FORMAT parquet)
        $DuckDB$
        )
     $Step2$
  -- 3. verify the file reads back, capture the row count as $v
  ~> $Step3$
        SELECT count(*) AS v
        FROM read_parquet('az://lake/archive/orders_2023.parquet')
     $Step3$ |=> 'v'
  -- 4. only now delete the cold rows from Postgres
  ~> $Step4$
        DELETE FROM orders WHERE order_date < DATE '2024-01-01'
     $Step4$
  -- 5. record what happened
  ~> $Step5$
        INSERT INTO archive_log(archived_year, n_rows, blob)
        VALUES (2023, $n, 'az://lake/archive/orders_2023.parquet')
     $Step5$,
  'durable-archive'   -- a label to find the run by
);

Two things about the write step (step 2), both carried over from the archive post:

• The blob write goes through duckdb.raw_query(), not plain COPY TO 'az://…' so that DuckDB authorizes the write according to its own secret and allow-list — the object-storage controls that PostgreSQL's server-file gate knows nothing about and would otherwise reject.
• Inside raw_query you're in DuckDB's catalog, so the table is pgduckdb.public.orders, not a bare orders.

And a word on the quoting: the steps are SQL statements declared as text arguments for df.start() and themselves include nested text. This would require multiple levels of double quotes in standard SQL. It is solved easily with PostgreSQL dollar quoting ($$ with an optional label inside).

Watching it run

df.start() returns immediately with an instance id. The worker executes the steps. You can check progress directly with the id or getting the last id with your label:

postgres=> SELECT instance_id, label, status, execution_count
           FROM df.list_instances()
           WHERE instance_id = (SELECT id FROM df.instances
                                WHERE label = 'durable-archive'
                                ORDER BY created_at DESC LIMIT 1)
;
 instance_id |      label      |  status   | execution_count
-------------+-----------------+-----------+-----------------
 b9b24177    | durable-archive | completed |               1

(1 row)

Each step is a node you can inspect — this is the checkpoint record:

postgres=> SELECT node_type, status, query
           FROM df.nodes
           WHERE instance_id = (SELECT id FROM df.instances
                                WHERE label = 'durable-archive'
                                ORDER BY created_at DESC LIMIT 1)
           ORDER BY created_at
;
 node_type |  status   |                                     query
-----------+-----------+-------------------------------------------------------------------------------
 SQL       | completed |                                                                              +
           |           |         SELECT count(*) AS n FROM orders WHERE order_date < DATE '2024-01-01'+
           |           |
 SQL       | completed |                                                                              +
           |           |         SELECT duckdb.raw_query(                                             +
           |           |         $DuckDB$                                                             +
           |           |            COPY (                                                            +
           |           |              SELECT order_id, order_date, amount                             +
           |           |              FROM pgduckdb.public.orders                                     +
           |           |              WHERE order_date < DATE '2024-01-01'                            +
           |           |            ) TO 'az://lake/archive/orders_2023.parquet' (FORMAT parquet)     +
           |           |         $DuckDB$                                                             +
           |           |         )                                                                    +
           |           |
 THEN      | completed |
 SQL       | completed |                                                                              +
           |           |         SELECT count(*) AS v                                                 +
           |           |         FROM read_parquet('az://lake/archive/orders_2023.parquet')           +
           |           |
 THEN      | completed |
 SQL       | completed |                                                                              +
           |           |         DELETE FROM orders WHERE order_date < DATE '2024-01-01'              +
           |           |
 THEN      | completed |
 SQL       | completed |                                                                              +
           |           |         INSERT INTO archive_log(archived_year, n_rows, blob)                 +
           |           |         VALUES (2023, $n, 'az://lake/archive/orders_2023.parquet')           +
           |           |
 THEN      | completed |
(9 rows)

Five SQL nodes, all completed — one per step: count, the raw_query COPY to Blob, the verify read, the DELETE, and the audit insert, in order.

It worked, end to end

As expected by the completed workflow, the archived file is in Blob that I can query though pg_duckdb functions:

postgres=> SELECT count(*) FROM read_parquet('az://lake/archive/orders_2023.parquet')
;
 count
-------
   364

(1 row)

The cold rows are gone from Postgres:

postgres=> SELECT count(*), min(order_date) FROM orders
;
 count |    min
-------+------------
   636 | 2024-01-01
(1 row)

The audit row is written in my custom table:

postgres=> SELECT archived_year, n_rows, blob FROM archive_log ORDER BY at
;
 archived_year | n_rows |                 blob
---------------+--------+---------------------------------------
          2023 |    364 | az://lake/archive/orders_2023.parquet

(1 row)

As we have seen in the previous post, history is still one query away, live rows unioned with the archived file, because pg_duckdb can run queries on heterogenous sources:

postgres=> SELECT yr, count(*) AS n
           FROM (
             SELECT extract(year FROM order_date)::int AS yr FROM orders
             UNION ALL
             SELECT extract(year FROM (r['order_date'])::date)::int
             FROM read_parquet('az://lake/archive/orders_2023.parquet') r
           ) u
           GROUP BY yr ORDER BY yr
;
  yr  |  n
------+-----
 2023 | 364
 2024 | 324
 2025 | 312

(3 rows)

2023 comes from the archived Blob file and 2024 and 2025 from the live Postgres table.

Why durable, not just a cron job

Steps 2→3→4 are export, verify, then delete — and because pg_durable checkpoints after each step, a crash between them is safe:

• Die after the export but before the delete, and the run resumes at the verify and delete steps on restart. The export is not repeated. The delete is not skipped.
• A failed step stops the workflow with its state intact in df.nodes, so you can see exactly where it stopped and why — no guessing, no half-done cleanup.

With a plain pg_cron + PL/pgSQL procedure you'd be rebuilding this yourself: a status table, retry counters, and a way to tell "exported but not yet deleted" apart from "nothing done yet." pg_durable keeps that bookkeeping inside Postgres.

Make it nightly

Wrap the same workflow in an eternal loop (@>) that waits on a cron schedule, and the archive runs on its own, forever, surviving restarts. Point the date window at a rolling boundary instead of a fixed year:

SELECT df.start(  
  @> (  
    $Step0$  
       SELECT (date_trunc('year', now()) - interval '1 year')::date AS cutoff  
    $Step0$ |=> 'cutoff'  
    ~> $Step1$  
         SELECT count(*) AS n FROM orders WHERE order_date < $cutoff  
       $Step1$ |=> 'n'   
    -- copy / verify 
    ~> $View$  
         CREATE OR REPLACE VIEW orders_all AS  
           SELECT order_id, order_date, amount FROM orders  
           WHERE order_date >= $cutoff          -- exclude rows about to be deleted  
           UNION ALL  
           SELECT order_id, order_date, amount  
           FROM read_parquet('az://lake/archive/orders_*.parquet')  
   $View$  
    -- delete / log steps, all referencing $cutoff ...  
    ~> df.wait_for_schedule('0 2 * * *')  
  ),  
  'nightly-archive'  
);

I added a step to (re)build a view that reads from the archives and adds the operational data. I'll explain how to query a partitioned data lake in a future post.

Takeaways

• pg_durable is another open-source extension on Azure Database for PostgreSQL, enabled like pg_duckdb (azure.extensions + shared_preload_libraries + restart), usable in your database database after CREATE EXTENSION pg_durable, and by your user after df.grant_usage(current_user).
• The archive becomes one df.start() graph: count → write to Blob (duckdb.raw_query) → verify → delete → log, checkpointed between steps.
• Durability closes the export-then-delete gap: a crash resumes at the next uncompleted step instead of re-exporting or deleting unsafely.
• @> + df.wait_for_schedule('0 2 * * *') turns it into a nightly job that survives restarts — no external scheduler, no status table.
• Everything is plain SQL with additional operators to define the workflow.
• Each step is a transaction, so intermediate results are durable — the workflow continues automatically after a failover.

You can visualize the workflow from VS Code with the PostgreSQL extension by Microsoft

Thread Pool in Percona Server and MySQL (Part 2) The previous Part 1 focused on explaining the idea of thread pool, configuration options and analysis of thread pool efficiency in Percona Server for MySQL 9.7.1-1 against the base line, which is the same server without thread pooling. We recommend reading the post with Part 1 … Continued

The post Thread Pool in Percona Server and MySQL (Part 2) appeared first on Percona.

Over the period, we have witnessed how MySQL versioning evolved from 8.0 to 9.7 LTS, and now it has entered a new era: a calendar-based model. Let’s first take a brief look at how MySQL versioning has evolved over time. Phase 1 – MySQL 8.0 (2018–2026) used a traditional single-track release model, with all new … Continued

The post Meet MySQL 26.7: A New Versioning Era Begins appeared first on Percona.

A Postgres table that only grows is a familiar problem: queries touch recent data, but old rows still consume storage and slow maintenance. With pg_duckdb, you can move cold data to Parquet in Azure Blob Storage and still query it — on its own and joined back to the live table — from the same connection. Here's the full cycle for a sales table: 10M rows over five years, with 2021 archived out.

Setup (same table as the previous post)

I'll use the same data as in the previous post DuckDB analytics on your PostgreSQL tables in Azure PostgreSQL:

SET duckdb.force_execution = false;

DROP TABLE IF EXISTS sales;
CREATE TABLE sales AS
SELECT g AS sale_id,
       (DATE '2021-01-01' + ((g * 7) % 1826))                       AS sale_date,
       1 + (g % 50000)  AS cust_id, 1 + (g % 2000) AS prod_id, 1 + (g % 5) AS channel_id,
       (ARRAY['US','CH','FR','DE','UK','JP','BR','IN'])[1 + (g % 8)] AS country,
       (ARRAY['Consumer','SMB','Enterprise'])[1 + (g % 3)]          AS segment,
       1 + (g % 10) AS qty, ((5 + (g % 500))::numeric(10,2))        AS amount
FROM generate_series(1, 10000000) g;
ANALYZE sales;

CREATE EXTENSION IF NOT EXISTS pg_duckdb;
SELECT duckdb.create_azure_secret('<blob connection string>');

I set force_execution = false, the default, in case you activated it when reproducing the previous post. The table needs to be built on the PostgreSQL executor. If DuckDB is enforced, it issues a warning: Binder Error: +(DATE, BIGINT) on this generate_series DDL and defaults back to Postgres.

Writing to Blob from inside Azure Database for PostgreSQL

One point up front, because it shapes the workflow. pg_duckdb can write Parquet to Azure Blob from inside the database — but on Azure Database for PostgreSQL the plain COPY … TO 'az://…' syntax is refused:

COPY (SELECT …) TO 'az://lake/archive/sales_2021.parquet' (FORMAT parquet);

--  ERROR: relative path not allowed for COPY to/from a file in Azure Database For PostgreSQL

On self-managed PostgreSQL, the pg_duckdb extension intercepts COPY ... TO 'az://...' and runs it in DuckDB for you. However, Azure ensures that extensions cannot bypass PostgreSQL security checks. It considers any path without a leading / as "relative", causing the az:// URL to be rejected. This occurs before the server requires superuser privileges or pg_write_server_files permissions for server-side file operations. Azure allows PostgreSQL extensions but limits their ability to bypass the managed environment's security controls.

The way through is duckdb.raw_query, which runs the COPY inside the embedded DuckDB engine. This isn't a way around the platform's guard — it's a write to a different resource, governed by a different control: the Blob write is governed instead by DuckDB's own secret and allow-list (duckdb.allowed_directories, here az://), so the write lands through the layer that actually owns that resource. That's the in-database export used below. If you'd rather not write from the database at all, a client-side export works too — see the alternative at the end of Step 2.

Step 1 — Identify the cold slice

I'll consider sales prior to 2022 as old data to be archived:

postgres=> SELECT count(*) 
           FROM sales
;
  count
----------
 10000000

postgres=> SELECT count(*) 
           FROM sales 
           WHERE sale_date < DATE '2022-01-01'
;
  count
---------
 1998938

We will retain just under 2 million rows from 2021 in the operational database and archive the older data.

Step 2 — Export to Parquet in Blob (in-database)

Hand the COPY to the DuckDB engine with duckdb.raw_query(), writing straight to Blob. Keep types faithful — the source amount is already numeric(10,2), so DuckDB writes it as DECIMAL(10,2) and reads it back natively:

postgres=> SELECT duckdb.raw_query($$
             COPY (
               SELECT sale_id, sale_date, cust_id, prod_id, channel_id,
                      country, segment, qty, amount
               FROM pgduckdb.public.sales
               WHERE sale_date < DATE '2022-01-01'
             ) TO 'az://lake/archive/sales_2021.parquet' (FORMAT parquet)
           $$)
;
 raw_query
-----------

(1 row)

Four things to know:

• Qualify the table as pgduckdb.public.sales, not sales because raw_query runs inside the DuckDB engine, so it sees DuckDB's catalog, not PostgreSQL's search path. pg_duckdb mounts your Postgres tables under the pgduckdb.public.* catalog. A bare sales fails with Catalog Error: Table with name sales does not exist.
• duckdb.query() won't carry this — it only accepts a single SELECT, so the COPY must go through raw_query().
• The write target must be under the az:// allow-list (a local path fails with LocalFileSystem has been disabled by configuration).
• If the target blob already exists and was created by something other than DuckDB (an az storage blob upload, say), the write can fail with IO Error: AzureStorageFileSystem … 'InvalidBlobType'. DuckDB happily overwrites files it wrote itself, but not a blob another tool created with an incompatible type. Write to a fresh path, or delete the stale blob first.

The result is a sales_2021.parquet of about 17 MB for these ~2M rows, already in Blob — no upload step.

Alternative — client-side export

If you can't or don't want to write from the database, pull the rows out and write the file client-side, then upload it.

The DuckDB CLI can read Postgres directly:

-- in the duckdb CLI
ATTACH 'host=… dbname=… user=… password=… sslmode=require' AS pg (TYPE postgres);
COPY (
  SELECT sale_id, sale_date, cust_id, prod_id, channel_id,
         country, segment, qty, amount
  FROM pg.sales WHERE sale_date < DATE '2022-01-01'
) TO 'sales_2021.parquet' (FORMAT parquet);

The resulting file can be uploaded:

az storage blob upload \
  --account-name <account> --account-key <key> \
  -c lake -f sales_2021.parquet -n archive/sales_2021.parquet --overwrite

This method is a possible alternative, but using PostgreSQL with a DuckDB COPY command offers the best security because it keeps the data within the cloud services protected by their native authentication methods.

Step 3 — Read it back through pg_duckdb

You can read the files from PostgreSQL to verify their content:

postgres=> SELECT count(*) 
           FROM read_parquet('az://lake/archive/sales_2021.parquet')
;
  count
---------
 1998938

(1 row)

The 2021 data is queryable straight from Blob. Note that the count(*) didn't read all rows, as the row count is available in the Parquet metadata footer and DuckDB gets the count from it.

Step 4 — Delete the cold rows from Postgres

Now that it's safely in Blob and verified, reclaim the space for new inserts:

postgres=> DELETE FROM sales 
           WHERE sale_date < DATE '2022-01-01'
;

DELETE 1998938

postgres=> SELECT count(*) FROM sales
;
  count
---------
 8001062

(1 row)

If lifecycle management wasn't in place from the start and you've deleted a large amount of data, you may need to reorg the table to actually shrink it on disk (pg_repack and pg_squeeze are available on Azure Database for PostgreSQL). Plain DELETE only marks rows as dead — it doesn't free disk space. A subsequent VACUUM reclaims that space, but it doesn't return it to the operating system: the freed heap space becomes available for any new row inserted into that table (tracked via the Free Space Map). In B-Tree indexes, reclaimed space is only reusable by new index entries whose key value falls into the same page/block — because B-Tree pages are key-ordered, not by arbitrary new values.

Step 5 — Keep querying history, concatenated with live data

The 2021 rows are gone from Postgres, but they are accessible from Parquet, and one query can span both. Here is an aggregate of the revenue per year, with hot rows from Postgres and old ones from Blob:

postgres=> SELECT yr, count(*) AS n, sum(amount) AS revenue
           FROM (
             -- Operational data in PostgreSQL
             SELECT extract(year FROM sale_date)::int AS yr, amount
             FROM sales
             WHERE sale_date >= DATE '2022-01-01'
             -- Heterogeneous Federation 
             UNION ALL
             -- Archived data in Blob
             SELECT extract(year FROM (r['sale_date'])::date)::int,
                    (r['amount'])::numeric(10,2)
             FROM read_parquet('az://lake/archive/sales_2021.parquet') r   
           ) u
           GROUP BY yr ORDER BY yr
;

  yr  |    n    |   revenue
------+---------+--------------
 2021 | 1998938 | 508738403.00
 2022 | 1998897 | 508731504.00
 2023 | 1998897 | 508723449.00
 2024 | 2004372 | 510107354.00
 2025 | 1998896 | 508699290.00
(5 rows)

The execution plan shows one DuckDBScan with a UNION over the Postgres scan and the Blob read:

postgres=> EXPLAIN (ANALYZE, COSTS OFF)
           SELECT yr, count(*) AS n, sum(amount) AS revenue
           FROM (
             SELECT extract(year FROM sale_date)::int AS yr, amount
             FROM sales
             WHERE sale_date >= DATE '2022-01-01'
             UNION ALL
             SELECT extract(year FROM (r['sale_date'])::date)::int,
                    (r['amount'])::numeric(10,2)
             FROM read_parquet('az://lake/archive/sales_2021.parquet') r
           ) u
           GROUP BY yr ORDER BY yr
;
                   QUERY PLAN
-------------------------------------------------
 Custom Scan (DuckDBScan) (actual time=0.002..0.003 rows=0.00 loops=1)
   DuckDB Execution Plan:

 ┌─────────────────────────────────────┐
 │┌───────────────────────────────────┐│
 ││    Query Profiling Information    ││
 │└───────────────────────────────────┘│
 └─────────────────────────────────────┘
 ┌────────────────────────────────────────────────┐
 │┌──────────────────────────────────────────────┐│
 ││               Total Time: 6.45s              ││
 │└──────────────────────────────────────────────┘│
 └────────────────────────────────────────────────┘
 ┌───────────────────────────┐
 │           QUERY           │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │      EXPLAIN_ANALYZE      │
 │    ────────────────────   │
 │           0 rows          │
 │          (0.00s)          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │          ORDER_BY         │
 │    ────────────────────   │
 │          u.yr ASC         │
 │                           │
 │           5 rows          │
 │          (0.00s)          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │       HASH_GROUP_BY       │
 │    ────────────────────   │
 │         Groups: #0        │
 │                           │
 │        Aggregates:        │
 │        count_star()       │
 │          sum(#1)          │
 │                           │
 │           5 rows          │
 │          (0.21s)          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │         PROJECTION        │
 │    ────────────────────   │
 │             yr            │
 │           amount          │
 │                           │
 │      10,000,000 rows      │
 │          (0.00s)          │
 └─────────────┬─────────────┘
 ┌─────────────┴─────────────┐
 │           UNION           │
 │    ────────────────────   │
 │           0 rows          ├──────────────┐
 │          (0.00s)          │              │
 └─────────────┬─────────────┘              │
 ┌─────────────┴─────────────┐┌─────────────┴─────────────┐
 │         PROJECTION        ││         PROJECTION        │
 │    ────────────────────   ││    ────────────────────   │
 │             yr            ││          extract          │
 │           amount          ││             r             │
 │                           ││                           │
 │       8,001,062 rows      ││       1,998,938 rows      │
 │          (0.03s)          ││          (0.01s)          │
 └─────────────┬─────────────┘└─────────────┬─────────────┘
 ┌─────────────┴─────────────┐┌─────────────┴─────────────┐
 │         TABLE_SCAN        ││         TABLE_SCAN        │
 │    ────────────────────   ││    ────────────────────   │
 │        Table: sales       ││         Function:         │
 │                           ││        READ_PARQUET       │
 │        Projections:       ││                           │
 │         sale_date         ││        Projections:       │
 │           amount          ││         sale_date         │
 │          Filters:         ││           amount          │
 │  sale_date>='2022-01-01': ││    Total Files Read: 1    │
 │           :DATE           ││                           │
 │                           ││                           │
 │       8,001,062 rows      ││       1,998,938 rows      │
 │          (4.68s)          ││          (6.60s)          │
 └───────────────────────────┘└───────────────────────────┘
 Planning:
   Buffers: shared hit=1
 Planning Time: 446.973 ms
 Execution Time: 224.008 ms

The left leaf reads the live table (with the sale_date >= 2022 filter pushed into the scan). The right leaf reads the archive from Blob. DuckDB unions and aggregates them. Note you don't need SET duckdb.force_execution = true — a query that references read_parquet goes to DuckDB automatically.

Making it a routine

If you partition the table by time (pg_partman is available on Azure Database for PostgreSQL), archiving a month becomes: export that partition, upload, DROP the partition. It is far cheaper than a big DELETE + VACUUM.

Lay the archive out by period (archive/year=2021/…) so you can read one period or glob a range — I will cover that in a future post.

Keep the export faithful — the source amount is numeric(10,2), so DuckDB writes and reads it as DECIMAL(10,2) natively (a bare numeric falls back to the Postgres executor).

I will show a fully automated information lifecycle management with pg_durable in a future blog post.

Takeaways

• pg_duckdb on Azure Database for PostgreSQL writes Parquet to Blob via duckdb.raw_query($$ COPY (…) TO 'az://…' $$)
• read_parquet('az://…') makes archived data queryable again. UNION/join it with the live table in one statement.
• You reclaim Postgres storage without losing query access to the history.

In the next post, we will see how to join a live Postgres table with reference data in Blob.

Atlas Infinite, which dropped in public preview on September 29 alongside MongoDB 9.0, delivers a significant architectural update payload: compute-storage disaggregation. Separating compute from storage is not new, and there are well-known ways to do it. But what makes Atlas Infinite interesting is how MongoDB departs from the standard design to keep WiredTiger's efficiency and to ensure that the shared storage layer never sees customer data unencrypted.

I am giving a high level overview of the architecture here distilling from the MongoDB Engineering blog post by Abhishek Chauhan, who led the effort. For detailed coverage, I refer you to that post. It is a great read, and parts 2 and 3 will drop soon.

Why not earlier, and why now

Disaggregated databases have existed for some time, so why did MongoDB wait? This is mainly because MongoDB didn't need disaggregation to scale out: when a workload outgrew one machine, sharding spread it across many nodes in a seamless manner.  MongoDB was among the first databases to ship sharding with automatic chunk balancing and routing. Since horizontal scaling has been a first-class feature for over a decade,  MongoDB never faced the single-node limit that pushed other systems toward  disaggregation early. That bought us time to do disaggregation right, without regressing on WiredTiger's performance or its security model.

Disaggregation is a good idea for several reasons anyways, and MongoDB adopted it to address customer needs. For one, storage gravity has become the  bottleneck for very large clusters. In Atlas's shared-nothing architecture, each node in a three-node replica set has its own disk with a full copy of the  data. That means, operations like growing storage, adding a replica, or restoring a backup, would require copying the whole dataset. On a large cluster  that can take hours when customers expect minutes. And since compute and  storage come bundled, needing more of one means paying for more of both. This  is why customers want compute and storage to scale independently. Disaggregation helps in right-sizing the customers and giving them flexibility for scaling. Finally, recent agentic development patterns want cheap snapshots,  clones, and branches of production data.

How MongoDB departs from the traditional disaggregated design

The standard approach works something like this. Compute is stateless. It ships a physical redo log (a record of every change to every page) to a quorum-replicated log service. Page servers read that log and rebuild pages by replaying the changes, so they can produce any page as of any point in the log. Object storage sits underneath as the durable copy.

MongoDB adopted much of this approach, but it broke with the traditional design in several places in order to keep WiredTiger's performance characteristics and the security guarantees customers expect.

Let's start with WiredTiger, one of the most sophisticated storage engines in production. WiredTiger achieves high performance by deferring and batching work. It is built around checkpoint-based consistency: rather than logging each page modification one by one, it lets writes accumulate in memory. A page can absorb many mutations and get written to disk just once, at a checkpoint. This amortizes the cost of turning writes into readable state. (This is increasingly where the field is heading as modern engines keep converging on copy-on-write, multi-version, checkpointed designs.) Forcing WiredTiger to log every page change for disaggregation sake would cripple its efficiency.

Secondly, because WiredTiger is lock-free and parallel and evicts pages speculatively, there is no canonical page image: two nodes holding the same data hold different bytes. So "replicate the pages" has no single correct target to replicate toward, which breaks from the traditional disaggregated design.

Finally, MongoDB refused to let shared storage read customer data. A storage tier that rebuilds pages from a log must be able to read them, so security ends up resting on operational controls. The better approach is to treat unencrypted user data as toxic. Holding it is a big responsibility, so keep it encrypted for as much of its lifetime as you can. That matters most when storage is shared across tenants, where one mistake no longer stops at one customer.

Because MongoDB owns the engine, the server, and the cloud service, it could co-design across all three layers. While the traditional design makes storage smart (by reading the data and rebuilding any page on demand), MongoDB went the other way and made storage blind to preserve WiredTiger's efficiency and to guarantee that storage can never read customer data unencrypted.

The architecture

Compute (primary, hot standby, and optional read replicas) runs queries and transactions and keeps only caches and recent changes, all of which can be rebuilt from storage if the node is lost. The shared storage layer, written in Rust, has three parts. A Log Service does only one thing, consensus on the log, which keeps it simple and steady. A Page Service spreads each customer's pages across many servers in every zone, with page materializers feeding it from the log. Object storage is responsible for long-term durability, organized by an Object Index Service.

Difference 1: two logs instead of one. A write is durable once the oplog, MongoDB's logical operation log, is on a majority of Log Service replicas across three zones. Separately, at checkpoint time, WiredTiger emits a phylog of physical page changes, mostly deltas. Page servers organize these by page after receiving them through the log service. This allows WiredTiger to keep its batching. Pages are written once per checkpoint, not once per change. Durability and page materialization are fully separate, with only the oplog on the critical path.

Difference 2: storage never sees your unencrypted data. In Atlas Infinite, data is encrypted on the compute node before any byte goes down, and decrypted only there, inside the customer's network, with keys the customer can hold and revoke. Storage can't see documents, collection names, indexes, or even tenant boundaries. Even obtaining root access to the whole storage fleet would yield an attacker only opaque bytes. Contrast this to the traditional design where storage must read data to rebuild pages, so security is partly "trust us" and partly "best intentions".

Difference 3: fresh replicas without page-level logging. In contrast to the traditional disaggregated approach where storage can build any page at any log point, in MongoDB pages only advance at checkpoints, so a replica reading only shared pages could potentially lag by seconds. This is easy to address however on the standby and read replica. Note that page servers build checkpoint-consistent pages from the phylog. On top of that, the compute node applies the oplog (as it arrives from the Log Service) to a local ingest table. Reads combine both to construct the fresh data. This way MongoDB gets millisecond replica lag while keeping checkpoint batching.

Finally, a word on availability. Every cluster has a hot standby, so a primary failure is a quick handoff rather than starting a new node. Data lives in three forms at once (the replicated log, the page servers, and object storage), the fleet is split into isolated cells so trouble in one can't cascade into the next, and recovery parallelizes across many machines rather than falling to one.