Creating checkpoints by gaslighting a Postgres database

The future of databases is being rewritten by AI agents—80% of new databases are now created autonomously, often through 'fan out' strategies where dozens of agents spin up isolated Postgres instances to solve problems in parallel. But this creates a massive infrastructure headache: databases pile up, costs spiral, and cleanup is ignored because it costs extra tokens. Brian Clark, director of product at Databricks' Lakehouse team, reveals how their solution—Lakebase—uses a 'gaslighting' file system trick to make Postgres believe it's writing to a real disk while actually routing all writes to a virtual, shared storage layer. This enables instant branching with copy-on-write pointers, zero data duplication, and near-instant compute startup (under 500ms). The result? Thousands of ephemeral databases can be spun up and torn down in seconds, with no performance penalty. But the real breakthrough isn't speed—it's control: by decoupling compute from storage and centralizing access through Databricks' Unity Catalog, Lakebase eliminates the security nightmare of shared passwords and uncontrolled data access. It turns Postgres from a data silo into a temporary staging area, with the true source of truth living in a secure, governed lakehouse. This isn't just about developer speed—it's about making AI agents safe, accountable, and compliant.