With regard to performance, commitment, effort, dedication, there is no middle ground. Or you do something very well or not at all.
- Ayrton Senna
If you inspect modern databases, you will see that they typically store their data in three places:
When we say object storage native databases, what we mean is that their persistence layer is object storage and that they navigate this storage hierarchy with elegance. The amusement of this strategy is that it spreads data depending on how frequent it is used. Favor hot data in memory, warm data in disk, and cold data in object storage; this is the game.
Many databases elect to optimize within one particular niche of the compute storage hierarchy. Instead, we argue for hot-warm-cold harmony. The requirements coalesce in a way that appoints object storage as the sole dependency for durability. We highlight the levers that make the architecture possible along the way.
†This post attempts a "one shot" introduction. The chapters themselves are not useful independently.
When building systems, a straight-forward strategy is to build for the qualities of interest and reconcile with the tradeoffs as consequence. Historically, databases experienced many cycles of where compute lives and where storage lives. In the old world, the two qualities were coupled. Storage and compute were attached to the same machine; where the data lived is where the compute happened. But state has to live somewhere, right?
A few primitives were required to allow the ecosystem to detangle the two. We intend to The ability to fetch objects from S3 into cache looksl like a storage moving to compute maneuver. But you also want the ability to scale nodes at the will of the workload. If the data was not active, the idea is that you don't want any compute nodes on. Only when a worload was ready did you want to scale compute nodes; like shifting from 0 to N nodes. e will come across a new way to describe the architecture, which is compute-storage flexible. The convenience of bootstrapping onto object storage like S3 is that it already champions around eleven 9s of reliability. This used to require a lot of consensus, but in recent years, arrived at an inflection point that allows object storage to be a sole dependent on durability. As a result, we can think of local NVME SSDs and DRAM as a general cache layer.
Flexibility comes out in different ways. When serving compute nodes in the cache layer, they could just turn off. As long as a write gets acknowledged by object storage, all working data in memory and disk can die. One could also scale the compute nodes. This decoupling between compute and storage describes it as flexible.
The object storage layer is one end of the continuum. Its defining characteristic is that it provides cheap & durable memory, but the round trip latency is slow (~100ms). S3 for example is (~0.002/GB/Mo). This is the economic appeal and tradeoff of object storage.
When a write (PUT) is acknowledged on object storage, it becomes a durable & immutable object. Strategies typically vary on how to organize data after the fact. †Immutability means updates to an object do not happen inplace, but are replaced.
Think of the background maintenance as compaction (to appeal to space) or indexing (to appeal to structure).
A simple way to handle this territory is to push off that work for "later" or wait until certain thresholds. This is usually not worth it in the long run and often call for passive ways to compact and index. This is unsolved territory and variance usually depends on the workload that the database aims to support.
Write Ahead Log (WAL)
A log has two properties:
Anyone could replay the log and deterministically arrive at the same state.
A traditional WAL on disk supports frequent, small, sequential writes. You can think of the appends themselves as ~1 op but k ops to perform the fsync. An fsync ensures that an append on disk becomes durable.
You can also use one fsync across a volume of appends. Let's take a closer look at an example of a WAL on disk visually scaled to 100ms -> 1s.
The fsync operation itself costs ~1ms, but it's common to see folks config this latency down to as low as 1s to avoid it from contending with the IOPS.
It's useful to keep fast & simple writes, but it must be justified with responsibility. On disk, the bulk of the ops are used to append to the log with sequential write speed, then promise durability with fsync.
Obtaining durability on object storage looks similar, but at the object storage end of the storage continuum, the writes are of a different nature:
The standard rate on S3 for a write (PUT) is $0.005 per 1000 PUTs (or $0.000005/PUT). We will make an attempt to show the same WAL-on-Disk strategy applied to this object storage pricing model.
Let's place the WAL in object storage and naively mark 1 append = 1 PUT request.
The first issue you might notice is that object storage is latency bound. We could not possibly match the total volume of appends we were making on disk. A PUT request is ~100ms in latency.
There is huge waste in naively appending like this to object storage.
Instead, we can pool writes on an in-memory op buffer and let a single batched PUT request write the whole buffer as one WAL entry in object storage.
This is the same maneuver as the fsync on a WAL on disk. Let the writes accumulate, then commit them in one batch.
There's a strong case to amortize the latency and the cost across all of them. This is a key contribution towards the economic responsibility of the object storage layer.
One of the challenges with a WAL on disk is that it tangles the read and write paths. Writes are kept dumb in the interest of sequential appends to the log, but reads, and the mechanisms to pamper the reads (like compaction & sorting), end up contending for the same IOPS on a machine.
By placing the WAL on object storage, the architecture becomes extremely tolerant with respect to what goes on in the memory and disk layer. As long as an object gets acked by the WAL, compute nodes can die.
This is a useful property and the ways that object storage databases handle this tend to vary here but it typically looks like either of the following, both of which are simple in their own right.
One could dedicate writer nodes and reader nodes.
┏━━━━━━━━┓ ┏━ writer node ━━━━┓ ┏━━━━━━━━━┓
┃ ┃ ┃memory ███▓ ┃ ┃ ┃
┃ ┣┅┅┅╋▶ ┣━━┫ ┃
┃ ┃ ┃NVME ████████▓ ┃ ┃ ┃
┃ ┃ ┗━━━━━━━━━━━━━━━━━━┛ ┃ object ┃
┃ router ┃ ┃ storage ┃
┃ ┃ ┏━ reader node ━━━━┓ ┃ ┃
┃ ┃ ┃memory ███▓ ┃ ┃ ┃
━━▶ ┣┅┅┅╋▶ ┣━━┫ ┃
┃ ┃ ┃NVME ████████▓ ┃ ┃ ┃
┃ ┃ ┗━━━━━━━━━━━━━━━━━━┛ ┃ ┃
┃ ┃ ┃ ┃
┃ ┃ ┏━ ....... ━━━━━━━━┓ ┃ ┃
┃ ┃ ┃ ┃ ┃ ┃
┃ ┣┅┅┅┫ .... ┣━━┫ ┃
┃ ┃ ┃ ┃ ┃ ┃
┗━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━┛
With asymmetric read and write loads, this can be an attractive schema; simply scale towards more read or write nodes to meet its demand.
Another method is to deploy generic nodes that serve both read and write work.
╔════════════════════╗ ┏━ object storage ━━━━━┓
━━━━━━━┓ ║ ┏━ R/W node ━━━━┓ ║ ┃ ┃
┃ ║ ┃memory ███▓ ┃ ║ ┃/{org_id}/{namespace}/┃
router ├─┼▶┃ ┃ ╠─┼▶ ┃
┃ ║ ┃NVME ████████▓┃ ║ ┃/{org_id}/{namespace}/┃
▶ ┃ ║ ┗━━━━━━━━━━━━━━━━┛ ║ ┃ ┃
┃ ║ ║ ┃/{org_id}/{namespace}/┃
┃ ║ ┏━ ....... ━━━━━┓ ║ ┃ ┃
├─┼▶┃ ┃ ║ ┗━━━━━━━━━━━━━━━━━━━━━━┛
┃ ║ ┃ ... ┃ ║
┃ ║ ┃ ┃ ║
━━━━━━━┛ ║ ┗━━━━━━━━━━━━━━━━┛ ║
╚════════════════════╝
One could deploy a literal GCP or EC2 instance per namespace. It's also common to see a single R/W node handling multiple namespaces.
There are strong cases to make about any of these choices.
Notice how this substantiates the strategy of compute storage flexibility. Compute nodes died? Durability is covered by object storage. Want to scale compute? Add more VMs in the direction demanded from the system.
When you see either of these compute node schemas, proxy your thought to "how do they handle concurrent writes?"
Concurrency and race conditions are a nightmare for distributed systems. Every major mechanism involved in object storage native is directly or indirectly a method to answering the crisis of concurrency with simplicity.
For example, in a traditional LSM, a typical read follows
memtable╮ ╭▶L1╮ ╭▶LN
╰▶frozen memtable╯ ╰▶...╯
(pre-flush) └─────────┘
SSTables
├───────────── read path ─────────────▶
Object storage native employs the same principle, but it looks different. We will start with a naive picture of the architecture, then narratively bolt on key mechanisms, ultimately to arrive at a coherent picture of both the read and write paths.
Below is a writer node appending ID 0, ID 1, ID 2, ..., ID N to an "in-memory buffer" with the intent to accumulate appends to get batched into a PUT request as a durable object in the WAL in object storage. Next to it is a reader node repeatedly reading the ID 0.
Here the object storage starts empty, then a stream of PUT requests batch the in-memory op buffer into the WAL. Now that ID 0 is the WAL, it's prepared for a read. Think of LSM files (L1, L2, .., LN) as a compaction layer. They aim to organize WAL files and clean scenarios where multiple versions of the ID exist, sorting along the way. Any data structure could live here, but typically a self-organizing structure like an LSM.
As the system appends IDs [0, N) on a 0.5ns basis, we're interested in doing a search-freshest-data-first strategy just like a traditional LSM on disk.
The appeal to this strategy is to answer the scenario where ID 0 could get overwritten
ID 0 (v2)
memtable╮ ╲ ╭▶L1╮ ╭▶LN
╲ ╰▶frozen memtable╯ ╰▶...╯
ID 0 (v3) ╲
ID 0 (v1)
├───────────── read path ─────────────▶
At any point the read finds an ID 0, it stops; it found the freshest copy of ID 0.
What is missing from this story is that a proper LSM has a MANIFEST file. The manifest is a small file that tracks which file covers which range of keys.
Since the LSM is on object storage, each check is its own GET request and dominates the added latency.
Without the manifest, the read path produces unpredictable behavior:
mem╮ ╭▶WAL╮ ╭▶... = ???
╰▶nvme╯ ╰▶...╯
With a traditional LSM on disk, this manifest file is often overlooked in an ELI5 because reads happen much faster in memory/disk. On object storage, the unit of work is a ~100ms roundtrip, so the manifest becomes a largely important detail and is key to this object storage native architecture.
By referring to the manifest, you can establish predictability:
mem╮ ╭▶manifest╮ = 2RT
╰▶nvme╯ ╰▶[]
(no manifest) (w/ manifest)
┌───────────────┐ ┌───────────────┐
RTs ▲ ▲
6 │█░░░░░██░█░░██░░█┊░░░░░░░░░░░░░░░░░
4 │░░█░░█░░█░░█░░██░┊░░░░░░░░░░░░░░░░░
2 │░█░██░░░░░█░░░░░░┊█████████████████
0 ┼──────────────────────────────────▶ (reads)
where RT1 is to fetch the manifest and RT2+ to retrieve relevant files.
It appears that reading ID 0 is quite popular, but it's wedged in the cold depths of object storage. The game is hot-warm-cold harmony, so how does files ever get to the cache layer?
The first read for ID 0, it does 2 RTs on object storage:
We call this a cold query because it's slow, but on the way back, a copy of that L2 file is warmed onto NVMe. Subsequent warm queries are served from NVMe with no roundtrip to object storage. And because ID 0 keeps getting hit, its file is eventually promoted into memory, where it's served hot.
In reality, the number of round trips on the cold query depends on how the data is structured and how large the structure is.
Practically speaking, people are not literally reading ID 0 repeatedly, but rather data related to ID 0 down the road.
This becomes a game of speculation. Object storage native happens to be excellent as serving this type of work.
Once the cold query goes through, different databases will have their own policies on the cache hierarchy. Cache efficiency has a large runway for improvement.
What matters in the abstract is that the file can now be accessed at the cache layer.
We still have not answered what happens if ID 0 got overwritten. What if it got overwritten multiple times: ID 0 v2,..., ID 0 vN?
The system has no current way of checking if the cache has the freshest ID 0.
In this experiment, the cache is populated with a file from L2. After many reads, the writer node initiates a PUT request, which contains an ID 0 v2, which in turn means all subsequent reads on the cache layer are now stale!
How can the system be aware of this?
In brief, there are really only two ways to deal with this. When a write comes in, is it available to the next read? If the answer is yes, this is called Strong Consistency. When a write comes in and is not available to the next read, it's eventually available, so it's called Eventual Consistency.
Each mode affects how the system performs, so it's important to distinguish the two.
Strong Consistency
To be strongly consistent requires a mechanism involving the manifest with some cooperation on both the read and write paths.
Every written object gets stamped with an ETag. The manifest is the keeper of these ETags.
You can think of the manifest ETag as the main commit hash.
On a cold query, RT1 requests for the manifest to be put in memory; its ETAG comes with it. Every read is packaged with a conditional GET that asks object storage whether their manifest ETag is the same one in cache. If nothing changed, the conditional GET response says nothing was modified.
In this scenario, the initial ETag was a1b2. A WAL object was created and the manifest's ETag changed to c3d4. The next read's conditional GET catches the new ETag and pulls the new manifest. The manifest has the most updated pointers to where the freshest data lives, so the reader initiates another GET request to satisfy the read request.
This works because object storage vendors like Azure, GCP, and AWS keep a metadata layer that tracks each object's ETag. The conditional GET checks the manifest's ETag there in the metadata plane, not needing to look inside its contents. If the ETag hasn't changed, a 304 is returned.
The metadata check runs about 5-18ms depending on the vendor. This is the latency price every read must pay to stay current (strongly consistent).
The issue is that multiple objects can be in the WAL.
To find the fresh values, the reader has to exhaustively scan every WAL object: one GET per object. The bill is 1 GET for the ETag plus 1 GET per relevant changed object.
The above exhaustive search look brutally slow, but why fire the GET requests in series?
Fire them in parallel! This makes the scan appear as one round trip of latency for the same $.
This works because the requests are from the same ETag instance. The hope is that all the requests finish in a reasonable time. The exhaustive search is bounded by the slowest request.
In addition to this, remember how the WAL is responsible for replication? Exhaustive scans will replay this log over the indexed data in the cache layer.
Eventual Consistency
A lot of mechanics were required to explain Strong Consistency. Thankfully, the guide to Eventual Consistency is quite brief in comparison.
Turn off Conditional GET.
This will remove the floor latency to check the metadata layer, so that's a 5-18ms saving and the system will just serve cache.
Any policy that satisfies the "eventually" clause will work here. For example, a periodic timer which GETs the new manifest and data and updates the cache. So when the writer commits v2, the reader keeps serving stale v1 until the next refresh runs.
Concurrency and race conditions are a major issue for every distributed system.
There are two main retorts depending on the layout of the cache layer.
╔════════════════╗ ╔════════════════╗
║ ┏━ W node ━━━┓ ║ ║ ┏━ R/W node ━┓ ║
║ ┫PUT ID 0 v2 ┣ ║ ║ ┫PUT ID 0 v2 ┣ ║
║ ┗━━━━━━━━━━━━┛ ║ ║ ┗━━━━━━━━━━━━┛ ║
║ ┏━ W node ━━━┓ ║ ║ ┏━ R/W node ━┓ ║
║ ┫PUT ID 0 v2 ┣ ║ vs ║ ┫PUT ID 0 v2 ┣ ║
║ ┗━━━━━━━━━━━━┛ ║ ║ ┗━━━━━━━━━━━━┛ ║
║ ┏━ R node ━━━┓ ║ ║ ┏━ ... ━━━━━━┓ ║
║ ┫ ┣ ║ ║ ┫ ┣ ║
║ ┗━━━━━━━━━━━━┛ ║ ║ ┗━━━━━━━━━━━━┛ ║
╚════════════════╝ ╚════════════════╝
(A) (B)
The many responses for A:
The responses for A require coordination in some form. This hurts the architecture's flexibility.
The elegant response for B:
CAS checks the manifest against the version you read. Say two PUT requests for ID 0 v2 are initiated, both holding ETag a1b2. The first one lands and moves the manifest to c3d4. The latter is still matching on a1b2, which no longer matches, so object storage rejects it with a 412 and that writer retries against c3d4.
Another win for the metadata layer and it allows multiple in the cache layer to make requests without coordinating a leader.
A single writer is a simple answer, and a real one. Serialize every write through one node and the race cannot happen. The weakness is that the one node is now the ceiling. Every write funnels through it, and if it dies, writes pause until a replacement recovers its state from object storage. Reads can continue serving the whole time.
Compare and swap removes that ceiling by pushing coordination into object storage itself, and pays for it in contention: under load the losers burn roundtrips retrying.
The demo runs both on one clock. The single writer marches the manifest forward and every PUT lands. The fleet forks, one writer takes the 412, and retries against the new version.
Enjoy these closing remarks in the form of bullet points. (i'm too dumb to make it complicated)
What's left is for you to just check out these object storage native database docs! For the most part, they have very useful public-facing documents and without them, this blog would be nothing.
With so many resources available, I found it appropriate to point out the people who inspired this writing. They have made a profound impact on my thinking.
[1] Plenty of online resources on these topics but from now on, I am recommending Ben Dicken from PlanetScale: WAL and LSM
[2] Joel Gustafson (former Canvas of Protocol Labs) who references Jay Kreps (during LinkedIn).
C'est Tout
Andy Tran | 2026