Durability

This page tells you what a write survives, and what the database does at open after a crash or a power loss. WAL and Recovery explains the file format and the exact rules.

What a write survives

Event	Synced write	Unsynced write
Process crash, after Write returns	Survives	Survives
Power loss, after Write returns	Survives	Can be lost
Power loss, after a clean Dispose	Survives	Survives
Crash during Write	The batch is there whole, or not at all	The batch is there whole, or not at all

The reasons:

• A synced write (WriteOptions.Sync) flushes the WAL to the device before Write returns.
• An unsynced write is in the WAL file in the operating system when Write returns. The operating system keeps it when the process dies, but a power loss can lose it. This is the same as in RocksDB.
• Dispose flushes the current WAL file. So after a clean close, every write survives a power loss. RocksDB does not do this: its unsynced writes need SyncWAL before the close.
• A batch is one WAL record with a checksum. Recovery replays a record whole, or drops it whole.

A crash and a power loss are different

• A process crash can only cut the last write short. The operating system keeps everything else that the process wrote.
• A power loss can lose any sector written since the last flush, in any order. So the newest WAL file can end with damage that has intact records after it.

Solarium tells these two cases apart. Each WAL record carries the durable offset of its file: how much of the file a completed flush made durable when the record was appended.

What recovery does at open

At open, the WAL files replay in order, and the entries that an SSTable already holds are skipped. Then recovery decides what to do with any damage that it finds:

The open also fails, and changes nothing, for a directory with data files but no MANIFEST. A crash during a flush or a compaction leaves orphan files, and the next open deletes them. WAL and Recovery gives the exact rules.

The one case that can drop a synced write

The cut can drop a synced write only in one case: flushed data was damaged in a way that no crash causes, and no record was written after it. That is damage inside the last synced group of the file, with nothing appended since. Then nothing in the file shows that the group was flushed.

RocksDB has two recovery modes for this case:

• TolerateCorruptedTailRecords, which Kommander uses, fails the open after any power loss that left such a hole.
• PointInTimeRecovery drops everything after the first damage, whatever its cause.

Solarium opens after every kind of power loss that its tests simulate, and fails only on damage that no crash explains.

After a failure while running

A WAL, flush, or compaction failure stops all later writes until you open the database again. After a failed append, the tail of the WAL is unknown, so the database does not write after it. Reads continue. See Storage Failures.

How it is tested

The power-loss tests and the kill tests check these rules. They cut the power or kill a process at random points of a random workload, reopen the database, and check that it holds exactly the batches that it must hold.