Snapshots

Snapshots

VortexDB’s snapshot system provides point-in-time backups of your entire database, including both the index topology and stored data.

Overview

A snapshot captures:

• Index topology: The structure of your index (graph edges, tree nodes, etc.)
• Index metadata: Configuration specific to the index type
• Storage data: All vectors and their payloads
• Database metadata: Dimensions, timestamps, version info

How It Works

Creating a Snapshot

The snapshot process involves two parallel operations:

Index Serialization

The index serializes its topology and metadata into binary format:

pub trait SerializableIndex {
    fn serialize_topology(&self) -> Result<Vec<u8>>;
    fn serialize_metadata(&self) -> Result<Vec<u8>>;
    fn snapshot(&self) -> Result<IndexSnapshot>;
}

Storage Checkpoint

The storage backend creates a consistent checkpoint:

fn checkpoint_at(&self, path: &Path) -> Result<StorageCheckpoint>;

For RocksDB, this uses the native checkpoint API for efficiency.

Manifest Generation

A manifest file is created with:

• Snapshot UUID
• Timestamp
• Parser version (for compatibility)
• SHA-256 checksums of all files

Archive Creation

All files are bundled into a compressed tarball (.tar.gz).

Restoring a Snapshot

Extract Archive

The tarball is extracted to a temporary directory.

Verify Checksums

All file checksums are verified against the manifest.

Version Check

The parser version is checked for compatibility.

Restore Storage

The storage checkpoint is restored first.

Rebuild Index

The index is deserialized from topology and metadata, then populated with vectors from storage.

Index-Specific Serialization

Each index type has its own serialization format:

Flat Index

• Topology: List of point IDs in insertion order
• Metadata: Minimal (just magic bytes)
• Restore: O(n) - simply rebuild the vector list

KD-Tree

• Topology: Tree structure with node relationships
• Metadata: Dimension count, tree depth
• Restore: O(n) - rebuild tree structure, populate vectors

HNSW

• Topology: Multi-layer graph with all edges
• Metadata: Level multiplier, max connections, entry point
• Restore: O(n) - rebuild graph layers, populate vectors

Snapshot Data Structures

Snapshot Object

pub struct Snapshot {
    pub id: Uuid,                        // Unique identifier
    pub date: SystemTime,                // Creation timestamp
    pub sem_ver: Version,                // Parser version
    pub index_snapshot: IndexSnapshot,   // Index data
    pub storage_snapshot: StorageCheckpoint, // Storage data
    pub dimensions: usize,               // Vector dimensions
}

Index Snapshot

pub struct IndexSnapshot {
    pub index_type: IndexType,  // Flat, KDTree, or HNSW
    pub magic: Magic,           // 4-byte identifier
    pub topology_b: Vec<u8>,    // Serialized graph/tree
    pub metadata_b: Vec<u8>,    // Index-specific config
}

Storage Checkpoint

pub struct StorageCheckpoint {
    pub path: PathBuf,         // Path to checkpoint files
    pub storage_type: StorageType, // InMemory or RocksDB
}

Snapshot Engine

The SnapshotEngine manages snapshot creation and restoration:

Registry Interface

pub trait SnapshotRegistry: Send + Sync {
    // Add a new snapshot to the registry
    fn add_snapshot(&mut self, path: &Path) -> Result<Metadata>;

    // List snapshots with pagination
    fn list_snapshots(&mut self, limit: usize, offset: usize) -> Result<SnapshotMetaPage>;

    // Get the most recent snapshot
    fn get_latest_snapshot(&mut self) -> Result<Metadata>;

    // Get metadata for a specific snapshot
    fn get_metadata(&mut self, id: String) -> Result<Metadata>;

    // Remove a snapshot
    fn remove_snapshot(&mut self, id: String) -> Result<Metadata>;

    // Load and restore a snapshot
    fn load(&mut self, id: String, path: &Path) -> Result<VectorDbRestore>;
}

Manifest Format

The manifest (manifest.json) contains all metadata needed for restore:

{
  "snapshot_id": "550e8400-e29b-41d4-a716-446655440000",
  "created_at": "2026-03-31T12:00:00Z",
  "parser_version": "1.0.0",
  "dimensions": 384,
  "index_type": "HNSW",
  "files": {
    "index_metadata": {
      "filename": "hnsw-index-meta.bin",
      "checksum": "sha256:abc123..."
    },
    "index_topology": {
      "filename": "hnsw-index-topo.bin",
      "checksum": "sha256:def456..."
    },
    "storage": {
      "filename": "rocksdb-checkpoint/",
      "checksum": "sha256:789ghi..."
    }
  }
}

Next Steps

Architecture

Understand how components fit together

Indexers

Choose the right index for your use case