Storage Engine¶

GoVector's storage engine is responsible for persisting vector data to disk and loading it back into memory. This document explains how the storage engine works and how to optimize it for your use case.

📋 Storage Overview¶

GoVector uses a combination of technologies for efficient storage:

• BoltDB - Embedded key-value store for persistent storage
• Protocol Buffers - Efficient serialization format for vector data
• SQ8 Quantization - Optional 8-bit scalar quantization for memory optimization

🔧 Storage Architecture¶

Data Flow¶

flowchart TD
    A[Go Structs] --> B[Protocol Buffers]
    B --> C[Optional SQ8 Quantization]
    C --> D[BoltDB Storage]
    D --> E[BoltDB Retrieval]
    E --> F[Optional Dequantization]
    F --> G[Protocol Buffers]
    G --> H[Go Structs]

Components¶

1. Protocol Buffers: Serializes Go structs to compact binary format
2. BoltDB: Stores serialized data in a transactional key-value store
3. SQ8 Quantization: Compresses vectors to 8-bit integers when enabled

🚀 Persistence¶

Saving a Collection¶

// Save collection to disk
if err := collection.Save("/path/to/collection"); err != nil {
    log.Fatalf("Failed to save collection: %v", err)
}

Loading a Collection¶

// Load collection from disk
loadedCollection, err := core.LoadCollection("/path/to/collection")
if err != nil {
    log.Fatalf("Failed to load collection: %v", err)
}

📊 Storage Format¶

Collection Directory Structure¶

/path/to/collection/
├── metadata.json       # Collection configuration
├── points.db           # BoltDB database for points
└── index/             # Index files (HNSW graph)

Metadata Format¶

{
  "name": "my-collection",
  "vector_len": 768,
  "metric": "Cosine",
  "index_type": "HNSW",
  "quantize": false,
  "hnsw_config": {
    "m": 16,
    "ef_construction": 200,
    "ef_search": 10
  }
}

💡 Performance Optimization¶

For Write Performance¶

• Batch Operations: Use batch upserts for multiple points
• Disable Quantization: For write-heavy workloads, consider disabling SQ8 quantization
• SSD Storage: Use SSDs for faster write operations

For Read Performance¶

• Enable Quantization: SQ8 quantization reduces disk I/O and memory usage
• Memory Mapping: BoltDB uses memory mapping for faster reads
• Cache Warm-up: After loading a collection, perform a few search operations to warm up the cache

For Storage Efficiency¶

• Enable SQ8 Quantization: Reduces vector storage by ~75%
• Optimize Payload: Keep payloads small and avoid storing large objects
• Regular Compaction: BoltDB automatically compacts data, but large deletions may require manual compaction

📈 Storage Requirements¶

Vector Storage¶

Overhead¶

• BoltDB: ~10-15% overhead for transactional features
• Protocol Buffers: Minimal overhead (< 5%)
• Index Files: HNSW index adds ~20-30% overhead depending on M parameter

🚩 Common Issues¶

Slow Save/Load¶

• Cause: Large collection size or slow disk
• Solution: Use SSD storage, enable SQ8 quantization, consider smaller batch sizes

Corrupted Database¶

• Cause: Unexpected shutdown or disk error
• Solution: Regular backups, use journaling filesystems (ext4, APFS), implement proper error handling

High Disk Usage¶

• Cause: Large vectors or payloads, no quantization
• Solution: Enable SQ8 quantization, optimize payload size, consider using smaller vectors

🔗 Related Documentation¶

• Collection Management - Collection creation and configuration
• Data Model - Core data structures
• HNSW Index - HNSW index implementation
• Usage Modes - Different ways to use GoVector