Core Modules¶

Overview¶

GoVector consists of five core modules that collaborate to provide efficient vector storage and retrieval capabilities. Each module is designed with a specific responsibility and can operate independently while supporting integration with other modules.

Module Architecture¶

graph TB
subgraph "Core Modules"
Collection["Collection
Collection Management"]
Storage["Storage
Local Persistence"]
Flat["FlatIndex
Flat Index"]
HNSW["HNSWIndex
Graph Index"]
Quant["SQ8Quantizer
8-bit Scalar Quantization"]
end
Collection --> Flat
Collection --> HNSW
Collection --> Storage
Storage --> Quant

Collection¶

Responsibility: Provides unified vector management interfaces

Main Functions: - Vector CRUD operations (Create, Read, Update, Delete) - Unified search interface - Index type selection and switching - Filter conditions

Key Properties: | Property | Type | Description | |----------|------|-------------| | name | string | Collection name | | vectorLen | int | Vector dimension | | metric | Distance | Distance metric | | indexType | IndexType | Index type |

Storage¶

Responsibility: Manages persistent storage of vector data

Main Functions: - Vector data persistence (BoltDB) - Protobuf serialization - Collection management - Quantized data storage

Implementation Details:

type Storage struct {
    db        *bbolt.DB
    closed    bool
    quantizer Quantizer
    useQuant  bool
}

Key Features: - Key-value storage based on BoltDB - Protobuf-based efficient serialization - Support for quantized data storage - Thread-safe read and write operations

FlatIndex¶

Responsibility: Provides brute-force exact search capability

Main Functions: - Brute-force nearest neighbor search - Filtering support - Multiple distance metrics

Use Cases: - Small to medium-sized datasets ( < 10,000 vectors) - Scenarios requiring exact results - As baseline for algorithm comparison

Performance Characteristics: - Search time: O(n), linearly scales with dataset size - Memory usage: O(n) for vector storage - No index building overhead

HNSWIndex¶

Responsibility: Provides efficient approximate nearest neighbor search

Main Functions: - Hierarchical graph index construction - ANN search with configurable accuracy - Support for multiple distance metrics

Use Cases: - Large-scale datasets (> 10,000 vectors) - Latency-sensitive production environments - Scenarios with high query throughput requirements

Performance Characteristics: - Search time: O(log n) average - Construction time: O(n log n) - Memory overhead: Additional graph structure storage

Parameters: | Parameter | Default | Description | |-----------|---------|-------------| | M | 16 | Maximum connections per node | | EfConstruction | 200 | Construction candidate list size | | EfSearch | 64 | Search candidate list size |

SQ8Quantizer¶

Responsibility: Implements 8-bit scalar quantization for vector compression

Main Functions: - Vector lossy compression - Memory usage reduction - Integration with storage subsystem

Compression Principle:

The SQ8 algorithm compresses 32-bit floating-point vectors to 8-bit integers through the following steps:

1. Range Calculation: Find the minimum and maximum values in the vector
2. Linear Scaling: Map values from [min, max] to [0, 255]
3. Storage: Store the min, max values and quantized data

Benefits: - 4x memory reduction (float32 to uint8) - Minimal accuracy loss (typically < 1%) - Significant performance improvement for large datasets

Module Interactions¶

Data Flow During Upsert¶

sequenceDiagram
    participant Client
    participant Collection
    participant Flat/HNSW
    participant Storage
    participant Quantizer

    Client->>Collection: Upsert(points)
    Collection->>Collection: Validate points
    Collection->>Flat/HNSW: Add to index
    Collection->>Quantizer: Quantize (if enabled)
    Collection->>Storage: Persist points
    Storage->>Quantizer: Store quantized data
    Collection-->>Client: Success

Data Flow During Search¶

sequenceDiagram
    participant Client
    participant Collection
    participant Index
    participant Storage

    Client->>Collection: Search(query, topK)
    Collection->>Index: ANN search
    Index-->>Collection: Candidate results
    Collection->>Storage: Retrieve payload
    Collection-->>Client: ScoredPoint results

Dependency Relationships¶

┌─────────────────┐
│   Collection    │
└────────┬────────┘
         │
    ┌────┴────┬───────────┐
    ▼         ▼           ▼
┌───────┐ ┌───────┐ ┌──────────┐
│ Flat  │ │ HNSW  │ │ Storage  │
│ Index │ │ Index │ └────┬─────┘
└───────┘ └───────┘      │
                         ▼
                   ┌───────────┐
                   │ SQ8Quant  │
                   └───────────┘

Module Selection Guide¶

Extension Points¶

The module design supports extension through interfaces:

• Quantizer interface: Implement custom quantization algorithms
• Storage interface: Support other storage backends
• Index interface: Integrate other indexing algorithms

type Quantizer interface {
    Quantize(vector []float32) []byte
    Dequantize(data []byte) []float32
}