HNSW Index¶

Hierarchical Navigable Small World (HNSW) is the default index type in GoVector, providing efficient approximate nearest neighbor (ANN) search. This document explains how HNSW works and how to optimize it for your use case.

📋 HNSW Overview¶

HNSW is a state-of-the-art approximate nearest neighbor search algorithm that:

• Provides high recall (close to 100% for well-tuned parameters)
• Offers fast search performance (O(log N) complexity)
• Scales well with high-dimensional vectors (up to thousands of dimensions)
• Supports dynamic insertion of new vectors

🔧 HNSW Parameters¶

Core Parameters¶

🚀 Creating an HNSW Index¶

Embedded Mode¶

import (
    "github.com/yourusername/govector/core"
)

// Create a collection with HNSW index
collection, err := core.NewCollection(core.CollectionConfig{
    Name:            "my-collection",
    VectorLen:       768,
    Metric:          core.Cosine,
    IndexType:       core.HNSW,  // Use HNSW index
    M:               16,         // Number of connections per node
    EfConstruction:  200,        // Construction parameter
    EfSearch:        10,         // Search parameter
})
if err != nil {
    log.Fatalf("Failed to create collection: %v", err)
}

Microservice Mode¶

curl -X POST http://localhost:6333/collections \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-collection",
    "vectors": {
      "size": 768,
      "distance": "Cosine"
    },
    "hnsw_config": {
      "m": 16,
      "ef_construction": 200,
      "ef": 10
    }
  }'

📊 How HNSW Works¶

HNSW creates a multi-layered graph structure where:

1. Each layer is a navigable small world graph
2. Higher layers act as coarse guides to lower layers
3. Search starts at the top layer and navigates down to find nearest neighbors
4. Insertion follows a similar path, building connections at each level

Search Process¶

flowchart TD
    A[Start at top layer] --> B[Find nearest neighbor in current layer]
    B --> C{Is current layer the bottom?}
    C -->|No| D[Move to next lower layer]
    D --> B
    C -->|Yes| E[Return nearest neighbors]

Insertion Process¶

flowchart TD
    A[Start at top layer] --> B[Find nearest neighbors in current layer]
    B --> C{Is current layer the bottom?}
    C -->|No| D[Move to next lower layer]
    D --> B
    C -->|Yes| E[Insert node and build connections]
    E --> F[Potentially add node to higher layers]

💡 Optimization Tips¶

For Large Datasets¶

• Increase M to 24-32 for higher recall
• Increase EfConstruction to 400-1000 for better index quality
• Use EfSearch between 50-100 for high recall applications
• Enable SQ8 quantization to reduce memory usage

For Fast Search¶

• Decrease M to 8-12 for faster indexing and lower memory usage
• Set EfSearch to 10-20 for faster search (trading off recall)
• Use smaller vectors when possible (e.g., 384 dimensions instead of 768)

For Memory Constraints¶

• Enable SQ8 quantization to reduce memory by ~75%
• Use smaller M values (8-12)
• Consider Flat index for very small datasets (< 10,000 points)

📈 Performance Benchmarks¶

Search Performance¶

Memory Usage¶

🚩 Common Issues¶

Slow Index Build¶

• Cause: High EfConstruction value or large dataset
• Solution: Use appropriate EfConstruction for dataset size, consider batch insertion

Low Recall¶

• Cause: Low EfSearch or M values
• Solution: Increase EfSearch for search-time recall, increase M for index quality

High Memory Usage¶

• Cause: Large M value or no quantization
• Solution: Enable SQ8 quantization, reduce M value

🔗 Related Documentation¶

• Collection Management - Collection creation and configuration
• Data Model - Core data structures
• Storage Engine - Persistence and serialization
• Usage Modes - Different ways to use GoVector