Skip to content

Data Flow Analysis

This document provides systematic analysis of GoVector's data flow and concurrency control, covering three main data flow paths:

  • Write flow: Client request → Server → Collection → Storage → Index
  • Query flow: Client request → Server → Collection → Index → Storage
  • Filter flow: Index returns candidates → Storage fetches complete data → Apply filter conditions

Additionally, the document explains concurrency control mechanisms (read-write locks) and data consistency guarantee strategies, providing timing diagrams and flowcharts to help developers understand data processing logic in complex scenarios.

Project Structure

GoVector uses a modular design with clear layering:

  • Core library core: Collection management, index interface and implementation, storage engine, models and utility functions
  • API layer api: HTTP server providing Qdrant-compatible REST interface
  • Command line entry cmd/govector: Standalone microservice launcher
  • Examples and tests: Embedded examples, unit tests and integration tests
graph TB
subgraph "Application Layer"
Client["Client"]
end
subgraph "API Layer"
APIServer["HTTP Server
api.Server"]
end
subgraph "Core Library"
Collection["Collection"]
Storage["Storage"]
IndexIF["Index Interface VectorIndex"]
Flat["Flat Index FlatIndex"]
HNSW["HNSW Index HNSWIndex"]
Models["Models and Filtering"]
Math["Distance Metrics"]
Quant["Quantizer"]
end
Client --> APIServer
APIServer --> Collection
Collection --> Storage
Collection --> IndexIF
IndexIF --> Flat
IndexIF --> HNSW
Storage --> Quant
Collection --> Models
Collection --> Math

Core Components

  • Collection: Thread-safe collection manager responsible for write, query, delete operations; internally holds read-write lock, coordinates storage and index consistency.
  • Storage: bbolt-based persistence engine supporting vector quantization, metadata storage and batch read/write.
  • Index VectorIndex: Unified interface supporting FlatIndex (brute-force search) and HNSWIndex (approximate nearest neighbor).
  • Models and Filtering: PointStruct, Filter, Condition, supporting multiple filter types (exact, range, prefix, contains, regex).
  • Distance Metrics: Supports Cosine, Euclid, Dot three metrics for similarity calculation and sorting.
  • Quantizer: SQ8 quantizer compressing float vectors to 8-bit integers, reducing disk usage and memory pressure.

Architecture Overview

The diagram shows overall interaction paths from client to storage and index, along with dependency relationships between components.

graph TB
Client["Client"] --> HTTP["HTTP Request
api/server.go"]
HTTP --> Server["api.Server"]
Server --> Col["core.Collection"]
Col --> RW["Read-Write Lock
sync.RWMutex"]
Col --> Store["core.Storage"]
Col --> Idx["core.VectorIndex"]
Idx --> FlatIdx["FlatIndex"]
Idx --> HNSWIdx["HNSWIndex"]
Store --> BBolt["bbolt Persistence"]
Store --> Proto["Protocol Buffers"]
Store --> Quant["SQ8 Quantization"]
Col --> Filter["Filter
core/models.go"]
Col --> Dist["Distance Metrics
core/math.go"]

Detailed Component Analysis

Write Flow: Client Request → Server → Collection → Storage → Index

The write flow ensures "persist first, then index" order to guarantee data consistency; if index update fails, it tries to rollback storage layer changes.

sequenceDiagram
participant C as "Client"
participant S as "api.Server"
participant COL as "core.Collection"
participant ST as "core.Storage"
participant IDX as "core.VectorIndex"
C->>S : "PUT /collections/{name}/points"
S->>COL : "Upsert(points)"
COL->>COL : "Validate dimension/set version"
COL->>ST : "UpsertPoints(collection, points)"
ST-->>COL : "Success/failure"
COL->>IDX : "Upsert(points)"
IDX-->>COL : "Success/failure"
COL-->>S : "Return status"
S-->>C : "200 OK"

Key points and verification: - Dimension validation and version number setting completed in Collection.Upsert, ensuring data validity before write. - Persistence prioritized over index update, rollback attempted on failure (best-effort), avoiding inconsistency. - Concurrency control: Collection uses RWMutex, Upsert acquires write lock, ensuring only one write operation at a time.

Query Flow: Client Request → Server → Collection → Index → Storage

The query flow adopts "index first, retrieve from storage when necessary" strategy. FlatIndex directly traverses and filters in memory; HNSWIndex uses graph search combined with post-filtering strategy.

sequenceDiagram
participant C as "Client"
participant S as "api.Server"
participant COL as "core.Collection"
participant IDX as "core.VectorIndex"
participant ST as "core.Storage"
C->>S : "POST /collections/{name}/points/search"
S->>COL : "Search(query, filter, limit)"
COL->>IDX : "Search(query, filter, limit)"
alt FlatIndex
IDX-->>COL : "Candidate results (filtered)"
else HNSWIndex
IDX->>IDX : "Graph search (fetchK)"
IDX->>ST : "Fetch complete data by ID (optional)"
IDX-->>COL : "Candidate results (filtered)"
end
COL-->>S : "TopK results"
S-->>C : "200 OK"

Key points and verification: - FlatIndex performs filtering and distance calculation for all points in memory, sorts and truncates TopK. - HNSWIndex uses "over-sampling + post-filtering" strategy: expands fetchK based on filtering needs, then applies filtering and scoring to candidate set, finally returns TopK. - Concurrency control: Query uses read lock, allowing multiple concurrent queries.

Filter Flow: Index Returns Candidates → Storage Fetches Complete Data → Apply Filter Conditions

The filter flow emphasizes "index candidates first, then apply filtering". For FlatIndex, filtering is done directly in memory; for HNSWIndex, complete data needs to be fetched from storage or memory map before applying filtering.

flowchart TD
Start(["Start"]) --> Choose["Select index implementation"]
Choose --> |FlatIndex| FlatScan["Traverse in-memory point set"]
Choose --> |HNSWIndex| HNSWSearch["Graph search candidate set"]
FlatScan --> FlatFilter["Apply filter conditions"]
HNSWSearch --> FetchFull["Fetch complete data by ID (optional)"]
FlatFilter --> FlatSort["Sort by metric"]
FetchFull --> ApplyFilter["Apply filter conditions"]
ApplyFilter --> Sort["Sort by metric"]
FlatSort --> TopK["Truncate TopK"]
Sort --> TopK
TopK --> End(["End"])

Key points and verification: - MatchFilter supports Must/MustNot condition combinations, covering exact match, range, prefix, contains, regex types. - HNSWIndex's post-filtering strategy compensates for insufficient candidates caused by filtering by expanding fetchK.

Concurrency Control and Consistency

  • Collection uses sync.RWMutex:
  • Write (Upsert/Delete): Write lock, ensuring atomicity of persistence and index update.
  • Query (Search/Count): Read lock, allowing multiple concurrent queries.
  • Server uses mutex to protect collections map, avoiding races during concurrent collection registration/deletion.
  • Storage layer uses bbolt's transaction semantics (View/Update) to ensure read-write isolation and consistency.
  • Quantizer works transparently at storage layer without affecting upper-layer interface consistency.

Data Conversion and Serialization

  • Protocol serialization: PointStruct encoded/decoded through Protocol Buffers, supporting strings, integers, floats, booleans, byte arrays and other types.
  • Quantized storage: When quantization enabled, vectors compressed and stored in Payload, auto-decompressed on load.
  • Distance calculation: Calculates similarity or distance based on configured metric (Cosine/Euclid/Dot), used for sorting and return.

Dependency Analysis

  • Component coupling
  • Collection depends on Storage and VectorIndex, both decoupled through interfaces for easy implementation replacement.
  • Index implementations (Flat/HNSW) depend on Collection's Payload and ID mapping to support filtering and retrieval.
  • Server only depends on Collection interface, manages multiple collections through collections map.
  • External dependencies
  • bbolt: Local key-value database providing transaction and persistence capabilities.
  • coder/hnsw: HNSW graph search library providing efficient approximate nearest neighbor search.
  • protobuf: Cross-language serialization protocol for point structures and metadata storage.
graph LR
Server["api.Server"] --> Collection["core.Collection"]
Collection --> Storage["core.Storage"]
Collection --> VectorIndex["core.VectorIndex"]
VectorIndex --> Flat["FlatIndex"]
VectorIndex --> HNSW["HNSWIndex"]
Storage --> BBolt["bbolt"]
HNSW --> HNSWLib["coder/hnsw"]
Storage --> Proto["protobuf"]

Performance Considerations

  • Index selection
  • FlatIndex: Suitable for small-scale data, O(n) query, no training or maintenance overhead.
  • HNSWIndex: Suitable for large-scale data, approximate search, supports parameter tuning (M, EfSearch, etc.).
  • Filter strategy
  • HNSWIndex uses "over-sampling + post-filtering", improving hit rate in high-filter-rate scenarios.
  • Quantization
  • SQ8 quantization significantly reduces disk and memory usage, suitable for large-scale vector storage.
  • Concurrency
  • Query uses read lock supporting high-concurrency reads; write uses write lock avoiding blocking read path.

Troubleshooting Guide

  • Write failure rollback
  • Symptom: Index update failed but some data already written to storage.
  • Handling: Collection.Upsert tries to delete corresponding points from storage on index failure (best-effort), maintaining consistency.
  • Empty filter results
  • Symptom: Query returns empty results.
  • Diagnosis: Confirm Filter's Must/MustNot conditions are correct; check Payload type and key name.
  • Dimension mismatch
  • Symptom: Upsert/Search reports "dimension mismatch".
  • Handling: Ensure point vector dimension matches dimension specified when collection was created.
  • Server startup and shutdown
  • Symptom: Service fails to start or graceful shutdown fails.
  • Handling: Check port occupancy, database path permissions; use Stop(ctx) with timeout context.

Conclusion

GoVector achieves high-performance, scalable and consistent vector retrieval system through clear data flow boundaries and strict concurrency control. The write flow ensures persistence priority, the query flow flexibly switches between Flat and HNSW, and the filter flow balances accuracy and performance through post-filtering strategy. Combined with quantization and parameterized indexing, it can achieve good performance across different scales and scenarios.