Skip to content

Component Interactions

This document systematically outlines GoVector's component interaction relationships, focusing on:

  • Persistent interaction flow between Collection and Storage (write order, consistency guarantee, rollback strategy)
  • Index operation collaboration between Collection and VectorIndex (transparent switching of Flat/HNSW)
  • Business logic calls between Server and Collection (REST API to internal method mapping)
  • Interface abstraction design philosophy (how VectorIndex supports multi-strategy switching)
  • Inter-component communication protocols and data transfer formats (JSON/Protobuf)
  • Interaction timing diagrams for typical operation scenarios
  • Decoupling design principles and extensibility considerations

Project Structure

GoVector adopts a clear layered organization: the core library core provides vector collections, indexing, storage and model definitions; the api layer provides Qdrant-compatible HTTP service; cmd/govector provides standalone microservice entry; examples and tests cover key paths.

graph TB
subgraph "Application Entry"
CLI["Command Line Entry
cmd/govector/main.go"]
end
subgraph "API Layer"
Srv["HTTP Server
api/server.go"]
end
subgraph "Core Library"
Col["Collection
core/collection.go"]
Stor["Storage(BoltDB)
core/storage.go"]
VIface["VectorIndex Interface
core/index.go"]
Flat["FlatIndex Implementation
core/flat_index.go"]
HNSW["HNSWIndex Implementation
core/hnsw_index.go"]
Models["Data Models/Filters
core/models.go"]
end
CLI --> Srv
Srv --> Col
Col --> Stor
Col --> VIface
VIface --> Flat
VIface --> HNSW
Srv --> Models
Col --> Models
Stor --> Models

Core Components

  • Collection: User-facing logical container encapsulating Upsert/Search/Delete operations, coordinating Storage and VectorIndex, ensuring consistency between persistence and in-memory index.
  • Storage: bbolt-based local persistence responsible for Collection metadata and point data read/write, supports optional vector quantization.
  • VectorIndex Interface: Unified index abstraction shielding Flat and HNSW implementation differences, enabling transparent strategy switching.
  • FlatIndex: In-memory brute-force search suitable for small to medium scale data, provides exact results.
  • HNSWIndex: Graph-based approximate nearest neighbor search suitable for large-scale data, provides sub-linear complexity.
  • Server: Provides Qdrant-compatible REST API, manages Collection and forwards requests to Collection.
  • Data Models and Filters: PointStruct, ScoredPoint, Filter, Condition, supporting data structures and query filtering.

Architecture Overview

The diagram below shows the complete chain from HTTP requests to internal processing to persistence, and the internal Upsert/delete flow within Collection.

sequenceDiagram
participant Client as "Client"
participant API as "Server(HTTP)"
participant Col as "Collection"
participant Stor as "Storage"
participant Idx as "VectorIndex(Flat/HNSW)"
Client->>API : "POST /collections/{name}/points"
API->>Col : "Upsert(points)"
Col->>Col : "Validate dimension/set version"
Col->>Stor : "UpsertPoints(collection, points)"
Stor-->>Col : "Persistence success"
Col->>Idx : "Upsert(points)"
Idx-->>Col : "Index update success"
Col-->>API : "Return status"
API-->>Client : "200 OK"
Client->>API : "POST /collections/{name}/points/search"
API->>Col : "Search(vector, filter, limit)"
Col->>Idx : "Search(query, filter, topK)"
Idx-->>Col : "TopK results"
Col-->>API : "Return results"
API-->>Client : "200 OK"
Client->>API : "POST /collections/{name}/points/delete"
API->>Col : "Delete(points, filter)"
Col->>Stor : "DeletePoints(collection, ids)"
Col->>Idx : "Delete(id)"
Col-->>API : "Return delete count"
API-->>Client : "200 OK"

Detailed Component Analysis

Collection and Storage Persistence Interaction

  • Write order and consistency
  • Upsert: Persist first then update index, on failure try to delete written points from storage (best-effort rollback).
  • Delete: Delete points from storage first, then from index.
  • Storage metadata
  • Collection metadata saved in special bucket, rebuilt from metadata after restart.
  • Protocol and format
  • Point data uses Protobuf serialization; metadata uses JSON.
  • Optional vector quantization: Compress before storage, decompress on load.
flowchart TD
Start(["Upsert entry"]) --> Validate["Validate vector dimension/set version"]
Validate --> Persist["Storage.UpsertPoints"]
Persist --> PersistOK{"Persistence success?"}
PersistOK --> |No| Rollback["Rollback storage (best effort)"]
Rollback --> Error["Return error"]
PersistOK --> |Yes| UpdateIdx["VectorIndex.Upsert"]
UpdateIdx --> UpdateOK{"Index update success?"}
UpdateOK --> |No| Cleanup["Delete written points from storage (best effort)"]
Cleanup --> Error
UpdateOK --> Done(["Complete"])
StartDel(["Delete entry"]) --> Decide["Determine target ID list"]
Decide --> DelStore["Storage.DeletePoints"]
DelStore --> DelIdx["VectorIndex.Delete"]
DelIdx --> Done

Collection and VectorIndex Index Operations

  • Interface abstraction
  • VectorIndex defines capabilities like Upsert/Search/Delete/Count/GetIDsByFilter/DeleteByFilter, shielding implementation differences.
  • Strategy switching
  • NewCollection/NewCollectionWithParams selects FlatIndex or HNSWIndex based on useHNSW parameter.
  • HNSW supports parameterized configuration (M, EfConstruction, EfSearch, K), FlatIndex only needs distance metric.
  • Filtering and sorting
  • FlatIndex calculates distance for all points in memory and sorts.
  • HNSW uses graph search combined with post-filtering strategy (over-fetch + filter) to meet filtering requirements.
classDiagram
class VectorIndex {

    +Upsert(points) error
    +Search(query, filter, topK) []ScoredPoint
    +Delete(id) error
    +Count() int
    +GetIDsByFilter(filter) []string
    +DeleteByFilter(filter) ([]string, error)

}
class FlatIndex {

    -points map[string]*PointStruct
    -metric Distance
    +Upsert(points) error
    +Search(query, filter, topK) []ScoredPoint
    +Delete(id) error
    +Count() int
    +GetIDsByFilter(filter) []string
    +DeleteByFilter(filter) ([]string, error)

}
class HNSWIndex {

    -graph *Graph[string]
    -points map[string]*PointStruct
    -metric Distance
    -params HNSWParams
    +Upsert(points) error
    +Search(query, filter, topK) []ScoredPoint
    +Delete(id) error
    +Count() int
    +GetIDsByFilter(filter) []string
    +DeleteByFilter(filter) ([]string, error)

}
VectorIndex <|.. FlatIndex
VectorIndex <|.. HNSWIndex

Server and Collection Business Logic Calls

  • Routing and endpoints
  • /collections, /collections/{name}, /collections/{name}/points, /collections/{name}/points/search, /collections/{name}/points/delete.
  • Request parsing and forwarding
  • handleUpsert/handleSearch/handleDelete parse JSON requests into internal structures, call corresponding Collection methods.
  • Startup and recovery
  • Start automatically loads Collection metadata from storage and rebuilds Collection.
  • Error handling
  • Returns 404 for non-existent collections, 400 for invalid JSON, 500 for internal errors.
sequenceDiagram
participant Client as "Client"
participant S as "Server"
participant C as "Collection"
participant I as "VectorIndex"
participant D as "Storage"
Client->>S : "POST /collections"
S->>C : "NewCollectionWithParams(...)"
C->>D : "SaveCollectionMeta"
C->>I : "Initialize index"
S-->>Client : "200 OK"
Client->>S : "PUT /collections/{name}/points"
S->>C : "Upsert(points)"
C->>D : "UpsertPoints"
C->>I : "Upsert"
S-->>Client : "200 OK"
Client->>S : "POST /collections/{name}/points/search"
S->>C : "Search(vector, filter, limit)"
C->>I : "Search"
S-->>Client : "200 OK"
Client->>S : "POST /collections/{name}/points/delete"
S->>C : "Delete(points, filter)"
C->>D : "DeletePoints"
C->>I : "Delete"
S-->>Client : "200 OK"

Data Models and Filters

  • PointStruct/ScoredPoint: Carries vector, version number and metadata.
  • Filter/Condition: Supports must/must_not conditions, types include exact, range, prefix, contains, regex.
  • Matching algorithm: MatchFilter iterates through must/must_not, matches condition by condition; range, prefix, contains, regex have dedicated functions.
flowchart TD
F["Filter(Must/MustNot)"] --> CheckMust{"Iterate Must"}
CheckMust --> Cond1["Condition(Key, Type, Match/Range)"]
Cond1 --> MatchType{"Match type?"}
MatchType --> |exact| Exact["Value equals"]
MatchType --> |range| Range["Range judgment"]
MatchType --> |prefix| Prefix["Prefix match"]
MatchType --> |contains| Contains["Contains match"]
MatchType --> |regex| Regex["Regex match"]
Exact --> NextMust["Next Must"]
Range --> NextMust
Prefix --> NextMust
Contains --> NextMust
Regex --> NextMust
NextMust --> CheckMust
CheckMust --> |All satisfied| CheckMustNot{"Iterate MustNot"}
CheckMustNot --> Cond2["Condition match?"]
Cond2 --> |Match exists| Reject["Reject"]
Cond2 --> |None match| Accept["Accept"]
CheckMustNot --> |All satisfied| Accept

Dependency Analysis

  • Component coupling and cohesion
  • Collection couples with Storage and VectorIndex through interfaces, cohesive within "collection" semantics, clear responsibilities.
  • Server only depends on Collection interface, does not directly care about underlying storage or index implementation.
  • External dependencies
  • bbolt for persistence; coder/hnsw for HNSW graph search; Protobuf for serialization.
  • Circular dependencies
  • No circular dependencies found, module boundaries are clear.
graph LR
Server["Server"] --> Collection["Collection"]
Collection --> Storage["Storage"]
Collection --> VectorIndex["VectorIndex"]
VectorIndex --> Flat["FlatIndex"]
VectorIndex --> HNSW["HNSWIndex"]
Server --> Models["Models(Filter/Point)"]
Collection --> Models
Storage --> Models

Performance Considerations

  • Index selection
  • Small-scale data recommends FlatIndex, avoids construction overhead; large-scale data recommends HNSWIndex, adjust parameters according to scenario (M, EfConstruction, EfSearch, K).
  • Write path
  • Upsert persists first then updates index, ensuring data not lost after crash; but may produce brief inconsistency window, recommend combining with idempotent strategy in high-concurrency scenarios.
  • Query path
  • HNSW uses post-filtering strategy; the heavier the filter, the larger fetchK needs to be to reduce missed candidates; can consider adding filtering during graph traversal (current implementation is post-filtering).
  • Storage optimization
  • Enabling SQ8 quantization can significantly reduce disk usage, auto-decompresses on load, pay attention to CPU pressure and precision trade-off.

Troubleshooting Guide

  • Common errors and localization
  • Collection does not exist: Server returns 404 when handling /points/* if collection not found.
  • JSON parsing failed: handleUpsert/handleSearch/handleDelete return 400 for invalid JSON.
  • Write failed: Collection.Upsert tries best-effort rollback of storage-side writes if index update fails.
  • Dimension mismatch: Upsert/Search/Delete validates vector dimension before operation, returns error on mismatch.
  • Test references
  • Collection tests cover no-storage/with-storage, Upsert/search/delete, error scenarios.
  • Server tests cover end-to-end verification of create/delete/list/get collections, Upsert/Search/Delete.

Conclusion

GoVector achieves high availability and high performance of "embedded vector database" through clear interface abstraction and layered design: - VectorIndex interface enables transparent Flat/HNSW strategy switching, facilitating scale-based evolution. - Strict write order and rollback strategy between Collection and Storage ensure data consistency. - Server uses Qdrant-compatible API as unified entry, facilitating integration and migration. - For extensibility, new VectorIndex implementations (like IVF/PQ) or alternative storage backends (like LSM) can be added to meet different scenario requirements.