Skip to content

Data Consistency Issues

This guide focuses on GoVector data consistency issues, covering diagnosis and repair methods for data integrity issues such as data loss, duplicate data, index inconsistency; provides backup and recovery strategies based on BoltDB (including maintenance and repair); details transaction processing, concurrent access control and data synchronization mechanisms; provides data verification tool usage and repair processes; and summarizes preventive best practices and monitoring recommendations.

Project Structure

GoVector uses a clear layered modular design: - Command-line entry is responsible for starting service, loading storage and collections, and registering HTTP API. - API layer provides Qdrant-compatible REST interfaces, managing collection and point operations. - Core layer contains collection, index, storage, quantization and mathematical calculation modules. - Storage layer uses BoltDB as the persistence backend, combined with Protobuf serialization to achieve highly reliable local storage.

graph TB
subgraph "Application Entry"
CLI["Command-line entry
cmd/govector/main.go"]
end
subgraph "API Layer"
APIServer["HTTP API Server
api/server.go"]
end
subgraph "Core Layer"
Collection["Collection
core/collection.go"]
IndexIF["Index interface
core/index.go"]
HNSW["HNSW Index
core/hnsw_index.go"]
Flat["Flat Index
core/flat_index.go"]
Storage["Storage engine
core/storage.go"]
Quant["Quantizer
core/quantization.go"]
Math["Distance metric
core/math.go"]
Models["Data model
core/models.go"]
end
CLI --> APIServer
APIServer --> Collection
Collection --> Storage
Collection --> IndexIF
IndexIF --> HNSW
IndexIF --> Flat
Storage --> Quant
Collection --> Math
APIServer --> Models

Core Components

  • Collection: Thread-safe vector collection, responsible for version number generation, persisting to disk first then updating in-memory index, dual-write consistency and rollback on delete.
  • Storage: Persistence layer based on BoltDB, providing collection buckets, metadata bucket, batch insert/delete, list and load capabilities; supports optional vector quantization.
  • Index (VectorIndex): Unified interface, supports HNSW and flat index, provides add/delete/update/search and filtering capabilities.
  • API Server: Provides Qdrant-compatible REST interfaces, manages collection and point operations, supports graceful shutdown.
  • Models and Filtering: Defines point structure, scored result, filter conditions and matching logic.
  • Math and Quantization: Provides distance metrics and 8-bit scalar quantization, balancing accuracy and space efficiency.

Architecture Overview

The diagram below shows the complete chain from API request to persistence and index update, as well as key nodes for error rollback and consistency guarantee.

sequenceDiagram
participant Client as "Client"
participant API as "API Server
api/server.go"
participant Col as "Collection
core/collection.go"
participant Store as "Storage
core/storage.go"
participant Idx as "Index
core/index.go/HNSW/Flat"
Client->>API : "PUT /collections/{name}/points"
API->>Col : "Upsert(points)"
Col->>Col : "Generate version/validate dimension"
Col->>Store : "UpsertPoints(persist first)"
Store-->>Col : "Success/failure"
Col->>Idx : "Upsert(update in-memory index)"
Idx-->>Col : "Success/failure"
Col-->>API : "Return status"
API-->>Client : "Response"
Note over Col,Store : "If index update fails, try to delete written points to maintain consistency"

Detailed Component Analysis

Storage and Persistence (BoltDB)

  • Collection buckets: Each collection corresponds to a bucket, key is point ID, value is Protobuf serialized point object.
  • Metadata bucket: Saves collection configuration and parameters, used for automatic collection rebuild after restart.
  • Batch operations: UpsertPoints/DeletePoints execute in a single transaction, ensuring atomicity.
  • Close semantics: Close is idempotent and can be called multiple times, ensures data flush.
flowchart TD
Start(["Start Upsert"]) --> CheckClosed{"Storage closed?"}
CheckClosed --> |Yes| ErrClosed["Return error"]
CheckClosed --> |No| TxBegin["Start bbolt transaction"]
TxBegin --> GetBucket["Get collection bucket"]
GetBucket --> Exists{"Bucket exists?"}
Exists --> |No| ErrBucket["Return collection not found error"]
Exists --> |Yes| LoopPoints["Iterate points to write"]
LoopPoints --> QuantCheck{"Quantization enabled?"}
QuantCheck --> |Yes| Quantize["Compress vector and store in payload"]
QuantCheck --> |No| KeepOrig["Keep original vector"]
Quantize --> Marshal["Protobuf serialize"]
KeepOrig --> Marshal
Marshal --> Put["Put(ID, data)"]
Put --> NextPoint{"More points?"}
NextPoint --> |Yes| LoopPoints
NextPoint --> |No| Commit["Commit transaction"]
Commit --> Done(["Done"])
ErrBucket --> Done
ErrClosed --> Done

Collection and Consistency (Upsert/Delete)

  • Version number: Each Upsert generates nanosecond timestamp as version number, convenient for subsequent audit and sorting.
  • Write order: Write to storage first, then update index; if index update fails, try to delete written points, best-effort rollback.
  • Delete strategy: Delete from storage first, then delete from index; if index deletion fails, keep residual points in storage to avoid data loss.
sequenceDiagram
participant Col as "Collection"
participant Store as "Storage"
participant Idx as "Index"
Col->>Store : "UpsertPoints"
Store-->>Col : "Success/failure"
alt Success
Col->>Idx : "Upsert"
Idx-->>Col : "Success/failure"
alt Failure
Col->>Store : "DeletePoints(rollback)"
Store-->>Col : "Best effort"
end
else Failure
Col-->>Caller : "Return error"
end

Index and Filtering (HNSW/Flat)

  • HNSW: Supports Cosine/Euclid/Dot distance functions; search phase uses "oversampling + post-filter" strategy to ensure filtering accuracy.
  • Flat index: Brute-force search, suitable for small scale or scenarios requiring exact results.
  • Filtering: Performs exact/range/prefix/contains/regex match on payload fields through MatchFilter.
classDiagram
class VectorIndex {
    +Upsert(points) error
    +Search(query, filter, topK) []ScoredPoint
    +Delete(id) error
    +Count() int
    +GetIDsByFilter(filter) []string
    +DeleteByFilter(filter) ([]string, error)
}
class HNSWIndex {
    -graph Graph
    -points map
    -metric Distance
    -params HNSWParams
    +Upsert(...)
    +Search(...)
    +Delete(...)
    +Count()
    +GetIDsByFilter(...)
    +DeleteByFilter(...)
}
class FlatIndex {
    -points map
    -metric Distance
    +Upsert(...)
    +Search(...)
    +Delete(...)
    +Count()
    +GetIDsByFilter(...)
    +DeleteByFilter(...)
}
VectorIndex <|.. HNSWIndex
VectorIndex <|.. FlatIndex

API and Concurrency Control

  • API server locks internal collection map to ensure thread safety of collection registration and query.
  • HTTP layer returns 400 for request parsing failure, 500 for business errors, 404 for collection not found.
  • Graceful shutdown: Supports timeout context, ensuring sufficient time for cleanup during shutdown.
sequenceDiagram
participant Client as "Client"
participant API as "API Server"
participant Col as "Collection"
participant Store as "Storage"
Client->>API : "POST /collections/{name}/points/delete"
API->>API : "Parse request/validate JSON"
API->>Col : "Delete(points, filter)"
Col->>Store : "DeletePoints(delete from storage first)"
Store-->>Col : "Return"
Col->>Col : "Delete from index"
Col-->>API : "Return delete count"
API-->>Client : "200 + result"

Dependency Analysis

  • Storage depends on bbolt and Protobuf; optional dependency on quantizer.
  • Collection depends on storage and index interface; index implementation depends on mathematical metrics.
  • API server depends on collection and storage; command-line entry depends on API and storage.

```mermaid