Skip to content

Upgrade and Migration

This guide is for GoVector users upgrading from older versions to newer versions, providing version compatibility checking methods, pre-upgrade preparation, progressive upgrade strategies (blue-green and rolling), data format and migration steps (vectors and metadata), configuration file differences and migration注意事项, rollback and emergency handling, upgrade testing and verification methods, and multi-node cluster coordinated upgrade steps. The goal is to help you complete upgrades with low risk and traceable manner.

Project Structure

GoVector uses a layered organization of "command-line entry + API service + core engine":

  • Command-line entry: Start independent service or embedded library
  • API layer: Provides Qdrant-compatible REST interfaces
  • Core layer: Collection management, index (HNSW/Flat), storage (BoltDB + Protobuf)
  • Build and release: Cross-platform packaging and Homebrew formula
graph TB
subgraph "Runtime"
CLI["Command-line entry
cmd/govector/main.go"]
API["HTTP API Server
api/server.go"]
CORE["Core engine
core/collection.go
core/storage.go"]
STORE["Persistent storage
BoltDB + Protobuf"]
end
subgraph "Build and Release"
MK["Build script
Makefile"]
REL["Release script
scripts/build_release.sh"]
HOMEBREW["Homebrew formula
scripts/release/govector.rb"]
end
CLI --> API --> CORE --> STORE
MK --> CLI
REL --> CLI
HOMEBREW --> CLI

Core Components

  • Storage and serialization: Local storage based on bbolt, uses Protobuf to serialize point data; supports optional vector quantization (SQ8).
  • Collection and metadata: Each collection corresponds to a bucket in bbolt, and collection configuration (dimension, metric, whether HNSW is enabled and parameters) is saved in a special metadata bucket.
  • Index: Supports Flat (exact) and HNSW (approximate) two index types, HNSW parameters are adjustable.
  • API: Provides REST interfaces for collection management and point operations, automatically loads persisted collections.
  • Quantizer: Provides 8-bit scalar quantization interface and implementation, used to reduce disk usage and memory pressure.

Architecture Overview

The diagram below shows key points of system interaction and data flow changes before and after upgrade, facilitating identification of compatibility risks and migration paths.

graph TB
subgraph "Before Upgrade"
P1["Old version service process"]
D1["Old version data file
govector.db"]
end
subgraph "After Upgrade"
P2["New version service process"]
D2["New version data file
govector.db"]
end
subgraph "Migration tool/process"
MIG["Migration tool/script"]
end
P1 --> D1
P2 --> D2
MIG --> D1
MIG --> D2

Detailed Component Analysis

Version Compatibility Check and Pre-upgrade Preparation

  • Language and dependency versions
  • Go version requirement: The Go version declared in the module must meet the new version requirements.
  • External dependencies: Changes in HNSW, bbolt, Protobuf versions may affect behavior or performance.
  • Data files and metadata
  • Collection metadata is stored in a special metadata bucket, automatically loaded on restart.
  • On startup, collection metadata is read and collection instances are rebuilt, ensuring configuration consistency.
  • Configuration items and parameters
  • HNSW parameters (M, EfConstruction, EfSearch, K) are persisted in collection metadata and can be restored after upgrade.
  • Quantization switch and quantizer implementation are determined at storage initialization, can remain consistent after upgrade or switched as needed.

Progressive Upgrade Strategies

  • Blue-green deployment
  • Prepare two environments: current (blue) and newly created (green).
  • Switch traffic from blue to green, close blue after verification is correct.
  • Suitable for scenarios requiring zero downtime and quick rollback.
  • Rolling upgrade
  • Restart nodes in batches, replacing only some instances at a time, continuously monitoring health status.
  • Suitable for online services, reducing the impact of single change.
  • Multi-node cluster coordination
  • Unify versions and configurations, upgrade control plane first (e.g., configuration center), then rolling upgrade nodes one by one.
  • Keep external service available during upgrade, downgrade to read-only or brief downtime window if necessary.

[This section is general strategy description, not directly analyzing specific files, hence no section source]

Data Format Changes and Migration Steps

  • Protobuf data model
  • Point structure contains id, vector array, payload map; scored result contains id, version, score, payload.
  • Conditions and filters support multiple matching types (exact, range, prefix, contains, regex).
  • Vector data migration
  • If quantization is enabled, vectors are compressed before storage and decompressed on load; can keep or switch quantization strategy after upgrade.
  • During migration, recommend stopping writes first, exporting all collections, then importing to new version database.
  • Metadata migration
  • Collection metadata (name, dimension, metric, HNSW switch and parameters) is stored in metadata bucket, automatically restored after upgrade.
  • During migration, ensure metadata bucket exists and is readable to avoid startup failure.
flowchart TD
Start(["Start migration"]) --> StopWrites["Stop writes/freeze writes"]
StopWrites --> Export["Export collection data"]
Export --> Convert["Convert format/parameters as needed"]
Convert --> Import["Import to new version database"]
Import --> Verify["Verify data integrity and query correctness"]
Verify --> Resume["Resume writes/switch traffic"]
Resume --> End(["End"])

Configuration File Version Differences and Migration Notes

  • Command-line parameters
  • Parameters such as service port, database path, whether to enable HNSW are parsed at startup.
  • If new parameters are added after upgrade, supplement in startup script.
  • API configuration
  • API server automatically loads persisted collections, no need to manually maintain collection list.
  • Homebrew service configuration
  • Service formula defines default port, database path and startup parameters, need to synchronize updates after upgrade.
  • Build and release
  • Cross-platform packaging script generates platform archives and checksums, version number and checksums need to be synchronized.

Upgrade Rollback Strategy and Emergency Handling

  • Rollback strategy
  • Keep old version binary and data file copies, rollback by restoring old version and restarting.
  • If quantization is enabled, can close quantization after rollback to avoid compatibility issues.
  • Emergency handling
  • Startup failure: Check logs and whether metadata bucket exists; confirm collection dimension and metric are consistent.
  • Query exception: Check index parameters and filter conditions; if necessary, fall back to Flat index for comparative verification.
  • Write failure: Check storage error logs; confirm bbolt transaction committed successfully; rollback and retry if necessary.

Upgrade Testing Process and Verification Methods

  • Functionality verification
  • Create/delete collection, batch write, search and filtering, delete and other basic operations.
  • Performance verification
  • Compare latency and throughput under same hardware and data scale, ensure upgrade does not introduce performance regression.
  • Compatibility verification
  • Compare query result consistency across versions, especially when enabling/disabling quantization.
  • Stress testing
  • Simulate high-concurrency writes and queries, observe system stability and resource usage.

Dependency Analysis

  • External dependencies
  • HNSW graph algorithm library: Provides approximate nearest neighbor search capability.
  • bbolt: Key-value database, provides transactions and persistence.
  • Protobuf: Serializes points and metadata.
  • Internal coupling
  • API layer depends on core engine; core engine depends on storage layer; collection and index cooperate with each other.
  • Quantizer is pluggable, does not affect upper-layer interfaces and data model.
graph LR
API["api/server.go"] --> COL["core/collection.go"]
COL --> IDX_H["core/hnsw_index.go"]
COL --> IDX_F["core/flat_index.go"]
COL --> ST["core/storage.go"]
ST --> PB["core/proto/point.proto"]
ST --> BB["bbolt"]
IDX_H --> HNSW["coder/hnsw"]

Performance Considerations

  • HNSW parameter tuning
  • M, EfConstruction, EfSearch, K affect build and query performance, need to adjust based on business scenarios.
  • Quantization strategy
  • SQ8 can significantly reduce storage and memory usage, but introduces compression/decompression overhead; need to trade off accuracy and performance.
  • Concurrency and locking
  • Both collection and index use read-write locks for protection, recommend evaluating lock contention under high concurrency after upgrade.

Troubleshooting Guide

  • Startup failure
  • Check database file permissions and path; confirm metadata bucket exists and is readable.
  • Query exception
  • Verify query vector dimension matches collection dimension; check filter conditions and payload types.
  • Write failure
  • Check storage error logs; confirm bbolt transaction committed successfully; rollback and retry if necessary.
  • Service graceful shutdown
  • Triggered by signals, ensures requests are completed and resources released.

Conclusion

By following the compatibility checking, progressive upgrade strategies, data migration steps and testing verification processes in this guide, you can safely upgrade GoVector from old to new versions. It is recommended to first rehearse in non-production environments, then execute in batches in production environments, and prepare rollback plans and emergency response processes.

Appendix

  • Build and release
  • Use Makefile to build binaries; use release script to generate cross-platform archives and Homebrew formula.
  • Version number and checksum

  • Release script automatically updates version number and platform checksums, ensuring installation consistency.

  • scripts/release/govector.rb:4