Skip to content

Collection Management API

This document is for users and maintainers of the Collection Management API, systematically describing HTTP endpoints related to collections:

  • POST /collections: Create collection
  • DELETE /collections/{name}: Delete collection
  • GET /collections: List collections
  • GET /collections/{name}: Get collection info

It also explains in detail the role and value ranges of collection configuration parameters (such as vector_size, distance, hnsw, parameters), and provides abundant JSON examples to demonstrate different types of collection creation methods (including HNSW parameter configuration and distance metric selection). Finally, it describes collection lifecycle management and status inspection methods.

Project Structure

The service uses a layered architecture:

  • API Layer: Provides HTTP interface, routes to specific handler functions
  • Core Business Layer: Collection abstraction, index engine (Flat/HNSW), storage engine (BoltDB)
  • Command Line Entry: Starts standalone microservice mode
graph TB
subgraph "API Layer"
S["Server
HTTP Routing and Handling"]
end
subgraph "Core Business Layer"
C["Collection
Collection Abstraction"]
I["Index
Flat/HNSW"]
ST["Storage
Persistence (BoltDB)"]
end
subgraph "Command Line Entry"
M["main.go
Start Service"]
end
S --> C
C --> I
C --> ST
M --> S

Core Components

  • Server: HTTP server responsible for routing and collection management endpoint handling
  • Collection: Collection abstraction encapsulating vector dimension, distance metric, index, and storage
  • Index: Index interface supporting both Flat and HNSW implementations
  • Storage: Local persistence engine based on BoltDB and Protobuf
  • HNSWParams: HNSW index tunable parameter set

Architecture Overview

The key flows for Collection Management API are as follows: - Create collection: Parse request body, validate parameters, construct Collection and register to Server - Delete collection: Remove collection instance from Server - List collections: Iterate through Server's collections and return basic info - Get collection: Return name, dimension, distance metric, and point count of specified collection

sequenceDiagram
participant Client as "Client"
participant Server as "Server"
participant Col as "Collection"
participant Store as "Storage"
Client->>Server : "POST /collections"
Server->>Server : "Parse request body
Validate parameters"
Server->>Col : "NewCollectionWithParams(...)"
Col->>Store : "EnsureCollection / SaveCollectionMeta"
Server->>Server : "Register to collections map"
Server-->>Client : "200 OK + result"
Client->>Server : "DELETE /collections/{name}"
Server->>Server : "Remove from map"
Server-->>Client : "200 OK + result"
Client->>Server : "GET /collections"
Server->>Server : "Iterate collections"
Server-->>Client : "200 OK + list"
Client->>Server : "GET /collections/{name}"
Server->>Col : "Query collection info"
Server-->>Client : "200 OK + details"

Detailed Component Analysis

Endpoint Specifications and Behavior

POST /collections (Create Collection)

  • Request body fields
  • name: string, unique collection identifier
  • vector_size: positive integer, vector dimension
  • distance: string, distance metric, supports "euclidean"/"Euclid", "dot"/"Dot", "cosine"/"Cosine"
  • hnsw: boolean, whether to use HNSW index (optional)
  • parameters: object, HNSW parameters (optional)
    • m: integer, maximum connections per node
    • ef_construction: integer, candidate list size during construction
    • ef_search: integer, candidate list size during search
    • k: integer, number of neighbors to return
  • Success response
  • Status code: 200
  • Return fields: status, result.name, result.vector_size, result.distance, result.hnsw, result.parameters, result.operation
  • Error response
  • 400: JSON parsing failed, vector_size not positive, invalid distance
  • 409: Collection already exists
  • 500: Internal error (creation failed)

Example request (create HNSW collection with custom parameters) - Request body - name: "mycollection" - vectorsize: 128 - distance: "euclidean" - hnsw: true - parameters: { "m": 16, "efconstruction": 128, "efsearch": 64, "k": 10 }

Example response - Status code: 200 - Return body contains collection metadata and parameters

DELETE /collections/{name} (Delete Collection)

  • Path parameters
  • name: collection name
  • Success response
  • Status code: 200
  • Return fields: status, result.operation, result.collection
  • Error response
  • 404: Collection does not exist

GET /collections (List Collections)

  • Success response
  • Status code: 200
  • Return fields: status, result (array), each item contains name, vectorsize, distance, pointscount
  • Error response
  • None (always 200)

GET /collections/{name} (Get Collection Info)

  • Path parameters
  • name: collection name
  • Success response
  • Status code: 200
  • Return fields: status, result.name, result.vectorsize, result.distance, result.pointscount
  • Error response
  • 404: Collection does not exist

Collection Configuration Parameters Explained

  • vector_size (Vector Dimension)
  • Type: Positive integer
  • Purpose: Fixed dimension of vectors in the collection; must match during insert and query
  • Value range: > 0

  • Impact: Memory usage, index construction cost, query performance

  • distance (Distance Metric)

  • Type: String
  • Values: "euclidean"/"Euclid", "dot"/"Dot", "cosine"/"Cosine"
  • Impact: Similarity calculation method, determines index and search strategy

  • Note: HNSW has different distance function adaptations for different metrics

  • hnsw (Enable HNSW)

  • Type: Boolean
  • Purpose: Controls whether to use HNSW approximate nearest neighbor index or Flat linear index

  • Default: true (can be controlled via parameters when starting from command line)

  • parameters (HNSW Parameters)

  • m: Maximum connections per node; larger makes the graph denser, higher recall but increased memory and construction time
  • ef_construction: Candidate list size during construction phase; larger means higher recall but slower construction
  • ef_search: Candidate list size during search phase; larger means higher recall but increased latency
  • k: Number of neighbors to return (HNSW search will over-sample then filter)

Collection Lifecycle and Status Inspection

  • Lifecycle
  • Create: Server.handleCreateCollection parses request, validates parameters, creates Collection and registers to Server
  • Running: Server maintains collections map, provides queries and operations
  • Delete: Server.handleDeleteCollection removes collection from map

  • Load: Server.Start automatically loads collection metadata from Storage and rebuilds collections on startup

  • Status Inspection

  • List: GET /collections returns basic collection info (includes points_count)
  • Details: GET /collections/{name} returns detailed collection info (includes points_count)

  • Existence: Delete and query endpoints return 404 when collection not found

  • Data Consistency

  • Collection.Upsert writes to Storage first, then updates in-memory index; if index update fails, attempts to rollback Storage write
  • Collection.Delete supports deletion by ID or by filter condition, deletes from Storage first then from index

End-to-End Sequence Diagram (Create Collection)

sequenceDiagram
participant Client as "Client"
participant Server as "Server"
participant Col as "Collection"
participant Store as "Storage"
Client->>Server : "POST /collections"
Server->>Server : "Parse JSON
Validate vector_size > 0"
Server->>Server : "Parse distance -> Metric"
Server->>Server : "Parse parameters -> HNSWParams"
Server->>Col : "NewCollectionWithParams(name, dim, metric, store, hnsw, params)"
Col->>Store : "EnsureCollection / SaveCollectionMeta"
Server->>Server : "collections[name] = col"
Server-->>Client : "200 OK + result"

Processing Logic Flowchart (Create Collection)

flowchart TD
Start(["Enter handleCreateCollection"]) --> Parse["Parse JSON request body"]
Parse --> ValidJSON{"JSON valid?"}
ValidJSON --> |No| Err400["Return 400 Bad Request"]
ValidJSON --> |Yes| CheckExist["Check if collection exists"]
CheckExist --> Exists{"Already exists?"}
Exists --> |Yes| Err409["Return 409 Conflict"]
Exists --> |No| ValidateDim["Validate vector_size > 0"]
ValidateDim --> DimOK{"Dimension valid?"}
DimOK --> |No| Err400
DimOK --> ParseMetric["Parse distance -> Metric"]
ParseMetric --> MetricOK{"Metric valid?"}
MetricOK --> |No| Err400
MetricOK --> ParseParams["Parse parameters -> HNSWParams"]
ParseParams --> CreateCol["NewCollectionWithParams(...)"]
CreateCol --> Reg["Register to collections map"]
Reg --> Ok["Return 200 OK + result"]

Dependency Analysis

graph LR
A["api/server.go"] --> B["core/collection.go"]
A --> C["core/models.go"]
B --> D["core/hnsw_index.go"]
B --> E["core/storage.go"]
F["cmd/govector/main.go"] --> A

Performance Considerations

  • HNSW vs Flat Selection
  • HNSW: Suitable for large-scale data, low query latency, but higher construction and memory overhead
  • Flat: Simple and direct, suitable for small-scale or temporary scenarios
  • HNSW Parameter Tuning
  • m: Affects graph density and recall; recommend starting from default value and gradually increasing while observing effects
  • ef_construction: Trade-off between construction quality and speed; larger is slower but better recall
  • ef_search: Trade-off between query latency and recall; larger is slower but more accurate
  • k: Search over-sampling multiplier; recommended to increase appropriately when using filters
  • Distance Metrics
  • Cosine: Often used for normalized vectors, suitable for text/image retrieval
  • Euclidean: Suitable for non-normalized Euclidean space
  • Dot: Direction-sensitive dot product; be aware of sign flip impact
  • Storage and Persistence
  • Enabling persistence allows automatic loading of collection metadata and point sets after restart
  • Supports vector quantization (SQ8) to reduce disk usage and memory pressure

Troubleshooting Guide

  • Error creating collection
  • 400: Check JSON format, whether vector_size is positive, whether distance is an allowed value
  • 409: Collection name conflict; change name or delete old collection first
  • 500: Internal error; check service logs to locate specific cause
  • Error deleting collection
  • 404: Collection does not exist; confirm name is correct
  • Error querying collection
  • 404: Collection does not exist; confirm name is correct
  • Data inconsistency
  • When Upsert/delete fails, service tries to rollback as much as possible, but still recommend checking Storage and index status

Conclusion

The Collection Management API provides Qdrant-compatible REST interfaces, covering collection creation, deletion, listing, and information queries. Through configuration options like vector_size, distance, hnsw, and parameters, users can flexibly select index types and parameters, balancing recall, latency, and resource consumption. With persistence and quantization capabilities, high-performance and reliable vector search service can be achieved in a standalone environment.