Learning¶

6.1 Learning Path Overview¶

flowchart TD
    subgraph Getting Started
        A["Understand basic concepts"] --> B["Vector basics"]
        B --> C["Similarity metrics"]
        C --> D["Embedding basics"]
    end

    subgraph Intermediate
        E["Index algorithms"] --> F["HNSW principles"]
        F --> G["ANN principles"]
        G --> H["Distance metric comparison"]
    end

    subgraph Hands-on
        I["Vector database selection"] --> J["Milvus"]
        J --> K["Qdrant"]
        K --> L["Pinecone"]
    end

    subgraph Advanced
        M["RAG development"] --> N["Recommendation systems"]
        N --> O["Multi-modal retrieval"]
        O --> P["AI Agent architecture"]
    end

    A --> E
    D --> I
    H --> M

    style A fill:#e1f5fe
    style E fill:#fff3e0
    style I fill:#c8e6c9
    style M fill:#e1f5fe

6.2 Getting Started Phase (1-2 weeks)¶

Learning Objectives¶

Recommended Learning Resources¶

Online Tutorials¶

1. Vector Database Fundamentals
2. Pinecone Blog: "What is a Vector Database?"

3. Qdrant Blog: "Vector Search Explained"

4. Embedding Getting Started

5. OpenAI Embeddings Guide
6. Sentence-Transformers Documentation

Hands-on Projects¶

# Your first vector search program as a beginner
from sentence_transformers import SentenceTransformer
import numpy as np

# 1. Load pre-trained model
model = SentenceTransformer('all-MiniLM-L6-v2')

# 2. Prepare data
sentences = [
    "I like eating apples",
    "iPhone is Apple company's product",
    "The weather is nice today",
    "Apple is a fruit"
]

# 3. Generate vectors
vectors = model.encode(sentences)

# 4. Query
query = "How's the iPhone?"
query_vector = model.encode([query])

# 5. Calculate similarity
similarities = np.dot(query_vector, vectors.T)[0]
most_similar_idx = np.argmax(similarities)

print(f"Most relevant sentence: {sentences[most_similar_idx]}")
print(f"Similarity: {similarities[most_similar_idx]:.3f}")

6.3 Intermediate Phase (2-4 weeks)¶

Learning Objectives¶

Core Algorithm Deep Dive¶

HNSW Algorithm Learning Path¶

graph TD
    subgraph HNSW Learning Steps
        A["Understand graph structure"] --> B["Understand layering concept"]
        B --> C["Understand greedy search"]
        C --> D["Understand parameter tuning"]
    end

    subgraph Learning Resources
        E["HNSW paper"]
        F["FAISS documentation"]
        G["Qdrant technical blog"]
    end

    E --> A
    F --> B
    G --> C

Key Parameter Tuning Guide¶

6.4 Hands-on Phase (4-8 weeks)¶

Vector Database Selection¶

graph TD
    subgraph Open Source Self-Hosted
        A["Milvus
Most full-featured
K8s friendly"] --> A1["Large-scale deployment"]
        B["Qdrant
Rust implementation
Excellent performance"] --> B1["High performance requirements"]
        C["Weaviate
Built-in vectorization
Ready to use"] --> C1["Quick prototyping"]
    end

    subgraph Cloud Services
        D["Pinecone
Fully managed
Easy to use"] --> D1["Don't want DevOps"]
        E["Azure AI Search
Enterprise-grade
Microsoft ecosystem"] --> E1["Enterprise users"]
    end

    subgraph Lightweight
        F["Chroma
LLM-oriented
Local运行"] --> F1["Personal projects
RAG prototyping"]
        G["FAISS
Facebook open source
Research首选"] --> G1["Deep customization"]
    end

Mainstream Vector Database Comparison¶

Quick Start Examples¶

Milvus Example¶

from pymilvus import connections, Collection, CollectionSchema, Field, data_types
from sentence_transformers import SentenceTransformer

# 1. Connect to Milvus
connections.connect(host="localhost", port="19530")

# 2. Define Collection Schema
fields = [
    Field(name="id", type=DataTypes.INT64, is_primary=True),
    Field(name="text", type=DataTypes.VARCHAR, max_length=600),
    Field(name="embedding", type=DataTypes.FLOAT_VECTOR, dim=384)
]
schema = CollectionSchema(fields, "text collection")

# 3. Create Collection
collection = Collection("texts", schema)
collection.create_index("embedding", {"index_type": "HNSW", "params": {"M": 16, "efConstruction": 200}})

# 4. Insert data
model = SentenceTransformer('all-MiniLM-L6-v2')
texts = ["Text 1", "Text 2", "Text 3"]
vectors = model.encode(texts)

entities = [
    [1, 2, 3],
    texts,
    vectors.tolist()
]
collection.insert(entities)

# 5. Search
query_vector = model.encode(["Query text"])
results = collection.search([query_vector], "embedding", {"params": {"ef": 100}}, limit=10)

Qdrant Example¶

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
from sentence_transformers import SentenceTransformer

# 1. Connect to Qdrant
client = QdrantClient(host="localhost", port=6333)

# 2. Create Collection
client.create_collection(
    collection_name="texts",
    vectors_config=VectorParams(size=384, distance=Distance.COSINE)
)

# 3. Insert data
model = SentenceTransformer('all-MiniLM-L6-v2')
texts = ["Text 1", "Text 2", "Text 3"]
vectors = model.encode(texts).tolist()

points = [
    PointStruct(id=idx, vector=vec, payload={"text": text})
    for idx, (vec, text) in enumerate(zip(vectors, texts))
]
client.upsert(collection_name="texts", points=points)

# 4. Search
query_vector = model.encode(["Query text"]).tolist()
results = client.search(collection_name="texts", query_vector=query_vector, limit=10)

6.5 Advanced Phase (8+ weeks)¶

RAG Development Learning Path¶

flowchart LR
    subgraph RAG Components
        A["Document Processing"] --> B["Vector Retrieval"]
        B --> C["Prompt Engineering"]
        C --> D["Generation Model"]
    end

    subgraph Tech Stack
        E["LangChain"]
        F["LlamaIndex"]
        G["Semantic Kernel"]
    end

    A -.->|Tools| E
    A -.->|Tools| F
    B -.->|Tools| E
    B -.->|Tools| F

RAG Advanced Techniques¶

6.6 Recommended Learning Resources¶

Official Documentation¶

Technical Blogs¶

Video Courses¶

Open Source Projects¶

6.7 Learning Suggestions¶

Progressive Learning Plan¶

gantt
    title Vector Database Learning Plan
    dateFormat  YYYY-MM-DD
    section Getting Started
    Understand basic concepts       :a1, 2024-01-01, 7d
    Learn vector operations         :a2, 2024-01-08, 7d
    Embedding basics               :a3, 2024-01-15, 7d
    section Intermediate
    HNSW algorithm                :b1, 2024-01-22, 14d
    ANN principles                 :b2, 2024-02-05, 14d
    section Hands-on
    Milvus deployment              :c1, 2024-02-19, 14d
    RAG development                :c2, 2024-03-04, 21d

FAQ¶

Q: How to choose between vector databases and traditional databases?

A: If your application needs: - Semantic understanding → Vector database - Exact matching → Traditional database - Both → Use both (e.g., PostgreSQL + pgvector)

Q: How to choose vector dimension?

A: Reference these principles: - General text: 384-768 dimensions (Sentence-Transformers) - High-quality semantics: 1024-1536 dimensions (OpenAI) - Image features: 512-2048 dimensions - Trade-off: Higher dimension = higher accuracy, but also higher storage and retrieval cost

Q: How to evaluate vector database performance?

A: Main metrics: - QPS: Queries per second - Latency: P50/P99 query latency - Recall: Match rate between ANN results and exact results - Memory usage: Index size

Next Learning Directions¶

Chapter Summary¶

Key Points:

1. Learning Path:
2. Getting started (1-2 weeks): Basic concepts, embedding, similarity metrics
3. Intermediate (2-4 weeks): Index algorithms, ANN principles, parameter tuning
4. Hands-on (4-8 weeks): Database deployment, API usage, project development

5. Advanced (8+ weeks): RAG, recommendation systems, multi-modal

6. Database Selection:

7. Production environment: Milvus, Qdrant
8. Quick prototyping: Chroma, Pinecone

9. Research/customization: FAISS

10. Resource Recommendations:

11. Official documentation is the best learning material
12. GitHub open source projects are great for practice
13. Technical blogs for latest developments

End of Document

Congratulations on completing all the content of the Vector Database Getting Started Guide! You now have mastery of:

• ✅ Basic concepts of vector databases
• ✅ Technology development history
• ✅ Core principles and index structures
• ✅ Basic concepts and terminology
• ✅ Typical application scenarios
• ✅ Learning paths and resource recommendations

Happy learning! Put this knowledge into practice in real projects 🚀

Return to Table of Contents