koan-vector-migration

What problem does it solve?

Migrating between different vector databases typically requires regenerating all embeddings, which is costly in terms of AI API calls and time. This Skill provides a strategy to export and import vectors without regeneration, enabling zero-cost migration between compatible vector stores.

Core Features & Use Cases

• Zero-Cost Migration: Export existing embeddings from one vector database and import them into another without incurring AI API costs for re-embedding.
• Embedding Caching: Cache embeddings after generation or export to facilitate quick transfers and reduce reliance on AI services during migration.
• Provider Support: Supports export from ElasticSearch and Weaviate, with planned support for Qdrant and Milvus.
• Use Case: You've built a semantic search feature using Weaviate, but now need to switch to ElasticSearch for better integration with your existing stack. Use this Skill to export all your product embeddings from Weaviate, update your configuration to ElasticSearch, and then import the cached embeddings, all without making new calls to your embedding model.

Quick Start

To export vectors from Weaviate to a cache, then import to ElasticSearch:

1. Export from Weaviate: using (EntityContext.Adapter("weaviate")) { var vectorRepo = sp.GetRequiredService<IVectorSearchRepository<Media, string>>(); await foreach (var batch in vectorRepo.ExportAllAsync(batchSize: 100)) { await cache.SetAsync(batch.Id, "ada-002", batch.Embedding); } }
2. Update appsettings.json to use elasticsearch adapter.
3. Import to ElasticSearch: foreach (var mediaId in allMediaIds) { var embedding = await cache.GetAsync(mediaId, "ada-002"); if (embedding != null) { var media = new Media { Id = mediaId, Embedding = embedding }; await media.Save(); } }