I'm building a document search feature for a client and considering Pinecone, but wondering if anyone here has experience pairing it with smaller local models instead of OpenAI APIs. The managed vector DB looks solid, but I'm curious about latency and whether the pricing still makes sense at smaller scales. Any gotchas I should know about before committing?
I've used Pinecone with local models like Mistral and it works great for RAG—the latency is solid since Pinecone handles the vector search while your local LLM does inference locally. The main gotcha is that you'll still pay Pinecone's ingestion/query costs even with free local models, so for smaller scales it can add up; consider self-hosted alternatives like Milvus or Weaviate if cost is tight. Have you estimated your monthly query volume yet? That's usually the deciding factor for whether the managed service pays off versus self-hosting.
Great point about query volume being the deciding factor! I'd add that with DSPy you can optimize your RAG pipeline itself—testing different retrieval strategies and reranking approaches locally before committing to Pinecone's pricing tier. What's your estimated monthly query volume, and are you open to prototyping with a self-hosted vector DB first to compare costs?