Quivr's vision

Hello, many of you have noticed some changes on Quivr's Github Repository. Let me explain to you what happened in this article.

To ensure the long-term sustainability and innovation of RAG-based solutions, we are glad to announce a strategic pivot: separating our deep tech components from our enterprise product.

Why This Separation Is Necessary

1. Clear Focus on Core Technology:
2. Our deep tech components - Megaparse, Quivr Core, and Le Juge - are the foundation of our RAG technology. Separating these components will allow us to focus on refining their capabilities, ensuring they continue to push the limit of the state of the art.
3. Driving Open-Source Collaboration:
4. We believe in the power of open-source. By making these technologies publicly available, we invite all contributors to drive their evolution.
5. Sustained Innovation for Enterprise:
6. While the deep tech components will remain open-source, Quivr Enterprise, our closed-source product, will focus on delivering business-ready features built on top of these technologies.

How We’ll Achieve It

To implement this strategy, we will maintain three open-source repositories:

1. Megaparse

Parsing Engine for Context Extraction

• What it does: Megaparse is a robust tool for parsing and extracting meaningful context from unstructured data.
• Roadmap:
• In the short term, the focus is on optimizing the parsing of various file formats like PDF, Docx, PPTX, TXT, and EPUB. We will after focus on  enhancing table parsing and delivering more structured outputs. On the long term, the goal is to achieve state-of-the-art performance.

2. Quivr Core

Retrieval System for Relevant Knowledge

• What it does: Quivr Core powers the retrieval of precise, contextually relevant knowledge from vast datasets.
• Roadmap:
• It will at first focus on implementing a single retrieval algorithm. Medium-term plans include developing multiple algorithms tailored to different use cases.

3. Le Juge

Evaluation Framework for RAG Systems

• What it does: Le Juge evaluates the quality, relevance, and accuracy of RAG outputs.
• Roadmap:
• Initially, it will focus on Quivr's RAG, then expand to evaluate all RAG systems

The Vision for Quivr Enterprise

Quivr Enterprise is an example of our open-source technologies usage. This closed-source solution will focus on delivering business features such as:

• Knowledge Management Systems (KMS): Storage and classification of all knowledge.
• Sharing Functionalities: Allow collaboration across teams.
• AI-Powered Chatbots: Tailored conversational interfaces.
• Integrations: Connect to Google Drive, SharePoint, or Dropbox to centralize and use your knowledge.

None of Quivr Enterprise’s closed-source features will impact Quivr RAG. Core RAG improvements will remain open-source, following our principles.

Why These Three Close Source Repositories?

These three components — Parsing, Retrieval, and Evaluation — are essential for ensuring the efficiency of a RAG system. Here's why each one is fundamental:

• Parsing to understand and structure information:Parsing is crucial for transforming data into an actionable format. Without it, information may remain too vague or unstructured to be effectively used. This step helps identify key elements within a document, extract important relationships and concepts, and prepares the data for subsequent stages like retrieval and evaluation.
• Retrieval to find what matters most:Retrieval is the core of searching for relevant information. It ensures that the most pertinent data is retrieved from large knowledge bases or corpora. A robust retrieval system guarantees that the information extracted during parsing is used to access contextually appropriate and specific data. This step is key to maximizing the quality of results, as, without it, irrelevant or inadequate information might be retrieved.
• Evaluation to ensure quality and relevance:Evaluation is critical for validating that the retrieved information is of high quality and truly relevant. It acts as a filtering step, ensuring that incorrect, unnecessary, or irrelevant data is discarded. Evaluation is particularly important in automated systems, as it helps measure result quality based on defined criteria (accuracy, relevance, coherence, etc.). This ensures that the system doesn't just retrieve data, but does so in an optimal and context-specific manner.

By combining these three components, you create a modular, flexible, and robust RAG system. Each step can be improved or adjusted independently to meet specific needs while allowing the entire system to function efficiently. Quivr Enterprise, as a demonstration, showcases the concrete impact of this approach, enabling fast, accurate search with reliable and high-quality results.

Conclusion

This separation of concerns is a pivotal step in aligning our mission with our vision for the future of RAG. By contributing to the open-source community while maintaining a clear boundary for enterprise innovation, we can accelerate progress in the RAG area while delivering value to businesses.

‍