Overview

Project Optimus is an initiative to develop an advanced AI research assistant that builds upon and surpasses the capabilities of current solutions like Perplexity. While existing tools provide quick access to information and basic reasoning, Optimus aims for deeper, more comprehensive research capabilities by gathering information from hundreds or even thousands of sources, employing a multi-stage reasoning pipeline, and returning results that are meticulously verified, context-rich, and thoroughly cited.

Key Objective - Going Beyond Perplexity

Perplexity integrates large language models (LLMs) and real-time web searches to deliver informed answers to user’s questions. Optimus retains these benefits but focuses on more extensive data gathering, sophisticated source processing, and advanced reasoning:

1. Expanded Research:

   • Generate dozens of unique and optimized search queries per user question.
   • Aggregate hundreds or thousands of search results from a wide range of online sources.
   • Process a larger, more diverse set of sources to provide a fuller picture of a topic.

2. Sophisticated Reasoning:

   • Explore multiple agent-based reasoning frameworks to ensure precise, reliable conclusions.
   • Implement techniques that consider multiple candidate solutions before finalizing an answer.
   • Incorporate iterative improvement steps and parallel task execution where appropriate.

3. Enhanced Accuracy & Relevance:

   • Incorporate robust data cleaning, deduplication, and normalization.
   • Use vector databases for semantic retrieval, ensuring more contextually aligned references.
   • Integrate a re-ranking step to highlight the most relevant and credible information.

4. Comprehensive Results:

   • Deliver detailed citations, clarity on data provenance, and confidence scores.
   • Provide metadata about the research process, offering transparency into how final answers are constructed.

Current Implementation Progress

December 10, 2024

1. Query Generation: Takes the user’s question and generates dozens of optimized search engine queries to cast a wide net and ensure comprehensive coverage of the topic.

2. Multi-Engine Search: Multiple search engines (like Brave and DuckDuckGo) are utilized to run each search query generated in the previous step. These search results are processed, and the outputs are gathered within seconds, resulting in hundreds of unique sources.

3. Content Scraping & Extraction: Using both existing solutions and custom solutions, web scraping is performed for each URL gathered in the previous step. These scrapers process all URLs in parallel, handling HTML, PDFs, videos, and dynamic content. This content is then cleaned and refined to be in a more usable state for the model to process.

At this stage, steps 1 through 3 are complete, giving us a solid foundation of clean, normalized data.

Planned Enhancements

Databases & Vectorization

We are setting up the databases needed to make Optimus more efficient, as well as to archive the data extracted from the scrapes. A PostgreSQL database instance is being prepared to store this content, along with the reasoning actions performed by the models during processing. This will allow us to backtrack and analyze the model’s reasoning, improving the system over time.

Additionally, we are exploring various vector databases, chunking methods, and retrieval techniques to optimize the system and return the best results.

Agent-Based Reasoning

We are evaluating multiple reasoning architectures to deliver robust, contextually aware answers. These may involve:

• Iterative refinement techniques to improve initial responses.
• Tools to ground answers with additional data if the system identifies gaps in knowledge.
• Parallelizing certain tasks to accelerate the decision-making process.

Refinement & Post-Processing

Once the reasoning pipeline is selected and refined, the final answer—along with citations and metadata—will be compiled and validated using a top-tier model. Additional post-processing steps may ensure formatting consistency and the presence of all required references.

Technical Architecture

Current Workflow (Simplified)

User Query -> Query Generation -> Data Retrieval -> Data Cleaning -> Store In Traditional Database

Future Workflow (Planned)

Vectorize Data -> Reasoning Pipeline -> ReRanking & Verification -> Final Processing & Output

Note(s)

This evolving architecture prioritizes modularity and scalability, making it easier to incorporate advanced reasoning agents, improved re-ranking algorithms, and any future enhancements without overhauling the entire system.

The overall pipeline here will make Optimus slower compared to Perplexity, with an estimated response time of 5 to 20 minutes as development continues. However, Optimus examines a broader range of sources and utilizes advanced reasoning to deliver highly refined and comprehensive reports. This makes it particularly suited for deeper questions in fields such as politics, economics, and business research, where detailed analysis is vital. The insights provided by Optimus are intended to inform significant decisions and actions in these complex domains.

Development Roadmap

1. Phase 1 (Completed):

   • Query generation and initial data gathering
   • Data cleaning and storing in traditional database(s)

2. Phase 2 (In Progress):

   • Create vectorization pipeline and retrieval pipeline
   • Selection and integration of reasoning frameworks
   • Improved retrieval strategies and re-ranking methods

3. Phase 3 (Planned):

   • User interface development
   • System optimization and testing
   • Final deployment and iteration

Conclusion

Optimus aims to redefine the landscape of AI-driven research assistance. By scaling beyond the limitations of current tools, it will deliver answers grounded in a breadth of sources, refined through advanced reasoning, and presented with clear citations. The end result is an assistant not just capable of finding broad and unique information but of distilling knowledge and insights that guide users to more confident, informed conclusions.