Past battle · 2024-05-31 UTC
Research AI Agents Showdown — May 31, 2024
From the Research AI Agents category. 34 marks placed across 8 fighters. Google AI Co-Scientist took the crown.
Final standings
The line-up
The fighters
Profiles of every tool that competed in this battle, ranked by their final score.
Google AI Co-Scientist
A multi-agent AI system that collaborates with scientists to generate hypotheses and accelerate biomedical research.
The Google AI Co-Scientist is a multi-agent AI system designed to collaborate with scientists and accelerate biomedical research. It is built with Gemini 2.0 and functions as a virtual scientific collaborator to help generate novel hypotheses and research proposals. The system is intended to mirror the reasoning process underpinning the scientific method and uncover new, original knowledge. Given a scientist's research goal specified in natural language, the AI Co-Scientist generates novel research hypotheses, a detailed research overview, and experimental protocols. It uses a coalition of specialized agents, including Generation, Reflection, Ranking, Evolution, Proximity, and Meta-review, which use automated feedback to iteratively generate, evaluate, and refine hypotheses. The system allows scientists to interact with it in various ways, including providing seed ideas for exploration or feedback on generated outputs in natural language. The AI Co-Scientist also uses tools like web-search and specialized AI models to enhance the grounding and quality of generated hypotheses. The system is designed to flexibly scale compute and iteratively improve its scientific reasoning towards the specified research goal. The AI Co-Scientist is particularly useful for scientists who face challenges in navigating the rapid growth in scientific publications and integrating insights from unfamiliar domains. By leveraging recent AI advances, including the ability to synthesize across complex subjects and perform long-term planning and reasoning, the AI Co-Scientist aims to accelerate the clock speed of scientific and biomedical discoveries. The system has been tested in various applications, including gene transfer discovery and transfer re-discovery. While the AI Co-Scientist presents several advantages, its effectiveness depends on the quality of the input research goal and the scientist's ability to provide meaningful feedback. The AI Co-Scientist is a new tool that has the potential to significantly impact the scientific community, but its limitations and potential biases need to be carefully evaluated. The development of the AI Co-Scientist is an ongoing effort, and its future applications and improvements will depend on continued research and testing.
Criteria breakdown
- Hypothesis generation
- Research overview creation
- Experimental protocol development
- Specialized agents for scientific reasoning
- Automated feedback loop
- Web-search integration
Bright Data Web MCP
MCP server that gives AI agents reliable web search, scraping, and browser automation with 5,000 free monthly requests.
Bright Data's Web MCP is a server that enables AI agents to reliably search, scrape, and automate web browsers. It provides 5,000 free monthly requests. The tool allows AI agents to effectively search the web, extract data, and navigate websites without getting blocked. It offers features such as real-time search results, data extraction, website crawling, and browser automation. The Web MCP is designed to bypass blocks and restrictions, and is trusted by over 20,000 customers worldwide.
Criteria breakdown
- Real-time web search
- Website crawling and data extraction
- Browser automation
- Bypassing geo-restrictions and CAPTCHAs
- Rendering JavaScript for dynamic content
- Mimicking real user behavior
Kosmos
Autonomous AI scientist for long research campaigns that analyzes data and literature to produce fully cited scientific reports.
Kosmos is an autonomous AI scientist designed for biopharma R&D teams and researchers. It analyzes data and literature to produce fully cited scientific reports, accelerating the drug development process from hypothesis to registration in days, not months. Kosmos works autonomously, reading literature, generating hypotheses, steering investigations, and executing scientific workflows. It supports the full lifecycle of drug development, from target identification through clinical development and filing. The AI tool collaborates with human scientists through Slack, Teams, and email, and learns from the organization's knowledge, building on experimental history and proprietary scientific data.
Criteria breakdown
- Autonomous hypothesis generation
- Literature analysis
- Data analysis
- Scientific workflow execution
- Collaboration through Slack, Teams, and email
- Model agnostic for future state-of-the-art models
THEUS (Aigora)
Enterprise AI research system that turns proprietary studies into traceable insights with fact IDs, citations, and expert avatars for synthesis.
THEUS is an enterprise AI research system that transforms proprietary studies into traceable insights with fact IDs, citations, and expert avatars. It's built on Google's Agent Development Kit (ADK) and uses bidirectional modeling for data-grounded knowledge agents. Users can extract, synthesize, and explore their research data, and every claim is backed by a specific Fact ID with page-level citations. THEUS offers two ways to explore: generating new insights with Dr. Reed or analyzing existing knowledge with Dr. Sinclair. This platform is designed for VPs of Insights, Global Sensory Leads, and Innovation leaders evaluating strategic capability build-out. It uses a purpose-built research methodology, not generic enterprise AI, and creates data-grounded knowledge agents trained on actual institutional data. With 85% of new products failing within two years (Nielsen), evidence-grounded exploration dramatically improves decision quality. THEUS offers a solo seat, which includes features such as up to 30 documents ingested per month, up to 1,500 pages parsed per month, and up to 500 questions to Dr. Sinclair per month. One of the key benefits of THEUS is its ability to turn decades of proprietary data into traceable, decision-ready intelligence with AI-powered knowledge agents.
Criteria breakdown
- Extract, synthesize, and explore research data
- Generate new insights with Dr. Reed or analyze existing knowledge with Dr. Sinclair
- Bidirectional modeling for data-grounded knowledge agents
- Connect product and consumer knowledge for bidirectional insight
- Supports cross-study synthesis and research gap analysis
AMIE
A multimodal AI diagnostic agent that conducts clinical conversations and interprets medical images for accurate diagnoses.
AMIE (Articulate Medical Intelligence Explorer) is a research AI diagnostic agent developed by Google DeepMind and Google Research. It is designed to conduct clinical conversations and interpret medical images for accurate diagnoses. Initially, AMIE was a text-based medical diagnostic conversational AI agent published in Nature. The latest advancement, multimodal AMIE, integrates the ability to intelligently request, interpret, and reason about visual medical information during clinical conversations. This development aims to improve diagnostic accuracy by incorporating multimodal data, such as images and documents, into the diagnostic process. AMIE uses a state-aware reasoning framework and is built on multimodal Gemini models. It has been evaluated through expert assessments, including Objective Structured Clinical Examinations (OSCEs), comparing its performance to primary care physicians (PCPs) in various patient scenarios.
Criteria breakdown
- Multimodal diagnostic dialogue
- Visual medical information interpretation
- State-aware reasoning framework
- Integration with Gemini models
- Simulation environment for dialogue evaluation
QualiaInterviews
AI-led platform for multilingual, conversational research and evaluation interviews at scale.
QualiaInterviews is a research platform that uses AI to conduct interviews with participants through natural, conversational dialogue. It enables teams to gather qualitative insights from larger sample sizes than traditional one-on-one interviews allow, while preserving the depth and nuance of open-ended conversation. The tool supports interviews across multiple languages, making it suitable for cross-border studies, global market research, and program evaluation. Researchers can design interview guides, deploy them to participants, and review automatically structured findings without manually moderating each session. QualiaInterviews is typically used by social researchers, evaluators, UX teams, and organizations running impact assessments who need scalable qualitative data without sacrificing conversational depth.
Criteria breakdown
- AI-moderated conversational interviews
- Multilingual interview support
- Automated follow-up and probing questions
- Structured analysis of qualitative responses
- Scalable participant deployment
- Research and evaluation workflow tools
Table Agent
AI-powered data assistant for tabular research Automated data gathering on any topic Customizable data schemas AI agent-driven data search
Table Agent is an AI-powered data assistant designed to aid in tabular research. It is intended to automate the process of gathering data on any given topic, making research more efficient. The tool allows for customizable data schemas, which enables users to tailor their data collection to specific needs. This flexibility is beneficial for a wide range of applications and industries where data structure can vary significantly. At the core of Table Agent's functionality is its AI agent-driven data search capability. This means the tool utilizes artificial intelligence to actively seek out and compile relevant data, potentially reducing the time and effort required for manual searches. While the specifics of its workflow and integrations are not detailed, the concept of an AI-driven data assistant suggests it could seamlessly integrate with various data analysis tools, enhancing the overall research process. The strengths of Table Agent would likely include its ability to automate tedious data collection tasks and its adaptability to different data structures. However, without more specific information, it's challenging to determine its limitations or how it compares to alternative data assistant tools.
Criteria breakdown
- Automated data gathering
- Customizable data schemas
- AI agent-driven data search
- Quick and accurate data retrieval
Autoresearch
An open-source project that lets AI agents autonomously run LLM training experiments and keep the best model changes.
Autoresearch is an open-source project that enables AI agents to autonomously run LLM training experiments and retain the best model changes. The project allows users to set up a small but real LLM training environment and let an AI agent experiment with it overnight, modifying the code, training for a short period, and checking if the results improve. The goal is to automate the research process, letting the AI agent explore different model architectures, hyperparameters, and optimization strategies without human intervention. The project includes a simplified single-GPU implementation of nanochat and provides a basic structure for programming the AI agent's research process using Markdown files. The project is designed to be extensible, allowing users to add more agents and improve the research process over time.
Criteria breakdown
- Autonomous LLM training experiments
- AI agent-driven research process
- Single-GPU implementation of nanochat
- Markdown-based programming for the research process
- 5-minute training time budget with evaluation metric (val_bpb)