What is the Gaia benchmark for generalist AI agents? (Answer from Perplexity)

on

The GAIA benchmark is a comprehensive evaluation framework designed to assess the capabilities of general AI assistants by testing fundamental cognitive abilities that humans find simple but current AI systems struggle with. Introduced by researchers including Meta's team, it focuses on real-world problem-solving through reasoning, multi-modality handling, web browsing, and tool usage14.

Key Features

Multi-level structure:
Tasks are divided into three difficulty tiers:

  • Level 1: Solvable by top-tier LLMs (67.92% accuracy for leading models)

  • Level 2: Requires advanced tool integration (54.65% top accuracy)

  • Level 3: Demands human-like adaptability (30.77% best performance)1

Task diversity:
Includes 450-466 questions spanning:

  • Visual puzzles requiring image analysis

  • Web navigation challenges

  • Multi-step calculations with real-world data

  • Tool-assisted problem solving12

Performance Insights

Current leaderboard standings (as of March 2025) show:

ModelOverall AccuracyCost (USD)
Claude-3-5-Sonnet57.58%$260.19
Claude-3-7-Sonnet56.36%$409.01
GPT-4o Mini13.94%$18.381

Notably, humans achieve 92% accuracy on these tasks compared to 15% for GPT-4 with plugins, highlighting the benchmark's effectiveness in exposing AI limitations48.

Evaluation Dimensions

  1. Task Execution: Completion rate and error frequency

  2. Adaptability: Response to novel scenarios

  3. Efficiency: Computational cost and time metrics

  4. Generalization: Performance on unseen task variants24

The benchmark serves as a critical milestone for AGI development, emphasizing human-like robustness over narrow expertise. By focusing on everyday challenges rather than specialized domains, GAIA provides a clearer measure of progress toward truly general artificial intelligence48.

Citations:

  1. https://hal.cs.princeton.edu/gaia
  2. https://workos.com/blog/gaia-benchmark-evaluating-intelligent-agents
  3. https://www.reddit.com/r/singularity/comments/181vthx/benchmark_for_general_ai_assistant_by_meta/
  4. https://ai.meta.com/research/publications/gaia-a-benchmark-for-general-ai-assistants/
  5. https://arxiv.org/abs/2311.12983
  6. https://openreview.net/forum?id=fibxvahvs3
  7. https://huggingface.co/gaia-benchmark
  8. https://ukgovernmentbeis.github.io/inspect_evals/evals/assistants/gaia/

Answer from Perplexity