🤯 AI's New Benchmark: Smarter Than Ever? 🚀

WANDR: A New Benchmark for Knowledge Work Agents
This chapter introduces WANDR, Perplexity’s open benchmark designed to rigorously evaluate agents tasked with wide and deep knowledge collection. It highlights the limitations of existing benchmarks and the core functionality of WANDR.

The Leak: Addressing the Gap in Benchmark Design
Traditional benchmarks for evaluating AI agents often focus on single, definitive answers, failing to capture the complexity of real-world knowledge work. Perplexity recognized this gap and developed WANDR, an open benchmark centered around 500 realistic, challenging data-collection tasks. This benchmark prioritizes the ability of agents to build large, evidence-backed collections rather than simply providing a single, correct answer.

Technical Specs: The Core of WANDR
WANDR’s architecture is built around two key demands: “wide” discovery – identifying a large, often open-ended set of qualifying entities – and “deep” investigation – thoroughly researching each entity to support claims with evidence. This combined approach creates a significantly more demanding test for knowledge work agents. The system employs a composable qualification key hierarchy, allowing for flexible task definitions, exemplified by the ‘ceo_cfo_appointmentstask’ which requires data on 70+ US companies with CEO/CFO appointments.

Next Steps: Task Construction and Evaluation
The construction of WANDR tasks utilizes a semi-automated pipeline, starting from de-identified patterns observed in production environments. This pipeline, involving seeding, authoring, admission, and curation, is guided by an author-critic loop and mechanical linting. The resulting tasks average 50 members and 245 records, totaling 170,495 source-backed records, categorized into 167 lower, 166 middle, and 167 higher difficulty levels.

Precision, Recall, and Soft Scores: Measuring Agent Performance
The evaluation of WANDR tasks employs a sophisticated scoring system based on precision, recall, and soft scores. Precision measures the quality of the submitted data, while recall measures quality-adjusted completion, addressing any shortcomings with zero values. Soft scores provide partial credit for incomplete members, and hard scores only reward members whose full subtree is correct. This nuanced approach allows for detailed analysis of agent performance. ---

Agent Evaluation: A Multi-faceted Approach
This chapter delves into the methodology used to assess the performance of six production systems on the WANDR benchmark. It details the cost variations, the limitations of the benchmark, and key findings regarding agent capabilities and challenges.

Cost Across Settings: A Significant Factor
The execution of the WANDR benchmark revealed significant cost variations across different systems, ranging from $0.03 to $324.83 per task. This cost differential highlights the resource-intensive nature of the benchmark and underscores the importance of optimizing agent efficiency.

Four Key Findings: Understanding WANDR’s Challenges
The evaluation process uncovered four critical insights regarding WANDR and the agents operating within it. First, partial progress is common, but complete coverage remains elusive. Second, scale significantly compounds the challenges, particularly within deeper hierarchies. Third, discovery represents a major bottleneck, with top-level discovery completion rates varying across systems. Finally, the quality of extracted evidence – specifically page usability and excerpt support – often falls short.

Real-World Applications: Mapping WANDR to Practical Tasks
WANDR’s design reflects real-world knowledge work scenarios. It tests patterns used by market analysts, due-diligence teams, and talent sourcing professionals. The benchmark's per-record grading allows for precise localization of failures, enabling targeted improvements in agent performance. ---

The Hierarchy: A Detailed Examination
This section provides a deeper dive into the structure of the WANDR benchmark, specifically focusing on the composable qualification key hierarchy and the ‘ceo_cfo_appointmentstask’ as a prime example. It highlights the flexibility of the system and its ability to represent diverse search patterns.

Composable Qualification Key Hierarchy: Flexibility in Task Definition
The core of WANDR’s design is its composable qualification key hierarchy. This structure allows for the creation of complex tasks by combining qualifying entities – such as company(n) -> employee(m) -> url(k) – into nested searches and matrices. This flexibility enables the creation of tasks that mirror real-world data collection needs.

ceo_cfo_appointmentstask: A Concrete Example
The ‘ceo_cfo_appointmentstask’ serves as a practical demonstration of the hierarchy’s capabilities. This task requires gathering information on at least 70 US-based companies, each with a CEO or CFO appointment announced between March 1 and April 30, 2026. The agent must supply one authoritative appointment page for each company, and a listing-authority page for additional context. ---

The Pipeline: Automation and Refinement
This chapter details the automated pipeline used to construct the WANDR tasks, emphasizing the iterative process of authoring, criticism, and refinement. It outlines the four stages of the pipeline: seeding, authoring, admission, and curation.

Seeding, Authoring, Admission, and Curation: A Four-Stage Process
The WANDR task construction pipeline utilizes a four-stage process: seeding, where initial task ideas are generated; authoring, where tasks are fleshed out with specific requirements; admission, where tasks are vetted for quality and relevance; and curation, where tasks are refined and optimized. This interleaved author-critic loop, coupled with mechanical linting, ensures high-quality tasks.

Author-Critic Loop and Mechanical Linting: Ensuring Quality
The author-critic loop allows for continuous feedback and improvement of the tasks, while mechanical linting enforces consistent formatting and adherence to the benchmark’s guidelines. This combination of human and automated oversight results in a robust and reliable set of tasks. ---

Results and Analysis: A Detailed Breakdown
This section presents a detailed analysis of the results obtained from running six production systems on the WANDR benchmark. It quantifies agent performance using metrics like precision, recall, and soft F1 scores.

Soft F1 Score: Measuring Overall Performance
The primary metric used to evaluate agent performance is the soft F1 score, which provides a holistic measure of accuracy and completeness. Perplexity’s Search as Code (SaC) system achieved a soft F1 of 0.447 at the high setting, representing a significant advancement in knowledge work agent capabilities.

Precision and Recall: Understanding Performance Trade-offs
The analysis reveals a consistent trend: soft recall consistently falls below soft precision. This indicates that while agents can often successfully identify a large number of entities (precision), they frequently fail to fully investigate and complete the required evidence (recall).

Scaling Challenges: The Impact of Hierarchy Depth
The benchmark highlights the challenges associated with scaling up knowledge work agents. Deeper hierarchies exacerbate the problem, as each branch adds a potential failure point. ---

Key Findings Revisited: Consolidation of Insights
This concluding chapter summarizes the key findings from the WANDR evaluation, reinforcing the significance of the benchmark and its implications for the development of knowledge work agents.

Partial Progress and Complete Coverage: A Crucial Distinction
The benchmark consistently demonstrates that partial progress is common, but complete coverage remains a significant challenge. Agents often struggle to address all aspects of a task, highlighting the need for more robust and versatile agents.

Discovery Bottleneck: The First Structural Challenge
Discovery completion rates represent the first major structural bottleneck in the WANDR benchmark. The varying completion rates across systems underscore the difficulty of automatically identifying relevant entities.

Usable Page vs. Complete Evidence: The Final Hurdle
Ultimately, the most significant challenge lies in extracting usable page content and transforming it into complete, supported evidence. Perplexity’s findings—41.4% of pages miss a substantive requirement, and 57.5% of excerpts fail to support the full claim—highlight the complexity of this final stage.