AI’s Biology Fail? 🤯 LifeSciBench Reveals Truth 🔬

LIFE SCIENCE BENCH: A NEW PARADIGM FOR AI MODEL EVALUATION
OpenAI’s LifeSciBench represents a significant advancement in evaluating large language models (LLMs) within the complex domain of scientific research. Recognizing the limitations of traditional benchmarks, which often suffer from saturation and lack of real-world relevance, LifeSciBench directly addresses this gap. The benchmark consists of 750 expert-authored tasks, meticulously designed to mirror the challenges faced by scientists. These tasks are organized across seven workflows – including evidence handling, design & optimization, and scientific communication – and seven biological domains, ranging from genomics and medicinal chemistry to clinical and translational science. Each task is structured around a prompt, supporting artifacts, and a detailed grading rubric, ensuring a rigorous and nuanced assessment of the model’s capabilities. The creation of this benchmark involved a collaborative effort of 173 expert scientists, each holding a Ph.D. and boasting significant experience in biotechnology or pharmaceutical research, further solidifying its credibility and practical value.

THE CORE MECHANICS: RUBRICS, ARTIFACTS, AND METRICS
The core of LifeSciBench’s methodology lies in its use of comprehensive rubrics. These rubrics contain a staggering 19,020 criteria, averaging approximately 25 criteria per task, each rewarding a specific, concrete property such as a factual assertion, a logical reasoning step, or a numerical answer within a defined tolerance. Grading is conducted against these rubrics, not against single reference strings, allowing for a more granular and accurate evaluation. To augment the rubrics, the benchmark incorporates 1,062 artifacts – including sequences, figures, tables, PDFs, and chemical structures – that models are expected to utilize. Approximately 53% of tasks require at least one artifact, highlighting the importance of integrating external data sources. Performance is then measured using two key metrics: normalized rubric score and task pass rate. The normalized rubric score provides a comprehensive measure of a model’s overall performance, while the task pass rate offers a simpler indication of success, defined as scoring at or above 70% on the task. This dual-metric approach provides a richer understanding of model capabilities.

MODEL PERFORMANCE AND KEY FINDINGS
OpenAI evaluated five models – GPT-Rosalind, GPT-5.5, Gemini 3.1 Pro, and two others – within a single-turn setting, with unrestricted internet browsing permitted. GPT-Rosalind emerged as the top performer, achieving the highest per-task mean on 386 of the 750 tasks and significantly boosting the overall pass rate from 25.7% to 36.1%. However, pass rates remained relatively modest across all models. Notably, Gemini 3.1 Pro led on 214 tasks, suggesting potential strengths in specific areas. Despite these findings, it's crucial to acknowledge that aggregate scores can mask task-specific strengths. Certain workflows, particularly Design, Optimization, and Prediction, proved more challenging, with GPT-Rosalind achieving a pass rate of only 30.7%. Analysis also presented difficulties, with a pass rate of 30.3%. The reliance on artifacts presented a bottleneck, with GPT-Rosalind experiencing a significant drop in performance (from 45.1% to 28.1%) when required to utilize them. Exact outputs demonstrated the greatest difficulty, with sequence and structure criterion success rates ranging from 46.9% to 18.0% across models. While GPT-Rosalind showed a marginal improvement (+0.001) over GPT-5.5 on generate/construct items, models consistently stalled mid-task, with a substantial number of tasks earning at least 50% rubric credit yet still failing the task. Ultimately, only 171 tasks (22.8%) were fully passed, and 261 tasks (34.8%) had a best-model pass rate below 20%, indicating significant headroom for further development.