What an expert task looks like
Tests that catch the mistakes agents actually make
Generic benchmarks reward fluent answers. Expert tasks check the method: the right statistical test, the confounders, the claim the evidence can support. This example reflects a failure seen in real co-scientist runs.
QuestionBulk RNA-seq from 40 tumors, responders vs non-responders. Which genes associate with resistance?
HintRaw counts, not TPM. Treatment line and tumor purity differ between groups.
AnswerCount-based model (DESeq2 or edgeR) with line and purity as covariates; filter low-count genes; report FDR. A rank-sum test per gene fails the task.
DifficultyHARD