Statistical Frameworks for Uncertainty in Agentic Systems

Hypotheses are central to information acquisition, decision-making, and discovery. Scientific AI agents powered by Large Language Models (LLMs) can now automate data analysis and propose scientific hypotheses from data. However, a missing piece is the "reward" signal, that is, whether the generated hypotheses are worth costly follow-up validation, since LLMs are prone to hallucinations and errors. In addition, the sheer volume of hypotheses LLMs can generate makes manual validation impractical.

We therefore argue the role of statistical evidence as a reward signal for AI-in-the-loop scientific discovery. In this talk, we present recent efforts on developing agentic frameworks that enable automated validation and generation of hypotheses with valid statistical evidence. The key idea is to design high-level statistical principles for the exploration of AI agent to guarantee error control while allowing flexibility and diversity. First, we present POPPER, an agentic framework for rigorous automated validation of free-form, abstract, and high-level hypotheses. Guided by Karl Popper's principle of falsification, Popper validates a hypothesis using LLM agents that design and execute falsification experiments targeting its measurable implications. A novel sequential testing framework ensures strict Type-I error control while actively gathering evidence from diverse observations. We demonstrate that Popper delivers robust error control, high power, and scalability. Compared to human scientists, Popper achieved comparable performance in validating complex biological hypotheses while reducing time by 10 folds. Second, we present ongoing work on agentic frameworks to orchestrate hypotheses generation and validation to produce candidate scientific hypotheses with Type-I error control.

AI systems are moving from one-shot prediction to sequential interaction. For example, an agent may need many steps to work through a complex task. But how can we trust the output produced by a black-box system whose behavior unfolds over time? This question becomes especially challenging in realistic settings where (i) interaction and evaluation are expensive and therefore limited by budget constraints, and (ii) high-quality labeled data are scarce. In this talk, I will discuss how distribution-free statistical inference can provide reliability guarantees for modern AI systems under such resource constraints.

The first part argues that agentic systems call for a new reliability question. Rather than asking only whether an AI system will fail or succeed at a task, we ask a richer question: how long will it take until an important event occurs, if ever? For example, how many interactions are needed before a jailbreak succeeds, or before an agent completes a task? The key challenge here is censoring: with a limited interaction budget, we often stop observing before the event occurs, so the data are only partially labeled. Nevertheless, I will show that statistical principles can still provide distribution-free predictive lower bounds for time-to-event outcomes, and that careful allocation of the interaction budget can yield sharper bounds.

The second part focuses on reliable inference under limited labeled data. I will introduce General Synthetic-Powered Inference (GESPI), a framework that safely enhances the sample efficiency of statistical inference procedures by leveraging synthetic or auxiliary data. The method can wrap statistical tools such as conformal prediction, hypothesis testing, and multiple testing. Crucially, it provides distribution-free error control without assumptions on the quality of the synthetic data, while improving power and utility when the synthetic data are useful.

Evaluating generative AI models is a routine, but resource-intensive, process conducted repeatedly during model development. This talk will cover some recent methods for improving the statistical efficiency of model evaluation when cost or time constraints are a concern. In particular, I will discuss Collaborative Evaluation (CollabEval), a simple but principled method for exploiting dependencies between historical runs of different models on the same tasks to improve estimates of a new model's performance with fewer samples.

Uncertainty is typically used in error detection, routing to tools, and subagents safeguards against unnecessary spend. In this talk I will discuss three different use cases. First, I will explain how it can be used to enhance multi-agent debate. Then, I will demonstrate how it can help us measure bias in machine translation. And finally, I will argue that uncertainty is fundamental in forecasting future events.