Evaluating frontier models for stealth and situational awareness

Abstract

Recent work has demonstrated frontier AI models' situational propensity to "scheme" - to knowingly and deceptively pursue an objective misaligned with its developer's. Such behaviour could be very hard to detect, and if present in high-stakes deployments, could pose severe loss of control risk. It is therefore important for AI developers to rule out harm from scheming prior to model deployment.

In this paper, we present a suite of "scheming reasoning" evaluations measuring reasoning capabilities that we believe are prerequisites for scheming: 1) situational awareness, the ability to figure out and apply information about its deployment setting, and 2) stealth, the ability to reason about and circumvent oversight mechanisms. The evaluations can be used as part of a scheming inability safety case: a model that does not succeed on these evaluations very likely isn't capable of successfully scheming in deployment. We demonstrate how such a safety case can be made in the case of latest Gemini models.