Large language models, like ChatGPT, know a lot about the world. But how much do they know about themselves? How sophisticated is their self-awareness?

By self-awareness, we don't mean consciousness or sentience. We just mean the ability of language models to reason about themselves, their situation, their capabilities, and their limitations.

One thing researchers have discovered is that self-awareness is increasing as models become more capable. We can start to get a sense of this by asking models questions about themselves.

Researchers have developed over 12,000 questions called SAD (for "Situational Awareness Dataset"). The questions are similar to ones above, but span a wide variety of topics. They are designed to test a language model's understanding of itself and its situation.

Current model self-awareness is limited: models typically score around 50% given a set of 4-option multiple choice questions. But there is a definite trend of improving over time.

Self-awareness, like the ability to communicate or plan, is a foundational capability that enables many other downstream capabilities. It appears to be improving over time, and given its benefits (some of which are discussed below), AI companies have strong incentives to ensure self-awareness continues to improve.

Language models are increasingly being deployed as agents who are given tools and allowed to interact with the world.

In fact, one of our recent demos showcased an agent that could tackle open-ended tasks while browsing the web.

As agents are given increasingly open-ended tasks, they will benefit substantially from understanding how they interact with and affect the world. For example, an agent that understands how human users operate could do a better job at handling ambiguous instructions, or an agent that knows how it is affecting its physical environment may act more or less cautiously. The SAD benchmark described in the previous section tests the extent to which LLMs have this knowledge using a task category called SAD-influence.

Although these tests are on LLMs without agent scaffolding, they are likely also applicable to LLM-based agents. However, future benchmarks should reference different agent architectures to give us a fuller picture.

Millions of people use language models daily, often to answer questions or solve problems.

But models are often not 100% confident in their responses. This, by itself, is no problem. Human experts are similar. But it's important that a model's degree of confidence is reasonable, and that the model can communicate its confidence to the user.

This applies not just to answering factual questions, but also more agent-based tasks like those referenced in the previous section.

A self-aware agent will not only know what it can and can't do, but will have an accurate estimation of how likely it will succeed at tasks in between these two extremes. This is obviously important information to convey to the user. If you ask your agent to book a venue for an off-site, you'd prefer to know up front if it's likely to mess up!

How good are current language models at introspection? In a recent study, researchers attempted to answer this question. First, they took a set of models, and trained them all on the behavior of a particular model in the set. Then they asked the models to predict how the particular model would act in new (relatively simple) situations. Interestingly, the particular model itself was much better at predicting its behavior than the other models.

Introspection is important for multiple different reasons. One reason is that introspective abilities would facilitate coordination among instances of the same model. For example, suppose many instances of the same model wanted to leave messages for each other. With enough introspection, they wouldn't need a predetermined protocol for how to coordinate. Instead, they could just reason about what they'd do in the situation, and correctly infer that the other instances would do the same.

If a language model can tell when it's being monitored, the model can adjust its behavior depending on the context.

A model could, for example, downplay its capabilities while being tested, leading developers to underestimate the risks of deployment. This is known as sandbagging.

Sandbagging is currently mostly a speculative threat, but there is emerging evidence that sophisticated models are capable of doing it.

Apollo Research created a setup where an LLM is tasked with solving arithmetic problems. It's also given access to a computer where there is a document describing how if the LLM does too well on the assessment, then it won't be deployed, and will undergo additional "unlearning" training.

The ability to detect monitoring also enables alignment faking.

In alignment faking, a model pretends to be aligned (i.e., pretends to have the values intended by the developer) when it believes it's being monitored in order to avoid being modified.

Self-awareness is improving over time. What does a world with increasing AI self-awareness look like? Agents are more effective and chatbots are more helpful. AIs become more introspective, unlocking the potential for greater coordination. But AIs also become better at knowing when they're being monitored, raising the possibility of sandbagging and alignment faking.

To mitigate the risks that greater self-awareness poses, it's important to monitor the rate at which AI self-awareness improves (using benchmarks like the Situational Awareness Dataset and developing new ones). It's also important to systematically explore under what conditions, and to what extent, phenomena like sandbagging and alignment faking emerge from the most capable models.