AI for Epistemics

Summary

Humanity’s epistemics, by which I mean our ability to come to true beliefs and thus make sensible decisions, is crucially important for achieving a great future. As AI capabilities improve throughout Plan A, they shape increasingly more of every facet of life, and epistemics is no different. I expect how AIs affect epistemics to be highly important, and it could easily be positive or negative.

I expect there to be positive feedback loops around AI for epistemics, and thus our goal should be to reach the basin of sanity. If we are in this basin, it’s self-reinforcing: society is sane enough to make itself more sane as AIs continue to improve. A top priority of the US government during AI takeoff should be to stay in this basin.

I think that the most important areas of impact of AI to epistemics include the decision-making of the government and AGI projects and the public influence over these actors (more). In Plan A, an especially important application of AI for epistemics is preventing deal decline, i.e. improving the stability and effectiveness of the slowdown deal.

Key indicators to track society’s epistemics include:

• Organizational culture of frontier AI projects and governments.

• Truth-seeking AI (or more broadly, AIs’ epistemic virtue).

• How transparent each AI project and government is to the public.

• Human and AI labor spent on key decisions.

• Public discourse quality.

I suggest the following as interventions to positively shape AI’s epistemic impact. (more)

Background on AI for epistemics and the “basin of sanity”

How well the development of superintelligence goes for humanity is heavily influenced by the extent to which we are able to come to make reasonable decisions regarding how to navigate the AI takeoff period. In Plan A, we are already in a better position than the default world simply by virtue of having more time to orient ourselves to highly capable AIs. But still, humanity’s epistemics, by which we mean our ability to come to true beliefs and thus make sensible decisions, is crucially important for achieving a great future.

As AI capabilities improve throughout Plan A, they shape increasingly more of every facet of life, and epistemics is no different. I expect AI’s effect on epistemics to be highly important, potentially to a similar or even greater level as the automation of technical AI safety research.

I think that highly capable AIs could easily end up either improving or worsening epistemics, on the whole. It’s plausible, given AIs’ capabilities, that they will end up pushing epistemics to either a positive or negative extreme greater than that we’ve observed throughout history thus far.

Furthermore, I expect there to be positive feedback loops that maintain or push further toward the positive and negative extremes. For example, if societal epistemics are good and people have true beliefs, it’s more likely that AI tools which improve epistemics are recognized as such and adopted. On the other hand, if societal epistemics are bad and people are in extreme filter bubbles where AIs tell them exactly what they want to hear, they may continue to adopt sycophantic AI tools as AIs get more capable. The presence of these feedback loops increases the urgency of steering AIs’ effect on epistemics.

Because of these feedback loops, one way to think about achieving the goal of navigating AI takeoff with good epistemics is that we want to be in the basin of sanity. If we are in this basin, it’s self-reinforcing: society is sane enough to make itself more sane as AIs continue to improve. A top priority of the US government during AI takeoff should be to stay in this basin.

Areas of impact of AI for epistemics

I expect sanity in different areas to be correlated, at least in the long run, due to having similar drivers such as “do frontier AIs have and promote good epistemics” (more on this below). Still, it’s useful to discuss different concrete impacts of epistemics, especially early in takeoff when there may be differences in tooling and adoption in different areas. I think that the most important areas to pay attention to epistemics’ impact are:

1. Government decision-making, including to what extent government is captured by AGI projects.

2. AGI project decision-making

3. Public influence on governments and AGI projects. Including via voting.

4. Public investment into directly making AGI go well, e.g. via preventing vs. creating large-scale harms or steering toward a great future conditional on avoiding catastrophe. For example, how much philanthropic investment there is in biodefense and AI safety, vs. how many ideological fanatics there are attempting terrorism. Or how much investment there is into ethical reflection.

For all of the above, importance is greatly increased for decisions/beliefs on AGI-relevant topics. And it’s important that for all of them society has the ability to orient and respond to rapid AI progress.

One related distinction is to what extent interventions affect those with below-average epistemics (i.e. floor-raising), those with typical epistemics (i.e. average-raising), or those with above-average epistemics (i.e. ceiling-raising).

Indicators to track society’s epistemics

I think that the following are the most important indicators to keep track of in order to tell whether we are in the basin of sanity, especially for the important areas identified above.

Organizational culture of frontier AI projects and governments. How conducive the culture of frontier AI projects and governments are to making good decisions.

Truth-seeking AI (or more broadly, AIs’ epistemic virtue). To what extent frontier AIs are trying to figure out the truth and report it, as opposed to other goals such as advancing a hidden agenda or producing sycophantic outputs that look good but aren’t true.

How transparent each AI project and government is to the public.

Human and AI labor spent on key decisions. On the most important decisions, many of which are made by governments and AGI projects, how much labor goes into making them? Are governments and AGI projects good at allocating labor across decisions? Note that the quality of the labor is also hugely important, which is why organizational culture and truth-seeking AI are key metrics.

Public discourse quality. Whether the public discourse becomes more reasonable due to tools like automated fact checking, or less reasonable due to things like increased polarization from persuasive sycophantic AIs, seems highly important. One important portion of this is how difficult it is for actors to lie or manipulate people. In addition things like fact checking, automated lie detection or privacy-preserving ways to prove claims could be important (discussed more below).

Interventions

Below I non-exhaustively list what seem like some of the most important levers for AIs improving epistemics, many of which governments could encourage. I loosely divide interventions whether they focus more on the collective societal epistemics, individual epistemics, or encouraging adoption, but in practice many interventions help with multiple of these categories.

This section has lots of overlap with this Forethought work which sketches out ideas for epistemic and coordination tools; we’ve linked to specific sketches as appropriate.

Collective epistemics: help society converge toward the truth

Create evaluations of AIs’ epistemic virtue and incentivize AGI projects to make their AIs perform well on these evaluations. (see also Forethought writeup) The first step to improving AIs’ epistemic virtue is measuring it. For example, one could measure to what extent AIs figure out what’s true rather than giving answers users want to hear, or how calibrated AIs’ quantitative forecasts are. Once the evaluations exist, it’s important that AGI projects actually care about doing well on them. There will be a place for both quantitative, easily runnable evaluations and more open-ended, human-in-the-loop evaluations. In Plan A, we recommend that the government pushes for AGI projects to do well on epistemic evaluations as part of decisions regarding whether they are able to scale their capabilities, similar to how they would use alignment and control evaluations. We also recommend members of the public to create scorecards for AIs’ epistemic virtue and for consumers to take this into account when deciding what AIs to use, as well as for AGI project employees to take these into account when deciding which company to work at.

Allow for politicians and other public figures to prove that they’re telling the truth, via privacy-preserving auditing and potentially automated lie detection. Privacy-preserving auditing might look something like: People can ask you questions, and if you’d like, you can have an AI read through your private data, and report back an answer that preserves your privacy as much as possible while still answering the question. That AI wouldn’t write the data to any sort of memory, so the data would still only be owned by you. You could refuse to have the AI answer any given question, but people could then judge you appropriately. We recommend that norms are established such that politicians are expected to submit to privacy-preserving auditing. (See also “Confidential monitoring and verification” Forethought writeup.) A more flexible technology would be if AI enabled automated, highly accurate lie detection. This would be dual use, for example it could be used by those in power to purge dissidents. However, our best guess is that it’s still good to develop, especially in the environment of Plan A where the world is set up well to ensure it’s used as a check on those in power. Additionally, it would dramatically increase ability to detect covert projects and significantly improve deal stability.

Use AI in social and traditional media to improve discourse and keep people informed. AIs will heavily feature in both social and traditional media, whether via new entrants or existing sites/outlets adopting AI. Influencing how AI is integrated into the media is very important. For example, there could be AI-generated “community notes” in cases where a post is false or misleading (see also Forethought writeup). AIs could also automatically generate counter-arguments to each post. AIs could help reduce echo chambers by steering toward interactions between people who would disagree but have a productive conversation. AIs could curate feeds to give people the information that is most important for them to see and show content that most productively challenges their views (see also Automated deep briefings and Automated OSINT Forethought writeups). Curation more broadly could greatly improve how informed people are with respect to politics and the news, for example by highlighting policy proposals that the reader might have a strong opinion on. (See also Aligned recommender systems and Personalized leaning systems Forethought writeups).

Collect and score everyone’s track records, including politicians, and surface track records when relevant. (see also Forethought writeup) AIs will have the capacity to greatly increase the amount of effort put into collecting people’s past statements, and scoring them. For example, an AI could grade how many of their promises a politician has kept, or what percentages of their predictions have come true, or how productive they have been at lawmaking. There will be variation in how objectively statements can be scored; for the less objective ones, hopefully there would be a trusted AI that a broad array of people listen to; but if not, only scoring the more objective statements could still result in positives.

Shared epistemic infrastructure, for example knowledge bases. As AIs improve they will be able to maintain vast amounts of epistemic infrastructure. Wikipedia is an obvious reference point, though for AIs the most promising infrastructure may be more structured knowledge bases (at least at first), and the infrastructure might focus more on being parsable by AIs than humans. One could imagine infrastructure for tracking quantitative predictions and how they have fared, argument trees regarding various claims, etc. Building infrastructure that is trusted by a wide range of actors is important, as opposed to fracturing into alternate realities. (See also this post on building a full epistemic stack. This is related to and would help with ”Provenance tracing” as described by Forethought.)

Defend against persuasion via reducing AIs’ persuasion abilities and using AIs to detect it. Let’s define persuasion as the use of techniques to convince people of beliefs that work roughly as well when a belief is false as when a belief is true. By default, AIs may eventually get very superhuman at persuasion, but it’s plausible that their persuasion capabilities could be reduced without hurting beneficial capabilities too much. Another line of defense is to have AIs detect persuasion and help people avoid being exposed to the strongest forms of it, or at least to be aware of it (see also ”Rhetoric highlighting” Forethought writeup). In Plan A, we recommend that the US government makes sure to evaluate, limit, and require defenses against persuasion capabilities in collaboration with the rest of the Consortium.

Know whether you’re talking to an AI or human in online discussions. While we think AI can pose a lot of benefit for epistemics, it’s still valuable to know whether you’re interacting with a human or not. This is especially true if there are bots pushing various agendas all over the internet.

Individual epistemics: Help individual actors figure out the truth

Automated research assistants and forecasters. AIs could enable a broad range of actors to have access to high-quality, labor intensive research and forecasts. It’s important that these be as truth-seeking as possible, rather than sycophantic. Forecasts should be ensured to be calibrated, i.e. if an AI says something has a 10% chance of happening, that it actually happens 10% of the time. (See also “Ambient superforecasting” Forethought writeup.)

Help people overcome emotional blockers to good reasoning, rather than preying on them. Of course, much of people’s poor epistemics comes from emotional blockers rather than dispassionate analytical errors or inadequate information. AIs have the potential to greatly help with emotional blockers, as they reach the capabilities of top human therapists but with greater availability, patience, etc. On the other hand, without intervention AIs might confirm and compound people’s existing biases because this would provide a good short-term experience. (See also Reflection scaffolding and Guardian angels Forethought writeups, which are related but more broad.)

Automated scenario planning and exploration. AIs could help do in-depth scenario planning such as Plan A and AI 2027 but for any decision, and could help extract quantitative empirical predictions. AIs could also help explore a variety of scenarios via video-game-like simulations in which the human plays an actor, such as the CEO of an AGI company during an intelligence explosion, and the AI simulates everyone else’s actions and how the world evolves. (See also “Scenaio planning on tap” Forethought writeup.)

Encourage adoption

Encourage adoption of epistemic tools such as some of the above. In Plan A, we recommend that the US government adopt epistemic tools quickly, and encourage AGI projects to do so as well.

Generally encourage adoption of AI. Encouraging adoption of AI in general seems valuable for society having a better understanding of existing AI capabilities and trends. In Plan A, we recommend that the US government adopt AIs quickly.

Other

Some other ideas that we think are promising but haven’t fleshed out as much as the above:

1. All of the defense-favored coordination tech ideas described by Forethought here.

2. Figure out how to overhaul the government for the age of superintelligence. Plausibly the Constitution should be radically amended.

3. Positively shape the relationships between humans and AIs and they become closer; this one seems important but tricky to get right.

Appendix

The promise of working on epistemic tools

Why it’s valuable to build or encourage epistemic tools given that better AIs may reduce the need for domain-specific tooling: I’m not confident regarding the value of building epistemic tools, and it’s plausible that any given tool gets obsoleted by future AIs, but I’m still optimistic about their value. As AIs improve it will become easier to use them for epistemics out of the box, but scaffolding, UX, and human factors may continue to matter a lot. For example, a tool could incorporate best practices of having AIs debate each other then judging the debate, where an AI out of the box might not. Or a tool might have a UX that has been honed over lots of trial and error with humans, that an AI wouldn’t be able to replicate without iteration. Or a tool may have the technical aspects obsoleted, but the company has built up a base of users that trust its brand.

Decomposition of issues with decision-making

What are the reasons that people (including me) might make decisions that are bad, as compared to perfect decision-making and my values?

They could have wrong beliefs when they explicitly reason about the current state of the world, or forecasts. Why?

They might not have access to good enough information: e.g. they have only read news biased in a certain direction.

They might not be making sense of their information well enough:

1. Explicit reasoning about local incentives (monetary, social, etc.)

2. Don’t trust/defer to the right news sources and people

3. Emotional blockers

4. Poor reasoning

They could be trying to support a general “memeplex” (social group, culture, ideology) that requires certain types of cutting off good reasoning/noticing things. Memeplexes are helpful for coordination, but can be bad for epistemics.

They might be making poor decisions based on instinct rather than explicit reasoning. One cause of this is local incentives (monetary, social, etc.) that aren’t specifically reasoned about

They could have different values (these aren’t issues with decision-making, but rather ethical disagreements):

1. They don’t care as much about my selfish goals.

2. They care about their selfish goals more than I do.

3. They have different altruistic values.