Preliminaries

It should be possible for almost all humans to get almost everything that they want from a post-ASI world. With huge numbers of robots capable of almost any task a human can do, it will be possible to provide everyone with the living standard that only the richest people today enjoy. People could spend their time doing what's most meaningful to them—raising children, creating art, or spending time with friends. With ASI-driven advances in medicine, we'll be able to treat mental illness, chronic pain, aging, cancer, and other diseases that shorten or worsen people's lives today. In short, if we achieve the future described in Plan A, there will be enough to go around to give everyone a very good life. Many of the tough tradeoffs that are the focus of modern politics will be resolved. So in some sense, these decisions need not be extremely fraught.

Yet, at the same time, the decisions we make are extremely high-stakes. If we handle things well, then we could create an enormously flourishing future for our descendants. But there are many ways we could waste this opportunity. We could fight over the resources, leaving a few people in control of the future and everyone else with nothing, or burning vast resources in internecine conflict. Even if we avoid those pitfalls, we could still misuse resources—squandering them on positional goods or failing to leverage ASI to figure out how best to achieve what we truly care about.

We want a governance system that will allow us to make wise decisions from a foundation of abundance and cooperation.

This document focuses on how to decide what to do with resources outside the solar system (henceforth referred to as “space resources”). This is an important aspect of post-ASI governance—the overwhelming majority of accessible resources are outside of the solar system and the overwhelming majority of future beings may live outside our solar system—but it is not the only important question.

Epistemic status

We’re currently not confident in how the treaty signatories in Plan A should approach post-ASI governance: it depends on a lot of details about people’s preferences, technological capabilities, and optimal coordination mechanisms that are only partly worked out.

Luckily, by the time the treaty signatories are making these decisions, they should have superintelligent automated researchers investigating the nature of the universe, doing moral philosophy, and figuring out optimal coordination mechanisms. And we’ll have AI therapists, advisors, and negotiators to help us understand our preferences and make deals with other actors.

So we think the actual best plan is to defer many of these questions until we have ASI-driven advice and coordination. However, there is value in concreteness and in sharing best guesses, so we wanted to provide some analysis. But we wouldn’t want readers to anchor too much on the ideas we present below.

What’s special about post-ASI governance?

Post-ASI governance will differ from historical precedent in several important ways.

Technological progress may quickly unlock new pools of resources that no one has previously exploited. The most important example is resources outside of our solar system.

• Because these resources have been unused, there are no historical claims to navigate. We should aim to govern them in a just and equitable way.

• There will likely be extreme abundance relative to the present day. With just a tiny fraction of available resources, we’ll probably be able to give everyone alive today a long, happy, healthy, and extremely prosperous life by current standards.

• We think that the most important question that governance will have to face is what to do with the huge quantities of remaining resources after we’ve ensured that everyone alive has enough to live an incredible life. Some salient options include:

  • Personal consumption. People could develop ever-more-expensive ways to derive enjoyment from resources. But most resources are probably poorly suited to use for personal consumption. Most resources are outside our galaxy and thus millions of light-years away. If you want to stay on Earth and consume anything—matter, energy, information—derived from another galaxy, you’d have to wait millions of years. If, on the other hand, you left Earth for another galaxy, then if you ever returned, you’d find that millions of years had passed.

  • Positional goods. People could spend resources on a zero-sum competition to have the most impressive galactic-scale art projects, or the largest number of statues of themselves. Positional goods could soak up a lot of resources. While there are probably generally diminishing returns to consuming more resources as people get richer, this might not hold for positional goods, since status is relative. We think it would probably be a mistake to spend huge quantities of resources on this type of zero-sum competition.

  • Satisfying moral preferences. We could create societies full of flourishing people, or enormous nature reserves. We think this is the most promising way to use the cosmic endowment. But then the question becomes: how should we decide which moral preferences to pursue?

Technological progress will also change how we make governance decisions:

• We may be able to make decisions with lasting effects far into the future.

  • For example, we could hand over resources or hard power to ASI systems that have been aligned to a particular set of values. This will likely give us the option of creating extremely long-lasting institutions.

  • Typically later generations adopt somewhat different values than their parents, and as a result society’s values change over time. But new technologies may enable parents to exercise tighter control over the values that their children develop.

• People will have access to superhuman advice and coordination assistance.

  • This means people will be better able to identify and advocate for policies that truly achieve what they care about.

  • But we’ll probably also need to make our governance structure more robust than ever to people trying to game it, either individually or in a group. Just because a governance structure has not been severely exploited in the past does not guarantee that it won’t be severely exploited in the future.

And we will have large responsibilities to make these decisions go well for people who aren’t able to participate:

• The overwhelming majority of the affected parties probably won’t be able to participate in the decision to govern space directly, because they won’t have been born yet. This is already true to some extent—in modern democracies, citizens set policies for non-citizens, adults decide for children, humans decide for animals—but in the post-ASI setting this becomes far more extreme. Present people may make decisions that affect the lives of vast numbers of people who cannot give input. The responsibility to get these decisions right is enormous.

Evaluating proposals

In this appendix, we’re evaluating proposals based on how well they achieve the following desiderata, which we believe will be broadly appealing across different value systems.

• Preference satisfaction. The governance system should try to satisfy all the participants’ preferences as much as possible. Some aspects of this include:

  • Paretopianism. Given a massive windfall, it should be possible to make nearly everyone vastly better off than they are today, and a good proposal should do exactly that.

  • Limit extreme downside. Our post-ASI governance proposal should have a very low chance of producing outcomes widely considered extremely bad, such as huge amounts of suffering, or all of space being destroyed.

  • Satisfy bounded views. If someone has a preference or moral view that is not very costly to satisfy, we should satisfy it. For example, on a cosmic scale, it’s fairly cheap to ensure that every human alive in 2040 has a long, flourishing life.

• Satisfy common-sense ethics. The process should not violate common-sense ethical views. This weighs against proposals where a small number of people predictably end up controlling the fate of the universe or where future generations end up totally disempowered.

• Satisfy objective morality. Some aspects of ethics may be objective and discoverable. If so, we should try to identify and conform to them.

These next criteria aren’t terminal goals, but we think that processes that satisfy them are more likely to achieve the criteria above.

• Coordination. The governance system should enable coordination on large-scale collective action problems. For example, if we encounter alien civilizations, we may need to coordinate to establish friendly relations with them. We may also need to coordinate to avoid actions that risk destroying all of Earth-originating civilization, or to avoid space races where factions compete to claim as many resources as possible, burning huge amounts of resources in the process. These goals probably require some minimal government functions: protecting property rights, enforcing prohibitions, and potentially collecting taxes to fund public goods.

• Moral trade. It might be possible to nearly satisfy many different moral viewpoints simultaneously. For example, some views might especially care about resources close to Earth or the near future, while others might be more indifferent across time and space: we might satisfy both by letting the former choose what happens in the Milky Way while the latter choose what happens in the rest of the universe. Here are two types of trade that we want our proposal to facilitate.

  • Bilateral trade. This is particularly well suited for satisfying uncommon preferences. These trades are most straightforward in market systems, where two people can literally trade deeds to resources or sign agreements about how they will use their resources. But it’s possible that some voting systems–e.g., the Vickrey-Clarke-Groves mechanism—could also achieve similar results.

  • Coordination on moral public goods. People might share some values while also holding idiosyncratic preferences—individually preferring to pursue their own preferences, but collectively preferring that everyone work toward the shared values.

  Coordination on moral public goods is most straightforward in voting systems, since people can just vote to devote resources to shared projects. It’s possible that voluntary contracts—where people pledge to spend resources on shared goods if others do the same—could achieve similar results, but this is less straightforward because of free-riding incentives.

  It might be particularly effective to coordinate to switch from funding positional goods to funding moral public goods. If done in a way that leaves people's relative ability to purchase positional goods unchanged, then this is great for everyone: relative status doesn't change, but people are able to satisfy their non-positional preferences, too.

• Reflection. ASI will likely uncover novel considerations and empirical facts highly relevant to how people want to use space resources (e.g., are there objective moral truths, and what are they?). We want each person to have the time and resources necessary to think carefully about what they value before making important decisions, and we want to ensure that we invest in ASI-fuelled research on important questions to inform people’s decisions.

• Pluralism. Humans today have many different visions of the good. It’s possible that after deep reflection with ASI we’ll converge on a single set of shared values. More likely, we’ll still have some disagreements. Where people have different value systems, we prefer proposals that will allocate resources to pursue all of them.

• Avoid brittle decision-making structures. There are likely many important considerations that we’re not yet aware of. Where possible, we favor flexible decision-making structures that won’t make it difficult to solve unforeseen problems.

• Fairness. Each living adult should have an opportunity to give equal input into the process. We don’t think that this criterion necessarily means that each person ends up controlling an equal quantity of resources. For one thing, some people might care a lot about what happens with resources closer to Earth while not caring much about resources in distant galaxies, and we’re happy for them to trade with people who care a lot about distant galaxies. Nor do we think that this criterion requires that everyone is equally happy with the outcome. For instance, if some decision is made by vote, then some people might end up voting for the losing side. But we want a neutral process that doesn’t privilege any particular people or viewpoints over others.

• Political feasibility. Unlike the other criteria, this is purely pragmatic: a proposal must be acceptable to the relevant decision-makers to be implemented. It’s unclear who these decision-makers would be, but they could include the citizens and/or leadership of the US, China, or other powerful countries. It might also include AI company CEOs or shareholders.

Summary of the most promising options for allocating space resources

We think that it’s best to decide on a specific structure for allocating space resources in 2040 at the time that the treaty is decided (more discussion here).

We’re currently fairly uncertain about what the best structure is, and that decision should be deferred until decision-makers have access to aligned ASI advice. But currently, these seem like the best object-level proposals.

(We discuss alternative proposals that we like less here).

Other important decision points include:

• At what time should votes, preference elicitation, or distribution of resources happen? (discussion here)

• Should later generations get a say? (discussion here)

• Should we prefer a hybrid approach that allocates some resources according to one proposal and allocates other resources according to other proposals? (discussion here)

• How do we enable each decision-maker to make the best decisions by their own lights? (discussion here)

Some available options

Here, we discuss options for governing future resources. We begin by describing a couple of notable options which seem relatively unpromising on the criteria above, then turn to proposals that we find more promising. In both cases, we’re trying to cover important and illustrative cases, not every possible outcome.

Unpromising options

Anarchy

No property rights system is established for space resources. To use a resource, an actor must either be the first to claim it or seize it from another actor. Actors can do whatever they want with resources they currently control—there’s no governance system for prohibiting particularly bad behaviors or any other forms of large-scale coordination (e.g., to fund public goods).

Drawbacks: there could be significant negative-sum conflict over resources, and insecure property claims would make trade—including moral trade—harder.

Anarchy is worse than the feasible alternatives. We do expect that it is not a stable long-term equilibrium. Given much more advanced tools for coordination and communication, we expect people to create ad hoc governance structures. But a government that emerges organically from anarchy could be worse than one designed intentionally. In particular, it might favor the interests of those who were able to claim and defend lots of space resources early on.

Finders-keepers property rights

Actors can claim unclaimed resources or they can buy resources from earlier owners, but they can’t seize claimed resources without the owner’s permission (similar to homesteading). As before, no effective system exists for large-scale coordination or public goods provision, and no governance system can enforce prohibitions.

Strengths: we think this is an improvement on anarchy. Once a claim is recognised, the holder no longer needs to spend resources defending it. Moreover, secure and widely recognised property claims make trade possible.

Drawbacks: value is no longer wasted contesting claimed resources but is still wasted in the race to claim them. It also seems likely that a small number of powerful actors—perhaps whoever can send out a wave of probes first—will claim most resources. And this proposal has no mechanism for public goods provision or enforcement of prohibitions, so it forgoes potentially large gains from moral public goods and from preventing egregiously unpopular uses of resources.

Next, we discuss governance proposals that we think improve on these scenarios according to the criteria above.

More promising options

Direct endowment and trade

Each person receives title to a portion of all future resources, which gives them the right to decide how those resources will be used in the future. For example, people can sell their shares of future resources, gift them to other people or to trusts, or make bilateral deals with each other about how they’ll use their resources. Property rights are well-enforced in practice. As before, no effective system exists for large-scale coordination or public goods provision, and no governance system can enforce prohibitions.

(A related approach: auction titles to all space resources and then redistribute the proceeds equally. In theory, if we make some standard assumptions, this scheme would result in the same distribution of ownership as direct endowment.)

Strengths: when property rights are clear and well-established from the outset, bilateral trade—including moral trade—becomes easier. Compared to finders-keepers, there are no longer incentives to waste resources through racing.

Since each property-holder is free to decide what to do with their share of the future, at least some resources will be devoted to pursuing most moral views, making it likely that cheap-to-satisfy views will be satisfied.

Drawbacks: moral public goods would have to be funded through voluntary contracts, and it’s not clear whether voluntary contracts will be sufficient to ensure that moral public goods are funded at the optimal level, since individuals will face a strong incentive to free-ride.

Since direct endowment doesn’t involve a way to impose universal prohibitions on how to use future resources, the same drawbacks carry over from anarchy and finders-keepers: if there are some uses of resources which are egregiously bad by most people’s lights, but preferred by a few actors, it may be hard to prevent those uses, and bad actors could even threaten those uses to extract concessions. This is the mirror image of the moral public goods drawback: there is no easy way to coordinate to prevent moral public bads.

A fuzzier concern: we think it’s probably better for the current generation to think of themselves as stewards who seek to create the best future by their own lights in the space that they control, rather than as “space emperors” who try to squeeze the maximum personal enjoyment or aggrandizement out of their share of resources. If people get unilateral control over space resources and they’re allowed to buy and sell them like normal property, we worry that this might push people to think of the resources as a personal windfall rather than a serious moral responsibility.

Uncertainties and decision points: We’d also need to make a call on decisions like:

• Who gets resources? (For example, should resources be reserved for future generations?)

• Should people be allowed to sell, transfer, or wager their resources immediately? Or do we want to—somewhat paternalistically—enforce a waiting period for people to carefully consider their options?

• What is the relationship of a galaxy’s “owner” to the future inhabitants of the galaxy?

One important variant on this is to add some mechanisms for preventing people from using resources in egregiously bad ways. In the section below, we discuss some possibilities.

Voting on collective uses of resources

Everyone votes on proposals for what to do with space resources. These proposals could range from very concrete (e.g., “create digital minds according to this specification”), to very meta (e.g., “delay doing anything with space resources right now, and re-assess with another vote X years from now” or “allocate some future resources to private ownership”), or something in between.

Strengths. Compared to the other proposals so far, it’s easier to fund moral public goods—if people prefer resources to go to a shared project rather than divided among all voters, they can vote for that proposal.

It’s also a more flexible plan: once future resources are transferred to private ownership, recovering the ability to make collective decisions might be impossible even if almost everyone wanted it. With some kind of regular vote, the electorate can switch to a different plan at the time of the vote, including agreeing to delay doing anything with space resources.

Drawbacks: All known voting systems have significant problems. Some are simply inefficient: for instance, majority rule faces the familiar problem that the majority view can succeed when the majority wouldn’t deem the minority's preferred option much worse, but the minority greatly dislikes the majority’s preferred option. Others incentivize people to misreport their preferences, individually or through collusion, to benefit themselves at the expense of overall value. This problem is especially serious when people know the details of the voting system well in advance and have ample time to develop strategies to game it with ASI assistance.

A further drawback is that voting requires synchronized decisions: everyone must vote at the same time. If the vote happens in 2100, you vote according to your preferences at that time, even if you would have preferred more time to reflect first. In principle, people could vote to delay the vote. But you could still face situations where a majority wants to vote now while a minority wants to keep reflecting, and the minority gets overruled. Contrast this with direct endowment, where each person can reflect for as long as they want before deciding what to do with their resources.

Uncertainties and decision points: How badly can strategic voting distort outcomes in practice, and can novel technology (e.g., lie-detection technology) mitigate this? How well does voting handle bilateral trades between people with rare preferences? And we'd need to decide: when do we hold the vote, who is included, and which voting system do we use?

Direct preference aggregation

Similar to voting, but instead of people casting votes, AI systems try to directly ascertain people’s preferences. For example, AI systems might build a profile of a person’s preferences from all their writing, extensive interviews perhaps under lie detection, and/or brain scans. Then, everyone’s preferences are aggregated to decide what to do with space resources. As with voting, the decision might be quite meta—e.g., wait 100 years and then do preference aggregation again.

It’s not clear how best to aggregate preference, but here’s a sketch of a proposal. Everyone is first assigned 1/N of future resources as a baseline—their fallback if no gains from trade are found. An AI arbiter then identifies reassignments of resources that would improve everyone's preference satisfaction above this baseline, splitting the gains fairly among all beneficiaries. This handles bilateral moral trade: if person A most wants statues of herself and person B most wants statues of himself, but both value statues of George Washington at 60% as much, the arbiter directs their combined resources toward Washington statues, leaving both better off than their baseline. It can also handle moral public goods: if funding some public good would improve everyone's welfare, the arbiter taxes each person in proportion to how much they value that good (a Lindahl tax).

Strengths. Direct preference aggregation is a modification of the voting proposal above and inherits its strengths. Its major additional advantage is that it makes strategic manipulation more difficult than in voting systems.

Drawbacks. Eliciting people’s honest preferences would require some novel technology and it’s not clear what this process would look like. It’s possible that this process would be invasive, time-consuming, or unpleasant for participants (e.g., an interview while hooked up to a lie detector). It seems unjust and contrary to common-sense morality to require that people go through a procedure that they strongly object to in order to have input into the allocation of future resources.

There are a few more technical issues. For instance, people could strategically modify their preferences before assessment, breaking the main reason for trying to directly elicit people’s preferences (although mitigations might be possible—e.g., governments could ban people from modifying their preferences using novel technologies or require that they register their pre-modification preferences). We also haven’t spelled out a specific algorithm for doing the preference aggregation, and it could be difficult to avoid perverse or hard-to-predict outcomes, especially if many people are best represented as having inconsistent or incoherent preferences. Indeed, the question of how to represent someone’s preferences, including on questions they haven’t explicitly considered, seems very contentious and underdetermined.

Defer to superintelligence

Instead of any particular method of inferring and aggregating people’s preferences, humanity builds aligned superintelligence to plan on its behalf and give it control of nearly all space resources.

For example, we could start with a “council” of superintelligent delegates, each starting from one of many diverse moral and philosophical traditions. Those delegates then engage in deep reflection and deliberation with one another. The delegates can initially be trained to be reliable and accurate on all verifiable domains, like forecasting and mathematics, and not post-trained to strongly avoid novel, ‘weird’, or otherwise unpopular views just because they are currently unpopular. If the superintelligences converged onto a single moral view, then they could devote space resources to that plan. If the superintelligences continued to have different views, they could come to a compromise plan via voting, direct endowment, or direct preference aggregation.

Or we could design an ASI system to be motivated to discover and act on objective moral truths, if they exist. (If they don’t exist, then it would revert to a backup plan—perhaps one of the proposals above).

Strengths. Determining the best plan under a particular moral view probably requires discovering and grappling with lots of novel and counterintuitive considerations. The resulting plan might seem very weird and alien to humans, and they might not be motivated to act on it, even if they’re assured by a superintelligence that it is the morally best course of action. People might also have competing personal interests that take precedence over doing the right thing. But an ASI could potentially be designed to have no competing personal interests. These considerations are particularly strong if moral realism is true, and a plan like this might be our best shot at devoting substantial resources to pursuing objective morality if it exists.

Drawbacks. The ultimate outcome of this process might depend a lot on the details of the process—e.g., what initial viewpoints were represented on the “council” of deliberating ASIs? If this is true, who should decide? It seems rather unfair for the treaty signatories to get to choose.

We think that a process like this might be the best ultimate use of space resources. But if so it should probably result from one of the proposals above. For example, if many people found moral realism compelling after reflection, they could vote to spend public resources on the most morally valuable uses (or express a preference in direct preference aggregation). (It’s unclear whether the direct endowment proposal is sufficient for this: handing off resources to morally-aligned ASI is probably a moral public good in this scenario, so we’d expect it to be underfunded if people were individually deciding how to spend their resources).

Other decision points

Should we commit to a process right away?

The treaty signatories could approach this decision in three ways:

1. Decide on a specific process in 2040.

   1. E.g., schedule an election in 2070 using quadratic voting where all living people over the age of 16 vote on what to do with all extrasolar resources.

2. Decide on a process in 2040 for picking a process later.

   1. E.g., schedule an election in 2070 using approval voting, where all living people over the age of 16 vote on what decision process to use.

   2. E.g., design an ASI arbiter who will do a bunch of research and then pick a process that best satisfies some abstract desiderata put forward by the signatories of the Treaty in 2040.

3. Make no explicit decision about a process or a process for picking a process.

   1. It’s not clear we’d end up deciding what to do with space, under this option. Perhaps at some point one of the signatories to the Treaty would decide that they wanted to colonize space and either do so unilaterally or negotiate with the other signatories to be allowed to do so.

Option 3 is probably worse than options 1 or 2. Having a concrete proposal—even a meta-level one about how to select an object-level process—allows public scrutiny, which we expect would push the proposal to be fairer and more principled. Without a concrete proposal, two failure modes become more likely. First, parties who especially wanted space resources might start lobbying the Treaty governments, and those who are well-connected or have invested heavily in lobbying would end up with outsized influence over how space resources are used. Second, some parties might start unilaterally colonizing space, which could trigger a space race. Governments could ban unilateral colonization, but that would require a plan for lifting the ban—at what point are people actually allowed to colonize?—which brings us back to option 2.

We’re less confident about whether option 1 or option 2 is better.

The main argument for option 2 over option 1 is that we’ll probably get more information about how to design a good process after 2040.

• We might discover new voting or preference aggregation systems that avoid issues with systems that we know about. However, we’re relatively pessimistic about learning much on this front after 2040, given existing impossibility results showing that voting systems can't simultaneously satisfy a set of desirable features. Also, designing new preference aggregation systems is a relatively well-scoped, tractable mathematical problem so we would be somewhat surprised if we didn’t have the answer in 2040 after devoting a small amount of ASI labor to it.

• We might learn how well particular processes are likely to work in practice.

  • For instance, we might learn about strategies that people are actually using to try to game the current system and adapt the rules accordingly, e.g., if we have rules intended to prevent parents from controlling their children’s values to an excessive degree, and it turns out that some parents can find loopholes to exert a lot of control, we could adapt the rules in response).

• We might discover new tools or capabilities that enable a better process.

  • For instance, we might develop new methods for noninvasively scanning people’s brains to determine their preferences.

• We might discover new considerations that affect the optimal preference aggregation process.

  • For instance, we might identify valuable forms of moral trade that we want our preference aggregation process to accommodate.

On the other hand, people will also accumulate information about the likely object-level consequences of each possible process. For example, suppose that we decided to vote in 2070 to decide on a procedure for deciding what to do with space, choosing between the policies of “hold a majority vote and do whatever the winning proposal says” and “direct endowment.” And at this time, it turns out that 50.1% of people favor colonizing the Milky Way while leaving the rest of space as a permanent nature preserve, while 49.9% have plans to colonize more distant galaxies.

In 2070, it would be clear that if we chose the majority vote, then all space resources would be allocated to the majority’s colonize-the-Milky-Way-and-nothing-else plan, but if we went with the direct endowment plan, then some galaxies would be left as nature preserves, while others would be colonized by the minority. Members of the colonize-the-Milky-Way-and-nothing-else coalition would then be strongly tempted to favor the majority vote when choosing a process.

At best, the process-revision vote becomes redundant—we might as well have voted directly on the object-level outcome. At worst, we lose the opportunity for people to agree on a process grounded in principles like fairness or Paretopianism.

One possible way around this would be to delegate process revisions to an AI arbiter that adjusts the process based on principles established in 2040, without regard for which factions benefit from any given change. This approach relies heavily on being able to verifiably align an AI system to a set of principles. It also requires being able to identify principles in 2040 that are specific enough for the AI arbiter to apply without heavy extrapolation, but not so specific that we forfeit much of the upside of deferring process decisions to a later date. But if it’s possible, this might be a good way to incorporate new information in a principled way.

Early decisions vs. late decisions

Once we’ve picked an object-level process—like a vote, a preference aggregation procedure, or directly distributing deeds to space resources—there’s a question of when to run that process.

Here are some advantages of delaying the process:

• People have more time to reflect on what they care about and how they endorse using space resources before it’s time to make a decision. This matters most for voting and direct preference aggregation, where everyone must decide simultaneously.

• People may have access to more information at the time of the decision—e.g., a clearer sense of what is technically feasible to do with space resources.

• It’s possible to incorporate later generations into the process—e.g., allow them a vote, take their preferences into account for direct preference aggregation, or distribute resources directly. This seems good on grounds of common-sense ethics, as it’s plausibly unfair for the current generation to make decisions about the use of space resources on behalf of all future generations.

Here are some advantages of running the process relatively early:

• Some types of potentially valuable moral trade are possible only if participants are uncertain about the future, and people’s uncertainty about the future will probably decrease over time.

  • People might want to bet on the outcomes of some reflective processes. For example, Alice might care deeply about maximizing positive impact if moral realism turns out to be true and discoverable, but care little about having resources if there are no objective moral truths. If Bob values resources equally regardless of whether moral realism is true, Alice and Bob might make the following bet: if ASIs discover objective moral truths, then Bob gives some of his resources to Alice. Otherwise, Alice pays Bob (more discussion of these wagers here).

  • People might also want to bet on empirical facts. For instance, we may still have uncertainty about how many galaxies are within the reachable universe and which of those galaxies are already occupied by aliens. A fairly risk-averse person might buy resources that are likely reachable and unoccupied, in exchange for a larger amount of more uncertain resources, while a more risk-tolerant person might take the other side of the trade.

  • These deals are most straightforward under direct endowment, but they could also work in voting systems, e.g., a coalition of moral realists and subjectivists might be willing to vote for a proposal that spent resources in the most moral way (conditional on moral truths existing and being discoverable) and otherwise distribute the resources to the subjectivists’ preferred purposes.

• Delaying decisions also gives people more time to figure out how to strategically misrepresent their preferences ahead of a vote or preference aggregation process.

• Running the process earlier may allow more flexibility.

  • If an early vote is scheduled and delaying seems wise at the time of the vote, voters can simply vote to defer to a later vote. If resources are distributed early, recipients can take their time deciding what to do with them.

  • On the other hand, if the process is scheduled for later, then we can’t make an early decision even if that would have been wise.

Who should participate in the process?

In this section, we discuss who should participate in deciding how to use space resources. This is distinct from who should receive other rights, such as autonomy over their own lives or the right not to be mistreated. We discuss more about how those basic rights could be assured in a later section.

People with substantial bargaining power in 2040

It’s unclear who these people will be, but some plausible candidates are: AI company CEOs, AI company shareholders, the US president (and advisors), the US electorate, Chinese leadership, or the leadership and/or electorate of other countries that participate in the treaty.

This is probably the default outcome. It’s also very unfair, contrary to common-sense ethics, and could lead to many people’s cheap-to-satisfy preferences being neglected unless powerful people are sympathetic to them. We’re also concerned that if many people expect that this is how decisions will be made, then they’ll focus on jockeying over power during the intelligence explosion rather than cooperating to avert risks.

All living humans

Another option is to share power among all humans who are alive in 2040, as in the Plan A scenario. This is fairer and more in line with common-sense ethics than the previous plan. And, to the extent that decision-makers can credibly signal that this will happen, committing to this plan might make people more cooperative during the intelligence explosion (although it’s unclear how credibly decision-makers could commit to this). Unlike proposals that share power with later generations (discussed next), this option makes it possible to make decisions relatively early, which might be important.

On the other hand, it’s unclear how politically feasible this will be. And it doesn’t include everyone who might deserve a say in how space resources are used—it excludes future people and digital minds.

Some plausible options:

• Include all living humans. This seems like a fairly natural coordination point.

• Include all living humans in some capacity, but give people with bargaining power extra weight. For instance, more powerful people might get larger endowments or have their votes or preferences more heavily weighted. This might be a more politically feasible compromise.

Later generations

“Future” generations are the generations born after the intelligence explosion. The “current” generation is the set of humans alive in 2040. We could include future generations in decisions by:

• Delaying votes (or rounds of preference aggregation) until future generations are adults.

• Reserving some decisions for future generations, e.g., allowing the current generation to vote on how to use 10% of space resources, but letting later generations decide at later votes how to use the remaining 90%, or directly distributing deeds to 10% of resources to the current generation, but saving 90% of resources for later generations.

One reason to share power with later generations is that it’s plausibly unfair not to. However, if we share power with future generations, then this potentially incentivizes parents to have as many children as possible and indoctrinate them to share the parents’ beliefs and values. This could let parents swing a later vote or ensure that a larger share of resources goes to people who share their values.

It’s not clear how big of a problem this is. On the one hand, it’s currently quite rare for ideological groups to deliberately try to have as many children as possible in order to increase their political and economic power. And we expect that most people would find the prospect of having huge numbers of children and brainwashing them to vote the parents’ way abhorrent. On the other hand, new technologies might lower the personal costs of having huge numbers of children (e.g., artificial wombs, robot caretakers). Other technologies could increase parents' ability to transmit their values to children (e.g., through genetic engineering or novel techniques based on advances in neuroscience or psychology). And a small number of bad actors might be able to substantially increase their population share within a few generations through rapid reproduction.

One way to navigate this is to place guardrails around the rate of reproduction or the extent of control—say, trying to ensure that parents could not use novel technology to reproduce substantially faster nor exercise substantially greater control over their children’s values than early-21st century parents were able to.

Some plausible options:

• Don’t include future generations directly. Instead, let the current generation decide whether and how to include them.

  • For example, if decisions were made by a vote, the current generation could vote to delay decisions until a later vote, where future generations could participate. (And analogously, under direct preference aggregation, if the current generation prefers to reserve some power for future generations, then that would happen).

  • Under an equal distribution proposal, the current generation could share resources with their descendants or donate to trusts that gifted resources to later generations.

    • (Although power granted to future generations is plausibly a moral public good, so we might expect it to be undersupplied in the equal distribution proposal).

• Include later generations, with restrictions to prevent strategic reproduction and/or indoctrination of one’s children. These restrictions could be chosen by a vote of the current generation.

• Include later generations, with no restrictions to prevent strategic reproduction or indoctrination of one’s children.

Digital minds

Another difficult question is whether and how digital minds should have input into decisions about space resources. Digital minds could include AI systems or simulated human brains.

In the Plan A scenario, by 2040, there may be huge numbers of digital minds. Some of these minds could be very sophisticated, with long-term stable preferences, human or superhuman intelligence, and perhaps even subjective experience. On some views, such minds have moral status and deserve a say in future governance.

But this is pretty controversial and might remain so in 2050. Different people could have wildly different views depending on their beliefs about what makes a being worthy of inclusion in the political process. It seems challenging to decide in a value-neutral way which digital minds should be empowered, and choices here could radically affect the composition of the electorate.

There are further difficulties in integrating digital minds into the decision process. While it’s conceptually straightforward to give each human adult an equal stake in the decision—e.g., one vote, one share of space resources—it’s less clear how to individuate digital minds. Should two copies of the same mind running on different computers each receive a vote? If not, how different do two minds have to be before they count as different individuals?

The issues of strategic reproduction and indoctrination discussed above are even more severe for digital minds than for human children. New digital minds could be created as fast as copying data from one computer to another. And the creators of digital minds may have a huge amount of fine-grained control over the values and behavior of those minds. Given sufficient compute, it would be trivial to spin up billions of minds that sincerely want to vote exactly the way their creator tells them.

Some plausible options:

• Include a subset of digital minds where the concerns about gaming the process through strategic reproduction and indoctrination are less strong.

  • Some possible subsets include:

    • Allow digital minds who were created before 2040 to participate in the process, perhaps with some overall cap on the share of total votes or resources they could receive.

      • (We would still somehow need to resolve the issue of individuating digital minds and deciding which digital minds deserved votes).

    • Allow digital minds that are uploads of deceased biological humans to participate in the process, one upload per original human.

    • Allow people to “adopt” digital minds and enfranchise those adopted digital minds, but only if the creators exercise no more control over the values and beliefs of those digital minds nor are producing digital minds at a faster rate than parents in 2026 could with biological children.

  • Most likely, most digital minds would not be enfranchised under any of these proposals.

• Don’t include digital minds directly. Instead, let biological humans decide whether and how to include them.

  • How this decision is made would depend on the proposal:

    • For example, if decisions were made by a vote, the biological humans could vote on a proposal to defer decisions to a later vote where digital minds were allowed to participate.

    • Under a direct endowment proposal, humans who received resources could give resources to digital minds. (Although, again, this would plausibly be a moral public good and we might expect it to be underfunded under the direct endowment proposal).

One process or many processes?

Designing a governance system involves many tradeoffs—for example, direct endowment and voting both have disadvantages. Should we hedge our bets with hybrid approaches that combine multiple processes?

Here are some examples of hybrid approaches:

• 50% of space resources are distributed directly to the population at a set time, and 50% are allocated according to the outcome of one or more votes.

• Each country (or each treaty signatory) is allocated a share of space resources. These countries each pick their own processes for deciding how to use space resources.

• 10% of space resources will be distributed in 2040, 10% will be distributed in 100 years, 10% will be distributed in 1000 years, and so on.

The main advantage of hybrid approaches is that different processes can direct resources toward different kinds of values. For example, votes may be particularly likely to fund widely shared values, whereas direct resource endowment might be better suited towards ensuring that rarer preferences are funded. By adopting a hybrid approach, we ensure that a broader set of values each get at least some resources allocated to them.

The main disadvantage of hybrid approaches is that one process might generate outcomes that other processes would regard as substantially harmful. This can be mitigated through some of the strategies discussed in the next section.

Mitigating downside risks

Under some governance regimes, individuals will be able to unilaterally decide how to use space resources. This has some strong upsides: people can create stuff that’s valuable to them even if others are indifferent to it, which makes it more likely that bounded views are satisfied and makes it more likely that nearly everyone is much better off. But it also has downsides: people might use their resources in ways that impose large negative externalities on others.

Some specific examples include:

• Seizing other people’s resources. Early settlers might try to use their star systems to launch probes to settle star systems earmarked for others who hadn’t yet laid physical claim on them, which could initiate a (potentially highly negative-sum) space race. Or people might attack star systems where others had already settled.

• Uses of resources that most people find intrinsically repugnant. For example, some people might make themselves absolute rulers over societies where nearly all citizens are oppressed or enslaved. Sadists might create new beings that experience immense pain.

• Risking the destruction of other people’s resources. People might deliberately destroy the universe (e.g., by initiating false vacuum decay).

• Taking other actions with severe negative externalities on others, e.g., starting a war with space-faring alien civilizations.

We need to strike a balance. If we’re too permissive, then a few bad actors could cause enormous damage. But if we’re too restrictive, then we risk banning things that are enormously valuable to a few people just because the majority finds them a bit offputting.

Here are some potential systems for mitigating these downside risks.

Market. If someone doesn’t like how someone else is using their resources, they can pay them to stop.

Strengths: under this proposal, an activity is only stopped when its cost to others exceeds its benefit to the actor, and they’re compensated for their loss. Thus, this proposal is less vulnerable to “tyranny of the majority” worries. Another strength is that we don’t have to collectively decide far in advance what to ban. If someone wants someone else to stop doing something, they can make an offer at any time (although, late into the future, as decision-makers spread out from Earth it might become more difficult to negotiate these trades).

Drawbacks: this could create perverse incentives: some people might do things that others find objectionable so that they’ll be paid to stop. This might result in even more objectionable behavior that would have happened without that incentive. And if a single actor wants to do something a large group objects to, the group may struggle to coordinate payment because of free-rider problems.

Voting on bans. We could prohibit activities that a supermajority of the population (say, 80%) votes to ban.

Strengths: avoids both of the drawbacks of the market solution—since there’s no transfer of resources from the people who want to ban to the people who would otherwise do the banned activity, there’s no collective action problem among the former to raise money to pay off the latter. And no one is incentivized to do objectionable things in hopes of a payout.

Drawbacks: a majority that very slightly disliked an activity could ban it even if it was very important to a small minority. Since there’s no transfer of resources, there’s no check to make sure the ban is sufficiently valuable to supporters to justify the cost to opponents.

If we went with this system, we’d have to decide when and how often to vote on which activities to ban.

• If we did a single early vote, we might fail to ban activities that were later widely considered bad.

  • Voters might not know of those activities at the time of the vote (e.g., because the activities are only possible due to unanticipated technology). Or voters might not be in reflective equilibrium at the time of the vote. For instance, perhaps they would later, on reflection, decide that they cared about the welfare of digital minds.

  • We might be able to circumvent these issues by adopting bans with broad, non-specific language and creating an AI arbiter to interpret the ban. But the arbiter may have to do nontrivial amounts of extrapolation and there might not be a single canonical way to do that extrapolation.

• If we had a recurring vote or a single late vote, then voters might have more time to develop and carry out strategies to game the voting system before the vote(s) (see discussion above).

Currently, we think that at least the following activities should be banned:

• Torture of conscious beings, including digital minds and non-human animals that are sufficiently likely to be capable of having strongly negatively-valenced experiences.

  • It might take non-trivial reflection by advanced ASI to figure out the precise bounds of this.

• Unilaterally destroying Earth-originating civilization, or taking actions that bear a significant risk of destroying all of Earth-originating civilization.

• Seizing other people’s property without permission.

• Making and/or carrying out threats to destroy other people’s resources, in order to extort them.

• Making and/or carrying out threats to create large amounts of disvalue by someone else’s lights in order to extort resources from them.

Bans decided by treaty signatories. Treaty signatories could negotiate bans before signing the treaty. This has many of the same strengths and drawbacks as bans decided by a single early vote.

Cancelling out resources. We could allow people to forfeit control of their resources in exchange for removing someone else’s control of their resources at a k:1 ratio (for k > 1).

Strengths: as with the Market approach, by imposing a cost on the people who want the ban, it ensures that people don’t block valuable activities that they only slightly dislike. And unlike the Market approach, since there’s no compensation for the person carrying out the activity, there’s no perverse incentives to do stuff that others dislike in search of a payout.

Drawbacks: as with the Market approach, a large group seeking to prevent a particular use of resources may struggle to coordinate.

Reflection

Under most of the proposals discussed in this appendix, individuals will need to make extremely high-stakes decisions, for example:

• Should I sell galaxies allocated to me in an auction? Should I save them to pass onto my children? Should I donate them to a trust, and which trust should I donate to? Should I wait until I’ve thought more to decide, and how should I go about thinking about it?

• Should I vote for a particular activity to be permanently banned throughout the universe? Should I vote for public resources to be allocated to a particular activity?

We want people to make wise decisions that reflect their values.

Making these decisions well will likely require resolving open scientific and philosophical questions. The specific questions will vary with individual values, but the following areas are likely to be relevant to a wide range of people:

Improved understanding of morally significant concepts. Many people consider consciousness and preferences to be morally significant—it’s wrong to inflict suffering on a conscious being and good to help people get what they want. But our current scientific and philosophical understanding of consciousness is insufficient to answer important questions like: which minds are conscious and which of their experiences have positive or negative valence—especially as our technological capabilities expand to allow creation of minds very different from human minds in substrate or structure.

Moral truths. There might be objective and discoverable moral truths.

Acausal trade. The reachable universe might be only a small part of what exists. There are regions of our universe outside our light cone, and there may even be other universes. Although we cannot causally interact with beings outside our light cone, our own choices may give us some evidence about how those beings will behave. On some decision theories (e.g., evidential decision theory), this should affect what we do. For instance, we might want to “trade” with these beings, doing stuff that matters to them in exchange for them doing stuff that matters to us. If this argument works out, our acausal impact could plausibly dwarf our causal impact.

Evidential Cooperation in Large Worlds (ECL) is one such mechanism. The multiverse contains many causally disconnected agents whose decisions may be correlated with ours: that is, when we observe ourselves making some decision, we should update about their choices too. So if we commit to pursuing multiverse-wide compromise values, this gives us evidence that huge numbers of other agents are doing the same. And this could be great—there might be substantial gains from trade if everyone pursues compromise values rather than their own.

These are just a few examples of potential crucial considerations that we might want to consider before making important decisions.

Deciding how to reflect

People will also need to think carefully about how they want to make these decisions, including how they want to use new technologies:

Deciding when to learn information. While it’s probably important to learn the answers to questions like those above before making object-level decisions about how to use space resources, there might be costs to learning information too early. For example, there might be positive-sum trades that can only be made before the parties to the deal know some empirical results. One example of this dynamic is wagers (discussed above). More speculatively, if we end up wanting to do ECL, it might be necessary to commit to doing ECL before we become confident whether other agents will do ECL—our decision only gives evidence about everyone else’s decision if we’re not sure what others will decide.

Engaging with ASI-generated arguments. ASI will likely be able to research the philosophical and scientific questions above more quickly and rigorously than humans, giving us access to rapid intellectual progress on how to best live out our values.

But ASI could also direct enormous intellectual effort toward crafting extremely persuasive arguments. And it’s not clear that the most compelling arguments will necessarily support true positions or positions we would endorse holding. If ASI is optimizing for maximally compelling arguments, then we might end up believing whatever ideology the ASI was subtly biased toward, or perhaps whatever ideologies are easiest to argue for compellingly.

One safeguard: ASI could refuse to generate superhumanly compelling arguments for claims it knows to be empirically false, in the same way that AIs today refuse to assist with creating biological weapons. But for some questions we care about, there might not be an empirical fact of the matter. For these cases, we need to think about how we want to be persuaded—what counts as legitimate moral suasion and what counts as illicit manipulation. It's not clear where the line is—is vividly describing the suffering of a particular being emotionally manipulative, or just helping you understand the true stakes of your decisions? And the line might be different for different people.

Delegating to aligned ASI. Another option is to delegate reflection to an aligned ASI. We will probably be able to design ASI systems that are much better suited to careful reflection than humans. These systems would be smarter, capable of understanding and integrating more complicated arguments or empirical results than any human.

Moreover, for many people, reflecting in a very careful way might be less pleasant than other ways of spending their time. An ideal reflection process might involve carefully controlling your memetic environment (if it’s possible for ASI to develop illegitimate, yet highly compelling arguments or ideologies). It might require modifying your mind to make yourself smarter, more reflective, or more free of biases that you don’t endorse. In general, it’s not clear that “optimal lifestyle for human flourishing” and “optimal lifestyle for figuring out the most idealized form of one’s values” will be the same for most people. Offloading reflection onto an ASI system that is better suited for the process might let those people get the best of both worlds.

Digital reflection has other advantages difficult to replicate on a biological substrate, including running many parallel reflection processes, reverting to earlier checkpoints, and potentially greater transparency about the trajectory that one’s values took over time. (This last is particularly important in versions of this proposal where many people’s preferences are aggregated together, since people might have incentives to strategically self-modify or misrepresent their preferences. Carrying out a digital reflection with a legible trace could let people verify that they hadn’t done so).

Delegating to future generations. Another option is to delegate decisions to one’s descendants or to future generations more broadly. If so, how should one raise their children? Parents will plausibly have much greater capacity to shape their children’s values—how should they exercise this power?

We’re not sure what the right answers are, and indeed the right answers may vary from person to person based on their values, but we think these are important decisions and want to help people make them carefully.

The role of governance

It’s important for the governance structure to facilitate reflection.

• Timing of decisions. Delaying decisions about how to use space resources gives decision-makers more time to think. Although ASI labor may speed up the rate of scientific and philosophical research, many people will likely need years or decades to integrate those findings into their thinking. Additionally, reaching these decisions may require substantial introspection into what the decision-maker truly values, which might also be time-consuming.

  • For governance structures where the timing of an important decision is scheduled in advance—e.g., a vote or a preference aggregation—this consideration favors delaying that decision (although there are considerations in the other direction, discussed above).

  • An important advantage of direct endowment governance structures is that they allow individuals to reflect for as long as needed before making object-level decisions. If individuals were still prone to hasty decisions, we could—somewhat paternalistically—require them to wait a few decades before selling or committing their resources to a trust.

• Subsidizing reflection. Governments could offer all voters or resource recipients access to ASI labor to help them decide how to use those resources. Governments could also directly fund research into open scientific and philosophical questions likely to bear on decisions many people will face.

• Avoiding perverse incentives. Governance structures could incentivize people to warp their reflective process or interfere in others’ reflective processes. For example:

  • Voting systems that offer voting rights to future generations could incentivize strategic reproduction and indoctrination.

  • Voting systems (with lie detection) or direct preference aggregation systems could incentivize people to modify their own preferences strategically to gain an advantage (just as traditional voting systems incentivize strategic voting).

The answers to some of the questions above could have implications for which governance procedures are best. One important tradeoff among the governance proposals is between enabling coordination on (moral) public goods and enabling individuals to pursue their own values. ASI-powered research into these questions could shed light on:

• Whether there are important public goods that we might want to coordinate on. For example, perhaps acausal trade has large fixed costs—e.g., running lots of simulations of alien civilizations to figure out the distribution of values that trade partners have—and it would be wise to ensure that there are governance mechanisms that let us pool our resources to do that.

• The degree to which humans’ moral views converge after endorsed reflection. If people converge substantially, this favors governance proposals with coordination mechanisms that let them pool resources to pursue shared values. If there’s a lot of divergence, this favors governance proposals that allow each person to pursue their own values.

Takes

Mia’s takes

• My actual take is that we should wait for aligned ASI and ask it to design a governance proposal that meets the desiderata above (and perhaps more that I haven’t thought of).

• It seems very important that everyone making irreversible high-stakes decisions carefully reflect on what they care about with the help of aligned ASI advisors. To make this more likely, governments should probably fund lots of public-domain ASI research into scientific and philosophical questions that are likely to be of broad interest (like the questions above). They should also subsidize ASI compute used by people thinking about these decisions.

• Likewise, it seems important that decisions about the use of resources beyond the solar system are made for altruistic or moral reasons, not selfish ones.

  • We should guarantee that all humans alive in 2040 and their descendants get enough to support them indefinitely at a very comfortable standard of living (i.e., at least the material standard of living of a multi-millionaire in 2026). This will hopefully lower the personal stakes of the decision about what to do with space and encourage people to be more high-minded when they vote.

    • It is also great on common-sense ethical grounds—on a cosmic scale, it’s relatively cheap to satisfy people’s preferences, and so we should do that.

• I think coordination capacity—where we can collectively use resources to provide (moral) public goods—is potentially really important and it's not clear that we'll be able to do that with voluntary contracts. Unless we're confident that this is possible, it seems good to maintain capacity to coordinate, i.e., have a government (or a few governments) that control substantial amounts of resources (or which are empowered to levy taxes if they need to).

• On the other hand, I also want to enable people with unusual value systems to spend some resources pursuing their values.

• For these reasons, my top choice is direct preference aggregation if it’s technically feasible and not so unpleasant that many people opt out, perhaps combined with automated reflection (like what’s described here).

• If that doesn’t work, my next favorite option is a hackier solution that ensures that some resources go to individuals and some resources are decided collectively. Here’s one way of approaching this.

  • At least 5% of resources are guaranteed to be directly granted to humans alive in 2040.

  • By default, the remaining 95% of resources will also be directly granted to humans alive today, unless a supermajority votes that it should instead be spent on public goods.

  • If highly reliable lie detection is available, governments could use that to check for strategic voting and disqualify people who were sufficiently likely to be executing these strategies.

  • Governments could also have ASI systems refuse to assist with attempts to coordinate to vote strategically (in the same way that ASI systems will presumably continue to refuse to assist with other harmful requests—e.g., making bioweapons).

• We will also vote on bans and on the process to use to amend the list of bans (e.g., should we have recurring votes?). There should be a high threshold for bans, perhaps 90%.

Thomas’s takes

• I agree that we should wait for aligned ASI and ask it to design a good governance proposal. The most important things for us to do now are (1) avoid losing control of the future, e.g. via misaligned AI takeover or a human dictator getting control, (2) distribute the power reasonably, and (3) try to do philosophy well.

• I think there’s a good chance that in practice, the best path forward will be to lock in some governance structure over the future, because deliberation within a fixed structure (like the ones discussed above) will go better than in a looser structure (like today’s government) where, e.g., more power will accrue to people who are, for example, more politically well connected.

• My tentative favorite concrete guess for what to do above is “direct endowment” with some form of amendment process with a very high threshold (e.g. 90% approval). I think it’s very similar to voting based proposals, where you “vote” by making trades with other people about what to do with your resources. The public goods concern is the strongest counterargument, and if there is a voting system discovered that addresses this concern without adding additional, even more dire consequences, then I would change to wanting that.

• All voting systems that we know of are screwed up, and I would not want them to be used for something as high stakes as deciding about what to do in the future. But I’m pretty optimistic about this changing, for two reasons:

  • I think there’s a good chance that ASIs (and humans who think about it for longer) will come up with voting systems that incentivize honest reporting of preferences, and choose a good policy that aggregates the votes.

  • I think that we might be able to do preference elicitation along the lines of “direct preference aggregation” above. Once you have honest preferences, the problem of designing a good voting system seems much easier.

• I care a lot about power in the future going to virtuous people. I think that it would be really bad if a huge fraction of the power in the future goes to selfish people or terminally evil people (e.g. people who want to cause massive amounts of suffering). Of course, different people have different moral views that are reasonable, and I don’t think it’s fair, for example, to claim that everyone should follow my moral views. But I do think that it’s reasonable to exclude people with sufficiently bad moral views, as agreed on from the perspective of the rest of society, from having a large amount of power over the future, for example, to exclude sociopaths from many of the preference aggregation systems above.

• I think that any attempt to have “normal governance” continue is not going to work, because our current systems will be totally exploited by actors using superintelligence to gain power. I think that we’re going to need to make massive changes to our system of government really quickly in order to maintain a reasonable ongoing distribution of power.