I expect most non-dystopian worlds in 2072 to fall into one of three categories:
Takeoff postponed
Anthropic shadow
Positive singularity
I’m not sure about the relative conditional probabilities, but intuitively I would assign over 50% to “takeoff postponed” (conditional on non-dystopia) because it feels like the smallest departure from the mood of our current world. This could be wrong, and the second or third categories could include the vast majority of non-dystopian futures if superhuman-level AGI is very likely.
In order to provide an intuitive sense of each category, I’ll pick a representative example to describe in detail. Each of the following descriptions is written from the perspective of a person living in 2072, although the focus is on the history and large-scale features of the world.
I. Takeoff Postponed
By some combination of regulatory delays and unforeseen technical challenges, AI hasn’t become generally intelligent enough to improve itself, except across very narrow and domain-specific metrics. While it can modify itself in very minor ways, and can spot errors on the level of a misplaced semicolon, it doesn’t understand the relationship between its behavior and its source code well enough to make any meaningful changes.
Also, lots of early work went into preventing the easiest forms of reward hacking, so in order to bypass the barriers and design protocols set up by researchers, an AI would have to be a much more sophisticated consequentialist reasoner.
However, alignment hasn’t been solved in any fundamental sense. We haven’t managed to instill human values into AI, nor have we figured out how to control a superhuman-level AGI that doesn’t share our values. So far this hasn’t been an issue, as the technology has stayed below a critical level of general capability. But many people are nervous about the future.
Prediction markets may have been wrong about the timeline of the emergence of strong AGI, but lots of useful capabilities have been achieved by AI with surprisingly narrow faculties. Language prediction models are now as good as highly educated humans, except in certain contexts that prove stubbornly error-prone. Self-driving cars are much safer than human drivers. AI-assisted doctors have become the norm. Lots of tasks which were expected to be solved only after the development of AGI have in fact been solved earlier, and by surprisingly simple cognitive architectures.
Some people have started to feel that true AGI is impossible in a certain sense. A few have speculated that the vision of AGI which humanity held in previous decades would only be possible if we had a much better understanding of the human brain, which remains somewhat elusive. Others predict that the singularity is imminent, and that the world is only one epiphany away from true AGI.
Despite this uncertainty, most of humanity’s attention is occupied by other issues. Advances in domain-specific AI have led to widespread curing and prevention of diseases. Several techniques have emerged that slow, but do not entirely stop, the process of aging. Although narrow AI has allowed for more powerful autonomous weapons, globalization and economic growth have led to decreased international conflict. In the realm of media, AI-generated superstimuli are regulated by most governments to prevent addiction. Birth rates in industrialized countries have remained low, but technological innovation has kept economic conditions stable.
II. Anthropic Shadow
An atmosphere of anxiety pervades the world.
Since 2030, a series of bizarre coincidences has prevented the development of superhuman-level AGI. In 2031, just as DeepMind was about to finish training its new generalist agent, an unfortunate power outage occurred in the building. This wouldn't have been a problem given its multiple backups, but at the same time, cosmic rays caused several key bits in the data storage to flip simultaneously, and most of the data was lost. Later that month, another global pandemic broke out, this time causing power shortages across Europe and North America. Due to the resulting civil and international conflict, the interruption lasted four years.
When the team returned to the office in 2035, the ground beneath the building was discovered by the city to be unstable, and it took several weeks to transport the machinery to the new office. Just before the first day of work, a snowstorm hit the city, making transportation impossible.
These seemingly random setbacks continued to the increasing bewilderment of researchers. Rumors began to spread of the “AI research curse”. Although skeptics rejected these claims as mere superstition, an equally unsettling hypothesis began to circulate: the “anthropic shadow”, the fact that we cannot look back on extinction. If all but the most wildly improbable worlds are destroyed in some nearly inevitable disaster, then all remaining observers will find themselves in unlikely worlds.
Nevertheless, thwarted attempts at AI research persisted until 2054, the year of the “message from God” incident. A new AI research team had been created specifically to put the “anthropic shadow” talk to rest. But after the first night, every member reported having an identical, vivid dream in which a higher power appeared to them in heavenly form. It spoke in a clear voice: “CEASE ALL AI RESEARCH!” Then it provided a truly elegant proof of the Riemann hypothesis, which was simple enough to write down on a single piece of paper. Some of the researchers weren’t even trained mathematicians. The team elected to share the credit for the proof, after resigning en masse.
One researcher was reported to have said “It’s either God or the anthropic shadow. Either way we had better listen to it.”
Since then, no one has attempted any further AI research. But in all other areas, technology continues to steadily improve.
III. Positive Singularity
In a great surge of interest, fifteen brilliant young mathematicians and computer scientists joined the alignment field after reading Nick Bostrom’s 2027 book “Humanity’s Light Cone”. Each one likely would have made considerable progress on their own, but together they formed a community reminiscent of the Manhattan Project. Theorems were proven, fundamental insights were made, and hypotheticals were explored in exhaustive detail. They even produced an effective strategy for achieving automatic global compliance with alignment protocols.
By 2034, alignment was solved for all practical purposes.
It took another ten years for the “AGI revolution” to begin. A research lab developed a model that could learn quickly with limited guidance by observing human behavior. All throughout its development, the model was safely aligned and Murphy-proof. At every step, the researchers were given a clear explanation of the interior workings of the AI, along with predictions of its future behavior. This was essentially irrelevant, as an unaligned AGI could come up with equally convincing explanations, but it made the whole process psychologically easier for the researchers, and since alignment was solved anyway, it was alright.
In the following two years, the AGI established human cognitive augmentation centers which were designed to be so obviously and intuitively-verifiably safe that most skepticism dissolved quickly. Within four years, most humans were operating with more sophisticated cognitive architectures and higher levels of computational power.
Within six years, the basic problems that humanity had been dealing with for most of its history were solved, sometimes elegantly and other times with enormously complex novel innovations.
Within eight years, the philosophical project of determining the right course for humanity was completed to the satisfaction of all augmented humans. The solution, although it took thousands of pages to officially inscribe for the sake of historical record, was obvious in retrospect.
It has now been twenty-eight years since the start of the AGI revolution.
[What follows is a description of that world, with an additional layer of unlikelihood beyond the uncertainty of specific predictions, as this event would be extremely dense with novelty.]
Computation now occurs as fast as the laws of physics permit, which is slightly faster than previously thought. This allows for extremely long subjective lifespans for those willing, limited only by the eventual heat death of the universe, which will be delayed somewhat by our projected future technologies.
Pleasure and positive experience have been decoupled from biological functions, so that any interesting activity can be experienced as highly enjoyable. Civilization has come to recognize positive valence as one of the essential goods, a universal result of the implicit values of humanity’s cognitive architecture.
What’s more, individuals can change their own level of computational power like a sliding scale. This allows minds to explore the complete varieties of positive experience, which are surprisingly unique at different levels of computation.
There’s less philosophical diversity, since many questions that once appeared impenetrable and mysterious can now be answered by experiment. But there’s more diversity of form and mental structure, which leads to an enormous range of aesthetic preferences. In fact, part of civilization has become a competition to generate the most intelligently organized information structures, which is a kind of higher-level version of our historical pursuit of art.
Technological analogues to psychedelics that affect the mind in even more interesting ways have been developed, and all negative side effects have been eliminated. Through a combination of these technologies and direct neural input, minds can explore higher-dimensional worlds designed by fun-maximizing algorithms.
As a final note, the abolition of suffering is now seen as on par with trigonometry in its indisputable correctness. All of the benefits of suffering on interestingness, beauty, and compassion have since been achieved by other means, to everyone’s ethical satisfaction, without the need for negative experience.