Main thesis

  • Upcoming waves of technologies (AI and gene editing) will destabilize societies and nation-states to the point where chaotic failed states or dystopian surveillance states become the likely default outcomes
  • Constraining technology so that it continues to do far more good than bad (containment) seems almost impossible: it needs to happen in every lab and company in every country forever despite overwhelming incentives to the contrary, but it remains indispensable.
  • This makes it the greatest challenge of the coming decades, likely even more pressing and much less appreciated than global warming.

Coming wave technologies

  • Synthetic intelligence: machine learning
  • Synthetic biology/life: gene editing and synthesis
  • Support technologies: robotics, quantum computing, fusion energy

Coming wave technology characteristics that make containment difficult

  • Asymmetric impact: risks (like upsides) created by small teams extend to entire societies (engineered pandemic, disinformation campaign) - just like a single person can now broadcast globally (through the information wave with Internet and social media), a single person will be able to act globally
  • Extremely fast evolution (at the pace of software)
  • General purpose
  • Autonomy: potential beyond use as tools (AI that chooses its own goals, gene editing that starts a new species)

Incentives against containment

  • Geopolitical competition among leading nation states: the attitude of behaving like adversaries are engaged in an arms race (commonplace in governments and think tanks) is a self fulfilling prophecy (classic prisoner’s dilemma) - started with AlphaGo in China
  • Global research ecosystem valuing curiosity and openness, operating through evidence-based open peer review (arXiv, GitHub) - openness is not just to enable falsifiability/criticism in Science but also for prestige and as a track record for researchers
  • Financial upsides: unbounded possible upsides lead to huge investments from big tech and VCs
  • Need to address global challenges of this century: global warming, care for aging population and its replacement in the workforce
  • Ego: making history, beating others, impressing others

The coming wave can be characterized by the broad availability of power

  • Power is the ability to affect the world towards goals (including the behavior of others)
  • Technology has always been a major (the most important?) source of power, but this wave
    • Will bring more power than any prior technology: an army of the world’s best experts on any subject with the ability to autonomously take action towards your goals
    • More broadly accessible than ever before: everyone with a smartphone and Internet - the same will apply to physical power (robots) with the falling cost of hardware

This will place a huge stress on societies and nation states

  • Unprecedented pressure on the nation states to provide security, stability, and public services against empowered bad actors: in the short term, disinformation campaigns through synthetic media; in the long term, pretty much anything is possible
  • Even ignoring bad actors, the simple widespread presence of such powerful technologies can have unintended disastrous consequences (lab leak)
  • Labor market disruptions: in the short term, AI will automate large portions of today’s jobs, both intellectual and physical, leading to underemployed, insecure, and angry populations
  • Concentration of power for corporations: the best will enjoy compounding returns on intelligence (use ⇒ data + deployment experience + attract best engineers ⇒ better product ⇒ more use) and challenge nation states - this coincidence of centralizing and decentralizing forces was already a feature of the Internet

It will likely lead to a tightening of the nation state’s grip on power: unprecedented levels of surveillance in the name of security

  • This is made much easier than ever before through AI as China is already showing today
  • If this seems unimaginable, think of Covid lockdowns

The alternative path of trying to stop technological evolution, even if it were possible (it doesn’t seem to be), is not a viable option

  • Our modern societies are predicated on constant growth, only possible through technological progress
  • We simply cannot deal with this century’s problems (global warming, aging population) without new Science and technology - maintaining, let alone improving, standards of living needs technology
  • Static societies are bound to collapse, unable to invent their way out of problems

What can we do about this? 10 ideas for containment, concentric circles of guardrails around technology

  • Technical safety: isolate systems from the world and ensure we maintain control - a proportion (say 20%) of research from leading organizations should be mandatorily dedicated to safety research and share findings
  • Transparency and accountability: build trust by proving safety and being accountable to governments and cross-organization institutions thar track failures, KYC requirements on APIs, screening of all synthesized DNA, cryptographic tracking of model sharing, verification enforced through code backdoor by authorities
  • Choke points: Nvidia makes all chips, all manufactured by TSMC in Taiwan, with equipment from ASML in the Netherlands; the AI frontier is a handful of industrial labs - use choke points to slow down progress and get more time to operationalize containment
  • Builder responsibility: front line technology builders have a responsibility - more than anyone else, although we need diverse perspectives - to be critics and remember that technology is just a means to an end
  • Company responsibility: shareholder returns as a company’s goal is poorly suited to containment, we need to reconcile profits and social purpose in company legal structures - this will be difficult: shareholder capitalism works because it is simple and clear
  • Government governance
    • Governments should understand what data their population generates and the latest research to leverage AI - to make this possible, they should have in-house world class researchers and engineers (with compensation aligned with the private sector)
    • They should log all ways technology causes harm
    • Frontier AI labs and models should be licensed after a certain threshold of capabilities
    • Fiscal policy is crucial to avoid extreme concentration of power in the hands of mega corporations and fund public services in an era of increasing state fragility: greater tax on older forms of capital, new tax on automation systems
  • International cooperation: new global institutions starting with a focus on transparency and accountability
  • A self-critical culture of openness in sharing failures to learn together (like the aviation industry) instead of today’s culture of secrecy about failures
  • Popular buy in: large change requires the average person to care and demand change (abolition of slavery, women vote, civil rights, global warming)
  • Coherence: containment is an emergent phenomenon of the interaction between all guardrails, safety is not a destination but an ongoing dynamic and unstable process, a narrow and never ending path