Notable reviews of arguments for AGI ruin

Here’s a collection of reviews of the arguments that artificial general intelligence represents an existential risk to humanity. They vary greatly in length and style. I may update this from time to time.

  • Here, from my perspective, are some different true things that could be said, to contradict various false things that various different people seem to believe, about why AGI would be survivable on anything remotely remotely resembling the current pathway, or any other pathway we can easily jump to.
  • This report examines what I see as the core argument for concern about existential risk from misaligned artificial intelligence. I proceed in two stages. First, I lay out a backdrop picture that informs such concern. On this picture, intelligent agency is an extremely powerful force, and creating agents much more intelligent than us is playing with fire -- especially given that if their objectives are problematic, such agents would plausibly have instrumental incentives to seek power over humans. Second, I formulate and evaluate a more specific six-premise argument that creating agents of this kind will lead to existential catastrophe by 2070. On this argument, by 2070: (1) it will become possible and financially feasible to build relevantly powerful and agentic AI systems; (2) there will be strong incentives to do so; (3) it will be much harder to build aligned (and relevantly powerful/agentic) AI systems than to build misaligned (and relevantly powerful/agentic) AI systems that are still superficially attractive to deploy; (4) some such misaligned systems will seek power over humans in high-impact ways; (5) this problem will scale to the full disempowerment of humanity; and (6) such disempowerment will constitute an existential catastrophe.
[continue reading]

Abstracts for July 2017

  • Modewise entanglement of Gaussian states
    Alonso Botero and Benni Reznik
    We address the decomposition of a multimode pure Gaussian state with respect to a bipartite division of the modes. For any such division the state can always be expressed as a product state involving entangled two-mode squeezed states and single-mode local states at each side. The character of entanglement of the state can therefore be understood modewise; that is, a given mode on one side is entangled with only one corresponding mode of the other, and therefore the total bipartite entanglement is the sum of the modewise entanglement. This decomposition is generally not applicable to all mixed Gaussian states. However, the result can be extended to a special family of “isotropic” states, characterized by a phase space covariance matrix with a completely degenerate symplectic spectrum.

    It is well known that, despite the misleading imagery conjured by the name, entanglement in a multipartite system cannot be understood in terms of pair-wise entanglement of the parts. Indeed, there are only N(N-1) pairs of N systems, but the number of qualitatively distinct types of entanglement scales exponentially in N. A good way to think about this is to recognize that a quantum state of a multipartite system is, in terms of parameters, much more akin to a classical probability distribution than a classical state. When we ask about the information stored in a probability distributions, there are lots and lots of “types” of information, and correlations can be much more complex than just knowing all the pairwise correlations. (“It’s not just that A knows something about B, it’s that A knows something about B conditional on a state of C, and that information can only be unlocked by knowing information from either D or E, depending on the state of F…”).

[continue reading]

Abstracts for March 2017

  • Recent progress in synthetic chemistry and molecular quantum optics has enabled demonstrations of the quantum mechanical wave–particle duality for complex particles, with masses exceeding 10 kDa. Future experiments with even larger objects will require new optical preparation and manipulation methods that shall profit from the possibility to cleave a well-defined molecular tag from a larger parent molecule. Here we present the design and synthesis of two model compounds as well as evidence for the photoinduced beam depletion in high vacuum in one case.

    The technique of using “laser grating”, in place of physical grating (slits), for producing spatial interference of molecules relies on the laser’s ability to ionize the molecule. (Once ionized, standing electric fields can sweep it out of the way.) But for some molecules, especially large nanoparticles, this is ineffective. Solution: attach a molecular tag to the nanoparticle that reliably cleaves in the presence of a laser, allowing the nanoparticle to be vacuumed up. Rad.

  • This chapter discusses the asymptotics, singularities, and the reduction of theories. The reduction must involve the study of limits—asymptotics. The reduction is obstructed by the fact that the limit is highly singular. In addition, the type of singularity is important, and the singularities are directly connected to the existence of emergent phenomena and underlie some of the most difficult and intensively studied problems in physics today. The chapter provides six examples of singular limits and emergent phenomena such as special relativity and statistical mechanics. Reduction in its simplest form is well illustrated by special relativity.
[continue reading]

Abstracts for October 2016

  • One of von Neumann's motivations for developing the theory of operator algebras and his and Murray's 1936 classification of factors was the question of possible decompositions of quantum systems into independent parts. For quantum systems with a finite number of degrees of freedom the simplest possibility, i.e. factors of type I in the terminology of Murray and von Neumann, are perfectly adequate. In relativistic quantum field theory (RQFT), on the other hand, factors of type III occur naturally. The same holds true in quantum statistical mechanics of infinite systems. In this brief review some physical consequences of the type III property of the von Neumann algebras corresponding to localized observables in RQFT and their difference from the type I case will be discussed. The cumulative effort of many people over more than 30 years has established a remarkable uniqueness result: The local algebras in RQFT are generically isomorphic to the unique, hyperfinite type III, factor in Connes' classification of 1973. Specific theories are characterized by the net structure of the collection of these isomorphic algebras for different space-time regions, i.e. the way they are embedded into each other.
    [arXiv:math-ph/0411058]

    One of the key subtleties about trying to study quantum information in a field theory is that you can’t formally decompose the Hilbert space into a tensor product of spatially local subsystems. The reasons are technical, and rarely explained well. This paper is an exception, giving an excellent introduction to the key ideas, in a manner accessible to a quantum (non-field) information theorist. (See related work by Yngvason this blogpost by Tobias Osborne and my previous discussion re: Reeh-Schielder theorem.)

[continue reading]

Abstracts for July-August 2016

  • Local dark matter searches with LISA
    Massimo Cerdonio, Roberto De Pietri, Philippe Jetzer and Mauro Sereno
    The drag-free satellites of LISA will maintain the test masses in geodesic motion over many years with residual accelerations at unprecedented small levels and time delay interferometry (TDI) will keep track of their differential positions at a level of picometers. This may allow investigations of fine details of the gravitational field in the solar system previously inaccessible. In this spirit, we present the concept of a method for measuring directly the gravitational effect of the density of diffuse local dark matter (LDM) with a constellation of a few drag-free satellites, by exploiting how peculiarly it would affect their relative motion. Using as a test-bed an idealized LISA with rigid arms, we find that the separation in time between the test masses is uniquely perturbed by the LDM, so that they acquire a differential breathing mode. Such an LDM signal is related to the LDM density within the orbits and has characteristic spectral components, with amplitudes increasing in time, at various frequencies of the dynamics of the constellation. This is the relevant result in that the LDM signal is brought to non-zero frequencies.
  • We review some recent developments in the statistical mechanics of isolated quantum systems. We provide a brief introduction to quantum thermalization, paying particular attention to the eigenstate thermalization hypothesis (ETH) and the resulting single-eigenstate statistical mechanics. We then focus on a class of systems that fail to quantum thermalize and whose eigenstates violate the ETH: These are the many-body Anderson-localized systems; their long-time properties are not captured by the conventional ensembles of quantum statistical mechanics.
[continue reading]

Abstracts for May-June 2016

Lots of matter interference experiments this time, because they are awesome.

  • Quantum Interference of a Microsphere
    H. Pino, J. Prat-Camps, K. Sinha, B. P. Venkatesh, and O. Romero-Isart
    We propose and analyze an all-magnetic scheme to perform a Young’s double slit experiment with a micron-sized superconducting sphere of mass 10^{13} amu. We show that its center of mass could be prepared in a spatial quantum superposition state with an extent of the order of half a micrometer. The scheme is based on magnetically levitating the sphere above a superconducting chip and letting it skate through a static magnetic potential landscape where it interacts for short intervals with quantum circuits. In this way a protocol for fast quantum interferometry is passively implemented. Such a table-top earth-based quantum experiment would operate in a parameter regime where gravitational energy scales become relevant. In particular we show that the faint parameter-free gravitationally-induced decoherence collapse model, proposed by Diósi and Penrose, could be unambiguously falsified.


    An extremely exciting and ambitious proposal. I have no ability to assess the technical feasibility, and my prior is that this is too hard, but the authors are solid. Their formalism and thinking is very clean, and hence quite abstracted away from the nitty gritty of the experiment.

  • Do the laws of quantum physics still hold for macroscopic objects -- this is at the heart of Schrodinger's cat paradox -- or do gravitation or yet unknown effects set a limit for massive particles? What is the fundamental relation between quantum physics and gravity? Ground-based experiments addressing these questions may soon face limitations due to limited free-fall times and the quality of vacuum and microgravity.
[continue reading]

Abstracts for March-April 2016

  • Unruh effect without trans-horizon entanglement
    Carlo Rovelli and Matteo Smerlak
    We estimate the transition rates of a uniformly accelerated Unruh-DeWitt detector coupled to a quantum field with reflecting conditions on a boundary plane (a “mirror”). We find that these are essentially indistinguishable from the usual Unruh rates, viz. that the Unruh effect persists in the presence of the mirror. This shows that the Unruh effect (thermality of detector rates) is not merely a consequence of the entanglement between left and right Rindler quanta in the Minkowski vacuum. Since in this setup the state of the field in the Rindler wedge is pure, we argue furthermore that the relevant entropy in the Unruh effect cannot be the von Neumann entanglement entropy. We suggest, an alternative, that it is the Shannon entropy associated with Heisenberg uncertainty.

    See also the related works by Gooding and Unruh, which connect to Pikovski et al. (blogged here).

  • What is the Entropy in Entropic Gravity?
    Sean M. Carroll and Grant N. Remmen
    We investigate theories in which gravity arises as a consequence of entropy. We distinguish between two approaches to this idea: holographic gravity, in which Einstein's equation arises from keeping entropy stationary in equilibrium under variations of the geometry and quantum state of a small region, and thermodynamic gravity, in which Einstein's equation emerges as a local equation of state from constraints on the area of a dynamical lightsheet in a fixed spacetime background. Examining holographic gravity, we argue that its underlying assumptions can be justified in part using recent results on the form of the modular energy in quantum field theory. For thermodynamic gravity, on the other hand, we find that it is difficult to formulate a self-consistent definition of the entropy, which represents an obstacle for this approach.
[continue reading]

Abstracts for February 2016

  • Non-Markovianity hinders Quantum Darwinism
    Fernando Galve, Roberta Zambrini, and Sabrina Maniscalco
    We investigate Quantum Darwinism and the emergence of a classical world from the quantum one in connection with the spectral properties of the environment. We use a microscopic model of quantum environment in which, by changing a simple system parameter, we can modify the information back flow from environment into the system, and therefore its non-Markovian character. We show that the presence of memory effects hinders the emergence of classical objective reality, linking these two apparently unrelated concepts via a unique dynamical feature related to decoherence factors.

    Galve and collaborators recognize that the recent Nat. Comm. by Brandao et al is not as universal as it is sometimes interpretted, because the records that are proved to exist can be trivial (no info). So Galve et al. correctly emphasize that Darwinism is dependent on the particular dynamics found in our universe, and the effectiveness of record production is in principle an open question.

    Their main model is a harmonic oscillator in an oscillator bath (with bilinear spatial couplings, as usual) and with a spectral density that is concentrated as a hump in some finite window. (See black line with grey shading in Fig 3.) They then vary the system’s frequency with respect to this window. Outside the window, the system and environment decouple and nothing happens. Inside the window, there is good productions of records and Darwinism. At the edges of the window, there is non-Markovianity as information about the system leaks into the environment but then flows back into the system from time to time. They measure non-Markovianity as the time when the fidelity between the system’s state at two different times is going up (rather than down monotonically, as it must for completely positive dynamics).

[continue reading]