Planetary Optimization and Advanced General Intelligences: the 'intelligence stairway'
The emergence of self-replicating physical systems 4 billion years ago marked the start of terrestrial evolution - which we view as a set of competing processes, optimising for the fitness of self-replicators. After evolution produced humans, however, a phase change occurred, because the optimisation power of humans, through technology, exceeded that of evolution. One can ask if future computers, smarter than their creators, could cause a similar phase transition. If so, then such AGIs (artificial general intelligences) will lead to a new AGI-driven "intelligence explosion" phase. Models indicate this may manifest itself as a sudden global ecological catastrophe. I discuss two ways of avoiding the catastrophe: either (i) collaboration (the "Friendly AI" approach), or (ii) artificial limitation of the AI-s to narrow domains. The latter approach is more pragmatic, and will require a "practical safety protocol" for AI developers to prevent accidental AGI's being produced.
