TL;DR
- Presents a formal argument that any rule-based system faces an infinite space of possible input-output transformations, and no instruction set can cover all of them because some transformations are mutually contradictory - saving and deleting the same input, for instance.
- Argues every instruction added to such a system implicitly creates at least one prohibition, forbidding some alternative action, so a system with n instructions carries n-plus prohibitions, meaning what the system cannot do always equals or exceeds what it can do.
- Notes this is irrelevant for finite, predictable input and output domains, but argues general AI must handle infinite and unpredictable inputs and outputs, so it needs to retain something close to the full space of possible transformations.
- Concludes that general AI therefore cannot be built through rule-based instruction programming, since every instruction added to enable one capability forbids others, and every attempted fix is itself a new instruction generating further prohibitions - complexity compounds rather than resolves as a system approaches general intelligence.
- Proposes an escape route: iterative or fractal systems, where one simple generating instruction spawns an unbounded variety of independent subsystems whose instructions and prohibitions never mathematically interfere with each other, keeping the parent system’s own complexity minimal, with a companion piece cited as a practical demonstration of the idea.
Key Takeaways
- The argument is a purely formal one about instruction-based systems in general, not a claim about the limits of any specific current AI architecture. Its force depends on accepting that general intelligence genuinely requires handling an unbounded, unpredictable space of transformations, not merely a very large finite one.
- Framing prohibitions as compounding faster than capabilities reframes the difficulty of general AI as a structural property of rule-writing itself, rather than a matter of needing more rules, more compute, or more careful engineers. That reframing is what motivates looking outside conventional instruction-based programming altogether.
- The proposed fractal escape route works by preventing instructions in different subsystems from ever having to interact. It buys unbounded behavioural variety without ever writing a global rule that has to reconcile every case, rather than resolving the contradictions the essay identifies.
- The piece functions as the theoretical companion to the site’s separate proposal for a disk-space-driven self-improving AI. The argument here supplies the reason that design uses subsystem generation rather than direct instruction; it is not a demonstration in itself.
Assume a system that uses instructions (rules or sets of rules) to transform inputs into outputs.
An infinite number of transformations are possible.
No instruction — however comprehensive — can incorporate all possible transformations, purely because some of them are mutually contradictory. The instruction “save input A” is mutually incompatible with the instruction “delete input A”, for example.
Because no instruction can be global, every instruction added to the system also adds at least one prohibition. The instruction “do A” forbids the system from not doing A. In practice, the addition of a new instruction will generally create more than one additional prohibition. “Add 2 to input A”, for example, forbids the system from adding 3 or 4 to input A, as well as deleting or ignoring it, converting it to a different data format, etc.
Thus, a system with n instructions will also contain n + x prohibitions, where x is an integer equal to or greater than zero.
This means that the number of things that the system cannot do will always be greater than or equal to the number of things that it can do.
When dealing with finite and predictable sets of inputs and/or outputs, this is of no concern, since their finite and predictable nature limits the number and nature of the transformations required of the system.
General AI, however, must be able to deal with infinite and unpredictable input and output sets. Therefore, it must retain the capacity to perform all (or close to all) possible transformations.
This means that a general AI cannot be programmed through the establishment of rule-based instructions, since every instruction added to enable it to carry out a particular transformation will forbid it from carrying out one or more others. To avoid or mitigate the effects of these prohibitions, one or more exceptions must be written, which — being themselves instructions — create new prohibitions of their own.
In other words, the closer programmers approach the levels of complexity required for general AI, the more complex the problem becomes.
To avoid this problem, it is necessary to have recourse to iterative or fractal systems, via which an infinitely complex set of subsystems may be created through a simple initial instruction. Under such a system, the initial instruction generates a set of subsystems of potentially infinite variety. While these subsystems have instructions and (therefore) prohibitions of their own, there is no mathematical interference between the instructions and prohibitions of each system. Instructing System 1 to perform transformation A and thereby forbidding it from performing transformation B has no effect on System 2’s ability or inability to perform transformation B. Thus, the parent system retains its initial level of Kolmogorov simplicity. Because it is only ever subject to one instruction — the sub-system generation instruction — the number of prohibitions to which it is subject does not increase.
In theory, using this method, it is possible to create a complex system with just one global prohibition. We demonstrate how this could be done.