The Ingenuity Gap

Welcome to the "Ingenuity Theory" page of this website. A brief note of introduction:

In The Ingenuity Gap, I don't provide much detail on the theory that underpins my thinking about ingenuity and its role in modern societies. There are a number of difficult technical issues that I mention only in passing in the book. In the following pages, I have begun to address some of these issues, and I would be delighted to receive feedback on my ideas in the Forum section of this website.

Below, I discuss or will soon discuss:

The advantages and disadvantages of the "sets of instructions" definition of ingenuity that I adopt.
How we might usefully measure ingenuity.
How we might usefully think about the "quality" of ingenuity.
How our social, economic, and political values affect the requirement for ingenuity and its supply.
A more refined categorization of ingenuity types than the simple technical/social distinction used in the book.
The complete set of factors (as I see it) driving up our requirement for ingenuity in today's world and the specific causal relations among these factors.
The benchmark against which we should measure our rising requirement for ingenuity. (In my previous writings, I've used a standard that I call the "constant-satisfaction requirement," which is the amount of ingenuity required to maintain a society's aggregate utility. See my original "Ingenuity Gap" article, cited below.)
How our time horizon (or what economists would call our "discount rate") affects our current requirement for ingenuity.
A systematic account of the factors that constrain the supply of ingenuity.
The recursive nature of ingenuity supply (i.e., that we need ingenuity to generate and deliver ingenuity).
The distribution and use of ingenuity within an economy and society (a matter that is quite distinct from the strict issue of supply).
And finally, why the "political will" rebuttal to my argument (i.e., that the real problem in our world is not an ingenuity gap but a lack of political will) is bogus.

I have a good deal of material on all these issues already available, but much of it is in note form. As I convert these notes into fully elaborated arguments, I am delivering them to this page. Readers interested in an early attempt to deal with some of the above theoretical issues should read my original "Ingenuity Gap" article, which appeared in the journal Population and Development Review in 1995; this article can be found at www.homerdixon.com/projects/ingen/ingen.htm.

INGENUITY THEORY

The Value Added of the Ingenuity Gap Concept

The first question I must address is: What is the "value added" of this approach? Some might argue that when I say we're facing an ingenuity gap, I'm saying nothing more than we have a problem that can't be solved.

But in our common usage, problems and their solutions are, as philosophers would say, "internally related"--that is, they are defined in terms of their relationships to each other. The difficulty of a given problem can't be assessed independently of our judgement about the solutions available to address it, and the quality of a given solution can't be assessed without reference to the problem it is intended to address. This means that problems and their solutions tend to be conflated with each other; it's difficult to examine one without sliding into an examination of the other.

The ingenuity gap approach, however, allows us, at least in principal, to separate problems from solutions and examine them independently. A given problem can be understood as a particular requirement for ingenuity, and if we can work out a way of measuring ingenuity (something I discuss below), then we can assess the nature or difficulty of the problem without reference to its putative solutions. Similarly, a given solution becomes a particular amount and type of ingenuity supplied, something that it is again possible to assess, at least in principal, without reference to the problem the ingenuity is intended to solve. Problems and solutions are reduced to a common, but independent and objective yardstick of comparison ingenuity.

Moreover, once problems and solutions are separated from each other, we can more effectively examine the many factors that can make our problems harder, as well as the many factors that can constrain our ability to solve those problems. I have found this separation of the two sides of the gap--ingenuity requirement and supply--to be a particularly helpful intellectual discipline.

For example, there are various kinds of constraints on scientific progress. Some affect the supply of scientific ingenuity and others affect the requirement for this ingenuity. Human cognitive limits, which restrict our ability to grasp the workings of complex systems in their entirety, are a supply-side constraint. On the other hand, the intrinsic difficulty of scientific problems, something that is a function of the fundamental characteristics of our natural world, is independent of our cognitive ability, and is a key factor determining our overall requirement for scientific ingenuity. Sometimes this requirement can exceed our cognitive ability, producing an ingenuity gap. But when we clearly distinguish between requirement and supply effects, we can say more than this: we can say whether the ingenuity gap is getting wider or narrower because features of our external world are changing (that is, because our scientific problems are getting intrinsically harder or easier) or because our cognitive ability is changing. (Normally admittedly, it's the former, but we shouldn't exclude the possibility of the latter--see my discussion in chapter 13 of the effects of toxins on brain development and performance.)

So, distinguishing between ingenuity requirement and supply is a more powerful analytical move than it might seem at first. It allows us to distinguish between changes in our world in a way that we couldn't otherwise. When we say that we face an ingenuity gap, we highlight a key question: Are the forces making this problem hard to solve affecting ingenuity requirement or supply?

Measuring Ingenuity Quantity: The Utility of the "Sets of Instructions" Approach

This brings us to the second key question: How do we measure ingenuity? Implicit in any discussion of requirement and supply is the assumption that it is possible, at least in principal, to measure ingenuity. In The Ingenuity Gap I define ingenuity as "sets of instructions that tell us how to arrange the constituent parts of our physical and social worlds in ways that help us achieve our goals." I go on to suggest that one measure of ingenuity might be the length of this set of instructions. (I'm assuming that a "set of instructions" consists of an ordered list of statements, in which each statement describes an action that one must execute--in the specific sequence prescribed by the list--if one is to achieve one's goal. Therefore, each additional action specified in the list lengthens that set of instructions.) By this account, a longer set of instructions represents more ingenuity.

This raises some tricky issues, however. For one thing, some of the instructions might simply be repetitions of ones that appeared earlier in the set; so you could have a long set of instructions in which many of the instructions don't provide much novelty or extra content. I think this is a manageable difficulty however, because we can stipulate that the amount of ingenuity in a proposed solution to a given problem is represented by the most compact set of instructions that describes that solution. We can make a set of instructions that contains a lot of repetitions of actions more compact, perhaps, by using "repeat" or "do loop" commands, similar to those used in computer programming languages.

Readers familiar with my book will see a resonance between this approach to measuring ingenuity and some suggested methods (reviewed at the end of chapter 4, especially in the endnotes) for measuring a system's complexity. A system's "algorithmic complexity" is represented by the length of a computer algorithm that adequately reproduces that system's behavior. Software engineers are also developing methods for gauging the complexity of computer programs, as a means of standardizing software development and better identifying bugs. These methods suggest that, at least in principal, the general task of developing a measure of ingenuity quantity is tractable.

Measuring Ingenuity Quality

The bigger challenge, though, is to measure the quality of ingenuity. Using the "length of the set of instructions" as a yardstick really only allows us to measure the quantity of ingenuity. There is a trap here, though: we must be careful to ensure that our measure of the quality of the ingenuity represented by a particular set of instructions is independent of our assessment of the effectiveness of the solution that the set of instructions represents. In other words, we can't say that better ingenuity solves our problems better. Otherwise, we end up conflating ingenuity requirement and supply, because as the problems and challenges we face get harder, our existing solutions are less effective, which suggests (if we measure the quality of ingenuity by its effectiveness in providing solutions) that the amount of ingenuity we have available to us declines. This conflation of requirement and supply negates the main advantage of the whole approach, which is to allow the independent analysis of requirement and supply. I identify and discuss this difficulty in chapter 9 of the book (in "Ingenuity and Wealth," pp. 230-1).

We shouldn't underestimate the difficulty of measuring the quality of ingenuity, especially if we need a measure that's independent of the ingenuity's effectiveness as a solution. I've puzzled over this problem for years, and I've concluded that we can learn much from discussions in philosophy of science about standards for "theory choice"--that is, the about the criteria scientists use to decide whether one scientific theory is better than another as an explanation or as a predictor of a given phenomenon in our world. It seems reasonable to assume that the criteria we use to determine whether one scientific theory is "better" than another can also help us determine whether one practical idea for addressing a given problem is "better" than another.

Three criteria of theory choice, widely cited by philosophers of science, are:

parsimony;
congruity with already established scientific theories (a better theory is one that fits better with already established understandings of the world); and,
explanatory scope (a better theory is one that makes better "sense" of a wider range of empirical data than other theories, especially data that appear to contradict those other theories what philosophers of science call "anomalous data").

Parsimony and explanatory scope often pull in opposite directions: increasing the latter often requires more complex theories that sacrifice some of the former. One way of getting around this problem is to combine the two criteria into an "efficiency" or "bang-for-the-buck" measure--by such a measure, the better theory provides greater explanatory return for a given amount of cognitive effort invested in understanding and using the theory.

So, in general, better scientific theories are simpler ones that explain more and that are more congruent with our existing understanding of the world. How do we apply these criteria to measuring the quality of ingenuity? It seems fairly easy to transfer parsimony and congruity to this new role. Thus it would seem that, all other things being equal, a set of instructions would represent higher quality ingenuity than another if it's simpler or shorter and if it's more congruent with existing ways of solving our problems. Note that we have a tension here between our measures of quantity and quality of ingenuity: a longer set of instructions might represent a greater quantity but a lower quality of ingenuity. That seems entirely reasonable: in various spheres of our lives, we often note that there is a trade-off between quantity and quality. Note, too, that by these criteria, "out-of-the-box" or lateral thinking (which, by definition, is incongruous with conventional thinking) is not valuable in and of itself.

The third criterion--explanatory scope--is more difficult to use as a gauge of ingenuity quality, since the point of ingenuity, unlike scientific theories, is not explanation of anything. Rather, the point of ingenuity is to provide us with solutions to our problems. But I've already established that we can't use "effectiveness as a solution" as a criterion.

There is, however, another implication of the"explanatory scope" criterion. If we adopt a realist ontology (that is, if we assume that there exists a "real" world independent of the human mind), and if we also adopt a "correspondence theory of truth" epistemology (that is, if we assume that our scientific theories are more "true" to the extent that they better correspond to or "map" onto that real world), then a scientific theory with greater explanatory scope is generally one that provides a more accurate representation of the world. By analogy, higher quality ingenuity would consist of sets of instructions that correspond better to the "real" or "true" characteristics of the world in which these instructions are to be implemented. There are difficult issues here, of course: first, we might rebel against the tight straightjacket imposed by these realist ontological and epistemological positions; second, what we mean by "correspond" is not entirely obvious (my general sense is that any high-quality set of instructions must have a close relationship to the world in which it's to be implemented, but the nature of that relationship has to be further specified); and, third, whether one set of instructions corresponds better or worse to reality is often a subject of great dispute.

Regarding the first issue, I've always been suspicious of realist ontologies and epistemologies, so I've accepted their relevance and value in this context with some reluctance. But I've come to conclude that we need to adopt a realist perspective if we are to avoid "internal" or "effectiveness" measures of ingenuity quality--"internal" in the sense that quality of a given set of instructions is gauged by its relationship to the problem it's intended to address. If we adopt a realist perspective, we can potentially gauge ingenuity quality by external or exogenous criteria--that is, by criteria independent of the relationship between the ingenuity and the given problem.

As for the second issue, what would a "correspondence" relationship between ingenuity and the real world consist of? How can we tell that one set of instructions "corresponds" better to the real world than another? To address this question, I find it helpful to return to my thought experiment (at the beginning of chapter 8) in which I'm trying to escape from my office.

Let's say one set of instructions for getting the flag up the chimney is identical to another, except for one small difference: set "A" says we should screw the pieces of the pole together inside the fireplace, while set "B" says we should screw them together outside the fireplace. (In other words, the two sets of instructions are identical except for the order of the instructions.) Of course, set A allows us to incrementally extend the pole up the chimney, whereas set B will present problems because the pole will be too long, when it's all connected together, to put inside the fireplace and up the chimney. That set A is better than B is a function of basic laws of physics and mechanics (relating to such things as the ductility of the metal in the pole, the solidity of the materials in the fireplace and chimney, and the mechanics of trying to get a pole of a certain length through a space of certain dimensions). Notice that we're saying that set A is better in this respect than B without any reference to whether A actually solves better than B our overall problem of being trapped in the office. Rather, set A simply "fits" the real world better than set B. This captures what I mean by a "correspondence" relationship between ingenuity and the real world.

Regarding the third issue, I'm not sure that it's a real problem. Yes, we are surrounded by disputes over whether one solution or another corresponds better to reality. When two people vehemently disagree about the value (the quality) of their respective solutions to a problem, the source of that disagreement is often their differing judgement about the nature of the world (the "real characteristics" of that world, if you will) in which their solutions are to be implemented. But the existence of such dispute does not weaken the argument I'm presenting here, because these are often entirely legitimate disputes over entirely reasonable questions.

In sum, as a first approximation, I believe we can think about ingenuity quality as a joint function of the parsimony, congruity, and "fit" (or "correspondence" with the real world) of the set of instructions in question. Note something important: by this strategy for thinking about ingenuity, we might sometimes supply large quantities of high-quality ingenuity to address the problems around us without solving those problems. In fact, sometimes the most effective solution might involve less and lower-quality ingenuity than another.