The study discussed in the book, Psychology of Intelligence Analysis (Heuer, 1999), details the results of an experiment whereby 23 NATO military officers accustomed to reading intelligence reports were tasked with assigning probability of occurrence given "verbal expressions of uncertainty" typically found in an intelligence report.
Having recently read Michael Lewis' latest book, The Undoing Project,
Heuer's work struck a familiar chord. Lewis retells the careers of noted Israeli psychologists, Amos Tversky and Daniel Kahneman, and their groundbreaking work on decision analysis and bias. Turns out that Heuer cites Tversky and Kahneman in his work.
The results of this study are summarized in Figure 18 from Heuer Chapter 12.
Each analyst was given a number of sentences typically found in intelligence reports and asked to assign a probability to each. Sentences were the same with the exception of the verbal expression of uncertainty. For example, a sentence might begin, "There is a very good chance that..."
The descriptors are based Kent's 1964 work (see Kent) who suggested standardizing phrases and their associated probability ranges.
A similar experiment was repeated in 2015 based on input from 46 Reddit users (see Zonination).
Since a number of the descriptors resulted in either overlapping ranges or the same median in the Zonination experiment, I reduced the number of classifications to a more manageable level. My results below:
What do these results suggest? To me, the rows suggest an odds ratio using verbal expressions of uncertainty rather than precise odds. Also, the distribution of responses are variable depending at which row of the table one finds oneself. Some responses have narrow ranges with few outliers; others have wider ranges will more outliers. Think of this table as a CRT with Combat Odds (Statements) along the Y-axis and Combat Results spread out along the X-axis.
What does this have to do with wargame design? Maybe something; maybe nothing. I could envision situational modifiers moving a result either up or down the rows either increasing or decreasing the probability of success.
In most games, the exact probability (or range of probabilities) of a particular outcome is known or can be computed with relative certainty. Did our historical counterparts have that precision in results? Not likely. They assessed the outcome based on limited evidence at hand. In their own decision-making process, commanders assessed and concluded that an attack had a "very good chance" of succeeding.
A few questions for thought until the next time I pick up this topic:
- What if we approach game design from this mindset?
- If this experiment was repeated with the survey given to wargamers, would the probability assignments look similar?
- With our experience in games and game theory, do we assign probabilities differently from intelligence analysts?
- Is there interest in conducting a similar experiment?