The Smug Baldy Speaks

In this article, I’ll outline a proposal to utilize a well known conceptual framework to increase our ability to rigorously and dispassionately examine the accuracy of psychic predictions of future events. In part one, I propose two sets of guidelines related to data recoding and validation, as well as data analysis. In later parts, I will apply these methods to the analysis of specific data sets garnered from predictions made by various psychics over the past few years. I invite your comments and suggestions.

I’ve been on a bit of a jag recently, looking at what I consider to be silly, spooky, and outright loony predictions made by self-proclaimed psychics. Also, I have recently tried my hand at scoring the accuracy of some of these, with somewhat mixed results. Obviously, there is the potential to allow bias to influence one’s tests as well as one’s data selection, and since I freely admit that I’m skeptical of the claims of psychic powers, I want to minimize the likelihood of bias and errors. To that end I’ve decided to put together a simple procedure for scoring psychic predictions.

This proposal offers guidelines in two major areas:

1. What data should be considered valid when one is assessing the accuracy or validity of predictions made by persons purporting to have psychic ability?
2. What statistics are important with respect to the data sets in question, and how to calculate them?

There are several benefits of developing such guidelines. First and foremost, these will allow us to treat diverse sets of predictions in a consistent manner. Second, by applying consistent analysis techniques, this effort will reduce the likelihood of biased interpretations. Third, this effort is built upon a well established theoretical framework - Signal Detection Theory.

Guideline 1: What data should be considered valid when assessing psychic predictions?

I believe that data from psychic predictions should only be subject to statistical analysis if the following conditions are met:

1. There must be an actual prediction that is “specific enough” to test. No statements will be accepted that do not specify enough context surrounding the predicted event in order to determine if the event actually did or did not occur as predicted. A psychic prediction that is “specific enough to test” can be operationally defined as one in which at least three of the “Wh” questions (Who, What, Where, Why, When) are specified, one of which must be some time frame (When) in the future.

   Examples of non-predictions:

   1. Celebrity X will get married. (Only specifies who and what, and does not specify when).
   2. Next Fall, it will be very warm. (Specifies when, but not where - and actually doesn’t specify what. “Very warm” isn’t specific.)
2. We will distinguish positive from negative predictions: Valid prediction can be either positive or negative. A positive prediction is defined as a prediction that a certain event will occur. A negative prediction is defined as a prediction that a certain event will not occur. In either case, however, the prediction is specific enough to test as outlined above.
3. There must be no advanced knowledge of the predicted event: This means that we will discard predictions of events that occur prior to the publication of the prediction, as well as predictions of events that are scheduled to occur and the schedule is available prior to the prediction. Predictions about events that are likely to occur will be considered as long as there is not any a verifiable published report that shows that the event is scheduled or planned.
4. No Compound Predictions: A compound prediction is simply more than one prediction in a single statement. If a psychic says X will occur and that Y will also occur, this will be treated as two distinct predictions, assuming the other prediction guidelines have been met.

Guideline 2: Important statistics, and how to calculate them

The conceptual framework that forms the basis of this guideline is known as Signal Detection Theory, which is described in Wikipedia as follows:

Signal detection theory (SDT) is used when psychologists want to measure the way we make decisions under conditions of uncertainty, such as how we would perceive distances in foggy conditions. SDT assumes that the decision maker is not a passive receiver of information, but an active decision-maker who makes difficult perceptual judgments under conditions of uncertainty.

SDT can be directly applied to situations in which skilled observers need to be able to discriminate target signals from background noise. For example, SDT has been applied to the accuracy of radar operators, medical diagnosis, psychology and neuropsychology. In addition to these, I believe psychic predictions can also be analyzed using SDT, since psychics are not supposed to be passive observers, but are rather active decision makers that claim to use their abilities to make difficult judgments concerning future events under conditions of uncertainty.

Applying SDT to psychic predictions is rather simple:

1. Each valid prediction is assigned to one of four categories based on the following table:
   

   	Event Occurred	Event Did Not Occur
   Event Predicted To Occur	Hit	False Alarm
   Event Predicted To Not Occur	Miss	Correct Rejection

   A Hit occurs when a positively predicted event actually occurs. For example, if celebrity X is predicted to get married and they do.

   A False Alarm occurs when a predicted event does not occur. For example, if celebrity X is predicted to get married and they don’t actually get married. (This is similar to “crying wolf”).

   A Miss occurs when an event that is predicted not to occur actually does occur. For example, if celebrity X is predicted to not get married and they do.

   A Correct Rejection occurs when an event that is predicted not to occur actually does not occur. For example, if celebrity X is predicted to not get married and they don’t actually get married.

   Within this framework, the psychic can be correct by getting a Hit or by getting a Correct Rejection. Misses and False Alarms are considered errors.

Statistics of Interest:

2. Hit Rate (P_Hit): This is the ratio of correctly predicted events with respect to the total number of events that actually occurred. Using the table above, it is calculated as:

   Hits / (Hits + Misses)

3. Proportion False Alarms (P_FA): This is the ratio of incorrectly predicted events to the total number of events that were predicted to occur, and is calculated as:

   False Alarms / ( Hits + False Alarms )

4. Proportion Correct (P_C): This is the total proportion of hits and correct rejections to the total number of events, and is calculated as:

   (Hits + Correct Rejections) / (Hits + Misses + False Alarms + Correct Rejections)

There are some potential anomalies with psychic predictions. One anomaly could be caused by the extent to which psychics tend to self-select the events that they will predict to occur or not occur. This has a bearing on the calculation of both hits and misses, since we can only count the events that were the subject of the psychic prediction. For example, if in 2004, a psychic didn’t mention in their weather predictions for 2005 that the hurricane season would result in the most catastrophic storm in American history (Hurricane Katrina), then we cannot count this as a miss (even though any real psychic shouldn’t have missed this one). The reason for this is the psychic picked the events to predict, which limits the scope of events we can track. During analysis, this could artificially inflate the hit rate to nearly 100% since there could be virtually no misses.

In Part II of this article, I’ll be examining Sylvia Browne’s 2007 prediction data within the context of these guidelines and present the results.