Philip Tetlock Calibration Audit

Evaluate forecast accuracy using Philip Tetlock's calibration audit method. Track assigned probability percentages against real outcomes to eliminate overconfidence.

5 time blocks, 25 minutes in total.

Time blocks

Gather Resolved Predictions: 5 min
Pull forecast logs with clear deadlines and explicit probability score
Code Binary Outcomes: 4 min
Assign a 1 if the event occurred or 0 if it did not occur
Compute Brier Score and Bins: 6 min
Calculate total mean squared error and group forecasts into buckets
Assess Calibration Curve: 5 min
Compare assigned probability percentages against actual hit rates
Record Adjustments and Base Rates: 5 min
Note systematic biases and adjust reference class probabilities

About this routine

Philip Tetlock's calibration audit comes from his decades of research into political forecasting and decision science at the Good Judgment Project. Tetlock demonstrated that superforecasters are not naturally psychic, but rather disciplined estimators who track their confidence levels meticulously over time.

This 25 minute review process takes numerical probabilities assigned to past predictions and compares them against actual outcomes. By calculating your Brier score and plotting forecast bins, you discover whether an 80 percent confidence claim actually comes true 80 percent of the time.

Regularly conducting a calibration audit trains your mind to weigh base rates accurately and strip away emotional bias. You can use a visual timer to stay disciplined during each phase of calculation and analysis.

Why this routine works

Identifies systematic overconfidence or underconfidence across probability ranges
Improves decision quality by forcing clear numerical probability estimates
Provides an objective Brier score to measure forecasting accuracy over time
Refines reference class base rates for future strategic decisions

FAQ

What is Philip Tetlock's calibration audit?

It is a systematic review method developed by political scientist Philip Tetlock to measure forecast accuracy. You compare numerical probabilities assigned to past predictions against actual binary outcomes.

How long does a calibration audit take?

This structured review takes 25 minutes to complete, covering data gathering, Brier score calculation, and bias identification.

What is a Brier score in forecasting?

A Brier score is a mathematical measure of forecast accuracy. It calculates the mean squared difference between predicted probabilities and actual binary outcomes.

Sources

Superforecasting: The Art and Science of Prediction, Philip E. Tetlock and Dan Gardner
Expert Political Judgment, Philip E. Tetlock