Skip to content

Innocent P-hacking

· 3 min read

As a researcher, reading and interpreting academic papers has become unavoidable in my daily routine. Lately, perhaps due to a Baader-Meinhof phenomenon, I have come across several papers exhibiting what I’d call non-malicious p-hacking, often stemming from an insufficient number of simulations.

About p-values

If you are not familiar with uncertainty estimation or p-values, allow me to provide a brief, albeit not entirely accurate, intuition: a p-value represents the probability that a measured (or observed) value is not merely a result of error or chance. Consider the following scenario: you flip a recently forged coin 20 times and get 14 heads (a 70% rate). How do you determine whether this coin is fair or not?

Let’s use a proof by contradiction. Imagine we have a theoretical-ideal fair coin and observe the values using it as a “reference” or null model:

[1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1]
[1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1]
(repeating this process a million times)
[1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1]

In this simulated world where a fair coin is flipped, how many times do we observe a rate equal to or higher than 70%? 57839 times, 5.87%, or 0.0587 (the exact value may vary a bit if you run the simulation yourself). By tradition, we use the values of 0.05 (or 0.01 sometimes) as a cutoff to determine whether something is “significant” or not.

In summary, in a simulated world where a fair coin is flipped, we still see 5.78% with 14 or more heads. Therefore, we cannot claim that what we observed is exceptionally unusual, and we cannot prove the coin is unfair (or, using a more statistical phrasing, we do not have enough statistical evidence to reject the possibility that the coin is fair).

Note that in this peculiar scenario, we know the experiment can be modeled using a binomial distribution. Therefore, we could be tempted to think there is a method to calculate the p-value analytically or in a closed form. However, while the variance of our estimate can be reasonably calculated, the p-value and the confidence intervals are particularly challenging to obtain.

The P-hacking issue

P-hacking is a pejorative term to highlight experiments that manipulate methods to achieve p-values below 0.05, passing them off as “statistically significant.” In our example, 14 out of 20 heads are not highly uncommon for a fair coin; it’s just a fortunate situation. However, the probability of seeing 14 or more heads is dangerously close to 0.05.

The interesting aspect here is that the “p-value” measured through simulations depends on the number of simulations conducted. We may get inconsistent results with fewer simulations, leading to incorrect reporting.

Thus, let’s make a meta-simulation of the simulations with different simulation sizes (assuming that creating a large number of simulations is computationally challenging):

  • With 10 simulations per scenario, 4506 out of 10000 (45.06%) agree that the coin could be fair. 54.94% will suggest that the coin was rigged!
  • With 100 simulations per scenario, 6914 out of 10000 (69.14%) agree with our simulations. 30.86% will suggest that the coin was unfair! Almost one-third of the cases!
  • With 1000 simulations per scenario, 8731 out of 10000 (87.31%) agree that the coin is fair.
  • With 10000 simulations per scenario, 9998 out of 10000 (99.98%) agree that the coin is fair.

This illustrates a common issue with simulations. Even in simple experiments like coin flipping, estimating p-values through simulations should be cautiously approached, primarily when used as a fixed threshold, because it could lead to diverse conclusions.

For real-world examples of this mistake in academic papers, I recommend Googling these:

EEG p-value "100 surrogate" "significance" site:https://www.ncbi.nlm.nih.gov

EEG p-value "500 surrogate" "significance" site:https://www.ncbi.nlm.nih.gov