The Percentile Trap: A Good Leaderboard Lives or Dies on Sample and Cohort

The most dangerous sentence in data scouting is "the numbers don’t lie." The numbers themselves may not, but a leaderboard built from them depends entirely on two design choices. First, which players go into the sample (sample size). Second, who each player is measured against (the cohort). Skip either decision and the table stops being information and becomes noise.

Every example below is calculated directly from 2024 J1 data (API-Football), and shows why this site’s gems, map and percentiles use the rules they do.

Trap #1: sample size — the lure of 100%

Sort duel win rate with no conditions and the top fills with 100%. The outright leader in 2024 J1 is Cerezo Osaka goalkeeper Kim Jin-Hyeon, who won all 28 of his 28 duels (100%). "The goalkeeper is the best in a physical battle" is obvious nonsense. Below him come players who won all one or two of their duels. Dribbling behaves the same way: the unfiltered success-rate table is topped by tiny samples — one attempt, one success (100%), or nine of nine.

The fix is a minimum-attempts floor. Require 200+ duels on the season and the field narrows to 74 players, whose average settles at a stable 49.8%. For dribbles, the baseline is 40+ attempts (43 players, averaging 45.0% success). That is why every leaderboard on this site states its sample threshold — "30+ shots," "200+ duels." A rate ranking with no floor is better left unread.

Trap #2: the cohort — full-backs and strikers play a different game

Filter the sample properly and a second trap remains: putting different positions in one column. Even inside that same 200+ cohort, duel win rate splits sharply by position — defenders average 56.1%, midfielders 50.6%, forwards 42.2%. Defenders contest winnable duels (stopping progress in their own half); forwards face hard ones (in front of a packed opposition goal).

So rank across positions and the table measures position, not ability (see our duel win-rate column for the full case). A 55% defender is ordinary; a 55% forward is elite for his role. The same number carries opposite meanings.

How we apply it

Against these two traps the site runs three layers of rules. First, a minimum-attempts floor per metric strips out tiny samples. Second, percentiles and the player map are always computed inside position groups — full-backs against full-backs, forwards against forwards. Third, a minutes floor (about 900, roughly ten matches) removes anyone below it from gems, the map and percentiles entirely; 240 players cleared that bar in the 2024 data.

These rules make the rankings less flashy — the 100% outliers disappear. But that is the point. Scouting does not need the eye-catching number; it needs the number that repeats next season. Fix the sample and the cohort first, and only then does what remains become a signal.