Game Leaderboards: What a Replay Study Can Show

Separate a comparison from a reason to return

Game leaderboards place a score beside other results. Whether that comparison makes somebody return after leaving the game is a further question. Looking at a rank, wanting to improve it and actually starting another session are different events. A useful evaluation decides which event matters before calling the feature engaging.

Imagine a fictional tile puzzle that shows your best result after each attempt. Adding five other names changes the information available, but your original score remains. If you want to understand the list’s contribution, record what the comparison adds and which behavior would count as a response to it.

A broken leaderboard created three historical windows

Pedersen and colleagues’ 2017 manuscript compared 4,553 new Quantum Moves players in three separate six-hour windows in April 2016. One group saw a loading error and its own score, another a Top 5 list, and another nearby ranks. These were successive live configurations, not randomly assigned conditions.

The broken list matters to the interpretation. Removing a ranking while retaining personal feedback differs from removing every indication of progress. An error message also differs from a deliberately designed screen without a list. Preserve those details before describing the comparison as simply competition versus no competition.

The replay definition belongs beside the percentages

They counted a replay when a break of at least 30 minutes began within the window and play later resumed; records were checked through June 4. Reported replay proportions were 8.6%, 7.3% and 5.6% for the broken, Top 5 and relative lists, respectively. Fisher’s exact comparisons found no significant differences after Bonferroni correction.

These are reported return visits under one definition, rather than evidence that players enjoyed one screen more. Numerical order alone cannot pick the better interface. A null comparison also does not prove that the displays produced identical behavior. A useful account retains the uncertainty instead of turning the result into either a universal benefit or a reason to remove every leaderboard.

The calendar is part of the comparison

The windows fell on April 15, 17 and 27. The relative-list group contained 554 new players, compared with 2,124 in the Top 5 group; the authors described declining publicity-driven traffic. They did not measure players’ attitudes toward the lists.

Different arrival dates can bring different visitors and circumstances alongside a changed interface. An account being new does not establish its owner’s motive, previous gaming experience or interest in the subject. A numerical pattern therefore needs its sampling story attached. Without that story, a change associated with the list could be credited entirely to the list.

Keep the separate physics claim in its historical context

The manuscript cites a separate 2016 physics paper that Nature retracted in 2020 after a code error invalidated its quantitative results. This discussion addresses reported return visits, without endorsing those earlier physics claims.

An activity log and a scientific optimization result require different checks. The existence of a return record cannot validate a claim about an algorithm, and the retraction notice supplies no independent verification of the return records. Read the older manuscript with those boundaries visible rather than treating its introductory success narrative as evidence for the interface comparison.

Give a future leaderboard test a concrete target

Return to the fictional tile puzzle. A proposed test could keep scoring, puzzle availability and personal feedback fixed while comparing a top list with nearby results. Assigning versions during the same period would address the calendar problem more directly. Specify the return interval before collecting data, and keep returning, score improvement and stated preference as separate outcomes.

This is an original study proposal, not a tested recipe. The manuscript and its arXiv record concern one historical comparison. For a design decision, ask what the list communicates, who encountered it and which observed action supports the conclusion. Those details give leaderboard design a question that a result can actually answer.

Keep exploring

FOLLOW THE SOURCE

Sources & further reading

Have a correction? Tell our editorial team.

✦ NICE. YOU LEARNED SOMETHING.