Tuning with data

Why this matters

"It feels too hard" is useful feedback, but it does not tell you which level, how much too hard, or whether your change helped. Studios combine playtest observation with simple numbers: how many players finished each level, how many attempts it took, where they quit. You do not need a data team to do this. A list of session records, a few small functions and a clear target turn vague opinions into decisions you can check.

What to record

Decide the question first, then log the minimum that answers it. For difficulty, a per-attempt record is enough:

const sessions = [
  { player: "p1", level: 1, completed: true,  deaths: 1, seconds: 42 },
  { player: "p1", level: 2, completed: false, deaths: 6, seconds: 180 },
  { player: "p2", level: 1, completed: true,  deaths: 0, seconds: 35 },
  { player: "p2", level: 2, completed: true,  deaths: 3, seconds: 150 },
];

In a jam game, these records can come from a notes sheet you fill in while watching, a console.log you copy out, or a simple download button. Respect privacy: record play, not personal data, and tell testers what you collect.

From records to rates

The completion rateCompleted attempts at a level divided by all attempts at it. of a level is completed attempts divided by all attempts:

function completionRates(sessions) {
  const totals = {};
  for (const s of sessions) {
    totals[s.level] ??= { done: 0, all: 0 };
    totals[s.level].all += 1;
    if (s.completed) totals[s.level].done += 1;
  }
  const rates = {};
  for (const level in totals) rates[level] = totals[level].done / totals[level].all;
  return rates;
}
// { 1: 1, 2: 0.5 }

Look at other numbers too: average deaths, medianThe middle value of a sorted list; less distorted by one stuck player than the average. time (one stuck player can inflate an average), and drop-offPlayers who never attempt the next level, a sign of where people quit. (players who never attempted the next level). A level with a fine completion rate but ten deaths per clear may still be frustrating.

Set a target, then adjust

Numbers need a goal. A designer might decide: "early levels should be completed by about 80% of attempts; the final boss by about 50%." With a target, you can turn a difficulty parameter (enemy speed, spawn interval, timer length) up or down by a small step:

function adjustDifficulty(value, rate, target, step, min, max) {
  if (rate < target) value -= step;   // too hard: make it easier
  else if (rate > target) value += step;   // too easy: make it harder
  return Math.max(min, Math.min(max, value));
}

Small steps and clamped ranges matter. Large jumps overshoot ("now it is trivial"), and an unclamped value can drift into nonsense like negative enemy speed. Change one parameter at a time between playtests, or you will not know which change caused the result.

Seeded randomness for fair comparisons

Random spawns make two playtests hard to compare: maybe level 2 just rolled more enemies this time. A seededDescribes a random generator that produces the same sequence every time from the same starting seed. random number generator produces the same sequence every time from the same seed, so you can replay identical conditions and simulate many runs:

function makeRng(seed) {           // mulberry32: small, fast, good enough for games
  let a = seed >>> 0;
  return function () {
    a = (a + 0x6D2B79F5) >>> 0;
    let t = a;
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

const rng = makeRng(42);
rng(); // always the same first value for seed 42

Math.random() cannot be seeded, so games that need replays, daily challenges or procedural levels ship their own generator. Many roguelikes let players share a seed to play the same map.

Simulating before playtesting

With a seeded RNG you can run a crude simulationMany cheap computed runs that model your assumptions to choose what to playtest, not a replacement for real players.: if a level kills a player with probability p per encounter and has n encounters, how many of 1,000 simulated runs survive? Simulations are cheap and catch obvious problems, but they model your assumptions, not real people. Use them to choose what to test, then confirm with humans.

Data is not the whole story

The Extra Credits video makes the point that balance is about meaningful choices, not equal numbers. Data tells you where players struggle; watching them tells you why. A 40% completion rate might mean the level is too hard, or that the exit is hard to find, or that a control is confusing. Pair every chart with observations.

Common mistakes

• Averages hiding outliers: use medians or look at the distribution.
• Tiny samples: five testers is enough to spot big problems, not to tune by 2%.
• Changing several things at once, so results cannot be attributed.
• No target, so every number looks "about right".
• Optimising the metric, not the experience (a level everyone completes because it is boring).

Quick recap

• Log just enough per attempt: level, completed, deaths, time.
• Completion rate = completed / attempts; also check deaths, medians and drop-off.
• Set targets; adjust one parameter by small, clamped steps.
• Seeded RNGs make runs repeatable and simulations possible.
• Combine numbers with observation.

Try it

Open the Do tab to write completionRates, hardestLevel, adjustDifficulty and a seeded simulateRuns.