Penalty Analysis for JAR Data: How to Turn Just-About-Right Ratings Into a Fix List

A JAR question asks whether an attribute has too little, about the right amount or too much of something, for example sweetness or crispness. It is asked next to an overall liking question, so each respondent gives a direction on each attribute and a liking score for the whole product.

Q Research Software's penalty analysis walkthrough treats the JAR variable as three ordered categories: "Not enough", "Just about right" and "Too much". If your scale has more points, it says to merge them into three (Q Research Software).

Keep one attribute per question. A question that mixes two attributes cannot be turned into a fix.

MetricGate's documentation gives the formula as penalty = drop × (% affected / 100), where the drop is the average liking loss for affected consumers and % affected is the share not at JAR (MetricGate). Its calculator page puts it as "the mean drop weighted by the proportion of respondents in that group" and tests each group against the JAR group with a two-sample t-test (MetricGate calculator).

Steps:

1. Collapse each JAR scale into three groups.
2. Compute mean overall liking for each group.
3. Drop = mean liking of the JAR group minus mean liking of the not-right group.
4. Penalty = drop × share of all respondents in that group.
5. Sum the "too little" and "too much" penalties for an attribute total, as Q's walkthrough does. Q floors negative drops at zero.

All numbers below are illustrative. A snack is tested by 200 people, who rate overall liking on a 9-point scale.

Sweetness totals 0.216 + 0.360 = 0.576. Spice totals 0.018 + 0.350 = 0.368. The biggest single penalties are sweetness too much (0.360) and spice too much (0.350).

Now pool the not-right groups, as MetricGate's mean-drop page does. For sweetness, the 76 not-right people average (36 × 6.2 + 40 × 5.6) ÷ 76 = 5.88. The drop is 7.4 − 5.88 = 1.52 and the penalty is 1.52 × 0.38 = 0.576, the same total. But the pooled figure hides that sweetness splits in two directions. A fix that adds sweetness would help the 36 and hurt the 40. MetricGate itself warns that pooling "can mask opposite-direction problems".

MetricGate's calculator says groups exceeding "roughly 20%" of respondents with a significant penalty flag attributes to fix. Treat that as a heuristic, not a rule. Its mean-drop page adds that rankings shift with the chosen percent-affected and drop thresholds, so you should report the thresholds you used. Decide them before you look at the chart.

In the example, spice too much (25%, drop 1.4) clears a 20% line. Sweetness too much sits exactly on it, so judge it by its significance test and your own risk appetite. Spice too little (6%, 12 people) is too small to read. MetricGate's calculator notes that small groups "produce unstable, non-actionable penalties even if the mean drop looks large".

Plot percent not right on one axis and the penalty (or drop) on the other, as Q does. Attributes at the top right are the ones to worry about.

• Reading penalties as cause. MetricGate says penalties are descriptive and "do not by themselves establish a causal effect on liking". People who dislike a product often also call it too sweet.
• Ignoring small groups. Always show the base next to the penalty.
• Attributes that overlap. If two JAR items are about related qualities, their penalties partly count the same dissatisfaction. Check how the two items correlate.
• Weighted data. Q's group sizes are weighted, so shares depend on your weighting. See our survey weighting guide.
• Treating a fix list as a recipe. A penalty says which attribute to look at, not how much to change it. Retest the revised product.

Poseidon, our analytics engine, lists penalty/JAR among the formulas it selects by question type. Whatever tool you use, ask to see the group sizes, the drops and the thresholds, not only a ranked list.

The Hercules panel is app-based, so it reaches smartphone users only, and a product test that needs physical use needs a way to get the product to people. The Free plan includes 30 responses, which is enough to test the JAR wording on a phone.