How reliable is a short self-assessment?

A measure is reliable when it gives consistent answers. For a self-assessment that means two things. The questions about one pattern should agree with each other, because they are meant to point at the same habit. And if nothing about you had changed, taking the test again should place you close to where you were before.

Reliability is not the same as being right. A test can be very consistent and still measure something narrower than its name suggests. Consistency is the floor a test has to clear, not proof that it means what it says.

Persona's checks focus on the first kind: how well the questions on each pattern agree.

For every pattern, Studio, the part of Persona where tests are built, works out Cronbach's alpha (Cronbach, 1951). Alpha compares how much the answers to each question vary with how much the pattern's totals vary. When the questions move together, alpha is high; when they pull in different directions, it falls.

Studio only works out alpha when it has enough to go on: at least 10 people who answered every question of the pattern, and at least 2 questions. Studio previews never count, so test runs by the authors cannot change the numbers.

• Consistent: alpha of 0.80 or above.
• Acceptable: from 0.70.
• Weak: from 0.50.
• Inconsistent: below 0.50.
• Items disagree: below 0, which usually means a question is scored the wrong way round.

Studio also shows, for each question, how closely its answers track the rest of its pattern. A question whose answers barely track the others, with a correlation below 0.30, is flagged for the authors to rewrite or remove.

An alpha worked out from a small group can be far from the value a larger group would give. So Studio puts a 95% interval around every alpha, using Feldt's method (Feldt, 1965). With few people the interval is wide; as more people take the test, it narrows.

When the interval reaches below 0, Studio marks the pattern as uncertain: there are too few people to judge it either way. That is a reason to wait for more results, not a verdict on the questions.

Alpha depends on how many questions a pattern has as well as on how well they agree, so a pattern with only a couple of questions starts at a disadvantage (Cortina, 1993). Studies of very short personality measures find the same trade-off in practice: they save time and lose some precision (Credé and others, 2012).

Alpha is also not the whole story. It rests on assumptions about the questions that are rarely exactly true, and it says nothing about whether a test measures what its name suggests (Sijtsma, 2009). Persona treats it as one check among several, beside the publish checklist and the authors' own review.

This is why every Persona result has a confidence level, which looks at how many of the asked questions you answered; a short test can stay at Moderate even when you answered all of them. It is also why a test's public page is kept out of search results while the test has fewer than 6 scored questions.

Every Persona test asks you to describe yourself. That makes it quick and personal, and it also means the result reflects how you see your own habits. People tend to lean, a little, toward the answer they would like to be true; researchers call this socially desirable responding (Paulhus, 1984).

• Answer for how you usually act, not how you act on your best day or how you would like to act.
• Think of real, recent situations when a statement feels abstract.
• Use the middle of the scale when it is honestly true, not as a way to avoid choosing.

How long each answer took is recorded too. It helps authors find questions people find confusing, and it is used only for group statistics about the test, never for your result.

Reliability figures are working tools for the people who build tests. Studio shows them to Persona managers; public test pages do not, because a figure from a small group can mislead more than it helps. They also do not decide whether a version can be published: publishing is held back by structural problems, such as gaps between bands or missing consent text, so a new test can collect the results its checks need.

For you, as someone taking a test, the practical reading is simple. Treat a result from a short test as a first sketch, look at its confidence level, read both bands when a position is near a band edge, and compare a retake with your earlier result rather than relying on one sitting. For the rest of a result, see How to read your Persona result.