How a Provyn score is made

A Provyn credential says that a named person completed one timed assessment, under the integrity controls listed on the credential, and scored at or above that assessment's pass mark. It records the score as a percent of the points available in the questions that person was given, the date, the question bank version, and, where there is enough data, a percentile and sub-scores by competency.

It does not claim:

• that the score predicts job performance. We have not shown that it does;
• that the assessment is accredited, certified or endorsed by any standards body or government;
• that the pass mark was set by a formal standard-setting study (see the known limits below);
• anything about skills the assessment did not test.

The Standards for Educational and Psychological Testing (AERA, APA and NCME) informed what this page reports: how items are developed and reviewed, how scores are produced, their precision, how the comparison group is defined, fairness, and security. That is a list of topics we chose to cover. It is not a claim that Provyn conforms to the Standards or has been evaluated against them.

Each assessment has a question bank written by Provyn. Question types are multiple choice, short answer, ordering, matching, fill in the code, code, open scenario, system design (a diagram plus notes), and step by step incident simulations. Every question carries a point value, a correct answer or accepted answers where one exists, and a written grading rubric where it needs judgement.

Every question is tagged to one competency (for example, query tuning in SQL). Each assessment has two to five competencies, and each competency needs at least 4 questions in the bank. Automated checks run on every change to a bank and block it from shipping if, among other things, question ids clash, a multiple-choice key points at an option that does not exist, an ordering key is not a valid order, matching answers are ambiguous, or a competency falls below its minimum. New questions are written as proposals, which are not served to candidates until they are reviewed and approved with the answers visible.

Each attempt records the version of the bank it was taken on, so a later edit to a bank does not change what an earlier credential means.

When you start, the server draws your questions and stores the list on your attempt, so the runner, the scorer and any reviewer all see the same set. Retired questions are never drawn.

• Size. 25 scored questions. If a bank has 25 or fewer, you get every scored question in a random order. System Design (Canvas) draws 4 design problems instead, from a bank of 8. Assessments that offer an optional junior, mid-level or senior level draw a different number for that level, shown on the level picker before you start.
• Competency balance. The 25 are split across competencies in proportion to their share of the bank, with at least one from each, so one attempt cannot be mostly about one area.
• Exposure control. Within each competency, questions are drawn at random but weighted away from those that appeared most often in the last 200 attempts at that assessment. With no history, the draw is uniform.
• Order. The chosen questions are shuffled, so position carries no information.

At the moment these banks are no larger than one draw, so every candidate on them sees the same questions in a different order: Data Structures and Algorithms, TypeScript (Advanced), Incident Response, Sales Development (SDR).

A new question can be run as an unscored trial before it counts. When a bank has trial questions, each attempt carries one of them in addition to the 25 scored questions, never in place of one. The least-seen trial question is chosen, so each gathers data evenly. It looks like any other question, the runner tells you an attempt may include one, and it has no effect on your score, pass or fail, percentile or competency scores.

A trial question starts counting only after it has been answered by at least 20 people and its statistics (below) show it separates stronger from weaker candidates, is neither too easy nor too hard, and shows no sign of a leak. A person makes that change; it is not automatic.

Automatic scoring. Multiple choice questions are right or wrong. Short answers are compared with the accepted answers, ignoring case and surrounding spaces. Matching and fill in the code questions give partial credit for each correct pair or blank, unless the question says otherwise. Incident simulations add up the points of the option you chose at each step. If every question in your attempt is scored this way, your result is final when you submit (unless it arrives well past the time limit; see integrity below).

Ordering questions with 5 or more items give partial credit for how close your order is to the correct one. Every pair of items counts: a pair in the same relative order as the answer key counts for you, a pair the other way round counts against you. Your credit is the share of pairs in order minus the share out of order (Kendall's tau), never below zero, and you earn that fraction of the points, rounded down. The exact order earns full points. An order with at least as many pairs out of order as in order (the exact reverse, for one) earns 0, and so does an answer that leaves out or repeats an item. For example, swapping two neighbours in a list of five puts 9 of 10 pairs in order, which earns 8 of 10 points. A list of 4 or fewer items is right or wrong only (with so few pairs, partial credit would reward guessing), a question can be marked right or wrong only, and attempts started before October 11, 2026 keep the right or wrong rule they were taken under, including when they are re-scored.

Each attempt starts every ordering list in its own scrambled order, which is never the answer and earns nothing as it stands. In lists of 5 or more items, at least as many of its pairs are out of order as in order, as close to half as the list allows, and reversing it or moving an item from one end to the other scores no better on average than a random order. The question counts as unanswered until you move an item, or choose "This order is my answer" if the order shown is already the one you want to give.

Answers that need judgement (code, open scenarios, system designs, and short answers that match no accepted answer) put the attempt on hold for review. Blank answers, and code left as the starter code, score zero without further grading. A background job picks up held attempts every few minutes:

• SQL is run. Your query and the author's reference query are both executed against a test copy of the question's database (Postgres, isolated per run, with a time limit). If the results match, the query is run again on that database plus extra rows generated for your attempt alone; a query that hard-codes the expected answer fails that second check. Matching results on both earn full marks. Anything else goes to the rubric with the execution result attached, because the rubric can give partial credit.
• Two independent AI passes. Other answers are graded against the question's rubric by a large language model (hosted by Together AI; see sub-processors), twice, in parallel, with different prompts. The passes agree when both are confident (at least 0.8 on the model's own 0 to 1 scale) and their scores are within 10% of the points at stake. An agreed grade is the average of the two, rounded down. A disagreement is marked for the reviewer with both rationales.
• A person confirms. AI grades are suggestions. A Provyn reviewer sees each one, with its rationale and any execution result, and confirms or changes it before the attempt is finalized. No AI grade finalizes a result on its own. We record how often reviewers agree with the AI; we have set a bar of at least 50 comparisons agreeing 95% of the time before we would even consider changing that, and it has not been met.
• Drift checks. Every week a set of gold-standard answers, each with a known correct score range, is graded by the live grader. If a confident grade falls outside its range, AI grading for that question is paused: its suggestions are still shown, but always flagged for a person, until a later check passes.

We aim to finish reviews within 2 business days; the measured turnaround is published on the Trust Center.

Score. Points earned divided by the points available in the questions you were given (trial questions excluded), as a percent. Questions carry different point values, so a code question counts for more than a single multiple-choice question.

Range. A score from a couple of dozen questions is an estimate. A different draw would land nearby, not on the same number. The result screen shows a 95% range for the automatically scored part (a Wilson interval on the share of questions answered correctly), when there are at least five such questions. It is a sampling range, not a guarantee.

Competency scores. Computed by the same scorer, one competency at a time, over only the questions you were asked. A competency is left off the credential, rather than shown as 0%, when your attempt had fewer than 3 of its questions or fewer than 10 points available.

Percentile. Your score compared with other people's finalized attempts on the same assessment and, where the assessment offers levels, the same level (junior attempts only against junior attempts; attempts on the standard form, including those from before levels existed, only against each other; levels are never combined): each person counted once at their best attempt, your own other attempts excluded, integrity-flagged attempts excluded, and ties placed at the middle of the tied group, so no one is shown as 0th or 100th. It is fixed when the attempt is finalized. With fewer than 10 other people it reads “cohort forming”; from 10 to 99 it shows a band (Top 25%, Top 50%, Below median); only at 100 or more does it show an exact number. Full rules are at how percentiles are calculated, and each assessment's score distribution is on its calibration page (for example, SQL).

Two-stage assessments only. An assessment can be delivered in two parts, where part 2 is chosen to be easier or harder depending on how part 1 went. This is off unless it has been switched on for that assessment. For those attempts a raw percent would favour whoever got the easier part 2, so the score is based on estimated ability instead: a Rasch model, using each question's calibrated difficulty, estimates your ability from every scored question you were given in both parts (a question counts as right only at full points, because the difficulties were calibrated on right and wrong answers, and an unanswered question counts as wrong), and your score is the percent you would be expected to earn on the whole calibrated question bank at that ability. That score is the one compared with the pass mark, used for the percentile and shown on the credential, and the range shown is its 95% range. The raw points, with partial credit as above, are kept with the attempt and match its competency scores. An attempt that goes to a person for grading is scored on raw points as described above.

You pass when your score is at or above the assessment's pass mark. Current pass marks:

• 80% for every assessment.

When you choose an optional junior, mid-level or senior level, that level has its own pass mark, shown on the level picker before you start.

A passing credential can carry a mastery level:

• Mastery: 90% or more.
• Practitioner: 85% to below 90%.
• Apprentice: a pass below 85%.

These thresholds are the same for every assessment today. Pass marks and levels were set by the people who wrote each assessment. They have not yet been set or checked by a formal standard-setting study. Levels are currently recorded when a reviewer finalizes a passing attempt; an attempt that passes on automatically scored questions alone does not show a level.

Every attempt is timed and runs with the candidate's consent to monitoring, shown before it starts. The runner records tab and window switches, fullscreen exits, copy and paste events, certain keyboard shortcuts, keystroke timing (when keys are pressed, not what is typed), and mouse movement patterns. With the candidate's consent it also takes camera snapshots. Each credential states which of these controls ran and shows a short public integrity report with the counts; detailed data stays private to the candidate and reviewers.

These signals are context for a reviewer. None of them lowers a score or fails an attempt automatically. A submission more than 10 minutes past its time limit (after any approved extra time) is always held for a person. A daily scan compares code answers across candidates on the same question and flags identical or structurally near-identical pairs for review. An attempt a reviewer flags for integrity is failed and excluded from every percentile and question statistic, and the candidate can contest the flag with a person. Counts of flags and contest outcomes are published on the Trust Center.

Candidates can request extra time before an attempt (1.25, 1.5 or 2 times the standard limit). We ask what adjustment you need, never why, and collect no documents. An accommodated attempt is scored and ranked exactly like any other, and nothing on the credential says it was accommodated.

Scoring never reads anything about who you are. We have built an optional, voluntary demographic question so that pass rates can be compared across groups (flagging any group whose pass rate is under four fifths of the highest group's, and reporting nothing for groups under 10 people). It is switched off while legal review is pending, so we have no data yet on whether any assessment has different outcomes for different groups.

For each question we track, over finalized attempts that were asked it and not flagged for integrity: how often it is answered correctly (a blank counts as wrong), and how much better the top 27% of candidates do on it than the bottom 27%. Results are withheld until a question has 20 responses. A question is flagged when more than 95% or fewer than 20% get it right, when it barely separates stronger from weaker candidates or separates them the wrong way, or when its correct rate in the last 30 days jumps well above its earlier rate, which is a common sign of a leak. We also fit a Rasch (item response theory) model as a diagnostic. Scores themselves are the simple percent described above, not model estimates.

A reviewer can retire a flagged question. It stops being drawn into new attempts; attempts that already hold it keep it and are scored normally, so no one's result changes. A bank cannot be thinned below 60% of its size, or below six questions.

Leak watch. Every week an automated job takes the next 15 active questions, in rotation across every bank, and searches the public web for a distinctive ten-word phrase from each. A match on any site other than Provyn's retires the question immediately (subject to the same depth limit) and alerts us. The most-seen tenth of each bank is listed internally as the first to be replaced.

• Small samples. Most assessments have been taken by few people so far. That is why many credentials show a band or “cohort forming” rather than a percentile, and why many questions do not yet have enough responses for reliable statistics.
• Pass marks are author judgement. They have not been set by a standard-setting study (such as a panel of practising engineers rating each question), and mastery thresholds are one default for all assessments.
• No predictive validity evidence. We have not shown that scores predict performance in a job.
• Some banks are shallow. Where a bank is no larger than one draw, every candidate sees the same questions, which makes them easier to share.
• AI grading needs a person. AI grades are suggestions that a reviewer confirms. That keeps a person accountable for every judgement, and it also means a reviewer's judgement is part of the score. How often reviewers and the AI agree is tracked internally but not yet published.
• Precision. The 95% range covers only automatically scored questions and only sampling variation. It says nothing about whether a question measures what it is meant to.
• Integrity controls run in the browser. They raise the effort needed to cheat and give reviewers context. They cannot prove that nobody helped.
• Leak watch is partial. It searches only the public web, a slice at a time, for an exact phrase. Paraphrased or private leaks are caught, if at all, only by the jump in correct rates.
• No fairness data yet. See above.