SigmaLab methodology demonstrator

From question-bank data to defensible assessment decisions.

This environment demonstrates how item responses can be structured, tested and translated into evidence-led recommendations for question-bank review, candidate diagnostics and assessment design.

Original prototypeLearning & readiness flowQuestion bank, diagnostic, mock assessment and study planning.

Redesigned demonstratorItem & cohort analyticsFacility, discrimination, omission, timing and reliability indicators.

Evidence safeguardNo outcome inflationNo real-candidate performance gain or official deployment is claimed.

Bank blueprint

306

items represented in the use case

Analysed sample

demonstrator items

Synthetic attempts

anonymised simulated candidates

Pass rate

—

threshold set at 70%

Items to review

—

rule-based analytical flags

Score distribution

Synthetic results across one diagnostic assessment form.

Mean —

Performance by domain

Mean correct-response rate across the analytical sample.

Target ≥ 70%

Priority item signals

Items requiring human review—not automatic deletion.

Decision workflow

A traceable path from responses to an approved recommendation.

StructureMap every item to domain, competency, intended difficulty, version and status.

AnalyseCalculate item and cohort indicators using an agreed analysis plan.

InvestigateCombine statistical flags with expert review and stakeholder context.

RecommendRetain, revise, re-key, reclassify or retire—with a recorded rationale.

VerifyMonitor revised items in a subsequent test window before final acceptance.

01 / Question bank

Structured item inventory

A governed bank links each question to content, competency, intended difficulty, version and review status. The demonstrator displays 24 sample items from a 306-item blueprint.

Sample content

Question wording is illustrative and is not an official EOPPEP question bank.

Item	Domain	Competency	Prompt	Intended level	Version	Status

02 / Item analysis

See how each question behaves.

The engine identifies unusual item behaviour. Every flag is a prompt for expert review, not a machine-made verdict.

Select a row to inspect response pattern, distractor choices and the suggested human-review action.

Item	Domain	Facility	Discrimination	Omission	Mean time	Priority

03 / Cohort performance

Understand the test—not just the average score.

A cohort view shows score distribution, domain coverage, pass-rate sensitivity and overall internal consistency.

72 synthetic attempts

These figures demonstrate calculation logic only.

Candidates

synthetic IDs only

Mean score

—

across 24 items

Median score

—

distribution midpoint

Pass rate

—

at 70% threshold

KR-20

—

illustrative reliability indicator

Score distribution

Counts in ten-percentage-point score bands.

Domain performance

Mean correct rate and content coverage.

Pass-threshold sensitivity

How pass rate changes under alternative policy thresholds.

Threshold selection is a policy and standard-setting decision. This chart does not recommend a pass mark by itself.

Interpretation guardrails

What the dashboard is designed to prevent.

No single-metric decisionsFacility, discrimination, omissions and content validity are reviewed together.

No automatic item removalStatistical signals trigger expert investigation and an auditable decision.

No demographic inference hereFairness analysis requires lawful, appropriate data and an agreed governance basis.

04 / Candidate diagnostics

Turn results into a focused learning path.

The original preparation concept remains visible here, but as a secondary layer informed by assessment data.

Candidate C-001

Synthetic diagnostic profile

Synthetic

—

Readiness score

Personalised study plan

Generated from the candidate’s two lowest-performing domains.

Human-adjustable

Candidate-level interpretation

A defensible diagnostic statement avoids overclaiming what a short assessment can prove.

Observed evidenceResponses, omissions, response time and topic-level score in this attempt.

Permitted inferenceAreas that may benefit from review, followed by further assessment.

Not inferredPermanent ability, employability or future performance from one test alone.

05 / Recommendations

Convert analytical signals into an auditable action plan.

The report layer separates evidence, interpretation, recommended action, owner and verification step.

Priority review register

Automatically populated from the analytical sample.

—

Recommended governance cycle

Indicative delivery workflow for a live assignment.

Data and bank auditValidate schema, versions, missingness, duplicate records and answer keys.

Analysis plan approvalAgree metrics, thresholds, exclusions, subgroups and reporting conventions before analysis.

Item and test analysisRun reproducible calculations and preserve a decision trail.

Expert review workshopInterpret flags with subject-matter experts and operational stakeholders.

Revision and verificationRe-test amended items and compare subsequent behaviour against the agreed baseline.

Executive analytical statement

Demonstration wording suitable for a methodology section—not a claim about real outcomes.

This synthetic demonstration identifies item-level signals that merit expert review, including low discrimination, extreme facility, high omission and extended response time. The recommended next step is a governed review of flagged items against content validity, answer-key accuracy and operational context, followed by verification in a subsequent test window.

06 / Method & evidence

What this demonstrator proves—and what it does not.

This page protects the credibility of the tender submission by keeping delivered work, later demonstration work and unsupported claims clearly separated.

Claim-control register

Implemented previously

Original functional prototype

A working front-end preparation environment was designed and used in a one-to-one learning context.

Structured sample question bank
Diagnostic quiz interaction
Mock-exam and timer flows
Readiness visualisation
Flashcards and study-plan logic

Added as demonstrator

Assessment analytics framework

These extensions demonstrate SigmaLab’s proposed methodology using synthetic response data.

Item facility and discrimination
Omission and response-time signals
Distractor-effectiveness view
Score distribution and KR-20
Priority register and recommendations

Not claimed

Unsupported experience or outcomes

The demonstrator must not be presented as evidence of facts that did not occur.

No official EOPPEP commissioning or endorsement
No production institutional deployment
No large real-candidate dataset analysed
No measured performance improvement
No automated high-stakes decision-making

Metric definitions

Definitions used in this functional demonstration.

Facility indexProportion of all candidates who answered the item correctly. A high value indicates an easier item in this sample.

Discrimination indexDifference in correct-response rate between the top and bottom 27% of candidates, ranked by total score.

Omission rateProportion of candidates who did not submit an answer for the item.

KR-20An internal-consistency indicator for dichotomously scored items. Interpretation requires sample and test-context judgement.

Tender-use guidance

How to position this asset accurately.

✓

Use as a technical methodology demonstratorIt provides inspectable evidence of the proposed analysis and reporting approach.

✓

Use as a capability sampleIt demonstrates product design, analytical logic and human-review safeguards.

Do not use as an official client reference by itselfA qualifying reference requires truthful contractual evidence, dates, scope, value and a verifiable client contact where required.

Check the final procurement documentsOnly the contracting authority’s criteria determine whether a project qualifies as prior similar experience.