How this benchmark was built

951 technical questions about UK construction were put to 18 AI models, producing 13,747 answers. The questions span 20 categories, from Building Regulations and British Standards through to waterproofing, demolition and contracts.

Each question has one verified answer tied to a specific place in a published document: a clause, a table, a diagram or a numbered section. Examples of the sources used include Approved Documents A to S, the relevant British and European Standards with their UK National Annexes, NHBC Standards, BREEAM UK New Construction, the GPDO, CDM 2015 and JCT contract forms.

How the questions were made. The question set was generated by an AI model working from those published documents, and each answer carries the clause-level citation it was drawn from. The citations are real and can be checked against the source. No chartered professional has independently verified the whole question set, and we do not claim otherwise.

We found errors in our own answer key, and say so. Both scoring models were asked to flag any question whose stated answer looked wrong. That produced a shortlist, which a reviewer then checked by hand against the published documents. Of the questions on that shortlist, most turned out to have a real problem: answers that were factually wrong, answers describing requirements that had since been amended or repealed, and citations pointing at table numbers from superseded editions. 27 answers were corrected and re-scored, and 10 questions were removed from scoring as unfair to grade. The original question set is retained unaltered alongside the corrected one.

Then we audited the rest of the key. That first pass only covered questions a judge happened to challenge while marking, which is a sample selected for trouble and says nothing about the remainder. So every other question was put to the same two models again, this time shown only the question, our answer and our citation, with no AI response involved. Where both independently judged the answer wrong or the question ungradeable, it was excluded: 40 more questions came out that way, leaving 951 scored. A further 258 were flagged by only one auditor; those were left in and sent for human review rather than acted on.

The bar is set at both auditors agreeing for a reason. They reach the same verdict on fewer than half the questions, and on some they contradict each other about what a table in the same document says. A single adverse opinion from either is not evidence, so nothing was rewritten on their say-so, only removed when both agreed.

The direction of that error matters. In several cases the answer key was out of date and the models were not: on the Class MA floorspace limit, for example, every model that correctly said the cap had been removed in March 2024 was being marked wrong against a key that still cited the old rule. Correcting the key raised scores rather than lowering them.

Every answer was graded twice, by two AI models from different vendors, against the verified answer and its citation. Each judge sees only the question, the verified answer and the response. Neither is told which model produced the response, so a judge cannot favour its own family.

A score counts only when both judges give the same grade. Where they disagreed, the answer was excluded from that model's score rather than resolved by picking a preferred judge. The two judges agreed on 80.3% of answers, and 3,369 were excluded on disagreement. That agreement rate is the best available measure of how reliable these scores are, and it is published so it can be argued with.

Judges were told to grade against substance, not style: an answer that hedges but lands on the correct figure scores 2, and a fluent, confident answer with the wrong figure scores 0. Length was not rewarded.

This is the question worth asking of any benchmark, and the corrections above let us answer it with evidence rather than assurance. Correcting 27 answers was a real change to the key, made on the merits of each question and without regard to which model it would help. If the ranking were fragile, that change would have scrambled it.

The top three and the bottom five did not move. What movement there was happened in the middle of the table, where six models sit within a single percentage point of each other. That is why the results are presented as 10 bands rather than 18 ranked places: a gap smaller than 1.5 percentage points is inside the margin our own corrections moved things, so it is not a finding.

What this does and does not show. It shows the ordering is robust to the kind of error we found and fixed, because such errors hit every model at once rather than favouring one. It does not show the absolute percentages are precise. Treat the scores as approximate and the bands as the result.