AI in plan review: questions to ask any vendor

AI plan review has gone from a pitch deck promise to a crowded vendor category in under two years. That's good news for the profession — competition pushes quality up. It is also a warning: when everyone claims accuracy, the only way to tell real work from marketing is to ask the questions that a demo cannot paper over.

Here are the questions I would ask any vendor, including us.

1. How big is your evaluation set, and what kinds of plans are in it?

Any serious AI team runs an evaluation set — a held-out collection of real plans with known correct answers that the model is graded against on every change. The size of that set, and the diversity of the plans in it, is the single best proxy for how well the tool will work on your work.

Ask: how many plans are in your evaluation set? How many jurisdictions? What project types — ADU, remodel, new construction, multifamily? Can you show me a sample?

If the answer is vague, that is the answer.

2. What is your accuracy — broken out by check type?

A single overall accuracy number is almost meaningless. A tool can be ninety-five percent accurate on pass/fail checks for rooms that obviously meet code and still miss the structural comments that matter. Ask for accuracy broken out by the categories of checks you actually spend your time on: fire and life safety, plumbing, and structural.

Ask them to define what counts as a correct answer. "Agrees with a senior reviewer" is a real definition. "Matches our internal expectations" is not.

3. How do you update for code cycles?

The CRC, CEC, and Title 24 all update on multi-year cycles. Ask how the vendor keeps pace. Who writes the code-to-check mapping? How is it validated? How long between a code taking effect and the tool supporting it?

A vendor that hand-waves this is a vendor whose product will be out of date the next code update.

4. Can I run it on a plan of my choosing, right now?

This is the question that ends most vendor conversations that should end. A confident vendor will let you upload one of your own plans and see the draft output in minutes. A nervous vendor will schedule a demo and show you cherry-picked examples.

If the tool cannot survive a plan you picked, it will not survive the plans you review next week.

Closing

The plan review profession is earned, and any AI that wants to be part of it should have to earn its place too. These questions are the ones we ask ourselves internally. We think every vendor should have to answer them.