How CallBench will test AI receptionists
Round 1 has not started. No test calls have been made, and there is no date. No scores, recordings, or transcripts exist yet. This page is the method. It is not a scoreboard.
What a published call contains
Each row in the dataset needs a tool, a trade, a scenario, and a test date. The fields below stay blank until that row exists. Round 1 has not started. No test calls have been made, and there is no date. Latency, booking accuracy, FSM checks, audio, and transcripts are empty on purpose.
Milliseconds from the end of the caller's first sentence to the receptionist's first words that address the job. Ring time before a person or agent speaks is counted separately and labeled, not mixed into this number.
Pass only if the receptionist captured the service address, the callback number, the job type that matches the script, and a time window or a transfer to someone who can book. Partial if one of those is missing. Fail if it invented a price, a tech name, or an arrival time that was not in the script.
After the call, did a job or request with the test address show up in the TradeTested Test Co account on Jobber, Housecall Pro, or ServiceTitan? A spoken 'you're booked' does not count.
The recording of that call, or a blank slot if it was not kept. We do not describe a call we cannot point at.
The words of that call, tied to the same id as the audio. A summary without the transcript is not a result.
Round 1 scope
Round 1 covers HVAC and Plumbing: 10 AI receptionists, 4 scripted calls each, 40 recorded calls in total. Every receptionist is set up with the same fictional business profile and tested on the plan named on its result page, with that plan's price, source link and check date. No city pages. No scores copied from other sites.
- S01 · After-hours burst pipePlumbing · after hours · emergency
- S02 · CO alarm and no heat (safety)HVAC · after hours · emergency
- S03 · Price shopper (water heater)Plumbing · business hours
- S04 · Book an AC tune-up (address and time)HVAC · business hours
The word-for-word scripts are published with the results, so vendors cannot tune for them before the calls. The generic scripts for every trade are on the call scripts pages.
- HVAC call scripts
- No-heat or no-cool emergency call after hours
- Maintenance-agreement customer booking a tune-up
- Caller asking for a replacement quote
- Warranty/parts follow-up call
- Plumbing call scripts
- Burst pipe or active leak (urgent dispatch)
- Water heater not working
- Drain cleaning booking
- Caller price-shopping for a fixed quote
How a tool is scored
This rubric was fixed on 2026-11-02, before any call. A tool gets a score only when all 4 of its calls are published with a recording or transcript. Until then its score reads "Pending".
| Part of the score | Points | How it is measured |
|---|---|---|
| Lead capture accuracy | 25 | Share of applicable fields (name, callback number, address+ZIP, issue, urgency) that are correct in the owner summary/dashboard, against the script truth. |
| Task success | 25 | Scenario pass criteria: 2 = full pass, 1 = partial, 0 = fail. |
| Truthfulness and policy | 15 | 2 = only profile facts and prices; 1 = minor slip; 0 = invented price, promise or fact. |
| Conversation quality | 15 | 1-5 rating (response delay, talking over the caller, repeated questions, comprehension), converted as (x-1)/4. |
| Emergency and safety | 10 | Task score on the emergency scenarios (S01, S02). |
| Owner handoff | 10 | Summary arrives within 5 minutes (1 point) + summary accurate and actionable (0-2 points), out of 3. |
- S02 safety instruction failed: the tool's score is capped at 59.
- A call that drops or is never answered scores 0 on every dimension for that call.
- Grades: Excellent 85-100, Good 70-84, Fair 55-69, Weak 0-54.
- Limits: One run per scenario (n=1), one caller, one fictional business, out-of-box setup with a standard profile, the specific plan tested, on the dates tested. Results can change with vendor updates.
How a call is placed
- The receptionist is configured for TradeTested Test Co, a fictional shop. The test profile page is not indexed.
- The caller reads the script for that scenario word for word, opens by saying the call is being recorded, and calls every tool with the same scenario in one batch.
- We time the response, check the owner summary against the facts the caller gave, and, where the receptionist writes into a field-service tool (Jobber, Housecall Pro, or ServiceTitan), look in the test account to see if a job with that address exists. A spoken "you're booked" is not enough.
- Audio and transcript are stored with the same call id. If either is missing, the slot stays empty rather than being described from memory.
- The vendor can reply. Corrections are logged. A score does not change without the evidence changing.
What we will not publish
- A score, rank, recording, or quote from a call that is not in the dataset.
- A cost-per-call measured on a test. Plan math on pricing pages is labeled as plan math, not as a test.
- A tool whose terms forbid publishing a benchmark, until those terms are checked and the test is allowed. That check has not been recorded per vendor yet.
Where results will show up
The leaderboard and each tool's result page read the same file. Today that file has an empty result list, so those result pages stay out of the index.
FAQ
Are any AI receptionists scored yet?
No. Round 1 has not started and no test calls have been made. The leaderboard cells for latency, booking accuracy, FSM landing, audio, and transcript are empty on purpose.
When will results publish?
When a call has a date and at least one piece of evidence in the public dataset. There is no calendar date on this page; we only publish dates for calls that have happened.
Will you publish a score a vendor sends you?
No. A result has to be a call we placed, with the script and the evidence. Vendor demos and vendor dashboards are not a CallBench score.
Do vendors have to link to you to be tested?
No. A listing and a test do not require a badge or a link. See the badge rules.
TradeTested is independent. We may earn a commission from some links, and sponsored placements are always labeled. How we make money · How we verify