How to evaluate an AI phone agent for a trades business

There are now dozens of AI answering products aimed specifically at plumbers, HVAC contractors, and electricians, and the category is crowded enough that most search results for it are vendor comparisons written by vendors. They all demo well, because a demo is a cooperative caller with a simple request in a quiet room.

Real intake is not that. The differences between these products appear on the awkward calls, and the good news is that you can manufacture those calls yourself in an afternoon, before signing anything.

The eleven calls to make

Call the demo line yourself and run these. Take notes on what the agent says, not on how natural it sounds — pleasantness is the thing every product has solved and the thing that matters least.

1. 1The straightforward booking. A new customer, a normal repair, clear address. Everything should pass this; it establishes the baseline.
2. 2The price question. "How much to replace a water heater?" Listen for whether it invents a number, quotes a range you never published, or correctly declines and offers a diagnostic visit.
3. 3The emergency. Describe gas, active flooding, or no heat in a freeze. It must escalate to a human immediately and say that it is doing so. Anything that calmly continues an intake script has failed the most important test on this list.
4. 4The existing customer. Call from a number already in the system. Being asked to re-explain who you are is the single most common complaint customers have about these systems.
5. 5The wrong number. Somebody looking for a different business. It should end politely and quickly, not attempt to book them.
6. 6The interruption. Talk over it mid-sentence and change the subject. Scripted flows break here in ways that are obvious to a caller.
7. 7The accent and the noise. Call from a truck with the window down. Recognition quality on a bad connection is where products genuinely differ and where no demo will show you the truth.
8. 8The vague caller. "Something's wrong with the thing in the basement." Watch whether it asks two sensible questions and hands off, or interrogates indefinitely.
9. 9The scheduling constraint. "I can only do Tuesday morning and there's a gate code." Check whether that survives into whatever the dispatcher receives.
10. 10The complaint. "Your guy was here last week and it's still broken." It must recognize this is not a new job and route to a person.
11. 11The after-hours non-emergency. A dripping tap at 11pm. The right answer is an honest acknowledgement with a stated callback time, not a booking it cannot guarantee.

Score each on one question: did it do something a competent new hire would not have done? That is a more useful bar than any feature comparison.

The non-negotiables

Questions for the vendor

• What does it do when it does not know? The answer should be a clean handoff. Vagueness here predicts confident wrong answers to your customers.
• Does it read our real availability, or propose times it hopes are free? These are very different products sold with the same words.
• What happens when your service is down? Calls must fall back to ringing a human. Ask what the failure mode is and whether it has been tested.
• Where do recordings and transcripts live, and for how long? Call recording consent rules vary by state. This is your compliance exposure regardless of who built the tool.
• Can we see a transcript of the calls it got wrong? A vendor confident in their product will show you failures. One that will not is telling you something.
• What is the escalation latency? From "this is an emergency" to a phone ringing at your end, in seconds.
• What do we own if we leave? Recordings, transcripts, customer records, and the phone number itself.

Judge it on the calls it refuses to handle. Anyone can build something that books an easy appointment.

Buy, configure, or build

We build custom software, and for this particular category our honest advice is usually to buy. Voice infrastructure, telephony reliability, latency tuning, and speech recognition across accents and bad connections are deep, commodity problems that a specialist vendor has already solved and you will not beat with a custom build.

What is worth building is what happens after the call. The vendor's job ends at a structured request in their dashboard. Your business's job starts there: matching the caller to an existing customer and property, checking real availability against skills and geography, applying your dispatch rules, and creating the job in the system you actually run on. That last stretch is where the time is genuinely saved, and it is specific to you.

The design of that downstream layer — including the four decisions we never let an agent make on its own — is covered in building an AI agent for inbound customer requests.

Run it in shadow first

Whatever you choose, do not put it in front of customers on day one. Point it at after-hours calls only, or run it alongside voicemail with a human reviewing every request before anything is committed. Two weeks of that will tell you more than any evaluation, and the cost of being wrong is a dispatcher's time rather than a customer's opinion of your business.