How to Vet an AI Tool Before You Let It Into Your Practice
By Mike Woo-Ming, MD, MPH
There's a new AI tool for doctors every week now, and the pitch is always the same: it'll save you time, cut your admin, make you a better clinician, and it's basically magic. Some of these tools are genuinely great. Some are demos held together with tape. And a few will happily do something with your patients' data that you'd never have agreed to if anyone had spelled it out. The trick is telling them apart before you've wired one into your practice.
I've bought a lot of software in my life — I built and sold a software company, so I've been on both sides of the table. Let me give you the way I'd size up an AI tool now, as a physician, before I let it anywhere near a patient.
Start with the job, not the shiny object
The most expensive way to buy technology is to fall in love with a demo and then go looking for a problem it solves. Flip it around. Name the actual pain first. Is it that documentation is eating your evenings? That your inbox never empties? That scheduling is a mess? Get specific about the job you're hiring the tool to do.
Once you know the job, most of the flashy features stop mattering. You're not buying "AI." You're buying a solution to one concrete headache, and you can judge every tool by one question: does it actually take that headache away, or just add a new one with a nicer interface? Half the tools out there are answers in search of a question. Don't let them make their question your problem.
Follow the data
This is the one I won't let physicians hand-wave past. Before anything else, understand what happens to the information you feed the thing. Where does it go? Where is it stored, and for how long? Who at the company can see it? And the big one — is your patients' data being used to train the vendor's models, and can you say no?
Ask these in plain language and make them answer in plain language. A serious company that works with healthcare will have clear answers and the proper agreements ready to sign. If you get vague reassurances, buzzwords, or a sales rep who has to "check on that," treat it as your answer. You are the one accountable for that information. The vendor isn't the one whose name is on the chart.
Remember it's a draft, not an oracle
However good the tool is, it doesn't carry your license and it doesn't carry your judgment. AI tools mishear, they smooth over the messy details, and every so often they'll state something completely wrong with total confidence. That's not a reason to avoid them. It's a reason to keep yourself firmly in the loop.
The right mental model is a sharp, fast, tireless assistant who hands you a first draft. You'd never sign an assistant's work without reading it, and you shouldn't here either. Whatever the tool produces under your name — a note, a message, a summary — is your professional word the moment you approve it. Build the habit of checking before you sign, and you get the speed without handing over the responsibility.
Run a small, real pilot
Don't roll a tool out to your whole practice on the strength of a sales call. Try it small and try it real. Give it a couple of weeks on your actual work, with your actual patients and your actual messy edge cases, not the tidy examples in the demo.
Pay attention to the honest question: is this genuinely saving me time and mental load, or am I now spending my "saved" time fixing its mistakes? Watch how it does on the weird cases, because the weird cases are where these tools fall apart. And notice how your patients react if it's visible to them. A short pilot costs you a little effort now and saves you from a painful, tangled rollout later.
Know what leaving looks like
Nobody thinks about the exit when they're excited about a new tool, and that's exactly when you should. If you cancel in a year, can you get your data out in a usable form? Or are you about to get locked into a company because everything you need is trapped inside their system?
Ask about it up front. The good vendors don't flinch at the question — they expect it. The ones who get squirmy when you ask how to leave are telling you something useful about what it'll feel like to be their customer. Give yourself a door before you walk in the room.
Use them with curiosity and a little skepticism
I'm genuinely optimistic about where this is going. Used well, these tools can haul away the administrative junk that's been burning good doctors out and hand you back time for the part of the job that actually needed a human. That's a real gift, and I don't want you to sit it out.
But optimism and diligence aren't opposites. Get clear on the job, follow the data, keep your judgment in your own hands, pilot before you commit, and make sure you can walk away. Do that, and you get the upside without the nasty surprises. The doctors who'll do best with AI aren't the ones who trust it blindly or the ones who refuse to touch it — they're the ones who bring a curious mind and a healthy dose of skepticism to every tool that comes knocking.
Dr. Mike Woo-Ming is a Mayo Clinic-trained physician, practice owner, and founder of BootstrapMD. He's the author of The Positioned Physician: Earn More, Work Smart, Love Medicine Again. Learn more at michaelwooming.com.
This is general educational information, not legal, medical, or compliance advice.
