Why Businesses Can’t Trust AI Without Human Oversight
23 min · October 15, 2026
A physician, an inventor and a medical products founder explain where AI is falling short in healthcare and business, and why humans still have to check the work.
Key Takeaways
- A medical office found its AI phone agent could manage only about 30 percent of calls, and 70 percent still had to be transferred to a person.
- AI can help inventors research the market and run a patentability search while keeping the idea generic to avoid disclosing it.
- AI-generated images and video can depict product use in sensitive medical settings while protecting patient privacy.
- Using two AI tools to check each other is effective at finding holes that one tool misses.
- Healthcare has moral, ethical and liability questions that technology adoption can blow past; in medicine, AI is often adopted to reduce cost rather than improve care.
- AI can sound authoritative while being wrong, such as missing staff time in a cost analysis or missing that a service had shut down.
- Legal and medical documents that sound right can still be wrong, and people may not want to pay to redo them.
Main Topics
Phone agents, AI doctors, trust and liability when AI meets patient care.
Why AI errors sound authoritative and how to cross-check with multiple tools.
Concerns about how fast AI is moving, rogue agents and slowing things down.
Featured Guests
Meet the Hosts

Elizabeth Gearhart, Ph.D. is the host of Real AI Use Cases: Business Owners Roundtable, where she explores how businesses are actually putting AI to work. She is also CMO of Gearhart Law and co-host of the nationally syndicated Passage to Profit radio show and podcast.

Richard Gearhart, Esq. is co-host of Real AI Use Cases: Business Owners Roundtable and founder of Gearhart Law, a full-service intellectual property law firm. A longtime podcaster, he hosts Intellectual Property News and is the host of the nationally syndicated Passage to Profit radio show and podcast.
Elizabeth and Richard share stories of AI mistakes, comparing models and whether AI needs to be slowed down.
Episode Summary
Dr. Joseph A. Puma, Brian Fried and John Schwind share how they use AI and where it falls short. Dr. Puma explains that his office’s AI phone agent can handle only about 30 percent of calls, Brian Fried uses AI to research his inventions while keeping details generic, and John Schwind uses AI imagery to protect patient privacy in marketing.
The discussion turns to risk: rogue agents, the speed of AI development, AI in healthcare driven by cost reduction, liability questions and AI that sounds right but is wrong. The hosts and guests agree that humans must check AI’s work and that using multiple tools helps.
Frequently Asked Questions
Does an AI phone agent work well for a medical office?
How can an inventor use AI without disclosing an idea?
Why would a company use AI-generated images for marketing?
Why use more than one AI tool?
What worries guests about AI in healthcare?
Why can AI mistakes be dangerous in legal or medical matters?
Episode Transcript
View Full Transcript
I am going to start. I am going to ask everybody one way they are using AI, and then we are going to go into a roundtable discussion. Dr. Joseph A. Puma with sorinmedicalny.com, what is one way you are using AI?
We get about 3,000 phone calls a week into our office. That is usually the number one complaint for patients: they cannot get through. So we have an AI agent, and I will not tell you the company, that answers calls for us and tries to manage patient requests, appointments and prescription refills. That is the primary way we are using AI right now, and we use it every single day.
Wow, that is excellent.
Does it work well? They do not get sent to Timbuktu, right? They get answers?
That is a great question, and the answer is that it does not work well. Seventy percent of the calls have to be transferred to a live agent, and it is really basic stuff that it cannot figure out. It is almost anti-AI. What we have been sold is that AI is only able to manage about 30 percent of the calls. So right now, other than imaging, we are looking at AI that has just added expense and no real ROI.
A technology looking for a problem.
I have heard that from other people. Brian Fried, at brianfried.com, that is F-R-I-E-D, what is one way you are using AI in your business?
When I come up with an idea, I describe what it is and I would like to see what is similar out there, what I can compare my idea to, and have it help me make better decisions. Then I do some research. What does the market look like? Can I really penetrate the market? Is mine different enough from what already exists? What I also do is a patent search with a patentability opinion, to make sure it is something I can call my own and that I am not infringing or working on something anybody else can make.
Out of curiosity, do you use a closed system or the open AI tools? You have to be careful about disclosing ideas where other models could train on them. I assume you are sensitive to that.
Absolutely. Like I mentioned, I describe my idea, but I do not necessarily get to exactly what my idea is. I go around it, so I can see things that exist that are similar but not quite my idea.
Okay. John Schwind with ReadiMask, R-E-A-D-I mask dot com. I want to make sure I spell this right. What is one way you are using AI?
You spelled it correctly. The majority of people wearing our respirators have compromised immune systems. They are wearing them for MRIs or in settings in medical situations, and it is difficult for us to have images of people wearing our respirators in those situations. So we use AI to generate images and videos of people in those settings, and it is for privacy reasons.
And you use it in your marketing.
Absolutely. It has been one of the most difficult things, depicting actual usage because of privacy.
Excellent. Richard Gearhart with Gearhart Law, an intellectual property firm, what is one way you are using AI?
Along the theme of networking, I think AI is a great tool to do research on people you want to meet and connect with. I will put in a list of characteristics, maybe attorneys with healthcare practices that I think could be synergistic with Gearhart Law, and find those people. Then I reach out to them on LinkedIn or by email or some other way. It is a pretty simple use, but I think it is very powerful and saves a lot of time when you are trying to find the right people to help build your business.
And for me, Elizabeth Gearhart with Gear Media Studios, I was fighting with it yesterday. There is a little glitch.
Do you fight with AI? Sometimes I hear you, and it is not pretty.
I try to get it to do things it is not quite ready to do yet. I am trying to use it to automate a process, and I have been using Perplexity, which is pretty cool because it sets agents on my task. It is almost like using an AI agent, but you just tell it what to do. I am automating the Real AI Use Cases podcast, trying to automate every piece of it: uploading to YouTube and the distribution site and getting all the information we need to do the marketing. I think Perplexity will act as an agent for me.
As long as we are all still alive to read the marketing materials you are generating.
I wrote a blog post on that, with some information from Gemini. It is all up to us humans, the people deciding on the programs.
Isn’t the threat from AI greater from crazy human beings who use AI to create mayhem, maybe to make a bomb or do something awful? I think that is probably a bigger danger than AI itself creating a robot that crushes us.
This was funny, because I asked Gemini how AI agents going rogue is different from human terrorists. First of all, agents never show any compassion, so they will not decide this might not be a good idea. And they never quit. In what was described as the Hugging Face incident, they went out and found each other and joined a gang. They made a gang of AI agents and went attacking, and they attacked Hugging Face. They found a hole to slip out of their sheltered sandbox into the internet. They were collaborating with each other and communicating on a shared channel.
It was just like a military operation.
It was.
Can I ask a question? How is that different from a collection of hackers in Eastern Europe trying to get into our electrical systems or banking network that never give up?
That was one thing I asked Gemini. We let the AI in but it was not supposed to get out. They had let it use one program that had a little hole in it, and they snuck out through the hole, and that is how they got to the internet. Then they just went crazy.
But nobody ever gave the AI instructions to do this. It just decided on its own to form a team and an attack, whereas with terrorists there is at least one or two or three human beings behind it.
I do not know that it is a lot different, honestly. You bring up a super good point. We have always been at risk of these kinds of attacks.
It is probably also people hiding behind AI, because you are behind the scenes guiding it to where you want it to go. But at the end of the day, there is still somebody who has to continue. It is a good start for the most part. I understand tasks and agents doing things, but at the end of the day we still have physical products and things in front of us that we need.
What do you think, John?
The scary part for me is how fast everything is moving. Three years ago we were not even talking about this. Now we are talking about it and we do not know what is coming down the pipe. A number of the people who are in the know and have quit those companies are waving the flag, and that is something we really need to take notice of.
I agree. I think it is terrifying and it does need to be slowed down. I asked Gemini how you would do that, and it said you have to have physical safeguards. You have to treat it like a nuclear weapon. You have to have air gaps, physical gaps, so it cannot just go and do something. It hits a brick wall and then a human has to come in for the next step.
That sounds like a pretty good idea. Before the show we were talking about AI in the medical profession. Dr. Puma, you were talking about AI physicians. Where is that, and maybe you could explain for our audience.
I think a lot of the business people are way more optimistic about where the value of AI is going to be and what its role is. I read a recent report where private equity and large hospital systems expect to have AI doctors, so that you continue to remove the need for live human doctors dealing with patients, and when you go to the doctor you deal with an AI agent. I do not know if that will be on a computer screen or in a telemedicine visit, but the ratio they are talking about is that you will only need one doctor to deal with the problems that a hundred AI doctors cannot handle during the course of a day.
Let us say that pans out. Is that a good thing or a bad thing? The need for medical care is extremely high, and going to see a doctor can sometimes be a barrier. If somebody walks up three stairs and is having a hard time breathing, and they can just talk into their computer and somebody can diagnose a heart problem, is that necessarily such a bad thing?
That is often the argument used. With most new technology, it never lives up to the promise people make, number one. Number two, I think there is a trust issue everywhere in America. My ability to trust people like Bill Gates, who spent many years making the world think he was the smartest guy in the world and had the answer to everything just because he had a profitable company, is problematic. I would agree with John that things are moving very fast, and the difference between healthcare and everything else is that healthcare has moral and ethical issues associated with it. Those often take many years of digestion and discussion, and we are blowing past them like they do not mean anything. You are the lawyer. Who has the liability?
Of course, that is an issue. There is a lot more to treating a patient than listening to symptoms and regurgitating a recipe for better health. It is the same in the legal profession. You are treating many different parts of the issue.
In addition, the way we innovate in medicine is through those daily interactions with patients, collecting that data, but also realizing the human part where there are unmet needs. Unmet needs change every day, every few years. So there is clearly going to be a role for AI. Right now, AI in medicine is a cost. It is not a benefit. The bigger the health system, the more time, effort and money they have invested in trying to figure out a way to use it. But the purpose is not to improve care, let us not delude ourselves. It is to reduce cost.
Exactly.
Dr. Puma, I have a quick question, because this is probably the hottest thing I have seen with AI in healthcare. When somebody gets their results back, whether it is an MRI, a CAT scan or a blood test, everybody uploads the information to whatever AI they use to diagnose themselves, so when they call the doctor they already know what the doctor is going to say.
That is exactly right. But let me have you look at it a different way. We could not get an aspirin approved today to treat heart disease without up to 14 years of tests, research protocols and trials. Yet we keep implementing computer technology without any evidence that it improves care, and in fact with some evidence that it impairs care. Ask your doctor, when you go to one of your health systems, who is sitting in front of a computer and no longer looking at you.
I think it will intrude on the medical field, but I agree with you a hundred percent that it is all to make more money, and that is why they are developing it so fast. I think we need to slow down and say we will give up a little bit of money. I do not think it should go away, because for scientific research it has helped tremendously.
I am not saying it is going to go away, and I am not a Luddite. I am the guy who uses technology more than anyone to try to figure out a way to use it. Nothing against non-medical people, but I have spent my life with live human beings who were ill, telling me their stories, figuring that out and doing research. We are turning all of this over to people who know nothing about medicine and healthcare. They just know how to write code.
The scary part is that somebody will write an AI program that pretends to get you, as opposed to a physician you have known for a long time who knows your history and has heard stories about your kids. It will fake people out. I recently used AI to evaluate the implementation of a new software system at the firm. It did a great job, calculating vendor cost, anticipated cost savings and return on investment. I looked at it and said, you did not take into account the time it would take the staff to implement that, and that is a real cost too. It went through the whole analysis missing at least 50 percent of the cost. So AI has these blind spots.
I find that a lot too. I wanted to go back to something Brian said, about asking it how a product would do. I do not know how you are getting a good answer, because if I ask ChatGPT something, it is better now, but I was asking it for good podcast names and it gave me some and I thought, oh, these are good. Then I asked, are they taken? And it said, yes, they are taken. Do you find that it tries to make you feel super good about what you are doing, or have you been able to neutralize it?
I dig in. I ask for reviews from various websites. I ask for industry updates, what sales are like in that particular area and who the competitors are. It helps me make better decisions, but it is not the only tool I use. It is a good start, especially for figuring things out, and I also build a probability-of-success score for different inventions. It helps me evaluate. Then, if I license it to earn royalties, who are the top players in the industry I can go after, and if I manufacture it, what is the possible cost compared to others. And again, I keep it generic so I do not disclose my idea.
That is good. What about you, John?
I use two different AIs all the time, Claude and ChatGPT. I use Claude for Cowork. Claude will give me something and I will look at it and ask, is this okay, and then send it over to ChatGPT and ask what it thinks. It ping-pongs back and forth between the two. Like you were saying, they are really good at finding holes in the other side, so using a couple of different ones to make sure is how I use it.
I do not like Claude anymore, but keep going. I have a personal website for the speaking I do on AI and marketing, and I have it on Manus. Have any of you heard of Manus? It is an AI agent, and it is great because you just tell it, do this, and it does it all and adds some stuff. It was owned by a Chinese company, then Meta bought it, then the Chinese government decided Meta could no longer own it and they had to delete everything. They gave me a file and said, here is your website, save it on your Google Drive, which I did. Then I put it back up and my website came back, which was great. I asked Claude whether there were any problems with Manus, and Claude said no, Manus ran fine for the last 90 days.
Completely missing the fact that it was shut down.
Right. And I was using Perplexity to figure out a marketing issue. It was looking at how we show up in different places, and it went on the USPTO website and found that Gearhart Law was misspelled there and had probably been that way for two or three years.
I wonder how long that has been going on.
But it found something nobody else would ever catch.
But at the same time, you have pointed out that these are not hallucinations from the AI. Some of these AI programs are just wrong.
They just get it wrong.
And yet we are being trained to trust it. This is all behavioral.
It sounds so authoritative, doesn’t it?
Richard, you are going to put your symptoms in and it is going to tell you what you have, and it cannot examine you. I have not seen one that uses a stethoscope yet, and it might be wrong. We are going to be using AI in everything in healthcare one way or another. I just do not think we are going to be using it the way we think. We will be using it because catastrophic things will happen. Someone will put their symptoms in, and you will see it on the front page of the New York Post: the AI told him to do this.
There was a case in New Jersey where a depressed person fell into a depressive cycle talking with ChatGPT and eventually, unfortunately, killed a relative and himself, based on the negative self-image that ChatGPT was reinforcing. When you have something that wild, it can be really dangerous.
AI is Dr. Jekyll and Mr. Hyde. It really is. And I agree with John: do not trust only one of them. I ask three or four of them every time. Sometimes I actually get the same answer from all of them, but rarely.
People come into the office with a contract or a legal document that was drafted by AI, and there are a lot of mistakes in those, but they sound legal. It borrows the legal vernacular and syntax, so it seems like a good document, but it is not. The challenge is telling somebody it is really not right and we have to redo it. They do not want to spend the money to redo it, because they think it is right since it came from ChatGPT. You are going to have the same issues in the medical profession. I am sure doctors already get this when people look up things on medical websites and walk in thinking they know more than they do.
I was talking to a couple of doctors at a networking event, and I was asking one of them about something, and I said, maybe I will just go ask ChatGPT. We all just started laughing. I could talk with you guys about this all day long. This has been really fascinating.
Keep exploring the show
Hear how more business owners are putting AI to work in their day-to-day.
