Kevin Fox: Before trusting an AI to tell you about stuff you don’t know, ask it to tell you about things you are an expert in

Several years ago, I listened to a few episodes of Stuff You Should Know. This is a podcast with two guys who spend a brief time learning about a topic before making an episode about it. I enjoyed it until I heard an episode on a topic I happen to know a lot about and realized how full it was of mistakes and misconceptions. They filled in the gaps with stories that seemed plausible to them. It was painful to listen to, and I unsubscribed to that podcast. If they got it so wrong on a topic I know about, they likely do the same with topics I know less about.

As a starting point, or out of curiosity, I do sometimes ask AI on a topic. If it’s important, I always check with other sources. Often, it is surprisingly good, and sometimes it’s way off.

I also often do what this quote suggests. I ask it questions on topics I know a lot about to see how well it does. When I started with ChatGPT, I asked it about topics I know a lot about, and I could see the misconceptions it had and how it would fill in gaps with plausible – and often completely wrong – information. For instance, I asked ChatGPT about the origin and history of Breema, and since there is little to no information about it online, it made up something that sounded plausible. That seems to be the nature of AI as it is right now.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.