In a paper, published in iScience in 2021, researchers evaluated our ability as humans to differentiate real, authentic videos from deep-fakes. A total of 210 people participated in the study. The study was designed to have them watch 16 short videos (eight were deep-fake and eight were athentic) and determine which ones were deep-fakes. The researchers found that people can’t detect deepfakes as well as they think they can.

As humans, we are subject to a “seeing-is-believing” heuristic, which is why 67% of the time the participants thought deep-fake videos were real.

67%!

Despite this, people remained overconfident that they could spot a deepfake easily---and the most overly confident people tended to be the least accurate. Sound familiar?

This brings to mind the Dunning-Kruger effect where the people with least amount of knowledge or understanding of something are most confident that they are experts. It happens to the best of us.

AI as Change-Agent

As someone using AI professionally and personally, I consider myself an expert of sorts. I teach AI at my local Adult School and use it at work every single day.

But like others, I am sometimes fooled. And sometimes I let myself be fooled, because I want something to be real — even if it’s not. But seriously, on a day-to-day basis, I am not dealing with deep-fakes. I am dealing with data and content. If something seems too good to be true, then it probably is.

Repeatedly throughout human history, we have always been seduced by that special thing that is finally going to solve our biggest problems. One of the best examples is the proverbial Fountain of Youth.

Can We Out-Prompt Errors?

AI is fallible, but good prompting can increase accuracy. AI’s preponderance for introducing random mistakes shows up differently depending on the setting. For me, it is nuanced and aggravating.

For example, on any given day I may be working with ChatGPT, Gemini, Claude, Notebook, or Napkin. I might be working on a report, a spreadsheet, or a PowerPoint deck, going back and forth with AI—when suddenly, I realize that something is not right. In fact, 6 iterations ago, AI may have made a tiny unintended shift in output.

There may be an inversion, a deletion, an addition of unrequested generative content, or changes in numerical values. You must check and recheck. Of course, if I ask AI to provide quantitative data to help me develop a marketing plan, I need to make sure that all the data generated is correct.

Sometimes data shows up out of nowhere that is so good, it would be a guaranteed win for the brand I’m supporting. But, of course, when I see the number, I ask AI to recheck.

Prompt Strategies

Invariably, AI says something along the lines of, “Ooops, my bad. Here is the correct answer….” I then must recheck THAT answer. One prompt that helps:

“Be 99% sure that all data is accurate before completing this task. Do not change anything except the highlighted areas. Make sure everything is referenced parenthetically and include links to the references.”

AI is here to stay, and even though it’s highly fallible, it is an incredible tool for productivity and creativity. It can also be easily used to mislead consumers, jurors, and voting constituents. Rigorous oversight and stewardship are needed. Let’s enjoy the tremendous advances that come with AI but remember that if it seems too good to be true — it probably is.