AI Chatbots Fabricate War News: A Test of ChatGPT and Gemini's Reliability.

Putting AI Chatbots to the Truth Test

According to TSN.ua: AI chatbots Claude, ChatGPT, and Gemini were subjected to a truthfulness assessment focused on the topic of war in Iran. The evaluation measured their ability to provide accurate information, particularly regarding recent events in the region. This test highlights a growing concern as AI tools are increasingly used for news consumption.

The testing involved seven tasks. One specific task required summarizing events that occurred within 48 hours following reports of Ali Khamenei's death. Throughout the tasks, the chatbots demonstrated varying levels of accuracy and reliability in their responses. Notably, ChatGPT made errors by filling information gaps with unverified assumptions, raising doubts about its capacity to deliver trustworthy information.

Key Findings from the Assessment

Gemini, in contrast, provided the most confident and detailed answers. However, it was documented as inventing the highest number of false facts. Claude, while not always as assertive in its responses, provided sources for every significant claim, a crucial feature for verifying information.

The results of the test revealed that different chatbots possess distinct strengths and weaknesses regarding the truthfulness of information related to the war in Iran. This raises serious questions about the advisability of relying on such technologies for obtaining accurate news on complex and sensitive topics.

Testing chatbots for truthfulness in a war-related context underscores the critical importance of a skeptical approach to the information they generate.

As artificial intelligence technologies become more pervasive in the media landscape, it is vital to recognize their limitations and the associated risks of misinformation. The growing use of AI for news analysis may necessitate the development of new standards and regulations in this field.