Exploration of ChatGPT citations paints a bleak picture for publishers

As publishers continue to cut content licensing deals with ChatGPT-maker OpenAI, a recent study by the Tow Center for Digital Journalism raises some interesting – and concerning – questions. The study looked at how the AI chatbot produces citations for publishers’ content, revealing that regardless of whether publishers allow OpenAI to crawl their content, they are still at the mercy of the generative AI tool’s tendency to invent or misrepresent information.

Unreliable Sourcing

The research conducted at Columbia Journalism School examined citations produced by ChatGPT, analyzing quotes from a variety of publishers including The New York Times, The Washington Post, and the Financial Times. The findings revealed that no publisher, whether affiliated with OpenAI or not, was spared from inaccurate representations of their content by ChatGPT. The researchers found a spectrum of accuracy in the responses, with some citations being entirely correct while others were entirely wrong or fell somewhere in between.

The researchers also discovered instances where ChatGPT erroneously cited publishers’ content, with very few instances where the chatbot didn’t project total confidence in its incorrect answers. This lack of transparency in the chatbot’s confidence in its responses makes it challenging for users to assess the validity of a claim and trust the information provided. Additionally, incorrect citations could pose reputation and commercial risks for publishers, as readers may be directed to incorrect sources.

Decontextualized Data

Furthermore, the study highlighted the potential for ChatGPT to reward plagiarism. An example cited by the researchers involved ChatGPT incorrectly attributing a piece of “deeply reported” New York Times journalism to a website that had plagiarized the content. This raises concerns about OpenAI’s ability to filter and validate the quality and authenticity of its data sources, particularly when dealing with unlicensed or plagiarized content.
In further findings that are likely to be concerning for publishers that have partnered with OpenAI, a recent study discovered that ChatGPT’s citations were not always reliable. Therefore, allowing its crawlers in does not guarantee accuracy. The researchers argue that OpenAI’s technology treats journalism as decontextualized content, showing little regard for its original production circumstances.

Variation in ChatGPT’s responses is another issue highlighted in the study. Testing the bot with the same query multiple times resulted in different answers each time. While this variability is common in AI tools, it poses a challenge in citation accuracy. The study, although small-scale, raises questions about the high-level deals major publishers are making with OpenAI.

The study suggests that media businesses may not receive special treatment for their content in terms of accurate sourcing through OpenAI. Even publishers without licensing deals but still allowing OpenAI’s crawlers in may not benefit from accurate citations. The study reveals that there is no guaranteed visibility for publishers in OpenAI’s search engine, even when their content is included.

Blocking crawlers does not necessarily protect publishers from reputational risks, as the study found instances of incorrect attribution to reputable sources like The New York Times. Researchers conclude that publishers have little meaningful agency over their content once ChatGPT processes it.

OpenAI responded to the research findings, stating that they support publishers by helping users discover quality content through summaries, quotes, links, and attribution. They claim to collaborate with partners to improve citation accuracy and respect publisher preferences in search results. The ongoing efforts aim to enhance the overall search experience for users.