On the Use of Chatbots in Writing Scientific Manuscripts
Ethan Vishniac American Astronomical Society (AAS)
This was published in the Bulletin of the AAS: Vishniac, E. T. (2023). Editorial: On the Use of Chatbots in Writing Scientific Manuscripts. https://doi.org/10.3847/25c2cfeb.c3619710]
In recent months there has been a mounting wave of discussion on the ability of large language modules (i.e., Chatbots such as ChatGPT), a recent development in artificial intelligence, to produce meaningful text (for examples of this discussion in the scientific literature see here, and here). Enthusiasts, or pessimists, have described these programs as the first stages in a process that will eliminate human writers and ultimately take over creative tasks throughout society. More measured views have touted this as a valuable tool for communication, with the potential to present information in ways that produce novel insights. In one famous case, an early example (from 2022!) was described by a software engineer as sentient. While much of the output is bland and predictable, these programs can produce nightmarish content when stimulated by a novel line of questioning.
The larger societal issues this raises are clearly beyond the scope of our journals, but some of these concerns are directly relevant to our work. These programs are trained to generate novel texts that can be difficult to distinguish from those written by humans. This raises the question of whether or not such programs have a legitimate role in scientific writing, and even whether or not they should be listed as authors. Proponents of their use suggest that they can rewrite confusing and garbled text and produce clear and incisive prose. On the other hand, they are prone to inventing references to support their arguments. At best, they are a tool for authors who find it difficult to write clearly in English but who have marshalled their arguments and references and prepared a rough draft of a manuscript. In this sense, they are an improved version of the Microsoft paperclip, which was sent to electronic heaven in 2003 because most users found him annoying and unhelpful. At worst, these programs can take a vaguely written text and produce a clear and emphatic argument for a point not intended by the authors, complete with fictitious references.
With this in mind, we offer two editorial guidelines for the use of chatbots in preparing manuscripts for submission to one of the journals of the AAS. First, these programs are not, in any sense, authors of the manuscript. They cannot explain their reasoning or be held accountable for the contents of the manuscript. They are a tool. Responsibility for the accuracy (or otherwise) of the submission remains with the (human) author or authors. Second, since their use can affect the contents of a manuscript more profoundly than, for example, the use of Microsoft Word or even the more sophisticated Grammarly, we expect authors to acknowledge their use and cite them as they would any other significant piece of software. Citing commercial software in the same style as scholarly citations may present difficulties. We urge authors to use whatever sources are most useful to readers, i.e. as detailed a description of the software as possible and/or a link to the software itself. Although these programs will surely evolve substantially in the near future, we think these guidelines should cover their use for years to come.
Finally, as an illustration of the promise and peril of its use, I asked ChatGPT (OpenAI 2020) whether or not it should be used to write a scientific paper. Here is the last sentence in its response.
While ChatGPT can be a useful tool in the writing process, it should not be relied upon as the sole source of information or analysis in a scientific paper.
This is a mostly accurate statement, generated entirely by ChatGPT. The only error is the use of the words “the sole” in place of “a”.
OpenAI (2020), "Introducing ChatGPT", Published on 30 November 2022; Last accessed on 01 March 2023, https://openai.com/blog/chatgpt