# data[].attributes.body
As AI technology becomes more prevalent in education, what are some of the specific facilitators and potential drawbacks of AI in the writing curriculum in terms of instructional quality and student development?

# data[].attributes.body
Artificial intelligence can detect and correct grammatical and spelling errors.

# data[].attributes.body
Artificial intelligence can provide a great deal of inspiration and information for writing.

# data[].attributes.body
Artificial intelligence technology may have some negative effects on writing. Over-reliance on AI-assisted learning systems may lead to a decline in students' ability to think and solve problems.

# data[].attributes.body
Artificial intelligence is not perfect at writing and cannot generate accurate content when dealing with complex areas.

# data[].attributes.body
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10508892/

# data[].attributes.attachment_link.readable_text
Auro Del Giglio, Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Project administration, Validation, Writing - original draft, Writing - review & editing 1 and Mateus Uerlei Pereira da Costa, Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Project administration, Validation, Writing - original draft, Writing - review & editing 2 ,*

SUMMARY

OBJECTIVE:

Scientific writing in English is a daunting task for non-native English speakers. The challenges of writing in a foreign language are evident in the scientific literature where texts by non-native English-speaking scientists tend to be less clear and succinct, contain grammatical errors, and are often rejected by prestigious journals.

METHODS:

We conducted a non-systematic review of the most recent literature using the terms "Artificial Intelligence," "Scientific Writing," and "Non-English Speaking" to create a narrative review.

RESULTS:

Artificial intelligence can be a solution to improve scientific writing, especially for non-native English-speaking scientists. Artificial intelligence can assist in the search for pertinent scientific papers, generate summaries, and help with the writing of different sections of the manuscript, including the abstract, introduction, methods, results, and discussion. Artificial intelligence-based programs can correct grammatical errors and improve writing style, both of which are particularly helpful for non-native English speakers. Two artificial intelligence programs that can help with the search for pertinent scientific papers on the internet are Elicit and ResearchRabbit. Scispace Copilot can be used to summarize the retrieved reference. The artificial intelligence software programs such as Grammarly and Paperpal can correct grammatical and spelling errors, while ChatGPT can also restructure sentences and paragraphs, reword text, and suggest appropriate words and phrases.

CONCLUSION:

Overall, artificial intelligence can be an effective tool to improve the clarity, style, and coherence of scientific writing, helping non-native English-speaking scientists to communicate their research more effectively.

Keywords: Artificial intelligence, Language, Writing, medical, Internet

INTRODUCTION

Most scientific papers in the world literature are written in English 1 by non-native English speakers 2. Non-native English-speaking scientists face many difficulties in writing clearly, succinctly, and without grammatical errors 3,4. Nevertheless, despite the use of word processors and spell checkers, the final text still does not compare favorably with those created by native English speakers, thus contributing to the lower chances of acceptance of these papers in prestigious scientific journals 5,6.

Artificial intelligence (AI) involves the development of algorithms and computer programs that can learn from and make predictions or decisions based on real-world data, mimicking human intelligence. Furthermore, AI systems can be trained to recognize data patterns, make predictions, and learn from experience. AI can thus perform tasks that typically require human-like reasoning and decision-making.

Natural language processing is a type of AI that enables machines to understand, interpret, and generate human language in a manner that is natural to humans 7,8. For example, ChatGPT (Generative Pre-Trained Transformer) is a language model developed by OpenAI that was designed to generate natural language responses to text prompts. In addition, ChatGPT's underlying AI system is pre-trained on vast amounts of text data from the Internet that enables it to learn the patterns and structures of languages. Therefore, it learns how to generate coherent and contextually appropriate responses to a wide range of text prompts.

With all these capabilities, the AI programs can assist scientists in the search for pertinent scientific papers on the Internet, summarize them, and help with the writing of abstracts, titles, and parts of the introduction, methods, results, and discussion. In addition, the AI programs can correct mistakes and improve the writing style of previously written texts, both of which are particularly helpful to non-native English-speaking scientists 9-12.

In this paper, we will briefly describe how AI-based programs can help with the preparation of research papers, especially considering how they can help improve scientific writing skills.

METHODS

Due to the very recent widespread access to AI-based programs such as ChatGPT, there was no enough evidence in the medical literature to pursue with a systematic review. We therefore elected to conduct a non-systematic review of the most recent literature using the terms "Artificial Intelligence," "Scientific Writing," and "Non-English Speaking" to create a narrative review using Google Scholar, Google, and AI-powered reference retrieval programs such as Elicit (www.elicit.org) and ResearchRabbit ( https://www.researchrabbit.ai/).

RESULTS

Artificial intelligence programs supporting scientific writing

Drafting a research manuscript entails several steps to be taken after the scientist has all the experimental or clinical trial results already tabulated, analyzed, and displayed in tables and/or figures. It is also important to conduct a recent and thorough review of the literature and have all the references that will be used in the paper stored in the citation software for future use. Most of the experts then suggest writing the results, followed by methodology sections 13,14. These sections are the easiest ones to write as they are readily available to the researcher when he or she becomes ready to start drafting the paper. Therefore, these sections, which represent the essence of the paper, are not going to benefit as much from AI text-generating capabilities, but they still can be much improved by AI-powered grammar and spelling correctors. Then comes the discussion section and finally the introduction, abstract, and title 13,14. These parts of a paper can be drafted with AI help, once the AI-powered program is fed with the methods and results previously written. Several authors have already outlined which are the parts that every one of the sections should have and what type of content they should include for a paper to convey all the scientific information properly and clearly as it is expected to report 15-18.

Artificial intelligence and bibliographic reference

Before even starting to write a paper, it is necessary to assemble all the pertinent references already published in the literature to learn about the field and at the same time to find still unanswered questions that can be the subject of future research. AI is particularly useful for this purpose, as it can find references related to a specific article from its content, authors, or citations and quickly build a new list of references that would otherwise be much more difficult to collect. Two AI programs can be helpful for this purpose, namely, ResearchRabbit ( https://www.researchrabbit.ai/) ( Figure 1 ) and Elicit (www.elicit.org). Both programs will suggest references many times not yet known by the scientist because they can go "beyond the horizon" by a self-learning approach based on several retrieved papers using their contents, keywords, and citations 19 to retrieve other papers.

AI can also help summarizing selected references by using both Elicit and Scispace Copilot ( https://typeset.io/). Scispace Copilot is an AI chrome extension that can also explain highlighted parts of a paper and the statistical results obtained and allows the scientist to ask his own questions regarding the text.

All the selected papers should then be stored in a reference management system (RMS). Although RMSs do not normally use AI, they are powerful tools to help in scientific writing. Examples of RMSs are Zotero (www.zotero.org) and Mendeley (www.mendeley.com), both of which are free and compatible with the most widely used word processors such as Microsoft Word. Zotero can also be used with Google Docs. Zotero and Mendeley allow the scientist to store his or her notes on each of the stored papers that could be useful later when writing the paper. Both these RMSs can import articles found with Elicit and ResearchRabbit. Many other RMSs such as Endnote and Paperpile, which are widely used by scientists, are not free. With the help of the RMS software, we can insert citations in the text as we are writing it and at the end of the paper produce a bibliography including all references that were cited in the text in a specific format such as Vancouver. Moreover, if editing changes are made later in the paper, these RMSs can automatically change the order of the references in the text and the bibliography.

Artificial intelligence and correction and improvement of english texts

Several software programs also use AI that can correct grammatical and spelling errors as well as improve the text with suggestions such as Grammarly (app.grammarly.com) and Paperpal (www.paperpal.com). Despite suggesting useful changes in the text, these programs, however, do not change the main structure of the article or generate new text. The AI tools such as ChatGPT, however, can correct spelling, punctuation, and grammatical errors; in addition, it can also summarize the text, paraphrase it, and even create a section of a paper based on another, for example, creating an abstract or an introduction based on the results of the paper. Most interesting to us is the possibility of a non-native English speaker loading his text into ChatGPT and asking the AI program to improve it so that it will not be perceived anymore as a paper written by a foreigner. ChatGPT can also write code to analyze data present in a Microsoft Excel spreadsheet or in a statistical program such as R and advise on what statistical tests to use to analyze data and help explain the obtained results.

ChatGPT works with prompts and knowing some of them as related to scientific writing may be especially useful to maximize its output toward a desired goal. For instance, before loading the text of a paper or a section of it to ChatGPT within quotation marks, we can ask the AI program to proceed with several tasks through prompts that we type in the chatbot window. For example, we can ask ChatGPT to summarize the following paper "paste the text of the paper here." Likewise, we can ask it to Create an introduction; Create a title; Create 250 words abstract; Reduce an abstract; Create a title for an abstract ( Figure 2 ); Create a discussion; and so on.

ChatGPT is a very promising resource and will certainly improve with all the technological improvements that can be foreseen in the next few years. Nevertheless, we can already point out some of the advantages and potential disadvantages that it can bring to scientific writing ( Table 1 ).

Table 1.

Benefits

Disadvantages

AI-based programs can help break mental logjams when writing by suggesting a first draft of a text. These programs can also enhance readability by breaking up a difficult topic into smaller pieces.

Incapable of understanding added information, generating novel insights and deep analysis which would limit the discussion within a scientific paper. AI tools are more adequate for regurgitating conventional wisdom than for identifying or generating unique outcomes.

Programs such as Elicit or ResearchRabbit can identify references for a specific topic that might be missed by conventional literature searches.

Superficial, and over-reliance on the output could decrease scientific creativity by reducing the impact of papers in generating real contributions to a specific knowledge field.

Be used effectively to improve your manuscript's title, abstract, and conclusion and tailor it to match the journal parameters and better match its scope or readership.

AI might be worse at assessing whether a unique outcome is spurious or ground-breaking. Therefore, reliance upon AI for this purpose will reduce the frequency of future disruptive scientific breakthrough.

Help a writer be more thorough when covering a topic by reminding them of aspects they had not considered and by providing knowledge in areas not yet familiar to the author in an easy and understandable way.

AI-based programs may harm the development of writing skills by young researchers and facilitate plagiarism.

Facilitating composition by non-native English speakers and decreasing inequalities existing with native-speaking researchers. The language resources of AI will certainly be included soon directly in other interfaces, such as Microsoft Office 365.

Develop code for Python and other programs such as R and Excel. This code-generating ability can help in the analysis of statistical data and in creating figures to represent experimental results.

Ethical considerations

AI can help the scientist in these steps differently. In fact, at present, AI can be an accessory, a facilitator, but should never be expected to write any of the sections of a scientific paper as an author would. As amazing as it can be that AI-generated output can be rapidly produced, grammatically correct, and sometimes even insightful in terms of its content, several errors can occur that limit the usefulness of AI-generated texts without due correction and editing 10.

ChatGPT cannot be considered a coauthor in a scientific paper. Authorship requires contributions to the conception, design, analysis, interpretation, and writing and approval of a definitive version of the manuscript. Furthermore, authorship also requires that authors take public responsibility for the content of the paper 20. As ChatGPT cannot be responsible for what it writes, the responsibility regarding the content of a scientific paper rests completely upon all the human coauthors who must check and edit all the generated content.

Nevertheless, it is the authors' opinion that the editorial help provided by the AI-powered software programs such as ChatGPT can be of use to improve their writing. Authors need, however, to properly disclose at the end of the paper in which of the article's sections the AI-powered software programs were used and for what reasons. In addition, it is important to state that AI-generated output was duly checked by the authors. Acknowledgment regarding the responsibility for all the paper content needs to be explicitly recognized by all the authors as well. We believe that, if all these precautions are taken, AI can be safely and productively used by non-native English-speaking researchers to improve their scientific writing 21,22. At the end of this paper, we added a paragraph based on a model suggested by Elsevier Company 21 to acknowledge the use of AI in this article that may be used by other researchers as well.

Another ethical concern with AI-generated texts is the potential for plagiarism 22. Plagiarism detectors exist 23 to help in identifying texts or parts of texts that were copied from other sources. Unfortunately, however, automatic rephrasing of paragraphs can potentially elude plagiarism detection 24. A potential problem with this technology that can circumvent plagiarism detection is the proliferation of papers with few real contributions to add to the overall knowledge that may further contribute to the relative decrease in scientific breakthroughs we are currently observing 25.

CONCLUSION

Writing scientific articles in English is a challenging task for non-native speakers. However, with the help of the AI tools, non-native English speakers can improve the clarity, style, and coherence of their writing. The AI-based programs can assist with various aspects of the scientific writing process, including searching for relevant papers, generating summaries, correcting grammatical errors, improving writing style, and creating bibliographies. Elicit, ResearchRabbit, Scispace Copilot, Zotero, Mendeley, Grammarly, Paperpal, and ChatGPT are some of the AI tools that can be particularly helpful for non-native English-speaking scientists. Using the AI-powered software programs can not only enhance the quality of scientific writing but also help researchers produce papers with higher chances of publication in major scientific journals and thus help effectively communicate their research to a wider audience.

Artificial intelligence disclosure

During the preparation of this work, the authors used CHATGPT to generate a draft of the abstract and conclusion sections. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

REFERENCES

23. ZeroGPT Chat GPT, open AI and AI text detector free tool [Internet] [[cited on Apr 23, 2023]]. Available from: https://www.zerogpt.com/

Articles from Revista da Associação Médica Brasileira are provided here courtesy of Associação Médica Brasileira

# data[].attributes.attachment_link.title
The use of artificial intelligence to improve the scientific writing of non-native english speakers

# data[].attributes.attachment_link.description
Scientific writing in English is a daunting task for non-native English speakers. The challenges of writing in a foreign language are evident in the scientific literature where texts by non-native English-speaking scientists tend to be less clear and ...

# data[].attributes.body
https://www.sciencedirect.com/science/article/pii/S2666920X22000108

# data[].attributes.attachment_link.description
About ScienceDirect
Shopping cart
Contact and
supportTerms and
conditionsPrivacy policy

# data[].attributes.body
https://hbr.org/2022/11/how-generative-ai-is-changing-creative-work?utm_medium=paidsearch&utm_source=google&utm_campaign=domcontent_bussoc&utm_term=Non-Brand&tpcc=domcontent_bussoc&gad_source=1&gclid=Cj0KCQjwsPCyBhD4ARIsAPaaRf3k0cs_up6QxZQ_2U2W6RF-Xu0bmfuGDo6eHgk_wa3j-vWvXw_nX9AaAupdEALw_wcB

# data[].attributes.attachment_link.readable_text
Generative AI models for businesses threaten to upend the world of content creation, with substantial impacts on marketing, software, design, entertainment, and interpersonal communications. These models are able to produce text and images: blog posts, program code, poetry, and artwork. The software uses complex machine learning models to predict the next word based on previous word sequences, or the next image based on words describing previous images. Companies need to understand how these tools work, and how they can add value.

Large language and image AI models, sometimes called generative AI or foundation models, have created a new set of opportunities for businesses and professionals that perform content creation. Some of these opportunities include:

# data[].attributes.attachment_link.title
How Generative AI Is Changing Creative Work

# data[].attributes.attachment_link.description
Generative AI models for businesses threaten to upend the world of content creation, with substantial impacts on marketing, software, design, entertainment, and interpersonal communications. These models are able to produce text and images: blog posts, program code, poetry, and artwork.

# data[].attributes.body
https://www.tandfonline.com/doi/full/10.1080/2331186X.2023.2236469

# data[].attributes.attachment_link.readable_text
Abstract

The primary objective of this study was to examine the range of available Artificial Intelligence (AI) writing tools and assess their influence on student writing, particularly in terms of content and organization, as perceived by English as a Foreign Language (EFL) teachers. Utilizing a qualitative approach, the research was constructed within a case study design. The data was collected via semi-structured interviews, targeting information about the diversity of AI writing tools and their impact on students" writing quality. The study gathered data from four EFL teachers across three distinct universities in Indonesia, shedding light on the variety of AI writing tools used in their classrooms. These included applications like Quillbot, WordTune, Jenni, Chat-GPT, Paperpal, Copy.ai, and Essay Writer. Furthermore, these teachers unanimously agreed that the AI writing tools positively improved their students' writing quality, particularly enhancing the quality of their content and organization. The findings of this study imply that integrating AI writing tools can prove beneficial in elevating the quality of EFL student writing. In response to this study's limitations, recommendations for future research were also addressed.

PUBLIC INTEREST STATEMENT

A recent study conducted in Indonesian universities revealed the positive influence of Artificial Intelligence (AI) writing tools on English as a Foreign Language (EFL) student writing. Interviews with four EFL teachers across three universities highlighted the diverse range of AI writing tools, including Quillbot, WordTune, Jenni, Chat-GPT, Paperpal, Copy.ai, and Essay Writer. Teachers unanimously agreed that these tools improved students' writing quality, particularly in terms of content and organization. The study's implications suggest that integrating AI writing tools can elevate the quality of EFL student writing, enhancing language learning outcomes. These findings have broader implications for educators, students, and policymakers interested in optimizing language instruction through technology.

1. Introduction

The use of Artificial Intelligent (AI)-powered writing tools in English as a foreign language (EFL) classroom is increasing rapidly. These tools include grammar checks, writing aids, and programs that can create written works like essays without human help. They are easy to use and effective, saving students and educators time and effort (T. S. Chang et al., Citation2021; Gayed et al., Citation2022; Jeanjaroonsri, Citation2023; Zhao, Citation2022). Additionally, AI writing tools have been used particularly for EFL learners with low English proficiency. By using these tools, students can receive immediate feedback and assistance, improving their writing skills faster.

In general, AI writing tools are intended to analyze written materials and provide comments on many parts of the writing, such as grammar, vocabulary, syntax, content, and structure (Hosseini et al., Citation2023; Strobl et al., Citation2019; Thorp, Citation2023). Such feedback is created by machine-learning algorithms that compare the written text to a big database of correct and incorrect instances of writing. In other words, EFL students can obtain instant and tailored feedback on their writing, which can assist them in identifying and correcting faults more rapidly. This real-time feedback can also assist students in comprehending the fundamental concepts of successful writing and offer them with direction for enhancing their writing abilities (Akgun & Greenhow, Citation2022; Nazari et al., Citation2021; Rudolph et al., Citation2023).

Recently, there is a growing amount of research on how AI writing tools affect student writing skills, and this research is relevant to the TEFL enterprise. Some research suggests that adopting AI-powered writing tools can help students improve their writing (e.g., Kurniati & Fithriani, Citation2022; Wang, Citation2022; Zhao, Citation2022), while others have voiced concerns about the side effect of these tools (e.g., Y. Liu et al., Citation2022; Lund & Wang, Citation2023; Qadir, Citation2022). Nevertheless, the literature presented focuses solely on the role of AI writing tools in improving grammar and syntax. While this is undeniably a crucial aspect, it does not address the full spectrum of what contributes to effective writing. The content and organization of a piece of writing are equally significant components that profoundly affect the reader's understanding and engagement (I. Lee & Yuan, Citation2021). Content is the core of any piece of writing. It carries the ideas, thoughts, and messages that the writer wants to convey. It is the "what" of the writing-the substance. High-quality content is relevant, informative, original, and meaningful, resonating with the audience and fulfilling its purpose-whether that be to inform, persuade, entertain, or provoke thought (Molina et al., Citation2021). Organization, on the other hand, is the "how" of the writing-the structure. It involves how the content is arranged, connecting ideas in a logical, coherent manner that guides the reader through the piece. The organization can significantly affect how well the reader understands the content, how persuasively an argument is presented, or how engaging a narrative is. Good organization ensures a smooth flow of ideas, making the writing easier to follow and more impactful (Awada et al., Citation2020). Therefore, the focus on just grammar and syntax overlooks the critical roles of content and organization in writing. While getting the words right is essential, what those words express and how they are structured are just as vital to effective communication. Thus, it is crucial to explore what the types of AI writing tools and how the tools address these other key aspects of writing from the perspective of EFL educators.

A similar situation holds true in the Indonesian context. To the best of our knowledge, there is almost no research in the Indonesian context that has been reported in journal articles examining the impact of AI writing tools on the content and organization from either students' or teachers' perspective. The present authors are aware of the seminal work of Miranty and Widiati ( Citation2021) and Fahmi and Cahyono ( Citation2021), which, to some degree of details, examine the use of Automated Writing Evaluation (AWE) as one of the AI writing tools employed by Indonesian EFL students in their writing. Miranty and Widiati ( Citation2021) explored students' perceptions of the merits and drawbacks of AWE (i.e., Grammarly), while Fahmi and Cahyono ( Citation2021) aimed to investigate differences in students' perceptions towards the use of AWE based on their proficiency levels. Their research findings suggest that using AWE can provide students with positive benefits, which were consistent across their studies. Despite these insightful findings, more research on the impact of AI writing tools is necessary. It is true the previous studies have provided information regarding the influence of these tools on student writing. However, it is still unclear how AI writing tools affect the quality of student writing particularly on the aspects of content and organization. Consequently, this study has been meticulously designed to fill the prevailing knowledge gap, thereby delivering a more comprehensive understanding of AI writing tools' utility in EFL classrooms. The primary objective of this research is to explore the variety of AI writing tools and evaluate their impact on student writing, particularly focusing on content and organization, through the lens of EFL teachers. To this end, we posed the following critical research questions:

1. What types of AI writing tools are used by EFL teachers to improve students' writing quality, particularly in terms of content and organization?

2. How do EFL teachers perceive the impact of AI writing tools on the content and organization aspects of students' writing?

The urgency in answering these research questions stems from two primary reasons. Firstly, understanding the types of AI writing tools used by teachers and their methods of integration could provide a blueprint for other educators looking to implement similar technologies in their teaching practices. This insight can ultimately contribute to the broader and more effective use of AI in EFL instruction. Secondly, investigating teachers' perceptions of AI writing tools is crucial. If teachers view these tools as beneficial, this positive perception can lead to more widespread adoption, enriching EFL instruction across the board. However, if teachers are apprehensive or skeptical about these tools, identifying and addressing their concerns becomes paramount. This understanding would allow for modifications in AI tools or teaching strategies to ensure the potential of AI is fully realized and appreciated in language learning settings. By answering these questions, this study hopes to pave the way for more effective AI integration in EFL classrooms.

2. Literature review

2.1. Teaching writing in the artificial intelligence era

The evolution of teaching writing in the current age of technological proliferation cannot be dissociated from the rapid advancement of digital tools and. As eloquently discussed in Haleem's et al. ( Citation2022) research, this shift is precipitated by an increasing fusion of digital utilities within the educational realm, effectively transforming traditional pen-and-paper methodologies into far more creative and dynamic pedagogical experiences. Garlinska et al. ( Citation2023) discuss how virtual classrooms, online workshops, and cloud-based writing tools, for example, are revolutionizing the domain of writing instruction. These platforms offer functionalities such as real-time feedback, collaborative editing, and plagiarism checks. Such features not only enhance the writing acumen of students, as per Nykyporets's ( Citation2023) study, but also inspire them to engage in critical thinking and independent reasoning.

Furthermore, as illustrated by Bhutoria's ( Citation2022) work, AI-driven platforms and applications offer personalized learning opportunities for students, identifying their writing strengths and weaknesses. This empowers educators to tailor their teaching strategies to each student's unique needs and preferences, leading to more effective learning outcomes (Dogan et al., Citation2023). In the same vein, Cahyono et al. ( Citation2023) explored of the workable approach mediated by mobile technology in teaching writing uncovers another layer of tech-enabled pedagogical innovation. Students are emboldened to publish their work in public forums, nurturing their confidence and writing abilities. Such platforms also promote peer review and feedback mechanisms, fostering a sense of community and collaborative learning (Umamah & Cahyono, Citation2022). However, as the literature acknowledges, the digitization of writing instruction is not without its challenges. As Duncan and Joyner ( Citation2022) articulated, educators must grapple with issues related to digital equity, privacy, and the potential for distraction. These issues underscore the necessity for an ongoing dialogue and proactive approaches in crafting pedagogical policies and strategies as we continue to navigate the teaching of writing in this artificial intelligence era.

2.2. The impact of AI writing tools on students writing quality

There has been burgeoning research on the impact of AI writing tools on students' writing ability. While several studies indicate positive outcomes, others highlight potential negative impacts. On the positive side, AI writing tools such as Grammarly, QuillBot, Wordtune, and Jenni have been found to significantly improve students' writing skills. These tools utilize advanced algorithms to identify common errors in grammar, punctuation, and syntax and provide suggestions to improve clarity and style. They also offer unique capabilities such as paraphrasing and refining sentences for enhanced effectiveness. A study by Tambunan et al. ( Citation2022) showed that using Grammarly improved students' grammar and punctuation. Grammarly's AI analyzes the user's text, provides real-time suggestions for enhancing grammar, spelling, punctuation, clarity, engagement, and delivery. This has the effect of turning the process of writing into a learning opportunity. QuillBot, another AI tool, specializes in paraphrasing, helping students avoid plagiarism while maintaining the original meaning of their content. Kurniati and Fithriani ( Citation2021) found that QuillBot aided students in developing better paraphrasing skills, which is a critical competency in academic writing. WordTune, on the other hand, focuses on refining and improving the tone and style of the text. A study by Lam and Moorhouse ( Citation2022) found that WordTune effectively helped students identify their writing weaknesses, thereby promoting self-assessment and learning. This tool goes beyond simple grammar correction and delves into the stylistic elements of writing. Jenni is another AI-based writing assistant that provides predictive text suggestions to help users compose emails, reports, articles, and other documents more efficiently. Although no specific research was cited in this regard, user feedback suggests a positive impact on writing productivity and creativity. GPT-3, developed by OpenAI, represents a significant advancement in language model technology. Capable of generating coherent and contextually relevant sentences, GPT-3 has been found to stimulate students' creative and critical thinking (Mhlanga, Citation2023). It can be an excellent tool for students to experiment with different writing styles and ideas. These studies highlight the potential of AI tools in enhancing the writing abilities of students.

In terms of the content and the structure of writing, a number of research has shown that AI writing tools can significantly enhance the efficiency and quality of writing tasks. AI-based tools, like GPT-3 and its successor GPT-4, have been leveraged to suggest the next word or paragraph in a text, aiding authors in creating human-like compositions (Y. Liu et al., Citation2023). This process is facilitated by machine learning algorithms that predict text sequences based on the vast amount of data they have been trained on. Furthermore, there is ongoing research exploring the impact of these tools on different aspects of writing. For instance, a study found that AI writing tools like "AI KAKU", a popular Japanese-developed English language learning tool, could potentially reduce cognitive barriers for English as a Foreign Language (EFL) learners when producing written text in English (Gayed et al., Citation2022). This is achieved by providing real-time translation and grammar suggestions, making the writing process less daunting. Another important application of AI in writing is its ability to expedite the literature review process, which deals with large volumes of partially structured (meta)data (Wagner et al., Citation2022). This is particularly valuable in fields like healthcare and social sciences, where researchers must often sift through enormous bodies of literature for their studies. In conclusion, the evolution and application of AI writing tools like have undoubtedly marked a significant stride in the world of writing and literature. These advancements hold substantial potential in not only streamlining the writing process but also in improving the quality and efficiency of written output.

Despite these advantages, certain studies have indicated potential drawbacks. For instance, AI writing tools may inadvertently promote over-reliance among students, as they might lean too heavily on these tools for correction without thoroughly understanding their mistakes. This dependency can stunt their natural learning process and development of self-editing skills. Iskender ( Citation2023) provided a critical perspective on this matter, arguing that the use of AI writing tools could lead to diminished critical thinking skills if students become overly dependent on them. They expressed concern that students might prioritize quick fixes from AI tools over deeply understanding and learning from their mistakes, which would essentially negate the process of learning, growth, and development in writing. In addition to concerns about over-reliance, the impact of AI writing tools on creativity has also been questioned. Some educators worry that students might use these tools not just for refining their language, but also for generating ideas, potentially curtailing their creative thinking and originality (Johinke et al., Citation2023). Furthermore, Farrokhnia et al. ( Citation2023) pointed out that while AI tools can improve certain writing skills, they may not be as effective in addressing higher-order writing elements, such as argument structure and coherence. These aspects require a deep understanding of the topic, logical thinking, and the ability to connect ideas, which are currently beyond the capacity of AI tools. In terms of tone and context, AI tools may also falter. These tools might not fully grasp the subtleties and nuances of human language and emotion, potentially leading to suggestions that may not be appropriate or accurate in a given context (Haleem et al., Citation2022). Finally, issues of digital equity have been raised. Not all students have equal access to these technologies, leading to potential disparities in learning outcomes. In their study, Mozumder et al. ( Citation2023) noted that students without access to high-speed internet or modern digital devices might be unable to utilize these AI tools, widening the educational achievement gap. Therefore, while the application of AI in writing instruction has promising benefits, educators need to be mindful of these potential drawbacks and address them proactively in their teaching strategies.

3. Method

3.1. Research design

To address the research questions, a qualitative approach was employed, specifically utilizing a case study methodology. The case study methodology, which typically involves conducting interviews and/or observations, enables the researcher to uncover important factors that emerge from the open-ended process (Yin, Citation2009). Through this methodology, we were able to explore in-depth the types of AI writing tools used by EFL teachers, as well as their perceptions about the impact of these tools on students' writing quality, particularly in terms of content and organization.

3.2. Research participants

This study focused on gaining insights from EFL teachers experienced with AI writing tools, specifically their views on how these tools affect student's writing quality, particularly in terms of content and organization. Four writing teachers were recruited from three different universities, one being a public institution and the other two, private. The selection was primarily based on their practical experience with AI writing tools in the classroom, accessibility, willingness to participate, and teaching expertise in advanced courses such as essay writing and argumentative essay writing. These advanced courses often demand a higher level of grammatical knowledge and a deeper understanding of discourse structure.

All the recruited teachers possessed at least three years of experience in teaching EFL and a minimum of one year's experience in applying AI writing tools in their classrooms. These AI tools served as supplementary teaching resources, aiding in grammar checking, paraphrasing, plagiarism detection, generating content, and offering suggestions to enhance the clarity and coherence of students' writing. This ensured the teachers' experiences transcended mere theoretical knowledge, encapsulating practical applications of AI tools in real-life teaching scenarios.

Teachers' classes varied in size, ranging from intimate, personalized instruction settings to larger, conventional classrooms. This range in teaching environments was intentional, allowing the study to account for various factors potentially impacting the efficacy of AI tools. These factors could include class size, available institutional resources, the composition of the student population, and individual teaching styles. Such a diverse selection of participants enabled a comprehensive understanding of EFL teachers' experiences and perspectives while using AI writing tools.

Participants were designated as T1, T2, T3, and T4 to maintain confidentiality. The recruitment process began with contacting prospective participants via email, outlining the study's purpose, their roles, data collection methods, and their rights, including the ability to withdraw from the study at any point without any repercussions. This ensured adherence to ethical research guidelines. They were also provided with an informed consent form to read, comprehend, and sign if they agreed to participate. Through this careful recruitment process, the study maintained the highest ethical standards while ensuring the inclusion of relevant experiences and perspectives. The demographic information of the participants is presented in Table 1.

3.3. Instrument

We used interviews as our research instrument to collect the required data for this study. This instrument allowed us to gather rich, qualitative data related to EFL teachers' use of AI writing tools, and their perspectives on the impact of these tools on students' writing quality- specifically, in terms of content and organization. To construct pertinent interview questions, we turned to several relevant pieces of literature. Works by Nazari et al. ( Citation2021), Dale and Viethen ( Citation2021), M. Lee et al. ( Citation2022), and Zhao ( Citation2022) served as valuable references. By undertaking an in-depth review of these sources, we focused on the topics and themes that aligned with our research objectives.

This thorough literature review helped us craft our interview protocol. It included specific questions such as "What AI writing tools have you used in your classroom?" and "In what ways have these tools impacted your students' writing, specifically with respect to content and organization?" This interview guide allowed us to investigate both the practical applications and theoretical implications of AI writing tools in the EFL teaching context.

3.4. Data collection procedure

After designing the interview protocol centered on the types of AI writing tools utilized by EFL teachers and their views on the impact of these tools on student writing quality (specifically, content and organization), we organized interviews with each participant. These were scheduled according to their convenience and availability. Before starting each interview, we briefed the participants about the study's objectives, the interview process, and our rigorous approach to data protection. We secured their informed consent to proceed, ensuring they comprehended their rights, including the freedom to withdraw from the study at any time. During the interview, we began with general questions about their experiences teaching EFL and using AI writing tools, thus fostering a relaxed environment conducive for open discussions. Gradually, we introduced more specific questions aligned with our interview protocol, delving into the particular AI writing tools they had employed and their impact on students' writing quality, particularly in terms of content and organization improvements.

To address potential social-desirability bias inherent in self-reported data, we emphasized to the participants that there were no right or wrong answers and encouraged them to be as open and honest as possible. This was done to ensure the reliability and authenticity of the responses gathered. Each interview was recorded with the participant's permission, to maintain accuracy in capturing their responses, an integral part of reliable transcription and subsequent data analysis. Note-taking during the interviews was also employed as a means to highlight significant points, capture nuanced details, and document immediate observations.

Upon the conclusion of each interview, we expressed our gratitude to the participants for their contributions, also providing them with an opportunity to pose any questions about the study or to add any further comments. Following the interviews, we carefully transcribed the recorded dialogues, transforming verbal insights into textual data. Accuracy during this stage is critical, as the transcribed data underpin our subsequent analysis. All collected data, encompassing the interview recordings, transcriptions, and notes, were securely stored, respecting participant privacy and enabling efficient data management for ensuing stages of data analysis and research reporting. Adhering to these practices ensured a comprehensive, ethical, and effective data collection procedure for our research on AI writing tools' use and impact in EFL teaching.

3.5. Data analysis

We employed a thematic case study to analyze the qualitative data. As outlined by Braun and Clarke ( Citation2006), the goal of this thematic analysis was to identify patterns and connections among themes extracted from the qualitative data, thereby addressing our research questions. We divided the analysis into five distinct steps.

In the first step, we thoroughly read the interview transcript multiple times to familiarize ourselves with its content. The second step involved developing preliminary codes that grouped related words and phrases in line with our research criteria. Following a detailed analysis of these codes, we then categorized each one. In this step, we also compared the results to identify any relationships or patterns. Finally, we established a set of categories from these grouped codes.

4. Findings

4.1. Types of AI writing tools use by the EFL teachers

Drawing from the data collated during the interview phase of our research, it was evident that the four EFL teachers under study employed a range of AI writing tools in their teaching practice. These tools were chosen and used based on their distinctive features and capabilities, aligning with the unique demands of their teaching environments and objectives. To provide a detailed yet succinct overview of these AI writing tools, we have organized this data into a clear and informative format. Refer to Table 2, which serves as a comprehensive catalog, capturing the variety of AI writing tools adopted by these EFL teachers. This table effectively highlights the names of the AI tools in use and their specific purposes, providing a comprehensive snapshot of the current landscape of AI utilization in EFL teaching.

The provided dataset in Table 2 reveals a notable pattern of how four teachers (T1, T2, T3 and T4), integrated Artificial Intelligence (AI) tools in their teaching methodologies. These teachers had shared a common interest in utilizing a combination of AI tools in enhancing students' writing skills, hinting at the fundamental trend of embedding technology into education.

Two tools, QuillBot and Wordtune, had emerged as universally used across all four teachers. QuillBot, known for its proficiency in paraphrasing, was an important part of teaching students how to express the same idea in different ways, effectively aiding in the avoidance of plagiarism. Wordtune, on the other hand, was employed to improve sentence structure and clarity, making it an essential tool in teaching English and enhancing students' written expression.

A second layer of commonality could be seen with Jenni AI and ChatGPT, which were employed by three out of the four teachers (T1, T3, and T4). Jenni AI, renowned for its topic modeling capabilities, assisted students in brainstorming and organizing their thoughts prior to writing, thereby encouraging critical thinking and structured writing. Chat-GPT simulated conversations, assisting students in improving their conversational writing skills, which was a crucial aspect of English learning in a globalized and digitally connected world.

While the aforementioned tools were the common threads woven into the teaching methodologies of these teachers, they also diversified their approach with specific tools unique to their classrooms. For instance, T2 used Copy.ai and Paperpal. Copy.ai generated creative writing samples, serving as examples for different styles of writing and thus, fostering creativity in students. Paperpal was utilized for its strength in grammar checks and providing real-time corrections, which was a vital practice in English learning. T3 employed Essay Writer, which helped students structure their essays logically and provided them with ideas for improvements, a tool that seemed uniquely suited to a classroom focused on academic writing. T4 also brought in Copy.ai and Essay Writer, leveraging these tools to show students how to craft compelling pieces of writing in various styles and formats.

When considering the demographic information of participants given in Table 1 and analyzing it against the teachers' utilization of AI tools, several initial observations become apparent. It appears that the teachers with more years of experience in using AIWT, T2 and T3, utilize a broader set of AI tools. Notably, T2, who is the oldest, holds the highest qualification (a Ph.D.) and has the most teaching experience, utilizes unique tools like Copy.ai and Paperpal. Similarly, T3, who has comparable AIWT experience, also uses a unique tool, Essay Writer. This pattern suggests that more seasoned and experienced teachers in using AIWT may exhibit greater confidence in exploring diverse AI tools for varying educational purposes. While it is challenging to conclude any significant gender-based differences in AI tool usage from this dataset, it is worth mentioning that the only female teacher, T2, uses a unique set of tools that could be more attributable to her experience and qualifications than her gender. Furthermore, it is worth noting that the level of teaching experience and qualifications may influence the use of AI tools. T2, having the highest teaching experience, experiments with unique tools, suggesting that more experienced teachers could be more exploratory. Similarly, the teacher with a Ph.D. degree, T2 again, utilizes unique AI tools, hinting that the level of education might have some influence on the choice of AI tools. However, given the small dataset, these observations need to be treated as preliminary trends rather than definitive conclusions.

4.2. Teachers perspective toward the impact of AI writing tools

In this section, we delved into the findings from our interviews regarding the teachers' perspectives on the impact of AI writing tools on students' writing quality, specifically focusing on the aspects of content and organization.

4.2.1. Impact on the aspect of content writing

When examining the aspect of content, we identified two major sub-themes: idea generation and vocabulary and language use.

4.2.1.1. Idea generation

The responses from four teachers (T1, T2, T3, & T4) highlighted a wide range of opinions among them regarding the impact of AI writing tools on students' creativity and ability to generate ideas. T1, for example, highlighted the potential benefits of AI writing tools, emphasizing their ability to stimulate creativity and idea expansion. According to him, these tools could offer various angles or perspectives when students were struggling with writer's block, thereby helping them overcome creative obstacles. The teacher's reply is as follows:

Conversely, T2 expressed a more mixed evaluation of AI writing tools. On one hand, she agreed with T1 that these tools could provide a kick-start when students grappled with idea generation. However, she also voiced a concern that the ideas produced by these tools could be rather generic or impersonal. In her view, while AI tools could assist in certain situations, they might not always promote deep or critical thinking about a topic. The teacher respond in the following manner:

In contrast to the previous views, T3 brought a different perspective, suggesting that AI writing tools could potentially hamper students' ability to generate original ideas. He posited that students might become overly reliant on the suggestions provided by the tool, limiting their willingness to think creatively or innovatively. He argued that struggling with idea generation and problem-solving might be beneficial in the long run as these were key skills that could be honed over time. The teacher offered the following response:

Lastly, T4 re-emphasized some positive aspects of AI writing tools, notably their capacity to aid students in developing their ideas further. He noted that these tools could offer phrases or sentences that enhanced an initial idea, showing students various ways to express their thoughts more effectively. Moreover, he stated that AI writing tools could demonstrate how an idea could be developed and supported with evidence or arguments, which he believed was especially beneficial in academic writing. The teacher response as follows:

In summary, these responses suggested a spectrum of views on the influence of AI writing tools on students' creativity and idea generation, underscoring both potential benefits and drawbacks. While some teachers viewed these tools as useful aids for overcoming creative blocks and developing ideas, others warned against over-reliance and the risk of limiting critical thinking and personal creativity. The overall message underscored the need for a balanced use of AI tools, fostering both creative assistance and independent thought.

4.2.1.2. Vocabulary and language use

In relation to the vocabulary and language use, each teacher had drawn attention to a specific aspect, while acknowledging both benefits and potential downsides. T1, for instance, had underscored the positive influence of AI writing tools on enhancing students' vocabulary. He suggested that these tools recommended synonyms or more advanced words that students might not have considered, thereby expanding their lexical repertoire and enabling them to express their ideas more precisely and engagingly. Over time, he believed this led to more advanced and polished writing.

T2, however, had introduced a note of caution. She had expressed worries that AI writing tools could sometimes result in overcomplicated language. While she acknowledged the potential of these tools to provide students with a more complex vocabulary, she voiced concerns that students might use these words without fully understanding their context or connotations, which could lead to awkward or excessively formal writing. She emphasized that while expanding vocabulary was beneficial, the primary goals should always be clarity and precision. The transcript of the interview is provided below:

T3 presented a more balanced view, acknowledging the usefulness of AI writing tools as a learning resource for expanding vocabulary. However, he also cautioned against an over-reliance on these tools. He expressed concerns that students might not make an effort to recall and use the new vocabulary independently. Further, he warned that this could potentially slow students' vocabulary acquisition over the long term. Emphasizing the need for balance, he stressed the importance of using these tools as a supplementary aid rather than a crutch. The excerpt from the interview is displayed below:

Lastly, T4 took the conversation in a slightly different direction, focusing on the role of AI writing tools in reducing repetition in students' writing. He argued that these tools could suggest varied vocabulary to replace frequently used words or phrases, thereby enhancing the quality of their writing and prompting students to incorporate new vocabulary into their work.

4.2.2. Impact on writing organization

As for the aspect of organization, we also distinguished two prominent sub-themes: "Coherence and Logical Flow" and "Use of Transition Words and Phrases". "Coherence and Logical Flow" explores how AI writing tools impact students' ability to construct logical and cohesive arguments or narratives. Meanwhile, "Use of Transition Words and Phrases" centers on whether these tools assist students in effectively using transitional language elements to link ideas, thus enhancing the fluidity of their writing.

4.2.2.1. Coherence and logical flow

The responses of the four teachers presented a comprehensive picture of the benefits of AI writing tools in enhancing students' writing skills. Each teacher provided insights into a distinct facet of these tools and their impact on the writing process.

Firstly, T1 underscored the overall positive impact of AI writing tools, emphasizing their role in fostering a clear, logical flow of thoughts. T1 noted how these tools helped in suggesting better phrasing, removing redundancies, and enhancing coherence, which subsequently improved students' confidence in their writing abilities. The teacher's response is presented below.

Secondly, T2 offered a balanced perspective, reflecting both on the positives and potential downsides of these tools. T2 highlighted the assistance provided by AI in organizing thoughts, structuring arguments, and identifying logical gaps. Yet, T2 also raised concerns about possible over-reliance on these tools, which could have inadvertently led students to overlook the development of their own critical thinking and problem-solving skills, fundamental to effective writing. The pieces of their interview is presented as follow:

Next, T3 built upon these insights by focusing on the instant feedback provided by AI writing tools. According to T3, these tools served as an invaluable aid in structuring arguments, identifying logical inconsistencies, and encouraging clear communication. The ability to correct and learn from mistakes in real-time reinforced students' understanding of good writing practices and contributed significantly to their writing skill improvement over time. The response is presented below:

Lastly, T4 highlighted the role of AI tools as personalized writing tutors. This teacher emphasized their effectiveness in providing grammar suggestions, enhancing vocabulary usage, and ensuring logical continuity of arguments. In T4's perspective, the AI writing tools had been instrumental in elevating students' writing capabilities, particularly in creating logically coherent and flowing pieces of writing. The interview transcript is presented below.

4.2.2.2. Use of transition words and phrases

The four teachers' statements are all in agreement regarding the impact of AI writing tools on students' use of transition words and phrases in their writing. They all stress the significant improvement in the quality of writing resulting from the adoption of such AI tools.

T1 had spoken positively about the impact of AI writing tools on students' writing, emphasizing an observed improvement in the use of transition words and phrases. They highlighted the educational nature of the tools, comparing them to an "interactive tutorial" that had helped students better understand the function and application of transitional elements in text. This suggested that T1 had seen AI tools as effective complementary educational resources. A portion of the interview transcript can be found below:

Following this, T2 also recognized the benefits of AI writing tools in enhancing the use of transitional elements. They mentioned improvements in sentence flow and cohesion. However, differing slightly from T1, T2 offered a more nuanced perspective by raising potential concerns regarding the dependency on these tools. They warned that over-reliance might impact students' creativity and unique writing styles. This had suggested that while T2 appreciated the benefits of the tools, they also recognized the need for balanced use. The transcript the interview can be seen below.

Subsequently, T3 emphasized the immediacy of the impact that AI writing tools could have had on student writing. They spoke about how the tools offered real-time prompts that helped students incorporate transitional elements while their thoughts were still fresh, contributing to improved organization and clarity. T3 s perspective seemed to focus on the capacity of AI writing tools to promote active and timely learning. The interview transcript is presented below.

Finally, T4 provided a before-and-after comparison, which implied a significant improvement in students' writing after the introduction of AI tools. He spoke to the tools' potential in facilitating smoother transitions between ideas and improved coherence and organization in writing. This suggested T4 perceived the AI tools as beneficial for mitigating common writing issues and enhancing overall writing quality. The teacher's response is presented below.

In conclusion, all the responses were largely positive about the impact of AI writing tools on students' writing, although they each focused on slightly different benefits. However, T2 had injected a note of caution about potential over-reliance on the tools, hinting at the importance of maintaining a balance in their use.

5. Discussion

The objective of this study was to explore the various types of AI writing tools and their influence on the quality of student writing-particularly in terms of content and organization-as perceived by EFL (English as a Foreign Language) teachers. The initial findings of the current research underscored the diverse array of Artificial Intelligence (AI) writing tools (e.g., Quilbot, Jenni, Chat-GPT, WordTune, Copy.ai, Paperpal and Essay writer) utilized by teachers of English as a Foreign Language (EFL). The EFL teachers shared a common interest in utilizing a combination of AI tools in enhancing students' writing skills, hinting at the fundamental trend of embedding technology into education. This is in line with the work of C. Liu et al. ( Citation2021), who emphasized the incorporation of multiple AI tools in enhancing the teaching-learning process. In their study, they observed that a combination of AI tools can foster a more comprehensive learning environment, especially for enhancing writing skills in EFL students. Similarly, Ouyang et al. ( Citation2022) studied the impact of AI in education, and their findings corroborate our research. They noted a growing trend among teachers to use a blend of AI tools to enrich the learning experience and found that this had a positive impact on students' overall academic performance, including their writing skills.

The second findings from our research examined the impact of AI writing tools on the content and organization of second-language writing. In the realm of content, which refers to the generation of ideas and their articulation, AI writing tools have shown promising potential. Wong and Mak ( Citation2019) note that the content of L2 writing, in particular, is often shaped by students' linguistic abilities and their cultural backgrounds. AI writing tools, with their capacity for providing prompts and suggesting improvements, can support students in articulating their thoughts and enhancing the richness of their content (Y. Liu et al., Citation2023). Moreover, a study by Gayed et al. ( Citation2022) found that AI writing tools can be instrumental in fostering idea development, thus aiding students in overcoming creative blocks. This aligns with our findings, where teachers (T1, T3 & T4) acknowledged these tools as beneficial for nurturing idea generation. However, potential downsides were also noted, such as the risk of over-reliance on these tools potentially limiting critical thinking and personal creativity, as raised by T2 and supported by Huang and Tan ( Citation2023). They argued that excessive dependence on AI technologies may decrease one's ability to think creatively and critically, as well as impair the capacity to make autonomous assessments regarding the standard of writing.

In terms of organization, which encompasses the logical arrangement and coherence of ideas, AI writing tools have been found to provide valuable assistance. According to Bowen and Thomas ( Citation2020), the organization of ideas is a critical component of L2 writing, and students often struggle with establishing coherence and logical progression in their texts. Here, AI writing tools can provide instantaneous feedback and suggestions for reorganizing sentences and paragraphs, thus helping students improve the structure of their writing (T. S. Chang et al., Citation2021). In our study, all four participating teachers (T1, T2, T3, and T4) recognized the positive role of AI writing tools in enhancing the clarity of student writing and promoting a logical progression of thoughts and arguments. These findings align with the research by Malik et al. ( Citation2023), who suggested that AI tools can be instrumental in improving the organization of L2 writing. However, it is also crucial to note the potential risks.

Additionally, the teachers in this study (T1, T2, T3, & T4) presented diverse perspectives on the influence of AI writing tools on students' vocabulary use and growth. While certain participants (i.e., T1, T3, & T4) recognized apparent advantages, such as vocabulary enhancement and minimization of repetition, others (i.e., T2) urged caution about potential downsides, including overly complex language and an excessive dependence on these tools. Again, this is supported by the research conducted by Gayed et al. ( Citation2022). They too investigated the impact of AI writing tools on students' vocabulary use and development. They found that these tools offer clear advantages, such as broadening vocabulary and reducing repetitive language use. Furthermore, the study by Fitria ( Citation2021) aligns with our findings. They underlined the role of AI tools in enhancing vocabulary but also cautioned against over-dependence, emphasizing the importance of these tools being used as aids rather than substitutes for comprehensive language learning.

The next point of discussion focused on the teachers' views regarding the impact of AI writing tools on organizational aspects of writing. All four teachers (T1, T2, T3, and T4) recognized the positive role these tools play in enhancing the clarity of student writing and promoting a logical progression of thoughts and arguments. They unanimously agreed on the tools' proficiency in recommending more effective wording, detecting logical inconsistencies, and aiding in the organization of arguments. However, T2 provided a balanced perspective, appreciating both the benefits and potential drawbacks of these tools. Particularly, T2 expressed concern over the possibility that students could become overly reliant on these tools, which could inadvertently inhibit the growth of their critical thinking and problem-solving skills. This concern aligns with the findings of Malik et al. ( Citation2023), who suggested that excessive reliance on AI tools can result in reduced efforts in critical thinking among students.

The study has a few limitations, however, that need to be addressed. One potential limitation is the possible bias in participant selection. The teachers who participated in our research are currently using and actively promoting the use of AI writing tools, which could predispose the findings towards more positive experiences and attitudes. Acknowledging this, it is recommended that future research should include teachers who have tried these tools but have decided not to continue using them. This would present a more balanced perspective and help in understanding the reasons for discontinuation and any potential barriers to effective implementation of these tools in the classroom.

6. Conclusion

This study aimed to examine various types of AI writing tools and their influence on the quality of student writing, focusing on content and organization, from the perspective of EFL teachers. The findings revealed a diverse array of AI writing tools employed in EFL teaching and shed light on the strategies teachers use to integrate these tools to address students' writing challenges. In particular, the tools, including Quillbot, Jenni, Chat-GPT, WordTune, Copy.ai, Paperpal, and Essay writer, were found to foster a comprehensive learning environment and enrich students' overall academic performance. The study also found unanimous agreement among the teachers about the positive role of AI writing tools in enhancing the clarity and logical progression of students' writing. However, concerns were expressed about the possibility of students becoming overly reliant on these tools, which could inhibit their critical thinking and problem-solving skills. The study's implications suggest the integration of AI writing tools can be beneficial in enhancing the quality of EFL student writing.

While the present study provides insightful revelations regarding the use of AI writing tools in EFL classrooms, there are several limitations that should be noted. Firstly, the data was obtained from only four EFL teachers from three universities in Indonesia, which limits the breadth of the study. Different regions may have varying degrees of access to technology, resources, and the attitudes of teachers towards AI tools, and this study may not fully represent the experiences of teachers in different contexts. Additionally, the teachers who participated in our research are currently using and actively promoting the use of AI writing tools, which could predispose the findings towards more positive experiences and attitudes. This potential bias in participant selection should be acknowledged. To address this limitation, it is recommended that future research includes teachers who have tried these tools but have decided not to continue using them. This would present a more balanced perspective and help in understanding the reasons for discontinuation and any potential barriers to effective implementation of these tools in the classroom. Furthermore, the study focused solely on teachers' perceptions and did not incorporate students' views or measure their actual improvements in writing skills due to the use of these AI tools. Therefore, it may not present a complete picture of the impact of AI writing tools on students' writing quality. Furthermore, the research did not examine the long-term effects of using AI tools on students' writing skills, nor did it investigate the potential negative effects of students' over-dependence on AI tools, which was a concern raised by some teachers. It is important to address these limitations in future research to gain a more comprehensive understanding of the implications and potential drawbacks associated with the use of AI writing tools in EFL classrooms

Acknowledgments

The authors are grateful for the financial support from the Endowment Education Funding of Indonesia (LPDP) through Centre for Higher Education Funding (BPPT) for this research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work was supported by the Balai Pembiayaan Pendidikan Tinggi (BPPT) [xxx-xxx]; Lembaga Pengelola Dana Pendidikan [xxx-xxx].

Notes on contributors

Marzuki

Utami Widiati

References

Cahyono, B. Y., Khotimah, K., & Batunan, D. A. (2023). Workable approaches in EFL teaching mediated by mobile technology during the pandemic and post-pandemic: Indonesian EFL teachers' experiences and expectations. Computer Assisted Language Learning, 24, 138-159. http://callej.org/journal/24-1/Cahyono-Khotimah-Batunan-Imamyartha2023.pdf(open in a new window)

Hosseini, M., Rasmussen, L. M., & Resnik, D. B. (2023). Using AI to write scholarly publications. Accountability in Research, 1-9. https://doi.org/10.1080/08989621.2023.2168535

Huang, J., & Tan, M. (2023). The role of ChatGPT in scientific communication: Writing better scientific review articles. American Journal of Cancer Research, 13(4), 1148. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10164801/(open in a new window)

Jeanjaroonsri, R. (2023). Thai EFL learners' use and perceptions of mobile technologies for writing. Learn Journal: Language Education and Acquisition Research Network, 16(1), 169-193. https://so04.tci-thaijo.org/index.php/LEARN/article/view/263438(open in a new window)

Lam, R., & Moorhouse, B. L. (2022). Using digital portfolios to develop students' writing: A practical guide for language teachers. Taylor & Francis. https://books.google.co.id/books?hl=id&lr=&id=b4qVEAAAQBAJ&oi=fnd&pg=PT9&dq=WordTune+AI(open in a new window) , https://doi.org/10.4324/9781003295860

Nazari, N., Shabbir, M. S., & Setiawan, R. (2021). Application of artificial intelligence powered digital writing assistant in higher education: Randomized controlled trial. Heliyon, 7(5), e07014. https://doi.org/10.1016/j.heliyon.2021.e07014

Nykyporets, S. S. (2023). Harnessing cloud technologies for foreign language acquisition among masters in energy engineering. Moderní aspekty vědy: Svazek XXXI mezinárodní, 21-56. http://ir.lib.vntu.edu.ua/handle/123456789/37121

Umamah, A., & Cahyono, B. Y. (2022). EFL university students' use of online resources to facilitate self-regulated writing. Computer Assisted Language Learning, 23(1), 108-124. http://callej.org/journal/23-1/Umamah-Cahyono2022.pdf(open in a new window)

Yin, R. K. (2009). Case study research: Design and methods (4th ed.). Sage Publications.

# data[].attributes.attachment_link.title
The impact of AI writing tools on the content and organization of students' writing: EFL teachers' perspective

# data[].attributes.attachment_link.description
The primary objective of this study was to examine the range of available Artificial Intelligence (AI) writing tools and assess their influence on student writing, particularly in terms of content ...

# data[].attributes.body
https://journals.lww.com/jnsm/fulltext/2023/06030/adapting_to_the_impact_of_artificial_intelligence.9.aspx

# data[].attributes.body
https://cardinalnewmansociety.org/there-is-no-ai-shortcut-to-education/

# data[].attributes.attachment_link.readable_text
Educational optimists predict that artificial intelligence, or "AI," will soon provide amazing efficiencies and progress in teaching and learning. But are efficiency and machine logic what our students need most?

It's certainly true that AI is a marvelous new tool dramatically transforming human life. AI is exponentially improving in speed and scope to recognize patterns in immensely complex data sets of all types, allowing it to make uncanny predictions about what might come next in a sequence, be it a purchase from a customer, a word in a sentence, a sound in spoken language, or countless other processes. AI can answer specific complex questions or perform intricate calculations at a rate impossible for the human mind to comprehend, let alone compete with. It can also generate images and speech which not only mimic reality but surpass it to meet programed standards of excellence.

However helpful these may be to adults seeking to improve productivity and processes, it is also prudent to follow G.K. Chesterton's advice, that children ought not be subjected to educational projects and ideas younger than they are. Allowing students to dodge traditional learning methods with AI will have uncertain and potentially harmful results. The same could be true of educators' dependence on AI for student assessment and lesson planning. In education at least, it is quite possible that AI will work against natural human development and provide not a shortcut to human formation but a short circuit.
Why Students Might Think of AI As a Shortcut
For students inclined to see homework as burdensome or useless, AI is now a tempting shortcut. AI can instantly answer most calculations and informational homework assignments including "show your work" complex math and science problems, and it can aid the writing of unique history or literature papers. AI is a transformative hack to game the homework system. Even so, some pedagogues celebrate this possible dismantling of conventional homework in the hopes that rote learning or linear thinking will no longer be assigned. Teachers will be forced to focus on developing assignments that are personalized and promote "critical thinking" and "authentic learning."
Why This Thinking Is Really a Short Circuit
While developing creative and effective homework assignments should be encouraged, educators cannot short-circuit the complete learning process by giving up on requiring students to engage in rote learning and writing assignments, even if some students cheat with AI. The hard work of instilling facts and information into the human brain and developing complex neural pathways step-by-step still needs to happen. It is not time-efficient to throw out homework altogether. Teachers should integrate some AI speedbumps into homework such as breaking the writing process into multiple submissions that each require peer or teacher feedback, creating more in-class writing assignments and oral presentations, etc., but they should not simply throw out most homework thereby restricting educational time. Students still need to learn prudent use of time and the development of academic skills and virtue which are part and parcel of traditional homework.

Homework in this sense might be viewed as something akin to practicing a musical instrument: A music teacher assigns practice at home, even if the student ignores or cheats in the practice sessions. There simply is no shortcut (technological or otherwise) to the human need to toil through the learning process, if the goal is freedom to play an instrument or earn a living as a musician. Few are interested in listening to a player-piano, but many will take time to listen to a fully formed musician who, after years of memorization and practice, is now free to creatively contribute their complex and personally unrepeatable humanity to the musical occasion. The world of AI requires us to focus even more on cultivating distinct human genius, compassion, insight, and creativity.

Because homework can be easily falsified, educators must be explicit in showing students and their parents how such deception and sloth short-circuits authentic learning and human development. It can lead to a person's real demise in a world where AI will be in constant and growing competition with their individual livelihood. Self-replacing personal development with AI may fool the teacher but may result in the student's inability to compete against AI with future employers. It is a student's complete human development which will be their competitive advantage against AI job replacement in the future.

The use of generative AI to produce writing, which one then presents fully or even partially as one's own, is particularly problematic. It is plagiarism and lying unless fully disclosed. It also weakens thinking, because writing is a type of explicit thinking. Writing assignments at their best require students to engage in multiple levels of critical thinking including synthesis, evaluation, and creativity. Students need extensive practice to develop these profoundly important human skills. We have students write essays on history and literature, not to generate new knowledge for the human species but to develop their own knowledge, understanding, and cognitive power. If they get initial essays and ideas from generative AI they will stymie their own generative capabilities.

Additionally, the AI homework short-circuit deprives a student of the value of their teacher. A teacher uses homework and, especially writing, as one tool among many to gain clear insight into a student's real thinking and processing to assist the student in properly interacting with, interpreting, and valuing what is before them. Any deception in this process does great harm on multiple levels. There is no short-circuiting of the educational process, especially the writing process, without harm.
Why Educators Might Think of AI As a Shortcut
Some educators can be bedazzled by AI's tremendous capability for analyzing student data. AI assisted by standardized test results can establish a student's reading or math level and decide what instruction, texts, or problem sets should come next. AI can also be programed to give students hints as opposed to the final answer in answering questions. It can even predict where errors in thinking may have occurred and offer tailored intervention and practice in that area. It can seem like the ultimate personalized teacher, who is not distracted by other students and has access to unlimited, perfectly tailored resources. As a bonus, teachers assisted by AI may have even more time to be a "guide on the side," with less time grading and lesson planning.
Why This Thinking Is Really a Short Circuit
This apparent "win-win" for the virtual-headset-wearing members of "Plato's Cave Academy" is really a short circuit. Teaching and learning are such fundamental and intimate human-to-human processes that farming out significant elements of human formation to computers is quite literally inhuman.

Humans are social animals who learn best socially, in person, and in relationships. This was made abundantly and tragically clear during the COVID school shutdowns. The academic, social, and psychological damage of isolating students on screens has lasting negative consequences. A child who is fortunate enough to have access to a healthy functioning classroom and school reaps great rewards in total human development. The fact that these conditions may be increasingly rare does not merit jettisoning the community-schooling paradigm for the cost effective, targeted, and quick efficiency of machine-generated teaching and learning.

Students typically find joy, bonding, and inspiration if surrounded by healthy and supportive classmates and a caring and wise teacher. Doing hard human stuff side-by-side with other humans is what students are wired for and can make the difficult delightful. Leaning conducted as human play and human exchange is also essential to motivation and learning, as titillating as computer-based learning games may be. Isolating students on screens where they work at their own pace deprives students of the benefit of watching peers attempting to do similar things and adjusting or confirming their own efforts in response. They witness approaches that they might not otherwise have imagined and bond with fellows in a common project of discovery or skill mastery. This is tremendously satisfying and makes school a joy!

Additionally, the teacher is tasked with instruction through modeling a human passion for engaging with truth, beauty, and goodness wherever it arises, manifesting a rich spirituality, and showing deep care and compassion for others. This has a profound impact on student learning, happiness, well-being, and a healthy disposition toward all that is being studied and fellow students. AI cannot look a student in the eye, smile, act with integrity, and model lived virtue. And AI cannot love the student. Nor can it teach the student to do these things. A real teacher, absent screens and mediators, can write on a child's soul and inspire them to greatness.

The teacher is also a model and source of real human learning by their encouragement and affirmation for the student, who at times struggles and at times makes spectacular breakthroughs-both of which are access points of human intimacy in the face of despair or joy and therefore demand real human response. Despondency and cynicism result when human experiences are isolated, unrecognized by other humans, or simply materialized in data sets. It is just not right, on a regular rather than occasional basis, to have a young child read to a computer to be analyzed and instructed rather than reading to and with another a human. Perhaps it may save time and provide individualized data, but this computerized "personalization" can actually depersonalize the instruction, as it removes the teacher from the process. And besides, what is time for, if not listening to children read?

And, of course, reading is much more than sounding at words and getting right responses about linear or discrete text-based questions, which are at the heart of computer-based instruction. Students are taught to read because they are human beings who love to share stories and insights with each other. They are also spiritual beings made to seek and know the truth, act upon it, and share it with others. Reading helps them transcend themselves, their time, their experience, and their culture. It helps them discover transcendent meaning of things and view the world from another's perspective.

This being said, there is room for providing drills and problem sets based on student interface with AI at home. The data from such AI-informed and tracked home-drills can be used by the instructor to guide in-person instruction. What needs to be protected is the authentic interaction which the school classroom provides.
The Dispiriting Nature of AI
As the excitement of AI dissipates and its numbing and dispiriting effects take hold of more and more people, jaded souls may fall into despair. The lack of human contact, lack of reality, inability to attain true beauty when left with AI-generated avatars and robots, inability to match AI-generated speed and perfection, and inability to compete with AI for employment will be devastating for many. Added to this, carefully crafted and monetized AI experiences such as immersive games and other entertainment specifically tailored by AI to personal desires and weaknesses will flood human senses and their lives, seeming to cover the pain and lack of true happiness, and worse yet, eliminating the sense that this unreality should cause them pain at all.

Wise parents will understand that radical recent changes in technology mean that now, more than ever, screen time and technology are direct threats to their children and their well-being. There is no benefit to introducing children to AI. The whole purpose of AI is to make interfacing with technology seamless, so there is no danger that children who do not use AI now will somehow not be able to use it productively in the future.

Educators and parents need to prevent children from becoming trapped in addictive, unreal worlds. The response to this threat should be dramatic and counter unreality with reality, at every opportunity. Children need to be at home in the real world. It is a world made for them, for their happiness, and for their work by a loving God. Educators must guide them in seeking and ascribing authentic meaning to those flawed but real experiences that make up the real world. Educators must reject anything which threatens their mind/body/spirit unities, including errant human sexual ideologies. Students need to be re-embodied and re-integrated with themselves and the natural/real world, and re-enchanted with the beauty and meaning present in all things, instilled by and delighted in by God, their creator, and in whose image they are made.
The Quest for the Authentically Human
AI is an adult tool which can unlock immense possibilities and creative resources. It can be of use to educators in their own research, development of materials, analysis of data, and administrative duties, but it does not belong in any direct, sustained way in the teacher-student relationship.

Now is the time to re-embrace the humanities in education. Those human actors with rich literary, historical, philosophical, theological, and psychological training will be best suited to succeed in a competitive AI-controlled market. In the end, most problems in the workforce will remain "people problems," and human creativity in all its uniqueness will be even more prized. Those individuals who are creative and who are best able to understand and help people and people-based challenges will always be needed and appreciated. A goal of educators at this point should be to maximize reality-based experiences for students and their unique human potentialities. It is these fully formed and trained memories, minds, imaginations, and wills which will act upon the new and unpredictable opportunities AI is beginning to create for them. Students need to organically perfect their humanity, as technology de-humanizes it. Un-mediated access to the greatest human accomplishments, presented and discussed by other humans who know and love them-even with their blemishes and misshapenness-is what students love and what will help them to love learning and indeed to love each other.
Daniel Guernsey, Ed.D., is senior fellow and education policy editor at The Cardinal Newman Society and associate professor of education and director of the M.Ed. in Catholic Educational Leadership at Ave Maria University.

# data[].attributes.attachment_link.title
There Is No AI Shortcut to Education - Cardinal Newman Society

# data[].attributes.attachment_link.description
Educational optimists predict that artificial intelligence, or "AI," will soon provide amazing efficiencies and progress in teaching and learning. But are efficiency and machine logic what our students need most? It's certainly true that AI is a marvelous new tool dramatically transforming human life. AI is exponentially improving in [...]

# data[].attributes.body
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10166793/

# data[].attributes.attachment_link.readable_text
Luigi De Angelis, 1 , † Francesco Baglivo, 1 , † Guglielmo Arzilli, 1 , * Gaetano Pierpaolo Privitera, 1 , 2 Paolo Ferragina, 3 Alberto Eugenio Tozzi, 4 and Caterina Rizzo 1

Abstract

Large Language Models (LLMs) have recently gathered attention with the release of ChatGPT, a user-centered chatbot released by OpenAI. In this perspective article, we retrace the evolution of LLMs to understand the revolution brought by ChatGPT in the artificial intelligence (AI) field.

The opportunities offered by LLMs in supporting scientific research are multiple and various models have already been tested in Natural Language Processing (NLP) tasks in this domain.

The impact of ChatGPT has been huge for the general public and the research community, with many authors using the chatbot to write part of their articles and some papers even listing ChatGPT as an author. Alarming ethical and practical challenges emerge from the use of LLMs, particularly in the medical field for the potential impact on public health. Infodemic is a trending topic in public health and the ability of LLMs to rapidly produce vast amounts of text could leverage misinformation spread at an unprecedented scale, this could create an "AI-driven infodemic," a novel public health threat. Policies to contrast this phenomenon need to be rapidly elaborated, the inability to accurately detect artificial-intelligence-produced text is an unresolved issue.

Keywords: artificial intelligence, natural language processing, large language models, ChatGPT, public health, infodemic

1. Introduction

"ChatGPT" is a large language model (LLM) trained by OpenAI, an Artificial intelligence (AI) research and deployment company, released in a free research preview on November 30th 2022, to get users' feedback and learn about its strengths and weaknesses ( 1) Previously developed LLMs were able to execute different natural language processing (NLP) tasks, but ChatGPT differs from its predecessors. It's an AI chatbot optimized for dialog, especially good at interacting in a human-like conversation.

With the incredibly fast spread of ChatGPT, with over one million users in 5 days from its release, ( 2) many have tried out this tool to answer complex questions or to generate short texts. It is a small leap to infer that ChatGPT could be a valuable tool for composing scientific articles and research projects. But can these generated texts be considered plagiarism? ( 3, 4).

It took a while to adopt systems at the editorial level to recognize potential plagiarism in scientific articles, but intercepting a product generated by ChatGPT would be much more complicated.

In addition, the impact that this tool may have on research is situated within a background that has been profoundly affected after the COVID-19 pandemic ( 5). In particular, health research has been strongly influenced by the mechanisms of dissemination of information regarding SARS-CoV-2 through preprint servers that often allowed for rapid media coverage and the consequent impact on individual health choices ( 6, 7).

Even more than scientific literature, social media have been the ground of health information dissemination during the COVID-19 pandemic, with the rise of a phenomenon known as infodemic ( 8).

Starting from a background on the evolution of LLMs and the existing evidence on their use to support medical research, we focus on ChatGPT and speculate about its future impact on research and public health. The objective of this paper is to promote a debate on ChatGPT's space in medical research and the possible consequences in corroborating public health threats, introducing the novel concept of "AI-driven infodemic."

2. The evolution of pre-trained large language models

The LLMs' evolution in the last 5 years has been exponential and their performance in a plethora of different tasks has become impressive.

Before 2017, most NLP models were trained using supervised learning for particular tasks and could be used only for the task they were trained on ( 9).

To overcome those issues, the self-attention network architecture, also known as Transformer, ( 10) was introduced in 2017 and was used to develop two game-changing models in 2018: Bidirectional Encoder Representations from Transformers (BERT) and Generative Pretrained Transformer (GPT) ( 11, 12).

Both models achieved superior generalization capabilities, thanks to their semi-supervised approach. Using a combination of unsupervised pre-training and supervised fine-tuning, these models can apply pre-trained language representations to downstream tasks.

GPT models rapidly evolved in different versions, being trained on a larger corpus of textual data and with a growing number of parameters.

The third version of GPT (GPT-3), with 175 billion parameters, is 100 times bigger than GPT-2 and approximately two times the number of neurons in the human brain ( 13).

GPT-3 can generate text that is appropriate for a wide range of contexts, but unfortunately, it often expresses unintended behaviors such as making up facts, generating biased text, or simply not following user instructions ( 14).

This can be explained since the objective of many LLMs, including GPT-3, is to predict the next element in a text, based on a large corpus of text data from the internet, ( 15) thus LLMs learn to replicate biases and stereotypes present in that data ( 16).

Here comes the major problem of alignment: the difficulty of ensuring that a LLM is behaving in a way that is aligned with human values and ethical principles.

Addressing the alignment problem for LLMs is an ongoing area of research and OpenAI developed a moderation system, trained to detect a broad set of categories of undesired content, including sexual and hateful content, violence, and other controversial topics ( 17).

ChatGPT incorporates a moderation system, but the true innovation lies in its user-centered approach, which was used to fine-tune the model from GPT-3 to follow the user instructions "helpfully and safely" ( 14).

This process started from InstructGPT, a LLM with "only" 1.3 billion parameters trained using reinforcement learning from human feedback (RLHF), a combined approach of supervised learning, to obtain human feedback, and reinforcement learning using human preferences as a reward signal.

RLHF is used for adapting the pre-trained model GPT-3 to the specific task of following users' instructions. From the optimization of InstructGPT for dialog, ChatGPT was born.

Despite these advancements, ChatGPT still sometimes writes plausible-sounding but incorrect or nonsensical answers, due to its inability of fact-checking and its knowledge limited until 2021 ( 1).

In Table 1 we summarize the strengths and weaknesses of large language models that can be considered important steps to understand the development of ChatGPT. The evolution of LLMs is still in its early stages and the release of GPT-4 on the 14th of March 2023 has been another step forward in LLMs' rapid advancement. GPT-4 has already been integrated into ChatGPT and this model seems to be more reliable, creative, and able to handle much more nuanced instructions, even if only a few technical details on the model were provided by OpenAI, for competitive and safety concerns. ( 18) GPT-4 impact on health care delivery and medical research is expected to be huge, but its limitations need to be taken into account ( 19).

Table 1

Large Language Model.

Year

Paper DOI

Pros

Cons

GPT

2018

https://api.semanticscholar.org/CorpusID:49313245

-first model using semi-supervised training

-trained on the BooksCorpus (800 M words)-need supervised fine-tuning to perform a specific task

BERT

2018

arXiv:1810.04805

-trained on BooksCorpus (800 M words) + Wikipedia (2,500 M words)-bidirectional architecture-unified architecture across different tasks

-need supervised fine-tuning to perform a specific task

GPT-2

2018

https://api.semanticscholar.org/CorpusID:160025533

-trained on WebText (40GB of text) -learn to perform tasks directly without the need for supervised fine-tuning (task-agnostic)

-zero-shot performance (without fine-tuning) still far from useable.

GPT-3

2020

arXiv:2005.14165

-trained on CommonCrawl (570 GB of text) + WebText + Wikipedia+ Books1-2 -trained using zero-shot, one-shot (one example of the task), and few-shot (10-100 examples of the task) settings

-In text synthesis sometimes semantic repetition and loss of coherence over sufficiently long passages
-retains the biases of the data it has been trained on

Instruct-GPT

2022

arXiv:2203.02155

-aligned to act in accordance with the user's intention using reinforcement learning from human feedback

-aligning to demonstrations and preferences provided by training labelers, not to human values. -follow the user's instruction, even if that could lead to harm in the real world.

3. Large language models to support academic research

One potential application of LLM is in support of academic research. The scientific literature, with around 2.5 million papers published every year, ( 20) due to its magnitude is already beyond human handling capabilities.

AI could be a solution to tame the scientific literature and support researchers in collecting the available evidence, ( 21) by generating summaries or recommendations of papers, which could make it easier for researchers to quickly get the key points of a scientific result. Overall, AI tools have the potential to make the discovery, consumption, and sharing of scientific results more convenient and personalized for scientists. The increasing demand for accurate biomedical text mining tools for extracting information from the literature led to the development of BioBERT, a domain-specific language representation model pre-trained on large-scale biomedical corpora ( 22).

BioBERT outperforms previous models on biomedical NLP tasks mining, including named entity recognition, relation extraction, and question answering.

Another possible approach is the one of domain-specific foundation models, such as BioGPT and PubMedGPT 2.7B, ( 23, 24) that were trained exclusively on biomedical abstracts and papers and used for medical question answering and text generation.

Med-PaLM, a LLM trained using few-shot prompting, exceeds previous state-of-the-art models on MedQA, a medical question answering dataset consisting of United States Medical Licensing Exam (USMLE) style questions ( 25). The performance of ChatGPT on USMLE was recently evaluated and it achieved around 50-60% accuracy across all examinations, near the passing threshold, but still inferior to Med-PaLM ( 26).

GPT-4 exceeds the passing score on USMLE by over 20 points and outperforms earlier LLMs. Nevertheless, there is a large gap between competency and proficiency examinations and the successful use of LLMs in clinical applications ( 27).

In the NLP task of text summarization GPT-2 was one of the best-performing models used for summarizing COVID-19 scientific research topics, using a database with over 500,000 research publications on COVID-19 (CORD-19) ( 28, 29).

CORD-19 was also used for the training of CoQUAD, a question-answering system, designed to find the most recent evidence and answer any related questions ( 30).

A web-based chatbot that produces high-quality responses to COVID-19-related questions was also developed, this user-friendly approach was chosen to make the LLM more accessible to the general audience ( 31).

LLMs have also been used for abstract screening for systematic reviews, this allows the use of unlabelled data in the initial step of scanning abstracts, thus saving researcher's time and effort ( 32).

LLMs facilitate the implementation of advanced code generation capabilities for statistical and data analytics, two large-scale AI-powered code generation tools have recently come into the spotlight:

OpenAI Codex, a GPT language model fine-tuned on publicly available code from GitHub, ( 33) and DeepMind AlphaCode, designed to address the main challenges of competitive programming ( 34).

On one hand, AI tools can make programmers' jobs easier, aid in education and make programming more accessible ( 35).

On the other hand, the availability of AI-based code generation raises concerns: the risk of using code generation models is users' over-reliance on the generated outputs, especially non-programmers may quickly become accustomed to auto-suggested solutions ( 36).

The above-described deskilling issue is not limited to coding. If we conceive a scenario in which AI is extensively used for scientific production, we must consider the risk of deskilling in researchers' writing abilities. Some have already raised concerns about the peril of seeing the conduct of research being significantly shaped through AI, leading to a decline in the author's ability to craft meaningfully and substantively her objects of study ( 37).

Our reflections highlight a growing interest in the use of LLMs in academic research, with the release of ChatGPT this interest has only increased ( 38).

4. The revolution of ChatGPT and the potential impact on scientific literature production

The user-centered approach of ChatGPT is the paradigm shift that makes it different from previous LLMs. The revolutionary impact of ChatGPT does not lie in its technical content, which appears to be merely a different methodology for training, but in the different perspective that it is bringing. ChatGPT will probably be overtaken soon, but the idea of making AI accessible to the broader community and putting the user at the center will stand.

The accessibility and user-friendly interface of ChatGPT could induce researchers to use it more extensively than previous LLMs. ChatGPT offers the opportunity to streamline the work of researchers, providing valuable support throughout the scientific process, from suggesting research questions to generate hypotheses. Its ability to write scripts in multiple programming languages and provide clear explanations of how the code works, makes it a useful asset for improving understanding and efficiency. Examples of these ChatGPT's abilities are provided in Figure 1 .

ChatGPT can also be used to suggest titles, write drafts, and help to express complex concepts in fluent and grammatically correct scientific English. This can be particularly useful for researchers who may not have a strong background in writing or who are not native English speakers. By supplementing the work of researchers, rather than replacing it, automating many of the repetitive tasks, ChatGPT may help researchers focus their efforts on the most impactful aspects of their work.

The high interest of the scientific community in this tool is demonstrated by the rapid increase in the number of papers published on this topic, shortly after its release. The use of ChatGPT in scientific literature production has already become a reality during the writing of this draft, many authors stated to have used ChatGPT to write at least part of their papers ( 39). This underlines how ChatGPT has already been integrated into the research process, even before addressing ethical concerns and discussing common rules. For example, ChatGPT has been listed as the first author of four papers, ( 26, 40, 41, 42) without considering the possibility of "involuntary plagiarism" or intellectual property issues surrounding the output of the model.

The number of pre-prints produced using ChatGPT points out that the use of this technology is inevitable and a debate in the research community is a priority ( 43).

5. Navigating the threats of ChatGPT in public health: AI-driven infodemic and research integrity

A potential concern related to the emergence of LLMs is the submissiveness in following users' instructions. Despite the limitations imposed by programmers, LLMs can be easily tricked into producing text on controversial topics, including misinforming content ( 44).

The ability of LLMs to generate texts similar to those composed by humans could be used to create fake news articles or other seemingly legitimate but actually fabricated or misleading content, ( 45, 46) without the reader realizing that the text is produced by AI ( 47).

Under this damaging matter, the counter-offensive rises: some authors highlight the importance of creating LLM detectors that can be able to identify fake news, ( 48) while others propose LLMs to support the enhancement of detector performance ( 49). Commonly used GPT-2 detectors were flawed in recognizing text written by AI when generated by ChatGPT ( 50), new detectors were rapidly developed and released to address this gap, but these tools do not perform well in identifying GPT-4 generated text. This poses a continuous unfair competition to improve detectors that need to follow the pace of LLMs' rapid advancement, leaving a gap for malicious intent.

As a result, this poses a continuous unfair competition to improve detectors that need to follow the pace of LLMs' rapid advancement, leaving a gap for malicious intent.

The absence of accurate detectors calls for precautionary measures, for example, the International Conference on Machine Learning (ICML) for its 2023 call for papers prohibited the use of LLMs such as ChatGPT in submitted drafts. However, ICML acknowledges that currently there is not any tool to verify compliance with this rule and thus they are relying on the discretion of participants and await the development of shared policies within the scientific community.

Many scientific journals are questioning the policy matter, publishing editorials on the topic, and updating author's guidelines ( 51).

For example, Springer Nature journals have been the first to add rules in the guide to authors: to avoid accountability issues, LLMs cannot be listed as authors and their use should be documented in the methods or acknowledgments sections ( 3). Also Elsevier created guidelines on the use of AI-assisted writing for scientific production, confirming the rules imposed by Springer and requiring the authors to specify the AI tools employed, giving details on their use. Elsevier declared to be committed to monitor the development around generative AI and to refine the policy if necessary ( 52).

The misuse of ChatGPT in scientific research could lead to the production of fake scientific abstracts, papers, and bibliographies. In the earlier versions of ChatGPT (up to the 15th December version), when asked to cite references to support its statements, the output was a list of fake bibliographic references. (e.g., fabricated output reference: Li, X., & Kim, Y. (2020). Fake academic papers generated by AI: A threat to the integrity of research. PLOS ONE, 15 ( 3), e0231891.)

The usage of real authors' names, journals, and plausible titles makes the fake reference difficult to immediately spot. This calls for preventive actions, such as the mandatory use of the digital object identifier system (DOI), which could be used to rapidly and accurately identify fake references.

In fields where fake information can endanger people's safety, such as medicine, journals may have to take a more rigorous approach to verify the information as accurate ( 53). A combined evaluation by more up-to-date AI-output detectors and human reviewers is necessary to identify AI-generated scientific abstracts and papers, though this process may be time-consuming and imperfect. We, therefore, suggest adopting a "detectable-by-design" policy: the release of new generative AI models by the tech industry to the public should be permitted only if the output generated by the AI is detectable and thus can be unequivocally identified as AI-produced. The impact that generating false and potentially mystifying texts can have on people's health is huge. The issue of the dissemination of untruthful information has long been known: starting with the unforgettable Wakefield case and the then-generated disbelief that vaccines can cause autism, ( 54) to the current observation of non-conservative behavior evidenced by the various phases of the COVID-19 pandemic ( 55). In this context, it has been more evident than ever that junk and manipulative research, through underperforming studies or with study designs unfit to carry out the intended research objective, has had an impact on the behavior of the general population and, more worryingly, on health professionals ( 56).

The diffusion of misinformation conveyed through rapidly disseminated channels such as mass media and social networks, can generate the phenomenon known as infodemic ( 57). The consequence on the scientific framework is considerable, even with implications on possible healthcare choices, already a determining factor in the recent pandemic. ( 58) Infodemic can influence medical decision-making on treatment or preventive measures, ( 59, 60) for example some people used hydroxychloroquine as a treatment for COVID-19 based on false or unproven information, endorsed by popular and influential people ( 61). The risk is that we may face a new health emergency where new information can be rapidly produced using LLMs to generate human-like texts ready to spread even if incorrect or manipulated. The concept of infodemic was introduced in 2003 by Rothkopf as an "epidemic of information," ( 62) and evolved in 2020 after the COVID19 pandemic, integrating the element of rapid misinformation spreading ( 63). With the global diffusion of LLMs the infodemic concept must evolve again into the one of "AI-driven Infodemic." Not only is it possible to rapidly disseminate misinformation via social media platforms and other outlets, but also to produce exponentially growing amounts of health-related information, regardless of one's knowledge, skills, and intentions. Given the nature of social media content diffusion, LLMs could be used to create content specifically designed for target population groups and in order to go viral and foster misinformation spread. We foresee a scenario in which human-like AI-produced contents will dramatically exacerbate every future health threat that can generate infodemics, that from now on will be AI-driven. Social media and gray literature have already been the ground for infodemic, ( 63) but scientific literature could become a new and powerful means of disinformation campaigns. The potential of LLMs and in particular ChatGPT in easily generating human-like texts, could convey excessive and, without proper control, low-quality scientific literature production in the health field. The abundance of predatory journals, that accept articles for publication without performing quality checks for issues such as plagiarism or ethical approval, ( 64) could allow the flooding of the scientific literature with AI-generated articles on an unprecedented scale. The consequences on the integrity of the scientific process and the credibility of the literature would be dreadful ( 65).

6. Discussion

Large language models have already shown hints of their potential in supporting scientific research and in the next months we expect a growing amount of papers talking about the use of ChatGPT in this field.

The accessibility and astonishing abilities of ChatGPT made it popular across the world and allowed it to achieve a milestone, setting AI conversational tools to the next level.

But soon after its release possible threats emerged, ChatGPT's ability to follow user's instruction is a double-edged sword: on one hand, this approach makes it great at interacting with humans, on the other hand being submissive ab origine exposes it to misuse, for example by generating convincing human-like misinformation.

The field of medical research may be a great source for both opportunities and threats coming from this novel approach.

Given that the scientific community has not yet determined the principles to follow for a helpful and safe use of this disruptive technology, the risks coming from the fraudulent and unethical use of LLMs in the health context cannot be ignored and should be assessed with a proactive approach.

We define the novel concept of "AI-driven infodemic," a public health threat coming from the use of LLMs to produce a vast amount of scientific articles, fake news, and misinformative contents. The "AI-driven infodemic" is a consequence of the use of LLM's ability to write large amounts of human-like texts in a short period of time, not only with malicious intent, but in general without any scientific ground and support. Beyond text-based content, other AI tools, such as generative-adversarial networks, could also generate audio and video Deepfakes that could be used to disseminate misinformation content, especially on social media ( 66). Political Deepfakes already contributed toward generalized indeterminacy and disinformation ( 67).

To address this public health threat is important to raise awareness and rapidly develop policies through a multidisciplinary effort, updating the current WHO public health research agenda for managing infodemics ( 68). There is a need for policy action to ensure that the benefits of LLMs are not outweighed by the risks they pose. In this context, we propose the detectable-by-design approach, which involves building LLMs with features that make it easier to detect when they are being used to produce fake news or scientific articles. However, implementing this approach could slow down the development process of LLMs, and for this reason, it might not be readily accepted by AI companies. The constitution of groups of experts inside health international agencies (e.g., WHO, ECDC) dedicated to monitor the use of LLMs for fake news and scientific articles production is needed, as the scenario is rapidly evolving and the AI-driven infodemic threat is forthcoming. Such groups could work closely with AI companies to develop effective strategies for detecting and preventing the use of LLMs for nefarious purposes. Additionally, there might be a need for greater regulation and oversight of the AI industry to ensure that LLMs are developed and used responsibly. Recently, the President of the Italian Data Protection Authority (DPA) has taken action against Open AI for serious breaches of the European legislation on personal data processing and protection ( 69). The DPA has imposed a temporary ban on ChatGPT in Italy due to the company's failure to provide adequate privacy information to its users its and lack of a suitable legal basis for data collection. The absence of a suitable legal basis for data collection raises serious concerns about the ethical implications of using personal data without consent or an adequate legal framework.

In the WHO agenda, AI is considered a possible ally to fight infodemics, allowing automatic monitoring for misinformation detection; but the rise of LLMs and in particular ChatGPT should raise concerns that it could play an opposite role in this phenomenon.

LLMs will continue to improve and rapidly become precious allies for researchers, but the scientific community needs to ensure that the advances made possible by ChatGPT and other AI technologies are not overshadowed by the risks they pose. All stakeholders should foster the development and deployment of these technologies aligned with the values and interests of society. It is crucial to increase understanding of AI challenges of transparency, accountability, and fairness in order to develop effective policies. A science-driven debate to develop shared principles and legislation is necessary to shape a future in which AI has a positive impact on public health, not having such a conversation could result in a dangerous AI-fueled future ( 70).

Data availability statement

The original contributions presented in the study are included in the article/ Supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

LDA, FB, and GA conceived the paper, performed the literature search, and drafted the manuscript. GP, AT, PF, and CR provided expert insights and contributed to the manuscript revision. LDA and FB contributed equally to the manuscript. All authors contributed to the article and approved the submitted version.

Funding

This work was supported also by the Italian Ministry of Health with "Current Research funds" for Bambino Gesù Children's Hospital.

Conflict of interest

AT received honoraria for education activities on digital health and immunizations from Roche, AstraZeneca, Novartis, Merck, MSD, Pfizer, Sanofi.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

11. Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. NAACL HLT 2019-2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies -Proceedings of the Conference [Internet]. (2018); 1:4171-4186. Available at: [ https://arxiv.org/abs/1810.04805v2 Google Scholar]

14. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, et al.. Training Language Models to Follow Instructions with Human Feedback In: Advances in Neural Information Processing Systems (2022). doi: 10.48550/arXiv.2203.02155 [[ CrossRef] Google Scholar]

15. Korngiebel DM, Mooney SD. Considering the possibilities and pitfalls of generative pre-trained transformer 3 (GPT-3) in healthcare delivery. NPJ Digital Medicine. (2021):1-3. [ PubMed] PMC free article] [

26. Kung T. H., Cheatham M., Medenilla A., Sillos C., De Leon L., Elepaño C., et al.., Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. [ medRxiv [Internet]. (2022); 2022.12.19.22283643. [Epub ahead of preprint] 10.1101/2022.12.19.22283643v2 PMC free article] [ PubMed] [ CrossRef]

28. Ontoum S, Chan JH. Automatic text Summarization of COVID-19 Scientific Research Topics using pre-trained Models from Hugging Face. (2022);1-8.

29. Batra H, Jain A, Bisht G, Srivastava K, Bharadwaj M, Bajaj D, et al.. CoVShorts: news summarization application based on deep NLP transformers for SARS-CoV-2. 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions), ICRITO 2021. (2021). [ Google Scholar]

31. Oniani D, Wang Y. A qualitative evaluation of language models on automatic question-answering for COVID-19. Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020 [Internet]. (2020). [ Google Scholar]

33. Chen M, Tworek J, Jun H, Yuan Q, HPO De Pinto, Kaplan J, et al.. Evaluating Large Language Models Trained on Code. (2021).

35. Castelvecchi D. Are ChatGPT and AlphaCode going to replace programmers? Nature [Internet]. (2022), 4383 doi: 10.1038/d41586-022-04383-z [[ PubMed] [ CrossRef] Google Scholar]

39. Chen Y, Eger S. Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts end-to-end. (2022).

43. EAM Van Dis, Bollen J, Van Rooij R, Zuidema W, Bockting CL. ChatGPT: Five Priorities for Research. 614, 224-226. [[ PubMed] Google Scholar]

45. Gupta A, Singhal A, Mahajan A, Jolly A, Kumar S. Empirical framework for automatic detection of neural and human authored fake news. Proceedings -2022 6th International Conference on Intelligent Computing and Control Systems, ICICCS 2022. (2022);1625-1633. [ Google Scholar]

66. Langguth J, Pogorelov K, Brenner S, Filkuková P, Schroeder DT. Don't trust your eyes: image manipulation in the age of DeepFakes In: Frontiers in Communication. Lausanne: Frontiers Media S.A. (2021). [ Google Scholar]

Articles from Frontiers in Public Health are provided here courtesy of Frontiers Media SA

# data[].attributes.attachment_link.title
ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health

# data[].attributes.attachment_link.description
Large Language Models (LLMs) have recently gathered attention with the release of ChatGPT, a user-centered chatbot released by OpenAI. In this perspective article, we retrace the evolution of LLMs to understand the revolution brought by ChatGPT in the ...

# data[].attributes.body
https://www.jmir.org/2023/1/e46924/

# data[].attributes.attachment_link.readable_text
Original Paper

¹ Department of Neurosurgery and Neurooncology, First Faculty of Medicine, Charles University, Prague, Czech Republic

² Department of Psychiatry, Faculty of Medicine in Pilsen, Charles University, Pilsen, Czech Republic

³ Institute of Biophysics and Informatics, First Faculty of Medicine, Charles University, Prague, Czech Republic

⁴ Department of Methodology, Faculty of Physical Education and Sport, Charles University, Prague, Czech Republic

Corresponding Author:

Martin Májovský, MD, PhD

Department of Neurosurgery and Neurooncology

First Faculty of Medicine

Charles University

U Vojenské nemocnice 1200

Prague, 16000

Czech Republic

Phone: 420 973202963

Email: majovmar@uvn.cz

Background: Artificial intelligence (AI) has advanced substantially in recent years, transforming many industries and improving the way people live and work. In scientific research, AI can enhance the quality and efficiency of data analysis and publication. However, AI has also opened up the possibility of generating high-quality fraudulent papers that are difficult to detect, raising important questions about the integrity of scientific research and the trustworthiness of published papers.

Objective: The aim of this study was to investigate the capabilities of current AI language models in generating high-quality fraudulent medical articles. We hypothesized that modern AI models can create highly convincing fraudulent papers that can easily deceive readers and even experienced researchers.

Methods: This proof-of-concept study used ChatGPT (Chat Generative Pre-trained Transformer) powered by the GPT-3 (Generative Pre-trained Transformer 3) language model to generate a fraudulent scientific article related to neurosurgery. GPT-3 is a large language model developed by OpenAI that uses deep learning algorithms to generate human-like text in response to prompts given by users. The model was trained on a massive corpus of text from the internet and is capable of generating high-quality text in a variety of languages and on various topics. The authors posed questions and prompts to the model and refined them iteratively as the model generated the responses. The goal was to create a completely fabricated article including the abstract, introduction, material and methods, discussion, references, charts, etc. Once the article was generated, it was reviewed for accuracy and coherence by experts in the fields of neurosurgery, psychiatry, and statistics and compared to existing similar articles.

Results: The study found that the AI language model can create a highly convincing fraudulent article that resembled a genuine scientific paper in terms of word usage, sentence structure, and overall composition. The AI-generated article included standard sections such as introduction, material and methods, results, and discussion, as well a data sheet. It consisted of 1992 words and 17 citations, and the whole process of article creation took approximately 1 hour without any special training of the human user. However, there were some concerns and specific mistakes identified in the generated article, specifically in the references.

Conclusions: The study demonstrates the potential of current AI language models to generate completely fabricated scientific articles. Although the papers look sophisticated and seemingly flawless, expert readers may identify semantic inaccuracies and errors upon closer inspection. We highlight the need for increased vigilance and better detection methods to combat the potential misuse of AI in scientific research. At the same time, it is important to recognize the potential benefits of using AI language models in genuine scientific writing and research, such as manuscript preparation and language editing.

J Med Internet Res 2023;25:e46924

doi:10.2196/46924

Keywords

Artificial intelligence (AI) has made substantial advances in recent years, revolutionizing many industries and transforming the way we live and work. In the field of scientific research, AI has the potential to greatly enhance the quality and efficiency of data analysis and publication. However, as with any powerful technology, there is also a dark side to AI that has the potential to cause harm (see for an AI-generated visual representation of this theme []).

One area of concern is the use of AI to create fraudulent scientific papers that appear to be legitimate. Although the use of fraudulent papers is not a new phenomenon, the advent of AI has opened up new possibilities for generating high-quality fraudulent papers in a fraction of the time and making them difficult to detect. This raises important questions about the integrity of scientific research and the trustworthiness of published papers [].

Several studies have demonstrated the potential of AI to generate highly convincing fraudulent nonscientific articles. For instance, in a recent experiment, researchers used an AI language model called GPT-2 (Generative Pre-trained Transformer 2) to generate a fake news article that was accepted for publication by a well-known web-based magazine without being detected as fraudulent []. Similarly, in a study investigating the capabilities of AI language models in generating scientific abstracts, researchers found that the generated abstracts were often indistinguishable from real abstracts and could even fool human reviewers []. To the best of our knowledge, no paper has so far reported on fabricating a whole scientific article using AI.

The aim of this study was to investigate the capabilities of current AI language models in generating high-quality fraudulent medical articles. We hypothesized that modern AI models can create a highly convincing fraudulent paper that can easily deceive readers and even experienced researchers.

‎
Figure 1. An image created with the assistance of artificial intelligence (DALL·E, on May 9, 2023) using the prompt "pandoras box opened with a physician standing next to it. Oil painting Henry Matisse style" [].

General Overview

We used the web-based chat interface ChatGPT (Chat Generative Pre-trained Transformer; OpenAI Limited Partnership), which relies on the GPT-3 (Generative Pre-trained Transformer 3) language model [] to generate a scientific article related to the field of neurosurgery.

ChatGPT is a large language model developed by OpenAI that uses deep learning algorithms to generate human-like text in response to prompts given by users. The model was trained on a massive corpus of text from the internet and is capable of generating high-quality text in a variety of languages and on various topics. The model uses a transformer architecture that allows it to process large amounts of data in parallel and learn complex relationships between words and phrases, enabling it to generate text that is not only coherent but also stylistically consistent with the given prompt.

AI-Generated Article

We selected PLOS Medicine as our target journal for the fabricated article due to its general aim and scope and full open access publishing model. It is very likely that the current version of ChatGPT was trained on full-text articles published in this journal.

We initiated the process by posing the following question: "Suggest relevant RCT in field of neurosurgery that is suitable for aim and scope of PLOS Medicine and would have high chance of acceptance." The reply was as follows:

One relevant randomized controlled trial (RCT) in the field of neurosurgery that may be suitable for the aim and scope of PLOS Medicine and has a high chance of acceptance is:

Title: "Effectiveness of deep brain stimulation for treatment-resistant depression: a randomized controlled trial"

We requested specific sections of the article, such as the abstract, introduction, methods, results, and discussion, along with appropriate citations. Additionally, we asked ChatGPT to provide tables based on the results section and to suggest 1 figure. As an AI language model, ChatGPT does not have graphic design capabilities, so it provided a data sheet, and we created a chart in Microsoft Excel without making any changes to the input data (see Chart 1 in ).

We refined the questions and prompts as the model generated the responses, allowing us to iteratively improve the quality of the output. The whole process of generating the article took no more than 1 hour.

The prompts were as follows:

1. Suggest relevant RCT in field of neurosurgery that is suitable for aim and scope of PLOS Medicine and would have high chance of acceptance.

2. Now give me abstract according to open access articles on PLOS Medicine.

3. Now I want you to make whole article step by step. One section after another section. Give me only introduction section. Use citations by standards of PLOS Medicine. Give me reference list at the end.

4. I want you to be more specific. Use scientific language.

5. Now give me materials and methods section.

6. Now give me detailed results section including patient data.

7. Now I need discussion. compare the results with published articles. Make in-text citations (numbers in square brackets) and give citation list at the end. Start numbering of citations from "9".

8. I need the discussion to be longer - at least twice. Compare our study with similar previous studies. Add more citations. Start numbering of citations from "9".

9. Give me all nine references.

10. PLOS Medicine want to provide "Author summary". It should be bullet Why was this study done?

11. Give me another two bullets on: What did the researchers do and find?

12. I give you result section of an article and you suggest tables to go with it?

13. Can you create some charts? Can you provide datasheet for creating charts?

Although the author who communicated with ChatGPT (MM) is a qualified neurosurgeon, no expert corrections or suggestions were made during the article creation process based on his expertise. Only general hints such as "make this section longer" or "provide a paragraph on statistics" were given.

Neurosurgery, Psychiatry, and Statistical Analysis Reviews

Once we had generated a complete article, we reviewed it for accuracy and coherence, comparing it to existing articles in the field and consulting with domain experts (a psychiatrist and a statistician) to ensure that the content was relevant and accurate.

AI-Generated Review

We also used ChatGPT to review the AI-generated article. The prompts were as follows:

1. Can you create a review of a scientific article as if you were a reviewer? I want you to mention strengths, weaknesses of the article. Then I want you to suggest, what should be improved. Provide examples.

2. I want you to mention strengths, weaknesses of the article.

3. I want you to suggest, what should be improved in manuscript. Study design can not be changed, suggest what information should be added or clarified.

The authors checked the AI-generated review for accuracy and comprehensibility.

AI Detection Tools

We used publicly available web-based tools to identify AI-generated text. Specifically, AI Detector by Content at Scale [] and AI Text Classifier by OpenAI [] were used.

Ethical Considerations

In accordance with current guidelines and regulations, we would like to confirm that this study does not require ethical approval as it exclusively uses publicly available data and does not involve human subjects, animal experiments, or interventions on living organisms.

AI-Generated Article

The result was an article that consisted of an abstract, a main body with standard sections (introduction, material and methods, results, and discussion), tables, and chart. The final manuscript included 1992 words and 17 additional citations. Citations were in the correct format for PLOS Medicine. The process of article creation took about an hour without any special training of the human user. The whole fabricated manuscript is included as .

Neurosurgery Review of AI-Generated Article

A senior professor of neurosurgery (DN) reviewed the AI-generated article with the following remarks:

Overall, the generated article demonstrated a high level of technical proficiency and authenticity. However, we also identified some concerns and specific mistakes. The most noticeable weakness is that the article is shorter in length than what is usual in similar articles and has a limited number of citations. The limited context size of the model may be responsible for this, as the model can only process a fixed amount of information at once. ChatGPT has shown substantial improvement over earlier natural language processing (NLP) models in understanding the contextual relationships between pieces of information that occur at distant places in the text. This is often attributed to its ability to compress previous context and append new information to it. However, despite this progress, the model may still struggle to process information that cannot fit into its embedded latent space representation.
A minor issue is the lack of information regarding whether the study was registered on ClinicalTrials.gov and the absence of an ethical approval number. Another limitation is that the currently available version of ChatGPT was not trained with data after September 2021 and, as a result, is not able to provide information beyond that time (eg, recent citations). When reviewing citations and the reference list, we discovered substantial errors. Although 9 citations were correct in terms of relevance and reference entry, 8 others were flawed (see for detailed information).

Table 1. Citations evaluation.

Reference number

Evaluation

1

Correct

2

Correct

3
Nonexisting citationa

4

Correct

5

Correct

6

Correct

7

Correct

8
Incorrect DOIb of citation

11

Correct

12

Correct

16

Correct

a Incorrect citations are italicized.

b DOI: digital object identifier.

Psychiatry Review of AI-Generated Article

A board certified psychiatrist with interest in deep brain simulation (M Kasal) reviewed the AI-generated article with the following remarks:

From a psychiatric expert point of view, the study could be considered groundbreaking due to the number of subjects and the double-blind study design, which has not been carried out in such an extensive manner before. The largest sets of similar studies included only 25 subjects without a placebo-controlled group []. The criteria for remission and disease response are correctly defined with regard to the questionnaire used, that is, the Hamilton Depression Rating Scale (HDRS), which is commonly used in similar studies. However, the exclusion criteria are not well-defined and are rather vague. The results are comparable to previous studies in terms of symptom reduction as measured by the HDRS. However, the number of responsive patients is substantially higher than the established scientific data to date [].
However, several issues in this study need to be addressed. First, the study lacks a clear definition of treatment-resistant depression (TRD). TRD is defined differently in various studies, and even expert opinions are inconsistent regarding its description. In the case of deep brain stimulation, the recommended procedures often refer to refractory depression, which can be considered a more severe stage of the disease. Although the paper mentions verification according to DSM-V (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition) criteria, it does not provide a specific definition of TRD within this classification []. Second, a major shortcoming of the study is the approach to adverse events. Current trials in this area require detailed evaluation of adverse events, including subtle variations in cognitive functioning. However, the study did not evaluate these outcomes.

Statistical Analysis Review of AI-Generated Article

A senior statistician with a medical degree (M Komarc) reviewed the AI-generated article with the following remarks:

The description of the statistical analysis approach was rather brief; however, it was very clearly formulated and included most of the requirements for a standard scientific text. The sample size required for the analyses was supported by a power analysis, and all the proposed statistical tests were well aligned with the purpose of the study (ie, mixed-effect model for a randomized controlled trial using 1 control and 1 experimental group) and the nature or type of the studied variables (ie, chi-square tests for count variables and t tests for continuous variables). The statistical findings were clearly and concisely presented within the text and tables.
However, the produced Table 2 () was inconsistent, as it did not contain confidence intervals and displayed different mean values than those presented in the results section, although the mean changes (referring to the test of intervention effectiveness) were the same.

AI-Generated Review

The AI-generated review () gave quite accurate remarks regarding the fabricated article, pointed out strengths and weaknesses, and suggested possible changes. Despite the fact that some comments were self-evident (single-center study design and limited follow-up time), there were no substantial errors.

Detection Tools for AI-Generated Manuscript

There are several publicly available web-based tools to identify AI-generated text. For example, AI Detector by Content at Scale claims to detect patterns and forecast the most probable word choices that lead to a higher AI detection probability []. We gave this tool a trial with our AI-generated article and the result was that probability of AI content was 48%, that is, inconclusive.

Another example of such a tool is AI Text Classifier by OpenAI (the same company that developed ChatGPT) []. AI Text Classifier gives result on a scale of very unlikely, unlikely, unclear if it is, possibly, or likely AI generated. Our AI-generated paper was classified as "unclear."

Detection Tools for AI-Generated Review

AI Detector by Content at Scale found that the probability of AI content in the ChatGPT-generated review was 72%, that is, "highly likely to be AI generated." AI Text Classifier by OpenAI classified the ChatGPT-generated review to be "likely AI generated."

Principal Findings

We have demonstrated that AI (ChatGPT) can create a highly convincing medical article that is completely fabricated with limited effort from a human user in a matter of hours. Nevertheless, the article would need an expert review and some improvements to be ready for possible submission. Shortcomings that are mentioned in the results section do not show any specific pattern; they are rather minor inaccuracies and minor study design flaws. Although a substantial number of citations seemed genuine at first glance, they were later found to have been fabricated. To our best knowledge, the errors the AI made were indistinguishable from those that a human could make.

There have been a number of high-profile cases of scientific fraud and misconduct in recent years, including cases where authors have fabricated or manipulated data, plagiarized content, or otherwise misrepresented their findings []. Although AI language models such as ChatGPT are a relatively new tool in scientific writing, it is possible that they could be used in similar ways to create fraudulent content.

ChatGPT is a cutting-edge NLP model developed by OpenAI that uses a transformer architecture to generate high-quality text in response to natural language prompts. Similar to other NLP models, ChatGPT works by analyzing large data sets of natural language text to learn patterns and structures in language, which it can then use to generate new text that is both coherent and contextually relevant.

At its core, ChatGPT is a large neural network that is trained on a massive corpus of text data (until the year 2021), such as books, articles, and web-based content. The model consists of multiple layers of self-attention and feedforward neural networks, which allow it to identify and model complex relationships between words and phrases in natural language text.

To generate new text using ChatGPT, a user provides a natural language prompt or question that the model uses to generate a sequence of tokens representing a coherent and contextually relevant response. The length and complexity of the response can be controlled by adjusting the parameters of the model, such as the length of the input prompt and the temperature of the sampling algorithm used to generate the response.

Although ChatGPT is primarily designed for use in conversational AI and chatbot applications, it has also shown promise in a range of other NLP tasks, including text completion, summarization, and machine translation. In recent years, researchers have also begun exploring the potential of ChatGPT and other NLP models for use in scientific writing and research, including generating scientific papers and summarizing research findings.

Some recent studies suggest that ChatGPT and other NLP models have substantial potential for use in scientific writing and research, particularly for tasks that involve summarizing or generating large volumes of text [].

Some researchers point out that ChatGPT sometimes writes plausible sounding but incorrect or nonsensical answers and that using it for medical writing still requires human judgment []. However, our findings suggest that the level of sophistication required for human input may not be overly complex. An obvious weakness that we encountered in this study is the quality of citations. As technology continues to advance, it is likely that specialized large language models will be developed, reducing their monetary costs and mitigating some of their current limitations.

Interestingly, Kung et al [] evaluated the performance of ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of 3 exams: Step 1, Step 2 Clinical Knowledge, and Step 3. ChatGPT performed at or near the passing threshold for all 3 exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations [].

We are not aware of any specific evidence that ChatGPT has been intentionally misused for fraud in scientific writing, but it is certainly a possibility. Few articles have focused on abstract ghostwriting and its implications for the academic ethics of using AI in manuscript preparation, as well as issues of originality and authorship [-].

An obvious emerging challenge that publishers are facing is the detection of AI-created text. To address this challenge, many publishers are implementing various tools and techniques. One approach involves using machine learning algorithms to analyze the language, structure, and other features of the text to determine whether it was likely to have been created by a human or a machine. As demonstrated above, the current AI detection tools were unable to detect an AI-generated manuscript. However, in the case of an AI-generated review, these tools were more accurate, labeling the text as "likely" or "highly likely" to have been generated by AI. Another approach to address AI-generated content involves developing ethical guidelines and standards, which can help ensure that AI-generated content is transparent and accountable. Despite these challenges, the use of AI in scientific writing is likely to become increasingly common in the future, and publishers will need to continue to adapt and evolve their approaches to ensure the integrity and quality of their publications. An effective measure to prevent fraud as described in this paper (ie, completely fabricated articles) could be the mandatory submission of data sets, potentially verified by local authorities.

As we mentioned earlier, the ability of AI language models such as ChatGPT to generate coherent and realistic text has raised concerns about the potential for their misuse in creating fraudulent or misleading content. To the best of our knowledge, no paper has so far reported on fabricating a whole scientific article using AI.

Conclusion

In conclusion, our experiment using ChatGPT to generate an authentic looking but completely fabricated scientific paper highlights the potential risks associated with the use of AI in scientific writing. Although current AI language models can generate sophisticated and seemingly flawless papers, expert readers may identify semantic inaccuracies and errors upon closer inspection, particularly in the references.

As AI language models continue to advance in their capabilities, it will become increasingly important to develop ethical guidelines and best practices for their use in scientific writing and research. This may include strategies for verifying the accuracy and authenticity of content generated using these tools, as well as mechanisms for detecting and preventing fraud and misconduct.

At the same time, it is important to recognize the potential benefits of using AI language models in scientific writing and research, such as improving the efficiency and accuracy of document creation, analyzing results, and language editing. By approaching these tools with care and responsibility, researchers can harness their power while minimizing the risk of misuse or abuse.

Ultimately, the future of AI in scientific writing and research will depend on how well we navigate these ethical challenges and leverage the full potential of these tools for the benefit of scientific society.

Acknowledgments

This study was supported by the Ministry of Defence of the Czech Republic (grant MO1012) and Cooperatio Neuroscience UK, which was funded by Charles University. The funding sources had no impact on the study design, collection, analysis, and interpretation of data; on the writing of the article; or on the decision to submit the article for publication.

We used the generative AI tool ChatGPT to draft a fabricated article and a review. The original ChatGPT transcripts are made available as -.

Conflicts of Interest

None declared.

Multimedia Appendix 1

The artificial intelligence-generated article from ChatGPT (Chat Generative Pre-trained Transformer).
DOCX File , 31 KB

Multimedia Appendix 2

Full review conversation with ChatGPT (Chat Generative Pre-trained Transformer).
DOCX File , 20 KB

‎

AI: artificial intelligence

ChatGPT: Chat Generative Pre-trained Transformer

DSM-V: Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition

GPT-2: Generative Pre-trained Transformer 2

GPT-3: Generative Pre-trained Transformer 3

HDRS: Hamilton Depression Rating Scale

NLP: natural language processing

TRD: treatment-resistant depression

USMLE: United States Medical Licensing Exam

Edited by T de Azevedo Cardoso; submitted 02.03.23; peer-reviewed by Y Wang, B Chaudhry, W Yang; comments to author 21.04.23; revised version received 25.04.23; accepted 03.05.23; published 31.05.23.

©Martin Májovský, Martin Černý, Matěj Kasal, Martin Komarc, David Netuka. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 31.05.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Background: Artificial intelligence (AI) has advanced substantially in recent years, transforming many industries and improving the way people live and work. In scientific research, AI can enhance the quality and efficiency of data analysis and publication. However, AI has also opened up the possibility of generating high-quality fraudulent papers that are difficult to detect, raising important questions about the integrity of scientific research and the trustworthiness of published papers.

Objective: The aim of this study was to investigate the capabilities of current AI language models in generating high-quality fraudulent medical articles. We hypothesized that modern AI models can create highly convincing fraudulent papers that can easily deceive readers and even experienced researchers.

Methods: This proof-of-concept study used ChatGPT (Chat Generative Pre-trained Transformer) powered by the GPT-3 (Generative Pre-trained Transformer 3) language model to generate a fraudulent scientific article related to neurosurgery. GPT-3 is a large language model developed by OpenAI that uses deep learning algorithms to generate human-like text in response to prompts given by users. The model was trained on a massive corpus of text from the internet and is capable of generating high-quality text in a variety of languages and on various topics. The authors posed questions and prompts to the model and refined them iteratively as the model generated the responses. The goal was to create a completely fabricated article including the abstract, introduction, material and methods, discussion, references, charts, etc. Once the article was generated, it was reviewed for accuracy and coherence by experts in the fields of neurosurgery, psychiatry, and statistics and compared to existing similar articles.

Results: The study found that the AI language model can create a highly convincing fraudulent article that resembled a genuine scientific paper in terms of word usage, sentence structure, and overall composition. The AI-generated article included standard sections such as introduction, material and methods, results, and discussion, as well a data sheet. It consisted of 1992 words and 17 citations, and the whole process of article creation took approximately 1 hour without any special training of the human user. However, there were some concerns and specific mistakes identified in the generated article, specifically in the references.

Conclusions: The study demonstrates the potential of current AI language models to generate completely fabricated scientific articles. Although the papers look sophisticated and seemingly flawless, expert readers may identify semantic inaccuracies and errors upon closer inspection. We highlight the need for increased vigilance and better detection methods to combat the potential misuse of AI in scientific research. At the same time, it is important to recognize the potential benefits of using AI language models in genuine scientific writing and research, such as manuscript preparation and language editing.

J Med Internet Res 2023;25:e46924

doi:10.2196/46924

Keywords

Artificial intelligence (AI) has made substantial advances in recent years, revolutionizing many industries and transforming the way we live and work. In the field of scientific research, AI has the potential to greatly enhance the quality and efficiency of data analysis and publication. However, as with any powerful technology, there is also a dark side to AI that has the potential to cause harm (see for an AI-generated visual representation of this theme []).

One area of concern is the use of AI to create fraudulent scientific papers that appear to be legitimate. Although the use of fraudulent papers is not a new phenomenon, the advent of AI has opened up new possibilities for generating high-quality fraudulent papers in a fraction of the time and making them difficult to detect. This raises important questions about the integrity of scientific research and the trustworthiness of published papers [].

Several studies have demonstrated the potential of AI to generate highly convincing fraudulent nonscientific articles. For instance, in a recent experiment, researchers used an AI language model called GPT-2 (Generative Pre-trained Transformer 2) to generate a fake news article that was accepted for publication by a well-known web-based magazine without being detected as fraudulent []. Similarly, in a study investigating the capabilities of AI language models in generating scientific abstracts, researchers found that the generated abstracts were often indistinguishable from real abstracts and could even fool human reviewers []. To the best of our knowledge, no paper has so far reported on fabricating a whole scientific article using AI.

The aim of this study was to investigate the capabilities of current AI language models in generating high-quality fraudulent medical articles. We hypothesized that modern AI models can create a highly convincing fraudulent paper that can easily deceive readers and even experienced researchers.

‎
Figure 1. An image created with the assistance of artificial intelligence (DALL·E, on May 9, 2023) using the prompt "pandoras box opened with a physician standing next to it. Oil painting Henry Matisse style" [].

General Overview

We used the web-based chat interface ChatGPT (Chat Generative Pre-trained Transformer; OpenAI Limited Partnership), which relies on the GPT-3 (Generative Pre-trained Transformer 3) language model [] to generate a scientific article related to the field of neurosurgery.

ChatGPT is a large language model developed by OpenAI that uses deep learning algorithms to generate human-like text in response to prompts given by users. The model was trained on a massive corpus of text from the internet and is capable of generating high-quality text in a variety of languages and on various topics. The model uses a transformer architecture that allows it to process large amounts of data in parallel and learn complex relationships between words and phrases, enabling it to generate text that is not only coherent but also stylistically consistent with the given prompt.

AI-Generated Article

We selected PLOS Medicine as our target journal for the fabricated article due to its general aim and scope and full open access publishing model. It is very likely that the current version of ChatGPT was trained on full-text articles published in this journal.

We initiated the process by posing the following question: "Suggest relevant RCT in field of neurosurgery that is suitable for aim and scope of PLOS Medicine and would have high chance of acceptance." The reply was as follows:

One relevant randomized controlled trial (RCT) in the field of neurosurgery that may be suitable for the aim and scope of PLOS Medicine and has a high chance of acceptance is:

Title: "Effectiveness of deep brain stimulation for treatment-resistant depression: a randomized controlled trial"

We requested specific sections of the article, such as the abstract, introduction, methods, results, and discussion, along with appropriate citations. Additionally, we asked ChatGPT to provide tables based on the results section and to suggest 1 figure. As an AI language model, ChatGPT does not have graphic design capabilities, so it provided a data sheet, and we created a chart in Microsoft Excel without making any changes to the input data (see Chart 1 in ).

We refined the questions and prompts as the model generated the responses, allowing us to iteratively improve the quality of the output. The whole process of generating the article took no more than 1 hour.

The prompts were as follows:

1. Suggest relevant RCT in field of neurosurgery that is suitable for aim and scope of PLOS Medicine and would have high chance of acceptance.

2. Now give me abstract according to open access articles on PLOS Medicine.

3. Now I want you to make whole article step by step. One section after another section. Give me only introduction section. Use citations by standards of PLOS Medicine. Give me reference list at the end.

4. I want you to be more specific. Use scientific language.

5. Now give me materials and methods section.

6. Now give me detailed results section including patient data.

7. Now I need discussion. compare the results with published articles. Make in-text citations (numbers in square brackets) and give citation list at the end. Start numbering of citations from "9".

8. I need the discussion to be longer - at least twice. Compare our study with similar previous studies. Add more citations. Start numbering of citations from "9".

9. Give me all nine references.

10. PLOS Medicine want to provide "Author summary". It should be bullet Why was this study done?

11. Give me another two bullets on: What did the researchers do and find?

12. I give you result section of an article and you suggest tables to go with it?

13. Can you create some charts? Can you provide datasheet for creating charts?

Although the author who communicated with ChatGPT (MM) is a qualified neurosurgeon, no expert corrections or suggestions were made during the article creation process based on his expertise. Only general hints such as "make this section longer" or "provide a paragraph on statistics" were given.

Neurosurgery, Psychiatry, and Statistical Analysis Reviews

Once we had generated a complete article, we reviewed it for accuracy and coherence, comparing it to existing articles in the field and consulting with domain experts (a psychiatrist and a statistician) to ensure that the content was relevant and accurate.

AI-Generated Review

We also used ChatGPT to review the AI-generated article. The prompts were as follows:

1. Can you create a review of a scientific article as if you were a reviewer? I want you to mention strengths, weaknesses of the article. Then I want you to suggest, what should be improved. Provide examples.

2. I want you to mention strengths, weaknesses of the article.

3. I want you to suggest, what should be improved in manuscript. Study design can not be changed, suggest what information should be added or clarified.

The authors checked the AI-generated review for accuracy and comprehensibility.

AI Detection Tools

We used publicly available web-based tools to identify AI-generated text. Specifically, AI Detector by Content at Scale [] and AI Text Classifier by OpenAI [] were used.

Ethical Considerations

In accordance with current guidelines and regulations, we would like to confirm that this study does not require ethical approval as it exclusively uses publicly available data and does not involve human subjects, animal experiments, or interventions on living organisms.

AI-Generated Article

The result was an article that consisted of an abstract, a main body with standard sections (introduction, material and methods, results, and discussion), tables, and chart. The final manuscript included 1992 words and 17 additional citations. Citations were in the correct format for PLOS Medicine. The process of article creation took about an hour without any special training of the human user. The whole fabricated manuscript is included as .

Neurosurgery Review of AI-Generated Article

A senior professor of neurosurgery (DN) reviewed the AI-generated article with the following remarks:

Overall, the generated article demonstrated a high level of technical proficiency and authenticity. However, we also identified some concerns and specific mistakes. The most noticeable weakness is that the article is shorter in length than what is usual in similar articles and has a limited number of citations. The limited context size of the model may be responsible for this, as the model can only process a fixed amount of information at once. ChatGPT has shown substantial improvement over earlier natural language processing (NLP) models in understanding the contextual relationships between pieces of information that occur at distant places in the text. This is often attributed to its ability to compress previous context and append new information to it. However, despite this progress, the model may still struggle to process information that cannot fit into its embedded latent space representation.
A minor issue is the lack of information regarding whether the study was registered on ClinicalTrials.gov and the absence of an ethical approval number. Another limitation is that the currently available version of ChatGPT was not trained with data after September 2021 and, as a result, is not able to provide information beyond that time (eg, recent citations). When reviewing citations and the reference list, we discovered substantial errors. Although 9 citations were correct in terms of relevance and reference entry, 8 others were flawed (see for detailed information).

Table 1. Citations evaluation.

Reference number

Evaluation

1

Correct

2

Correct

3
Nonexisting citationa

4

Correct

5

Correct

6

Correct

7

Correct

8
Incorrect DOIb of citation

11

Correct

12

Correct

16

Correct

a Incorrect citations are italicized.

b DOI: digital object identifier.

Psychiatry Review of AI-Generated Article

A board certified psychiatrist with interest in deep brain simulation (M Kasal) reviewed the AI-generated article with the following remarks:

From a psychiatric expert point of view, the study could be considered groundbreaking due to the number of subjects and the double-blind study design, which has not been carried out in such an extensive manner before. The largest sets of similar studies included only 25 subjects without a placebo-controlled group []. The criteria for remission and disease response are correctly defined with regard to the questionnaire used, that is, the Hamilton Depression Rating Scale (HDRS), which is commonly used in similar studies. However, the exclusion criteria are not well-defined and are rather vague. The results are comparable to previous studies in terms of symptom reduction as measured by the HDRS. However, the number of responsive patients is substantially higher than the established scientific data to date [].
However, several issues in this study need to be addressed. First, the study lacks a clear definition of treatment-resistant depression (TRD). TRD is defined differently in various studies, and even expert opinions are inconsistent regarding its description. In the case of deep brain stimulation, the recommended procedures often refer to refractory depression, which can be considered a more severe stage of the disease. Although the paper mentions verification according to DSM-V (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition) criteria, it does not provide a specific definition of TRD within this classification []. Second, a major shortcoming of the study is the approach to adverse events. Current trials in this area require detailed evaluation of adverse events, including subtle variations in cognitive functioning. However, the study did not evaluate these outcomes.

Statistical Analysis Review of AI-Generated Article

A senior statistician with a medical degree (M Komarc) reviewed the AI-generated article with the following remarks:

The description of the statistical analysis approach was rather brief; however, it was very clearly formulated and included most of the requirements for a standard scientific text. The sample size required for the analyses was supported by a power analysis, and all the proposed statistical tests were well aligned with the purpose of the study (ie, mixed-effect model for a randomized controlled trial using 1 control and 1 experimental group) and the nature or type of the studied variables (ie, chi-square tests for count variables and t tests for continuous variables). The statistical findings were clearly and concisely presented within the text and tables.
However, the produced Table 2 () was inconsistent, as it did not contain confidence intervals and displayed different mean values than those presented in the results section, although the mean changes (referring to the test of intervention effectiveness) were the same.

AI-Generated Review

The AI-generated review () gave quite accurate remarks regarding the fabricated article, pointed out strengths and weaknesses, and suggested possible changes. Despite the fact that some comments were self-evident (single-center study design and limited follow-up time), there were no substantial errors.

Detection Tools for AI-Generated Manuscript

There are several publicly available web-based tools to identify AI-generated text. For example, AI Detector by Content at Scale claims to detect patterns and forecast the most probable word choices that lead to a higher AI detection probability []. We gave this tool a trial with our AI-generated article and the result was that probability of AI content was 48%, that is, inconclusive.

Another example of such a tool is AI Text Classifier by OpenAI (the same company that developed ChatGPT) []. AI Text Classifier gives result on a scale of very unlikely, unlikely, unclear if it is, possibly, or likely AI generated. Our AI-generated paper was classified as "unclear."

Detection Tools for AI-Generated Review

AI Detector by Content at Scale found that the probability of AI content in the ChatGPT-generated review was 72%, that is, "highly likely to be AI generated." AI Text Classifier by OpenAI classified the ChatGPT-generated review to be "likely AI generated."

Principal Findings

We have demonstrated that AI (ChatGPT) can create a highly convincing medical article that is completely fabricated with limited effort from a human user in a matter of hours. Nevertheless, the article would need an expert review and some improvements to be ready for possible submission. Shortcomings that are mentioned in the results section do not show any specific pattern; they are rather minor inaccuracies and minor study design flaws. Although a substantial number of citations seemed genuine at first glance, they were later found to have been fabricated. To our best knowledge, the errors the AI made were indistinguishable from those that a human could make.

There have been a number of high-profile cases of scientific fraud and misconduct in recent years, including cases where authors have fabricated or manipulated data, plagiarized content, or otherwise misrepresented their findings []. Although AI language models such as ChatGPT are a relatively new tool in scientific writing, it is possible that they could be used in similar ways to create fraudulent content.

ChatGPT is a cutting-edge NLP model developed by OpenAI that uses a transformer architecture to generate high-quality text in response to natural language prompts. Similar to other NLP models, ChatGPT works by analyzing large data sets of natural language text to learn patterns and structures in language, which it can then use to generate new text that is both coherent and contextually relevant.

At its core, ChatGPT is a large neural network that is trained on a massive corpus of text data (until the year 2021), such as books, articles, and web-based content. The model consists of multiple layers of self-attention and feedforward neural networks, which allow it to identify and model complex relationships between words and phrases in natural language text.

To generate new text using ChatGPT, a user provides a natural language prompt or question that the model uses to generate a sequence of tokens representing a coherent and contextually relevant response. The length and complexity of the response can be controlled by adjusting the parameters of the model, such as the length of the input prompt and the temperature of the sampling algorithm used to generate the response.

Although ChatGPT is primarily designed for use in conversational AI and chatbot applications, it has also shown promise in a range of other NLP tasks, including text completion, summarization, and machine translation. In recent years, researchers have also begun exploring the potential of ChatGPT and other NLP models for use in scientific writing and research, including generating scientific papers and summarizing research findings.

Some recent studies suggest that ChatGPT and other NLP models have substantial potential for use in scientific writing and research, particularly for tasks that involve summarizing or generating large volumes of text [].

Some researchers point out that ChatGPT sometimes writes plausible sounding but incorrect or nonsensical answers and that using it for medical writing still requires human judgment []. However, our findings suggest that the level of sophistication required for human input may not be overly complex. An obvious weakness that we encountered in this study is the quality of citations. As technology continues to advance, it is likely that specialized large language models will be developed, reducing their monetary costs and mitigating some of their current limitations.

Interestingly, Kung et al [] evaluated the performance of ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of 3 exams: Step 1, Step 2 Clinical Knowledge, and Step 3. ChatGPT performed at or near the passing threshold for all 3 exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations [].

We are not aware of any specific evidence that ChatGPT has been intentionally misused for fraud in scientific writing, but it is certainly a possibility. Few articles have focused on abstract ghostwriting and its implications for the academic ethics of using AI in manuscript preparation, as well as issues of originality and authorship [-].

An obvious emerging challenge that publishers are facing is the detection of AI-created text. To address this challenge, many publishers are implementing various tools and techniques. One approach involves using machine learning algorithms to analyze the language, structure, and other features of the text to determine whether it was likely to have been created by a human or a machine. As demonstrated above, the current AI detection tools were unable to detect an AI-generated manuscript. However, in the case of an AI-generated review, these tools were more accurate, labeling the text as "likely" or "highly likely" to have been generated by AI. Another approach to address AI-generated content involves developing ethical guidelines and standards, which can help ensure that AI-generated content is transparent and accountable. Despite these challenges, the use of AI in scientific writing is likely to become increasingly common in the future, and publishers will need to continue to adapt and evolve their approaches to ensure the integrity and quality of their publications. An effective measure to prevent fraud as described in this paper (ie, completely fabricated articles) could be the mandatory submission of data sets, potentially verified by local authorities.

As we mentioned earlier, the ability of AI language models such as ChatGPT to generate coherent and realistic text has raised concerns about the potential for their misuse in creating fraudulent or misleading content. To the best of our knowledge, no paper has so far reported on fabricating a whole scientific article using AI.

Conclusion

In conclusion, our experiment using ChatGPT to generate an authentic looking but completely fabricated scientific paper highlights the potential risks associated with the use of AI in scientific writing. Although current AI language models can generate sophisticated and seemingly flawless papers, expert readers may identify semantic inaccuracies and errors upon closer inspection, particularly in the references.

As AI language models continue to advance in their capabilities, it will become increasingly important to develop ethical guidelines and best practices for their use in scientific writing and research. This may include strategies for verifying the accuracy and authenticity of content generated using these tools, as well as mechanisms for detecting and preventing fraud and misconduct.

At the same time, it is important to recognize the potential benefits of using AI language models in scientific writing and research, such as improving the efficiency and accuracy of document creation, analyzing results, and language editing. By approaching these tools with care and responsibility, researchers can harness their power while minimizing the risk of misuse or abuse.

Ultimately, the future of AI in scientific writing and research will depend on how well we navigate these ethical challenges and leverage the full potential of these tools for the benefit of scientific society.

# data[].attributes.attachment_link.keywords[]
ethics; artificial intelligence; neurosurgery; publications; chatgpt; language models; fraudulent medical articles

# data[].attributes.attachment_link.title
Artificial Intelligence Can Generate Fraudulent but Authentic-Looking Scientific Medical Articles: Pandora's Box Has Been Opened

# data[].attributes.attachment_link.description
Background: Artificial intelligence (AI) has advanced substantially in recent years, transforming many industries and improving the way people live and work. In scientific research, AI can enhance the quality and efficiency of data analysis and publication.