Faculty Help: Generative AI Resource Guide: AI Detectors

Motivated by the growing role of AI in text generation and the potential misuse of generative tools, this study investigates key features that differentiate AI-generated text from human-authored content. We produce a corpus of AI-generated counterparts to 2.100 research paper abstracts, in order to compare formal linguistic and stylometric characteristics such as perplexity, grammar, n-gram distributions and function word frequencies between human- and AI-generated texts. Key findings indicate that human-written abstracts tend to exhibit higher perplexity, greater grammatical error, and more diverse n-gram distributions. To distinguish between the two types of texts we employ various machine learning algorithms, with our Random Forest implementation achieving a precision of 0.986 on unseen data. Notably, feature importance analysis reveals that perplexity, grammar, and n-gram distributions are highly influential in AI-detection classification. Our research contributes a nuanced study of discriminating characteristics of AI-generated text to the increasingly important field of AI authorship attribution.

The article highlights that current AI-generated text detectors, including those using watermarking techniques and neural network-based methods, are not reliable in practical scenarios. The authors demonstrate that these detectors can be easily circumvented through methods like recursive paraphrasing, which only slightly degrades text quality. Even language models protected by watermarking schemes are susceptible to spoofing attacks. These attacks can mislead detectors to incorrectly classify human-written text as AI-generated, posing risks of reputational damage to developers and highlighting the fundamental challenges in reliably detecting AI-generated text.

The results of this study support three main conclusions: Three AI text detectors – Copyleaks, TurnItIn, and Originality – have very high accuracy with all three sets of documents examined for this study: GPT-3.5 papers, GPT-4 papers, and human-generated papers. Most of the other detectors can distinguish between GPT-3.5 papers and human-generated papers with reasonably high accuracy. However, most are ineffective at distinguishing between GPT-4 papers and papers written by students. In general, a detector’s free or paid status is not a good indicator of its accuracy, nor is its appearance on the “best AI text detector” lists considered here.

Our tests reveal that Turnitin’s summative score does a reasonably good job of correctly identifying texts that are entirely or nearly entirely written by humans.

Turnitin is not a plagiarism detector, but you are the detector! Turnitin only matches text as Similarity Index, it does not check for plagiarism. Your judgment is crucial to interpreting the similarity report and detecting potential academic misconduct.

The study reveals that although the detection tool identified 91% of the experimental submissions as containing some AI-generated content, the total detected content was only 54.8%. This suggests that the use of adversarial techniques regarding prompt engineering is an effective method in evading AI detection tools and highlights that improvements to AI detection software are needed. Using the Turnitin AI detect tool, faculty reported 54.5% of the experimental submissions to the academic misconduct process, suggesting the need for increased awareness and training into these tools. Genuine submissions received a mean score of 54.4, whereas AI-generated content scored 52.3, indicating the comparable performance of GPT-4 in real-life situations.

Turnitin plagiarism detection software demonstrated the highest average similarity detection, followed by iThenticate plagiarism detection software, and then Ouriginal plagiarism detection software.

The results of this study revealed that the average accuracy of AI text detectors in identifying non-manipulated AI-generated content was 39.5%, with a 67% accuracy rate for human-written control samples. When adversarial techniques were applied to the AI-generated samples, the average accuracy of the detectors dropped further to 22.14%, with some techniques, such as adding spelling errors and increasing burstiness, proving highly effective in evading detection. Error analysis also highlighted the risk of false accusations and undetected cases. These findings underscore the limitations of current AI text-detection tools in accurately determining the authorship of a given piece of text, particularly when faced with deliberate attempts to obscure the nature of the sample.”

Only two of the 30 tested, free-to-use, AI detectors, Copyleaks and Undetectable AI, did manage to correctly detect all of the student essay sets of the two English language categorie…as human-written.” “…the results of the current study suggest that the bulk of the currently available AI detectors, especially the currently available free-to-use AI detectors, are not fit for purpose.

While the findings demonstrate the potential of Turnitin’s AI detection tool in supporting academic staff in detecting AI-generated content, the relatively low detection accuracy of the participants underscores the need for further training and awareness.” “The results demonstrated that the Turnitin AI detection tool was not particularly robust to the use of these adversarial techniques, and there will likely be a continued ‘arms race’ between the techniques employed to evade detection and the tools designed to detect GenAI content.

This paper exposes serious limitations of the state-of-the-art AI-generated text detection tools and their unsuitability for use as evidence of academic misconduct. Our findings do not confirm the claims presented by the systems. Tey too often present false positives and false negatives. Moreover, it is too easy to game the systems by using paraphrasing tools or machine translation. Therefore, our conclusion is that the systems we
tested should not be used in academic settings. Although text matching software also suffers from false positives and false negatives (Foltýnek et al. 2020), at least it is possible to provide evidence of potential misconduct. In the case of the detection tools for AI-generated text, this is not the case.

Text analysis is conducted on articles from 13 chemistry journals vs. ChatGPT The classification method uses 20 text features and XGBoost The model effectively discriminates human scientific writing from GPT-3.5 and GPT-4 AI text from prompts designed to obfuscate AI use is classified with 99% accuracy.

This study explores the effectiveness of various AI content detection tools in distinguishing between human and AI-generated content, focusing on ChatGPT Models 3.5 and 4. Using fifteen paragraphs from each model on engineering cooling towers and five human-written responses as controls, tools from OpenAI, Writer, Copyleaks, GPTZero, and CrossPlag were evaluated. Results indicate that these tools more accurately identified content from GPT 3.5 than from GPT 4, but struggled with human-written responses, often producing false positives and uncertain classifications. This highlights the necessity for ongoing development in AI detection technologies as AI-generated content continues to evolve.

The release of ChatGPT marked the beginning of a new era of AI-assisted plagiarism that disrupts traditional assessment practices in ESL composition. In the face of this challenge, educators are left with little guidance in controlling AI-assisted plagiarism, especially when conventional methods fail to detect AI-generated texts. One approach to managing AI-assisted plagiarism is using fine-tuned AI classifiers, such as RoBERTa, to identify machine-generated texts; however, the reliability of this approach is yet to be established. To address the challenge of AI-assisted plagiarism in ESL contexts, the present cross-disciplinary descriptive study examined the potential of two RoBERTa-based classifiers to control AI-assisted plagiarism on a dataset of 240 human-written and ChatGPT-generated essays. Data analysis revealed that both platforms could identify AI-generated texts, but their detection accuracy was inconsistent across the dataset.