Research Guides: Faculty Help: Generative AI Resource Guide: AI Detectors

AI Writing Detectors

Current research shows AI-generated text detection tools to be unreliable, inconsistent, and vulnerable to manipulation. Studies show these tools often produce conflicting results, even when analyzing similar content, and their accuracy can vary widely across contexts. They struggle with both false positives and false negatives, especially when faced with adversarial techniques like paraphrasing, spelling errors, or stylistic prompts. Some tools also exhibit bias, disproportionately flagging texts by non-native English speakers. Additionally, detectors often misclassify AI-polished human writing, highlighting their difficulty in gauging nuanced AI involvement. While a few tools show relatively better performance in isolated studies, no single detector is consistently effective. Given the fast-paced evolution of generative AI and detection tools, many researchers now consider reliable detection increasingly mathematically infeasible. Experts recommend a cautious, multi-layered approach, blending detection tools with human judgment rather than relying solely on automation.

AI detection tools show inconsistent and unreliable performance across studies.
Accuracy is impacted by adversarial techniques such as paraphrasing or formatting changes.
Tools may exhibit bias, especially against non-native English writers.
AI detectors often flag human text polished by AI as AI-generated.
No tool is universally effective; even top performers vary by study.
Reliable detection of AI text is considered mathematically difficult.
Human oversight remains essential for responsible evaluation.

NOTE: The above text was generated by Google NotebookLM, based off of all studies referenced in this section, then summarized by ChatGPT 40.

Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing | arXiv | May 2025
The growing use of large language models (LLMs) for text generation has led to widespread concerns about AI-generated content detection. However, an overlooked challenge is AI-polished text, where human-written content undergoes subtle refinements using AI tools. This raises a critical question: should minimally polished text be classified as AI-generated? Such classification can lead to false plagiarism accusations and misleading claims about AI prevalence in online content. In this study, we systematically evaluate twelve state-of-the-art AI-text detectors using our AI-Polished-Text Evaluation (APT-Eval) dataset, which contains 14.7K samples refined at varying AI-involvement levels. Our findings reveal that detectors frequently flag even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models. These limitations highlight the urgent need for more nuanced detection methodologies.
Are AI Detectors Good Enough | arXiv | March 2025
The rapid development of autoregressive Large Language Models (LLMs) has significantly improved the quality of generated texts, necessitating reliable machine-generated text detectors. A huge number of detectors and collections with AI fragments have emerged, and several detection methods even showed recognition quality up to 99.9% according to the target metrics in such collections. However, the quality of such detectors tends to drop dramatically in the wild, posing a question: Are detectors actually highly trustworthy or do their high benchmark scores come from the poor quality of evaluation datasets? In this paper, we emphasise the need for robust and qualitative methods for evaluating generated data to be secure against bias and low generalising ability of future model. We present a systematic review of datasets from competitions dedicated to AI-generated content detection and propose methods for evaluating the quality of datasets containing AI-generated fragments. In addition, we discuss the possibility of using high-quality generated data to achieve two goals: improving the training of detection models and improving the training datasets themselves. Our contribution aims to facilitate a better understanding of the dynamics between human and machine text, which will ultimately support the integrity of information in an
Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing | arXiv | Feb. 2025
The growing use of large language models (LLMs) for text generation has led to widespread concerns about AI-generated content detection. However, an overlooked challenge is AI-polished text, where human-written content undergoes subtle refinements using AI tools. This raises a critical question: should minimally polished text be classified as AI-generated? Misclassification can lead to false plagiarism accusations and misleading claims about AI prevalence in online content. In this study, we systematically evaluate eleven state-of-the-art AI-text detectors using our AI-Polished-Text Evaluation (APT-Eval) dataset, which contains 11.7K samples refined at varying AI-involvement levels. Our findings reveal that detectors frequently misclassify even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models. These limitations highlight the urgent need for more nuanced detection methodologies.
AI vs AI: How effective are Turnitin, ZeroGPT, GPTZero, and Writer AI in detecting text generated by ChatGPT, Perplexity, and Gemini? | J. of Applied Learning & Teaching | Jam. 2025
AI chatbots and LLMs have made a significant impact in a short time. Despite their benefits, they pose serious threats to academic integrity and ethics by generating human-like text, which is very hard to detect. Various AI-detection tools have been developed to tackle this issue. However, their effectiveness is questionable. This study investigates the performance of four AI-detection tools (Turnitin, ZeroGPT, GPTZero, and Writer AI) in detecting AI-generated text. That text was generated using three LLMs (ChatGPT, Perplexity, and Gemini). Furthermore, three adversarial techniques (edited through Grammarly, paraphrased through Quillbot, and 10%-20% editing by a human expert) were applied to see their effects on the performance of AI-detection tools. Turnitin turned out to be the most accurate and consistent one, with a 100% AI score even with the adversarial techniques. ZeroGPT and GPTZero also reported relatively high AI scores, especially with the original files and the first and third adversarial techniques. Among the three adversarial techniques, paraphrasing through Quillbot affected the performance of three AI-detection tools (ZeroGPT, GPTZero, and Writer AI) the most. Among the three LLMs, text generated through Perplexity was more accurately detected, while Gemini-generated text showed a relatively lower AI score. What was the most note-worthy was the fact that in many cases, even when the text was generated through the same LLM, and detected through the same AI-detection tool; different files showed different AI scores, further highlighting the inconsistencies among AI-detection tools.
Simple techniques to bypass GenAI text detectors: implications for inclusive education | Int. J. of Educational Technology in Higher Education | Sept. 2024
This study investigates the efficacy of six major Generative AI (GenAI) text detectors when confronted with machine-generated content modified to evade detection (n = 805). We compare these detectors to assess their reliability in identifying AI-generated text in educational settings, where they are increasingly used to address academic integrity concerns. Results show significant reductions in detector accuracy (17.4%) when faced with simple techniques to manipulate the AI generated content. The varying performances of GenAI tools and detectors indicate they cannot currently be recommended for determining academic integrity violations due to accuracy limitations and the potential for false accusation which undermines inclusive and fair assessment practices. However, these tools may support learning and academic integrity when used non-punitively. This study aims to guide educators and institutions in the critical implementation of AI text detectors in higher education, highlighting the importance of exploring alternatives to maintain inclusivity in the face of emerging technologies.
Generative AI detection in higher education assessments | Wiley | Sept. 2024
This chapter presents a critical analysis of generative AI (GenAI) detection tools in higher education assessments. The rapid advancement and widespread adoption of GenAI, particularly in education, necessitates a reevaluation of traditional academic integrity mechanisms. I explore the effectiveness, vulnerabilities, and ethical implications of AI detection tools in the context of preserving academic integrity. My analysis synthesizes insights from various case studies, newspaper articles, and student testimonies to scrutinize the practical and philosophical challenges associated with AI detection. I argue that reliance on detection mechanisms is misaligned with the educational landscape, where AI plays an increasing role. I advocate for a strategic shift toward robust assessment methods and educational policies that embrace GenAI usage while ensuring academic integrity and authenticity in assessments.
Detecting ChatGPT-generated essays in a large-scale writing assessment: Is there a bias against non-native English speakers? | Computers & Education | Aug. 2024
With the prevalence of generative AI tools like ChatGPT, automated detectors of AI-generated texts have been increasingly used in education to detect the misuse of these tools (e.g., cheating in assessments). Recently, the responsible use of these detectors has attracted a lot of attention. Research has shown that publicly available detectors are more likely to misclassify essays written by non-native English speakers as AI-generated than those written by native English speakers. In this study, we address these concerns by leveraging carefully sampled large-scale data from the Graduate Record Examinations (GRE) writing assessment. We developed multiple detectors of ChatGPT-generated essays based on linguistic features from the ETS e-rater engine and text perplexity features, and investigated their performance and potential bias. Results showed that our carefully constructed detectors not only achieved near-perfect detection accuracy, but also showed no evidence of bias disadvantaging non-native English speakers. Findings of this study contribute to the ongoing debates surrounding the formulation of policies for utilizing AI-generated content detectors in education.
Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Text | arXiv | June 2024
Large Language Models (LLMs) have revolutionized the field of Natural Language Generation (NLG) by demonstrating an impressive ability to generate human-like text. However, their widespread usage introduces challenges that necessitate thoughtful examination, ethical scrutiny, and responsible practices. In this study, we delve into these challenges, explore existing strategies for mitigating them, with a particular emphasis on identifying AI-generated text as the ultimate solution. Additionally, we assess the feasibility of detection from a theoretical perspective and propose novel research directions to address the current limitations in this domain
Accuracy pecking order – How 30 AI detectors stack up in detecting generative artificial intelligence content in university English L1 and English L2 student essays | Journal of Applied Learning & Teaching | April 2024
Only two of the 30 tested, free-to-use, AI detectors, Copyleaks and Undetectable AI, did manage to correctly detect all of the student essay sets of the two English language categorie…as human-written.” “…the results of the current study suggest that the bulk of the currently available AI detectors, especially the currently available free-to-use AI detectors, are not fit for purpose.
GenAI Detection Tools, Adversarial Techniques and Implications for Inclusivity in Higher Education | arxiv | March 2024
The results of this study revealed that the average accuracy of AI text detectors in identifying non-manipulated AI-generated content was 39.5%, with a 67% accuracy rate for human-written control samples. When adversarial techniques were applied to the AI-generated samples, the average accuracy of the detectors dropped further to 22.14%, with some techniques, such as adding spelling errors and increasing burstiness, proving highly effective in evading detection. Error analysis also highlighted the risk of false accusations and undetected cases. These findings underscore the limitations of current AI text-detection tools in accurately determining the authorship of a given piece of text, particularly when faced with deliberate attempts to obscure the nature of the sample.”
Raidar: geneRative AI Detection viA Rewriting | arXiv \ April 2024
We find that large language models (LLMs) are more likely to modify human-written text than AI-generated text when tasked with rewriting. This tendency arises because LLMs often perceive AI-generated text as high-quality, leading to fewer modifications. We introduce a method to detect AI-generated content by prompting LLMs to rewrite text and calculating the editing distance of the output. We dubbed our geneRative AI Detection viA Rewriting method Raidar. Raidar significantly improves the F1 detection scores of existing AI content detection models -- both academic and commercial -- across various domains, including News, creative writing, student essays, code, Yelp reviews, and arXiv papers, with gains of up to 29 points. Operating solely on word symbols without high-dimensional features, our method is compatible with black box LLMs, and is inherently robust on new content. Our results illustrate the unique imprint of machine-generated text through the lens of the machines themselves.
Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text? | ACM Digital Library | March 2024
Large language models (LLMs) such as ChatGPT are increasingly being used for various use cases, including text content generation at scale. Although detection methods for such AI-generated text exist already, we investigate ChatGPT's performance as a detector on such AI-generated text, inspired by works that use ChatGPT as a data labeler or annotator. We evaluate the zeroshot performance of ChatGPT in the task of human-written vs. AI-generated text detection, and perform experiments on publicly available datasets. We empirically investigate if ChatGPT is symmetrically effective in detecting AI-generated or human-written text. Our findings provide insight on how ChatGPT and similar LLMs may be leveraged in automated detection pipelines by simply focusing on solving a specific aspect of the problem and deriving the rest from that solution. All code and data is available at https://github.com/AmritaBh/ChatGPT-as-Detector.
Can AI-Generated Text be Reliably Detected? | arxiv | Feb 2024
The article highlights that current AI-generated text detectors, including those using watermarking techniques and neural network-based methods, are not reliable in practical scenarios. The authors demonstrate that these detectors can be easily circumvented through methods like recursive paraphrasing, which only slightly degrades text quality. Even language models protected by watermarking schemes are susceptible to spoofing attacks. These attacks can mislead detectors to incorrectly classify human-written text as AI-generated, posing risks of reputational damage to developers and highlighting the fundamental challenges in reliably detecting AI-generated text.
Comparing the similarity index across iThenticate, Ouriginal, and Turnitin plagiarism detection software | Journal of Physical Education and Sport | Feb 2024
Turnitin plagiarism detection software demonstrated the highest average similarity detection, followed by iThenticate plagiarism detection software, and then Ouriginal plagiarism detection software.
Turnitin Guide to Interpreting Similarity Reports for Academics | Wits University | 2024
Turnitin is not a plagiarism detector, but you are the detector! Turnitin only matches text as Similarity Index, it does not check for plagiarism. Your judgment is crucial to interpreting the similarity report and detecting potential academic misconduct.
Assessing AI Detectors in Identifying AI-Generated Code | ICSE-SEET '24: Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training | May 2024
Educators are increasingly concerned about the usage of Large Language Models (LLMs) such as ChatGPT in programming education, particularly regarding the potential exploitation of imperfections in Artificial Intelligence Generated Content (AIGC) Detectors for academic misconduct. In this paper, we present an empirical study where the LLM is examined for its attempts to bypass detection by AIGC Detectors. This is achieved by generating code in response to a given question using different variants. We collected a dataset comprising 5,069 samples, with each sample consisting of a textual description of a coding problem and its corresponding human-written Python solution codes. These samples were obtained from various sources, including 80 from Quescol, 3,264 from Kaggle, and 1,725 from Leet-Code. From the dataset, we created 13 sets of code problem variant prompts, which were used to instruct ChatGPT to generate the outputs. Subsequently, we assessed the performance of five AIGC detectors. Our results demonstrate that existing AIGC Detectors perform poorly in distinguishing between human-written code and AI-generated code.
Evaluating the Effectiveness of Turnitin’s AI Writing Indicator Model | Temple University
Our tests reveal that Turnitin’s summative score does a reasonably good job of correctly identifying texts that are entirely or nearly entirely written by humans.
Reviewing the performance of AI detection tools in differentiating between AI-generated and human-written texts: A literature and integrative hybrid review | J. of Applied Learning & Teaching | Feb. 2024
The purpose of this study was to review 17 articles published between January 2023 and November 2023 that dealt with the performance of AI detectors in differentiating between AI-generated and human- written texts. Employing a slightly modified version of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) protocol and an aggregated set of quality evaluation criteria adapted from A MeaSurement Tool to Assess systematic Reviews (AMSTAR) tool, the study was conducted from 1 October 2023 to 30 November 2023 and guided by six research questions. The study conducted its searches on eleven online databases, two Internet search engines, and one academic social networking site. The geolocation and authorship of the 17 reviewed articles were spread across twelve countries in both the Global North and the Global South. ChatGPT (in its two versions, GPT- 3.5 and GPT-4) was the sole AI text generator used or was one of the AI text generators in instances where more than one AI text generator had been used. Crossplag was the top-performing AI detection tool, followed by Copyleaks. Duplichecker and Writer were the worst-performing AI detection tools in instances in which they had been used. One of the major aspects flagged by the main findings of the 17 reviewed articles is the inconsistency of the detection efficacy of all the tested AI detectors and all the tested anti-plagiarism detection tools. Both sets of detection tools were found to lack detection reliability. As a result, this study recommends utilising both contemporary AI detectors and traditional anti-plagiarism detection tools, together with human reviewers/raters, in an ongoing search for differentiating between AI-generated and human- written texts.

Listing and linking to these resources does not indicate SFCC Library's endorsement of said resources (Editor's Note: I've actually seriously considered deleting this section altogether due to the controversy surrounding the use of these resources, but...)

Please keep in mind that the efficacy of each platform is not consistent. No one platform is 100% foolproof.
Sources are listed in alphabetical order
Many of these are free but simply require you to set up an account. But yes - some only allow a free trial period.

♦ means a score of 100% on David Gewirtz' AI Detector Test (Senior Contributing Editor for ZDNet)

What to do if you suspect unsanctioned use of Generative AI

Beforehand (“An ounce of prevention…”)
- Develop a Generative AI use policy for your course syllabus and/or assignments
- Draw students’ attention to that policy
- Talk openly with students about Generative AI
- Collect periodic writing samples from students to familiarize yourself with their writing style and voice
- Require students to provide links to all sources, and randomly spot-check those links
  (Some Generative AI platforms can now provide real, legitimate links to so-called 'sources'; however, those links often do not match the generated 'source').
Tell the student why you believe they may have used Generative AI in a way they were not supposed to
- Do you see phrasing in their writing that clearly indicates a Generative AI platform wrote it?
  - “I’m sorry but as a Large Language Model, I can’t….”
  - “Certainly! I’m happy to write that essay for you!”
- Is their writing style or vocabulary unexpectedly different than you’ve ever seen it?
- Does their writing not address the question or prompt in a way you’d expect?
Engage the student in a conversation about their work
- Are they able to engage and converse with you about their work or do they have trouble recalling key aspects?
- Ask the student to discuss both their thought and writing processes. Are they able to do this?
- Can they define terms/words that you believe may have been provided by Generative AI?
- Document this interaction.
Please consider very carefully before...
- Using AI detection software.
- Assuming that use of words like “delve,” “tapestry,” “landscape” etc automatically means the student used a Generative AI tool. Rather, compare the writing style with various student writing samples.
Further action needed?
- Consult your Chair or Dean
- Refer to the Student Code of Conduct - Policy 2-1

AI Humanizers - What are they?

The increasing use of Generative AI by students and faculty efforts to counter it have often been described as an arms race. One of the latest weapons in this race are AI 'humanizer' writing websites.

What are they?
AI humanizer writing websites are tools designed to make AI-generated text sound more natural, human-like, and less detectable as machine-written. They work by taking content created by an AI (like ChatGPT or similar tools) and rewriting or editing it to:

Improve tone and flow
Add natural language patterns (e.g., contractions, idioms, variability)
Avoid common structures or phrasing that AI detectors flag

Some use rule-based methods (applying specific linguistic tweaks), while others use additional AI models trained to mimic human writing styles. These tools are often used to bypass AI detection tools or improve readability.

At the time of this writing (Spring 2025) some of the more popular AI humanizer websites are AIHumanizer, WriteHuman, Humanize AI and AI Undetect but there are hundreds out there.

To address suspected AI humanizer use in student essays:

Detection Strategies

Analyze writing patterns: Humanizers may correct grammar but leave overly uniform tone or lack authentic emotional shifts. Compare current work to past submissions for sudden style changes.
Require process documentation: Ask for drafts, outlines, and AI prompts used. Verify consistency between stages.
Check contextual depth: Humanized text often remains superficial or misses assignment-specific details (e.g., personal observations, niche citations).

Conversation Approaches

Ask open-ended questions: “Walk me through your research process” or “How did you develop this argument?” Inability to discuss specifics may indicate AI use.
Focus on learning: Frame violations as growth opportunities. Discuss time management, citation norms, and the value of original thought.

Policy Adjustments

Explicitly ban humanizers in syllabi and define consequences.
Assign AI-proof tasks: Incorporate real-world observations, class discussions, or reflective elements.

Tools and Workflow

Combine AI detectors (e.g., GPTZero) with plagiarism checkers, as humanizers often paraphrase.
Use version history tracking in tools like Google Docs to monitor edits.

AI Detector Pro	BrandWell	Checker AI	Copyleaks	CrossPlag
GPTKit	GPT-2 Output	GPTZero	GPT Radar
	Grammarly Authorship	Monica ♦	Originality.ai ♦	PlagiarismCheck
QuillBot ♦	Sapling	Scribbr	SynthID	Turnitin
UndetectableAI ♦	Winston		Writer AI Content Detector	ZeroGPT ♦