Faculty Help: ChatGPT Comprehensive Resource Guide: AI Detectors

The results of this study support three main conclusions: Three AI text detectors – Copyleaks, TurnItIn, and Originality – have very high accuracy with all three sets of documents examined for this study: GPT-3.5 papers, GPT-4 papers, and human-generated papers. Most of the other detectors can distinguish between GPT-3.5 papers and human-generated papers with reasonably high accuracy. However, most are ineffective at distinguishing between GPT-4 papers and papers written by students. In general, a detector’s free or paid status is not a good indicator of its accuracy, nor is its appearance on the “best AI text detector” lists considered here.

Our tests reveal that Turnitin’s summative score does a reasonably good job of correctly identifying texts that are entirely or nearly entirely written by humans.

Turnitin is not a plagiarism detector, but you are the detector! Turnitin only matches text as Similarity Index, it does not check for plagiarism. Your judgment is crucial to interpreting the similarity report and detecting potential academic misconduct.

The study reveals that although the detection tool identified 91% of the experimental submissions as containing some AI-generated content, the total detected content was only 54.8%. This suggests that the use of adversarial techniques regarding prompt engineering is an effective method in evading AI detection tools and highlights that improvements to AI detection software are needed. Using the Turnitin AI detect tool, faculty reported 54.5% of the experimental submissions to the academic misconduct process, suggesting the need for increased awareness and training into these tools. Genuine submissions received a mean score of 54.4, whereas AI-generated content scored 52.3, indicating the comparable performance of GPT-4 in real-life situations.

Turnitin plagiarism detection software demonstrated the highest average similarity detection, followed by iThenticate plagiarism detection software, and then Ouriginal plagiarism detection software.

The results of this study revealed that the average accuracy of AI text detectors in identifying non-manipulated AI-generated content was 39.5%, with a 67% accuracy rate for human-written control samples. When adversarial techniques were applied to the AI-generated samples, the average accuracy of the detectors dropped further to 22.14%, with some techniques, such as adding spelling errors and increasing burstiness, proving highly effective in evading detection. Error analysis also highlighted the risk of false accusations and undetected cases. These findings underscore the limitations of current AI text-detection tools in accurately determining the authorship of a given piece of text, particularly when faced with deliberate attempts to obscure the nature of the sample.”

Only two of the 30 tested, free-to-use, AI detectors, Copyleaks and Undetectable AI, did manage to correctly detect all of the student essay sets of the two English language categorie…as human-written.” “…the results of the current study suggest that the bulk of the currently available AI detectors, especially the currently available free-to-use AI detectors, are not fit for purpose.

While the findings demonstrate the potential of Turnitin’s AI detection tool in supporting academic staff in detecting AI-generated content, the relatively low detection accuracy of the participants underscores the need for further training and awareness.” “The results demonstrated that the Turnitin AI detection tool was not particularly robust to the use of these adversarial techniques, and there will likely be a continued ‘arms race’ between the techniques employed to evade detection and the tools designed to detect GenAI content.

This paper exposes serious limitations of the state-of-the-art AI-generated text detection tools and their unsuitability for use as evidence of academic misconduct. Our findings do not confirm the claims presented by the systems. Tey too often present false positives and false negatives. Moreover, it is too easy to game the systems by using paraphrasing tools or machine translation. Therefore, our conclusion is that the systems we
tested should not be used in academic settings. Although text matching software also suffers from false positives and false negatives (Foltýnek et al. 2020), at least it is possible to provide evidence of potential misconduct. In the case of the detection tools for AI-generated text, this is not the case.