Something for academic colleagues interested in the new TurnItIn AI detection capability. Has anyone else tried a similar analysis to the one I describe below? More data would be helpful.
(For others, this is a commercial platform widely used for student assignment submission as it is strong on plagiarism detection. And now they claim (with caveats) that they have software that can detect the use of AI).
At my workplace, UNSW, generative AI is allowed to be used in specified coursework, and if used, its use must be correctly attributed.
I just analysed ~100 science student assignments where attributed use of genAI was allowed, to compare those that declared the use of genAI, to the results of the new TurnItIn AI detection function.
If I believe the results, there were 76% correct detections of AI not being used and 9% correct detections of declared AI use.
5% of assignments had AI declared/cited and were not detected.
10% of assignments had AI detected by TurnItIn but not declared/referenced by the students. Some of this I would think is poor citation / attribution of use by the students, but some I suspect are false positives.
@Andbaker Really interesting. What is Turnitin providing in terms of analysis, eg does it highlight specific sections as it does for other sources? Does it provide any reasoning attached to the highlights? What did you think from looking at the positives yourself?
I wonder if you could run an anonymous survey asking students if they had used it, and see how the %s line up.
@jroper
yes, it highlights the text for marker to see, but the students don't see any detail at all, not even the AI percentage score. No reasoning is given to the marker.
All my assignments are set up to allow students to use genAI and the pros and cons are openly discussed in class. So I can ignore the TurnItIn AI check. That seems to be a good thing if this is a typical false positive and negative output.
@jroper
Yes, fairly confident is a good summary.
The same class submits another assignment Friday week. genAI is allowed and I'll stress the importance of correct attribution. That should be a useful comparison.