On AI Detectors Regarding College Applications

post by Kaustubh Kislay (kaustubh-kislay) · 2024-11-27T20:25:48.151Z · LW · GW · 2 comments

Contents

  What are AI Detectors
  The Flaws
  How should you use AI Detectors
  How should Admission Officers use AI Detectors
None
2 comments

What are AI Detectors

You've probably already used them before, websites like GPTZero, ZeroGPT, Grammarly, Quilbot, all have their own AI Detectors. AI Detectors can be a combination of Pre-trained LLMS, Statistical Models, and Ml models using NLP(Natural Language Processing). The model will analyze linguistic patterns, sentence structures, and statistical measures like perplexity, the predictability of text, and burstiness, the sentence variety, to distinguish AI-generated content. 

The Flaws

How should you use AI Detectors

  1. do not use AI to write full essays in the first place(for now). If you only sprinkle in AI there is no reason as to why someone should be able to make a reasonable accusation of your writing being AI. But even better just write the essay yourself and don't stress since you are a honest writer.
  2. Use multiple detectors. AI detectors generally share the same evaluation techniques but there are key differences between each detector that make it unique. Use maybe 3 or 4 'reputable' detectors that give good performance/accuracy and evaluate the same piece of text using them. You only need to worry if multiple/majority are returning extreme values. For example if 2 return 100% and the other 2 return 0% that is still highly suspicions of AI writing. If they all return 50-60 percent there's not much need to worry about anything. But even then there is still good chance of false positive/negative, just a little less than using just 1 detector.

How should Admission Officers use AI Detectors

AI detectors are at a stage of development where they still give a significant amount of false positive/negative predictions. If you base your evaluations of someone's personal statement off of an AI detector you are being extremely irresponsible. Judging how much AI is in someone's essay based on your own judgment can be even more irresponsible. And combining the two will not improve your accuracy I assure you. I would suggest only worrying about the quality of the essay and keep AI out of your mind(for now). Generally AI written personal statements are bad to begin with. This is likely what's best for both the AO and the applicant. Applicants will not have to worry about any bullshit evaluation an AI detector gives, and you as an AO will have one less thing to worry about when evaluating essays.

2 comments

Comments sorted by top scores.

comment by Viliam · 2024-11-29T15:08:57.707Z · LW(p) · GW(p)

For the human who writes the text, is there a simple way to protect yourself from becoming a false positive? Such as, make a very unlikely typo (that wouldn't appear in LLM's training text).

Replies from: kaustubh-kislay
comment by Kaustubh Kislay (kaustubh-kislay) · 2024-12-06T00:45:33.437Z · LW(p) · GW(p)

inputing some variance into text goes along with exactly what makes human text human. Though there are some safeguards LLMs have in regards to tokenization in order to handle prompting typos, so in the same way there will be some ways to handle typos in actual writing. But inputing uncommon variance into text as to make it more 'human' would be the best way to avoid AI detectors.