What 99.65% accuracy taught me about AI security
A few years ago I lost money to an online scam. Later, a ransomware attack locked my files. Neither attack was clever, but both worked, and both left me with the same question: how do you stop something that looks normal until it is too late?
That question followed me into my MS in Artificial Intelligence. Today it has a new twist. The message that tricks you no longer has to be written by a person. A language model can write a convincing bank alert, a fake job offer or a heartfelt plea for help in seconds, in good grammar, without the spelling mistakes we were all taught to look for.
So for my thesis I set out to build a detector: a model that reads a piece of text and decides whether a human or an AI wrote it.
What I built
I trained and compared four deep learning models on a balanced dataset of about 500,000 texts, half written by people and half generated by AI:
- an LSTM, which reads text word by word and keeps a running memory;
- a fine-tuned BERT, a transformer that looks at every word in the context of the whole passage;
- TF-IDF + BERT + LightGBM, which mixes classic word statistics with BERT features and a gradient-boosted classifier;
- a hybrid BERT + LSTM, where BERT understands the context and an LSTM on top reads it as a sequence.
The hybrid won: 99.65% accuracy and an F1-score of 0.995.
I was proud of that number, and I still am. But the more I thought about where a detector like this would actually be used, the more I realised the number is where the security question starts, not where it ends.
Lesson 1: accuracy depends on what the world looks like
My dataset was balanced: one human text for every AI text. That is the right way to train and compare models. It is not what an inbox looks like.
Here is a simple thought experiment. Imagine 10,000 messages where only 100 are AI-generated, and say the detector is wrong 0.35% of the time on both kinds of text:
- it catches almost all of the 100 AI messages;
- but it also wrongly flags about 35 of the 9,900 human ones.
Out of roughly 135 flagged messages, about a quarter are real people being accused of being machines. The model has not changed at all, only the world it works in.
In security this is called the base-rate problem, and it is why SOC analysts drown in false alarms. A detector is only as useful as its false-positive rate at the scale it runs.
Lesson 2: attackers read the same papers we do
A spam filter or a malware scanner does not face a fixed opponent. It faces someone who adapts. AI-text detection is no different. Paraphrasing a text, mixing human and machine writing, or switching to a newer model are all ways to slip past a detector trained on yesterday’s data.
A high score on a test set tells you how well a model handles the attacks it has already seen. It says much less about the ones it hasn’t. This is the field of adversarial machine learning, and it is the main reason I want my next step to be in AI security itself. Building a detector is one half of the job. Making it hold up when someone is actively trying to break it is the other half.
Lesson 3: language is a security gap
Most public datasets and detection tools are built for English. The scam messages people receive in Pakistan often aren’t in English. They come in Urdu, or in Roman Urdu, Urdu typed in English letters, with its own spellings and slang.
A detector that has never seen “aap ka account block ho gaya hai” will not protect the people who receive that message. That is why my current project is a scam and phishing detector for English, Urdu and Roman Urdu. The attackers already speak these languages, so our defences need to as well.
Lesson 4: a detector should give a signal, not a verdict
Put the first three lessons together and one conclusion stands out. A model like mine should not be the judge. It should be one signal among several, alongside who sent the message, where the link points and whether the request makes sense, and the final decision on anything that matters should stay with a person.
That is less exciting than “99.65% accurate AI detector”, but it is how good security works: layers, not silver bullets.
Where I’m going next
My thesis taught me how to build models that separate real from fake. It also showed me how much is still open: detectors that cope with real-world base rates, that hold up against adaptive attackers, and that work in the languages people actually use.
That is the work I want to do next, in research and in security operations. If you are working on similar problems, I would be glad to hear from you. You can find me through the contact form.
A paper based on my thesis is in preparation. I will share it here when it is out.