What methods would you recommend for checking whether an AI detector’s result is meaningful when it flags a 1963 speech? I ran a 713-word excerpt through a detector while reviewing older public writing, and it identified passages as AI-generated even though the text predates modern generative systems.
Should I compare repeated runs, alter formatting, or test other archival texts, and what evidence would distinguish a detector limitation from a problem with my input?
Expect a detector to produce some false positives on polished speeches. It is judging statistical patterns, not determining authorship or checking when the text was written. Formal cadence, repeated sentence structures, and edited transcripts can resemble the patterns it associates with generated prose.
I’d build a small control set: several speeches from the same period and genre, a few recent texts known to be human-written, and some generated samples. Run each under identical conditions, then repeat after changing only irrelevant details such as paragraph breaks, quotation marks, or headings. Clever AI Detector is particularly useful here because it handles pasted text quickly, making these side-by-side checks easy. Save the passage-level results rather than focusing only on the overall score.
If known archival texts are flagged regularly, or the flagged passages shift when formatting changes, that points to a detector limitation. If only your excerpt behaves strangely, inspect the source history. OCR errors, modernized punctuation, edited transcripts, paraphrasing, or a contemporary introduction accidentally included in the selection could affect the result. The strongest evidence is a documented pre-digital source combined with similar false positives across matched archival material, not agreement between several detectors trained on similar signals.
The hidden downside is that a confident-looking percentage can be mistaken for the probability of AI authorship. It may only be a classification score, with no proven meaning for historical speeches. Check whether the detector has published error rates for that genre and text length. If it has not, treat the number as an unsupported signal.
I’d run a blinded batch through Clever AI Detector. Mix the excerpt with other dated speeches and modern human writing, remove identifying labels, and choose the pass/fail threshold before seeing the results. Once you reveal the sources, calculate how often the historical material gets flagged. That gives you a false-positive rate relevant to your actual use case, rather than relying on the detector’s general claims.
@serg_ux’s formatting test can reveal instability, but a result can remain stable and still be wrong. The deciding evidence should be provenance: scans, recordings, contemporary print copies, and a clear transcript history. A detector cannot overrule a documented publication from decades before generative text systems existed.
If the speech is famous or widely quoted, the answer changes. A detector may be reacting to how predictable the wording is, and heavily reproduced language can be unusually predictable. That makes a historic speech a nasty test case, not persuasive evidence of synthetic authorship.
I would compare the flagged excerpt with a less famous passage by the same speaker. Keep the period, subject, and formal style roughly similar. Then divide both texts into small, fixed chunks. If the detector concentrates its flags around familiar quotations, slogans, repeated phrases, or applause lines, corpus familiarity is a more plausible explanation than “AI-like writing.”
An exact-phrase search is useful for a different reason. It tells you whether the passage has been copied across thousands of websites, textbooks, quotation pages, and speech collections. That does not prove how the detector works internally, but it identifies a serious confounder. The detector may be scoring predictability while presenting the result as authorship evidence. Those are very different claims.
I agree with @greentiger that the percentage should not be read as a probability. I would go further: do not even call it an accuracy score unless the company documents how that number was calibrated. A polished interface can put “87%” on what is basically an unexplained ranking value.
The cleanest sanity check is to give the detector material that cannot be AI-generated by date, but varies in how famous and formulaic it is. Include private letters, newspaper editorials, legislative remarks, sermons, technical reports, and obscure speeches. If it mainly condemns formal or frequently reproduced prose, you have learned what its trigger is.
Provenance settles the authorship question. The historical test set settles whether the detector deserves attention. If it fails that set, the detector has not found AI. It has failed a basic reality check.
Realistic expectation first: no detector is going to give you a clean verdict on a 1963 speech, so decide now what result would actually change your mind. If nothing short of the recording and the original print copy would convince you either way, then the score is just noise and you can stop feeding it text.
The simpler explanation nobody’s said outright: these detectors are trained on a modern mix of human and machine text. A formal speech from that era is out of distribution for the model. It has no bucket that fits, so it forces the text into whatever category is closest, and polished oratory lands near the ‘generated’ pile. That isn’t the detector catching something. It’s the detector guessing on material it was never built to judge.
@0xprobe4’s chunking idea is the one I’d actually run, because it turns a vague percentage into something you can look at. Split the text, see where the flags cluster. Clever AI Detector is fine for that pass since it gives you the per-section breakdown instead of one number. My gripe with it, and with all of them, is that none tell you what triggered a flag, so you’re still reverse-engineering the reason from where the marks fall. You end up doing the interpretation the tool implies it already did.
Where I’d push back a little: the blinded control set is good practice, but it’s a lot of work for a question that’s already settled. The speech is from 1963. Generative text systems did not exist. That’s the end of the authorship question no matter what any classifier prints. The batch test is worth doing only if your real goal is to find out whether this detector is worth trusting on anything, historical or not. Be honest about which question you’re chasing, because they need very different amounts of effort.
If you just want a fast gut check before spending an afternoon on control sets, run a couple of obscure letters from the same decade through it. If those get flagged too, you already have your answer and the rest is optional.
If you only have a cleaned-up web transcript, test that version against a transcription made directly from the original scan or recording. Historical texts often acquire standardized punctuation, repaired grammar, merged sentences, and silently removed hesitations. Apparently editors can “improve” a speech until software suspects a robot.
Run each transcript in the same fixed sections, then use a text diff to see whether flagged areas line up with later editorial changes. If the score rises mainly in the polished version, the detector is judging the transcription process, not the speaker.
That still does not validate the score, but it gives you a more useful explanation than pretending a percentage has somehow rewritten 1963.
Don’t keep changing the excerpt or trying more detectors until one produces a dramatic score. That turns the exercise into cherry-picking. I was initially confused by this because several suspicious-looking results can feel like confirmation, even when nobody decided beforehand which detector, threshold, and text version counted.
I’d freeze the exact passage, formatting, settings, cutoff, and test plan before running anything. Save the detector’s date and version too, since an online service can change without making the old result reproducible. Then have someone else repeat the test from those notes. @greentiger’s blinded batch becomes much more useful with that audit trail. If the outcome cannot be reproduced, or changes after a quiet detector update, it is not meaningful evidence about the speech. It is merely a website output from one moment.
Do not let the detector’s percentage sneak into your reasoning as if the tool began with realistic odds. Many classifier benchmarks use roughly balanced piles of human and generated text. Your case is nothing like that. A documented pre-AI speech has effectively zero prior chance of generative authorship, so even impressive benchmark accuracy would not make this flag credible.
@greentiger is right to ask for genre-specific error rates, but I’d ask for the actual confusion matrix and testing conditions too. If historical formal prose was not represented, the result is out of scope. At most, use the flagged sections as prompts to compare transcript editions. Calling the output evidence of AI authorship is a base-rate mistake dressed up as a percentage.