A wrong security rule is worse than a missing one
I spent weeks building an LLM-based code security scanner. The hardest lesson had nothing to do with prompts: a detection rule that is subtly wrong hides real bugs and invents fake ones at the same time. Here is what that looks like in practice.
For the last few months I have been building a code security scanner that runs on an LLM engine. The idea is simple to say and hard to do well: point it at a repository, and it reports injection, weak crypto, missing security headers, leaked secrets, dependency problems and a few dozen other classes of issue, each with the file and line and a suggested fix. I ended up with 44 detection classes. The product is called CodeHarden.
I expected the hard part to be prompt engineering. It was not. The hard part was writing the detection rules the engine reads, because of one property that took me a while to respect: the rules become ground truth. The model treats what I wrote as fact. And that means a rule that is subtly wrong is worse than no rule at all, because it fails in two directions at once. It marks real vulnerabilities as safe, and it flags safe code as vulnerable. A gap in coverage only costs you the bugs you never looked for. A wrong rule costs you the bugs you did look for, plus the trust of everyone who reads a false alarm.
The first time I wrote the rules, I had a lot of them almost right. Almost right is exactly the dangerous zone. A few examples from the cleanup, because the specifics are more convincing than the principle.
I had a rule that said Laravel's Auth::attempt does not equalise timing between a missing
user and a wrong password, which would leak whether an account exists. When I actually read
the framework source, the conclusion was right but the reason was wrong: modern Laravel does
equalise the timing, just not by doing constant work. It wraps the check in a Timebox that
pads the whole operation to a fixed budget. If I had shipped my original explanation, the
fix suggestion would have pointed developers at the wrong thing. Worse, a reviewer who
"corrected" it to "Laravel does not equalise timing at all" would have flipped a true
finding into a false one.
I had a rule about logging that assumed a Logback pattern without %ex silently drops stack
traces. Read the source and it turns out PatternLayout appends a throwable converter
automatically, so the stack trace is still there. The real leak is narrower and easier to
miss: an exception message printed outside the masked %msg field. If your rule describes
the wrong mechanism, you scan for the wrong shape and walk past the actual problem.
And a favourite: I assumed Go's gorilla/csrf would panic on a too-short authentication
key, the way a lot of libraries do. It does not. There is no length check in the source at
all. It silently fails token validation instead, which from an auditor's point of view is
worse than a panic, because nothing crashes and nothing warns. The system just quietly stops
protecting you.
The pattern in all three is the same. "Sounds right" is not a standard. Security claims have a way of being 90 percent true, and the missing 10 percent is where the vulnerability lives. So the rule I settled on, and the rule the whole engine now runs on, is that every technical claim has to be checked against a primary source before it ships: the specification text, the framework's own source code, the RFC. Not a blog post, not memory, not the confident output of another model.
That is also why I do not trust a single pass. When I write or fix a rule, one agent drafts it, a second reviews it with instructions to find only faults and praise nothing, and a third is told to verify independently before changing anything and to leave it alone if it cannot refute it. Then I do structural checks and sample the critical claims myself. It sounds paranoid. It is the only thing that reliably catches the reviewer being wrong too, which happened more than once.
There is a broader point here about AI in security tooling, and it is not the usual hype or the usual doom. An LLM is a genuinely good reader of code in context. It will notice things a regex never could. But it inherits the correctness of what you tell it with no friction at all, which means the discipline moves upstream, into the rules and the verification. The model is not the product. The verified knowledge behind it is.
That conviction is baked into CodeHarden in a way I am a little stubborn about. It scans static code and configuration, and every report says so in plain language. It does not do live testing, network testing or penetration testing, and it does not imply that it does. I would rather tell you exactly what a scan covers than let a clean report mean something it does not. If you have read this far, that is the whole philosophy: be precise about what you know, and honest about what you do not. codeharden.com is launching soon.