The limits: confident, wrong, and without ground truth

The single most important thing a defender must understand about AI is this: it produces confident language, not verified fact. A model can call a benign PowerShell script malicious, or wave through a real attack, and it will sound equally sure either way.

Three limits matter most in security work:

No ground truth. A model does not know what actually happened on your network. It pattern-matches against text. It has no access to the real process tree, the real packet capture, or the real intent unless you give it that evidence, and even then it can misread it.

Confident false positives and false negatives. A wrong verdict delivered with authority is more dangerous than an obvious guess, because it invites you to stop checking. In a SOC, a missed true positive can mean a breach.

Staleness and gaps. A model may not know about a threat that emerged last week, and it will rarely say so. It fills gaps with plausible text.

None of this makes AI useless. It makes AI a fast, tireless assistant whose output is a hypothesis, never a finding. Treat every AI verdict the way you would treat a tip from an unverified source: worth investigating, never worth acting on alone. The skill you are building is not blind trust. It is calibrated suspicion.