Sentinel analytics rules you have not tested are rules you do not have
The rule is called "Suspicious sign-in from unfamiliar location". It has been enabled for fourteen months. It is scheduled every five minutes, it has a severity of High, and it has never produced a single alert.
Two explanations. Either nothing suspicious has happened in fourteen months, or the rule does not work. Sentinel analytics rules give you no way to tell those apart from the portal, because a rule that cannot run does not report an error. It returns zero rows, which looks identical to a quiet estate.
Why sentinel analytics rules fail quietly
The causes are dull, which is why they persist.
The table does not exist in your workspace. Someone copied a rule that
queries SecurityEvent, and you never deployed the agent that populates it. The
query is valid KQL. It simply has nothing to run against, and Sentinel treats an
empty table the same as an empty result.
A column was renamed. Schemas change. TimeGenerated is stable, but plenty
of connector-specific columns are not, and a project of a column that no longer
exists narrows your result to nothing rather than raising an error.
The connector was never enabled. The most common version of the first
problem. AuditLogs and SigninLogs require the Entra ID connector. If somebody
set up the workspace and somebody else imported the rules, nobody necessarily
checked.
The rule was written for a different tenant's layout. Custom tables, different table names, a different Log Analytics workspace structure. Community rulesets are enormously useful and they were written against somebody else's estate.
Nothing has matched yet, legitimately. This is a real answer, and it is indistinguishable from all of the above unless you check.
What a validator should check
The point is to make "this rule works" a fact you have established rather than an assumption. Five checks, in this order, because each one is only meaningful if the previous passed.
It parses. Genuine KQL parsing rather than a regular expression hunting for keywords. A missing pipe or an unbalanced bracket should fail at your desk.
Every table referenced exists in the workspace you are targeting, checked against the live schema rather than a list of what usually exists.
Every column referenced exists on those tables. This is the one that catches schema drift, and it is the reason parsing alone is not enough.
It returns rows against historic data. Run the logic over a wide lookback, ninety days rather than five minutes. If a privilege escalation rule finds no role assignments in ninety days, either your estate is unusually static or the rule is wrong, and both are worth knowing.
Severity and MITRE tactics are populated. An unclassified High alert is one nobody triages, and an alert nobody triages is worse than no alert because it consumes the attention you were saving.
The first three should block deployment. The fourth should warn, because a detection that is legitimately quiet is fine. Microsoft's guidance on custom analytics rules covers the rule mechanics; the validation layer is the part you have to build or buy.
Two categories first
For a small estate, configuration drift and privilege escalation are where to start, and everything else can wait.
Configuration drift is the security control that quietly stopped applying. A Conditional Access policy modified or deleted. A storage account made public. A diagnostic setting removed, which is the one that covers the attacker's tracks and almost nobody watches. These fire rarely, and when they fire they are almost always worth reading.
Privilege escalation is access that appeared. A role assignment at subscription scope. A new credential on an existing app registration. A guest elevated. Same property: rare, and interesting when it happens.
Both categories share the thing that makes them viable for one person. They alert on change rather than on volume, so they do not require somebody watching a screen. They require somebody reading an email occasionally.
A rule worth having
Conditional Access modification, which is the one people are most surprised to need until the first time somebody excludes themselves from MFA:
AuditLogs
| where TimeGenerated > ago(1d)
| where OperationName has_any (
"Update conditional access policy",
"Delete conditional access policy",
"Add conditional access policy")
| extend Actor = tostring(InitiatedBy.user.userPrincipalName)
| extend PolicyName = tostring(TargetResources[0].displayName)
| extend Modified = tostring(TargetResources[0].modifiedProperties)
| project TimeGenerated, OperationName, PolicyName, Actor, Modified, Result
| order by TimeGenerated desc
Run that over ninety days on your own workspace before you schedule it. If it
returns nothing, either nobody has touched Conditional Access in three months,
or AuditLogs is not being ingested. Find out which.
Fewer rules, proven
The instinct when setting up Sentinel is to import everything available, on the grounds that more detection is more security. It is not, for two reasons.
Rules you have not validated give you false confidence, which is worse than no confidence because it stops you looking. And rules that fire often get muted, which trains you to ignore the channel the real alert will arrive in.
Twelve rules that provably run, each of which you could explain to somebody, is a better position than sixty imported ones where you could not say which are live. That is the argument behind AzClean Detections, and the ingestion cost note covers the other constraint on how many you can afford to feed.
Questions
How do I tell whether a rule has ever fired?
Check the incidents and alerts generated by that rule name over a long window rather than the rule blade. A rule with no alerts in ninety days is a question, not necessarily a fault, but it should always be a question you have asked.
Does running validation queries cost me anything?
Queries against data already ingested are not charged per query on the standard model, but they do consume workspace resources and there are limits on concurrent queries. Validating twelve rules over ninety days is not a meaningful cost. Ingestion is where the money goes, which the Sentinel cost note covers.
Should I validate the built-in Microsoft rule templates too?
Yes, and for the same reason. A built-in template can reference a table your workspace does not have, because the templates cover connectors you may not have enabled. Being Microsoft's rule does not make it a working rule in your tenant.
What about rules that need a connector I am not paying for?
Disable them rather than leaving them enabled and empty. An enabled rule with no data source is the exact thing this note is about, and leaving it in place makes your rule list untrustworthy.
Is a rule that returns nothing over ninety days always broken?
No. Some detections should be silent for years, which is the point of them. The distinction is whether you have confirmed it can run, and validating tables and columns answers that without needing the event to have occurred.