Why E8 assurance rarely stays automated for long

We've tried to build a fully automated Essential Eight assurance pipeline more than once. Each attempt got further than the last, and each one stopped in the same place: a control that couldn't be verified without a person deciding whether what the system reported was actually true.

Anyone promising continuous, out-of-the-box E8 automation is skipping that part. Two of the eight controls hold up to full automation, and a third gets close. The rest need someone with application context, change authority, or the judgement to sign off on an exception, and no amount of tooling changes that.

The audit record points the same way. In the ANAO's 2025–26 interim audit of major Commonwealth entities, one of the 22 entities required to comply reported full implementation across the Essential Eight, and self-assessed compliance fell across seven of the eight strategies, partly because entities changed how they assessed. The numbers went down as the assessment got more honest. What follows is where the line between automatable and not actually sits, and why it doesn't move much no matter how the tooling improves.

The controls that resist automation don't share an industry or a technology. What they share is a structural problem: the thing enforcing the control and the thing reporting on the control are the same untrusted node. Call it the difference between closed-loop and open-loop systems.

A closed-loop system is one where the enforcer and the record-keeper are the same authority. Entra ID doesn't just require MFA, it is the record of whether MFA happened, which method was used, and whether it succeeded. There's no second system to reconcile against, no gap between the claim and the reality to bridge. You ask Entra a question, and the answer is authoritative by construction.

An open-loop system is one where you're asking a device to grade its own homework. AppLocker or WDAC policy gets pushed from Intune or Azure Policy, and the deployment tooling will happily report that policy as "compliant". But whether the policy is actually blocking anything, rather than sitting in audit mode, or having silently failed to apply, is only knowable by asking the endpoint itself, the same endpoint the control exists to protect. Before that endpoint's self-report can be trusted, someone has to independently establish that the reporting channel is honest. That's a one-time validation at minimum, and arguably a recurring one, and it's exactly the kind of judgement call that resists generic automation.

Multi-factor authentication is an identity provider's problem, full stop. Entra or Okta enforces it and reports on it in the same breath. Phishing-resistant MFA (ML3) is even more tightly closed-loop, since the credential type itself (FIDO2, Windows Hello for Business, certificate) is part of the same authoritative record.

Restrict administrative privileges looks like an identity problem but isn't purely one. Privilege isn't a property of an account, it's a property of an account in the context of a specific application's authorisation model. The IdP knows the claim; it doesn't know what that claim means inside every downstream app. This is why "we deployed PIM" and "we restricted admin privilege" aren't the same sentence.

Regular backups is an infrastructure and data-estate ownership question, and it has its own signal problem. Unauthorised access to a backup is a strong, unambiguous signal that the control has failed. But there's no equivalent positive signal that the control is holding. You're proving a negative, and the only failure mode is silence itself failing silently.

Application control is a standard operating environment and support problem before it's a security one. The question underneath it is what operating environment a user is actually afforded, and how that holds up against BYOD and remote work.

User application hardening is where the browser lives, and the browser is the hard part: a modern browser is closer to an operating system than an application. The settings themselves push cleanly through Intune. Whether they hold against a user who installs a second browser, or a vendor who ships their product as Electron with a whole Chromium inside it, is a fleet-knowledge question, and the dashboard doesn't have fleet knowledge.

Patch operating systems and patch applications look like server admin work, and mostly they are, except for the "no longer supported, replace it" clause at the bottom of every maturity level. That clause isn't a patching problem, it's an application modernisation problem wearing a patching costume, and it doesn't automate at all.

Microsoft Office macros is a legacy problem with a heavy finance-team footprint, and it ties directly back into privilege: a macro is effectively an ad hoc elevated execution path running under the document-opener's own credentials.

Even where a control is well-defined, the signal telling you it's satisfied isn't always trustworthy in the same way. There are three flavours:

• Upstream-anchored. The claim is checked against a trusted third party. What needs patching comes from Microsoft's vulnerability intelligence, not from the machine being graded, and the software inventory comes from a sensor that is independent of the tooling doing the patching.
• Local, event-based. Something happened and was logged. Good for catching a breach after the fact, useless for proving a baseline is holding, since the absence of the event proves nothing.
• Absence-based. Compliance is inferred from nothing bad being logged. This is the weakest of the three, because the system can fail silently and the silence looks identical to success.

Patching sits in the first category, which is what makes it automatable. Backups and most of the ML3 logging requirements sit in the third, which is what makes them the hardest to assure continuously, not just implement once.

Phishing-resistant MFA, checked directly against Entra's own sign-in record. The output is every user and app combination with no phishing-resistant sign-in in the last 30 days:

SigninLogs
| where TimeGenerated > ago(30d)
| where ResultType == "0"
| mv-expand AuthDetail = parse_json(AuthenticationDetails)
| extend AuthMethod = tostring(AuthDetail.authenticationMethod)
| where tobool(AuthDetail.succeeded) == true
| extend IsPhishingResistant = AuthMethod in ("FIDO2 security key",
    "Windows Hello for Business", "X.509 Certificate")
| summarize PhishingResistantSignIns = countif(IsPhishingResistant),
    TotalSignIns = count()
    by UserPrincipalName, AppDisplayName
| where PhishingResistantSignIns == 0

Patching within SLA for actively exploited vulnerabilities. The exploit intelligence comes from Microsoft, not the endpoint. The software inventory comes from the Defender sensor, which sits on the endpoint but is independent of the pipeline doing the patching, and that separation between the tool doing the work and the tool grading it is what makes the signal trustworthy:

DeviceTvmSoftwareVulnerabilities
| where VulnerabilitySeverityLevel in ("Critical", "High")
| join kind=inner (
    DeviceTvmSoftwareVulnerabilitiesKB
    | where IsExploitAvailable == true
    | project CveId, PublishedDate
) on CveId
| extend HoursSinceDisclosure = datetime_diff('hour', now(), PublishedDate)
| where HoursSinceDisclosure > 48
| project DeviceName, SoftwareName, SoftwareVersion, CveId,
    PublishedDate, HoursSinceDisclosure, RecommendedSecurityUpdate

One caveat on the clock: this measures from CVE publication, and the 48-hour SLA strictly runs from when a patch became available, so a device can show up here before it's able to comply. Treat the output as a triage list, not a verdict.

Application control enforcement is the conditional one. Before the endpoint's self-report means anything, someone has to independently establish that the reporting channel is honest, and keep establishing it. This is the query that catches a policy showing "compliant" in Intune while the endpoint is still quietly running in audit mode:

Event
| where TimeGenerated > ago(7d)
| where Source == "Microsoft-Windows-AppLocker"
| where EventID in (8002, 8003, 8004, 8005, 8006, 8007)
| summarize
    Allowed   = countif(EventID in (8002, 8005)),
    AuditOnly = countif(EventID in (8003, 8006)),
    Enforced  = countif(EventID in (8004, 8007))
    by Computer
| extend Status = case(
    Enforced > 0,  "Enforcing and blocking",
    AuditOnly > 0, "Audit-only, not yet enforcing",
    Allowed > 0,   "Engine alive, nothing blocked yet",
    "No signal, treat as not applied")

That last one is the whole argument in one query. The policy can be compliant while the control does nothing. The allow events (8002, 8005) prove the AppLocker engine is alive, but a machine that's enforcing with nothing to block still looks the same as one in audit mode with no violations. Closing that gap means asking the endpoint directly, and having already checked that the endpoint isn't lying.

None of this is an argument against pursuing ML2 or ML3. It's an argument against believing a vendor who says the whole model can be pushed to "set and forget". The controls split cleanly into two groups: the ones where a well-configured platform genuinely does the ongoing work (identity, patching against upstream intelligence), and the ones where the platform gives you a lever but a person still has to pull it, check it, and keep checking it (privilege boundaries per application, application control enforcement, backup integrity, anything resting on an absence-based signal).

Knowing which group a control falls into changes what you budget for. The first group is a configuration cost. The second is an ongoing one, and it doesn't shrink because the tooling improved. It shrinks because someone is still deciding, on a schedule, whether what the system is telling them is actually true.