5 CIS M365 Controls That Automated Tools Cannot Check

Automated security tools — Secure Score, ScubaGear, Maester — pull configuration data from your M365 tenant and compare it against benchmark recommendations. For controls that involve checking whether a setting is on or off, whether a policy exists in Entra ID, or whether a logging feature is enabled, automated tools are efficient and accurate.

But every version of the CIS M365 Foundations Benchmark includes controls classified as manual — the exact count varies by version, but the gap is consistent. These controls cannot be fully evaluated by any automated tool. They require an assessor to navigate admin portals, verify configurations against organizational context, and confirm that the implementation meets the benchmark’s intent.

These are not obscure edge cases. They cover emergency access, cloud security monitoring, external sharing governance, app management, and Power BI tenant controls — areas where a misconfiguration creates real exposure. Here are five that illustrate why the manual controls matter.

1. Emergency Access Accounts Are Defined and Monitored (Controls 1.1.2 and 2.2.1)

What the benchmark requires:

Control 1.1.2 requires organizations to define two emergency access accounts — also called “break-glass” accounts — for scenarios where normal administrative accounts are unavailable. These accounts must meet specific criteria: cloud-only identities on the .onmicrosoft.com domain, passwords at least 16 characters and randomly generated, excluded from Conditional Access policies, and not assigned to any specific user. The credentials must be physically secured — FIDO2 security keys locked in a fireproof location, passwords optionally split into separate pieces stored in separate locations.

Control 2.2.1 requires that sign-in activity from these emergency accounts is actively monitored. The organization must configure an activity policy in Defender for Cloud Apps (or equivalent) that triggers high-severity alerts and notifications to administrators whenever an emergency account signs in.

Why automation cannot fully check it:

A script can query Entra ID for accounts with the Global Administrator role. It cannot verify that the accounts meet the break-glass criteria — that credentials are physically secured in a fireproof location, that the accounts use the .onmicrosoft.com domain rather than the organization’s vanity domain, that FIDO2 keys are stored separately from passwords, or that the organization has documented policies and procedures for when and how these accounts may be used.

For monitoring (2.2.1), an automated tool can check whether an activity policy exists in Defender for Cloud Apps. It cannot verify that the policy is correctly scoped to the specific emergency accounts, that the notification recipients are current, or that the alerts have been tested.

What assessments typically find:

Emergency access accounts either do not exist, or they exist but do not meet the benchmark criteria. Common gaps: accounts use the organization’s vanity domain (making them vulnerable to domain-level disruptions), passwords are stored in a shared password manager accessible to the same people who hold normal admin accounts (defeating the purpose), and monitoring is not configured — no one would know if the accounts were used outside an actual emergency.

2. Microsoft Defender for Cloud Apps Is Enabled and Configured (Control 2.4.3)

What the benchmark requires:

Microsoft Defender for Cloud Apps (MDCA) must be enabled with specific configurations: the Microsoft 365 and Microsoft Azure app connectors must be connected, Microsoft Defender for Endpoint integration must be enabled, and file monitoring must be active. MDCA provides the CASB layer that detects suspicious activity across M365 — impossible travel, suspicious inbox rules, inbox forwarding anomalies, new country detections, and activity from anonymous IP addresses.

Why automation cannot fully check it:

MDCA configuration spans multiple admin portals and integration points. An automated tool can check whether MDCA is licensed. It cannot reliably verify that app connectors are in a “Connected” state (not just created but stalled), that Defender for Endpoint integration is actively enforcing app access, that file monitoring is enabled, or that the configuration aligns with the organization’s risk profile.

The benchmark’s audit procedure requires navigating to the Microsoft 365 Defender portal, checking Settings > Cloud apps > App connectors for connected status, verifying Cloud Discovery > Microsoft Defender for Endpoint integration, and confirming Information Protection > Files monitoring — a multi-step manual walkthrough across several portal sections.

What assessments typically find:

MDCA is licensed but partially configured. The most common gap: app connectors were set up during initial deployment but one or both show a stale connection status. File monitoring is disabled — it requires explicit enablement and is off by default. Defender for Endpoint integration exists but “Enforce app access” is not checked, meaning MDCA is observing but not blocking. The organization has a CASB in name but not in function.

3. External Sharing Is Restricted by Security Group (Control 7.2.8)

What the benchmark requires:

SharePoint and OneDrive external sharing must be restricted so that only users in designated security groups can share content externally. This is a global setting — it applies to all SharePoint sites and OneDrive accounts and cannot be overridden at the site level.

The benchmark requires that Allow only users in specific security groups to share externally is checked in the SharePoint admin center, and that the security groups are defined in accordance with organizational policy.

Why automation cannot fully check it:

An automated tool can query the SharePoint Online sharing configuration via PowerShell or the admin API. It can detect whether the “restrict to security groups” toggle is enabled. What it cannot evaluate is whether the security groups are appropriately defined — whether the membership reflects the organization’s actual sharing needs, whether the groups are reviewed periodically, and whether the configuration aligns with organizational policy for external collaboration.

The benchmark explicitly requires the security groups to be “defined in accordance with company procedure.” That phrase requires an assessor to verify that a procedure exists, that the groups match the procedure, and that someone is responsible for maintaining the membership.

What assessments typically find:

External sharing is either unrestricted (any user can share externally) or restricted to a security group that was created at deployment and never reviewed. The group contains every user in the organization — technically satisfying the checkbox while providing no actual restriction. Or the group was properly scoped at creation but has grown through help desk tickets (“I need to share a file with a vendor”) without anyone reviewing whether the additions are still appropriate.

4. Teams App Permission Policies Are Configured (Control 8.4.1)

What the benchmark requires:

Microsoft Teams app permission policies must be configured to control which classes of applications users can install. The benchmark specifies: Microsoft apps may be allowed by default (or more restrictive), third-party apps must be set to not allow users to install by default, and custom apps must be set to not allow users to install by default.

This is evaluated in the Teams admin center under Teams apps > Manage apps > Org-wide app settings.

Why automation cannot fully check it:

Teams app policies span multiple configuration layers — org-wide settings, per-user permission policies, and app setup policies. An automated tool can query the Teams admin API for the org-wide defaults. It cannot reliably verify the full policy stack: whether per-user policies override the org-wide defaults, whether specific third-party apps have been individually allowed through exception processes, or whether the “Manage apps” list has been reviewed to remove apps that were approved temporarily but never cleaned up.

The benchmark’s audit procedure requires checking both the org-wide settings and the individual app states — a manual review of the Teams admin portal to verify the overall posture.

What assessments typically find:

Org-wide settings allow third-party app installation by default. This is the most common finding because it is the default configuration in new M365 tenants — and most organizations never change it. Users install third-party Teams apps for project management, scheduling, file sharing, and other workflows without any review of the app’s data access permissions. The result: dozens of third-party applications with access to Teams data, chat content, and channel files that no one in the organization explicitly approved.

5. Power BI Guest Access and Service Principal Controls (Controls 9.1.1 and 9.1.10)

What the benchmark requires:

Control 9.1.1 requires that guest user access to Microsoft Fabric (Power BI) is either disabled entirely or enabled only for specific security groups. The default state allows all B2B guests to access Power BI content they have permissions to — the benchmark requires restricting this.

Control 9.1.10 requires that service principal access to Fabric public APIs is either disabled or restricted to specific security groups. Service principals can perform create, read, update, and delete operations through APIs — unrestricted access means any registered service principal can programmatically interact with the organization’s Power BI content.

These are two of twelve Power BI controls in the benchmark, all of which are manual.

Why automation cannot fully check it:

Power BI tenant settings are managed through the Microsoft Fabric admin portal — a separate interface from the main M365 admin centers. Most automated M365 assessment tools do not cover the Fabric admin portal. The settings require navigating to app.powerbi.com/admin-portal > Tenant settings, then scrolling through configuration sections to verify each control individually.

Even for tools that do query Power BI settings via API, the benchmark requires evaluating whether the security group restrictions are “defined in accordance with company procedure” — the same organizational context check that automated tools cannot perform.

What assessments typically find:

Power BI controls are the most frequently missed category in CIS M365 assessments — not because they are difficult to configure, but because most organizations and their IT teams do not think of Power BI as a security-sensitive surface. The Fabric admin portal is visited infrequently, and the default settings allow broad access.

Guest access is enabled for all B2B users. Service principals have unrestricted API access. Publish to web is enabled, allowing any user to create publicly accessible links to reports containing potentially sensitive data. Twelve controls, all manual, all in their default (non-compliant) state.

The Pattern

These five controls — and the other manual controls in the benchmark — share a common trait: they involve configurations that require navigating specific admin portals, verifying settings against organizational context, and confirming that security groups, policies, and procedures are defined and maintained. Automated tools can check toggles. They cannot verify intent.

The majority of controls that automated tools cover are necessary. But the manual controls they miss include some of the highest-risk areas in the M365 environment: the emergency accounts that provide unlimited access, the CASB that detects compromise, the sharing controls that govern data leaving the organization, the app policies that control what third-party code runs in your tenant, and the Power BI settings that most organizations have never reviewed.

A CIS M365 Benchmark assessment covers every control — both the automated ones and the manual ones that require an assessor to navigate portals, review configurations, and evaluate organizational context. The automated tools are useful for monitoring between assessments. They are not a substitute for the full evaluation.

---

Genesis delivers CIS M365 Benchmark assessments covering 100% of controls — automated configuration checks and manual reviews of every control in the benchmark, including those that no automated tool can reach. For MSPs and vCISOs, wholesale pricing with white-label delivery.

See what your automated tools miss. Request a gap comparison — send your current Secure Score or ScubaGear output and we will identify which manual controls have never been evaluated in your environment.

Contact us to request a manual control gap review.