Test a data security coverage claim
Define the store, sensitive content and required action. Then test what was inspected, what changed and what the team must operate.
- Require evidence for the exact store and control, including exclusions and prerequisites.
- Measure missed data, incorrect findings and failed actions against a known test set.
- Compare costs and operating work for the same tested scope.
Define an observable outcome
Write one requirement per workflow. Include the data store, content, user or identity, action and acceptable delay. Agree on pass criteria with the person who owns the data before the vendor configures the test.
For example, require a synthetic customer export in SharePoint to remain accessible to an approved partner while becoming inaccessible through an anonymous link. A successful result needs both access checks and a record of the change. A finding in a dashboard alone does not satisfy this requirement.
For discovery, specify whether every sensitive object must be identified or whether a resource-level profile is sufficient. Record the accepted sampling policy and any exceptions.
Match the evidence to the claim
Purview Data Map documents BigQuery metadata and lineage without automatic classification. Its S3 connector supports classification but does not apply data policies. Ask for the connector's capability table, not a list of supported logos. [1]
Google distinguishes resource profiles from inspection findings for individual sensitive values. These outputs answer different questions. Specify the required output before comparing scan results. [2]
| Required outcome | Evidence that can demonstrate it |
|---|---|
| Locate a sensitive object | The seeded object's location, detected type and scan result, including misses and exclusions. |
| Remove excessive access | Effective access before and after the change, preserved legitimate access and an audit event. |
| Mask a sensitive column | The same query returns the expected masked or readable values for each tested identity and role. |
| Prevent a transfer | The attempted transfer denied through the required app, device and identity, with timing recorded. |
Use a test set with known answers
Create an approved test area with synthetic sensitive examples and harmless lookalikes. Keep a separate inventory of each object's expected result. Include common formats, rare formats, encrypted files, old content and recently changed content that matter to your environment.
Macie skips unsupported formats and storage classes. It does not analyze images or objects in Glacier Deep Archive. An excluded object must remain visible in the evaluation record rather than count as a clean result. [3]
- Record the submitted, eligible, inspected, sampled, skipped and failed populations. Keep overlapping categories separate.
- Count known sensitive examples found and missed. Count harmless examples incorrectly flagged.
- Group failures by format, source, permissions and scan mode so an overall percentage cannot conceal a required workflow's failure.
- Repeat after adding and changing content. Record the delay until results reflect the change.
Verify the action and its timing
For a sharing test, use a public link, an organization-wide link and a named legitimate recipient. Check effective access from each identity before and after remediation. Include inherited permissions. Record approvals, audit events and the process for restoring legitimate access.
Purview documents different blocking behavior for different DLP conditions and actions. One externally shared content rule with Block everyone can leave a short guest-access window. Repeat the test for the exact configured rule and measure access before enforcement. [4]
For transfer prevention, repeat the same content through each required channel. Record whether the attempt was blocked, allowed with an override or only logged. Keep these outcomes separate from changes to access permissions.
For warehouse masking, test authorized users, restricted users and configured exceptions through tables, views and application accounts. Change a policy and measure when query results reflect it. Check what evidence records the query and the policy change, then test rollback. The ALTR and Immuta comparison identifies processing and policy conditions to include in this test.
Record prerequisites and operating work
Google's S3 profiling requires Security Command Center Enterprise and an AWS connector with discovery permissions. Optional exported sample findings include detected strings. Include the required product tier, permissions and output destination in the evaluation scope. [5]
Ask the operator to repeat a scan, investigate an incorrect result, change a policy and recover a failed connector. Record active working time separately from elapsed scan time. Identify which steps require vendor assistance and who owns them after the evaluation.
| Keep in the decision record | What to capture |
|---|---|
| Configuration | Product edition, connector version, permissions, scan mode, exclusions and policy settings. |
| Data handling | Where content is processed, metadata and samples retained, region and deletion process. |
| Results | Expected versus observed outcomes, missed examples, incorrect findings and time to action. |
| Cost | Licenses, inspected volume, scan frequency, cloud requests, compute and operator work for this scope. |
| Unresolved items | The failed requirement, owner, proposed remedy and a date to repeat the test. |
Decide from the tested scope
Reject a proposed deployment that fails a mandatory workflow unless the accountable owner accepts a documented exception. Evaluate an additional product against the failed requirement using the same dataset and pass criteria. Preserve working controls and price the additional coverage and operating work.
Use the platform comparison to identify documented differences. The Purview evaluation guide applies this process to existing Microsoft entitlements. Public documentation can inform a shortlist, but it cannot supply the observed results of your evaluation.