Six month Copilot review framework for federal programs

Why a six-month Copilot review is a required control gate

At six months, a Copilot pilot should undergo a formal review that tests whether it delivers measurable mission value while meeting governance, safety, privacy, accessibility, records, security, and acquisition obligations. OMB M-24-10 requires agencies to establish AI governance, maintain an AI use-case inventory, and apply elevated safeguards to rights-impacting and safety-impacting AI uses, including impact assessments, testing and evaluation, independent evaluation, transparency, and appropriate human alternatives or oversight¹. NIST’s AI Risk Management Framework (AI RMF 1.0) calls for post-deployment measurement, monitoring, and risk treatment across its Govern, Map, Measure, and Manage functions². The six‑month review operationalizes these requirements before scaling deployments, consistent with Executive Order 14110’s direction for safe, secure, and trustworthy AI use in government³.

Scope the review to the use case and risk designation

• Confirm the AI use case is recorded in the agency’s AI inventory, with its risk designation (rights-impacting or safety-impacting) and governance approvals documented per M-24-10¹.
• Identify all system boundaries and data flows: Microsoft 365 Copilot, Azure OpenAI-based services, connectors, and any external plugins or APIs that could change the risk posture, consistent with the AI RMF’s Map function².

Governance, impact, and oversight

• Verify completion and currency of AI impact/risk assessments required by M-24-10 for rights-impacting or safety-impacting use cases, including documented human oversight or appropriate alternatives for affected individuals as applicable¹.
• Ensure pre-deployment and ongoing testing and evaluation are documented commensurate with impact, including independent evaluation where required by M-24-10¹.
• Align monitoring and change management with AI RMF Measure and Manage functions, including defined triggers for model or policy updates and rollback criteria².
• Cross-check governance artifacts against GAO’s AI accountability dimensions (governance, data, performance, and monitoring) to ensure completeness⁴.

Security authorization and cloud boundary

• Confirm all components run within authorized environments and align with the agency ATO. Use FedRAMP’s standardized approach for security assessment of cloud services as the baseline for cloud risk evaluation⁵.

• For Microsoft platforms used:

  • Azure Government: validate service scope against Azure Government’s FedRAMP High and applicable DoD Impact Level authorizations as documented by Microsoft⁶.
  • Microsoft 365 US Government: verify the tenant environment (GCC, GCC High, or DoD) and its data residency and personnel screening characteristics match the agency’s requirement profile⁷.

• Map implemented controls to NIST SP 800-53 Rev. 5 (for example AC, AU, CM, IR, PL, RA, SA, SC, SI) and ensure continuous monitoring is in place per NIST SP 800-137⁸⁹.

Privacy and data protection

• Validate that Privacy Impact Assessments are current and reflect AI-specific data uses, consistent with OMB Circular A-130 requirements for agency privacy programs and PIAs¹⁰.
• For Copilot for Microsoft 365, confirm data handling aligns with Microsoft’s documented behavior: Copilot access is governed by existing Microsoft 365 permissions, compliance, and privacy controls, and tenant content is not used to train the foundation models¹¹.
• For Azure OpenAI Service, confirm Microsoft’s documented data protections are in effect for the tenant: customer inputs/outputs are not used to train OpenAI models and applicable retention settings are configured; apply built-in safety system protections as appropriate¹²¹³.

Logging, auditability, and records

• Ensure audit logging aligns with the agency’s M-21-31 logging maturity targets, including capture of relevant prompts, responses, model parameters (where feasible), and administrative actions to support investigation and remediation¹⁴.
• For Microsoft 365 Copilot, enable and validate audit events in Microsoft Purview for supported Copilot activities and confirm retention settings appropriate for the system’s risk and mission needs¹⁵.
• Determine whether prompts, model outputs, or decision artifacts meet the Federal Records Act definition of records and, if so, manage them under approved records schedules, consistent with 44 U.S.C. 3301¹⁶.
• Verify accessibility of audit trails and records for eDiscovery and oversight in line with AI RMF’s Measure/Manage guidance on traceability and accountability².

Accessibility and user experience

• Validate Section 508 conformance of user-facing Copilot experiences and any custom integrations, consistent with the ICT Standards and Guidelines at 36 CFR Part 1194¹⁷.
• Provide accessible alternatives where needed and document UX testing with assistive technologies¹⁷.

Performance and value measurement

• Establish and review baseline-to-pilot comparisons on mission KPIs (task completion time, error rates, customer experience), with statistically sound sampling where applicable, consistent with AI RMF’s emphasis on post-deployment measurement².
• Include quality assurance via human review of samples, red-team findings disposition, and drift monitoring thresholds tied to risk tolerances².

Incident management and safeguards

• Define what constitutes an AI incident for the use case and ensure alignment with M-24-10 incident handling and reporting expectations; validate escalation paths and communications templates¹.
• Confirm technical safeguards for misuse and model safety (e.g., abuse monitoring, content filtering) are configured and tested; align with CISA’s secure AI system development guidance for logging, testing, and deployment controls¹⁸.
• Ensure continuous monitoring processes can detect and respond to degradations or policy violations per NIST SP 800-137⁹.

Acquisition and software supply chain

• Verify that software and services used in the Copilot stack meet OMB M-22-18 requirements for secure software development practices, including obtaining producer self-attestations (and SBOMs where required by agency policy)¹⁹.
• For cloud services, use FedRAMP security assessment artifacts to inform the ATO package and ongoing risk reviews⁵.
• Ensure contracts and TOUs explicitly prohibit unauthorized training on agency data and define data residency, retention, and deletion obligations consistent with agency policy and A-130¹⁰.

Decision gates and documentation

• Make an authorization decision to continue, scale, pause, or retire, documenting risk acceptance and required mitigations in the system authorization package per the NIST Risk Management Framework²⁰.
• Record governance board decisions, conditions for scale, and a plan for ongoing monitoring (metrics, tests, and incident drills) consistent with AI RMF Manage function².

Microsoft-aligned control checks (use where applicable)

• Microsoft 365 Copilot

  • Validate that Copilot respects existing M365 permissions and data boundaries and that tenant content is not used to train foundation models¹¹.
  • Confirm Microsoft Purview Audit logging is enabled for Copilot activities and retention meets investigative and records requirements¹⁵.
  • Validate that DLP, sensitivity labels, and conditional access policies in the tenant apply to Copilot-assisted workflows per the product’s documented alignment with existing M365 compliance controls¹¹.

• Azure OpenAI-based copilots

  • Confirm Microsoft’s documented data handling is in effect for the deployment (no use of customer data for model training) and that safety system settings are aligned to mission risk¹²¹³.
  • Ensure deployment is within the agency’s authorized Azure boundary (e.g., Azure Government as required) and inherits applicable FedRAMP/DoD controls from the platform⁶.

• Azure Government and Microsoft 365 US Government

  • Cross-check service scope and control inheritance against Azure Government’s compliance attestations and the selected Microsoft 365 US Government environment characteristics (GCC, GCC High, DoD) for data residency and personnel screening where required⁶⁷.

Artifacts to bring to the six-month review

• Governance: AI use-case inventory entry, risk designation, approvals, and AI impact/risk assessment (per M-24-10)¹.
• Testing and evaluation: pre-deployment test plans/results, independent evaluation reports (if applicable), red-team findings and mitigations¹.
• Security: control mappings to NIST SP 800-53, FedRAMP/ATO boundary diagram, POA&Ms, continuous monitoring plan⁸⁵⁹.
• Privacy: updated PIA, data flow diagrams, data retention/deletion configuration evidence, contractual data protections (no training on agency data)¹⁰.
• Audit and records: M-21-31-aligned logging configuration, sample audit events (e.g., Purview for Copilot), records determination and schedules for prompts/outputs¹⁴¹⁵¹⁶.
• Accessibility: Section 508 test results and remediation plans¹⁷.
• Performance: KPI baselines, pilot metrics, error analyses, and human review samples consistent with AI RMF measurement guidance².
• Incident: incident definitions, runbooks, tabletop results, escalation contacts, reporting templates aligned to M-24-10¹.
• Acquisition/SCRM: M-22-18 software attestations, applicable SBOMs, contractual clauses for data protection, model update/change control terms¹⁹.
• Authorization decision: documented decision per NIST RMF with conditions and monitoring commitments²⁰.

¹: OMB M-24-10 — Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence — https://www.whitehouse.gov/omb/memoranda/2024/m-24-10-advancing-governance-innovation-and-risk-management-for-agency-use-of-artificial-intelligence/ ²: NIST Artificial Intelligence Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/artificial-intelligence/executive-order-safe-secure-and-trustworthy-artificial-intelligence ³: Executive Order 14110 — Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence — https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/ ¹⁰: OMB Circular A-130 — Managing Information as a Strategic Resource — https://www.whitehouse.gov/omb/management/office-of-the-federal-chief-information-officer/circular-a-130/ ¹⁶: 44 U.S.C. 3301 — Definition of records — https://www.law.cornell.edu/uscode/text/44/3301 ¹⁷: 36 CFR Part 1194 — Information and Communication Technology (ICT) Standards and Guidelines — https://www.access-board.gov/ict/ ¹⁴: OMB M-21-31 — Improving the Federal Government’s Investigative and Remediation Capabilities Related to Cybersecurity Incidents — https://www.whitehouse.gov/omb/memoranda/2021/m-21-31/ ⁵: FedRAMP — About the Program — https://www.fedramp.gov/about/ ⁶: Azure Government compliance offerings — https://learn.microsoft.com/azure/azure-government/compliance/azure-government-compliance ⁷: Microsoft 365 US Government environments — https://learn.microsoft.com/microsoft-365/enterprise/microsoft-365-government?view=o365-worldwide ¹¹: Data, privacy, and security for Copilot for Microsoft 365 — https://learn.microsoft.com/microsoft-365-copilot/microsoft-365-copilot-privacy?view=o365-worldwide ¹⁵: Audit Copilot for Microsoft 365 activities in Microsoft Purview — https://learn.microsoft.com/purview/audit/copilot-for-microsoft-365-auditing?view=o365-worldwide ¹²: Azure OpenAI Service data, privacy, and security — https://learn.microsoft.com/azure/ai-services/openai/concepts/data-privacy ¹³: Azure OpenAI Service safety system and content filtering — https://learn.microsoft.com/azure/ai-services/openai/concepts/safety-system ⁸: NIST SP 800-53 Rev. 5 — Security and Privacy Controls for Information Systems and Organizations — https://csrc.nist.gov/publications/detail/sp/800-53/rev-5/final ⁹: NIST SP 800-137 — Information Security Continuous Monitoring (ISCM) for Federal Information Systems and Organizations — https://csrc.nist.gov/publications/detail/sp/800-137/final ¹⁹: OMB M-22-18 — Enhancing the Security of the Software Supply Chain through Secure Software Development Practices — https://www.whitehouse.gov/omb/memoranda/2022/m-22-18/ ¹⁸: CISA — Guidelines for Secure AI System Development — https://www.cisa.gov/resources-tools/resources/guidelines-secure-ai-system-development ⁴: GAO-21-519SP — Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities — https://www.gao.gov/products/gao-21-519sp ²⁰: NIST SP 800-37 Rev. 2 — Risk Management Framework for Information Systems and Organizations — https://csrc.nist.gov/publications/detail/sp/800-37/rev-2/final