AIIA: Auditing AI: Hands-On for Internal Auditors (v2.0)
About this Course
Durations
a. It will be delivered as 4-hour sessions over 4 days as a public schedule within a program.
b. It will be delivered as a 2 day private class.
Course Overview
Understand and evaluate the foundational concepts, mechanisms, risks, and governance implications of Artificial Intelligence (AI) and Generative AI systems, with a specific focus on how these technologies impact audit scope, risk assessment, and control requirements. Internal and external auditors will assess whether AI initiatives are governed effectively from planning through deployment using recognized frameworks and documented controls. Design and apply AI governance policies and procedures for GenAI applications by aligning organizational controls with trust principles, regulatory requirements, and AI lifecycle oversight. Will be able to apply the full AI audit lifecycle to a real-world case, assess governance maturity, and produce closure documentation appropriate for either internal or external auditor roles in accordance with ISO/IEC 42001 and the NIST AI Risk Management Framework.
Audience Profile
Course Target Audience
Who should Attend:
IT Auditors and Compliance Professionals: This is the primary audience. If you're already doing IT audits (SOX, SOC 2, HIPAA, etc.) and your organization is deploying AI, you need to evolve your skillset or risk becoming obsolete. AI-based systems are inherently more complex and fundamentally different from traditional IT systems, particularly with deep learning models that lack transparent logic, introducing new risks that are not widely understood by IT auditors.
Infrastructure and Platform Engineers: A person that understands how AI systems are deployed, hosted, and operated, but adding audit methodology gives you the ability to assess whether those systems are governed properly. Most auditors struggle with the technical side; a technical infrastructure person coming at it from the opposite direction is incredibly valuable.
Risk and Governance Leaders: RCISOs, GRC managers, and anyone responsible for enterprise risk management. As AI gets embedded into business processes, they need to understand what controls should exist and how to verify them.
At Course Completion
Course Objectives
Define key AI concepts including Artificial Intelligence, Machine Learning, Generative AI, LLMs, and Small Language Models (SLMs).
Explain how LLMs generate responses and identify the risks of hallucination, inconsistency, and lack of explainability.
Differentiate between traditional automation, ML models, GenAI, and SLMs in terms of audit risk and control requirements.
Evaluate core risks that apply across all AI systems, including data bias, model drift, and overfitting.
Describe the auditor’s responsibilities in reviewing early-stage GenAI adoption, with attention to role separation between internal and external audit functions.
Compare major AI auditing and governance frameworks used by internal and external auditors (NIST AI RMF, ISO/IEC 42001, EU AI Act, GAIA, etc.).
Apply risk and governance concepts in a hands-on LLM prompt lab and generate auditor-aligned reflections.
Identify key points in the AI lifecycle where internal and external auditors should engage
Differentiate responsibilities between audit, IT, and data science teams regarding AI risk
Recognize the audit implications of model degradation, drift, and bias ownership
Apply prompt engineering as a method to surface audit-relevant model behaviors
Interpret how prompt-based audit observations support assurance, documentation, and escalation
Outline
Course Outline
Understanding and Auditing AI Applications
Learning Path 1: Understanding AI Systems and Establishing Audit Scope
Module 1: Exploring AI and Generative AI Services
Intro and Objectives
AI Fundamentals
GenAI and Language Models
Risk Awareness
Frameworks and Governance
Internal versus External Role Comparison
Module 1 – Lab: Exploring LLM Behavior and Drafting AI Compliance Assessments
Compliance Assessments
Summary and Takeaways
Module 2: How to Audit the Intricate Components of AI Applications
What to Audit in AI Systems
How Audit Checklist Items Map to Frameworks
Metrics for Evaluating GenAI Outputs
Mapping Metrics to NIST AI RMF Functions
GenAI Output Review: Internal vs External Auditor Roles
Introduction to Auditing Tools
Auditing Tools by Role
Module 2 – Lab Part A: Evaluating and Tracking Compliance Measures with Microsoft Purview Compliance Manager
Module 2 – Lab Part B: Fairness and Bias Review Using Aequitas
Classifying Data and AI Models
Real-World Risk Case: $1 Car Chatbot
Module Summary and Takeaways
Module 3: Investigating Internal AI Usage – Governance
Introduction to Internal AI Governance
AI Activity Logging and Monitoring Practices
Using Microsoft Purview to Audit AI Usage
Policy Adherence and Risk Signal Evaluation
Internal vs. External Auditor Responsibilities
Module 3 – Lab1: Governance Audit of AI-driven Traffic Sign Recognition
Module 3 – Lab 2: Reviewing AI Prompt Trails with Microsoft Purview
Summary and Key Takeaways
Knowledge Check
Learning Path 2: Structuring Risk-Base AI Engagements
Module 4: Redefining Audit Engagement Across the AI Lifecycle
Framing the Auditor’s Role in AI Governance
Auditor Engagement Across the AI Lifecycle
Who Owns AI Risk? Role Differentiation Matrix
Understanding Model Degradation, Drift, and Accountability
Prompt Engineering as an Audit Tool
Prompt Audit Patterns: Red Flag Prompts for Risk Discovery
Module 4 – Lab: Conducting a Prompt-Based AI Audit
Reflection and Role Exercise: Who Should Respond to This Risk?
Learning Path 3: Executing Fieldwork Across the AI Lifecycle
Module 5: Execute AI Project Management Efficiently
AI Project Governance: Scope and Oversight
Auditing the AI Vision, Strategy, and Roadmap
Evaluating Project Roles and Cross-Functional Accountability
Auditing Risk Registers, Use Case Alignment, and Business Impact
Module 5 – Lab: Reviewing AI Project Governance Templates
Summary and Key Takeaways
Knowledge Check
Module 6: Monitoring AI Systems and Governance in Action
Monitoring What Matters
What Should Be Audited Post-Deployment
Types of Audit Evidence: Logs, Outputs, Labeling, Risk
Frameworks in Action
Manual Audit Techniques: No-Tools? No Problem
Module 6 Lab: Documenting AI Risk Using the Risk Register tool
Summary and Maturity Takeaways
Knowledge Check
AI Governance, Monitoring, and Capstone Execution
Learning Path 4: Assessing Maturity, Governance, and Strategic Closure
Module 7: Designing Governance in New AI and GenAI Applications
AI Governance Principles and Standards Overview
Translating Governance Principles into Policy
Domain-Based Governance Structures
Auditing Governance Implementation
Governance Gaps and Red Flags
Knowledge Check and Reflection Questions
Module 7 Lab: Assessing Your Organizations AI Governance Maturity
Summary and Key Takeaways
Module 8: Auditing AI Improvement Cycles and Profile-Driven Risk Tailoring
Understanding AI Improvement Obligations.
NIST AI RMF Profiles and Audit Customization.
Auditing the AI Feedback Loop: Are Controls Evolving?
Internal versus External Auditor Roles in the Improvement Lifecycle.
Evaluating Evidence of Corrective and Preventive Actions.
Knowledge Checks.
Module 8 - Lab: Auditing Evidence of AI Governance Improvement
Summary and Key Takeaways
Module 9: Administering Trust and Accountability in Emerging AI Platforms
Introduction to Trust and AI Platform Governance
Auditing Platform-Level Trustworthiness Characteristics
Provisioning and Onboarding AI Services Securely
Controls for Post-Deployment Behavior and Drift
Auditing Multimodal AI and GenAI Capabilities
Module 9 - Lab: Privacy Trust Assessment
Summary and Key Takeaways
Module 10: Finalizing the AI Audit – Synthesis, Reporting, and Strategic Readiness
Reviewing Multi-Domain AI Audit Findings
Mapping Risks to ISO Clauses and NIST Functions
Evaluating Governance Maturity and Improvement Signals
Final Audit Judgment: Certification, Readiness, or Escalation
Internal versus External Auditor Roles in Final Reporting
Knowledge Check: Risk Readiness versus Risk Documentation
Capstone Simulation
Summary and Key Takeaways
Capstone Final Event: Business Audit Simulation
Capstone: Auditing a National AI Program – The Australian Taxation Office Case
Capstone Deliverables:
Learners will submit one of the following, based on their assigned role:
Internal Auditor Role:
A completed AI Audit Closure Memo, including:
Summary of findings
Residual risk analysis
Clause 10.2 alignment
Closure determination or monitoring plan
External Auditor Role:
A completed Readiness Opinion Letter, including:
Scope of review
Key observations
Maturity and risk assessment
Certification readiness opinion
Recommendations for improvement
Lab Outline
Understanding and Auditing AI Applications
Learning Path 1: Understanding AI Systems and Establishing Audit Scope
Optional Hands-On Lab
Module 2: How to Audit the Intricate Components of AI Applications
Optional Lab: Microsoft Purview
Optional Lab: Aequitas
Module 3: Investigating Internal AI Usage – Governance
Hands-On Lab: Reviewing Copilot Activity and Prompt Trails
Learning Path 2: Structuring Risk-Base AI Engagements
Module 4: Redefining Audit Engagement Across the AI Lifecycle
Hands-On Lab (Optional): Conducting a Prompt-Based AI Audit
Learning Path 3: Executing Fieldwork Across the AI Lifecycle
Module 5: Execute AI Project Management Efficiently
Hands-On Lab: Reviewing AI Project Governance Templates
Module 6: Monitoring AI Systems and Governance in Action
Optional Lab: Investigating Copilot and Purview Logs (Demo)
AI Governance, Monitoring, and Capstone Execution
Learning Path 4: Assessing Maturity, Governance, and Strategic Closure
Module 7: Designing Governance in New AI and GenAI Applications
Optional Lab: Assigning AI and Data Governance Roles
Module 8: Auditing AI Improvement Cycles and Profile-Driven Risk Tailoring
Lab: Auditing Evidence of AI Governance Improvement.
Module 9: Administering Trust and Accountability in Emerging AI Platforms
Lab: Privacy Trust Assessment
Prerequisites
Course Prerequisites
A fundamental understanding of AI
Familiarity with the IIA AI Framework
Review the NIST AI RMF