A Comprehensive AI Audit Work Program for Internal Audit Functions

Navigating the AI Audit Landscape: Provider vs. Deployer

The core challenge in auditing AI, as highlighted by the author, begins with correctly identifying an organization's role: is it an AI 'provider' or a 'deployer'? This distinction is crucial because it dictates the scope and depth of the audit. A deployer, using off-the-shelf AI tools, requires audits focused on contracts, configuration, and data flow. Conversely, a provider, who builds, trains, or substantially modifies AI models, faces more rigorous audit requirements, including validation, robustness, and extensive documentation. Internal auditors must continuously monitor for 'nexus' events, such as fine-tuning vendor models or using proprietary data for retrieval-augmented generation (RAG), which can subtly shift an organization from a deployer to a provider, thereby altering its regulatory obligations and audit needs.

Demystifying AI Audits: Expertise and the Three Lines Model

A common misconception is that AI audits demand deep data science expertise, often sidelining internal audit functions. The author refutes this, proposing a three-tiered approach to AI assurance. The first tier, focusing on governance, policies, processes, and compliance, is accessible to most internal auditors without specialized technical knowledge. The second tier, involving data and model-level testing, may require some re-performance of targeted checks, while the third, technical-security level, necessitates expert support for areas like adversarial robustness. This approach aligns with Global Internal Audit Standards, emphasizing risk-based scoping and the judicious use of external expertise. The article also stresses the importance of applying the Three Lines Model to AI governance, ensuring clear accountability for AI systems across the first (build/run), second (guardrails/oversight), and third (independent assurance) lines of defense.

Structuring an Effective AI Audit Program

The proposed AI Audit Work Program is structured across six domains, moving from foundational governance to technical security. These domains include:

• Governance and Oversight: Ensuring board oversight, a clear AI strategy, responsible-AI policies, system inventory, and risk classification.
• Risk Management: Integrating AI risks into the enterprise risk framework.
• Compliance and Internal Control: Addressing generative AI within the control framework, including acceptable-use policies and regulatory adherence (e.g., EU AI Act, FINMA).
• Data Governance and Quality: Focusing on the trustworthiness of data inputs and outputs, and vendor assurances.
• Model Lifecycle, Performance, and Trustworthiness: Covering validation, performance, bias, explainability, and human oversight.
• Technical Security, Resilience, and Incidents: Addressing LLM-specific security, adversarial threats, and incident response.

Each section of the program follows a consistent structure, moving from risk to control objective, expected control, evaluation criteria, tests, and documentation requirements. A critical aspect highlighted is the distinction between testing the design and operating effectiveness of controls, and the importance of robust evidence collection. The article also emphasizes the often-overlooked aspect of 'human oversight,' urging auditors to verify if human intervention is genuinely effective and not merely a rubber-stamping process. Finally, the program advocates for anchoring audits on principles rather than rapidly changing regulations, and for providing maturity ratings to offer actionable insights to management.

DOWNLOAD Audit Program