Auditing AI: A Critical and Evolving Role for Internal Audit

The Evolving Landscape of AI and Internal Audit's Mandate

The rapid integration of Artificial Intelligence (AI) into core business processes presents a transformative challenge for internal audit. While AI promises efficiency and innovation, its dynamic and adaptive nature fundamentally alters the traditional assurance paradigm. Internal audit's role is no longer just to verify initial implementation but to provide continuous assurance that AI systems are performing correctly, reliably, safely, and are delivering intended business value. This shift demands a re-evaluation of audit methodologies, moving away from static, point-in-time testing towards a more agile and continuous monitoring approach.

Rethinking AI Testing: Beyond Traditional Software Audits

Auditing AI is distinct from auditing traditional software. Unlike fixed code, AI agents learn and evolve, meaning their behavior can change over time. This necessitates a comprehensive testing framework that addresses not only initial functionality but also ongoing performance, consistency, and resistance to unintended outcomes. Key considerations for internal audit include:

• Defining "Working Properly": Establishing clear, measurable expectations and success criteria for AI performance.
• Scenario-Based Testing: Developing robust test sets that include historical data, edge cases, adversarial prompts, and ambiguous situations to thoroughly evaluate AI behavior.
• Continuous Monitoring: Implementing mechanisms to monitor AI in production, tracking performance metrics, user behavior changes, and potential drift over time.

The article highlights that a good AI test program should assess whether the AI performs tasks correctly, behaves reliably and safely, and creates business value. This multi-faceted approach is crucial for internal audit to provide meaningful assurance.

Practical Approaches and Future Implications for Internal Audit

The article, drawing insights from an AI itself, outlines a ten-layered testing framework for AI agents. This framework emphasizes defining clear success criteria, building comprehensive test scenarios (including adversarial and edge cases), testing accuracy, reliability, tool usage, hallucination resistance, and security. Crucially, it stresses the importance of testing end-to-end business outcomes and continuous monitoring in production. For internal audit, this implies a need to:

• Collaborate closely with IT and business units to understand AI models and their intended impact.
• Develop specialized skills in AI testing methodologies.
• Consider the feasibility of internal audit developing its own AI tools for continuous monitoring and testing of other AI systems.

The evolving nature of AI means that internal audit's role will be both critical and challenging, requiring adaptability and a forward-thinking approach to ensure organizational resilience and value creation in an AI-driven world.