AI Risks Stage by Stage: Operation and Monitoring - The Silent Decay of AI Models

The Inherent Instability of AI Models in Production

Unlike conventional software that, once functional, tends to remain so, AI models possess a unique characteristic: they can degrade silently over time, even without any code changes. This phenomenon, known as 'drift,' is the central challenge in the operation and monitoring phase of the AI lifecycle. Drift occurs because the world in which the model operates is constantly changing, rendering the model's initial training data and assumptions less relevant. This silent decay means that while the AI system appears to be running perfectly from a technical standpoint (no errors, good uptime), the quality of its decisions can be eroding, leading to significant business risks and financial losses, as exemplified by the Zillow case.

Understanding and Mitigating Model Drift

Model drift primarily manifests in two forms: data drift and concept drift. Data drift occurs when the input data a model receives in production deviates from its training data, even if the underlying task remains the same. For instance, a retail forecasting model trained on in-store purchases may become inaccurate with a surge in online orders. Concept drift, more insidious, happens when the relationship between inputs and correct outputs changes, making the model's fundamental assumptions obsolete, such as fraud detection models failing during unprecedented events like COVID-19. To counter these, organizations must implement:

• Continuous Monitoring: Beyond uptime and latency, actively measure the quality of the model's decisions.
• Clear Ownership: Assign specific individuals or teams responsibility for monitoring and maintaining model performance.
• Robust Logging and Retention: Ensure comprehensive logging of model operations to reconstruct past behavior and diagnose issues.
• Defined Incident Response: Establish clear protocols for responding to and escalating issues identified through monitoring.
• Third-Party Dependency Tracking: Monitor external models or APIs for changes that could impact your AI system's performance.

The Auditor's Evolving Role in AI Assurance

For internal audit and assurance professionals, auditing AI systems in operation requires a shift in perspective. While traditional IT audit principles like continuous monitoring, clear ownership, evidence retention, and incident response remain relevant, their application to AI is fundamentally different. The core challenge is that an AI system can be 'running' perfectly while simultaneously being 'wrong' in its outputs. Therefore, auditors must move beyond simply verifying that controls are operating to assessing whether the controls are effectively measuring if a still-running system is making sound decisions. This necessitates a deeper understanding of AI model behavior, drift mechanisms, and the specific metrics that indicate decision quality, rather than just system availability.