What are the safety considerations of using artificial intelligence in control systems?

1 AI and control systems

AI can be broadly defined as the capacity of a machine or software to carry out tasks that typically require human intelligence, such as problem-solving, decision-making, learning, and reasoning. Control systems are systems that utilize feedback loops to manage the behavior of a physical process, for instance, a robot, a vehicle, or a power plant. AI can be applied to control systems in various ways, including designing and tuning the control parameters and algorithms, learning from data and enhancing the control performance over time, adapting to changing conditions and environments, coordinating and communicating with other agents or systems, and acting autonomously and self-organizing.

2 Safety challenges of AI in control systems

While AI can offer many advantages for control systems, it also brings new sources of complexity and uncertainty that can harm the safety and dependability of the system. AI systems may be unpredictable and opaque, making them difficult to comprehend or explain when using complex or black-box models. Additionally, they may be vulnerable to errors, failures, or degradation due to hardware or software faults, data corruption, or external disturbances. Moreover, AI systems may also present ethical or moral dilemmas and raise legal or social issues. For instance, how to ensure accountability and transparency or who is liable for the results of their actions.

3 Safety approaches for AI in control systems

To address the safety challenges of AI in control systems, control engineers need to adopt a systematic and holistic approach that considers the entire life cycle of the system, from design and development to operation and maintenance. Such an approach includes verification and validation to check that the system meets the specified requirements and expectations of stakeholders, monitoring and diagnosis to observe and measure the system's behavior and performance, correction and recovery to restore the system to a normal or desired state, as well as assurance and certification to provide evidence that the system is safe and reliable. Verification and validation can be done using testing, simulation, analysis, or inspection. Monitoring and diagnosis can be performed using sensors, indicators, logs, or alarms. Correction and recovery can be achieved through feedback, redundancy, backup, or restart. Finally, assurance and certification can be done through documentation, audits, reviews, or ratings.

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?