How can you measure AI interpretability?

The two main reasons for AI interpretatability is: Integration with domain expertise: Domain experts will always want to add their subjective knowledge to the system’s decisions or forecasts in order to incorporate data either not considered by the AI or that is too new and hence not used for training the model. Additionally, adjustments by domain experts can enhance the accuracy in many settings, such as in demand forecasting, where market changes need to be incorporated fast. This process is difficult unless we have an understanding of why AI is making certain decisions and which features are being most relied on. SHAP values can be helpful in this regard as they provide global as well as local explanations.

Increased chance of adoption and success: At the end of the day as a data scientist you need to convince key stakeholders and executives to adopt your model, even if your out of sample accuracy is very high. Business stakeholders cannot be expected to take significant risks by deploying models they do not understand. Making your model interpretable will help business stakeholders understand the decisions and develop trust in your model.

In one of the my recent projects, we've implemented what we call "explainability scores" that allows business users to understand each AI result and what components of the model contributed to AI's recommendations. It allows the team to then troubleshoot recommendations, if reported to be inaccurate by the users. This score also provides transparency and allows business users to have a view into the AI model, without a high dependency on the engineering team to provide lengthy evaluations and reporting.

Measuring AI interpretability involves quantitative and qualitative assessment. Use metrics like LIME or SHAP values to quantify interpretability. Conduct user surveys and task performance evaluations to assess qualitative interpretability. Balance interpretability with performance, as highly interpretable models may sacrifice accuracy.

Measuring AI interpretability is a multifaceted task that depends on several dimensions. These include the type of AI system, the level of interpretability needed, the specific task, the domain, the user, and the system's purpose. Additionally, factors like internal complexity, interactions, problem clarity, impact, user knowledge, and intent are all crucial in assessing interpretability. Evaluating AI interpretability requires a comprehensive consideration of these dimensions, as they vary based on the context and objectives of the AI system.

Measuring quality of the output and benchmarking AI against accepted gold standard datasets allows you to understand how your model is performing, and whether it requires further fine-tuning. User satisfaction scores with the model output are also important to capture user intent and assess overall usefulness of the tool. If users are not finding the output appropriate or accurate, they would not be as satisfied with the tool, which could be contributing to lower satisfaction scores (as an example).

To measure AI interpretability, use a mix of user studies for qualitative insights and metrics like fidelity for technical assessment. Improvements can be made based on these evaluations. If fidelity is low, consider more transparent models or better feature engineering. If users find the model hard to understand, enhance explanation presentation, perhaps through a simplified dashboard. This targeted approach helps in building user trust and model accountability efficiently.

Evaluating AI interpretability is pivotal in ensuring transparency and user trust in artificial intelligence systems. Functional tests examine accuracy, complexity, consistency, and regulatory compliance of AI explanations. User tests gauge satisfaction, trust, understanding, and engagement, offering valuable insights into human interactions with AI. Impact tests assess social, ethical, and environmental consequences. Comprehensive documentation and transparent reporting foster accountability, enabling continuous improvement. Rigorous evaluation empowers AI developers to create accurate, understandable, and ethical systems, enhancing user trust and positively impacting society and the environment.