Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward

Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward

INTRODUCTION

Legions of businesses, governments, and nonprofits are starting to cash in on the value AI can deliver. Between 2017 and 2018, It is found the percentage of companies inserting at least one AI capability in their business processes more than doubled, with nearly all companies using AI reporting achieving some level of value. Not surprisingly, though, as AI amplifies business and society, CEOs are under the spotlight to ensure their company’s responsible use of AI systems beyond complying with the spirit and letter of applicable laws. Ethical debates are well underway about what’s “right” and “wrong” when it comes to high-stakes AI applications such as autonomous weapons and surveillance systems. And there’s an outpouring of concern and skepticism regarding how we can imbue AI systems with human ethical judgment when moral values often vary by culture and can be difficult to code in the software. While these big moral questions touch a select number of organizations, nearly all companies must grapple with another stratum of ethical considerations, because even seemingly innocuous uses of AI can have grave implications. Numerous instances of AI bias, discrimination, and privacy violations have already littered the news, leaving leaders rightly concerned about how to ensure that nothing bad happens as they deploy their AI systems. The best solution is almost certainly not to avoid the use of AI altogether—the value at stake can be too significant, and there are advantages to being early to the AI game. Organizations can instead ensure the responsible building and application of AI by surety confirms that AI outputs are fair, that new levels of personalization do not translate into discrimination, that data acquisition and use do not occur at the expense of consumer privacy, and that their organizations balance system performance with clarity into how AI systems make their predictions.

DEFINITION AND OTHER FEATURES

AI: A quest or a pursuit of a Definition

AI has been defined in many ways. Today, it comprises several techno-scientific branches. Altogether, AI paradigms still satisfy the classic definition provided by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon in their seminal Proposal for the Dartmouth Summer Research Project on AI, the founding document and later event that established the new field of AI in 1955: For the present purpose the AI problem is taken to be that of making a machine behave in ways that would be called intelligent if a human were so behaving. This is contrary to fact: If a human behaves intelligently. Just only means that the machine is intelligent, or even thinking. The latter scenario is a fallacy and spanks of superstition. Just because a dishwasher cleans the dishes as well or needs any intelligence to achieve its task. The same counterfactual understanding of AI underpins the Turing test, which, in this case, checks the ability of a machine to perform a task in such a way that the conclusion would be indistinguishable from the outcome of a human agent working to achieve the same task. The classic definition enables one to conceptualize AI as a growing resource of interactive, autonomous, and often self-learning agencies (in the ML sense), that can deal with tasks that would otherwise require human intelligence and intervention to be performed successfully. AI is defined on the basis of engineered outcomes and actions and so, in what follows, we shall treat AI as a reservoir of smart agency on tap. This is adequately general to grasp the many ways in which AI is discussed in the documents.

DEFINITION

AI is the branch of computer science that deals with the simulation of intelligent behavior in computers as regards their capacity to mimic, and ideally improve, human behavior. We have to achieve the simulation of human cognition and functions, including learning and problem-solving, are required. This simulation may limit itself to some simple predictable features, thus limiting human complexity. It is already having a major impact on society. The key questions are how, where, when, and by whom the impact of AI will be felt. As a result, many organizations have launched a wide range of initiatives to establish ethical principles for the adoption of socially beneficial AI. Sadly, the sheer volume of proposed principles threatens to become overwhelming and confusing, posing two potential problems. Either the various sets of ethical principles for AI are similar, leading to pointless repetition and redundancy, or, if they differ significantly, confusion and ambiguity will result instead. The worst outcome would be a ‘market for principles’ where stakeholders may be tempted to ‘shop’ for the most appealing ones. The problem of ‘principal proliferation’ we are facing. The results of a fine-grained analysis of several of the highest-profile sets of ethical principles for AI. We assess whether these principles are convergent, with a set of agreed-upon principles, or divergent, with significant dissent over what constitutes ‘ethical AI.’ The analysis finds a high degree of overlap among the sets of principles we analyze. We then identify a predominant framework consisting of five core principles for ethical AI. We found the limitations and assess the implications of this ethical framework for future efforts to create laws, rules, standards, and best practices for ethical AI in a wide range of contexts.

An integrated Framework of Five Principles for

AI became a self-standing discipline in the year 1955. After years of consistent and immense development over the last decades. AI recourses to ML to devise a predictive functioning based on data acquired from a given framework. The strength of ML is inherent in its capacity to acquire data without the presence of any clearly programmed. ML algorithms are autonomous and self-sufficient when performing their learning function. ML algorithms are the reason for their ubiquitous AI advancement. Besides, ML operations in data science and other applied fields are hypothesized in the context of a final decision-making application makes its prominence. Applications in our daily lives encompass fields, such as precision in agriculture, air combat, military training, education, finance, health care), human resources and recruiting, music composition, customer service, reliable engineering and maintenance, autonomous vehicles and traffic management, social-media news-feed, work scheduling and optimization and more others. In all these fields, an increasing number of functions are being ceded to algorithms to the detriment of human control, raising concern for the loss of fairness and equitability. Moreover, issues of garbage-in-garbage-out. may be prone to emerge in contexts when external control is entirely removed. This issue may be further exacerbated by the offer of new services of auto-ML where the entire algorithm development workflow is automatized and the residual human control practically removed. Decision-making on numerous aspects of our daily lives is being outsourced to ML algorithms and AI, motivated by speed and efficiency in the decision process. ML approaches—one of the typologies of algorithms underpinning AI are typically developed as black boxes. The implication is that ML code scripts are rarely scrutinized; interpretability is usually sacrificed in favor of usability and effectiveness. Room for improvement in practices associated with program development has also been flagged along other dimensions, including inter alia fairness, accuracy, accountability, and transparency. In this contribution, the production of guidelines and dedicated documents around these themes is discussed. The following applications of AI-driven decision-making are outlined: (a) risk assessment in the criminal justice system, and (b) autonomous vehicles, highlighting points of friction across ethical principles. Possible ways forward toward the implementation of governance on AI are finally examined.

In the following sections, we will (i) detail a series of research questions around the ethical principles in AI; (ii) take stock of the production of guidelines elaborated in the field; (iii) showcase their prominence in practical examples; and (iv) discuss actions towards the inclusion of these dimensions in the future of AI ethics.

Research questions on the ethics of artificial intelligence

Critical aspects of AI adoption have now had robust traction in the prevalent direction of influence. A main deficiency of ML styles is the fact these resort to proxies for driving trends, such as a person’s ZIP/PIN code or language connected to the capacity of an individual to pay back a loan or handle a job, respectively. However, these correlations may be biased, if not illegal.

Potential black swans in the code should also be considered. These have been documented, for instance, in the case of the Amazon website, for which errors, such as the quotation of plain items (often books) up to 10,000 dollars. have been reported. While errors about monetary values may be easy to spot, the situation may become more complex and less intelligible when common biases come to play. That is the reason why the number of guidelines on the topic of ethics in AI has been multiplying over the last few years. While reflections around the ethical implications of machines and automation deployment were already put forth in the ’50s and ’60s, the increasing use of AI in many fields raises new important questions about its aptness.  This stems from the complexity of the aspects undertaken and the plurality of views, stakes, and values at play. A basic aspect is how and to what extent the values and standpoints of the involved stakeholders have been taken care of in the design of the decision-making algorithm. Besides, this ex-ante evaluation, an ex-post evaluation would need to be put in place so as to monitor the consequences of AI-driven decisions in making winners and losers. To the cloud, it is prime to assess how and if ethical aspects have been included in the AI-driven decision-making implemented by asking questions such as: What are the most prominent ethical concerns raised by large-scale deployment of AI applications? How are these multiple dimensions interwoven? What are the actions the involved stakeholders are carrying out to address these concerns? What are possible ways forward to improve ML and AI development and use over their full life-cycle? We will first examine the production of relevant guidelines in the fields along with academic secondary literature. These aspects will then be discussed in the context of two applied cases: (i) recidivism-risk assessment in the criminal justice system, and (ii) autonomous vehicles.

Guidelines and secondary literature on AI ethics, its dimensions, and stakes

The production of loyal documents has been shooting up since 2016. Suggested reading on national and international AI strategies providing a comprehensive list of documents. France’s Digital Republic Act gives the right to an explanation as regards decisions on an individual made through the use of administrative algorithms. This law touches upon several aspects including How and to what extent the algorithmic processing contributed to the decision-making; Which data was processed and its source; How parameters were treated and weighted; and Which operations were carried out in the treatment. Sensitive governmental areas, such as national security and defense, and the private sector (the largest user and producer of ML algorithms by far) are excluded from this document. An international European initiative is the multi-stakeholder European Union High-Level Expert Group on Artificial Intelligence, which includes 52 experts from academia, civil society, and industry. The group produced a deliverable on the required criteria for AI trustworthiness. Even articles 21 and 22 of the recent European Union General Data Protection Regulation include passages functional to AI governance, although further action has been recently demanded from the European Parliament. Taking a cue, China has also been allocating efforts to privacy and data protection.

 As regards secondary literature, examined a list of statements/declarations elaborated since 2016 from multi-stakeholder organizations. A set of 47 principles has been identified, which mapped onto five all-embracing dimensions: beneficence, non-maleficence, autonomy, justice, and, explicability. The latter is a new dimension specifically accepted in the case of AI, while the others were already identified in the controversial domain of bioethics. After reviewing 84 documents, which were produced by several actors in the field, almost half of which were from private companies or governmental agencies. The classification proposed by Jobin & others is around a slightly different set of values: transparency, justice and fairness, non-maleficence, responsibility, and privacy. Other potentially relevant dimensions, such as accountability and responsibility, were rarely defined in the studies reviewed by these authors. Seven of the most prominent value statements from the AI/ML fields were examined in Greene & others: The Partnership on AI to Benefit People and Society; The Montreal Declaration for a Responsible Development of Artificial Intelligence; The Toronto Declaration Protecting the rights to equality and non-discrimination in machine-learning systems; OpenAI; The Centre for Humane Technology; Fairness, Accountability and Transparency in ML; Axon’s AI Ethics Board for Public Safety. Greene & others in 2019. They found seven common core elements across these documents: (i) design’s moral background (universal concerns, objectively measured); (ii) expert oversight; (iii) values-driven determinism; (iv) design as the locus of ethical scrutiny; (v) better building; (vi) stakeholder-driven legitimacy; and, (vii) machine translation. Mittelstadt in 2019 critically analyzed the current debate and actions in the field of AI ethics and noted that the dimensions addressed in AI ethics are converging towards those of medical ethics. However, this process appears problematic due to four main differences between medicine and the medical professionals on one side, and AI and its developers on the other. Firstly, the medical professional rests on common aims and fiduciary duties, which AI developers lack. Secondly, a formal profession with a set of clearly defined and governed good-behavior practices exists in medicine. This is not the case for AI, which also lacks a full understanding of the consequences of the actions enacted by algorithms. Thirdly, AI faces the difficulty of translating overarching principles into practices. Even its current setting of seeking maximum speed, efficiency, and profit clashes with the resource and time requirements of an ethical assessment and/or counseling. Finally, the accountability of professionals or institutions is at this stage mainly theoretical, having the vast majority of these guidelines been defined on a merely voluntary basis and hence with the total lack of a fractional scheme for non-compliance.

Machine-learning algorithms in the field of autonomous vehicles

The case of autonomous vehicles, also known as self-driving vehicles, poses different challenges as a continuity of decisions is to be enacted while the vehicle is moving. It is not a one-off decision as in the case of the assessment of recidivism risk. An exercise to appreciate the value generated by these decisions is the moral-machine experiment. Massachusetts Institute of Technology 2019 opined a serious game where users are requested to fulfill the function of an autonomous-vehicle decision-making algorithm in a situation of danger. This experiment entails performing choices that would prioritize the safety of some categories of users over others. For instance, choosing over the death of car occupants, pedestrians, or occupants of other vehicles, et cetera. While such extreme situations may be a simplification of reality, one cannot exclude that the algorithms driving an autonomous vehicle may find themselves in circumstances where their decisions may result in harming some of the involved parties. In practice, the issue would be framed by the algorithm in terms of a statistical trolley dilemma in the words of Bonnefon et al in 2019, whereby the risk of harm for some road users will be increased. This corresponds to a risk management situation by all means, with a number of nuances and inherent complexity.

Hence, autonomous vehicles are not bound to play the role of silver bullets, solving once and forever the vexing issue of traffic fatalities.  Further, the way decisions enacted could backfire in complex contexts in which the algorithms had no prognostic power, is an unpredictable issue one has to deal with. Coding algorithms that assure fairness in autonomous vehicles can be a very challenging issue. Contrasting and incommensurable dimensions are likely to emerge when designing an algorithm to reduce the harm of a given crash. For instance, in terms of material damage against human harm. Odds may emerge between the interest of the vehicle owner and passengers, on one side, and the collective interest of minimizing the overall harm, on the other. Minimizing the overall physical harm may be achieved by implementing an algorithm that, in the occasion of an unavoidable collision, would target the vehicles with the highest safety standards. The algorithm may also face a dilemma between a low probability of serious harm and a higher probability of mild harm. Unavoidable normative rules will need to be included in the decision-making algorithms to tackle these types of situations.

Accuracy in the context of self-autonomous vehicles rests on their capacity to correctly simulate the course of the events. While this is based on physics and can be informed by the numerous sensors these vehicles are equipped with, unforeseen events can still play a prominent role, and profoundly affect the vehicle's behavior and reactions. For instance, fatalities due to autonomous-vehicle malfunctioning were reported as caused by the following failures: (i) the incapability of perceiving a pedestrian as such (National Transport Safety Board 2018); (ii) the acceleration of the vehicle in a situation when braking was required due to contrasting instructions from different algorithms the vehicle was hinged upon (Smith, 2018). In this latter case, the complexity of autonomous vehicle algorithms was witnessed by the millions of lines of code composing their scripts, a universe no one fully understands in the words of The Guardian (Smith, 2018), so that the causality of the decisions made was practically impossible to scrutinize. Hence, no corrective action in the algorithm code may be possible at this stage, with no room for improvement in accuracy. One should also not forget that these algorithms are learning by direct experience and they may still end up conflicting with the initial set of ethical rules around which they have been conceived. Learning may occur through algorithms interaction taking place at a higher hierarchical level than the one imagined in the first place. This aspect would represent a further open issue to be taken into account in their development. It also poses further tension between the accuracy a vehicle manufacturer seeks and the capability to keep up the agreed fairness standards upstream from the algorithm development process.

POINTS OF FRICTION BETWEEN ETHICAL DIMENSIONS

Higher transparency is a common refrain when discussing the ethics of algorithms, in relation to dimensions such as how an algorithmic decision is arrived at, based on what assumptions, and how this could be corrected to incorporate feedback from the involved parties. Rudin in 2019 argued that the community of algorithm developers should go beyond explaining black-box models by developing interpretable models in the first place. On a larger scale, the use of open-source software in the context of ML applications has already been advocated for over a decade with an indirect call for tools to execute more interpretable and reproducible programming such as Jupyter Notebooks, available from 2015 onwards. However, publishing scripts expose their developers to the public scrutiny of professional programmers, who may find shortcomings in the development of the code. Ananny and Crawford in 2018 commented that resorting to full algorithmic transparency may not be an adequate means to address their ethical dimensions; opening up the black box would not suffice to disclose their modus operandi. Moreover, developers of algorithms may not be capable of explaining in plain language how a given tool works and what functional elements it is based on. A more socially relevant understanding would encompass the human/non-human interface (i.e., looking across the system rather than merely inside). Algorithmic complexity and all its implications unravel at this level, in terms of relationships rather than as mere self-standing properties.

Other authors pointed to possible points of friction between transparency and other relevant ethical dimensions. de Laat argues that transparency and accountability may even be at odds in the case of algorithms. Hence, he argues against full transparency along four main lines of reasoning: (i) leaking of privacy-sensitive data into the open; (ii) backfiring into an implicit invitation to game the system; (iii) harming the company property rights with negative consequences on their competitiveness (and on the developer's reputation as discussed above); (iv) inherent opacity of algorithms, whose interpretability maybe even hard for experts (see the example below about the code adopted in some models of autonomous vehicles). All these arguments suggest limitations to the full disclosure of algorithms, be it that the normative implications behind these objections should be carefully scrutinized. Raji & others 2020 suggest that a process of algorithmic auditing within the software-development company could help in tackling some of the ethical issues raised. Larger interpretability could be in principle achieved by using simpler algorithms, although this may come at the expense of accuracy. To this end, Watson and Floridi 2019 defined a formal framework for interpretable ML, where explanatory accuracy can be assessed against algorithmic simplicity and relevance. Loss in accuracy may be produced by the exclusion of politically critical features (such as gender, race, age, etc.) from the pool of training predictive variables. For instance, Amazon scrapped a gender-biased recruitment algorithm once it realized that despite excluding gender, the algorithm was resorting to surrogate gender variables to implement its decisions Dastin, 2018. This aspect points again to possible political issues of a trade-off between fairness, demanded by society, and algorithmic accuracy, demanded by, e.g., a private actor.

Fairness may be further hampered by reinforcement effects. This is the case of algorithms attributing credit scores, that have a reinforcement effect proportional to people's wealth that de facto rules out credit access for people in a more socially difficult condition O’Neil, 2016. According to Floridi and Cowls 2019 a prominent role is also played by the autonomy dimension; the possibility of refraining from ceding decision power to AI for overriding reasons (e.g., the gain of efficacy is not deemed fit to justify the loss of control over decision-making). In other words, machines' autonomy could be reduced in favor of human autonomy according to this meta-autonomy dimension.

MACHINE-LEARNING ALGORITHMS BASED ON AI IN THE FIELD OF CRIMINAL JUSTICE

ML algorithms have been extensively utilized to assist juridical debate in many states of the USA. This country faces the issue of the world’s highest incarcerated population, both in absolute and per-capita terms. The COMPAS algorithm, developed by the private company Northpointe, attributes a 2-year recidivism-risk score to arrested people. It also evaluates the risk of violent recidivism as a score. The fairness of the algorithm has been questioned in an investigative report, that examined a pool of cases where a recidivism score was attributed to >18,000 criminal defendants in Broward County, Florida, and flagged up a potential racial bias in the application of the algorithm. According to the authors of the report, the recidivism risk was systematically overestimated for black people: the decile distribution of white defendants was skewed towards the lower end. Conversely, the decile distribution of black defendants was only slightly decreasing towards the higher end. The risk of violent recidivism within 2 years followed a similar trend. This analysis was debunked by the company, which, however, refused to disclose the full details of its proprietary code. While the total number of variables amounts to about 140, only the core variables were disclosed. The race of the subject was not one of those.

Here, a crucial point is how this fairness is to be attained: whether it is more important a fair treatment across groups of individuals or within the same group. For instance, let us take the case of gender, where men are overrepresented in prison in comparison with women. As to account for this aspect, the algorithm may discount violent priors for men in order to reduce their recidivism-risk score. However, attaining this sort of algorithmic fairness would imply inequality of treatment across genders. Fairness could be further hampered by the combined use of this algorithm with other driving decisions on neighborhood police patrolling. The fact these algorithms may be prone to drive further patrolling in poor neighborhoods may result from a training bias as crimes occurring in public tend to be more frequently reported. One can easily understand how these algorithms may jointly produce a vicious cycle—more patrolling would lead to more arrests that would worsen the neighborhood average recidivism-risk score, which would in turn trigger more patrolling. All this would result in exacerbated inequalities, likewise the case of credit scores previously discussed in O’Neil, 2016. A potential point of friction may also emerge between the algorithm dimensions of fairness and accuracy. The latter may be theoretically defined as the classification error in terms of the rate of false positive (individuals labeled at risk of recidivism, that did not re-offend within 2 years) and false negative (individuals labeled at low risk of recidivism, that did re-offend within the same timeframe) (Loi and Christen, 2019). Different classification accuracy (the fraction of observed outcomes in disagreement with the predictions) and forecasting accuracy (the fraction of predictions in disagreement with the observed outcomes) may exist across different classes of individuals (e.g., black or white defendants). Seeking equal rates of false positives and false negatives across these two pools would imply a different forecasting error (and accuracy) given the different characteristics of the two different training pools available for the algorithm. Conversely, having the same forecasting accuracy would come at the expense of different classification errors between these two pools). Hence, a trade-off exists between these two different shades of fairness, which derives from the very statistical properties of the data population distributions the algorithm has been trained on. However, the decision-making rests again on the assumptions the algorithm developers have adopted, e.g., on the relative importance of false positives and false negatives (i.e., the weights attributed to the different typologies of errors, and the accuracy sought. When it comes to this point, an algorithm developer may decide (or be instructed) to train his/her algorithm to attribute, e.g., a five/ten/twenty times higher weight for a false negative (re-offender, low recidivism-risk score) in comparison with a false positive (non-re-offender, high recidivism-risk score).

As with all ML, an issue of transparency exists as no one knows what type of inference is drawn on the variables out of which the recidivism-risk score is estimated. Reverse-engineering exercises have been run so as to understand what are the key drivers of the observed scores. The algorithm seemed to behave differently from the intentions of their creators with a non-linear dependence on age and a weak correlation with one’s criminal history. These exercises showed that it is possible to implement interpretable classification algorithms that lead to a similar accuracy as COMPAS. Dressel and Farid in 2018. achieved this result by using a linear predictor-logistic regressor that made use of only two variables (age and the total number of previous convictions of the subject).

1. Clarify how values translate into the selection of AI applications.

Leaders must sharpen and unpack high-level value statements, using examples that show how each value translates into the real-world choices that analytics teams make on which processes (and decisions) should be candidates for automation. We have seen some great examples of companies using “mind maps” to turn corporate values into concrete guidance, both in terms of when to use AI and how. One European financial services organization systematically mapped its corporate values to AI reputational risks. The exercise led it to decide that, while AI could be used to recommend new services to clients, it should always include a “human in the loop” when advising the financially vulnerable or recently bereaved. In addition to leading mapping exercises, CEOs should ask both business and analytics leaders to explain how they interpret values in their work and how they use these values to make better decisions. This can jump-start conversations that identify and clear up any fuzzy areas.

2. Provide guidance on definitions and metrics used to evaluate AI for bias and fairness.

Value statements can also fall short when it comes to how concepts such as bias and fairness should be defined and measured in the context of assessing AI solutions. For example, as data scientists review an automated resume-screening system for gender bias, they could use a metric that ensures similar percentages of candidates are selected (known as parity) or one that is equally predictive of future successes among candidates (known as equal opportunity), or, if the company is striving for a more representative workforce, they can ensure the system recommends a diverse set of candidates.

As a result, leaders need to steer their organizations toward defining and setting metrics that best align AI with company values and goals. CEOs should make clear exactly what the company goals and values are in various contexts, ask teams to articulate values in the context of AI and encourage a collaborative process in choosing metrics. Following employee concerns over AI projects for the defense industry, Google developed a broad set of principles to define responsible AI and bias and then backed it with tools and training for employees. One technical training module on fairness has helped more than 21,000 employees learn about the ways that bias can crop up in training data and helped them master techniques to identify and mitigate them.

3. Advise on the hierarchy of company values.

AI development always involves trade-offs. For instance, when it comes to model development, there is often a perceived trade-off between the accuracy of an algorithm and the transparency of its decision-making, or how easily predictions can be explained to stakeholders. Too great a focus on accuracy can lead to the creation of “black box” algorithms in which no one can say for certain why an AI system made the recommendation it did. Likewise, the more data that models can analyze, the more accurate the predictions, but also, often, the greater the privacy concerns. What should a model-development team do in these instances if, for example, a company’s values state both to strive to build the best products and to always ensure customer satisfaction? A leader’s business judgment is necessary to help teams make the best decisions possible as they navigate trade-offs.

Leaders should also emphasize their organization’s diversity values and make sure they’re translating into diverse analytics teams. Diverse people bring a variety of experiences, which gives rise to not only the innovative approaches needed to solve tough problems but also those required to prevent bias. For example, an all-male team building a resume-scanning model might generate a hypothesis that continuous employment is an indicator of good job performance, overlooking the need to modify the hypothesis to address how maternity leave can affect career history. Gender diversity, however, isn’t enough. Leaders should also stress other types of diversity, such as different ages, ethnicities, disciplines, and backgrounds to ensure teams represent a broad range of experiences and perspectives.

Beyond values: Five areas demanding leadership

While ensuring that company values can be more easily applied to AI-development decisions is a foundational step in responsibly building AI, it’s not enough. There are too many instances in which well-intentioned and talented data science teams have accidentally waded into murky waters and their organizations were dragged into a riptide of negative press. Advances in AI techniques and the expanding use of AI only complicate matters by continually shifting the line that data scientists must walk. As a result, CEOs need to dig deeper, challenging analytics teams to evaluate their actions in the blistering heat of public opinion in five key areas.

1. Appropriate data acquisition

Data serve as the fuel for AI. In general, the more data used to train systems, the more accurate and insightful the predictions. However, pressure on analytics teams to innovate can lead to the use of third-party data or the repurposing of existing customer data in ways that, while not yet covered by regulations, are considered inappropriate by consumers. For example, a healthcare provider might buy data about its patients—such as what restaurants they frequent or how much TV they watch—from data brokers to help doctors better assess each patient’s health risk. While the health system believes the acquisition and use of this data are in the best interest of its patients (after all, office visits are short, and this knowledge can help guide its physicians as to a patient’s greatest risks), many patients might perceive this as an invasion of privacy and worry that the data might paint an incomplete picture of their lives and lead to redundant or inaccurate medical pieces of advice.

As a result, leaders must be vigilant in asking data-science teams where they acquire data from and how the data will be used and challenge them to consider how customers and society might react to their approach. For example, a financial institution that wanted to provide additional assistance to financially vulnerable customers developed capabilities to identify digital behaviors that indicated likely mental health issues. However, the organization chose not to include this dimension in the final AI system, because of potential customer reaction to such classification—despite the best of intentions.

2. Data-set suitability

Ensuring data sets accurately reflect all of the populations being analyzed is a rightfully hot topic, given that underrepresentation of groups can lead to different impacts for different cohorts. For example, a facial-recognition system trained on a data set that included far more images of white males can fail to identify women and people of color as a result. While racial, gender and other human biases top the list, leaders should also consider the impact of more mundane data biases, such as time-selection bias—where, for example, data sets used to train a predictive-maintenance algorithm to miss a failure because they draw on only nine months, rather than several years, of data. As history has shown, hard-working data scientists in the thick of deadlines and drowning in data can think they have covered all bases, when in fact they have not. For instance, we have seen many analytics teams exclude protected variables from a model input without checking whether there is a conflation with other input variables, such as zip codes and income data. As a result, leaders must ask data-science teams fairly granular questions to understand how they sampled the data to train their models. Do data sets reflect real-world populations? Have they included data that are relevant to minority groups? Will performance tests during model development and use uncover issues with the data set? What could we be missing?

3(a). Fairness of AI outputs

Even when data sets do reflect real-world populations, the AI outputs may still be unfair due to historical biases. Machine learning algorithms, which have driven the most recent advances in AI, detect patterns and make predictions and recommendations from data and experiences. They have no awareness of the context in which their decisions will be applied or the implications of these decisions. As a result, it is easy for historic human biases and judgment to cloud predictions—anything from which prisoners should get paroled to which customers should get loans or special offers to which job applicants should get interviews. Even sophisticated organizations can “sleepwalk” into industrializing and perpetuating historical bias, as a leading tech company found when it discovered that its resume-screening algorithm was discriminating in favor of male candidates (because, historically, men predominantly held the roles for the position to be filled).

Leaders, therefore, need to frame and ensure the adoption of a thoughtful process around “fairness by design”—first by establishing definitions and metrics for assessing fairness, as described earlier, and then continually challenging data-science teams to consider fairness during the full range of their work when doing the following: Choosing data. Maximizing fairness is not as clear-cut as removing a protected attribute or artificially accounting for historical bias. For instance, excluding gender in the resume-screening application might lead to a false sense of fairness, as the aforementioned technology company found out. Likewise, inflating the proportion of female applications included in the data sets may level the playing field for women but lead to unfair outcomes for other categories of applicants. It’s important for leaders to discuss with their teams what historical human biases might affect their AI systems and how the company can address them. In this example, to ensure a stronger representation of women applicants, a company might need to collect gender data to measure the impact of gender inclusion but not use that data in training the AI model. While collecting protected or sensitive attributes can be essential to demonstrate that an AI system is acting fairly, strong data governance is required to ensure that the data are not then used for any other purpose. Choosing “features” from the raw data. In building algorithms, data scientists must choose what elements (known as features) from the raw data an algorithm should consider. For the resume-screening system, these features might include the length of time applicants have been at their previous job, what level of education they have achieved, or which computer languages they are proficient in. Selecting these features is an iterative process and somewhat of an art. Data scientists typically work with experts from the business to generate hypotheses about what features to consider, identify the necessary data, test, and repeat. The challenge here is that if they measure model performance against a single metric (such as the accuracy of the prediction), they risk missing real-world realities that might compromise fairness. For example, a hypothesis to consider the length of time in prior roles might not account for frequent job changes military spouses often make due to relocations. Leaders who encourage their teams to use a wide range of metrics to evaluate model performance and to curate features under the auspices of larger business goals can help safeguard fairness. Developing, testing, and monitoring models. It’s easy to assume that if teams choose the right data and the right features, the resulting algorithm will deliver a fair outcome. However, there are at least a dozen commonly used modeling techniques, and different approaches (or combinations of them) can yield different results from the same data sets and features. During development, teams typically test model performance (for example, is the model performing as expected?) and are increasingly bringing in specially trained internal teams or external service providers to conduct further tests. The same rigor should be applied to testing models against the organization’s definition and metrics for fairness. And, just as model performance should be monitored throughout the life of an AI system, model fairness must also be monitored by risk teams to ensure that biases don’t emerge over time as the systems integrate new data into their decision-making. For instance, a leading pharmaceutical company used machine learning models to identify clinical-trial sites that were most at risk of having patient-safety or other compliance issues—the early versions of the models were repeatedly tested against on-the-ground reality, and the hundreds of users of the model outputs were trained to flag any anomalies.

4. Regulatory compliance and engagement

In the past, organizations outside of regulated industries such as banking and healthcare often had a lower bar when it came to data-privacy protections. With existing and emerging regulations, such as the General Data Protection Regulation (GDPR) or the California Consumer Privacy Act (CCPA), all leaders have had to reexamine how their organizations use customer data and interpret new regulatory issues such as the right to be evaluated by a human, the right to be forgotten, and automatic profiling. To that end, in the United States, telecommunication operators, for example, have pledged to stop selling data to location-aggregation services amid reports that private user data had been used without prior consent. It is incumbent on leaders not only to ask their teams what regulations might be applicable in their work and how they ensure compliance but also to make sure that their data science, regulatory, and legal teams collaborate to define clear compliance metrics for AI initiatives.

Additionally, leaders must encourage their organization to move from a compliance mindset to a co-creation mindset in which they share their company’s market and technical acumen in the development of new regulations. Recent work in the United Kingdom between the Financial Conduct Authority (FCA), the country’s banking regulator, and the banking industry offers a model for this new partnership approach. The FCA and banking industry have teamed in creating a “regulatory sandbox” where banks can experiment with AI approaches that challenge or lie outside of current regulatory norms, such as using new data to improve fraud detection or better predict a customer’s propensity to purchase products.

5. Explainability

There is a temptation to believe that as long as a complex model performs as expected, the benefits the model delivers outweigh its lack of explaining ability. Indeed, in some applications, not understanding how an algorithm made its prediction might be acceptable. For example, if a healthcare application uses image classification to conveniently, consistently, and accurately predict which skin blemishes are at high risk for skin cancer, a patient is unlikely to worry about whether the model uses the shade of the blemish, its shape, its proximity to another freckle, or any of a million other features to drive its recommendation. Ultimately, the patient’s concern is whether the recommendation is correct or not—and, if the patient is at high risk, what he or she can do about this prognosis.

In other cases, however, having an opaque model may be unacceptable (for example, it is reasonable for job or loan applicants to want to understand why they were turned down) and even a hindrance in adoption and use (for example, a store manager likely wants to understand why the system is recommending a particular product mix for his or her store before acting on the advice). The ability to explain model outputs to stakeholders is a major lever in ensuring compliance with expanding regulatory and public expectations and in fostering trust to accelerate adoption. And it offers domain experts, frontline workers, and data scientists a common language through which to discuss model outputs, so they can root out potential biases well before models are thrust into the limelight.

The nascent but rapidly maturing field of “explainable AI” (sometimes referred to as XAI) is starting to offer tools—such as SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-Agnostic Explanations), and activation atlases—that can remove the veil of mystery when it comes to AI predictions. To ensure model outputs can easily be explained to stakeholders, leaders must probe their data-science teams on the types of models they use by, for example, challenging teams to show that they have chosen the simplest performant model (not the latest deep neural network) and demanding the use of explainability techniques for naturally opaque techniques. One analytics team at a media company routinely uses such explainable AI techniques for its marketing reports, so the executive team can understand not only which customers are most likely to churn within a given period but also why. XAI allows the use of more performant predictive models while enabling the marketing team to take data-driven preventive actions to reduce churn.

There are no easy answers here. But leaders who sharpen and unpack their corporate values, build teams with a diversity of perspectives, create a language and a set of reference points to guide AI, and frequently engage with and challenge AI development teams to position themselves to create and use AI responsibly. Importantly, responsible AI builds trust with both employees and consumers. Employees will trust the insights AI delivers and be more willing to use them in their day-to-day work and help ideate new ways to use AI to create value. Consumer trust gives you the right to use consumer data appropriately, and it is these data that power and continually improve AI. Consumers will be willing to use your AI-infused products because of the trust they have in your organization, and happy to use them because they just keep getting better. It’s a virtuous cycle that drives brand reputation and an organization’s ability to innovate and compete and, most importantly, enables society to benefit from the power of AI rather than suffer from its unintended consequences.

Contrasting dimensions in terms of the theoretical framing of the issue also emerged from the review of Jobin & others in 2019.  as regards the interpretation of ethical principles, reasons for their importance, ownership, and responsibility of their implementation. This also applies to different ethical principles, resulting in the trade-offs previously discussed, difficulties in setting prioritization strategies, and operational and actual compliance with the guidelines. For instance, while private actors demand and try to cultivate trust from their users, this runs counter to the need for society to scrutinize the operation of algorithms in order to maintain developer accountability. Attributing responsibilities in complicated projects where many parties and developers may be involved, an issue known as the problem of many hands, may indeed be very difficult. Conflicts may also emerge between the requirements to overcome potential algorithm deficits in accuracy associated with large databases and the individual rights to privacy and autonomy of decision. Such conflicts may exacerbate tensions, further complicating agreeing on standards and practices.

In the following two sections, the issues and points of friction raised are examined in two practical case studies, criminal justice, and autonomous vehicles. These examples have been selected due to their prominence in the public debate on the ethical aspects of AI and ML algorithms.

DISCUSSION FOR REACHING CONCLUSIONS

It is examined the ethical dimensions affected by the application of algorithm-driven decision-making. These are entailed both ex-antes, in terms of the assumptions underpinning the algorithm development, and ex-post as regards the consequences upon society and social actors on whom the elaborated decisions are to be enforced. Decision-making-based algorithms rest inevitably on assumptions, even silent ones, such as the quality of data the algorithm is trained on, or the actual modeling relations adopted, with all the implied consequences. A decision-making algorithm will always be based on a formal system, which is a representation of a real system. As such, it will always be based on a restricted set of relevant relations, causes, and effects. It does not matter how complicated the algorithm may be or how many relations may be factored in, it will always represent the one-specific vision of the system being modeled. Eventually, the set of decision rules underpinning the AI algorithm derives from human-made assumptions, such as, where to define the boundary between action and no action, and between different possible choices. This can only take place at the human/non-human interface: the response of the algorithm is driven by these human-made assumptions and selection rules. Even the data on which an algorithm is trained is not an objective truth, they are dependent upon the context in which they have been produced. Tools for technically scrutinizing the potential behavior of an algorithm and its uncertainty already exist and could be included in the workflow of algorithm development. For instance, global sensitivity analysis may help in exploring how the uncertainty in the input parameters and modeling assumptions would affect the output. Additionally, a modeling of the modeling process would assist in the model transparency and in addressing questions such as: Are the results from a particular model more sensitive to changes in the model and the methods used to estimate its parameters, or to changes in the data?

Tools of post-normal-science inspiration for knowledge and modeling quality assessment could be adapted to the analysis of algorithms, such as the NUSAP (Numeral Unit Spread Assessment Pedigree) notation system for the management and communication of uncertainty and sensitivity auditing, respectively. Ultimately, developers should acknowledge the limits of AI, and what its ultimate function should be in the equivalent of a Hippocratic Oath for ML developers. An example comes from the field of financial modeling, with a manifesto elaborated on in the aftermath of the 2008 financial crisis. As to address these dimensions, value statements and guidelines have been elaborated by political and multi-stakeholder organizations. For instance, The Alan Turing Institute released a guide for responsible design and implementation of AI that covers the whole life cycle of design, use, and monitoring. However, the field of AI ethics is just in its infancy and it is still to be conceptualized how AI developments that encompass ethical dimensions could be attained. Some authors are pessimistic, such as Supiot who speaks of governance by numbers, where quantification is replacing the traditional decision-making system and profoundly affecting the pillar of equality of judgment. Trying to revert the current state of affairs may expose the first movers in the AI field to a competitive disadvantage. One should also not forget that points of friction across ethical dimensions may emerge, e.g., between transparency and accountability, or accuracy and fairness as highlighted in the case studies. Hence, the development process of the algorithm cannot be perfect in this setting, one has to be open to negotiation and unavoidably work with imperfections and clumsiness.

The development of decision-making algorithms remains quite obscure in spite of the concerns raised and the intentions manifested to address them. Attempts to expose to public scrutiny the algorithms developed are yet scant. As are the attempt to make the process more inclusive, with higher participation from all the stakeholders. Identifying a relevant pool of social actors may require an important effort in terms of stakeholders’ mapping so as to assure a complete, but also effective, governance in terms of the number of participants and simplicity of working procedures. The post-normal-science concept of extended peer communities could assist also in this endeavor. Example-based explanations) may also contribute to an effective engagement of all the parties by helping in bridging technical divides across developers, experts in other fields, and laypeople. An overarching meta-framework for the governance of AI in experimental technologies (i.e., robot use) has also been proposed. This initiative stems from the attempt to include all the forms of governance put forth and would rest on an integrated set of feedback and interactions across dimensions and actors. An interesting proposal comes from Berk in 2019, who asked for the intervention of super parties authorities to define standards of transparency, accuracy, and fairness for algorithm developers in line with the role of the Food and Drug Administration in the US and other regulatory bodies. A shared regulation could help in tackling the potential competitive disadvantage a first mover may suffer. The development pace of new algorithms would be necessarily reduced so as to comply with the standards defined and the required clearance processes. In this setting, seeking algorithm transparency would not be harmful to their developers as scrutiny would be delegated to entrusted intermediate parties, to take place behind closed doors.As noted by a perceptive reviewer, ML systems that keep learning are dangerous and hard to understand because they can quickly change. Thus, could an ML system with real-world consequences be “locked down” to increase transparency? If yes, the algorithm could become defective. If not, transparency today may not be helpful in understanding what the system does tomorrow. This issue could be tackled by hard-coding the set of rules on the behavior of the algorithm, once these are agreed upon among the involved stakeholders. This would prevent the algorithm-learning process from conflicting with the standards agreed upon. Making it mandatory to deposit these algorithms in a database owned and operated by this entrusted super-partes body could ease the development of this overall process.

CONCLUSION

Company values can offer a compass for the appropriate application of AI, but CEOs must provide employees with further guidance. CEOs often live by the numbers—profit, earnings before interest and taxes, and shareholder returns. These data often serve as hard evidence of CEO success or failure, but they’re certainly not the only measures. Among the softer, but equally important, success factors: making sound decisions that not only lead to the creation of value but also “do no harm.” While AI is quickly becoming a new tool in the CEO tool belt to drive revenues and profitability, it has also become clear that deploying AI requires careful management to prevent unintentional but significant damage, not only to brand reputation but, more importantly, to workers, individuals, and society as a whole. It may seem logical to delegate these concerns to data-science leaders and teams since they are the experts when it comes to understanding how AI works. However, we are finding through our work that the CEO’s role is vital to the consistent delivery of responsible AI systems and that the CEO needs to have at least a strong working knowledge of AI development to ensure he or she is asking the right questions to prevent potential ethical issues. In this article, we’ll provide this knowledge and a pragmatic approach for CEOs to ensure their teams are building AI that the organization can be proud of. In today’s business environment, where organizations often have a lot of moving parts, distributed decision-making, and workers who are empowered to innovate, company values serve as an important guide for employees—whether it is a marketing manager determining what ad campaign to run or a data scientist identifying where to use AI and how to build it. However, translating these values into practice when developing and using AI is not as straightforward as one might think. Short, high-level value statements do not always provide crystal-clear guidance in a world where “right” and “wrong” can be ambiguous and the line between innovative and offensive is thin. CEOs can provide critical guidance here in three key areas.