Английская Википедия:Explainable artificial intelligence

Материал из Онлайн справочника
Перейти к навигацииПерейти к поиску

Шаблон:Short description Шаблон:Other uses of Шаблон:Split Шаблон:Artificial intelligence

Explainable AI (XAI), often overlapping with Interpretable AI, or Explainable Machine Learning (XML), either refers to an AI system over which it is possible for humans to retain intellectual oversight, or to the methods to achieve this.[1] The main focus is usually on the reasoning behind the decisions or predictions made by the AI[2] which are made more understandable and transparent.[3] XAI counters the "black box" tendency of machine learning, where even the AI's designers cannot explain why it arrived at a specific decision.[4][5]

XAI hopes to help users of AI-powered systems perform more effectively by improving their understanding of how those systems reason.[6] XAI may be an implementation of the social right to explanation.[7] Even if there is no such legal right or regulatory requirement, XAI can improve the user experience of a product or service by helping end users trust that the AI is making good decisions. XAI aims to explain what has been done, what is being done, and what will be done next, and to unveil which information these actions are based on.[8] This makes it possible to confirm existing knowledge, challenge existing knowledge, and generate new assumptions.[9]

Machine learning (ML) algorithms used in AI can be categorized as white-box or black-box.[10] White-box models provide results that are understandable to experts in the domain. Black-box models, on the other hand, are extremely hard to explain and can hardly be understood even by domain experts.[11] XAI algorithms follow the three principles of transparency, interpretability, and explainability. A model is transparent “if the processes that extract model parameters from training data and generate labels from testing data can be described and motivated by the approach designer.”[12] Interpretability describes the possibility of comprehending the ML model and presenting the underlying basis for decision-making in a way that is understandable to humans.[13][14][15] Explainability is a concept that is recognized as important, but a consensus definition is not available.[12] One possibility is “the collection of features of the interpretable domain that have contributed, for a given example, to producing a decision (e.g., classification or regression)”.[16] If algorithms fulfill these principles, they provide a basis for justifying decisions, tracking them and thereby verifying them, improving the algorithms, and exploring new facts.[17]

Sometimes it is also possible to achieve a high-accuracy result with white-box ML algorithms. These algorithms have an interpretable structure that can be used to explain predictions.[18] Concept Bottleneck Models, which use concept-level abstractions to explain model reasoning, are examples of this and can be applied in both image[19] and text[20] prediction tasks. This is especially important in domains like medicine, defense, finance, and law, where it is crucial to understand decisions and build trust in the algorithms.[8] Many researchers argue that, at least for supervised machine learning, the way forward is symbolic regression, where the algorithm searches the space of mathematical expressions to find the model that best fits a given dataset.[21][22][23]

AI systems optimize behavior to satisfy a mathematically specified goal system chosen by the system designers, such as the command "maximize the accuracy of assessing how positive film reviews are in the test dataset." The AI may learn useful general rules from the test set, such as "reviews containing the word "horrible" are likely to be negative." However, it may also learn inappropriate rules, such as "reviews containing 'Daniel Day-Lewis' are usually positive"; such rules may be undesirable if they are likely to fail to generalize outside the training set, or if people consider the rule to be "cheating" or "unfair." A human can audit rules in an XAI to get an idea of how likely the system is to generalize to future real-world data outside the test set.[24]

Goals

Cooperation between agents – in this case, algorithms and humans – depends on trust. If humans are to accept algorithmic prescriptions, they need to trust them. Incompleteness in formal trust criteria is a barrier to optimization. Transparency, interpretability, and explainability are intermediate goals on the road to these more comprehensive trust criteria.[25] This is particularly relevant in medicine,[26] especially with clinical decision support systems (CDSS), in which medical professionals should be able to understand how and why a machine-based decision was made in order to trust the decision and augment their decision-making process.[27]

AI systems sometimes learn undesirable tricks that do an optimal job of satisfying explicit pre-programmed goals on the training data but do not reflect the more nuanced implicit desires of the human system designers or the full complexity of the domain data. For example, a 2017 system tasked with image recognition learned to "cheat" by looking for a copyright tag that happened to be associated with horse pictures rather than learning how to tell if a horse was actually pictured.[5] In another 2017 system, a supervised learning AI tasked with grasping items in a virtual world learned to cheat by placing its manipulator between the object and the viewer in a way such that it falsely appeared to be grasping the object.[28][29]

One transparency project, the DARPA XAI program, aims to produce "glass box" models that are explainable to a "human-in-the-loop" without greatly sacrificing AI performance. Human users of such a system can understand the AI's cognition (both in real-time and after the fact) and can determine whether to trust the AI.[30] Other applications of XAI are knowledge extraction from black-box models and model comparisons.[31] In the context of monitoring systems for ethical and socio-legal compliance, the term "glass box" is commonly used to refer to tools that track the inputs and outputs of the system in question, and provide value-based explanations for their behavior. These tools aim to ensure that the system operates in accordance with ethical and legal standards, and that its decision-making processes are transparent and accountable. The term "glass box" is often used in contrast to "black box" systems, which lack transparency and can be more difficult to monitor and regulate.[32] The term is also used to name a voice assistant that produces counterfactual statements as explanations.[33]

Explainability versus interpretability

There is a difference between the terms explainability and interpretability in the context of AI.[34]

Term Definition Source
Interpretability "level of understanding how the underlying (AI) technology works" ISO/IEC TR 29119-11:2020(en), 3.1.42[35]
Explainability "level of understanding how the AI-based system ... came up with a given result" ISO/IEC TR 29119-11:2020(en), 3.1.31[35]

History and methods

During the 1970s to 1990s, symbolic reasoning systems, such as MYCIN,[36] GUIDON,[37] SOPHIE,[38] and PROTOS[39][40] could represent, reason about, and explain their reasoning for diagnostic, instructional, or machine-learning (explanation-based learning) purposes. MYCIN, developed in the early 1970s as a research prototype for diagnosing bacteremia infections of the bloodstream, could explain[41] which of its hand-coded rules contributed to a diagnosis in a specific case. Research in intelligent tutoring systems resulted in developing systems such as SOPHIE that could act as an "articulate expert", explaining problem-solving strategy at a level the student could understand, so they would know what action to take next. For instance, SOPHIE could explain the qualitative reasoning behind its electronics troubleshooting, even though it ultimately relied on the SPICE circuit simulator. Similarly, GUIDON added tutorial rules to supplement MYCIN's domain-level rules so it could explain the strategy for medical diagnosis. Symbolic approaches to machine learning relying on explanation-based learning, such as PROTOS, made use of explicit representations of explanations expressed in a dedicated explanation language, both to explain their actions and to acquire new knowledge.[40]

In the 1980s through the early 1990s, truth maintenance systems (TMS) extended the capabilities of causal-reasoning, rule-based, and logic-based inference systems.[42]Шаблон:Rp A TMS explicitly tracks alternate lines of reasoning, justifications for conclusions, and lines of reasoning that lead to contradictions, allowing future reasoning to avoid these dead ends. To provide an explanation, they trace reasoning from conclusions to assumptions through rule operations or logical inferences, allowing explanations to be generated from the reasoning traces. As an example, consider a rule-based problem solver with just a few rules about Socrates that concludes he has died from poison: Шаблон:Blockquote

By the 1990s researchers began studying whether it is possible to meaningfully extract the non-hand-coded rules being generated by opaque trained neural networks.[43] Researchers in clinical expert systems creatingШаблон:Clarify neural network-powered decision support for clinicians sought to develop dynamic explanations that allow these technologies to be more trusted and trustworthy in practice.[7] In the 2010s public concerns about racial and other bias in the use of AI for criminal sentencing decisions and findings of creditworthiness may have led to increased demand for transparent artificial intelligence.[5] As a result, many academics and organizations are developing tools to help detect bias in their systems.[44]

Marvin Minsky et al. raised the issue that AI can function as a form of surveillance, with the biases inherent in surveillance, suggesting HI (Humanistic Intelligence) as a way to create a more fair and balanced "human-in-the-loop" AI.[45]

Modern complex AI techniques, such as deep learning and genetic algorithms, are naturally opaque.[46] To address this issue, methods have been developed to make new models more explainable and interpretable.[47][14][13][48][49][50] This includes layerwise relevance propagation (LRP), a technique for determining which features in a particular input vector contribute most strongly to a neural network's output.[51][52] Other techniques explain some particular prediction made by a (nonlinear) black-box model, a goal referred to as "local interpretability".[53][54][55][56][57][58] The mere transposition of the concepts of local interpretability into a remote context (where the black-box model is executed at a third party) is Шаблон:Vague.Шаблон:Clarify[59][60]

There has been work on making glass-box models which are more transparent to inspection.[18][61] This includes decision trees,[62] Bayesian networks, sparse linear models,[63] and more.[64] The Association for Computing Machinery Conference on Fairness, Accountability, and Transparency (ACM FAccT) was established in 2018 to study transparency and explainability in the context of socio-technical systems, many of which include artificial intelligence.[65][66]

Some techniques allow visualisations of the inputs to which individual software neurons respond to most strongly. Several groups found that neurons can be aggregated into circuits that perform human-comprehensible functions, some of which reliably arise across different networks trained independently.[67][68]

There are various techniques to extract compressed representations of the features of given inputs, which can then be analysed by standard clustering techniques. Alternatively, networks can be trained to output linguistic explanations of their behaviour, which are then directly human-interpretable.[69] Model behaviour can also be explained with reference to training data—for example, by evaluating which training inputs influenced a given behaviour the most.[70]

The use of explainable artificial intelligence (XAI) in pain research, specifically in understanding the role of electrodermal activity for automated pain recognition: hand-crafted features and deep learning models in pain recognition, highlighting the insights that simple hand-crafted features can yield comparative performances to deep learning models and that both traditional feature engineering and deep feature learning approaches rely on simple characteristics of the input time-series data.[71]

Regulation

As regulators, official bodies, and general users come to depend on AI-based dynamic systems, clearer accountability will be required for automated decision-making processes to ensure trust and transparency. The first global conference exclusively dedicated to this emerging discipline was the 2017 International Joint Conference on Artificial Intelligence: Workshop on Explainable Artificial Intelligence (XAI).[72]

The European Union introduced a right to explanation in the General Data Protection Right (GDPR) to address potential problems stemming from the rising importance of algorithms. The implementation of the regulation began in 2018. However, the right to explanation in GDPR covers only the local aspect of interpretability. In the United States, insurance companies are required to be able to explain their rate and coverage decisions.[73] In France the Loi pour une République numérique (Digital Republic Act) grants subjects the right to request and receive information pertaining to the implementation of algorithms that process data about them.

Limitations

Despite ongoing endeavors to enhance the explainability of AI models, they persist with several inherent limitations.

Adversarial parties

By making an AI system more explainable, we also reveal more of its inner workings. For example, the explainability method of feature importance identifies features or variables that are most important in determining the model's output, while the influential samples method identifies the training samples that are most influential in determining the output, given a particular input.[74] Adversarial parties could take advantage of this knowledge.

For example, competitor firms could replicate aspects of the original AI system in their own product, thus reducing competitive advantage.[75] An explainable AI system is also susceptible to being “gamed”—influenced in a way that undermines its intended purpose. One study gives the example of a predictive policing system; in this case, those who could potentially “game” the system are the criminals subject to the system's decisions. In this study, developers of the system discussed the issue of criminal gangs looking to illegally obtain passports, and they expressed concerns that, if given an idea of what factors might trigger an alert in the passport application process, those gangs would be able to “send guinea pigs” to test those triggers, eventually finding a loophole that would allow them to “reliably get passports from under the noses of the authorities”.[76]

Technical complexity

A fundamental barrier to making AI systems explainable is the technical complexity of such systems. End users often lack the coding knowledge required to understand software of any kind. Current methods used to explain AI are mainly technical ones, geared toward machine learning engineers for debugging purposes, rather than toward the end users who are ultimately affected by the system, causing “a gap between explainability in practice and the goal of transparency”.[74] Proposed solutions to address the issue of technical complexity include either promoting the coding education of the general public so technical explanations would be more accessible to end users, or providing explanations in layperson terms.[75]

The solution must avoid oversimplification. It is important to strike a balance between accuracy – how faithfully the explanation reflects the process of the AI system – and explainability – how well end users understand the process. This is a difficult balance to strike, since the complexity of machine learning makes it difficult for even ML engineers to fully understand, let alone non-experts.[74]

Understanding versus trust

The goal of explainability to end users of AI systems is to increase trust in the systems, even “address concerns about lack of ‘fairness’ and discriminatory effects”.[75] However, even with a good understanding of an AI system, end users may not necessarily trust the system.[77] In one study, participants were presented with combinations of white-box and black-box explanations, and static and interactive explanations of AI systems. While these explanations served to increase both their self-reported and objective understanding, it had no impact on their level of trust, which remained skeptical.[78]

This outcome was especially true for decisions that impacted the end user in a significant way, such as graduate school admissions. Participants judged algorithms to be too inflexible and unforgiving in comparison to human decision-makers; instead of rigidly adhering to a set of rules, humans are able to consider exceptional cases as well as appeals to their initial decision.[78] For such decisions, explainability will not necessarily cause end users to accept the use of decision-making algorithms. We will need to either turn to another method to increase trust and acceptance of decision-making algorithms, or question the need to rely solely on AI for such impactful decisions in the first place.

However, some emphasize that the purpose of explainability of artificial intelligence is not to merely increase users' trust in the system's decisions, but to calibrate the users' level of trust to the correct level.[79] According to this principle, too much or too little user trust in the AI system will harm the overall performance of the human-system unit. When the trust is excessive, the users are not critical of possible mistakes of the system and when the users do not have enough trust in the system, they will not exhaust the benefits inherent in it.

Criticism

Some scholars have suggested that explainability in AI should be considered a goal secondary to AI effectiveness, and that encouraging the exclusive development of XAI may limit the functionality of AI more broadly.[80][81] Critiques of XAI rely on developed concepts of mechanistic and empiric reasoning from evidence-based medicine to suggest that AI technologies can be clinically validated even when their function cannot be understood by their operators.[80]

Some researchers advocate the use of inherently interpretable machine learning models, rather than using post-hoc explanations in which a second model is created to explain the first. This is partly because post-hoc models increase the complexity in a decision pathway and partly because it is often unclear how faithfully a post-hoc explanation can mimic the computations of an entirely separate model.[18] However, another view is that what is important is that the explanation accomplishes the given task at hand, and whether it is pre or post-hoc doesn't matter. If a post-hoc explanation method helps a doctor diagnose cancer better, it is of secondary importance whether it is a correct/incorrect explanation.

The goals of XAI amount to a form of lossy compression that will become less effective as AI models grow in their number of parameters. Along with other factors this leads to a theoretical limit for explainability.[82]

Explainability in related fields

Explainability was studied also in related fields.

Voting

Cailloux and Endriss[83] present a method for explaining voting rules using the axioms that characterize them. They exemplify their method on the Borda voting rule .

Peters, Procaccia, Psomas and Zhou[84] present an algorithm for explaining the outcomes of the Borda rule using O(m2) explanations, and prove that this is tight in the worst case.

Participatory budgeting

Yang, Hausladen, Peters, Pournaras, Fricker and Helbing[85] present an empirical study of explainability in participatory budgeting. They compared the Greedy and the Equal Shares rules, and three types of explanations: mechanism explanation (a general explanation of how the aggregation rule works given the voting input), individual explanation (explaining how many voters had at least one approved project, at least 10000 CHF in approved projects), and group explanation (explaining how the budget is distributed among the districts and topics). They compared the perceived trustworthyness and fairness of Greedy and Equal Shares, before and after the explanations. They found out that, for MES, Mechanism explanation yields the highest increase in perceived fairness and trustworthiness; the second-highest was Group explanation. For Greedy, Mechanims explanation increases perceived trustworthiness but not fairness, whereas Individual explanation increases both perceived fairness and trustworthiness. Group explanation decreases the perceived fairness and trustworthiness.

Payoff allocation

Nizri, Azaria and Hazon[86] present an algorithm for computing explanations for the Shapley value. Given a coalitional game, their algorithm decomposes it to sub-games, for which it is easy to generate verbal explanations based on the axioms characterizing the Shapley value. The payoff allocation for each sub-game is perceived as fair, so the Shapley-based payoff allocation for the given game should seem fair as well. An experiment with 210 human subjects shows that, with their automatically generated explanations, subjects perceive Shapley-based payoff allocation as significantly fairer than with a general standard explanation.

See also

References

Шаблон:Reflist

External links

Шаблон:Differentiable computing

Партнерские ресурсы
Криптовалюты
Магазины
Хостинг
Разное

  1. ↑ Шаблон:Cite journal
  2. ↑ Шаблон:Cite journal
  3. ↑ Шаблон:Cite journal
  4. ↑ Шаблон:Cite journal
  5. ↑ 5,0 5,1 5,2 Шаблон:Cite news
  6. ↑ Шаблон:Cite journal
  7. ↑ 7,0 7,1 Шаблон:Cite journal
  8. ↑ 8,0 8,1 Шаблон:Cite journal
  9. ↑ Шаблон:Cite journal
  10. ↑ Шаблон:Cite journal
  11. ↑ Шаблон:Cite journal
  12. ↑ 12,0 12,1 Шаблон:Cite journal
  13. ↑ 13,0 13,1 Шаблон:Cite journal
  14. ↑ 14,0 14,1 Шаблон:Cite journal
  15. ↑ Шаблон:Cite web
  16. ↑ Шаблон:Cite journal
  17. ↑ Шаблон:Cite journal
  18. ↑ 18,0 18,1 18,2 Шаблон:Cite journal
  19. ↑ Шаблон:Cite conference
  20. ↑ Шаблон:Cite arXiv
  21. ↑ Шаблон:Cite journal
  22. ↑ Шаблон:Cite journal
  23. ↑ Шаблон:Citation
  24. ↑ Шаблон:Cite journal.
  25. ↑ Шаблон:Cite conference
  26. ↑ Шаблон:Cite journal
  27. ↑ Шаблон:Cite journal
  28. ↑ Шаблон:Cite news
  29. ↑ Шаблон:Cite news
  30. ↑ Шаблон:Cite web
  31. ↑ Шаблон:Cite journal
  32. ↑ Rai, Arun. "Explainable AI: From black box to glass box." Journal of the Academy of Marketing Science 48 (2020): 137-141.
  33. ↑ Шаблон:Cite book
  34. ↑ Шаблон:Cite journal
  35. ↑ 35,0 35,1 Шаблон:Cite ISO standard
  36. ↑ Шаблон:Cite journal
  37. ↑ Шаблон:Cite book
  38. ↑ Шаблон:Cite book
  39. ↑ Шаблон:Cite book
  40. ↑ 40,0 40,1 Шаблон:Cite book
  41. ↑ Шаблон:Cite book
  42. ↑ Шаблон:Cite book
  43. ↑ Шаблон:Cite journal
  44. ↑ Шаблон:Cite news
  45. ↑ Minsky, et al., "The Society of Intelligent Veillance" IEEE ISTAS2013, pages 13-17.
  46. ↑ Шаблон:Cite magazine
  47. ↑ Шаблон:Cite journal
  48. ↑ Шаблон:Cite arXiv
  49. ↑ Шаблон:Cite arXiv
  50. ↑ Шаблон:Cite book
  51. ↑ Шаблон:Cite journal
  52. ↑ Шаблон:Cite news
  53. ↑ Шаблон:Cite journal
  54. ↑ Шаблон:Cite journal
  55. ↑ Шаблон:Citation
  56. ↑ Шаблон:Cite journal
  57. ↑ Шаблон:Cite journal
  58. ↑ Шаблон:Cite journal
  59. ↑ Шаблон:Cite journal
  60. ↑ Шаблон:Cite journal
  61. ↑ Шаблон:Cite journal
  62. ↑ Шаблон:Cite journal
  63. ↑ Шаблон:Cite journal
  64. ↑ Bostrom, N., & Yudkowsky, E. (2014). The ethics of artificial intelligence. The Cambridge Handbook of Artificial Intelligence, 316-334.
  65. ↑ Шаблон:Cite web
  66. ↑ Шаблон:Cite news
  67. ↑ Шаблон:Cite journal
  68. ↑ Шаблон:Cite journal
  69. ↑ Шаблон:Cite book
  70. ↑ Шаблон:Cite journal
  71. ↑ Шаблон:Cite journal
  72. ↑ Шаблон:Cite web
  73. ↑ Шаблон:Cite news
  74. ↑ 74,0 74,1 74,2 Шаблон:Cite book
  75. ↑ 75,0 75,1 75,2 Шаблон:Cite journal
  76. ↑ Шаблон:Cite book
  77. ↑ Шаблон:Cite journal
  78. ↑ 78,0 78,1 Шаблон:Cite book
  79. ↑ Шаблон:Cite journal
  80. ↑ 80,0 80,1 Шаблон:Cite journal
  81. ↑ Шаблон:Cite journal
  82. ↑ Шаблон:Cite journal
  83. ↑ Шаблон:Cite journal
  84. ↑ Шаблон:Cite journal
  85. ↑ Шаблон:Cite arXiv
  86. ↑ Шаблон:Cite journal
Источник — http://wikihandbk.com/ruwiki/index.php?title=Английская_Википедия:Explainable_artificial_intelligence&oldid=14630334