Techniques for understanding how trained neural networks operate

Number: 11568212
Published: 2023-01-31
Filed: 2019-08-06
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Öztireli; Ahmet Cengiz et al.
CPC: G06N3/0464; G06F17/15; G06N3/047; G06N5/045; G06N3/045; G06F17/18
Verdict: Set aside generic ML interpretability research
Source: Google Patents · FreePatentsOnline

Abstract

In various embodiments, a relevance application quantifies how a trained neural network operates. In operation, the relevance application generates a set of input distributions based on a set of input points associated with the trained neural network. Each input distribution is characterized by a mean and a variance associated with a different neuron included in the trained neural network. The relevance application propagates the set of input distributions through a probabilistic neural network to generate at least a first output distribution. The probabilistic neural network is derived from at least a portion of the trained neural network. Based on the first output distribution, the relevance application computes a contribution of a first input point included in the set of input points to a difference between a first output point associated with a first output of the trained neural network and an estimated mean prediction associated with the first output.

Background

BACKGROUND Field of the Various Embodiments (1) Embodiments relate generally to artificial intelligence and neural networks and, more specifically, to techniques for understanding how trained neural networks operate. Description of the Related Art (2) A neural network is a collection of interconnected parameterizable units known as “neurons” that can be trained to recognize and interpret patterns in data based on multiple examples of those patterns. Trained neural networks are used to solve problems for a wide variety of tasks, such as image recognition, search engine filtering, playing board games, character animation, medical diagnosis, piloting drones, driving automobiles, etc. However, because a typical neural network includes a large number of parameters (hundreds, thousands, or even tens of thousands or more) that are automatically adjusted during training, trained neural networks are usually implemented without any proper understanding of what the trained neural network actually learned during training. Understanding what a trained neural network learned during training is important in many different contexts, such as when mistakes in the output of the trained neural network can have serious consequences (e.g., in automated vehicle implementations), when attempting to improve the accuracy of the trained neural network (e.g., when debugging errors), when attempting to implement portions of the trained neural network to solve a new or different problem, and when pruning

Claims

1. A computer-implemented method for quantifying how a trained neural network operates, the method comprising: generating a plurality of input distributions based on a first plurality of input points associated with the trained neural network, wherein each input distribution is represented as a mean and a variance associated with a different neuron included in the trained neural network; propagating the plurality of input distributions through a probabilistic neural network that is derived from at least a portion of the trained neural network to generate one or more output distributions; and based on a first output distribution included in the one or more output distributions, computing a contribution of a first input point included in the first plurality of input points to a difference between a first output point associated with a first output of the trained neural network and an estimated mean prediction associated with the first output. || 11. One or more non-transitory computer readable media including instructions that, when executed by one or more processors, cause the one or more processors to quantify how a trained neural network operates by performing the steps of: generating a plurality of input statistical sets based on a first plurality of input points associated with the trained neural network, wherein each input statistical set represents a distribution associated with a different neuron included in the trained neural network; propagating the plurality of input statistical sets through a probabilistic neural network that is derived from at least a portion of the trained neural network to generate one or more output statistical sets; and based on a first output statistical set included in the one or more output statistical sets, computing a contribution of a first input point included in the first plurality of input points to a difference between a first output point associated with a first output of the trained neural network and an estimated mean prediction associated with the first output. || 20. A system for quantifying how a trained neural network operates, the system comprising: one or more memories storing instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to: generate a plurality of input distributions based on a first plurality of input points associated with the trained neural network, wherein each input distribution is represented as a mean and a variance associated with a different neuron included in the trained neural network; propagate the plurality of input distributions through a probabilistic neural network that is derived from at least a portion of the trained neural network to generate one or more output distributions; and compute an estimated Shapley value based on a first output distribution included in the one or more output distributions, wherein the estimated Shapley value quantifies a contribution of a first input point included in the first plurality of input points to a difference between a first output point associated with a first output of the trained neural network and an estimated mean prediction associated with the first output.