Semiautomatic machine learning model improvement and benchmarking

Number: 10489722
Published: 2019-11-26
Filed: 2017-07-27
Assignee: Disney Enterprises, Inc.
Inventors: Farré Guiu; Miquel Angel et al.
CPC: G06F18/2155; G06N20/00; G06F18/217; G06V10/7784; G06F18/214; G06V10/776; G06F18/2178; G06V20/41; G06V10/774; G06N5/022
Verdict: Set aside semiautomatic ML model improvement/benchmarking grant dup
Source: Google Patents · FreePatentsOnline

Abstract

Systems, methods, and articles of manufacture to perform an operation comprising processing, by a machine learning (ML) algorithm and a ML model, a plurality of images in a first dataset, wherein the ML model was generated based on a plurality of images in a training dataset, receiving user input reviewing a respective set of tags applied to each image in the first data set as a result of the processing, identifying, based on a first confusion matrix generated based on the user input and the sets of tags applied to the images in the first data set, a first labeling error in the training dataset, determining a type of the first labeling error based on a second confusion matrix, and modifying the training dataset based on the determined type of the first labeling error.

Background

BACKGROUNDField of the Invention(1) Embodiments disclosed herein relate to machine learning. More specifically, embodiments disclosed herein relate to semiautomatic machine learning model improvement and benchmarking.Description of the Related Art(2) Many vendors offer machine learning (ML) algorithms to identify different types of elements in media files at high levels of accuracy. However, to provide high levels of accuracy, the algorithms must be trained based on a training dataset. Nevertheless, preparing an accurate training dataset to train the ML algorithms is difficult due to the need of keeping the dataset updated (e.g., cleaning the dataset, correcting errors in the dataset, adding more data, and the like). Furthermore, updating the dataset does not provide a description of the improvement or saturation of the ML algorithms after adding new edge cases.SUMMARY(3) In one embodiment, a method comprises processing, by a machine learning (ML) algorithm and a ML model, a plurality of images in a first dataset, wherein the ML model was generated based on a plurality of images in a training dataset, receiving user input reviewing a respective set of tags applied to each image in the first data set as a result of the processing, identifying, based on a first confusion matrix generated based on the user input and the sets of tags applied to the images in the first data set, a first labeling error in the training dataset, determining a type of the first labeling error based on

Claims

1. A method, comprising: processing, by a machine learning (ML) algorithm and a ML model, a plurality of images in a first dataset, wherein the ML model was generated based on a plurality of images in a training dataset; receiving user input reviewing a respective set of tags applied to each image in the first data set as a result of the processing; identifying, based on a first confusion matrix generated based on the user input and the sets of tags applied to the images in the first data set, a first labeling error in the training dataset; determining a type of the first labeling error based on a second confusion matrix; and modifying the training dataset based on the determined type of the first labeling error. 2. The method of claim 1, wherein the first confusion matrix comprises a row-normalized confusion matrix, and wherein identifying the first labeling error in the training dataset comprises at least one of: identifying at least two instances of a maximum value in a first row of the first confusion matrix, wherein the first row of the first confusion matrix corresponds to a first tag of at least one of the sets of tags; or upon determining that one instance of a maximum value occurs in a first row of the first confusion matrix, determining that a position of the one instance of the maximum value is not on a diagonal of the first confusion matrix. 3. The method of claim 1, wherein the second confusion matrix comprises a column-normalized matrix, and wherein determining the type of the first labeling error comprises: computing an absolute value of a difference of two values in a first column of the second confusion matrix, wherein when the computed absolute value does not exceed a threshold, the type of the first labeling error is a first type, and wherein when the computed absolute value exceeds the threshold, the type of the first labeling error is a second type, different than the first type. 4. The method of claim 3, wherein when the type of the first labeling error is determined to be the first type, modifying the training dataset comprises: identifying a plurality of images determined to be mislabeled with a first tag of at least one of the sets of tags by the ML algorithm and the ML model; and adding the identified plurality of images determined to be mislabeled with the first tag to the training dataset. 5. The method of claim 3, wherein when the type of the first labeling error is determined to be the second type, modifying the training dataset comprises: determining a second tag associated with the first column of the second confusion matrix; and relabeling the first tag as the second tag in the training dataset. 6. The method of claim 3, further comprising: rebuilding the ML model based on the modified training dataset; computing a score for a user providing the user input; determining a message associated with the computed score; and outputting an indication of the associated message to the user. 7. The method of claim 1, wherein processing the plurality of images in the first dataset by the ML algorithm and the ML model comprises determining whether at least one object is depicted in a respective image, and wherein at least one image of the plurality of images in the first dataset and the training dataset comprise frames of video. 8. A computer program product, comprising: a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by a processor to perform an operation comprising: processing, by a machine learning (ML) algorithm and a ML model, a plurality of images in a first dataset, wherein the ML model was generated based on a plurality of images in a training dataset; receiving user input reviewing a respective set of tags applied to each image in the first data set as a result of the processing; identifying, based on a first confusion matrix generated based on the user input and the sets of tags applied to the images in the first data set, a first labeling error in the training dataset; determining a type of the first labeling error based on a second confusion matrix; and modifying the training dataset based on the determined type of the first labeling error. 9. The computer program product of claim 8, wherein the first confusion matrix comprises a row-normalized confusion matrix, and wherein identifying the first labeling error in the training dataset comprises at least one of: identifying at least two instances of a maximum value in a first row of the first confusion matrix, wherein the first row of the first confusion matrix corresponds to a first tag of at least one of the sets of tags; upon determining that one instance of a maximum value occurs in a first row of the first confusion matrix, determining that a position of the one instance of the maximum value is not on a diagonal of the first confusion matrix. 10. The computer program product of claim 8, wherein the second confusion matrix comprises a column-normalized matrix, and wherein determining the type of the first labeling error comprises: computing an absolute value of a difference of two values in a first column of the second confusion matrix, wherein when the computed absolute value does not exceed a threshold, the type of the first labeling error is a first type, and wherein when the computed absolute value exceeds the threshold, the type of the first labeling error is a second type, different than the first type. 11. The computer program product of claim 10, wherein when the type of the first labeling error is determined to be the first type, modifying the training dataset comprises: identifying a plurality of images determined to be mislabeled with a first tag of at least one of the sets of tags by the ML algorithm and the ML model; and adding the identified plurality of images determined to be mislabeled with the first tag to the training dataset. 12. The computer program product of claim 10, wherein when the type of the first labeling error is determined to be the second type, modifying the training dataset comprises: determining a second tag associated with the first column of the second confusion matrix; and relabeling the first tag as the second tag in the training dataset. 13. The computer program product of claim 10, the operation further comprising: rebuilding the ML model based on the modified training dataset; computing a score for a user providing the user input; determining a message associated with the computed score; and outputting an indication of the associated message to the user. 14. The computer program product of claim 8, wherein processing the plurality of images in the first dataset by the ML algorithm and the ML model comprises determining whether at least one object is depicted in a respective image, and wherein at least one image of the plurality of images in the first dataset and the training dataset comprise frames of video. 15. A system, comprising: one or more computer processors; and a memory containing a program which when executed by the processors performs an operation comprising: processing, by a machine learning (ML) algorithm and a ML model, a plurality of images in a first dataset, wherein the ML model was generated based on a plurality of images in a training dataset; receiving user input reviewing a respective set of tags applied to each image in the first data set as a result of the processing; identifying, based on a first confusion matrix generated based on the user input and the sets of tags applied to the images in the first data set, a first labeling error in the training dataset; determining a type of the first labeling error based on a second confusion matrix; and modifying the training dataset based on the determined type of the first labeling error. 16. The system of claim 15, wherein the first confusion matrix comprises a row-normalized confusion matrix, and wherein identifying the first labeling error in the training dataset comprises at least one of: identifying at least two instances of a maximum value in a first row of the first confusion matrix, wherein the first row of the first confusion matrix corresponds to a first tag of at least one of the sets of tags; upon determining that one instance of a maximum value occurs in a first row of the first confusion matrix, determining that a position of the one instance of the maximum value is not on a diagonal of the first confusion matrix. 17. The system of claim 15, wherein the second confusion matrix comprises a column-normalized matrix, and wherein determining the type of the first labeling error comprises: computing an absolute value of a difference of two values in a first column of the second confusion matrix, wherein when the computed absolute value does not exceed a threshold, the type of the first labeling error is a first type, and wherein when the computed absolute value exceeds the threshold, the type of the first labeling error is a second type, different than the first type. 18. The system of claim 17, wherein when the type of the first labeling error is determined to be the first type, modifying the training dataset comprises: identifying a plurality of images determined to be mislabeled with a first tag of at least one of the sets of tags by the ML algorithm and the ML model; and adding the identified plurality of images determined to be mislabeled with the first tag to the training dataset. 19. The system of claim 17, wherein when the type of the first labeling error is determined to be the second type, modifying the training dataset comprises: determining a second tag associated with the first column of the second confusion matrix; and relabeling the first tag as the second tag in the training dataset. 20. The system of claim 17, wherein processing the plurality of images in the first dataset by the ML algorithm and the ML model comprises determining whether at least one object is depicted in a respective image, and wherein at least one image of the plurality of images in the first dataset and the training dataset comprise frames of video, the operation further comprising: rebuilding the ML model based on the modified training dataset; computing a score for a user providing the user input; determining a message associated with the computed score; and outputting an indication of the associated message to the user.