- Number
- 11645579
- Published
- 2023-05-09
- Filed
- 2019-12-20
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Farré Guiu; Miquel Angel et al.
- CPC
- G06N20/00; G06V20/41; G06V20/46; G06V10/7784; G06N3/0464; G06N3/09; G06F18/2185
- Verdict
- Set aside content review tagging, business
- Source
- Google Patents · FreePatentsOnline
Abstract
Techniques for machine learning optimization are provided. A video comprising a plurality of segments is received, and a first segment of the plurality of segments is processed with a machine learning (ML) model to generate a plurality of tags, where each of the plurality of tags indicates presence of an element in the first segment. A respective accuracy value is determined for each respective tag of the plurality of tags, where the respective accuracy value is based at least in part on a maturity score for the ML model. The first segment is classified as accurate, based on determining that an aggregate accuracy of tags corresponding to the first segment exceeds a predefined threshold. Upon classifying the first segment as accurate, the first segment is bypassed during a review process.
Background
BACKGROUND (1) The present disclosure relates to machine learning, and more specifically, to using machine learning to optimize a tagging process and reduce manual review. (2) Machine learning (ML) algorithms can be used to identify different types of elements in media files at high levels of accuracy. However, to provide high levels of accuracy, the algorithms must be trained based on a training dataset. Preparing an accurate and complete training dataset to train the ML algorithms is difficult due to the amount of data needed, as well as the need to keep the dataset updated (e.g., cleaning the dataset, correcting errors in the dataset, adding more data, and the like). Additionally, existing systems cannot provide transparency to ensure accuracy, nor can they facilitate or expedite review. SUMMARY (3) According to one embodiment of the present disclosure, a method is provided. The method includes receiving a video comprising a plurality of segments, and processing a first segment of the plurality of segments with a machine learning (ML) model to generate a plurality of tags, wherein each of the plurality of tags indicates presence of an element in the first segment. The method further includes determining, for each respective tag of the plurality of tags, a respective accuracy value, wherein the respective accuracy value is based at least in part on a maturity score for the ML model. Additionally, the method includes classifying the first segment as accurate, based on determ
Claims
1. A method, comprising: receiving a video comprising a plurality of segments; processing a first segment of the plurality of segments with a machine learning (ML) model to generate a plurality of tags, wherein each of the plurality of tags indicates presence of an element in the first segment; determining, for each respective tag of the plurality of tags, a respective accuracy value, wherein the respective accuracy value is based at least in part on a maturity score generated for the ML model based on aggregating a plurality of model-specific scores for a plurality of versions of the ML model, wherein a respective weight assigned to each respective version of the plurality of versions is inversely proportional to a respective age of the respective version; classifying the first segment as accurate, based on determining that an aggregate accuracy of tags corresponding to the first segment exceeds a predefined threshold; and upon classifying the first segment as accurate, bypassing the first segment during a review process. ||
6. A method, comprising: receiving a video comprising a plurality of segments; processing a first segment of the plurality of segments with a machine learning (ML) model to generate a plurality of tags, wherein each of the plurality of tags indicates presence of an element in the first segment; determining, for each respective tag of the plurality of tags, a respective accuracy value, wherein the respective accuracy value is based at least in part on a maturity score generated for the ML model based on how many times the ML model has correctly identified the element of the respective tag as being present in video segments previously processed with the ML model, compared to how many times the element has actually been present in the video segments previously processed with the ML model; classifying the first segment as accurate, based on determining that an aggregate accuracy of tags corresponding to the first segment exceeds a predefined threshold; and upon classifying the first segment as accurate, bypassing the first segment during a review process. ||
8. A non-transitory computer-readable medium containing computer program code that, when executed by operation of one or more computer processors, performs an operation comprising: receiving a video comprising a plurality of segments; processing a first segment of the plurality of segments with a machine learning (ML) model to generate a plurality of tags, wherein each of the plurality of tags indicates presence of an element in the first segment; determining, for each respective tag of the plurality of tags, a respective accuracy value, wherein the respective accuracy value is based at least in part on a maturity score generated for the ML model based on aggregating a plurality of model-specific scores for a plurality of versions of the ML model, wherein a respective weight assigned to each respective version of the plurality of versions is inversely proportional to a respective age of the respective version; classifying the first segment as accurate, based on determining that an aggregate accuracy of tags corresponding to the first segment exceeds a predefined threshold; and upon classifying the first segment as accurate, bypassing the first segment during a review process. ||
13. A system, comprising: one or more computer processors; and a memory containing a program which when executed by the one or more computer processors performs an operation, the operation comprising: receiving a video comprising a plurality of segments; processing a first segment of the plurality of segments with a machine learning (ML) model to generate a plurality of tags, wherein each of the plurality of tags indicates presence of an element in the first segment; determining, for each respective tag of the plurality of tags, a respective accuracy value, wherein the respective accuracy value is based at least in part on a maturity score generated for the ML model based on aggregating a plurality of model-specific scores for a plurality of versions of the ML model, wherein a respective weight assigned to each respective version of the plurality of versions is inversely proportional to a respective age of the respective version; classifying the first segment as accurate, based on determining that an aggregate accuracy of tags corresponding to the first segment exceeds a predefined threshold; and upon classifying the first segment as accurate, bypassing the first segment during a review process. ||
18. A non-transitory computer-readable medium containing computer program code that, when executed by operation of one or more computer processors, performs an operation comprising: receiving a video comprising a plurality of segments; processing a first segment of the plurality of segments with a machine learning (ML) model to generate a plurality of tags, wherein each of the plurality of tags indicates presence of an element in the first segment; determining, for each respective tag of the plurality of tags, a respective accuracy value, wherein the respective accuracy value is based at least in part on a maturity score generated for the ML model based on how many times the ML model has correctly identified the element of the respective tag as being present in video segments previously processed with the ML model, compared to how many times the element has actually been present in the video segments previously processed with the ML model; classifying the first segment as accurate, based on determining that an aggregate accuracy of tags corresponding to the first segment exceeds a predefined threshold; and upon classifying the first segment as accurate, bypassing the first segment during a review process. ||
20. A system, comprising: one or more computer processors; and a memory containing a program which when executed by the one or more computer processors performs an operation, the operation comprising: receiving a video comprising a plurality of segments; processing a first segment of the plurality of segments with a machine learning (ML) model to generate a plurality of tags, wherein each of the plurality of tags indicates presence of an element in the first segment; determining, for each respective tag of the plurality of tags, a respective accuracy value, wherein the respective accuracy value is based at least in part on a maturity score generated for the ML model based on how many times the ML model has correctly identified the element of the respective tag as being present in video segments previously processed with the ML model, compared to how many times the element has actually been present in the video segments previously processed with the ML model; classifying the first segment as accurate, based on determining that an aggregate accuracy of tags corresponding to the first segment exceeds a predefined threshold; and upon classifying the first segment as accurate, bypassing the first segment during a review process.