- Number
- 11741129
- Published
- 2023-08-29
- Filed
- 2021-08-06
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Farre Guiu; Miquel Angel et al.
- CPC
- G06F16/24573; G06F16/285; G06F16/75; G06N20/00
- Verdict
- Set aside content annotation taxonomy, business
- Source
- Google Patents · FreePatentsOnline
Abstract
According to one implementation, a system includes a computing platform having processing hardware, a system memory storing a software code; and a machine learning model based classifier. The processing hardware is configured to execute the software code to receive tagging quality assurance (QA) data including multiple terms applied as tags and corrections to those tags, to identify, using the tagging QA data, a first problematic term, and to classify, using the machine learning model based classifier, the first problematic term as one of confusing or flawed. The processing hardware is further configured to execute the software code to obtain, when the first problematic term is classified as confusing, a comparative sample for clarifying use of the first problematic term, and to obtain, when the first problematic term is classified as flawed, modification data for editing a predetermined annotation taxonomy including the first problematic term.
Background
BACKGROUND (1) Due to its popularity as a content medium, ever more video is being produced and made available to users. As a result, the efficiency with which video content can be annotated. i.e., “tagged,” and managed has become increasingly important to the producers, owners, and distributors of that video content. For example, annotation of video is an important part of the production process for television (TV) programming content and movies. (2) Tagging of video has traditionally been performed manually by human taggers, based on a predetermined set, or “taxonomy,” of terms that may be applied as tags, while quality assurance (QA) for the tagging process is typically performed by human QA reviewers. However, in a typical video production environment, there may be such a large number of videos to be annotated that manual tagging and review become impracticable. In response, various automated systems for performing content tagging and QA review have been developed or are in development. While offering efficiency advantages over traditional manual techniques, the performance of automated systems, like the performance of human taggers, depends to a significant extent on the relevance and specificity of the typically closed set of terms included in the annotation taxonomy. Consequently, there is a need in the art for systems and methods for enhancing the performance of automated and human taggers alike through the performance-based evolution of content annotation taxonomies.
Claims
1. A computing platform comprising: a processing hardware; a system memory storing a software code; and a machine learning model based classifier; the processing hardware configured to execute the software code to: receive tagging quality assurance (QA) data including a plurality of terms applied as tags by a trained machine learning model based automated tagging system; identify, using the tagging QA data, a problematic term of the plurality of terms; classify, using the machine learning model based classifier, the problematic term as one of a re-trainable term or a flawed term; when the problematic term is classified as the re-trainable term, adjust, using one or more parameters, the trained machine learning model based automated tagging system; and when the problematic term is classified as the flawed term, edit, using modification data, an annotation taxonomy. ||
6. A computing platform comprising: a processing hardware; a system memory storing a software code; and a machine learning model based classifier; the processing hardware configured to execute the software code to: receive tagging quality assurance (QA) data including a plurality of terms applied as tags; identify, using the tagging QA data, a problematic term of the plurality of terms; classify, using the machine learning model based classifier, the problematic term as one of a confusing term or a flawed term; obtain, when the problematic term is classified as the confusing term, a comparative sample for clarifying use of the problematic term as a tag; obtain, when the problematic term is classified as the flawed term, a modification data for editing an annotation taxonomy including the problematic term; receive another tagging QA data including another plurality of terms applied as tags by a trained machine learning model based automated tagging system, and a plurality of corrections to the tags; identify, using the another tagging QA data, an automated problematic term of the another plurality of terms; classify, using the another tagging QA data, the automated problematic term as one of a re-trainable term or another flawed term; obtain, when the automated problematic term is classified as the re-trainable term, one or more parameters for adjusting the trained machine learning model based automated tagging system; and obtain, when the automated problematic term is classified as the another flawed term, another modification data for editing the annotation taxonomy. ||
13. A method for use by a computing platform, the method comprising: receiving tagging quality assurance (QA) data including a plurality of terms applied as tags by a trained machine learning model based automated tagging system; identifying, using the tagging QA data, a problematic term of the plurality of terms; classifying, using the tagging QA data, the problematic term as one of a re-trainable term or a flawed term; when the problematic term is classified as the re-trainable term, adjusting, using one or more parameters, the trained machine learning model based automated tagging system; and when the problematic term is classified as the flawed term, editing, using modification data, an annotation taxonomy.