Quality Control Systems and Methods for Annotated Content

Number: 20210019576
Published: 2021-01-21
Filed: 2019-07-15
Assignee: Disney Enterprises, Inc.
Inventors: Farre Guiu; Miquel Angel, Petrillo; Matthew C., Martin; Marc Junyent, Accardo; Anthony M., Swerdlow; Avner, Alfaro Vendrell; Monica
CPC: G06V20/46; G06F18/24; G06V40/103; G06V10/993; G06V10/7784; G06V20/41; G06F18/217; G06V10/7788
Verdict: Set aside content annotation QC, business
Source: Google Patents · FreePatentsOnline

Abstract

According to one implementation, a quality control (QC) system for annotated content includes a computing platform having a hardware processor and a system memory storing an annotation culling software code. The hardware processor executes the annotation culling software code to receive multiple content sets annotated by an automated content classification engine, and obtain evaluations of the annotations applied by the automated content classification engine to the content sets. The hardware processor further executes the annotation culling software code to identify a sample size of the content sets for automated QC analysis of the annotations applied by the automated content classification engine, and cull the annotations applied by the automated content classification engine based on the evaluations when the number of annotated content sets equals the identified sample size.

Background

BACKGROUND

Due to its popularity as a content medium, ever more video is being produced and made available to users. As a result, the efficiency with which video content can be annotated and managed has become increasingly important to the producers of that video content. For example, annotation of video is an important part of the production process for television (TV) programming and movies.

Annotation of video has traditionally been performed manually by human annotators or taggers. However, in a typical video production environment, there may be such a large number of videos to be annotated that manual tagging becomes impracticable. In response, automated solutions for annotating content are being developed. While offering efficiency advantages over traditional manual tagging, automated systems are more prone to error than human taggers. Consequently, there is a need in the art for an automated solution for performing quality control (QC) of the tags applied to content by automated content annotation systems. SUMMARY

There are provided quality control (QC) systems for annotated content and QC methods for use by those systems, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims.

Claims

1. A quality control (QC) system for annotated content, the QC system comprising: a computing platform including a hardware processor and a system memory; an annotation culling software code stored in the system memory; the hardware processor configured to execute the annotation culling software code to: receive a plurality of content sets annotated by an automated content classification engine; obtain evaluations of annotations applied by the automated content classification engine to the plurality of content sets; identify a sample size of the plurality of content sets for automated QC analysis of the annotations applied by the automated content classification engine; and cull the annotations applied by the automated content classification engine based on the evaluations when the plurality of annotated content sets equals the identified sample size. || 10. A quality control (QC) system for annotated content, the QC system comprising: a computing platform including a hardware processor and a system memory; an annotation culling software code stored in the system memory; the hardware processor configured to execute the annotation culling software code to: receive a plurality of content sets annotated by an automated content classification engine; obtain evaluations of annotations applied by the automated content classification engine to the plurality of content sets; identify as non-validated annotation classes, classes of annotations applied by the automated content classification engine that are neither verified nor changed based on the evaluations; and cull the annotations applied by the automated content classification engine by discarding the non-validated annotation classes. || 16. A quality control (QC) system for annotated content, the QC system comprising: a computing platfoiin including a hardware processor and a system memory; an annotation culling software code stored in the system memory; the hardware processor configured to execute the annotation culling software code to: receive a plurality of content sets annotated by an automated content classification engine; obtain evaluations of annotations applied by the automated content classification engine to the plurality of content sets; determine a maturity index for each class of the annotations applied by the automated content classification engine based on the evaluations; classify each class as a mature class or an immature class wherein classes having respective maturity indexes equal to or exceeding a predetermined threshold are classified as mature classes and classes having respective maturity indexes less than the predetermined threshold are classified as immature classes,; and cull the annotations applied by the automated content classification engine by retaining annotations belonging to mature classes.