According to one implementation, a quality control (QC) system for annotated content includes a computing platform having a hardware processor and a system memory storing an annotation culling software code. The hardware processor executes the annotation culling software code to receive multiple content sets annotated by an automated content classification engine, and obtain evaluations of the annotations applied by the automated content classification engine to the content sets. The hardware processor further executes the annotation culling software code to identify a sample size of the content sets for automated QC analysis of the annotations applied by the automated content classification engine, and cull the annotations applied by the automated content classification engine based on the evaluations when the number of annotated content sets equals the identified sample size.
BACKGROUND (1) Due to its popularity as a content medium, ever more video is being produced and made available to users. As a result, the efficiency with which video content can be annotated and managed has become increasingly important to the producers of that video content. For example, annotation of video is an important part of the production process for television (TV) programming and movies. (2) Annotation of video has traditionally been performed manually by human annotators or taggers. However, in a typical video production environment, there may be such a large number of videos to be annotated that manual tagging becomes impracticable. In response, automated solutions for annotating content are being developed. While offering efficiency advantages over traditional manual tagging, automated systems are more prone to error than human taggers. Consequently, there is a need in the art for an automated solution for performing quality control (QC) of the tags applied to content by automated content annotation systems. SUMMARY (3) There are provided quality control (QC) systems for annotated content and QC methods for use by those systems, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims.
1. A quality control (QC) system for annotated content, the QC system comprising: a computing platform including a hardware processor and a system memory; an annotation culling software code stored in the system memory; the hardware processor configured to execute the annotation culling software code to: receive a plurality of content sets annotated by an automated content classification engine; obtain evaluations of annotations applied by the automated content classification engine to the plurality of content sets; identify a sample size of the plurality of content sets for automated QC analysis of the annotations applied by the automated content classification engine; determine a respective maturity index for each class of the annotations applied by the automated content classification engine based on the evaluations; and when the plurality of annotated content sets equals the identified sample size, cull the annotations applied by the automated content classification engine based on the respective maturity index for each class of the annotations. ||
10. A quality control (QC) system for annotated content, the QC system comprising: a computing platform including a hardware processor and a system memory; an annotation culling software code stored in the system memory; the hardware processor configured to execute the annotation culling software code to: receive a plurality of content sets annotated by an automated content classification engine; obtain evaluations of annotations applied by the automated content classification engine to the plurality of content sets; determine a sample size of the plurality of content sets for automated QC analysis of the annotations based in part on content features described by the annotations applied by the automated content classification engine; after determining the sample size, identify as non-validated annotation classes, classes of annotations applied by the automated content classification engine that are neither verified nor changed based on the evaluations; and cull the annotations applied by the automated content classification engine by discarding the non-validated annotation classes. ||
11. A quality control (QC) system for annotated content, the QC system comprising: a computing platform including a hardware processor and a system memory; an annotation culling software code stored in the system memory; the hardware processor configured to execute the annotation culling software code to: receive a plurality of content sets annotated by an automated content classification engine; obtain evaluations of annotations applied by the automated content classification engine to the plurality of content sets; identify a predetermined sample size of the plurality of content sets for automated QC analysis of the annotations; after identifying the predetermined sample size, identify as non-validated annotation classes, classes of annotations applied by the automated content classification engine that are neither verified nor changed based on the evaluations; and cull the annotations applied by the automated content classification engine by discarding the non-validated annotation classes. ||
15. A quality control (QC) system for annotated content, the QC system comprising: a computing platform including a hardware processor and a system memory; an annotation culling software code stored in the system memory; the hardware processor configured to execute the annotation culling software code to: receive a plurality of content sets annotated by an automated content classification engine; obtain evaluations of annotations applied by the automated content classification engine to the plurality of content sets; determine a respective maturity index for each class of the annotations applied by the automated content classification engine based on the evaluations; classify each class as mature or immature, wherein each class having the respective maturity index equal to or exceeding a predetermined threshold is classified as mature classes and each class having the respective maturity index less than the predetermined threshold is classified as immature; and cull the annotations applied by the automated content classification engine by retaining annotations belonging to classes classified as mature.