Tagging Performance Evaluation and Improvement

Number: 20220245554
Published: 2022-08-04
Filed: 2021-02-03
Assignee: Disney Enterprises, Inc.
Inventors: Farre Guiu; Miquel Angel, Porta Valles; Marcel, Martin; Marc Junyent, Badia Pujol; Jordi, Ovanessian; Melina
CPC: G06F40/169; G06Q10/06395
Verdict: Set aside content tagging QC, business
Source: Google Patents · FreePatentsOnline

Abstract

According to one implementation, a tagging performance evaluation system includes a computing platform having a hardware processor and a memory storing a software code. The hardware processor is configured to execute the software code to receive annotation data identifying content, annotation tags applied to the content, and one or more correction(s) to the annotation tags, to perform, using the annotation data, at least one of an evaluation of a tagging process resulting in application of the annotation tags to the content or an assessment of a correction process resulting in the correction(s), and to identify, based on the at least one of the evaluation or the assessment, one or more parameters for improving at least one of the tagging process or the correction process. At least one of the evaluation or the assessment is performed using a machine learning model of the tagging performance evaluation system.

Background

BACKGROUND

Due to its popularity as a content medium, ever more video is being produced and made available to users. As a result, the efficiency with which video content can be annotated, i.e., “tagged,” and managed has become increasingly important to the producers of that video content. For example, annotation of video is an important part of the production process for television (TV) programming content and movies.

Tagging of video has traditionally been performed manually by human taggers, while quality assurance (QA) for the tagging process is typically performed by human QA reviewers. However, in a typical video production environment, there may be such a large number of videos to be annotated that manual tagging and review become impracticable. In response, various automated systems for performing content tagging and QA review have been developed or are in development. While offering efficiency advantages over traditional manual techniques, automated systems, like human taggers and QA reviewers, are prone to error. Consequently, there is a need in the art for automated systems and methods for evaluating and improving the performance of the tagging and QA review processes performed as part of content annotation.

Claims

1. A tagging performance evaluation system comprising: a computing platform including a hardware processor and a system memory storing a software code; the hardware processor configured to execute the software code to: receive annotation data, the annotation data identifying a content, a plurality of annotation tags applied to the content, and one or more corrections to the plurality of annotation tags; perform, using the annotation data, at least one of an evaluation of a tagging process resulting in application of the plurality of annotation tags to the content or an assessment of a correction process resulting in the one or more corrections; and identify, based on the at least one of the evaluation or the assessment, one or more parameters for improving at least one of the tagging process or the correction process; wherein at least one of the evaluation or the assessment is performed using a machine learning model of the tagging performance evaluation system. || 11. A method for use by a tagging performance evaluation system including a computing platform having a hardware processor and a system memory storing a software code, the method comprising: receiving, by the software code executed by the hardware processor, annotation data, the annotation data identifying a content, a plurality of annotation tags applied to the content, and one or more corrections to the plurality of annotation tags; performing, by the software code executed by the hardware processor and using the annotation data, at least one of an evaluation of a tagging process resulting in application of the plurality of annotation tags to the content or an assessment of a correction process resulting in the one or more corrections; and identifying, by the software code executed by the hardware processor and based on the at least one of the evaluation or the assessment, one or more parameters for improving at least one of the tagging process or the correction process; wherein at least one of the evaluation or the assessment is performed using a machine learning model of the tagging performance evaluation system.