- Number
- 12711449
- Published
- 2026-08-18
- Filed
- 2021-02-03
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Farre Guiu; Miquel Angel, Porta Valles; Marcel, Martin; Marc Junyent, Badia Pujol; Jordi, Ovanessian; Melina
- CPC
- G06Q10/06395; G06F40/169
- Verdict
- Set aside search/metadata
- In edition
- 2026-W36
- Source
- Google Patents · FreePatentsOnline
The keeper's note
According to one implementation, a tagging performance evaluation system includes a computing platform having a hardware processor and a memory storing a software code.
Abstract
According to one implementation, a tagging performance evaluation system includes a computing platform having a hardware processor and a memory storing a software code. The hardware processor is configured to execute the software code to receive annotation data identifying content, annotation tags applied to the content, and one or more correction(s) to the annotation tags, to perform, using the annotation data, at least one of an evaluation of a tagging process resulting in application of the annotation tags to the content or an assessment of a correction process resulting in the correction(s), and to identify, based on the at least one of the evaluation or the assessment, one or more parameters for improving at least one of the tagging process or the correction process. At least one of the evaluation or the assessment is performed using a machine learning model of the tagging performance evaluation system.
Background
BACKGROUND (1) Due to its popularity as a content medium, ever more video is being produced and made available to users. As a result, the efficiency with which video content can be annotated, i.e., “tagged,” and managed has become increasingly important to the producers of that video content. For example, annotation of video is an important part of the production process for television (TV) programming content and movies. (2) Tagging of video has traditionally been performed manually by human taggers, while quality assurance (QA) for the tagging process is typically performed by human QA reviewers. However, in a typical video production environment, there may be such a large number of videos to be annotated that manual tagging and review become impracticable. In response, various automated systems for performing content tagging and QA review have been developed or are in development. While offering efficiency advantages over traditional manual techniques, automated systems, like human taggers and QA reviewers, are prone to error. Consequently, there is a need in the art for automated systems and methods for evaluating and improving the performance of the tagging and QA review processes performed as part of content annotation.
Claims
1. A system comprising: a computing platform including a hardware processor and a system memory storing a software code, an annotation evaluation machine learning model, and a correction assessment machine learning model; the hardware processor configured to execute the software code to: receive annotation data identifying a content and a plurality of annotation tags applied to the content in a tagging process performed by a tagging entity; receive one or more corrections to the plurality of annotation tags, the one or more corrections having been made in a correction process performed by a quality assurance (QA) entity, wherein at least one of (i) the tagging entity is a trained tagging machine learning model communicatively coupled to the system via a communication network, or (ii) the QA entity is a trained tag review and correction machine learning model communicatively coupled to the system via the communication network; perform, using the annotation evaluation machine learning model and the annotation data, an automated evaluation of the tagging process based on the plurality of annotation tags and the one or more corrections to the plurality of annotation tags; perform, using the correction assessment machine learning model, the automated evaluation of the tagging process and the annotation data based at least in part on how many corrections are included among the one or more corrections to the plurality of annotation tags; identify, based on at least one of the automated evaluation of the tagging process or the automated assessment of the correction process, one or more parameters for improving at least one of the tagging process or the correction process; and modify, based on the one or more parameters, one or more of stored weights or stored priorities of at least one of the trained tagging machine learning model communicatively coupled to the system or the trained tag review and correction machine learning model communicatively coupled to the system, wherein the one or more of the stored weights or stored priorities are updated using the one or more parameters as training feedback, the one or more parameters including at least one of a tagging performance history of the tagging entity or a correction history of the QA entity, to provide at least one of an improved tagging machine learning model or an improved tag review and correction machine learning model. ||
8. A method for use by a system including a computing platform having a hardware processor and a system memory storing a software code, an annotation evaluation machine learning model and a correction assessment machine learning model, the method comprising: receiving, by the software code executed by the hardware processor, annotation data identifying a content and a plurality of annotation tags applied to the content in a tagging process performed by a tagging entity; receiving, by the software code executed by the hardware processor, one or more corrections to the plurality of annotation tags, the one or more corrections having been made in a correction process performed by a quality assurance (QA) entity, wherein at least one of (i) the tagging entity is a trained tagging machine learning model communicatively coupled to the system via a communication network, or (ii) the QA entity is a trained tag review and correction machine learning model communicatively coupled to the system via the communication network; performing, by the software code executed by the hardware processor and using the annotation evaluation machine learning model and the annotation data, an automated evaluation of the tagging process based on the plurality of annotation tags and the one or more corrections to the plurality of annotation tags; performing, by the software code executed by the hardware processor and using the correction assessment machine learning model, the automated evaluation of the tagging process and the annotation data based at least in part on how many corrections are included among the one or more corrections to the plurality of annotation tags; identifying, by the software code executed by the hardware processor and based on at least one of the automated evaluation of the tagging process or the automated assessment of the correction process, one or more parameters for improving at least one of the tagging process or the correction process; and modifying, by the software code executed by the hardware processor and based on the one or more parameters, one or more of stored weights or stored priorities of at least one of the trained tagging machine learning model communicatively coupled to the system or the trained tag review and correction machine learning model communicatively coupled to the system, wherein the one or more of the stored weights or stored priorities are updated using the one or more parameters as training feedback, the one or more parameters including at least one of a tagging performance history of the tagging entity or a correction history of the QA entity, to provide at least one of an improved tagging machine learning model or an improved tag review and correction machine learning model.