MULTILINGUAL MULTIMEDIA TRANSLATION WITH SYNCHRONIZED AUDIO AND BODY GESTURES

Number: 20260281513
Published: 2026-09-17
Filed: 2025-03-11
Assignee: Disney Enterprises, Inc.
Inventors: NGUYEN; Duy
CPC: H04N21/8146; G06V40/20; H04N21/8106
Verdict: Set aside dropped in weekly review
In edition: 2026-W38
Source: Google Patents · FreePatentsOnline

The keeper's note

Multilingual multimedia translation by generating a synchronized multimedia asset based on a multimedia asset that is translated into a different language.

Abstract

Multilingual multimedia translation by generating a synchronized multimedia asset based on a multimedia asset that is translated into a different language. The multilingual multimedia translation segments the multimedia asset into an audio segment and a video segment. The multilingual translation uses the audio segment to generate a translated audio segment. The multilingual translation may use the video segment identify body gestures of individuals in the video segment, generate translated body gestures, and generate translated lip movements. Using the translated audio segment and any translated lip movements or translated body gestures, the multilingual translation generates a synchronized multimedia asset.

Background

BACKGROUND

In video content, language barriers pose a significant challenge to global communication and understanding. Traditional video translation methods primarily focus on verbal content, often neglecting the crucial aspects of non-verbal communication such as body language and gestures.

Current methods do not capture the nuances and cultural contexts embedded in body language, which are integral to effective communication. This can lead to misunderstandings or misinterpretations of the original content. Existing systems often provide translations in the form of subtitles or dubbed audio that are not synchronized with the speakers' body gestures and facial expressions, leading to a disjointed and less engaging viewer experience.

Content creators face difficulties in making their video content accessible and relatable to a global audience, as they must consider not only language barriers but also cultural differences in non-verbal communication.

The problem of multilingual video translation is typically solved using separate technologies for audio and subtitle translation. For example, subtitles and dubbed audio tracks are used to translate spoken language, but these solutions often lack synchronization with body gestures and facial expressions, leading to a disjointed viewer experience.

Claims

1. A method, comprising: accessing a multimedia asset depicting an individual speaking in a first language of a plurality of languages; extracting an audio segment of the individual speaking in the multimedia asset; identifying a first gesture depicted in the multimedia asset during a first portion of the audio segment; determining a first temporal alignment between an utterance made in the first portion of the audio segment and the first gesture; and generating a synchronized multimedia asset based on at least one of: modifying a temporal location of the first gesture in the multimedia asset or modifying a timing of a translated audio segment to set a second temporal alignment between the first gesture and a translation of the utterance in the translated audio segment. || 10. A method comprising: accessing a multimedia asset depicting an individual from a first culture; identifying a first gesture made by the individual in the multimedia asset; determining a second gesture in a second culture having a meaning that matches the meaning of the first gesture in the first culture, wherein the first and second gestures are different; and adding the second gesture to the multimedia asset to modify the multimedia asset to reflect the second culture. || 16. A system comprising: one or more computer processors; and a memory containing a program which when executed by the computer processors performs an operation comprising: accessing a multimedia asset depicting an individual speaking in a first language of a plurality of languages; extracting an audio segment of the individual speaking in the multimedia asset; identifying a first gesture depicted in the multimedia asset during a first portion of the audio segment; determining a first temporal alignment between an utterance made in the first portion of the audio segment and the first gesture; and generating a synchronized multimedia asset based on at least one of: modifying a temporal location of the first gesture in the multimedia asset or modifying a timing of a translated audio segment to set a second temporal alignment between the first gesture and a translation of the utterance in the translated audio segment.