AI-ASSISTED SOUND EFFECT EDITORIAL

Number: 20230136632
Published: 2023-05-04
Filed: 2021-11-01
Assignee: Lucasfilm Entertainment Company Ltd. LLC
Inventors: Tsingos; Nicolas et al.
CPC: G06F16/7834; G06F16/7867; G06F16/787; G06F18/214; G06N3/044; G06N3/045; G06N3/08; G06N3/084; G06V10/82; G06V20/40; G06V20/46
Verdict: Medium Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

AI-assisted sound-effect editorial/creative-ML audio tool.

Abstract

Some implementations of the disclosure relate to a method, comprising: obtaining, at a computing device, first video clip data including multiple sequential video frames, the multiple sequential video frames including at least a first video frame and a second video frame that occurs after the first video frame; inputting, at the computing device, the first video clip data into at least one trained model that automatically predicts, based on at least features of the first video frame and features of the second video frame, sound effect data corresponding to the second video frame; and determining, at the computing device, based on the sound effect data predicted for the second video frame, a first sound effect file corresponding to the second video frame.

Background

BRIEF SUMMARY OF THE DISCLOSURE

Implementations of the disclosure describe systems and methods that leverage machine learning to automatically determine sound effects for a given input video.

In one embodiment, a non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, cause a system to perform operations comprising: obtaining first video clip data including multiple sequential video frames, the multiple sequential video frames including at least a first video frame and a second video frame that occurs after the first video frame; inputting the first video clip data into at least one trained model that automatically predicts, based on at least features of the first video frame and features of the second video frame, sound effect data corresponding to the second video frame; and determining, based on the sound effect data predicted for the second video frame, a first sound effect file corresponding to the second video frame.

In some implementations, determining the first sound effect file corresponding to the second video frame comprises: mapping, using at least a sound effect datastore comprising multiple sound effect files that include the first sound effect file, the sound effect data predicted for the second video frame to the first sound effect file. In some implementations, the sound effect data predicted for the second video frame comprises a type or label; and mapping, using at least

Claims

1. A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, cause a system to perform operations comprising: obtaining first video clip data including multiple sequential video frames, the multiple sequential video frames including at least a first video frame and a second video frame that occurs after the first video frame; inputting the first video clip data into at least one trained model that automatically predicts, based on at least features of the first video frame and features of the second video frame, sound effect data corresponding to the second video frame; and determining, based on the sound effect data predicted for the second video frame, a first sound effect file corresponding to the second video frame. || 18. A system, comprising: one or more processors; and one or more non-transitory computer-readable mediums having executable instructions stored thereon that, when executed by the one or more processors, cause the system to perform operations comprising: obtaining first video clip data including multiple sequential video frames, the multiple sequential video frames including at least a first video frame and a second video frame that occurs after the first video frame; inputting the first video clip data into at least one trained model that automatically predicts, based on at least features of the first video frame and features of the second video frame, sound effect data corresponding to the second video frame; and determining, based on the sound effect data predicted for the second video frame, a first sound effect file corresponding to the second video frame. || 19. A method, comprising: obtaining, at a computing device, first video clip data including multiple sequential video frames, the multiple sequential video frames including at least a first video frame and a second video frame that occurs after the first video frame; inputting, at the computing device, the first video clip data into at least one trained model that automatically predicts, based on at least features of the first video frame and features of the second video frame, sound effect data corresponding to the second video frame; and determining, at the computing device, based on the sound effect data predicted for the second video frame, a first sound effect file corresponding to the second video frame.