ASPECT RATIO CONVERSION WITH MACHINE LEARNING

Number: 20200304755
Published: 2020-09-24
Filed: 2019-03-21
Assignee: Disney Enterprises, Inc.
Inventors: NARAYAN; Nimesh C., YAACOB; Yazmaliza, GRUBIN; Kari M., WAHLQUIST; Andrew J.
CPC: H04N21/2335; H04N21/4398; G06V20/35; G06V20/41; G06F18/41; G06V40/20; G06N3/09; G06N3/0464; G06N3/084; H04N21/234372; H04N21/44008; G06V40/172; H04N21/466; G06V20/46; H04N7/0122; G06N5/025; G06V40/168; G06F18/211; G06F3/04845; H04N21/440272
Verdict: Set aside aspect ratio conversion, generic video processing
Source: Google Patents · FreePatentsOnline

Abstract

Techniques are disclosed for converting image frames, such as the image frames of a motion picture, from one aspect ratio to another while predicting the pan and scan framing decisions that a human operator would make. In one configuration, one or more functions for predicting pan and scan framing decisions are determined, at least in part, via machine learning using training data that includes historical pan and scan conversions. The training data may be prepared by extracting features indicating visual and/or audio elements associated with particular shots, among other things. Function(s) may be determined, using machine learning, that take such extracted features as input and output predicted pan and scan framing decisions. Thereafter, the image frames of a received video may be converted between aspect ratios on a shot-by-shot basis, by extracting the same features and using the function(s) to make pan and scan framing predictions.

Background

BACKGROUNDField of the Disclosure

Aspects presented in this disclosure generally relate to converting image frames of a video between aspect ratios.Description of the Related Art

The aspect ratio of an image is a proportional relationship between a width of the image and a height of the image. Motion picture content is typically created for theatrical distribution in a wide-screen aspect ratio, such as the 2.39:1 aspect ratio, that is wider than aspect ratios supported by in-home display devices, such as the 1.78:1 aspect ratio for high-definition televisions. As a result, image frames of motion pictures in wide-screen aspect ratios need to be converted to smaller aspect ratios for home video distribution. One approach for aspect ratio conversion is pan and scan, which entails reframing image frames to preserve the primary focus of scenes depicted therein, while cropping out the rest of the image frames (e.g., the sides of wide-screen aspect ratio image frames). The faming decisions may include shifting the preserved window left or right to follow the action in a scene, creating the effect of a “pan” shot, and ensuring that the focus of each shot is preserved (i.e., “scanned”). Traditional pan and scan requires manual decisions to be made on how to represent on-screen action throughout a motion picture in a new aspect ratio, which can be both labor intensive and time consuming.SUMMARY

One aspect of this disclosure provides a computer-implemented method for c

Claims

1. A computer-implemented method for converting image frames between aspect ratios, the method comprising: determining using machine learning at least one aspect ratio conversion function based, at least in part, on (i) extracted features which indicate at least one of visual or audio elements associated with first image frames in shots from videos in a first aspect ratio, and (ii) manual framing decisions in pan and scan conversions of the first image frames to corresponding second image frames in a second aspect ratio; identifying shots in a received video; and for each image frame in each shot of the identified shots, converting the image frame from the first aspect ratio to the second aspect ratio using one of the at least one aspect ratio conversion function. 11. A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to convert image frames between aspect ratios, by performing operations comprising: determining using machine learning at least one aspect ratio conversion function based, at least in part, on (i) extracted features which indicate at least one of visual or audio elements associated with first image frames in shots from videos in a first aspect ratio, and (ii) manual framing decisions in pan and scan conversions of the first image frames to corresponding second image frames in a second aspect ratio; identifying shots in a received video; and for each image frame in each shot of the identified shots, converting the image frame from the first aspect ratio to the second aspect ratio using one of the at least one aspect ratio conversion function. 20. A system, comprising: one or more processors; and a memory containing a program that, when executed on the one or more processors, performs operations for converting image frames between aspect ratios, the operations comprising: determining using machine learning at least one aspect ratio conversion function based, at least in part, on (i) extracted features which indicate at least one of visual or audio elements associated with first image frames in shots from videos in a first aspect ratio, and (ii) manual framing decisions in pan and scan conversions of the first image frames to corresponding second image frames in a second aspect ratio, identifying shots in a received video, and for each image frame in each shot of the identified shots, converting the image frame from the first aspect ratio to the second aspect ratio using one of the at least one aspect ratio conversion function.