Described herein are apparatus, systems and methods for synchronizing a reference video with an input video. A method comprises extracting first motion data from the input video having a first set of frames, extracting second motion data from the reference video having a second set of frames, computing motion descriptors for each frame in the first set of frames and the second set of frames based on the first and second motion data, respectively, and non-linearly mapping the first set of frames to the second set of frames based on the motion descriptors, respectively.
BACKGROUND(1) Video synchronization aims to temporarily align a set of input videos acquired by multiple cameras. Video synchronization may be a fundamental step for many applications in computer vision, such as three-dimensional (“3D”) reconstruction from multiple cameras, video morphing, facial performance manipulation, spatial compositing, motion analysis, etc. When several cameras are simultaneously used to acquire multiple viewpoint shots of a scene, then synchronization may be trivially achieved by using timecode information or camera triggers.(2) In the absence of timecode information and camera triggers, the videos may be synchronized using a recorded audio track, wherein the synchronization finds a fixed time offset between the cameras. Furthermore, videos may be synchronized through manual alignment, such as by finding video frame correspondence and manually computing the required time offset. However, these techniques require time-consuming manual effort for video alignment.(3) Existing video synchronization methods have significant limitations. For instance, existing techniques typically require simultaneously acquired viewpoint shots of a scene, videos that are acquired in the same location, a fixed temporal offset between the cameras, specific use cases (e.g., videos of faces), a similar appearance change, a combination thereof, etc.SUMMARY(4) Described herein are apparatus, systems and methods for nonlinear synchronization of action videos.(5) A method of synch
1. A method of synchronizing a reference video with an input video, comprising: extracting first motion data from the input video having a first set of frames, wherein the input video includes a first scene during which a first action is performed; extracting second motion data from the reference video having a second set of frames, wherein the reference video includes a second scene during which a second action is performed, wherein the first action and the second action are different based on at least one of (i) a time when the first action is performed and a time when the second action is performed or (ii) a performer of the first action and a performer of the second action; computing motion descriptors for each frame in the first set of frames and each frame in the second set of frames based on the first motion data and the second motion data, respectively; generating a temporal correspondence between the first set of frames and the second set of frames using a non-linear mapping between the first set of frames and the second set of frames based on the motion descriptors; and generating a video sequence that includes the input video synchronized with the reference video based on the generated temporal correspondence.
8. A device, comprising: a processor coupled to a memory, wherein the processor is programmed for synchronizing a reference video with an input video by: extracting first motion data from the input video having a first set of frames, wherein the input videos includes a first scene during which a first action is performed; extracting second motion data from the reference video having a second set of frames, wherein the reference video includes a second scene during which a second action is performed, wherein the first action and the second action are different based on at least one of (i) a time when the first action is performed and a time when the second action is performed or (ii) a performer of the first action and a performer of the second action; computing motion descriptors for each frame in the first set of frames and each frame in the second set of frames based on the first motion data and the second motion data, respectively; generating a temporal correspondence between the first set of frames and the second set of frames using a non-linear mapping between the first set of frames and the second set of frames based on the motion descriptors; and generating a video sequence that includes the input video synchronized with the reference video based on the generated temporal correspondence.
15. A non-transitory computer readable storage medium with an executable program stored thereon, wherein the program instructs a microprocessor to perform operations to synchronize a reference video with an input video comprising: extracting first motion data from the input video having a first set of frames, wherein the input video includes a first scene during which a first action is performed; extracting second motion data from the reference video having a second set of frames, wherein the reference video includes a second scene during which a second action is performed, wherein the first action and the second action are different based on at least one of (i) a time when the first action is performed and a time when the second action is performed or (ii) a performer of the first action and a performer of the second action; computing motion descriptors for each frame in the first set of frames and each frame in the second set of frames based on the first motion data and the second motion data, respectively; generating a temporal correspondence between the first set of frames and the second set of frames using a non-linear mapping between the first set of frames and the second set of frames based on the motion descriptors; and generating a video sequence that includes the input video synchronized with the reference video based on the generated temporal correspondence.