System and method for synchronizing live video with external data sources

Number: 12184914
Published: 2024-12-31
Filed: 2022-12-19
Assignee: DISNEY ENTERPRISES, INC.
Inventors: Hyde; Andrew et al.
CPC: H04N21/4305; H04N21/26258; H04L65/75; H04N21/242; H04N21/8456
Verdict: Set aside broadcast synchronization, plumbing
Source: Google Patents · FreePatentsOnline

Abstract

Methods and systems for synchronizing video with external data sources by a user device where the video is grouped into segment video files having video frames, includes establishing a persistent connection between the user device and a synchronization server, receiving a sync message associated with the beginning of a video segment to be played by the user device, the sync message having an initial recording wall clock time stamp, calculating by the user device a recording wall clock time for each frame in the segment using the initial wall clock time and a frame rate, downloading the video segment file for playing on the user device, receiving by the user device, external data having a wall clock time stamp, and displaying on the user device the external data having the same time stamp as the recording time stamp of the video frame being displayed.

Background

BACKGROUND (1) It is common practice for computer-based servers to digitally stream media content (e.g., video, music, text, graphics, and the like), live or on-demand, to electronic devices, such as mobile smartphones, tablets and the like, for viewing by a user. Such media content is often provided to the user devices (or clients) via the internet using a distributed network of computer-based data file servers deployed in one or more data centers, also referred to as a content delivery network or content distribution network (collectively, CDN). (2) HTTP-based streaming methods, such as HLS (HTTP-based Live Media Streaming communication protocol), supported or implemented by products made by companies such as Apple, Inc., HDS (HTTP Dynamic Streaming), supported or implemented by products made by companies such as Adobe, Inc., and DASH (Dynamic Adaptive bit-rate Streaming over HTTP) or MPEG-DASH, are commonly used for streaming media content and operate using “HTTP-based streaming,” where media content is stored in “segments” or “chunks” of a predetermined length (e.g., 1 to 10 seconds) and a playlist (or manifest or index) file is provided that lists, or points to, each of the segments or chunks of the media stream for display or playback purposes by the user devices. (3) HLS typically works by generating a manifest file which is continually polled by the video player and contains the location (or address or URL) where the segments of video should be downloaded from. Within

Claims

1. A method for synchronizing streaming video with external data sources received by a user device, the video being grouped into segment video files having video frames, comprising: establishing a persistent connection between the user device and a synchronization server; receiving by the user device from the synchronization server via the persistent connection, a sync message associated with the beginning of a video segment having video frames to be played by the user device, the sync message having an initial recording wall clock time stamp corresponding to a first frame of the video segment; and calculating, by the user device, a recording wall clock time for each frame in the segment using the initial recording wall clock time stamp and a frame rate for the video segment. || 22. A method for synchronizing video with external data sources on a user device, the video grouped into segment video files having video frames, comprising: establishing a persistent connection between the user device and a synchronization server; receiving a sync message associated with the beginning of a video segment to be played by the user device, the sync message having at least an initial recording wall clock time stamp corresponding to a first frame of the video segment; determining by the user device, using the initial recording wall clock time stamp and a frame rate, a recording wall clock time for each video frame in the video segment; downloading the video segment file for playing on the user device; receiving by the user device, external data having an event wall clock time stamp; and synchronizing in realtime by the user device, the external data with the video segment file having the same time stamp. || 35. A method for synchronizing video with external data sources, comprising: receiving by a user device streaming encoded video content grouped into segment video files, each segment video file having a plurality of video frames and having an initial recording wall clock time stamp corresponding to a first frame of the video segment; downloading to the user device, via a segment delivery connection between the user device and a segment server, a video segment file from the segment server, for playing on the user device; extracting from the video segment file the initial recording wall clock time stamp of the video segment; calculating, by the user device, a recording wall clock time for each frame in the video segment using the initial recording wall clock time stamp and a frame rate for the video segment; receiving by the user device external data having an event wall clock time, the external data for a given event wall clock time stamp being received by the user device before the video segment having a corresponding time stamp; and displaying on the user device, the external data having the same wall clock time stamp as the wall clock time stamp of the video frame being displayed, thereby providing realtime synchronized video and external data displayed on the user device.