- Number
- 11521411
- Published
- 2022-12-06
- Filed
- 2020-10-22
- Assignee
- DISNEY ENTERPRISES, INC.
- Inventors
- Devassy; Jayadas, Walsh; Peter
- CPC
- G06T7/292; G06V40/23; G06V40/10; G06V10/82; G06T7/70; G06V20/42
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Multi-camera 3D body-part labeling/performance-capture technique (granted).
Abstract
A system and method for providing multi-camera 3D body part labeling and performance metrics includes receiving 2D image data and 3D depth data from a plurality image capture units (ICUs) each indicative of a scene viewed by the ICUs, the scene having at least one person, each ICU viewing the person from a different viewing position, determining 3D location data and visibility confidence level for the body parts from each ICU, using the 2D image data and the 3D depth data from each ICU, transforming the 3D location data for the body parts from each ICU to a common reference frame for body parts having at least a predetermined visibility confidence level, averaging the transformed, visible 3D body part locations from each ICU, and determining a performance metric of at least one of the body parts using the averaged 3D body part locations. The person may be a player in a sports scene.
Background
BACKGROUND (1) In combat sports, such as boxing, martial arts, mixed martial arts, and kick boxing, measurement of athlete performance using sensing technology has the potential to enable advanced insights into an athlete's performance. Such measurement requires the determining of the three dimensional (3D), e.g., X, Y, Z, location in space of specific body parts (semantic segmentation) of the athlete, especially when certain body parts are blocked (or occluded) from camera view, e.g., by the athlete's body, another athlete's body, an official/referee, or other occluding object/person. (2) Current techniques for semantic segmentation, such as use of two-dimensional images or video, do not provide the necessary data and do not account for such occlusions, causing the measured data to be unusable to accurately and repeatably measure an athlete's performance or metrics, especially when the athletes are close to each other, such as in boxing, martial arts, or other sports. (3) Accordingly, it would be desirable to have a system and method that overcomes the shortcomings of the prior art and provides an accurate and robust approach to measuring athlete performance in three dimensions (or 3D).
Claims
1. A method for providing multi-camera, 3D body part labeling and performance metrics, comprising: receiving 2D image data and 3D depth data from a plurality image capture units (ICUs), each of the 2D image data and the 3D depth data indicative of a sports scene viewed by the ICUs, the sports scene having a plurality of people, each ICU viewing the people from a different viewing position; identifying two players to be analyzed for performance from the plurality of people in the sports scene for each ICU, using the 2D image data for each ICU and using a person detection model; determining 3D location data and a visibility confidence level for a predetermined number of body parts for each of the two identified players from each ICU, using the 3D depth data from each ICU and using a pose estimation model; transforming the 3D location data for the body parts for each of the two players from each ICU to a common reference frame, for body parts having the confidence level of at least a predetermined acceptable confidence level, to create visible 3D body part common location data for the body parts for each ICU; averaging the visible 3D body part common location data for the body parts for each of the two players from each ICU, to create averaged 3D body part common location data; and determining a performance metric of at least one of the body parts for at least one of the two players using the averaged 3D body part common location data. ||
24. A method for providing multi-camera, 3D body part labeling and performance metrics, comprising: receiving 2D image data and 3D depth data from a plurality image capture units (ICUs), each of the 2D image data and the 3D depth data indicative of a sports scene viewed by the ICUs, the sports scene having a plurality of people, each ICU viewing the people from a different viewing position; identifying from the plurality of people, two closest players to each ICU in the sports scene for each ICU, using the 2D image data for each ICU and using a person detection model; determining 3D location data and a confidence level for a predetermined number of body parts for each of the two closest players from each ICU, using the 3D depth data from each ICU and using a pose estimation model, including a confidence level for each of the body parts; transforming the 3D location data for the body parts for each of the two closest players from each ICU to a common reference frame, for body parts having the confidence level of at least a predetermined acceptable confidence level, to create visible 3D body part common location data for the body parts for each ICU; averaging the visible 3D body part common location data for the body parts for each of the two closest players from each ICU, to create averaged 3D body part common location data; tracking the averaged 3D body part common location data for the body parts for the two closest players from a prior image frame to a current image frame; and determining a performance metric of at least one of the body parts for at least one of the two closest players using the averaged 3D body part common location data. ||
36. A method for providing multi-camera, 3D body part labeling and performance metrics, comprising: receiving 2D image data and 3D depth data from a plurality image capture units (ICUs) each of the 2D image data and the 3D depth data indicative of a scene viewed by the ICUs, the scene having at least one person to be analyzed in the scene, each ICU viewing the at least one person from a different viewing position; determining 3D location data and a visibility confidence level for a predetermined number of body parts for the at least one person from each ICU, using the 2D image data and the 3D depth data from each ICU; transforming the 3D location data for the body parts for the at least one person from each ICU to a common reference frame, for body parts having the confidence level of at least a predetermined acceptable confidence level, to create visible 3D body part common location data for the body parts for each ICU; averaging the visible 3D body part common location data for the body parts for the at least one person from each ICU, to create averaged 3D body part common location data; and determining a performance metric of at least one of the body parts for the at least one person using the averaged 3D body part common location data.