Large-scale environmental mapping in real-time by a robotic system

Number: 10484659
Published: 2019-11-19
Filed: 2017-08-31
Assignee: Disney Enterprises, Inc.
Inventors: Beardsley; Paul, Vempati; Anurag Sai, Nieto; Juan, Gilitschenski; Igor
CPC: H04N13/122; H04N13/271; G01C15/002; G01C21/20; G05D1/00; G05D1/0011; G05D1/0202; G05D1/0206; G05D1/0248; G05D1/027; G05D1/0274; G06T15/08; G06T17/00; G06T17/05; H04N13/25; H04N13/257; H04N13/296
Verdict: High Hardware
Source: Google Patents · FreePatentsOnline

The keeper's note

Large-scale environmental mapping real-time robotic system (Beardsley).

Abstract

According to one implementation, a robotic system for performing large-scale environmental mapping in real-time includes a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory, and a navigation unit communicatively coupled to the mobile reconnaissance unit and having a central processing unit (CPU) with a CPU memory. The robotic system begins a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor, and generates mapping data for populating a volumetric representation of the environment. The robotic system continues the 3D scan of the environment using the color sensor and the depth sensor, updates the mapping data, and partitions the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria.

Background

BACKGROUND(1) The generation of detailed and accurate representations of a local environment, using dense simultaneous localization and mapping (dense SLAM) methods, for example, can be important for robotics applications such as navigation and scene interpretation. Although one obstacle to successfully producing such environmental representations has been the processing overhead required by dense SLAM, advances in computing technology have made that particular obstacle less formidable. For example, powerful graphics processing units (GPUs) enabling dense SLAM algorithms to harness the power of parallelization are now widely available.(2) Several further challenges need to be addressed in order to make dense SLAM suitable for real-world applications, however. For example, conventional dense SLAM systems typically do not scale to large-scale environments because they are constrained by GPU memory, thus limiting the size of the area that can be mapped. Another limitation is the inability to handle rapid or abrupt camera motion, which is particularly problematic for agile aerial robotic vehicles, such as aerial drones, for example.SUMMARY(3) There are provided robotic systems and methods for performing large-scale environmental mapping in real-time, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims.

Claims

1. A robotic system comprising: a plurality of mobile reconnaissance units including a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory; a navigation unit communicatively coupled to the plurality of mobile reconnaissance units, the navigation unit having a central processing unit (CPU) with a CPU memory, wherein the navigation unit is a remote base station for all of the plurality of mobile reconnaissance units; the robotic system configured to: begin a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generate a mapping data for populating a volumetric representation of the environment produced using the GPU; continue the 3D scan of the environment using the color sensor and the depth sensor; update the mapping data based on the continued 3D scan; and partition the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit. 2. The robotic system of claim 1, wherein the navigation unit further comprises a display screen, and wherein the robotic system is further configured to display a visual representation of the environment of the mobile reconnaissance unit on the display screen. 3. The robotic system of claim 1, wherein the navigation unit is integrated with the mobile reconnaissance unit. 4. The robotic system of claim 1, wherein the mobile reconnaissance unit comprises an aerial vehicle. 5. The robotic system of claim 1, wherein the mobile reconnaissance unit comprises one of a wheeled vehicle, a legged vehicle, and a continuous track propulsion vehicle. 6. The robotic system of claim 1, wherein the mobile reconnaissance unit comprises a submersible vehicle. 7. A robotic system comprising: a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory; a navigation unit communicatively coupled to the mobile reconnaissance unit, the navigation unit having a central processing unit (CPU) with a CPU memory; the robotic system configured to: begin a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generate a mapping data for populating a volumetric representation of the environment produced using the GPU; continue the 3D scan of the environment using the color sensor and the depth sensor; update the mapping data based on the continued 3D scan; and partition the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit, wherein the mobile reconnaissance unit further comprises an inertial sensor, and wherein the robotic system is further configured to: detect an abrupt movement of the mobile reconnaissance unit during the second 3D scan, using the inertial sensor; generate a perturbation data corresponding to the abrupt movement; and correct the mapping data to compensate for the abrupt movement using the perturbation data. 8. The robotic system of claim 7, wherein the navigation unit is a remote base station for controlling the mobile reconnaissance unit. 9. The robotic system of claim 7, wherein the mobile reconnaissance unit is one of a plurality of mobile reconnaissance units communicatively coupled to the navigation unit, and wherein the navigation unit is a remote base station for all of the plurality of mobile reconnaissance units. 10. A robotic system comprising: a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory; a navigation unit communicatively coupled to the mobile reconnaissance unit, the navigation unit having a central processing unit (CPU) with a CPU memory; the robotic system configured to: begin a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generate a mapping data for populating a volumetric representation of the environment produced using the GPU; continue the 3D scan of the environment using the color sensor and the depth sensor; update the mapping data based on the continued 3D scan; and partition the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit, and wherein the color sensor is an RGB camera, and the RGB camera and the depth sensor form an integrated RGB-D sensor of the mobile reconnaissance unit. 11. A method for use by a robotic system comprising a plurality of mobile reconnaissance units including a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory, and further including a navigation unit communicatively coupled to the plurality of mobile reconnaissance units and having a central processing unit (CPU) with a CPU memory, the navigation unit being a remote base station for all of the plurality of mobile reconnaissance units, the method comprising: beginning a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generating a mapping data for populating a volumetric representation of the environment produced using the GPU; continuing the 3D scan of the environment using the color sensor and the depth sensor; updating the mapping data based on the continued 3D scan; and partitioning the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit. 12. The method of claim 11, wherein the navigation unit further includes a display screen, and wherein the method further comprises displaying a visual representation of the environment of the mobile reconnaissance unit on the display screen. 13. The method of claim 11, wherein the navigation unit is integrated with the mobile reconnaissance unit. 14. The method of claim 11, wherein the mobile reconnaissance unit comprises an aerial vehicle. 15. The method of claim 11, wherein the mobile reconnaissance unit comprises one of a wheeled vehicle, a legged vehicle, and a continuous track propulsion vehicle. 16. The method of claim 11, wherein the mobile reconnaissance unit comprises a submersible vehicle. 17. A method for use by a robotic system including a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory, and further including a navigation unit communicatively coupled to the mobile reconnaissance unit and having a central processing unit (CPU) with a CPU memory, the method comprising: beginning a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generating a mapping data for populating a volumetric representation of the environment produced using the GPU; continuing the 3D scan of the environment using the color sensor and the depth sensor; updating the mapping data based on the continued 3D scan; and partitioning the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit, wherein the mobile reconnaissance unit further includes an inertial sensor, and wherein the method further comprises: detecting a abrupt movement of the mobile reconnaissance unit during the second 3D scan, using the inertial sensor; generating a perturbation data corresponding to the abrupt movement; and correcting the mapping data to compensate for the abrupt movement using the perturbation data. 18. The method of claim 17, wherein the navigation unit is a remote base station for controlling the mobile reconnaissance unit. 19. The method of claim 17, wherein the mobile reconnaissance unit is one of a plurality of mobile reconnaissance units communicatively coupled to the navigation unit, and wherein the navigation unit is a remote base station for all of the plurality of mobile reconnaissance units. 20. A method for use by a robotic system including a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory, and further including a navigation unit communicatively coupled to the mobile reconnaissance unit and having a central processing unit (CPU) with a CPU memory, the method comprising: beginning a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generating a mapping data for populating a volumetric representation of the environment produced using the GPU; continuing the 3D scan of the environment using the color sensor and the depth sensor; updating the mapping data based on the continued 3D scan; and partitioning the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit, and wherein the color sensor is an RGB camera, and the RGB camera and the depth sensor form an integrated RGB-D sensor of the mobile reconnaissance unit.