- Number
- 20250370974
- Published
- 2025-12-04
- Filed
- 2024-05-29
- Assignee
- Disney Enterprises, Inc.
- Inventors
- SENERTH; Mark F. et al.
- CPC
- G06F16/2228
- Verdict
- Set aside data-lake analytics, business
- Source
- Google Patents · FreePatentsOnline
Abstract
Techniques for identifying and applying data relationships. These techniques include retrieving log data relating to data stored in a plurality of electronic repositories. The techniques further include transforming the log data to generate structured table data, including matching a pattern in the log data to generate the structured table data, and identifying a plurality of data relationships for the data stored in the plurality of electronic repositories. This includes identifying metadata associated with the data stored in a plurality of electronic repositories, and correlating the generated structured table data with the associated metadata. The techniques further include applying the identified plurality of data relationships to at least one of: (i) modify operation of a computer software job operating on the data stored in the plurality of electronic repositories or (ii) identify for removal a portion of the data stored in a plurality of electronic repositories.
Background
BACKGROUND
Managing and optimizing large-scale electronic data storage (e.g., large-scale data lakes) is a challenging problem. For example, failure to identify redundant, unused, or rarely-used data can result in excessive and inefficient data storage and management. Further, improperly managed electronic data storage can increase computational burdens for a variety of applications, and can pose security and compliance risks.
Claims
1. A computer-implemented method, comprising: retrieving log data relating to data stored in a plurality of electronic repositories; transforming the log data to generate structured table data, comprising: matching a pattern in the log data to generate the structured table data; identifying, using one or more computer processors, a plurality of data relationships for the data stored in the plurality of electronic repositories, comprising: identifying metadata associated with the data stored in a plurality of electronic repositories; and correlating the generated structured table data with the associated metadata; and applying the identified plurality of data relationships to at least one of: (i) modify operation of a computer software job operating on the data stored in the plurality of electronic repositories or (ii) identify for removal a portion of the data stored in a plurality of electronic repositories. ||
10. A non-transitory computer program product comprising: one or more non-transitory computer readable media containing, in any combination, computer program code that, when executed by operation of any combination of one or more processors, performs operations comprising: retrieving log data relating to data stored in a plurality of electronic repositories; transforming the log data to generate structured table data, comprising: matching a pattern in the log data to generate the structured table data; identifying, using one or more computer processors, a plurality of data relationships for the data stored in the plurality of electronic repositories, comprising: identifying metadata associated with the data stored in a plurality of electronic repositories; and correlating the generated structured table data with the associated metadata; and applying the identified plurality of data relationships to at least one of: (i) modify operation of a computer software job operating on the data stored in the plurality of electronic repositories or (ii) identify for removal a portion of the data stored in a plurality of electronic repositories. ||
16. A system, comprising: one or more processors; and one or more memories storing a program, which, when executed on any combination of the one or more processors, performs operations, the operations comprising: retrieving log data relating to data stored in a plurality of electronic repositories; transforming the log data to generate structured table data, comprising: matching a pattern in the log data to generate the structured table data; identifying, using one or more computer processors, a plurality of data relationships for the data stored in the plurality of electronic repositories, comprising: identifying metadata associated with the data stored in a plurality of electronic repositories; and correlating the generated structured table data with the associated metadata; and applying the identified plurality of data relationships to at least one of: (i) modify operation of a computer software job operating on the data stored in the plurality of electronic repositories or (ii) identify for removal a portion of the data stored in a plurality of electronic repositories.