Methods and systems for converting user data from disparate sources to bitmap data are described, where user data from a plurality of different data sources are conformed to a conformed user data set using a conform mapping schema, and the conformed data set is then analyzed to create a bitmap mapping schema which provides a mapping from the conformed user data set to a bitmap data set. The conformed user data set is then converted to a bitmap data set using the bitmap mapping schema and the current user data in the conformed user data set. A query user interface (UI) may be provided to allow clients or others to query the resulting bitmap data set.
BACKGROUND (1) Large and complex data management systems manage large quantities of data stored in many disparate data sources, such that querying and returning results from such systems in an efficient time frame is not possible. If such systems were faster or more efficient, they could be used to efficiently perform tasks such as determining potential audiences for targeted services or advertising. It is desirable to quickly obtain a complete picture (or view) of a user's preferences and behaviors/activities across multiple products (or applications) and platforms. However, this is difficult because users interact with numerous different products and platforms, each of which collects and generates information about the users in different formats and often saves them in different servers or data sources. (2) Also, querying user data for millions of users can take a long time to provide the query results, e.g., 30 min. to 10+ hours, depending on the search criteria, given the enormous amount of data (e.g., over 1 terabyte per day) that must be searched. (3) Accordingly, it would be desirable to have a method and system that provides very fast query results of a large quantity of user data which includes user data across multiple products and platforms.
1. A method for converting user data from a plurality of users and from a plurality of disparate data sources to bitmap data, comprising: receiving the user data from the plurality of data sources, the user data indicative of user attributes; receiving a conform mapping schema; creating a conformed user data set for the plurality of users using the conform mapping schema; wherein the creating the conformed user data set comprises reconciling conflicts in data types for the same attribute between different data sources using the conform mapping schema; analyzing the conformed user data set to obtain statistics about the user data for each user attribute; creating a bitmap mapping schema based on the statistics about the user data set; loading the conformed user data set into a bitmap data set using the bitmap mapping schema; wherein the analyzing is only performed initially and when the conformed user data set has changed and the bitmap mapping schema is updated only when the conformed user data set has changed; and providing a user interface (UI) where queries are performed on the bitmap data set, and which provides a plurality of selectable fields and conditions based on the bitmap data for a client to perform a search of the bitmap data set. ||
17. A method for converting user data from a plurality of users and from a plurality of disparate data sources to bitmap data, comprising: receiving the user data from the plurality of data sources, the user data indicative of user attributes; receiving a conform mapping schema; creating a conformed user data set for the plurality of users using the conform mapping schema; wherein the creating the conformed user data set comprises reconciling conflicts in data types for the same attribute between different data sources using the conform mapping schema; analyzing the conformed user data set to obtain statistics about the user data for each user attribute; creating a bitmap mapping schema based on the statistics about the user data set; loading the conformed user data set into a bitmap data set using the bitmap mapping schema; wherein the analyzing is only performed initially and when the conformed user data set has changed and the bitmap mapping schema is updated only when the conformed user data set has changed; and providing a user interface (UI) where queries are performed on the bitmap data set. ||
27. A method for improving the speed of obtaining query results from a massive, disparate data set, comprising: converting the data to bitmap data set format, comprising: receiving the data from the plurality of data sources, the user data indicative of user attributes; receiving a conform mapping schema; creating a conformed user data set for the plurality of users using the conform mapping schema; analyzing the conformed user data set to obtain statistics about the user data for each user attribute; creating a bitmap mapping schema based on the statistics about the user data set; and loading the conformed user data set into a bitmap data set using the bitmap mapping schema; wherein the creating the conformed user data set comprises reconciling conflicts in data types for the same attribute between different data sources using the conform mapping schema; wherein the analyzing is only performed initially and when the conformed user data set has changed and the bitmap mapping schema is updated only when the conformed user data set has changed; querying the bitmap data set; returning a query result; and wherein massive comprises larger than 1 million records, each record having a plurality of attributes and disparate comprises at least 3 different data sources in at least 3 different data formats or types.