InsightLake Metadata Store enables companies to obtain metadata from various data sources like databases, files, real time streams etc and store them for easy exploration and integration with other applications.
Metadata Store defines metadata layer on various data entities.
Metadata Store captures technical, operational and business domain metadata and stores them at central location for easy exploration.
Table schema, File type, format, AVRO JSON schema, tags are some of the technical information elements about data assets which Metadata Store captures and stores. Data profiling feature allows extraction of known technical metadata like data field type, size, min and max values, sample values etc. It also extracts derived information like geo, currency, business domain types etc. All types of metadata gets stored in SOLR based central store to allow fast exploration and REST based integration with other enterprise applications.
Metadata Store captures operational metadata from running jobs like number of records ingested, time duration, status etc.
Metadata Store allows companies to create metadata at various levels like data source, data location, store, field and cell. Company can provision tags and properties at these levels and access them in various data pipelines or applications.
Metadata Store stores metadata in SOLR based central storage. It provides fast metadata search and exploration capability. SOLR also exposes metadata store as REST apis, which applications can use to access metadata.
Any data element can be tagged for example a table or cell can be tagged as secure, which can then be used by security policy manager to automatically secure the access to the table or cell. Ingestion flow can be tokenized with tags, which can flow through end to end data pipeline for better lineage tracking. Other than tagging, properties (name, value) can be defined on data elements. For example on credit card column a property "Masking" with value "last 4" can be defined and used in business rules where ever data gets processed.
Metadata Store enables companies to put business context over technical metadata to provide clear business terms on top of physical data. It also increases the productivity of the enterprise. Generated glossary helps cross functional alignment between various business groups and provides a common business vocabulary across organization.