Data Catalog: Organizing and Discovering Data Assets

You have 3,000 tables in Snowflake. Business analysts need customer revenue data. Data scientists need feature engineering datasets. Engineers need to know which tables are actively maintained and which are deprecated. Without a catalog, people ask in Slack, get inconsistent answers, and eventually just pick a table that looks right.

A data catalog is a searchable inventory of your data assets. It stores metadata about tables, columns, pipelines, dashboards, and the people and processes associated with them. It is the difference between data as a black box and data as a governed, discoverable resource.

What a Data Catalog Does

At minimum, a data catalog records what data exists and where it lives. Modern catalogs do significantly more.

Discovery: Search for tables by name, column, tag, description, or owner. A data scientist looking for customer features can search “customer features” rather than guessing table names.

Lineage: Track how data flows from source to transformation to output. If a revenue number looks wrong, lineage shows which tables and pipelines contributed to it.

Documentation: Store human-readable descriptions of tables and columns. “This column represents the customer’s lifetime value in USD, calculated as sum of all orders minus returns.”

Governance: Enforce access policies, mark sensitive columns (PII, PHI), track data contracts.

Quality: Store data quality metrics, test results, and freshness SLAs.

Metadata: The Foundation

A data catalog is only as good as its metadata. There are several layers.

Technical metadata: Table names, column names and types, partition keys, indexes, file formats, storage locations, row counts, data volume.

{
  "table_name": "fact_orders",
  "schema": "analytics",
  "database": "snowflake",
  "columns": [
    { "name": "order_id", "type": "VARCHAR", "nullable": false },
    { "name": "customer_id", "type": "NUMBER", "nullable": false },
    { "name": "order_total_usd", "type": "NUMBER(10,2)", "nullable": false },
    { "name": "order_date", "type": "DATE", "nullable": false }
  ],
  "partition_keys": ["order_date"],
  "row_count": 1450000000,
  "bytes": 85899345920
}

Business metadata: Human-readable descriptions, ownership, business context, data contracts. This is the hardest metadata to maintain because it requires human input and upkeep.

Operational metadata: Last updated timestamp, data freshness, pipeline job status, quality test results, access patterns.

Structural metadata: Schema information, column-level statistics, key distributions.

How Data Catalogs Work

Data catalogs typically integrate with data platforms through connectors or automated scanning.

Automated discovery

Catalog tools connect to data warehouses, data lakes, and BI platforms and automatically scan for new tables. When a new table appears in Snowflake, the catalog picks it up within the next scan cycle.

# Example: OpenMetadata scanner configuration
# Scans Snowflake and ingests table metadata automatically
connectors:
  - type: snowflake
    config:
      host: account.snowflakecomputing.com
      username: catalog_service_account
      role: sysadmin
      databases:
        - analytics
        - production
      schemaFilter:
        include:
          - "analytics.*"
          - "production.customers"

Manual curation

Not everything can be automated. Business context, data contracts, and sensitive column markings require human input. The best catalogs combine automated scanning with curation workflows.

Curation targets business metadata — the descriptions, definitions, and classifications that no scanner can infer. A column named clv might show up in a scan as VARCHAR(10), but only a human knows it means customer lifetime value and should be marked sensitive. Owners and data stewards fill these gaps through a review interface or API. Most catalog tools surface untagged columns as curation tasks, prioritizing the most-used tables first.

Beyond descriptions, curation handles data contracts and column classifications. A contract records an agreed SLA between a producer and consumer — say, “this table refreshes every 6 hours and never contains nulls in the customer_id column.” When a producer changes the contract, the catalog alerts downstream consumers before they build on stale assumptions. Classifications like PII, PHI, or confidential sit at the column or table level and inform access policy.

The problem with manual curation is that it tends to happen once and then quietly dies. Teams annotate tables during the initial launch, feel good about it, and move on. Six months later the descriptions are stale or missing. Effective curation makes the work part of existing processes: require column descriptions in your dbt model definitions, assign data stewards to review high-traffic tables quarterly, and watch what percentage of tables lack business metadata — if it climbs, the catalog is drifting toward noise. Without a process, the scanner keeps running and the human context slowly evaporates.

Lineage extraction

Catalogs can derive lineage from multiple sources:

• SQL parsing: Parse transformation SQL to understand which tables feed into which
• Pipeline metadata: Pull lineage from Airflow DAGs or dbt models
• BI metadata: Extract which reports and dashboards use which tables

-- Example lineage query in OpenMetadata
SELECT
    from_table FullyQualifiedName,
    to_table FullyQualifiedName,
    sql_query
FROM lineage
WHERE sql_query ILIKE '%customer_id%'
ORDER BY to_table;

Popular Data Catalog Tools

Apache Atlas

Apache Atlas is the open-source standard for Hadoop ecosystems. It provides metadata management, lineage, and governance for data lakes built on Hive, Spark, and Kafka.

Atlas uses a type system for metadata objects. You define types (like “hive_table” or “kafka_topic”), create instances of those types, and attach classifications (like “PII” or “confidential”). Lineage is derived from job metadata collected from Spark and Hive.

Atlas plugs into the Hadoop stack through hooks that intercept operations in Hive, Spark, and Kafka. When a Spark job writes to a Parquet table, Atlas captures that relationship through the hook and builds lineage edges automatically. The hook approach means you do not configure lineage manually for every pipeline — it flows from the job execution itself. The catch is that Atlas only sees what the hooks cover. If you run a transformation outside the hook ecosystem (say, a Python script writing directly to S3), Atlas misses it.

Governance in Atlas centers on classifications and business metadata. Tag a column as PII and that classification propagates through lineage automatically — any table feeding into the tagged column inherits the marking. This propagation helps compliance teams track where sensitive data flows, but it requires the lineage graph to be accurate. Hook-based lineage is only as complete as the plugin coverage for your specific tools. Atlas ships hooks for the major Hadoop components; for everything else, you decide whether to write a custom hook or accept the gap.

The type system is Atlas’s most distinctive feature. Rather than treating all assets the same way, you define a hive_table type with specific attributes (name, owner, create time, location) and a kafka_topic type with different attributes (topic name, partition count, retention). You can subclass these to add your own attributes. Admins manage type definitions through the REST API or the web UI. For large organizations with complex governance requirements, this typed model is Atlas’s clearest advantage over flatter catalog designs.

DataHub

DataHub is a modern open-source metadata platform built on PostgreSQL and Elasticsearch. It provides real-time metadata updates through a streaming architecture and a GraphQL API for consumption.

DataHub’s strength is its extensibility. The metadata model is flexible, and there are pre-built connectors for Snowflake, BigQuery, Kafka, Airflow, and dbt.

The streaming architecture uses a metadata events (MCE) pipeline built on Kafka. When something changes in your data platform — a new table appears in Snowflake, a dbt model gets modified, a Kafka topic is created — DataHub ingests that event and updates its search index in near real-time. Kafka sits in the middle as a buffer and fan-out mechanism, so multiple consumers can react to metadata changes without polling the source systems. This architecture scales better than hook-based approaches for large, heterogeneous environments where many independent systems emit metadata simultaneously.

DataHub exposes everything through a GraphQL API, which makes building custom integrations more straightforward than the REST-heavy APIs of older catalog tools. You write GraphQL queries to browse lineage, search for datasets, or pull ownership metadata. The API also powers the web UI, so anything you can do in the UI is available programmatically. This matters when you want to automate curation tasks — syncing ownership from a Slack user directory, pushing column descriptions from a dbt model, or building a custom dashboard on top of catalog data.

The connector ecosystem is where DataHub earns its reputation for general-purpose use. Pre-built extractors pull metadata from Snowflake, BigQuery, Redshift, Looker, Tableau, Airflow, dbt, and more. Each connector runs as a scheduled ingestion job that talks to the source platform, extracts metadata, and publishes it to the Kafka MCE topic. Connector configuration lives in YAML, version-controlled and deployable through standard CI/CD. For teams with a mixed cloud stack, this connector coverage is usually the deciding factor over Atlas.

Amundsen

Amundsen, built by Lyft, focuses on search and discovery. It aggregates metadata from various sources into a searchable index optimized for data consumer workflows.

Amundsen’s philosophy is that the most important metadata is what helps people find and understand data quickly. It prioritizes the search experience over governance workflows.

The architecture separates indexing from serving. A group of publisher services pull metadata from source systems (Hive, dbt, Airflow, and others) and write to a search index backed by Elasticsearch. A separate frontend service handles the web UI and exposes a REST API. The separation means the search index can be rebuilt independently of the serving layer — useful when you want to reindex without downtime or test index changes in a staging environment before promoting them.

Amundsen’s data model centers on three entities: Table, Column, and User. Everything flows from these. A Table record holds the table name, schema, description, and owner. Column records attach to a Table and carry the column name, type, description, and sort order. User records represent owners and suggest people as experts for particular datasets. The model is deliberately narrow — if your catalog needs to track Kafka topics, ML models, or API endpoints, you either extend the model or choose a different tool. This simplicity works well for search-first use cases but becomes a limitation for governance-heavy environments.

The search ranking is Amundsen’s standout characteristic. It weights popularity signals — how often a table is queried, how frequently it appears in dashboards — to surface the most useful results first. A table that nobody uses drifts down the results. A table that analysts query daily rises. This relevance tuning is hard to replicate in catalogs that treat all assets equally. The cost is that popularity signals require instrumentation: Amundsen needs to see query logs or dashboard references to build those signals. Without that telemetry, the ranking degrades toward pure text match.

Cloud-native catalogs

AWS Glue Catalog, Google Cloud Data Catalog, and Azure Purview provide managed catalog services that integrate with their respective cloud platforms. If your data stack is heavily cloud-vendor specific, these reduce operational overhead.

Each service ties deeply into its provider’s ecosystem. AWS Glue Catalog shares a metastore with Athena, Lake Formation, and EMR, meaning any service that reads the Glue Data Catalog automatically sees your catalog entries without extra configuration. Similarly, Google Cloud Data Catalog integrates natively with BigQuery datasets, Cloud Storage buckets, and Dataproc Metastore, automatically surfacing metadata as you create resources. Azure Purview connects to Azure Data Factory, Cosmos DB, SQL Database, and more through a single unified view. The tight integration means less configuration work and automatic metadata freshness for resources you already manage in those platforms.

The tradeoff is lock-in. A catalog entry created in AWS Glue lives in the AWS ecosystem and does not port to Google Cloud or Azure. If you plan to migrate between cloud providers or run a multi-cloud environment, a cloud-native catalog becomes a migration liability. Additionally, each service has lineage capabilities specific to its own processing services, which means cross-cloud pipeline lineage requires a third-party catalog that can ingest from multiple clouds. The comparison table below summarizes the key dimensions.

Data catalog tool comparison:

Tool	Best for	Strengths	Weaknesses
Apache Atlas	Hadoop ecosystems	Native Hive/Spark/Kafka lineage, governance types	Steep setup, UI dated, slow
DataHub	General-purpose	PostgreSQL + Elasticsearch, GraphQL API, extensible connectors	Real-time metadata needs extra config
Amundsen	Search-first discovery	Lyfts search quality focus, lightweight	Less governance tooling out of the box
AWS Glue	AWS-only shops	Managed, integrates with Athena, Lake Formation	Lock-in, limited lineage beyond AWS
Google Data Catalog	GCP-only shops	Managed, integrates with BigQuery	Lock-in, regional limits
Azure Purview	Azure-only shops	Managed, integrates with ADF, Cosmos DB	Lock-in, cost at scale

Data Catalog in the Modern Stack

In a modern data stack, the catalog sits at the center and integrates with every platform.

flowchart TD
    Snowflake -->|metadata| Catalog
    BigQuery -->|metadata| Catalog
    S3 -->|metadata| Catalog
    Kafka -->|metadata| Catalog
    Airflow -->|lineage| Catalog
    dbt -->|lineage| Catalog
    Looker -->|dashboard metadata| Catalog
    Catalog -->|search, lineage| Analyst
    Catalog -->|policy enforcement| Governance

dbt has become a de facto metadata source for data catalogs. dbt models define transformations, and catalog tools extract model definitions and test results as metadata. The dbt manifest.json and catalog.json files are widely used as lineage sources.

For more on how transformation pipelines generate lineage, see Extract-Transform-Load and dbt.

Implementing a Data Catalog: Practical Steps

Start with what you have. Scanning an existing Snowflake instance and ingesting technical metadata takes hours, not weeks. Automate the basics before investing in curation.

Define ownership. Every table should have an owner (team or individual). Owners are responsible for documentation and are the first point of contact when questions arise.

Tag sensitive data. PII, PHI, financial data, and other regulated categories should be marked in the catalog. This enables governance workflows and access control.

Integrate lineage gradually. SQL parsing lineage is easiest to implement. Pipeline-level lineage requires integration with your orchestrator (Airflow, Dagster, Prefect).

Make the catalog the source of truth for data questions. When someone asks “which table has customer revenue data?”, the answer should come from the catalog, not from a Slack message.

When to Use and When Not to Use a Data Catalog

Use a data catalog when:

• You have more than a few dozen data assets (tables, streams, files) that multiple teams consume
• Discovery is a pain point — people ask in Slack instead of finding data themselves
• You need to track lineage for governance, debugging, or compliance
• You have regulated data (PII, PHI, financial) that requires column-level marking

Do not use a data catalog when:

• Your data stack is small and single-team — a shared spreadsheet may suffice until you hit real scale
• You cannot assign ownership — a catalog without owners decays into noise
• Your priority is speed to market and governance overhead will block pipelines
• Metadata curation requires more effort than the data work itself

Common Pitfalls

The catalog becomes stale. Automated scanning keeps technical metadata fresh. Business metadata and descriptions decay unless there is a process for maintaining them. Assign clear ownership and include metadata accuracy in team responsibilities.

Too many tags. Teams over-tag when first implementing governance. Start with essential classifications: PII, sensitive, deprecated. Expand only when the taxonomy has proven value.

Catalog is too hard to use. If searching the catalog is harder than asking in Slack, people will not use it. Invest in search quality and make the catalog the path of least resistance.

Observability for Data Catalogs

A data catalog quietly goes stale if nobody watches it.

What to track:

• Scan freshness: when was each platform last scanned? A catalog that has not touched Snowflake in a week is wrong more often than it is right.
• Untagged tables: what percentage of tables lack business descriptions or owners? High percentages mean the catalog is mostly noise.
• Search-to-click ratio: how often do searches end in a table visit? Low ratios mean the search results are not landing people where they need to go.
• Lineage coverage: what percentage of pipelines have lineage captured? Low coverage makes lineage useless for incident investigation.

Catalog health metrics to expose:

# Example: OpenMetadata health check
curl http://openmetadata:8585/api/v1/databaseServices | jq '.data[] | {
  name: .name,
  lastScan: .lastScanTimestamp,
  status: .connectionStatus
}'

Alert on: scan freshness older than 48 hours, connection status changes, sudden drops in search activity.

Capacity Estimation for Data Catalogs

Catalog capacity planning comes down to how many metadata objects you have and how often they change.

Metadata storage:

Each table with 50 columns generates roughly 5-10 KB of metadata per scan (column types, stats, descriptions, tags). For 3,000 tables, a single scan produces about 30 MB. Over a year with daily scans, raw metadata is roughly 11 GB before compression. PostgreSQL and Elasticsearch compress this significantly on disk.

Scan frequency vs freshness:

• Hourly scans: high freshness, high compute cost. Makes sense for fast-moving ETL environments where schemas change daily.
• Daily scans: fine for most analytical warehouses. 3,000 tables scanned in under 5 minutes with a well-indexed catalog.
• Weekly scans: acceptable for stable data warehouses that rarely change.

Search index sizing:

Amundsen and DataHub use Elasticsearch for search. Index size runs about 1-2 KB per table (name, descriptions, tags, column names). 3,000 tables need a 3-6 GB index. Budget 2x overhead for Elasticsearch.

Quick Recap

• A data catalog solves the discovery problem. When you have hundreds of tables, people should find data without asking in Slack.
• Automated scanning handles technical metadata. Human curation handles business context.
• Atlas, DataHub, and Amundsen are the main open-source choices. Cloud-native catalogs work best in single-vendor shops.
• SQL parsing, pipeline orchestrators, and BI tools all produce partial lineage — combine them.
• Watch scan freshness, untagged tables, and search effectiveness. A stale catalog is worse than no catalog.

Conclusion

A data catalog transforms data from an opaque resource into a governed, discoverable asset. It solves the discovery problem that emerges in any organization with more than a few dozen tables.

The best catalogs combine automated metadata collection with human curation. Automated scanning handles technical metadata at scale. Curation workflows handle business context that cannot be derived automatically.

Start with what you have. Catalog your existing tables before building sophisticated governance workflows. Discovery is the immediate pain point. Governance comes later.

For related reading on data pipeline components, see Pipeline Orchestration and Data Quality.