Data Formats: JSON, CSV, Parquet, Avro, and ORC Explained

Data formats are the language your data pipeline speaks. Move data from one system to another, and the format determines how much storage you burn, how fast you query, and whether downstream systems understand what they are reading. The wrong format for the wrong workload is a silent killer of performance.

This post covers the five most common formats in data engineering: JSON, CSV, Parquet, Avro, and ORC. We will look at structure, compression, schema handling, and when to use each.

JSON: Flexibility at a Cost

JSON (JavaScript Object Notation) is the dominant format for APIs and semi-structured data. It stores data as key-value pairs nested into objects and arrays. Human-readable, flexible, and everywhere.

{
  "order_id": "ORD-12345",
  "customer": {
    "id": "CUST-789",
    "name": "Priya Sharma",
    "email": "priya@example.com"
  },
  "items": [
    { "product": "wireless-headphones", "qty": 2, "price": 79.99 },
    { "product": "usb-c-cable", "qty": 1, "price": 12.99 }
  ],
  "total": 172.97,
  "timestamp": "2026-03-27T10:23:45Z"
}

JSON schema is self-describing. Every field name appears with every record. This makes it easy to evolve records without breaking readers, but it adds significant overhead. A billion JSON records each carry every field name, burning storage and slowing scans.

JSON works well for event payloads, configuration files, and data exchange between services. It is the wrong choice for analytical workloads scanning millions of rows.

JSON trade-offs

Aspect	Assessment
Readability	Excellent — human-readable
Schema evolution	Flexible — fields can be added without breaking readers
Storage efficiency	Poor — repeated field names
Query performance	Slow — requires parsing full record
Compression	Moderate — text formats compress reasonably

CSV: Simple but Fragile

CSV (Comma-Separated Values) is the lowest common denominator for tabular data exchange. It is a plain text format with one record per line and commas delimiting fields.

order_id,customer_id,product,qty,price,total
ORD-12345,CUST-789,wireless-headphones,2,79.99,172.97
ORD-12346,CUST-456,laptop-stand,1,45.00,45.00

The simplicity is the appeal. Every tool in existence reads CSV. The problems are equally fundamental: no schema definition, no type preservation, no handling of delimiters inside fields without quoting rules, and no efficient columnar access.

CSV has no standard way to represent dates versus strings versus numbers. A column of dates arrives as text that downstream systems must parse. Null values appear as empty fields or magic strings like “NULL” or “N/A” that nobody standardized.

CSV works for one-time data transfers and flat tabular exports. It is a poor choice for production pipelines that care about types, schema evolution, or query performance.

Parquet: Columnar for Analytical Workloads

Parquet is a columnar storage format built for analytical queries. It stores data column-by-column rather than row-by-row, which means scanning a specific column reads only that column’s data from disk.

Parquet was designed for the Hadoop ecosystem and is now the default format for modern data lakes. It supports complex nested data structures similar to JSON, but organizes them in columnar fashion.

import pyarrow.parquet as pq

# Write a Parquet file
table = pa.table({
    'order_id': ['ORD-12345', 'ORD-12346'],
    'customer_id': ['CUST-789', 'CUST-456'],
    'total': [172.97, 45.00]
})
pq.write_table(table, 'orders.parquet')

# Read only specific columns (column pruning)
table = pq.read_table('orders.parquet', columns=['order_id', 'total'])

Parquet files embed a schema in the file footer and encode data using techniques like dictionary encoding and run-length encoding. A column of repeated category values compresses dramatically because the same strings appear contiguously.

Typical compression ratios for Parquet are 5:1 to 10:1 compared to raw JSON. For analytical workloads that aggregate across columns, Parquet is the standard choice.

For more on how Parquet fits into data lakes versus data warehouses, see Data Warehousing.

Avro: Row-Based with Schema Evolution

Avro is a row-based binary format designed for serialization and data exchange, particularly in Hadoop and Kafka ecosystems. Unlike Parquet, Avro stores data row by row, which makes it better for write-heavy workloads and record-level access.

Avro’s distinguishing feature is its schema. The schema lives in a separate Schema Registry or at the top of the file. Records do not carry field names with every row. Instead, data is encoded against the schema, and the schema is what provides context.

{
  "type": "record",
  "name": "Order",
  "fields": [
    { "name": "order_id", "type": "string" },
    { "name": "customer_id", "type": "string" },
    { "name": "total", "type": "double" },
    {
      "name": "timestamp",
      "type": { "type": "long", "logicalType": "timestamp-millis" }
    }
  ]
}

Avro supports schema evolution. You can add a new field with a default value, or remove a field, without breaking readers operating on older schemas. This makes Avro well-suited for streaming pipelines where schemas evolve over time.

Avro uses binary encoding, which is compact and fast to serialize and deserialize. For Kafka producers and consumers that need to exchange records with evolving schemas, Avro with Confluent Schema Registry is a common pattern.

ORC: Hive’s Columnar Format

ORC (Optimized Row Columnar) is Hive’s columnar format, similar in concept to Parquet but optimized for Hive’s workload patterns. ORC was developed to address limitations in Hive’s original RCFile format.

# Example ORC file inspection
hive --orc-file-info --file orders.orc

ORC includes features like built-in indexes (bloom filters and statistics), ACID support for Hive transactions, and native support for complex types. It stores data in stripe-based columnar layout where each stripe contains column data for a set of rows.

Feature	Parquet	ORC
Hive ACID support	No	Yes
Built-in indexes	Limited	Yes (bloom filters, min/max)
Timestamp precision	Nanoseconds	Seconds
Nested structure support	Complex types	More explicit
Primary ecosystem	Spark, Drill, Presto	Hive

ORC makes sense when you are running Hive or Spark with heavy analytical aggregations and need the built-in index features for predicate pushdown. Parquet is more broadly supported across query engines and is the safer default for general-purpose data lake storage.

Choosing a Format

The right format depends on your workload pattern.

Workload	Recommended Format
APIs and web services	JSON
One-time data exchange	CSV
Analytical queries (read-heavy)	Parquet
Streaming with schema evolution	Avro
Hive/Spark with aggregations	ORC

For most modern data pipelines landing data in a data lake, Parquet is the default choice. For Kafka message serialization with evolving schemas, Avro with Schema Registry is the common pattern. CSV and JSON persist in data exchange scenarios where their trade-offs are acceptable.

Data Format Selection Flow

Choosing the right format is a decision tree. This diagram maps the selection logic:

flowchart TD
    Start[New format decision] --> Workload{Workload type?}
    Workload -->|APIs / Web| JSON
    Workload -->|One-time export| CSV
    Workload -->|Read-heavy analytics| Analytics{Column projections?}
    Analytics -->|Wide reads / aggregations| Parquet
    Analytics -->|Hive / Spark aggregations| ORC
    Workload -->|Streaming with schema changes| Avro
    Workload -->|General-purpose lake| Parquet

Start at the top and follow the path that matches your workload. If you land on multiple candidates, use the comparison table in the “Choosing a Format” section to break the tie.

Common Anti-Patterns

Using JSON for analytical workloads

The root problem is structural. JSON stores every field name alongside every value, which works fine for a handful of records but becomes catastrophic at scale. Picture a table with 50 columns and 1 billion rows. Each row repeats all 50 field names as strings. That overhead is not marginal. It is the dominant storage cost. A 20-byte column value might share the row with 200 bytes of field names.

Beyond storage, parsing is the second killer. JSON is a text format, and every record must be parsed character by character. Even with efficient parsers, scanning 1 billion JSON records to extract values from three columns means reading and deserializing all 1 billion records first. There is no column pruning, no skip-index, no efficient projection. The query engine throws away 99% of the work it just did.

Columnar formats solve this by storing column values contiguously. When you ask for the total column, the engine reads only the total column pages from disk. The field names appear once in the schema, not once per row. For aggregations, reporting, and OLAP queries, JSON is the wrong starting point. The performance gap is not a detail. It is the entire point.

Using CSV for typed data

The trouble starts before the data arrives. CSV has no type system, which means every value arrives as a string. A column of dates like 2026-03-27 is text. A column of prices like 172.97 is text. A column of booleans like true is text. Downstream systems spend their first hour parsing, validating, and converting every value before they can do anything useful with it.

Null handling is where CSV becomes genuinely fragile. There is no standard representation for nulls. The exporting tool might write an empty field, the string NULL, the string None, N/A, n/a, null, -, or a bare 0. Two systems that both claim to export CSV might produce output that is subtly incompatible. The importing system must guess at the null representation, and guessing wrong means nulls silently become strings or zeros.

Type uncertainty compounds in production pipelines. A column that looked like integers during development turns out to contain decimal values in production. A column of UUIDs has occasional NULL values that downstream string-length validation rejects. These are not edge cases. They are the normal state of CSV pipelines. The code that handles these edge cases tends to grow until it becomes its own parsing library. Use CSV for one-time, low-stakes data exchanges where building that robustness is not worth the investment.

Choosing Parquet for write-heavy workloads

Parquet’s columnar layout is optimized for read patterns, not write patterns. When you write to Parquet, the writer must accumulate rows in memory, group them by column, apply encoding (dictionary, run-length, delta), and then write column pages to disk. This buffering overhead is acceptable for batch writes but becomes a bottleneck for high-frequency streaming inserts. Each row insert triggers partial page flushes and re-encoding that defeat the compression efficiency Parquet is known for.

Avro sidesteps this by writing records sequentially without column-level buffering. A Kafka producer writing Avro records serializes each record and appends it to a partition file. There is no column reorganization, no page formation, no dictionary encoding on write. The schema is registered once. Records are encoded compactly and written immediately. This makes Avro the natural choice for write-heavy workloads where records arrive continuously rather than in batches.

The decision rule is straightforward. If your pipeline primarily appends data, reach for Avro or another row-based format. If your pipeline primarily reads and aggregates data, Parquet is the right tool. Many pipelines land streaming data as Avro and then compact it to Parquet for analytical reads. The write format and the read format serve different purposes.

Capacity Estimation

Storage planning comes down to format, compression ratios, and data shape. Benchmarks to anchor your estimates:

Format	Uncompressed vs JSON	Typical Compression	When to Re-estimate
JSON	1x (baseline)	3-5x (gzip)	Schema changes add fields
CSV	0.9-1.1x vs JSON	3-5x (gzip)	New delimiters or quoting
Parquet	1.2-1.5x vs JSON	5-10x (Snappy/Gzip)	After column cardinality changes
Avro	1.1-1.3x vs JSON	5-8x (deflate)	After schema changes
ORC	1.2-1.4x vs JSON	5-10x (zlib)	After Hive query patterns change

For a dataset of 1 billion records with 50 columns averaging 20 bytes each, raw JSON is approximately 1 TB. Parquet with Snappy compression lands around 100-150 GB. Calculate your compression ratio as: uncompressed_size / parquet_size = compression_ratio.

For related reading on how data formats interact with storage systems, see Object Storage and Database Replication.

Conclusion

Data formats are not interchangeable. CSV is simple but fragile. JSON is flexible but expensive to store and query. Parquet gives you columnar efficiency for analytical reads. Avro gives you compact binary encoding with schema evolution. ORC adds Hive-specific features like ACID and indexes.

Choose based on how your pipeline reads data. Row-based formats like Avro and CSV favor write throughput. Columnar formats like Parquet and ORC favor read-heavy analytical queries. The format is not a detail — it determines the performance and cost profile of everything downstream.

Quick Recap

• JSON: APIs and event payloads. Not for analytical workloads — field names bloat storage and slow scans.
• CSV: Universal compatibility. No type safety, no schema, no column pruning. One-time exports only.
• Parquet: Default for data lakes. 5-10x compression over JSON, fast aggregations on column reads.
• Avro: Row-based binary with schema evolution. Kafka and streaming pipelines with changing schemas.
• ORC: Hive ACID transactions, built-in bloom filters. Use when Parquet’s ecosystem support is not enough.
• Match format to read/write pattern — this is the decision that haunts pipelines at scale.