Parquet vs ORC vs Avro: Pick by Access Pattern, Not by Benchmark

Format arguments usually turn into a benchmark on someone’s laptop, and the results are useless because the format is not the variable that matters — the access pattern is. Avro losing a column scan benchmark to Parquet is not a finding. It is the design working correctly.

Here is how I actually choose, and the layout mistakes that cost far more than the format decision ever will.

Row versus column, which is the entire story

flowchart TD
  subgraph "Avro (row)"
    a1["row 1: id, name, amount, ts"]
    a2["row 2: id, name, amount, ts"]
    a3["row 3: id, name, amount, ts"]
  end
  subgraph "Parquet / ORC (columnar)"
    c1["id: 1,2,3…"]
    c2["name: a,b,c…"]
    c3["amount: 10,20,30…"]
    c4["ts: …"]
  end

Reading two columns of a 60-column table means reading two chunks, or all of every row.

Avro stores complete records sequentially. To read one column you deserialize the whole row. In exchange you get cheap appends, cheap whole-record reads, and a schema travelling with the data.

Parquet and ORC store values column by column, in blocks, with per-block statistics. Reading SELECT amount FROM orders WHERE order_date = '2026-08-01' touches one column chunk and skips blocks whose min/max cannot match. Similar values sit next to each other, so compression ratios are dramatically better than row storage.

That is the whole trade. Everything else is detail.

Where each one belongs

Avro: transport and landing. Kafka messages with a Schema Registry, the raw bronze landing zone where you write whatever arrived, and any place you append records one at a time. Avro’s schema evolution rules are the best of the three — named fields with defaults, real forward and backward compatibility — which is exactly what you want for messages produced by services you do not control.

Parquet: everything analytical. Silver and gold layers, anything a query engine touches, anything stored in Iceberg or Delta. Spark, Trino, DuckDB, Snowflake external tables, BigQuery external tables, Athena, polars, and pandas all read it natively and well. When there is no strong reason to do otherwise, this is the answer.

ORC: the Hive and Trino-on-Hive world. ORC has built-in lightweight indexes (row-group level min/max plus optional bloom filters) and is the format Hive ACID tables use. If you are operating a Hive metastore estate, ORC is often already the standard and switching gains you little. Trino reads both well. On a Hadoop-to-Iceberg exit, format conversion is usually the least interesting part of the migration.

The practical summary: unless you are in a Hive-native estate, use Parquet for storage and Avro for transport, and spend your attention on layout instead.

Layout beats format, every time

A well-partitioned, badly-sized Parquet dataset performs worse than a mediocre format with sane files.

Row group size. Parquet stores statistics per row group. The default target is around 128 MB. If your writer produces 4 MB files, each file is one small row group, min/max stats cover almost nothing, and the engine reads everything. This is the most common “Parquet is slow” complaint I get asked about, and it is a writer configuration problem.

File count. Ten thousand 2 MB files means ten thousand object store requests, ten thousand footer reads, and a query plan that spends more time listing than scanning. Target roughly 128 MB to 1 GB per file. Compaction is a scheduled job, not an optional optimization — the Hudi post makes the same argument for merge-on-read tables.

Column ordering and sorting. Sorting by your most common filter column before writing makes min/max statistics narrow and selective. Sorting by a high-cardinality random column makes every row group’s range cover everything, and pruning stops working. This is the same principle as Snowflake micro-partition pruning and ClickHouse’s ORDER BY, just in files you control.

# Spark: the three settings that decide whether pruning works.
(
    df.repartition("order_date")           # one date per file, not scattered
      .sortWithinPartitions("customer_id") # narrow min/max on the filter column
      .write
      .option("parquet.block.size", 256 * 1024 * 1024)
      .option("compression", "zstd")
      .partitionBy("order_date")
      .mode("overwrite")
      .parquet("s3://acme/silver/orders/")
)

Note what is not in there: nested partition directories by year/month/day/hour. Over-partitioning is how you produce the small-file problem in the first place. Partition by the column you filter on, at the granularity that keeps files above ~128 MB.

Compression, briefly

Snappy is fast to decode and compresses moderately. It is the historical Parquet default and a reasonable choice when queries are CPU-bound on decompression.

ZSTD compresses noticeably better at comparable read speed on modern hardware, and is now my default for anything stored longer than a week. Smaller files mean less object storage, less network transfer, and less scanned data on engines that bill by bytes.

GZIP compresses well and decodes slowly. Use it when storage cost dominates and query latency does not.

If you bill by bytes scanned — BigQuery and Athena — compression directly reduces the invoice, so the default should not be Snappy out of habit.

Schema evolution differs more than people expect

Avro resolves schemas by field name with defaults, and the Schema Registry enforces compatibility modes before a bad producer ships. This is the strongest story of the three and the reason it survives as a transport format.

Parquet evolution depends on the reader. Adding a nullable column is safe. Renaming is a new column plus a dropped one unless your table format tracks field IDs — which is exactly what Iceberg does, and why Iceberg-on-Parquet handles renames that raw Parquet directories cannot.

ORC is similar to Parquet, with Hive’s own rules layered on top.

The lesson: schema evolution is mostly a property of the table format (Iceberg, Delta, Hudi) rather than the file format. If evolution is your concern, that is the layer to fix. The open table formats sheet lays out which guarantees come from where.

Pitfalls

Storing analytics data as Avro because ingestion produced it. Landing in Avro is fine. Leaving silver in Avro means every query reads every column forever.

Over-partitioning into hourly directories. You will produce thousands of tiny files and destroy both listing performance and row-group statistics. Partition coarsely; sort within.

Writing Parquet from a streaming job without compaction. A micro-batch every minute writes a file every minute. Schedule compaction, or use a table format that does it for you.

Assuming compression makes small files fine. Compression does not reduce request count or footer overhead. File count is a separate problem with a separate fix.

Relying on SELECT * and then blaming the format. Columnar storage only pays off if you project columns. SELECT * on Parquet is Avro with extra steps.

Nested structures everywhere. Parquet handles nesting well, but deeply nested arrays exploded at query time are a modeling problem — flatten in silver, as in MongoDB to silver.

FAQ

Is Parquet always better than ORC? No, but it is the safer default outside Hive. ORC’s advantages are real inside the ecosystem built around it; outside, Parquet’s universal engine support matters more than any per-query delta.

Should I use Avro in the bronze layer? It is a good fit if you are landing Kafka messages and want the schema alongside the data. JSON is also common and much worse for storage cost. Either way, convert to Parquet before silver.

What about Arrow — is that a fourth option? Arrow is an in-memory format, not a storage format. Parquet on disk, Arrow in memory, and the two are designed to work together. Storing Arrow IPC files on disk is a niche choice for short-lived intermediates.

Does the format still matter if I use Iceberg or Delta? Less than you would think, because both default to Parquet and handle layout concerns for you. What still matters is target file size and sort order, which you configure at the table level.

How do I find my small-file problem? Count files and average size per partition. If the average is under 32 MB, you have one. Most engines expose this in table statistics, and object store inventory reports work for the rest.

What this means for data engineers

Use Avro where records travel one at a time and schemas change without your permission. Use Parquet for everything you query, and let a table format handle evolution and compaction on top of it. Use ORC when you are already standardized on it in a Hive estate, and do not spend a sprint converting.

Then stop thinking about formats and go measure your average file size. On every slow lake I have been handed, the problem was thousands of tiny files and a sort order that made statistics useless — never the four letters at the end of the filename.