File formats

Question

You are planning the deployment of Azure Data Lake Storage Gen2.
You have the following two reports that will access the data lake:
? Report1: Reads three columns from a file that contains 50 columns.
? Report2: Queries a single record based on a timestamp.
You need to recommend in which format to store the data in the data lake to support the reports. The solution must minimize read times.
What should you recommend for each report? and why?

Avro or parquet or csv or tsv

Eliana Blake · Answer

For Report1, where only three columns out of 50 need to be read, the recommended format would be Parquet. Parquet is a columnar storage format that is highly efficient for read-heavy workloads. Since Parquet organizes data by columns, it allows for better compression and minimizes the data that needs to be read from disk, resulting in faster query performance specifically when only a subset of columns is required to be retrieved. For Report2, which involves querying a single record based on a timestamp, Avro would be a suitable choice. Avro is a row-based format that provides efficient serialization and deserialization of data. It works well for querying specific records based on criteria like timestamps because it stores data along with its schema, making it easier to retrieve individual records without having to process the entire dataset. By storing data in Parquet for Report1 and Avro for Report2, you can optimize read times based on the specific requirements of each report, ensuring efficient data access in Azure Data Lake Storage Gen2.