#061 How to read data in JSON format in Azure Synapse Analytics

Create pipeline with data

• In Azure Synapse Studio, create a new pipeline and add a data copy activity.

JSON):

• In the pipeline, add an HTTP data source and configure to read data in JSON format.

from IBGE website:

• Go to IBGE website (https://servicodados.ibge.gov.br/api/docs) to get the URL of the data to be copied.

(IMPORT SCHEMA):

• Use the copy data activity to import the schema from the source data.

data (DATALAKE, PARQUET):

• Add a target step in the pipeline by configuring the destination as a Data Lake and choosing the Parquet format.

Some reasons to use the Parquet format:

• The Parquet format is storage-efficient and offers compression, schema support, and efficient reading for analytical processing.

(TRIGGER):

• Set up and run a trigger to start the pipeline and monitor the status of the run.

(RUN CONSUMPTION):

• Analyze the costs associated with running the pipeline using Azure monitoring and consumption tools.

PARQUET format file:

• If necessary, generate a script to read data from the Parquet file in the desired environment (for example, using Apache Spark).

(MAPPING):

• In the pipeline, map the columns from the source data to the target columns in Parquet format in the Data Lake.