Introduction
Azure data factory (ADF) is a cloud-based data ingestion and ETL (Extract, Transform, Load) tool. The data-driven workflow in ADF orchestrates and automates data movement and data transformation. Azure data factory helps organizations across the globe in making critical business decisions by collecting data from various sources such as e-commerce websites, supply chains, logistics, healthcare, etc., transforming that data into a usable and trusted resource using multiple operations like filtering, concatenation, sorting, etc., and loads that data into a destination store.
This article was published as a part of the Data Science Blogathon.
Basic Level Azure Data Factory Questions
Q1. What is Azure Data Factory?
ADF is a cloud-based data ingestion and ETL (Extract, Transform, Load) Azure service.ADF helps organizations across the globe in making critical business decisions by building complex ETL processes and scheduled event-driven workflows to process data which later can be used by various reporting tools for storytelling purposes.
Q2. How Azure Data Factory Makes the Process of Creating a Data Pipeline Easy?
ADF makes the process of creating a data pipeline easy by providing built-in connectors for data ingestion and orchestration, giving various activity options to perform operations such as copying data, for-each loop, look-up, etc., validating, publishing and monitoring pipelines, continuous integration, and continuous deployment support to the pipelines.
Q3. What are the Different Types of Activities Supported by Azure Data Factory?
Below are the different types of activities supported by ADF:
- Data Movement Activities: Activities used to move data from one data source to another in a data pipeline are known as Data movement activities. For example, copy activity can be used to copy data from ADLS to Azure SQL.
- Data Transformation Activities: Activities used to perform data transformation in a data pipeline are known as Data transformation activities. Data Flow Activity, Azure Functions Activity, Databricks Notebook Activity, etc., are examples of data transformation activities.
- Control Activities: Activities used to build conditional, sequential, or iterative conditional logic in a data pipeline are known as control activities. Lookup Activity, Until Activity, For-Each Activity, etc., are examples of control activities.
Q4. Solve the Project Scenario based on Question 1.
Your data team is building an ETL pipeline for a client. You want to generate output files from Azure Data Factory which are optimized for read-heavy analytical workloads and support the columnar format. What should be the file format of output files? The generated output files should have Parquet format as Parquet stores data in columns and are optimized for read-heavy analytical workloads.
Q5. What are Annotations in Azure Data Factory?
Annotations are additional informative tags that help in filtering and searching data factory resources such as datasets, pipelines, linked services, etc. For example, if you are working as a team lead for a large data processing project for a client ABC that uses ADF containing 10 pipelines. To avoid confusion in the data processing sequence, we can label each pipeline with its primary purpose: ingest, transform, or load using annotations. When we are monitoring pipelines, these annotations must be available to perform searching, grouping, and filtering.
Q6. Why do we need Azure Data Factory?
Azure Data Factory is a cloud-based data integration service that enables you to create, schedule, and manage data pipelines for ingesting, preparing, and transforming data from various sources to various destinations. It’s useful for ETL (Extract, Transform, Load) processes and data movement tasks. Data scientists can use it to move and transform data for analysis.
Q7. What is the integration runtime?
Integration Runtime is the compute infrastructure that Azure Data Factory uses to provide data integration capabilities across different network environments. It enables you to connect to on-premises data sources securely. For example, you can set up a self-hosted integration runtime to connect to your organization’s local database.
Q8. What is the difference between Azure Data Lake and Azure Data Warehouse?
Azure Data Lake Store is a large-scale data lake solution for big data analytics. Azure SQL Data Warehouse is a cloud-based data warehousing service. Data Lake Store is optimized for big data storage and analysis, while SQL Data Warehouse is designed for fast querying and analytical processing.
Q9. What is blob storage in Azure?
Answer: Azure Blob Storage is Microsoft’s object storage solution. It’s used to store and manage unstructured data, such as images, videos, documents, and backups. Data scientists can use Blob Storage to store datasets that they need for analysis.
Q10. What are the steps for creating ETL process in Azure Data Factory?
The ETL process in Azure Data Factory involves creating a pipeline that consists of activities. Activities can be data movement activities or data transformation activities. For example, you can use a Copy Data activity to move data from one storage account to another, and a Data Flow activity to transform and clean the data.
Intermediate Level Interview Questions
Q11. How can I schedule a pipeline?
Pipelines in Azure Data Factory can be scheduled using triggers. You can create time-based triggers that specify when a pipeline should run. For example, you can create a daily trigger to run a pipeline every day at a specific time.
Q12. Can I pass parameters to a pipeline run?
Yes, you can pass parameters to a pipeline run. Parameters allow you to parameterize various elements of a pipeline, such as input datasets, linked services, and activities. This makes pipelines more dynamic and reusable.
Q13. How do I handle null values in an activity output?
In a data transformation activity, you can use functions like coalesce() or ifnull() to handle null values. For example, in SQL-based transformations, you can use the COALESCE(column_name, replacement_value) function to replace null values with a specific value.
Q14. Explain the two levels of security in ADLS Gen2.
ADLS Gen2 (Azure Data Lake Storage Gen2) provides two levels of security: POSIX-like ACLs (Access Control Lists) and Azure Active Directory (Azure AD) integration. POSIX-like ACLs allow fine-grained control over data access. Azure AD integration enables secure authentication and authorization using Azure AD identities.
Advanced Azure Data Factory Questions
Q15. Solve the Project Scenario based on Question 2.
A data science company handles data processing for different clients. Your team is building an ADF pipeline to move user logs generated based on users’ activities on an e-commerce platform from an ADLS container to a database inside Azure Synapse dedicated SQL pool. The user logs are stored in container users in the following folder structure./user/{YYYY}/{MM}/{DD}/{HH}/{mm}
The earliest folder is /user/2021/01/02/00/00. The latest folder is /user/2021/01/17/01/45.
How would you configure the pipeline to trigger so that existing data must be loaded every 30 minutes, and up to two minutes delay in data arrival must be included in the time at which the data should have arrived?We can configure the pipeline to trigger using a tumbling window trigger with Recurrence: 30 minutes, Start time: 2021-01-01T00:00, and Delay: 2 minutes to achieve the above scenario.
The earliest folder is /user/2021/01/02/00/00. The latest folder is /user/2021/01/17/01/45.
How would you configure the pipeline to trigger so that existing data must be loaded every 30 minutes, and up to two minutes delay in data arrival must be included in the time at which the data should have arrived?We can configure the pipeline to trigger using a tumbling window trigger with Recurrence: 30 minutes, Start time: 2021-01-01T00:00, and Delay: 2 minutes to achieve the above scenario.
Q16. How Can Users Secure Their Data Store Credentials in ADF?
Users can secure their data store credentials in ADF by storing them in Azure Key Vault or encrypting them with certificates. Azure Key Vault is an Azure service used to securely store API keys, data store credentials, passwords, etc., to prevent unauthorized access. Developers can easily import or create keys, authorize users to access the key vault, and configure and manage the keys using Azure Key Vault.
Q17. State the Difference Between Pipeline Parameters and Variables in ADF.
Pipeline parameters are created using the “Parameters” tab in the pipeline and cannot be modified while a pipeline is running.
Source: learn.microsoft.com
Pipeline variables can be modified and set using Set variable activity during a pipeline run.
Source: learn.microsoft.com
Q18. Name Some Data Stores and File Formats Supported by Azure Data Factory.
Azure Data Factory supports various data stores such as Azure SQL, Azure Storage, Azure Databricks, HBase, Hive, Impala, MariaDB, Oracle, Cassandra, Amazon S3, MongoDB Atlas, etc. ADF supports various file formats such as Parquet, Avro, JSON, Delta, Excel, XML, Delimited text format, etc.
Q19. Which Activity of Azure Data Factory can be Used to Copy Data From Azure Blob Storage to Azure SQL?
Copy activity can be used in ADF to copy data from Azure Blob Storage to Azure SQL. Copy activity is used to copy data from between different data sources. Copy Activity reads data from the source store, performs column mapping, data compression/decompression based on data type and input and output dataset format, and writes the data into the destination data store.
Source: learn.microsoft.com
Q20. Can I define default values for the pipeline parameters?
Yes, you can define default values for pipeline parameters in Azure Data Factory. When defining a parameter in the pipeline, you can set a default value. This default value will be used if the parameter is not explicitly provided when triggering the pipeline.
Conclusion
Azure Data Factory (ADF) is a cloud-based data ingestion and ETL (Extract, Transform, Load) Azure service. The data-driven workflow in ADF orchestrates and automates data movement and data transformation. ADF helps developers to build complex ETL processes and scheduled event-driven workflows to process data which later can be used by various reporting tools for storytelling purposes. Below are some key points from the above article:
- We have seen how ADF makes the process of creating a data pipeline easy.
- We learned about approaches by which users can secure their data store credentials in ADF.
- We have seen the differences between pipeline parameters and variables in ADF.
- We got an understanding of how we can copy data from Azure Blob Storage to Azure SQL using ADF.
- Apart from this, we also saw some scenario-based questions on ADF.
The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.
Full Stack developer with 2.6 years of experience in developing applications using
Azure, SQL Server & Power BI. I love to read and write blogs. I am always willing to learn new technologies, easily adaptable to changes, and have a good time
management, problem-solving, logical and communication skills
