List of Activities in ADF
Azure Data Factory Azure Data Factory is a cloud-based data integration service that allows you to create, schedule, and manage data pipelines to move and transform data from various sources to various destinations. Here are some of the different types of activities in Azure Data Factory with examples:
1. Copy Activity:- The Copy activity is used to copy data from one data store to another. For example, you can use the Copy activity to copy data from an on-premises SQL Server database to an Azure SQL database.
2. Execute Pipeline Activity:- The Execute Pipeline activity is used to call another pipeline from within the current pipeline. For example, you can use this activity to execute a pipeline that contains a data transformation activity after the data has been copied.
3. Web Activity:- The Web activity is used to call a REST API endpoint or a web service. For example, you can use this activity to call an API to retrieve data from an external system.
4. Stored Procedure Activity:- The Stored Procedure activity is used to call a stored procedure in a SQL Server database. For example, you can use this activity to execute a stored procedure that performs a data transformation.
5. If Condition Activity:- The If Condition activity is used to create a conditional workflow in your pipeline. For example, you can use this activity to check if a file exists in a data store and only continue with the pipeline if the file is found.
6. For Each Activity:- The For Each activity is used to iterate over a collection of items and perform an action on each item. For example, you can use this activity to loop through a list of files and copy each file to a destination.
7. Lookup Activity:- The Lookup activity is used to retrieve metadata or a single value from a data store. For example, you can use this activity to get the schema of a table in a SQL Server database.
8. Set Variable Activity:- The Set Variable activity is used to set the value of a variable in a pipeline. For example, you can use this activity to set a variable that holds the current date or time.
9. Wait Activity: The Wait activity is used to pause the execution of a pipeline for a specified period of time. For example, you can use this activity to wait for a specific time to start a data transfer operation.
10.Filter Activity:- The Filter activity is used to filter data based on a specified condition. For example, you can use this activity to filter data based on a specific column value before transferring the data to a destination.
11.Join Activity:- The Join activity is used to join data from two or more sources. For example, you can use this activity to join data from two tables in a SQL Server database.
12.Union Activity:- The Union activity is used to combine data from two or more sources. For example, you can use this activity to combine data from two tables in a SQL Server database into a single destination.
13.Lookup Activity:- The Lookup activity is used to retrieve metadata or a single value from a data store. For example, you can use this activity to get the schema of a table in a SQL Server database.
14.Set Variable Activity:- The Set Variable activity is used to set the value of a variable in a pipeline. For example, you can use this activity to set a variable that holds the current date or time.
15.If Condition Activity:- The If Condition activity is used to create a conditional workflow in your pipeline. For example, you can use this activity to check if a file exists in a data store and only continue with the pipeline if the file is found.
16.Until Activity:- The Until activity is used to execute a loop until a specific condition is met. For example, you can use this activity to keep copying data until a specific file is found in a data store.
17.Mapping Data Flow Activity: The Mapping Data Flow activity is used to visually design and build data transformation logic using a drag-and-drop interface. For example, you can use this activity to transform data from one format to another, or to combine data from multiple sources.
18.Databricks Notebook Activity: The Databricks Notebook activity is used to run a Databricks notebook in a Databricks workspace. For example, you can use this activity to run a Python or Scala script to transform data.
19.HDInsight Hive Activity: The HDInsight Hive activity is used to execute Hive queries on an HDInsight cluster. For example, you can use this activity to transform data using HiveQL.
20.HDInsight Pig Activity: The HDInsight Pig activity is used to execute Pig scripts on an HDInsight cluster. For example, you can use this activity to transform data using Pig Latin.
21.HDInsight MapReduce Activity: The HDInsight MapReduce activity is used to execute MapReduce jobs on an HDInsight cluster. For example, you can use this activity to perform complex data transformations on large datasets.
22.Custom Activity: The Custom activity is used to run custom code in a data pipeline. For example, you can use this activity to run a PowerShell script to perform a specific task.
23.Execute SSIS Package Activity: The Execute SSIS Package activity is used to execute an SSIS package stored in an Azure Storage account or a SQL Server Integration Services (SSIS) catalog. For example, you can use this activity to perform data transformations using an existing SSIS package.
24.Delete Activity: The Delete activity is used to delete data from a data store. For example, you can use this activity to delete files from an Azure Blob Storage container.
25.Teradata Query Activity: The Teradata Query activity is used to execute queries on a Teradata database. For example, you can use this activity to extract data from a Teradata database.
26.Amazon S3 Storage Activity: The Amazon S3 Storage activity is used to copy data between an Amazon S3 storage account and an Azure Data Factory-supported data store. For example, you can use this activity to transfer data between an Amazon S3 storage account and an Azure Blob Storage account.
27.Azure Function Activity: The Azure Function activity is used to execute an Azure Function in a pipeline. For example, you can use this activity to perform custom data transformations using an Azure Function.
28.Wait Event Activity: The Wait Event activity is used to pause the execution of a pipeline until a specific event occurs. For example, you can use this activity to wait for a signal from an external system before proceeding with the pipeline.
29.Amazon Redshift Query Activity: The Amazon Redshift Query activity is used to execute queries on an Amazon Redshift database. For example, you can use this activity to extract data from an Amazon Redshift database.
30.Web Activity: The Web activity is used to call a REST API or a web endpoint from a pipeline. For example, you can use this activity to call an API to retrieve data or to perform an action.
31.Azure Analysis Services Activity: The Azure Analysis Services activity is used to execute a command or a query against an Azure Analysis Services database. For example, you can use this activity to refresh a cube in an Azure Analysis Services database.
32.SharePoint Online List Activity: The SharePoint Online List activity is used to copy data between a SharePoint Online list and an Azure Data Factory- supported data store. For example, you can use this activity to transfer data between a SharePoint Online list and an Azure SQL Database.
33.Stored Procedure Activity: The Stored Procedure activity is used to execute a stored procedure in a database. For example, you can use this activity to perform a custom data transformation using a stored procedure.
34.Lookup with a Stored Procedure Activity: The Lookup with a Stored Procedure activity is used to retrieve data from a database using a stored procedure. For example, you can use this activity to retrieve data from a SQL Server database using a stored procedure.
35.Copy Activity: The Copy activity is used to copy data between different data stores. For example, you can use this activity to copy data from an on-premises SQL Server database to an Azure Blob Storage container.
36.IF Condition Activity: The IF Condition activity is used to evaluate a Boolean expression and perform different actions based on the result. For example, you can use this activity to perform different data transformations based on a condition.
37.For Each Activity: The For Each activity is used to loop through a set of items and perform an action for each item. For example, you can use this activity to process a set of files stored in an Azure Blob Storage container.
38.Until Activity: The Until activity is used to repeatedly perform an action until a certain condition is met. For example, you can use this activity to keep polling a system until a certain status is returned.
39.Filter Activity: The Filter activity is used to filter data based on a condition. For example, you can use this activity to filter out data that does not meet certain criteria.
40.Set Variable Activity: The Set Variable activity is used to set the value of a variable that can be used in later activities. For example, you can use this activity to set a variable to the current date and time.
41.Azure Databricks Notebook Activity: The Azure Databricks Notebook activity is used to execute a Databricks notebook in a pipeline. For example, you can use this activity to perform advanced data processing and analytics using Databricks.
42.Lookup Activity: The Lookup activity is used to retrieve data from a data store. For example, you can use this activity to retrieve metadata from a file stored in Azure Blob Storage.
43.Wait Activity: The Wait activity is used to pause the execution of a pipeline for a specified amount of time. For example, you can use this activity to introduce a delay between two activities in a pipeline.
44.If Condition Branch Activity: The If Condition Branch activity is used to define the action that should be taken based on the result of an If Condition activity. For example, you can use this activity to perform different data transformations based on the result of the if Condition activity.
45.Get Metadata Activity: The Get Metadata activity is used to retrieve metadata about a file or folder stored in a data store. For example, you can use this activity to retrieve the size, type, and last modified date of a file stored in Azure Blob Storage.
46.Union Activity: The Union activity is used to combine the results of two or more data sources. For example, you can use this activity to combine the results of two different SQL queries into a single data set
No comments:
Post a Comment