Prerequisites
- Azure Subscription: You need an Azure subscription to use Azure Data Factory.
- Azure Data Factory: An existing Azure Data Factory with at least one pipeline.
- Service Principal: A service principal or a user account with necessary permissions to interact with Azure Data Factory via REST API.
- Tools:
curlfor making HTTP requests andjqfor processing JSON data (or equivalent tools).
Step 1: Obtain Azure REST API Endpoint
To execute an ADF pipeline, you'll need the appropriate REST API endpoint. Here's the endpoint format to trigger a pipeline run:
POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelines/{pipelineName}/createRun?api-version=2018-06-01
Replace {subscriptionId}, {resourceGroupName}, {factoryName}, and {pipelineName} with your subscription ID, resource group name, factory name, and pipeline name, respectively.
Step 2: Obtain an Azure Access Token
To make authorized API requests, you need an OAuth 2.0 access token from Azure Active Directory (AAD). This requires a service principal with appropriate permissions. Here's how you can get an access token:
// Replace these with your actual values private static readonly string TenantId = "your-tenant-id"; private static readonly string ClientId = "your-client-id"; private static readonly string ClientSecret = "your-client-secret"; private static readonly string SubscriptionId = "your-subscription-id"; private static readonly string ResourceGroupName = "your-resource-group-name"; private static readonly string FactoryName = "your-factory-name"; private static readonly string PipelineName = "your-pipeline-name";
Get Access Token
public static async Task<string> GetAccessTokenAsync()
{
using (var client = new HttpClient())
{
var requestContent = new StringContent(
$"resource=https://management.azure.com/&client_id={ClientId}&client_secret={ClientSecret}&grant_type=client_credentials",
Encoding.UTF8, "application/x-www-form-urlencoded"
);
var response = await client.PostAsync($"https://login.microsoftonline.com/{TenantId}/oauth2/token", requestContent);
var responseContent = await response.Content.ReadAsStringAsync();
if (!response.IsSuccessStatusCode)
{
throw new Exception($"Error getting access token: {responseContent}");
}
var json = JObject.Parse(responseContent);
return json["access_token"].ToString();
}
}
Call ADF Pipeline via Rest API:
public static async Task TriggerPipelineAsync(string accessToken)
{
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", $"Bearer {accessToken}");
var endpoint = $"https://management.azure.com/subscriptions/{SubscriptionId}/resourceGroups/{ResourceGroupName}/providers/Microsoft.DataFactory/factories/{FactoryName}/pipelines/{PipelineName}/createRun?api-version=2018-06-01";
var response = await client.PostAsync(endpoint, null);
var responseContent = await response.Content.ReadAsStringAsync();
if (!response.IsSuccessStatusCode)
{
throw new Exception($"Error triggering pipeline: {responseContent}");
}
Console.WriteLine("Pipeline triggered successfully.");
}
}
Main Method:
public static async Task Main(string[] args)
{
try
{
var accessToken = await GetAccessTokenAsync();
await TriggerPipelineAsync(accessToken);
}
catch (Exception ex)
{
Console.WriteLine($"An error occurred: {ex.Message}");
}
}
Explanation
- GetAccessTokenAsync: This function retrieves an OAuth 2.0 access token using the Service Principal credentials (tenant ID, client ID, and client secret).
- TriggerPipelineAsync: This function sends a POST request to the Azure Data Factory REST API endpoint to trigger the pipeline.
- Error Handling: The code includes basic error handling to catch and report any issues during authentication or pipeline execution.
Thank you.
0 comments:
Post a Comment