Trigger Fabric Data Pipeline from ADF/Synapse via Managed Identity

Problem Statement

Is it possible to trigger the Fabric Data Pipeline from Azure Data Factory / Synapse Pipeline via Managed Identity authentication?

Prerequisites

1. Fabric Data Pipeline
2. Azure Data Factory / Synapse Pipeline

Solution

Grant the ADF / Synapse, and give the member access to the workspace hosting the Fabric Data Pipeline.

GitHub Code for sample pipeline.

Step 1. Create 2 Pipeline parameters.

1. WorkspaceId
2. PipelineId

For which below are the values required.

1. WorkspaceId: This represents the Workspace ID hosting the fabric data pipeline.
   
2. PipelineId: This represents the Data Pipeline Id.
   

Step 2. One can leverage web activity to trigger the fabric data pipeline via the fabric REST API through managed identity authentication, as shown below.

URL: @concat(
    'https://api.fabric.microsoft.com/v1/workspaces/', 
    pipeline().parameters.WorkspaceId, 
    '/items/', 
    pipeline().parameters.PipelineId, 
    '/jobs/instances?jobType=Pipeline'
)

Resource: https://api.fabric.microsoft.com/

Step 3. The Trigger API call is asynchronous. Hence, you do not know whether the Fabric data pipeline execution has actually succeeded. The successful execution of the web activity only means that the Fabric data pipeline execution was a success. To check the status of the Fabric data pipeline one can leverage Web activity to trigger fabric data pipeline status via Fabric REST API through Managed Identity Authentication as below.

URL: @concat(
    'https://api.fabric.microsoft.com/v1/workspaces/', 
    pipeline().parameters.WorkspaceId, 
    '/items/', 
    pipeline().parameters.PipelineId, 
    '/jobs/instances/', 
    activity('TriggerFabricDataPipeline').output.id
)

We have to add a polling pattern to periodically check on the status of the execution status until it is complete. We start with an until activity. In the settings of the until loop, we set the expression so that the loop executes until the output of the above web activity is not equal to InProgress. When a Fabric data pipeline is in execution, “InProgress” is the status returned until it completes or fails.

Expression: @not(equals(activity('GetPipelineStatus').output.status, 'InProgress'))

Step 4. The Final activity is the IF activity that checks the execution status and leverages a Fail activity to fail the ADF pipeline in case of Fabric Data pipeline failure.

Expression: @equals(activity('GetPipelineStatus').output.status, 'Failed')

Fail message: @activity('GetPipelineStatus').output.failureReason.message  

Error code: @activity('GetPipelineStatus').output.failureReason.errorCode