Unlocking Cross-Tenant Magic: Azure Data Factory and Private Link

In the complex landscape of multi-tenant cloud environments, the need for seamless data connectivity across Azure tenants has become a critical challenge. Organizations must securely share data between their SaaS platforms and customer environments while maintaining the highest levels of security, privacy and efficiency.

Imagine your SaaS platform residing in one Azure tenant, while your customers operate in their own isolated domains. They rely on Azure-native tools, logic apps, and data pipelines to receive critical data. How do you bridge this gap? How can you ensure data flows smoothly while adhering to security best practices?

Our components

We’ll approach this challenge in a couple of steps. First, we’ll set up an Azure Data Factory (ADF) that will function as our nexus of data movement. ADF orchestrates data movement across various sources and destinations. Its components—pipelines, datasets, and triggers—form the backbone of cross-tenant data flows.

But data movement isn’t just about pipes and valves; it’s about secure pathways. Azure Private Link steps onto the stage as our guardian. It provides a shield against the public internet, allowing data to flow privately through dedicated connections. We rely on the usage of Private Endpoints, managed on ADF side, to connect the dots. This automatically creates a Private DNS Zone in our designated subscription, containing one record for every endpoint we connect to.

For permissions, we will leverage our trusty Azure Key vault to store, retrieve and audit the usage of secrets related to our cross-tenant magic.

Data storage is up next, we will use Azure Storage Accounts as source and destination storage. We assume our third party will provide us with a service principal that has Blob Data Contributor rights on the source storage.

Triggering the data movement should be our next point of concern. We’ll leverage Azure Logic Apps combined with HTTPS WebHooks to trigger our pipelines.

Design

For simplicity sake, let’s assume we’re extracting some form of data from a third party tenant (pictured on the right hand side as source storage absthirdparty001). To do this securely, our third party creates a service principal with a client id and secret that has Blob Data Contributor (mi-blobdatacontributor-001) rights on the source storage.

We’ll want to transform it in some way and place it into our own storage on our tenant (pictured on the left hand side as absmidpcloud001). We’ll leverage adfmidpcloud001 as our orchestrator for the data movement, create a managed private endpoint towards the third party storage and leverage Azure backbone to connect the dots (private link). Of course, we’ll want to not store anything in ADF itself, therefore leverage akvmidpcloud001 as the vault where we store clientid/secret of the target service principal.

We can set this pipeline to be triggered on a schedule, which we’ll do regardless, but also want it to trigger using a logic app lamidpcloud001. For this purpose, third party provider can create an event grid on the source storage account towards our logic app by publishing BlobCreated events towards HTTPS endpoint.

ADF – Creating the link towards own storage

We assume we’ve got all components ready, now we simply want to connect the dots from our design. From ADF Studio, let’s start by connecting our own storage. Simple, same tenant, follow the steps:

Create a new linked service, select blob

We love legacy, let’s go with XML in this example

Create a new linked service, give it a proper name

Start with creating a new integration runtime

Under VNet, enable configuration and click Create

I chose to use a system assigned MI to authenticate to absmidpcloud001, make sure you take note of object ID if you want to do the same since we’ll need to give it rights in a bit. Click Create.

Now that we’ve done the basic configuration, you’ll see the ADF wants to publish these changes before continuing. However, we still require the proper rights for our MI on the destination storage. Therefore, assign blob data contributor to the MI under access control on that storage account.

For simplicity sake, I’ve configured two containers: inbound and outbound. Under Outbound, I’ve placed a empty.xml file which contains some data we want to send to our third party in an initial test.

Heading back to our Data Factory, I’ve configured the empty.xml as file path. Make sure to select Publish All.

Publish all on top and Publish once more to confirm.

Of course, this is just to test connectivity to our own storage. We want to connect to third party storage and copy data first.

Configure KeyVault

We’re assuming our third party storage has an Azure storage account set up, with an app registration (service principal) that allows us to authenticate over OAuth2.0 towards the storage account. This service principal also has blob data contributor on the third party storage.

We need several parameters before starting:

third party storage account resource id
URI for requesting access token
client id
client secret
scope (always https://storage.azure.com/.default)

Let’s head to our KeyVault first, to enter the client secret required for accessing our source storage. Do note that if you get a permissions error, you assign yourself Key Vault Secrets Officer role to manage the secrets of this key vault.

Under Objects – Secrets, Generate a new secret:

Make sure you give the Secret a valid name and paste the value as well:

Give the secret a name and paste the value under Secret value, click Create

Once you’ve created the secret, it’s important to also make sure our data factory’s system assigned managed identity can access the key vault:

Access Control (IAM) – Add – Add role assignment

Head to Members, Managed Identity and Select members

Select your subscription, system-assigned MIs and your ADF, click Continue

We should get a notification whether this operation was successful afterwards!

Configure Key Vault on ADF

Next up, we need to link the Key Vault as a service on our ADF. Head back to the Data Factory and under Linked Services, create a Key Vault link:

Enter the name of your Key Vault, choose your Key vault after selecting your subscription, make sure you authenticate using the ADF’s MI and Test connection to linked service. Click Create

Once more, we have to publish all changes! Publish all -> Publish.

ADF – Creating the link towards third party storage

We’ve linked our own storage, we’ve created the required secrets and roles on them, now we need to link towards our third party storage existing in another tenant.

Once more, create a new linked service on our ADF. Almost all the same steps as we did for our own storage, with the difference that we’re going to create a Private Link first:

After the usual steps, assign a name, choose our created integratino runtime and enter the storage account details of our third party. Make sure you Create new for Managed private endpoint.

Give a logical name and enter the Resource ID under Target resource ID. Click Create.

You’ll notice the PE is created:

However, the state is stuck on Provisioning:

This is normal, we cannot approve private link requests for our third party storage as it’s managed in a different tenant. Logical, right? Ask your third party to approve your private link request. Here’s a rundown of what it looks like on their side:

The third party will see a pending connection under their networking – private endpoint connections. Once they approve, it goes to Approved state:

Now, back to our ADF, if we Refresh next to Provisioning (or Pending):

We still need to configure our authentication towards the third party storage:

I’m doing this Inline, you could also create a separate Credential. Enter the third party Tenant and Client ID, choose your AKV linked service for the secret we created earlier.

Click Create, Publish All – Publish. Easy does it!

ADF – Pipeline time

We’ve successfully created everything we need to copy over data from our third party. Create a new pipeline.

Create a Copy data activity, head to Source

Select your third party storage and the csv file you want to copy over. Either input manually or Browse.

Once this is done, we actually already know everything towards our third party is working correctly. Add a Sink next to dump it over to our own storage:

I’m not going to run through all the steps, but let’s assume we want to copy over to our inbound container into a delimited text file (csv format). Same steps as previously apply, only you select our own storage now.

We’ll see a nice green icon once we’ve configured this all:

Validate – Publish All – Publish. The usual.

ADF – Trigger once

Let’s test by manually adding a trigger:

After about a minute or so, we’ll see our trigger ran successfully:

If we validate in our inbound container, we notice a new file present:

Congratulations! You’ve copied over the file.

Automation using Logic Apps

Great! Our engineer can copy over data, manually, with a lot of effort. Now, to be honest, we don’t want to run this manually, right? Indeed. Let’s review our initial design:

That’s right, we still need to create our lamidpcloud001 to ensure files are copied over whenever a blob is created!

I’ll give you a very high-level overview of how to do this:

Create your Logic App with a HTTPS Webhook trigger
Create an ADF pipeline similar to the one we created, that copies over all data in their outbound container to our container
In the Logic App, specify to run the pipeline whenever our webhook is triggered
Ask your third party to create an event trigger on their storage
- On Blob Created, call our HTTPS Webhook URL

Et voila! We’ve automated our entire flow. Whenever our third party uploads a blob inside the container, our logic app is notified using a HTTPS webhook. This then triggers our pipeline to run and copy over the data from thid party to our own storage.

TL;DR

Hope you learned something today! In conclusion, we’ve unlocked the cross-tenant magic by seamlessly integrating Azure Data Factory, Private Link, Logic App, Key Vault, and Azure Storage Accounts. This powerful combination has enabled us to securely and efficiently manage data across different tenants, paving the way for enhanced collaboration and innovation.

At the time of publishing this post, I’ve already removed all the resources. Feel free to re-use the names ;-)!

As always, any questions -> [email protected]. Happy to help!