(In-progress) Getting started with Azure Data Factory
Image credit: MicrosoftStarting with Azure Data Factory.
In this blog I will focus on some fundamentals about Azure Data Factory:
- Definitions - What is Azure Data Factory.
- Where to start - How to start and pre-requisites.
- Connectivity - Some basics about connectivity
📖 Table of Contents
What is Azure Data Factory?
Azure Data Factory is a fully managed, cloud-based data integration service that helps you move, transform, and orchestrate data workflows at scale—without the headache of managing infrastructure. Get the full breakdown here: Azure Data Factory Overview
Licensing
In order to start with Azure Data Factory, licensing should be configured properly.
About the Frabric Trial pre-requisites
The Fabric trial requires a per-user Power BI license. So in order to login to App Fabric and use the Fabric Trial; You should assign the (trial) licenses for:
- Microsoft Fabric (Free)
- Power BI Premium Per User
How to purchase and assing these (trial) licenses is explained here: Puchase licenses or trials
You can also view the assigned licenses in the Azure Portal
Azure Data Factory Managed Virtual Network & Private Endpoints
Private endpoints in Azure Data Factory’s Managed Virtual Network (Managed VNet) function differently from regular private endpoints in Azure. Let’s break this down in a clear and structured way.
Azure Data Factory (ADF) offers a Managed Virtual Network (Managed VNet) to enhance security by fully isolating data integration processes from public internet access. This is useful for enterprises that need strict network security and want to ensure their data never leaves a controlled network environment.
Within this Managed VNet, you have the option to use Private Endpoints, which serve a different role compared to standard Azure Private Endpoints.
Managed VNet Architecture Diagram
Key Differences: ADF Managed Private Endpoints vs. Regular Azure Private Endpoints
✅ While using the ADF Managed Private Endpoint you can still use the Regular Azure Private Endpoint for connections outside of ADF. For example: When connecting to Azure SQL from ADF via the Managed Private Endpoint, you can still have applications from within Azure that use the same Azure SQL instanace.
| Feature | ADF Managed Private Endpoint | Regular Azure Private Endpoint |
|---|---|---|
| Scope of Use | Only within ADF’s Managed VNet | Can be used across Azure resources |
| Management | Fully managed by ADF, no need to set up a VNet manually | Requires manual setup within your own Azure VNet |
| Connectivity | Used only for connecting ADF to private resources (e.g., Azure SQL, Storage Accounts) within the same region | Can connect different services across VNets and regions |
| Security | ADF ensures private connections without needing extra firewall rules or public internet access | You manage firewall rules and NSG settings manually |
| Setup Process | Created within ADF UI under “Managed Private Endpoints” | Created in Azure Portal, PowerShell, or CLI under “Private Link” |
| Peering | No need to configure peering; ADF manages it internally | May require VNet peering between different services |
How ADF Managed Private Endpoints Work
ADF Managed VNet Activation
- When you enable the Managed Virtual Network in ADF, data movement and pipeline execution occur within a fully isolated network controlled by ADF.
Creating Managed Private Endpoints
- Inside this Managed VNet, ADF allows you to create “Managed Private Endpoints” to securely connect to Azure resources (like Azure SQL, Azure Storage, or Synapse).
Approval Process
- Unlike regular Private Endpoints, these endpoints need explicit approval from the target resource owner before they can be used.
Private Traffic Flow
- Once approved, all traffic between ADF and the connected resource flows privately, without exposure to the public internet.
Why Use ADF Managed Private Endpoints Instead of Regular Private Endpoints?
✅ Simplified Security: No need to configure VNets, firewall rules, or public access exceptions.
✅ Full Isolation: Ensures all data movement happens inside Azure’s backbone network, preventing exposure to public internet.
✅ Easier to Manage: No manual VNet peering or private DNS configurations—ADF takes care of it.
✅ Better Compliance: Helps meet regulatory requirements by keeping all data within a controlled network.
Step-by-step guide on setting up an ADF Managed Private Endpoint? 🚀
Resources
- Azure Data Factory resources and documentation
- Azure Data Factory documentation
- Workspaces in Fabric and Power BI
- What is Microsoft Fabric
- Data Factory Documentation Index | Microsoft Docs
- Azure Data Factory managed virtual network
- Quickstart: Get started with Azure Data Factory
- Manage network policies for private endpoints
