Updates from: 08/09/2021 03:05:20
Service Microsoft Docs article Related commit history on GitHub Change details
active-directory Concept Identity Protection Risks https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/identity-protection/concept-identity-protection-risks.md
These risks can be calculated in real-time or calculated offline using Microsoft
| | | | | Anonymous IP address | Real-time | This risk detection type indicates sign-ins from an anonymous IP address (for example, Tor browser or anonymous VPN). These IP addresses are typically used by actors who want to hide their login telemetry (IP address, location, device, etc.) for potentially malicious intent. | | Atypical travel | Offline | This risk detection type identifies two sign-ins originating from geographically distant locations, where at least one of the locations may also be atypical for the user, given past behavior. Among several other factors, this machine learning algorithm takes into account the time between the two sign-ins and the time it would have taken for the user to travel from the first location to the second, indicating that a different user is using the same credentials. <br><br> The algorithm ignores obvious "false positives" contributing to the impossible travel conditions, such as VPNs and locations regularly used by other users in the organization. The system has an initial learning period of the earliest of 14 days or 10 logins, during which it learns a new user's sign-in behavior. |
-| Anomalous Token| Offline | This detection indicates that there are abnormal characteristics in the token such as time active and authentication from unfamiliar IP address. |
+| Anomalous Token | Offline | This detection indicates that there are abnormal characteristics in the token such as time active and authentication from unfamiliar IP address. |
+| Token Issuer Anomaly | Offline | This risk detection indicates there is unusual activity with known attack patterns, such as updating trusted realm federation settings or changing a signing certificate. |
| Malware linked IP address | Offline | This risk detection type indicates sign-ins from IP addresses infected with malware that is known to actively communicate with a bot server. This detection is determined by correlating IP addresses of the user's device against IP addresses that were in contact with a bot server while the bot server was active. | | Suspicious browser | Offline | Suspicious browser detection indicates anomalous behavior based on suspicious sign-in activity across multiple tenants from different countries in the same browser. | | Unfamiliar sign-in properties | Real-time | This risk detection type considers past sign-in history (IP, Latitude / Longitude and ASN) to look for anomalous sign-ins. The system stores information about previous locations used by a user, and considers these "familiar" locations. The risk detection is triggered when the sign-in occurs from a location that's not already in the list of familiar locations. Newly created users will be in "learning mode" for a period of time in which unfamiliar sign-in properties risk detections will be turned off while our algorithms learn the user's behavior. The learning mode duration is dynamic and depends on how much time it takes the algorithm to gather enough information about the user's sign-in patterns. The minimum duration is five days. A user can go back into learning mode after a long period of inactivity. The system also ignores sign-ins from familiar devices, and locations that are geographically close to a familiar location. <br><br> We also run this detection for basic authentication (or legacy protocols). Because these protocols do not have modern properties such as client ID, there is limited telemetry to reduce false positives. We recommend our customers to move to modern authentication. |
active-directory Admin Units Add Manage Groups https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/admin-units-add-manage-groups.md
You can assign only individual groups to an administrative unit. There is no opt
* From the **Groups** pane:
- 1. In the Azure portal, go to **Azure AD**.
- 1. Select **Groups**, and then select the group that you want to assign to the administrative unit.
+ 1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+ 1. Select **Azure Active Directory** > **Groups**.
+
+ 1. Select the group that you want to assign to an administrative unit.
+ 1. On the left pane, select **Administrative units** to display a list of the administrative units that the group is assigned to. ![Screenshot of the "Assign to administrative unit" link on the "Administrative units" pane.](./media/admin-units-add-manage-groups/assign-to-group-1.png) 1. Select **Assign to administrative unit**.+ 1. On the right pane, select the administrative unit.
-* From the **Administrative units** > **All Groups** pane:
+* From the **Administrative units** > **Groups** pane:
- 1. In the Azure portal, go to **Azure AD**.
+ 1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+ 1. Select **Azure Active Directory** > **Administrative units**.
+
+ 1. Select an administrative unit that you want to add a group to.
- 1. On the left pane, select **Administrative units**, and then select **All Groups**.
- Any groups that are already assigned to the administrative unit are displayed on the right pane.
+ 1. Select **Groups**. Any groups that are already assigned to the administrative unit are displayed on the right pane.
1. On the **Groups** pane, select **Add**. The right pane lists all available groups in your Azure AD organization.
active-directory Admin Units Add Manage Users https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/admin-units-add-manage-users.md
You can assign users to administrative units individually or as a bulk operation
- Assign individual users from a user profile:
- 1. Sign in to the [Azure AD admin center](https://portal.azure.com).
+ 1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
- 1. Select **Users** and then, to open the user's profile, select the user to be assigned to an administrative unit.
+ 1. Select **Azure Active Directory** > **Users** and then, to open the user's profile, select the user to be assigned to an administrative unit.
1. Select **Administrative units**.
You can assign users to administrative units individually or as a bulk operation
- Assign individual users from an administrative unit:
- 1. Sign in to the [Azure AD admin center](https://portal.azure.com).
- 1. Select **Administrative units**, and then select the administrative unit where the user is to be assigned.
+ 1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+ 1. Select **Azure Active Directory** > **Administrative units**, and then select the administrative unit where the user is to be assigned.
+ 1. Select **All users**, select **Add member** and then, on the **Add member** pane, select one or more users that you want to assign to the administrative unit. ![Screenshot of the administrative unit "Users" pane for assigning a user to an administrative unit.](./media/admin-units-add-manage-users/assign-to-admin-unit.png) - Assign users as a bulk operation:
- 1. Sign in to the [Azure AD admin center](https://portal.azure.com).
+ 1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
- 1. Select **Administrative units**.
+ 1. Select **Azure Active Directory** > **Administrative units**.
1. Select the administrative unit to which you want to add users.
active-directory Admin Units Assign Roles https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/admin-units-assign-roles.md
You can assign a scoped role by using the Azure portal, PowerShell, or Microsoft
### Azure portal
-1. In the Azure portal, go to **Azure AD**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
-1. Select **Administrative units**, and then select the administrative unit that you want to assign a user role scope to.
+1. Select **Azure Active Directory** > **Administrative units** and then select the administrative unit that you want to assign a user role scope to.
1. On the left pane, select **Roles and administrators** to list all the available roles.
You can view a list of scoped admins by using the Azure portal, PowerShell, or M
You can view all the role assignments created with an administrative unit scope in the [Administrative units section of Azure AD](https://ms.portal.azure.com/?microsoft_aad_iam_adminunitprivatepreview=true&microsoft_aad_iam_rbacv2=true#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/AdminUnit).
-1. In the Azure portal, go to **Azure AD**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
-1. In the left pane, select **Administrative units**, and then select the administrative unit for the list of role assignments you want to view.
+1. Select **Azure Active Directory** > **Administrative units** and then select the administrative unit for the list of role assignments you want to view.
1. Select **Roles and administrators**, and then open a role to view the assignments in the administrative unit.
active-directory Admin Units Manage https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/admin-units-manage.md
You can add an administrative unit by using either the Azure portal or PowerShel
### Azure portal
-1. In the Azure portal, go to Azure AD. Then, on the left pane, select **Administrative units**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+1. Select **Azure Active Directory** > **Administrative units**.
![Screenshot of the "Administrative units" link in Azure AD.](./media/admin-units-manage/nav-to-admin-units.png)
In Azure AD, you can remove an administrative unit that you no longer need as a
### Azure portal
-1. In the Azure portal, go to **Azure AD**, and then select **Administrative units**.
-1. Select the administrative unit to be deleted, and then select **Delete**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+1. Select **Azure Active Directory** > **Administrative units**.
+
+1. Select the administrative unit to be deleted, and then select **Delete**.
+ 1. To confirm that you want to delete the administrative unit, select **Yes**. The administrative unit is deleted.
-![Screenshot of the administrative unit Delete button and confirmation window.](./media/admin-units-manage/select-admin-unit-to-delete.png)
+ ![Screenshot of the administrative unit Delete button and confirmation window.](./media/admin-units-manage/select-admin-unit-to-delete.png)
### PowerShell
active-directory Best Practices https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/best-practices.md
When planning your access control strategy, it's a best practice to manage to le
Follow these steps to help you find the right role.
-1. In the Azure portal, open [Roles and administrators](https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/RolesAndAdministrators) to see the list of Azure AD roles.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+1. Select **Azure Active Directory** > **Roles and administrators** to see the list of Azure AD roles.
1. Use the **Service** filter to narrow down the list of roles.
active-directory Custom Create https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/custom-create.md
For more information, see [Prerequisites to use PowerShell or Graph Explorer](pr
### Create a new custom role to grant access to manage app registrations
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com).
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+ 1. Select **Azure Active Directory** > **Roles and administrators** > **New custom role**. ![Create or edit roles from the Roles and administrators page](./media/custom-create/new-custom-role.png)
$roleAssignment = New-AzureADMSRoleAssignment -ResourceScope $resourceScope -Rol
Like built-in roles, custom roles are assigned by default at the default organization-wide scope to grant access permissions over all app registrations in your organization. Additionally, custom roles and some relevant built-in roles (depending on the type of Azure AD resource) can also be assigned at the scope of a single Azure AD resource. This allows you to give the user the permission to update credentials and basic properties of a single app without having to create a second custom role.
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com) with Application Developer permissions.
-1. Select **App registrations**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com) with Application Developer permissions.
+
+1. Select **Azure Active Directory** > **App registrations**.
+ 1. Select the app registration to which you are granting access to manage. You might have to select **All applications** to see the complete list of app registrations in your Azure AD organization. ![Select the app registration as a resource scope for a role assignment](./media/custom-create/appreg-all-apps.png)
Like built-in roles, custom roles are assigned by default at the default organiz
1. In the app registration, select **Roles and administrators**. If you haven't already created one, instructions are in the [preceding procedure](#create-a-new-custom-role-to-grant-access-to-manage-app-registrations). 1. Select the role to open the **Assignments** page.+ 1. Select **Add assignment** to add a user. The user will be granted any permissions over only the selected app registration. ## Next steps
active-directory Custom Enterprise Apps https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/custom-enterprise-apps.md
Granting the update permission is done in two steps:
>[!NOTE] > Custom roles are created and managed at an organization-wide level and are available only from the organization's Overview page.
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com).
-1. Select **Azure Active Directory**, select **Roles and administrators**, and then select **New custom role**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+1. Select **Azure Active Directory** > **Roles and administrators** and then select **New custom role**.
![Add a new custom role from the roles list in Azure AD](./media/custom-enterprise-apps/new-custom-role.png)
Granting the update permission is done in two steps:
### Assign the role to a user using the Azure portal
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com).
-1. Select **Azure Active Directory** and then select **Roles and administrators**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+1. Select **Azure Active Directory** > **Roles and administrators**.
+ 1. Select the **Grant permissions to manage user and group assignments** role. ![Open Roles and Administrators and search for the custom role](./media/custom-enterprise-apps/select-custom-role.png)
active-directory Groups Assign Role https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/groups-assign-role.md
For more information, see [Prerequisites to use PowerShell or Graph Explorer](pr
Assigning a group to an Azure AD role is similar to assigning users and service principals except that only groups that are role-assignable can be used. In the Azure portal, only groups that are role-assignable are displayed.
-1. Sign in to the [Azure AD admin center](https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/Overview).
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
-1. Select **Azure Active Directory** > **Roles and administrators**, and select the role you want to assign.
+1. Select **Azure Active Directory** > **Roles and administrators** and select the role you want to assign.
1. On the ***role name*** page, select > **Add assignment**.
active-directory Groups Create Eligible https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/groups-create-eligible.md
For more information, see [Prerequisites to use PowerShell or Graph Explorer](pr
## Azure portal
-1. Sign in to the [Azure AD admin center](https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/Overview).
-1. Select **Groups** > **All groups** > **New group**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+1. Select **Azure Active Directory** > **Groups** > **All groups** > **New group**.
[![Open Azure Active Directory and create a new group.](./media/groups-create-eligible/new-group.png "Open Azure Active Directory and create a new group.")](./media/groups-create-eligible/new-group.png#<lightbox>) 1. On the **New Group** tab, provide group type, name and description.+ 1. Turn on **Azure AD roles can be assigned to the group**. This switch is visible to only Privileged Role Administrators and Global Administrators because these are only two roles that can set the switch. [![Make the new group eligible for role assignment](./media/groups-create-eligible/eligible-switch.png "Make the new group eligible for role assignment")](./media/groups-create-eligible/eligible-switch.png#<lightbox>)
active-directory Groups Remove Assignment https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/groups-remove-assignment.md
For more information, see [Prerequisites to use PowerShell or Graph Explorer](pr
## Azure portal
-1. Sign in to the [Azure AD admin center](https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/Overview).
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
-1. Select **Roles and administrators** > ***role name***.
+1. Select **Azure Active Directory** > **Roles and administrators** > *role name*.
1. Select the group from which you want to remove the role assignment and select **Remove assignment**.
active-directory Groups View Assignments https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/groups-view-assignments.md
Title: View roles assigned to a group in Azure Active Directory | Microsoft Docs
-description: Learn how the roles assigned to a group can be viewed using Azure AD admin center. Viewing groups and assigned roles are default user permissions.
+description: Learn how the roles assigned to a group can be viewed using the Azure portal. Viewing groups and assigned roles are default user permissions.
# View roles assigned to a group in Azure Active Directory
-This section describes how the roles assigned to a group can be viewed using Azure AD admin center. Viewing groups and assigned roles are default user permissions.
+This section describes how the roles assigned to a group can be viewed using the Azure portal. Viewing groups and assigned roles are default user permissions.
## Prerequisites
For more information, see [Prerequisites to use PowerShell or Graph Explorer](pr
## Azure portal
-1. Sign in to the [Azure AD admin center](https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/Overview).
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
-1. Select the group that you are interested in.
+1. Select **Azure Active Directory** > **Groups**.
+
+1. Select a role-assignable group that you are interested in.
1. Select **Assigned roles**. You can now see all the Azure AD roles assigned to this group.
active-directory Manage Roles Portal https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/manage-roles-portal.md
Follow these steps to assign Azure AD roles using the Azure portal. Your experie
### Assign a role
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com).
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
1. Select **Azure Active Directory** > **Roles and administrators** to see the list of all available roles.
If you have [Azure AD Privileged Identity Management (PIM)](../privileged-identi
Follow these steps to assign roles using the [Roles and administrators](https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/RolesAndAdministrators) page. If you want to assign roles using the [Privileged Identity Management](https://portal.azure.com/#blade/Microsoft_Azure_PIMCommon/CommonMenuBlade/quickStart) page, see [Assign Azure AD roles in Privileged Identity Management](../privileged-identity-management/pim-how-to-add-role-to-user.md).
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com).
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
1. Select **Azure Active Directory** > **Roles and administrators** to see the list of all available roles.
active-directory My Staff Configure https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/my-staff-configure.md
To complete this article, you need the following resources and privileges:
* Each user who's enabled in the text message authentication method policy must be licensed, even if they don't use it. Each enabled user must have one of the following Azure AD or Microsoft 365 licenses: * [Azure AD Premium P1 or P2](https://www.microsoft.com/security/business/identity-access-management/azure-ad-pricing)
- * [Microsoft 365 (M365) F1 or F3](https://www.microsoft.com/licensing/news/m365-firstline-workers)
- * [Enterprise Mobility + Security (EMS) E3 or E5](https://www.microsoft.com/microsoft-365/enterprise-mobility-security/compare-plans-and-pricing) or [Microsoft 365 (M365) E3 or E5](https://www.microsoft.com/microsoft-365/compare-microsoft-365-enterprise-plans)
+ * [Microsoft 365 F1 or F3](https://www.microsoft.com/licensing/news/m365-firstline-workers)
+ * [Enterprise Mobility + Security (EMS) E3 or E5](https://www.microsoft.com/microsoft-365/enterprise-mobility-security/compare-plans-and-pricing) or [Microsoft 365 E3 or E5](https://www.microsoft.com/microsoft-365/compare-microsoft-365-enterprise-plans)
## How to enable My Staff Once you have configured administrative units, you can apply this scope to your users who access My Staff. Only users who are assigned an administrative role can access My Staff. To enable My Staff, complete the following steps:
-1. Sign into the Azure portal as a User Administrator.
-2. Browse to **Azure Active Directory** > **User settings** > **User feature previews** > **Manage user feature preview settings**.
-3. Under **Administrators can access My Staff**, you can choose to enable for all users, selected users, or no user access.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com) as a User Administrator.
+
+1. Select **Azure Active Directory** > **User settings** > **User feature previews** > **Manage user feature preview settings**.
+
+1. Under **Administrators can access My Staff**, you can choose to enable for all users, selected users, or no user access.
> [!Note] > Only users who've been assigned an admin role can access My Staff. If you enable My Staff for a user who is not assigned an admin role, they won't be able to access My Staff.
active-directory Prerequisites https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/prerequisites.md
To use AzureADPreview, follow these steps to make sure it is imported into the c
To manage Azure AD roles using the [Microsoft Graph API](/graph/overview) and [Graph Explorer](/graph/graph-explorer/graph-explorer-overview), you must do the following:
-1. In the Azure portal, open **Azure Active Directory**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
-1. Click **Enterprise applications**.
+1. Select **Azure Active Directory** > **Enterprise applications**.
-1. In the applications list, find and click **Graph explorer**.
+1. In the applications list, find and select **Graph explorer**.
-1. Click **Permissions**.
+1. Select **Permissions**.
-1. Click **Grant admin consent for Graph explorer**.
+1. Select **Grant admin consent for Graph explorer**.
![Screenshot showing the "Grant admin consent for Graph explorer" link.](./media/prerequisites/select-graph-explorer.png)
active-directory Quickstart App Registration Limits https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/quickstart-app-registration-limits.md
For more information, see [Prerequisites to use PowerShell or Graph Explorer](pr
### Create a custom role
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com).
-1. Select **Azure Active Directory**, select **Roles and administrators**, and then select **New custom role**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+1. Select **Azure Active Directory** > **Roles and administrators** and then select **New custom role**.
![Create or edit roles from the Roles and administrators page](./media/quickstart-app-registration-limits/new-custom-role.png)
For more information, see [Prerequisites to use PowerShell or Graph Explorer](pr
### Assign the role
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com).
-1. Select **Azure Active Directory** and then select **Roles and administrators**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+1. Select **Azure Active Directory** > **Roles and administrators**.
+ 1. Select the Application Registration Creator role and select **Add assignment**.+ 1. Select the desired user and click **Select** to add the user to the role. Done! In this quickstart, you successfully created a custom role with permission to create an unlimited number of app registrations, and then assign that role to a user.
active-directory Role Definitions List https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/role-definitions-list.md
For more information, see [Prerequisites to use PowerShell or Graph Explorer](pr
## Azure portal
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com) and select **Azure Active Directory**.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
-1. Select **Roles and administrators** to see the list of all available roles.
+1. Select **Azure Active Directory** > **Roles and administrators** to see the list of all available roles.
![list of roles in Azure portal](./media/role-definitions-list/view-roles-in-azure-active-directory.png)
active-directory Security Emergency Access https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/security-emergency-access.md
Organizations should monitor sign-in and audit log activity from the emergency a
### Obtain Object IDs of the break glass accounts
-1. Sign in to the [Azure portal](https://portal.azure.com) with an account assigned to the User Administrator role.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com) with an account assigned to the User Administrator role.
+ 1. Select **Azure Active Directory** > **Users**. 1. Search for the break-glass account and select the userΓÇÖs name. 1. Copy and save the Object ID attribute so that you can use it later.
active-directory View Assignments https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/active-directory/roles/view-assignments.md
For more information, see [Prerequisites to use PowerShell or Graph Explorer](pr
This procedure describes how to list role assignments with organization-wide scope.
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com).
-1. Select **Azure Active Directory**, select **Roles and administrators**, and then select a role to open it and view its properties.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+1. Select **Azure Active Directory** > **Roles and administrators** and then select a role to open it and view its properties.
+ 1. Select **Assignments** to list the role assignments. ![List role assignments and permissions when you open a role from the list](./media/view-assignments/role-assignments.png)
To download all assignments for a specific role, on the **Roles and administrato
This section describes how to list role assignments with single-application scope. This feature is currently in public preview.
-1. Sign in to the [Azure AD admin center](https://aad.portal.azure.com).
-1. Select **App registrations**, and then select the app registration to view its properties. You might have to select **All applications** to see the complete list of app registrations in your Azure AD organization.
+1. Sign in to the [Azure portal](https://portal.azure.com) or [Azure AD admin center](https://aad.portal.azure.com).
+
+1. Select **Azure Active Directory** > **App registrations**, and then select the app registration to view its properties. You might have to select **All applications** to see the complete list of app registrations in your Azure AD organization.
![Create or edit app registrations from the App registrations page](./media/view-assignments/app-reg-all-apps.png)
aks Monitor Aks https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/aks/monitor-aks.md
Last updated 07/29/2021
-# Monitoring Azure Kubernetes Service (AKS) machines with Azure Monitor
+# Monitoring Azure Kubernetes Service (AKS) with Azure Monitor
This scenario describes how to use Azure Monitor to monitor the health and performance of Azure Kubernetes Service (AKS). It includes collection of telemetry critical for monitoring, analysis and visualization of collected data to identify trends, and how to configure alerting to be proactively notified of critical issues. The [Cloud Monitoring Guide](/azure/cloud-adoption-framework/manage/monitor/) defines the [primary monitoring objectives](/azure/cloud-adoption-framework/strategy/monitoring-strategy#formulate-monitoring-requirements) you should focus on for your Azure resources. This scenario focuses on Health and Status monitoring using Azure Monitor.
If you're unsure about which resource logs to initially enable, use the recommen
| Category | Enable? | Destination | |:|:|:|
-| cluster-autoscale | Enable if autoscale is enabled | Log Analytics workspace |
+| cluster-autoscaler | Enable if autoscale is enabled | Log Analytics workspace |
| guard | Enable if Azure Active Directory is enabled | Log Analytics workspace | | kube-apiserver | Enable | Log Analytics workspace | | kube-audit | Enable | Azure storage. This keeps costs to a minimum yet retains the audit logs if they're required by an auditor. |
aks Servicemesh About https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/aks/servicemesh-about.md
Title: About service meshes
-description: Obtain an overview of service meshes, their architecture and capabilities, and what criteria you should consider when selecting one to deploy.
-
+description: Obtain an overview of service meshes, supported scenarios, selection criteria, and next steps to explore.
+ Previously updated : 10/09/2019- Last updated : 07/29/2021+ # About service meshes
These are some of the scenarios that can be enabled for your workloads when you
- **Traffic management and manipulation** - Create a policy on a service that will rate limit all traffic to a version of a service from a specific origin. Or a policy that applies a retry strategy to classes of failures between specified services. Mirror live traffic to new versions of services during a migration or to debug issues. Inject faults between services in a test environment to test resiliency. -- **Observability** - Gain insight into how your services are connected by analyzing the traffic that flows between them. Obtain metrics, logs, and traces for all traffic in cluster, and ingress/egress. Add distributed tracing abilities to your applications.-
-## Architecture
-
-A service mesh is typically composed of a control plane and the data plane.
-
-The **control plane** has a number of components that support managing the service mesh. This will typically include a management interface which could be a UI or an API. There will also typically be components that manage the rule and policy definitions that define how the service mesh should implement specific capabilities. There are also components that manage aspects of security like strong identity and certificates for mTLS. Service meshes will also typically have a metrics or observability component that collects and aggregates metrics and telemetry from the workloads.
-
-The **data plane** typically consists of a proxy that is transparently injected as a sidecar to your workloads. This proxy is configured to control all network traffic in and out of the pod containing your workload. This allows the proxy to be configured to secure traffic via mTLS, dynamically route traffic, apply policies to traffic and to collect metrics and tracing information.
-
-![Typical service mesh architecture](media/servicemesh/typical-architecture.png)
-
-## Capabilities
-
-Each of the service meshes have a natural fit and focus on supporting specific scenarios, but you'll typically find that most will implement a number of, if not all, of the following capabilities.
-
-### Traffic management
--- **Protocol** ΓÇô layer 7 (http, grpc)-- **Dynamic Routing** ΓÇô conditional, weighting, mirroring-- **Resiliency** ΓÇô timeouts, retries, circuit breakers-- **Policy** ΓÇô access control, rate limits, quotas-- **Testing** - fault injection-
-### Security
--- **Encryption** ΓÇô mTLS, certificate management, external CA-- **Strong Identity** ΓÇô SPIFFE or similar-- **Auth** ΓÇô authentication, authorisation-
-### Observability
--- **Metrics** ΓÇô golden metrics, prometheus, grafana-- **Tracing** - traces across workloads-- **Traffic** ΓÇô cluster, ingress/egress-
-### Mesh
--- **Supported Compute** - Kubernetes, virtual machines-- **Multi-cluster** - gateways, federation
+- **Observability** - Gain insight into how your services are connected the traffic that flows between them. Obtain metrics, logs, and traces for all traffic in cluster, and ingress/egress. Add distributed tracing abilities to your applications.
## Selection criteria
-Before you select a service mesh, ensure that you understand your requirements and the reasons for installing a service mesh. Try asking the following questions.
+Before you select a service mesh, ensure that you understand your requirements and the reasons for installing a service mesh. Ask the following questions.
- **Is an Ingress Controller sufficient for my needs?** - Sometimes having a capability like a/b testing or traffic splitting at the ingress is sufficient to support the required scenario. Don't add complexity to your environment with no upside. -- **Can my workloads and environment tolerate the additional overheads?** - All the additional components required to support the service mesh require additional resources like CPU and memory. In addition, all the proxies and their associated policy checks add latency to your traffic. If you have workloads that are very sensitive to latency or cannot provide the additional resources to cover the service mesh components, then re-consider.
+- **Can my workloads and environment tolerate the additional overheads?** - All the additional components required to support the service mesh require additional resources like cpu and memory. In addition, all the proxies and their associated policy checks add latency to your traffic. If you have workloads that are very sensitive to latency or cannot provide the additional resources to cover the service mesh components, then re-consider.
- **Is this adding additional complexity unnecessarily?** - If the reason for installing a service mesh is to gain a capability that is not necessarily critical to the business or operational teams, then consider whether the additional complexity of installation, maintenance, and configuration is worth it. - **Can this be adopted in an incremental approach?** - Some of the service meshes that provide a lot of capabilities can be adopted in a more incremental approach. Install just the components you need to ensure your success. Once you are more confident and additional capabilities are required, then explore those. Resist the urge to install *everything* from the start.
-If, after careful consideration, you decide that you need a service mesh to provide the capabilities required, then your next decision is *which service mesh?*
-
-Consider the following areas and which of them are most aligned with your requirements. This will guide you towards the best fit for your environment and workloads. The [Next steps](#next-steps) section will take you to further detailed information about specific service meshes and how they map to these areas.
--- **Technical** - traffic management, policy, security, observability--- **Business** - commercial support, foundation (CNCF), OSS license, governance--- **Operational** ΓÇô installation/upgrades, resource requirements, performance requirements, integrations (metrics, telemetry, dashboards, tools, SMI), mixed workloads (Linux and Windows node pools), compute (Kubernetes, virtual machines), multi-cluster--- **Security** - auth, identity, certificate management and rotation, pluggable external CA-- ## Next steps
-The following documentation provides more information about service meshes that you can try out on Azure Kubernetes Service (AKS):
+As a next step, explore Open Service Mesh (OSM) on Azure Kubernetes Service (AKS):
> [!div class="nextstepaction"]
-> [Learn more about Istio ...][istio-about]
+> [Learn more about OSM ...][osm-about]
-> [!div class="nextstepaction"]
-> [Learn more about Linkerd ...][linkerd-about]
+You can also explore the following service meshes on Azure Kubernetes Service (AKS) via the comprehensive project documentation available for each of them:
-> [!div class="nextstepaction"]
-> [Learn more about Consul ...][consul-about]
+- [Istio][istio]
+- [Linkerd][linkerd]
+- [Consul Connect][consul]
+
+If you'd like to understand more about the service mesh landscape, the broader set of available service meshes, tooling, and compliance, then explore:
+
+- [Layer 5's Service Mesh Landscape][service-mesh-landscape]
-You may also want to explore Service Mesh Interface (SMI), a standard interface for service meshes on Kubernetes:
+You may also want to explore the various service mesh standardization efforts:
- [Service Mesh Interface (SMI)][smi]
+- [Service Mesh Federation][smf]
+- [Service Mesh Performance (SMP)][smp]
<!-- LINKS - external -->
+[istio]: https://istio.io/latest/docs/setup/install/
+[linkerd]: https://linkerd.io/getting-started/
+[consul]: https://learn.hashicorp.com/tutorials/consul/service-mesh-deploy
+[service-mesh-landscape]: https://layer5.io/service-mesh-landscape
[smi]: https://smi-spec.io/
+[smf]: https://github.com/vmware/hamlet
+[smp]: https://github.com/service-mesh-performance/service-mesh-performance
<!-- LINKS - internal -->
-[istio-about]: ./servicemesh-istio-about.md
-[linkerd-about]: ./servicemesh-linkerd-about.md
-[consul-about]: ./servicemesh-consul-about.md
+[osm-about]: ./servicemesh-osm-about.md
aks Servicemesh Consul About https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/aks/servicemesh-consul-about.md
- Title: Overview of Consul
-description: Obtain an overview of Consul
-- Previously updated : 10/09/2019---
-# Consul
-
-## Overview
-
-[Consul][consul] is a multi data centre aware service networking solution to connect and secure services across runtime platforms. [Connect][consul-features] is the component that provides service mesh capabilities.
-
-## Architecture
-
-Consul provides a data plane that is composed of [Envoy][envoy-proxy]-based [sidecars][consul-sidecar] by default. Consul has a pluggable proxy architecture. These intelligent proxies control all network traffic in and out of your meshed apps and workloads.
-
-The control plane manages the configuration, and policy via the following [components][consul-architecture]:
--- **Server** - A Consul Agent running in Server mode that maintains Consul cluster state.--- **Client** - A Consul Agent running in lightweight Client Mode. Each compute node must have a Client agent running. This client brokers configuration and policy between the workloads and the Consul configuration. -
-The following architecture diagram demonstrates how the various components within the data plane and control plane interact.
-
-![Overview of Consul components and architecture.](media/servicemesh/consul/about-architecture.png)
--
-## Selection criteria
-
-It's important to understand and consider the following areas when evaluating Consul for your workloads:
--- [Consul Principles](#consul-principles)-- [Capabilities](#capabilities)-- [Scenarios](#scenarios)--
-### Consul principles
-
-The following principles [guide][consul-principles] the Consul project:
--- **API-Driven** - Codify all configuration and policy.--- **Run and Connect Anywhere** - Connect workloads across runtime platforms (Kubernetes, VMs, Serverless).--- **Extend and Integrate** - Securely connect workloads across infrastructure.--
-### Capabilities
-
-Consul provides the following set of capabilities:
--- **Mesh** ΓÇô gateway (multi data centre), virtual machines (out of cluster nodes), service sync, built in debugging option--- **Proxies** ΓÇô Envoy, built-in proxy, pluggable, l4 proxy available for Windows workloads--- **Traffic Management** ΓÇô routing, splitting, resolution--- **Policy** ΓÇô intentions, ACLs--- **Security** ΓÇô authorisation, authentication, encryption, SPIFFE-based identities, external CA (Vault), certificate management, and rotation--- **Observability** ΓÇô metrics, ui dashboard, prometheus, grafana--
-### Scenarios
-
-Consul is well suited to and suggested for the following scenarios:
--- Extending existing Consul connected workloads--- Compliance requirements around certificate management--- Multi cluster service mesh--- VM-based workloads to be included in the service mesh---
-## Next steps
-
-The following documentation describes how you can install Consul on Azure Kubernetes Service (AKS):
-
-> [!div class="nextstepaction"]
-> [Install Consul in Azure Kubernetes Service (AKS)][consul-install]
-
-You can also further explore Consul features and architecture:
--- [Consul Getting Started Tutorials][consul-getting-started]-- [Consul Features][consul-features]-- [Consul Architecture][consul-architecture]-- [Consul - How Connect Works][consul-how-connect-works]-
-<!-- LINKS - external -->
-[consul]: https://www.consul.io/mesh.html
-[consul-features]: https://www.consul.io/docs/connect/https://docsupdatetracker.net/index.html
-[consul-architecture]: https://www.consul.io/docs/internals/architecture.html
-[consul-sidecar]: https://www.consul.io/docs/connect/proxies.html
-[consul-how-connect-works]: https://www.consul.io/docs/connect/connect-internals.html
-[consul-principles]: https://www.consul.io/
-[consul-getting-started]:https://learn.hashicorp.com/consul?track=gs-consul-service-mesh#gs-consul-service-mesh
-
-[envoy-proxy]: https://www.envoyproxy.io/
-[grafana]: https://grafana.com/
-[prometheus]: https://prometheus.io/
-
-<!-- LINKS - internal -->
-[consul-install]: ./servicemesh-consul-install.md
aks Servicemesh Consul Install https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/aks/servicemesh-consul-install.md
- Title: Install Consul in Azure Kubernetes Service (AKS)
-description: Learn how to install and use Consul to create a service mesh in an Azure Kubernetes Service (AKS) cluster
-- Previously updated : 10/09/2019-
-zone_pivot_groups: client-operating-system
--
-# Install and use Consul in Azure Kubernetes Service (AKS)
-
-[Consul][consul-github] is an open-source service mesh that provides a key set of functionality across the microservices in a Kubernetes cluster. These features include service discovery, health checking, service segmentation, and observability. For more information about Consul, see the official [What is Consul?][consul-docs-concepts] documentation.
-
-This article shows you how to install Consul. The Consul components are installed into a Kubernetes cluster on AKS.
-
-> [!NOTE]
-> These instructions reference Consul version `1.6.0`, and use at least Helm version `2.14.2`.
->
-> The Consul `1.6.x` releases can be run against Kubernetes versions `1.13+`. You can find additional Consul versions at [GitHub - Consul Releases][consul-github-releases] and information about each of the releases at [Consul- Release Notes][consul-release-notes].
-
-In this article, you learn how to:
-
-> [!div class="checklist"]
-> * Install the Consul components on AKS
-> * Validate the Consul installation
-> * Uninstall Consul from AKS
-
-## Before you begin
-
-The steps detailed in this article assume that you've created an AKS cluster (Kubernetes `1.13` and above, with Kubernetes RBAC enabled) and have established a `kubectl` connection with the cluster. If you need help with any of these items, then see the [AKS quickstart][aks-quickstart]. Ensure that your cluster has at least 3 nodes in the Linux node pool.
-
-You'll need [Helm][helm] to follow these instructions and install Consul. It's recommended that you have the latest stable version correctly installed and configured in your cluster. If you need help with installing Helm, then see the [AKS Helm installation guidance][helm-install]. All Consul pods must also be scheduled to run on Linux nodes.
-
-This article separates the Consul installation guidance into several discrete steps. The end result is the same in structure as the official Consul installation [guidance][consul-install-k8].
-
-### Install the Consul components on AKS
-
-We'll start by downloading version `v0.10.0` of the Consul Helm chart. This version of the chart includes Consul version `1.6.0`.
----------
-Use Helm and the downloaded `consul-helm` chart to install the Consul components into the `consul` namespace in your AKS cluster.
-
-> [!NOTE]
-> **Installation options**
->
-> We are using the following options as part of our installation:
-> - `connectInject.enabled=true` - enable proxies to be injected into pods
-> - `client.enabled=true` - enable Consul clients to run on every node
-> - `client.grpc=true` - enable gRPC listener for connectInject
-> - `syncCatalog.enabled=true` - sync Kubernetes and Consul services
->
-> **Node selectors**
->
-> Consul currently must be scheduled to run on Linux nodes. If you have Windows Server nodes in your cluster, you must ensure that the Consul pods are only scheduled to run on Linux nodes. We'll use [node selectors][kubernetes-node-selectors] to make sure pods are scheduled to the correct nodes.
----------
-The `Consul` Helm chart deploys a number of objects. You can see the list from the output of your `helm install` command above. The deployment of the Consul components can take around 3 minutes to complete, depending on your cluster environment.
-
-At this point, you've deployed Consul to your AKS cluster. To ensure that we have a successful deployment of Consul, let's move on to the next section to validate the Consul installation.
-
-## Validate the Consul installation
-
-Confirm that the resources have been successfully created. Use the [kubectl get svc][kubectl-get] and [kubectl get pod][kubectl-get] commands to query the `consul` namespace, where the Consul components were installed by the `helm install` command:
-
-```console
-kubectl get svc --namespace consul --output wide
-kubectl get pod --namespace consul --output wide
-```
-
-The following example output shows the services and pods (scheduled on Linux nodes) that should now be running:
-
-```output
-NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
-consul ExternalName <none> consul.service.consul <none> 38s <none>
-consul-consul-connect-injector-svc ClusterIP 10.0.98.102 <none> 443/TCP 3m26s app=consul,component=connect-injector,release=consul
-consul-consul-dns ClusterIP 10.0.46.194 <none> 53/TCP,53/UDP 3m26s app=consul,hasDNS=true,release=consul
-consul-consul-server ClusterIP None <none> 8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP 3m26s app=consul,component=server,release=consul
-consul-consul-ui ClusterIP 10.0.50.188 <none> 80/TCP 3m26s app=consul,component=server,release=consul
-
-NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
-consul-consul-connect-injector-webhook-deployment-99f74fdbcr5zj 1/1 Running 0 3m9s 10.240.0.68 aks-linux-92468653-vmss000002 <none> <none>
-consul-consul-jbksc 1/1 Running 0 3m9s 10.240.0.44 aks-linux-92468653-vmss000001 <none> <none>
-consul-consul-jkwtq 1/1 Running 0 3m9s 10.240.0.70 aks-linux-92468653-vmss000002 <none> <none>
-consul-consul-server-0 1/1 Running 0 3m9s 10.240.0.91 aks-linux-92468653-vmss000002 <none> <none>
-consul-consul-server-1 1/1 Running 0 3m9s 10.240.0.38 aks-linux-92468653-vmss000001 <none> <none>
-consul-consul-server-2 1/1 Running 0 3m9s 10.240.0.10 aks-linux-92468653-vmss000000 <none> <none>
-consul-consul-sync-catalog-d846b79c-8ssr8 1/1 Running 2 3m9s 10.240.0.94 aks-linux-92468653-vmss000002 <none> <none>
-consul-consul-tz2t5 1/1 Running 0 3m9s 10.240.0.12 aks-linux-92468653-vmss000000 <none> <none>
-```
-
-All of the pods should show a status of `Running`. If your pods don't have these statuses, wait a minute or two until they do. If any pods report an issue, use the [kubectl describe pod][kubectl-describe] command to review their output and status.
-
-## Accessing the Consul UI
-
-The Consul UI was installed in our setup above and provides UI based configuration for Consul. The UI for Consul is not exposed publicly via an external ip address. To access the Consul user interface, use the [kubectl port-forward][kubectl-port-forward] command. This command creates a secure connection between your client machine and the relevant pod in your AKS cluster.
-
-```console
-kubectl port-forward -n consul svc/consul-consul-ui 8080:80
-```
-
-You can now open a browser and point it to `http://localhost:8080/ui` to open the Consul UI. You should see the following when you open the UI:
-
-![Consul UI](./media/servicemesh/consul/consul-ui.png)
-
-## Uninstall Consul from AKS
-
-> [!WARNING]
-> Deleting Consul from a running system may result in traffic related issues between your services. Ensure that you have made provisions for your system to still operate correctly without Consul before proceeding.
-
-### Remove Consul components and namespace
-
-To remove Consul from your AKS cluster, use the following commands. The `helm delete` commands will remove the `consul` chart, and the `kubectl delete namespace` command will remove the `consul` namespace.
-
-```console
-helm delete --purge consul
-kubectl delete namespace consul
-```
-
-## Next steps
-
-To explore more installation and configuration options for Consul, see the following official Consul articles:
--- [Consul - Helm installation guide][consul-install-k8]-- [Consul - Helm installation options][consul-install-helm-options]-
-You can also follow additional scenarios using:
--- [Consul Example Application][consul-app-example]-- [Consul Kubernetes Reference Architecture][consul-reference]-- [Consul Mesh Gateways][consul-mesh-gateways]-
-<!-- LINKS - external -->
-[Hashicorp]: https://hashicorp.com
-[cosul-github]: https://github.com/hashicorp/consul
-[helm]: https://helm.sh
-
-[consul-docs-concepts]: https://www.consul.io/docs/platform/k8s/https://docsupdatetracker.net/index.html
-[consul-github]: https://github.com/hashicorp/consul
-[consul-github-releases]: https://github.com/hashicorp/consul/releases
-[consul-release-notes]: https://github.com/hashicorp/consul/blob/master/CHANGELOG.md
-[consul-install-download]: https://www.consul.io/downloads.html
-[consul-install-k8]: https://learn.hashicorp.com/consul/kubernetes/kubernetes-deployment-guide
-[consul-install-helm-options]: https://www.consul.io/docs/platform/k8s/helm.html#configuration-values-
-[consul-mesh-gateways]: https://learn.hashicorp.com/consul/kubernetes/mesh-gateways
-[consul-reference]: https://learn.hashicorp.com/consul/kubernetes/kubernetes-reference
-[consul-app-example]: https://learn.hashicorp.com/consul?track=gs-consul-service-mesh#gs-consul-service-mesh
-[install-wsl]: /windows/wsl/install-win10
-
-[kubectl-get]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get
-[kubectl-describe]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe
-[kubectl-port-forward]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward
-[kubernetes-node-selectors]: ./concepts-clusters-workloads.md#node-selectors
-
-<!-- LINKS - internal -->
-[aks-quickstart]: ./kubernetes-walkthrough.md
-[consul-scenario-mtls]: ./consul-mtls.md
-[helm-install]: ./kubernetes-helm.md
aks Servicemesh Istio About https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/aks/servicemesh-istio-about.md
- Title: Overview of Istio
-description: Obtain an overview of Istio
-- Previously updated : 10/09/2019---
-# Istio
-
-## Overview
-
-[Istio][istio] is a full featured, customisable, and extensible service mesh.
-
-## Architecture
-
-Istio provides a data plane that is composed of [Envoy][envoy-proxy]-based sidecars. These intelligent proxies control all network traffic in and out of your meshed apps and workloads.
-
-The control plane manages the configuration, policy, and telemetry via the following [components][what-is-istio]:
--- **Mixer** - Enforces access control and usage policies. Collects telemetry from the proxies that is pushed into [Prometheus][prometheus].--- **Pilot** - Provides service discovery and traffic management policy/configuration for the proxies.--- **Citadel** - Provides identity and security capabilities that allow for mTLS between services.--- **Galley** - Abstracts and provides configuration to components.-
-The following architecture diagram demonstrates how the various components within the data plane and control plane interact.
--
-![Overview of Istio components and architecture.](media/servicemesh/istio/about-architecture.png)
--
-## Selection criteria
-
-It's important to understand and consider the following areas when evaluating Istio for your workloads:
--- [Design Goals](#design-goals)-- [Capabilities](#capabilities)-- [Scenarios](#scenarios)--
-### Design goals
-
-The following design goals [guide][design-goals] the Istio project:
--- **Maximize Transparency** - Allow adoption with the minimum amount of work to get real value from the system.--- **Extensibility** - Must be able to grow and adapt with changing needs.--- **Portability** - Run easily in different kinds of environments - cloud, on-premises.--- **Policy Uniformity** - Consistency in policy definition across variety of resources.--
-### Capabilities
-
-Istio provides the following set of capabilities:
--- **Mesh** ΓÇô gateways (multi-cluster), virtual machines (mesh expansion)--- **Traffic Management** ΓÇô routing, splitting, timeouts, circuit breakers, retries, ingress, egress--- **Policy** ΓÇô access control, rate limit, quota, custom policy adapters--- **Security** ΓÇô authentication (jwt), authorisation, encryption (mTLS), external CA (HashiCorp Vault)--- **Observability** ΓÇô golden metrics, mirror, tracing, custom adapters, prometheus, grafana-
-### Scenarios
-
-Istio is well suited to and suggested for the following scenarios:
--- Require extensibility and rich set of capabilities--- Mesh expansion to include VM based workloads--- Multi-cluster service mesh-
-## Next steps
-
-The following documentation describes how you can install Istio on Azure Kubernetes Service (AKS):
-
-> [!div class="nextstepaction"]
-> [Install Istio in Azure Kubernetes Service (AKS)][istio-install]
-
-You can also further explore Istio concepts and additional deployment models:
--- [Istio Concepts][what-is-istio]-- [Istio Deployment Models][deployment-models]-
-<!-- LINKS - external -->
-[istio]: https://istio.io
-[what-is-istio]: https://istio.io/docs/concepts/what-is-istio/
-[design-goals]: https://istio.io/docs/concepts/what-is-istio/#design-goals
-[deployment-models]: https://istio.io/docs/concepts/deployment-models/
-
-[envoy-proxy]: https://www.envoyproxy.io/
-[grafana]: https://grafana.com/
-[prometheus]: https://prometheus.io/
-
-<!-- LINKS - internal -->
-[istio-install]: ./servicemesh-istio-install.md
aks Servicemesh Istio Install https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/aks/servicemesh-istio-install.md
- Title: Install Istio in Azure Kubernetes Service (AKS)
-description: Learn how to install and use Istio to create a service mesh in an Azure Kubernetes Service (AKS) cluster
-- Previously updated : 10/02/2020-
-zone_pivot_groups: client-operating-system
--
-# Install and use Istio in Azure Kubernetes Service (AKS)
-
-[Istio][istio-github] is an open-source service mesh that provides a key set of functionality across the microservices in a Kubernetes cluster. These features include traffic management, service identity and security, policy enforcement, and observability. For more information about Istio, see the official [What is Istio?][istio-docs-concepts] documentation.
-
-This article shows you how to install Istio. The Istio `istioctl` client binary is installed onto your client machine and the Istio components are installed into a Kubernetes cluster on AKS.
-
-> [!NOTE]
-> The following instructions reference Istio version `1.7.3`.
->
-> The Istio `1.7.x` releases have been tested by the Istio team against Kubernetes version `1.16+`. You can find additional Istio versions at [GitHub - Istio Releases][istio-github-releases], information about each of the releases at [Istio News][istio-release-notes] and supported Kubernetes versions at [Istio General FAQ][istio-faq].
-
-In this article, you learn how to:
-
-> [!div class="checklist"]
-> * Download and install the Istio istioctl client binary
-> * Install Istio on AKS
-> * Validate the Istio installation
-> * Access the add-ons
-> * Uninstall Istio from AKS
-
-## Before you begin
-
-The steps detailed in this article assume that you've created an AKS cluster (Kubernetes `1.16` and above, with Kubernetes RBAC enabled) and have established a `kubectl` connection with the cluster. If you need help with any of these items, then see the [AKS quickstart][aks-quickstart].
-
-Make sure that you have read the [Istio Performance and Scalability](https://istio.io/docs/concepts/performance-and-scalability/) documentation to understand the additional resource requirements for running Istio in your AKS cluster. The core and memory requirements will vary based on your specific workload. Choose an appropriate number of nodes and VM size to cater for your setup.
-
-This article separates the Istio installation guidance into several discrete steps. The end result is the same in structure as the official Istio installation [guidance][istio-install-istioctl].
----------
-## Install the Istio Operator on AKS
-
-Istio provides an [Operator][istio-install-operator] to manage installation and updates to the Istio components within your AKS cluster. We'll install the Istio Operator using the `istioctl` client binary.
-
-```bash
-istioctl operator init
-```
-
-You should see something like the following output to confirm that the Istio Operator has been installed.
-
-```console
-Using operator Deployment image: docker.io/istio/operator:1.7.3
-Γ£ö Istio operator installed
-Γ£ö Installation complete
-```
-
-The Istio Operator is installed into the `istio-operator` namespace. Query the namespace.
-
-```bash
-kubectl get all -n istio-operator
-```
-
-You should see the following components deployed.
-
-```console
-NAME READY STATUS RESTARTS AGE
-pod/istio-operator-6d7958b7bf-wxgdc 1/1 Running 0 2m43s
-
-NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
-service/istio-operator ClusterIP 10.0.8.57 <none> 8383/TCP 2m43s
-
-NAME READY UP-TO-DATE AVAILABLE AGE
-deployment.apps/istio-operator 1/1 1 1 2m43s
-
-NAME DESIRED CURRENT READY AGE
-replicaset.apps/istio-operator-6d7958b7bf 1 1 1 2m43s
-```
-
-You can learn more about the Operator pattern and how it can help automate complex tasks via [kubernetes.io][kubernetes-operator].
--
-### Install Istio components
-
-Now that we've successfully installed the Istio Operator in our AKS cluster, it's time to install the Istio components.
-
-We will leverage the `default` [Istio Configuration Profile][istio-configuration-profiles] to build the [Istio Operator Spec][istio-control-plane].
-
-You can run the following `istioctl` command to view the configuration for the `default` Istio Configuration Profile.
-
-```bash
-istioctl profile dump default
-```
-
-> [!NOTE]
-> Istio currently must be scheduled to run on Linux nodes. If you have Windows Server nodes in your cluster, you must ensure that the Istio pods are only scheduled to run on Linux nodes. We'll use [node selectors][kubernetes-node-selectors] to make sure pods are scheduled to the correct nodes.
-
-> [!CAUTION]
-> The [Istio CNI][istio-feature-cni] Istio features are currently in [Alpha][istio-feature-stages], so thought should be given before enabling these.
-
-Create a file called `istio.aks.yaml` with the following content. This file will hold the [Istio Operator Spec][istio-control-plane] for configuring Istio.
-
-```yaml
-apiVersion: install.istio.io/v1alpha1
-kind: IstioOperator
-metadata:
- namespace: istio-system
- name: istio-control-plane
-spec:
- # Use the default profile as the base
- # More details at: https://istio.io/docs/setup/additional-setup/config-profiles/
- profile: default
- # Enable the addons that we will want to use
- addonComponents:
- grafana:
- enabled: true
- prometheus:
- enabled: true
- tracing:
- enabled: true
- kiali:
- enabled: true
- values:
- global:
- # Ensure that the Istio pods are only scheduled to run on Linux nodes
- defaultNodeSelector:
- beta.kubernetes.io/os: linux
- kiali:
- dashboard:
- auth:
- strategy: anonymous
-```
-
-Create the `istio-system` namespace and deploy the Istio Operator Spec to that namespace. The Istio Operator will be watching for the Istio Operator Spec and will use it to install and configure Istio in your AKS cluster.
-
-```bash
-kubectl create ns istio-system
-
-kubectl apply -f istio.aks.yaml
-```
-
-At this point, you've deployed Istio to your AKS cluster. To ensure that we have a successful deployment of Istio, let's move on to the next section to [Validate the Istio installation](#validate-the-istio-installation).
-
-## Validate the Istio installation
-
-Query the `istio-system` namespace, where the Istio and add-on components were installed by the Istio Operator:
-
-```bash
-kubectl get all -n istio-system
-```
-
-You should see the following components:
--- `istio*` - the Istio components-- `jaeger-*`, `tracing`, and `zipkin` - tracing addon-- `prometheus` - metrics addon-- `grafana` - analytics and monitoring dashboard addon-- `kiali` - service mesh dashboard addon-
-```console
-NAME READY STATUS RESTARTS AGE
-pod/grafana-7cf9794c74-mpfbp 1/1 Running 0 5m53s
-pod/istio-ingressgateway-86b5dbdcb9-ndrp5 1/1 Running 0 5m57s
-pod/istio-tracing-c98f4b8fc-zqklg 1/1 Running 0 82s
-pod/istiod-6965c56995-4ph9h 1/1 Running 0 6m15s
-pod/kiali-7b44985d68-p87zh 1/1 Running 0 81s
-pod/prometheus-6868989549-5ghzz 1/1 Running 0 81s
-
-NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
-service/grafana ClusterIP 10.0.226.39 <none> 3000/TCP 5m54s
-service/istio-ingressgateway LoadBalancer 10.0.143.56 20.53.72.254 15021:32166/TCP,80:31684/TCP,443:31302/TCP,15443:30863/TCP 5m57s
-service/istiod ClusterIP 10.0.211.228 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP 6m16s
-service/jaeger-agent ClusterIP None <none> 5775/UDP,6831/UDP,6832/UDP 82s
-service/jaeger-collector ClusterIP 10.0.7.62 <none> 14267/TCP,14268/TCP,14250/TCP 82s
-service/jaeger-collector-headless ClusterIP None <none> 14250/TCP 82s
-service/jaeger-query ClusterIP 10.0.52.172 <none> 16686/TCP 82s
-service/kiali ClusterIP 10.0.71.179 <none> 20001/TCP 82s
-service/prometheus ClusterIP 10.0.171.151 <none> 9090/TCP 82s
-service/tracing ClusterIP 10.0.195.137 <none> 80/TCP 82s
-service/zipkin ClusterIP 10.0.136.111 <none> 9411/TCP 82s
-
-NAME READY UP-TO-DATE AVAILABLE AGE
-deployment.apps/grafana 1/1 1 1 5m54s
-deployment.apps/istio-ingressgateway 1/1 1 1 5m58s
-deployment.apps/istio-tracing 1/1 1 1 83s
-deployment.apps/istiod 1/1 1 1 6m16s
-deployment.apps/kiali 1/1 1 1 83s
-deployment.apps/prometheus 1/1 1 1 82s
-
-NAME DESIRED CURRENT READY AGE
-replicaset.apps/grafana-7cf9794c74 1 1 1 5m54s
-replicaset.apps/istio-ingressgateway-86b5dbdcb9 1 1 1 5m58s
-replicaset.apps/istio-tracing-c98f4b8fc 1 1 1 83s
-replicaset.apps/istiod-6965c56995 1 1 1 6m16s
-replicaset.apps/kiali-7b44985d68 1 1 1 82s
-replicaset.apps/prometheus-6868989549 1 1 1 82s
-
-NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
-horizontalpodautoscaler.autoscaling/istio-ingressgateway Deployment/istio-ingressgateway 7%/80% 1 5 1 5m57s
-horizontalpodautoscaler.autoscaling/istiod Deployment/istiod 1%/80% 1 5 1 6m16s
-```
-
-You can also gain additional insight into the installation by watching the logs for the Istio Operator.
-
-```bash
-kubectl logs -n istio-operator -l name=istio-operator -f
-```
-
-If the `istio-ingressgateway` shows an external ip of `<pending>`, wait a few minutes until an IP address has been assigned by Azure networking.
-
-All of the pods should show a status of `Running`. If your pods don't have these statuses, wait a minute or two until they do. If any pods report an issue, use the [kubectl describe pod][kubectl-describe] command to review their output and status.
-
-## Accessing the add-ons
-
-A number of add-ons were installed by the Istio Operator that provide additional functionality. The web applications for the add-ons are **not** exposed publicly via an external ip address.
-
-To access the add-on user interfaces, use the `istioctl dashboard` command. This command uses [kubectl port-forward][kubectl-port-forward] and a random port to create a secure connection between your client machine and the relevant pod in your AKS cluster. It will then automatically open the add-on web application in your default browser.
-
-### Grafana
-
-The analytics and monitoring dashboards for Istio are provided by [Grafana][grafana]. Remember to use the credentials you created via the Grafana secret earlier when prompted. Open the Grafana dashboard securely as follows:
-
-```console
-istioctl dashboard grafana
-```
-
-### Prometheus
-
-Metrics for Istio are provided by [Prometheus][prometheus]. Open the Prometheus dashboard securely as follows:
-
-```console
-istioctl dashboard prometheus
-```
-
-### Jaeger
-
-Tracing within Istio is provided by [Jaeger][jaeger]. Open the Jaeger dashboard securely as follows:
-
-```console
-istioctl dashboard jaeger
-```
-
-### Kiali
-
-A service mesh observability dashboard is provided by [Kiali][kiali]. Remember to use the credentials you created via the Kiali secret earlier when prompted. Open the Kiali dashboard securely as follows:
-
-```console
-istioctl dashboard kiali
-```
-
-### Envoy
-
-A simple interface to the [Envoy][envoy] proxies is available. It provides configuration information and metrics for an Envoy proxy running in a specified pod. Open the Envoy interface securely as follows:
-
-```console
-istioctl dashboard envoy <pod-name>.<namespace>
-```
-
-## Uninstall Istio from AKS
-
-> [!WARNING]
-> Deleting Istio from a running system may result in traffic related issues between your services. Ensure that you have made provisions for your system to still operate correctly without Istio before proceeding.
-
-### Remove Istio
-
-To remove Istio from your AKS cluster, delete the `IstioOperator` resource named `istio-control-plane` that we added earlier. The Istio Operator will recognize that the Istio Operator Spec has been removed, and then delete all the associated Istio components.
-
-```bash
-kubectl delete istiooperator istio-control-plane -n istio-system
-```
-
-You can run the following to check when all the Istio components have been deleted.
-
-```bash
-kubectl get all -n istio-system
-```
-
-### Remove Istio Operator
-
-Once Istio has been successfully uninstalled, you can also remove the Istio Operator.
-
-```bash
-istioctl operator remove
-```
-
-And then finally, remove the `istio-` namespaces.
-
-```bash
-kubectl delete ns istio-system
-kubectl delete ns istio-operator
-```
-
-## Next steps
-
-To explore more installation and configuration options for Istio, see the following official Istio guidance:
--- [Istio - installation guides][istio-installation-guides]-
-You can also follow additional scenarios using:
--- [Istio Bookinfo Application example][istio-bookinfo-example]-
-<!-- LINKS - external -->
-[istio]: https://istio.io
-[helm]: https://helm.sh
-
-[istio-faq]: https://istio.io/faq/general/
-[istio-docs-concepts]: https://istio.io/docs/concepts/what-is-istio/
-[istio-github]: https://github.com/istio/istio
-[istio-github-releases]: https://github.com/istio/istio/releases
-[istio-release-notes]: https://istio.io/news/
-[istio-installation-guides]: https://istio.io/docs/setup/install/
-[istio-install-download]: https://istio.io/docs/setup/kubernetes/download-release/
-[istio-install-istioctl]: https://istio.io/docs/setup/install/istioctl/
-[istio-install-operator]: https://istio.io/latest/docs/setup/install/operator/
-[istio-configuration-profiles]: https://istio.io/docs/setup/additional-setup/config-profiles/
-[istio-control-plane]: https://istio.io/docs/reference/config/istio.operator.v1alpha1/
-[istio-bookinfo-example]: https://istio.io/docs/examples/bookinfo/
-
-[istio-feature-stages]: https://istio.io/about/feature-stages/
-[istio-feature-sds]: https://istio.io/docs/tasks/traffic-management/ingress/secure-ingress-sds/
-[istio-feature-cni]: https://istio.io/docs/setup/additional-setup/cni/
-
-[kubernetes-operator]: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
-[kubernetes-feature-sa-projected-volume]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection
-[kubernetes-crd]: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions
-[kubernetes-secrets]: https://kubernetes.io/docs/concepts/configuration/secret/
-[kubernetes-node-selectors]: ./concepts-clusters-workloads.md#node-selectors
-[kubectl-get]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get
-[kubectl-describe]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe
-[kubectl-port-forward]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward
-
-[grafana]: https://grafana.com/
-[prometheus]: https://prometheus.io/
-[jaeger]: https://www.jaegertracing.io/
-[kiali]: https://www.kiali.io/
-[envoy]: https://www.envoyproxy.io/
-
-<!-- LINKS - internal -->
-[aks-quickstart]: ./kubernetes-walkthrough.md
aks Servicemesh Istio Scenario Routing https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/aks/servicemesh-istio-scenario-routing.md
- Title: Use Istio for intelligent routing-
-description: Learn how to use Istio to provide intelligent routing and deploy canary releases in an Azure Kubernetes Service (AKS) cluster
-- Previously updated : 10/09/2019-
-zone_pivot_groups: client-operating-system
--
-# Use intelligent routing and canary releases with Istio in Azure Kubernetes Service (AKS)
-
-[Istio][istio-github] is an open-source service mesh that provides a key set of functionality across the microservices in a Kubernetes cluster. These features include traffic management, service identity and security, policy enforcement, and observability. For more information about Istio, see the official [What is Istio?][istio-docs-concepts] documentation.
-
-This article shows you how to use the traffic management functionality of Istio. A sample AKS voting app is used to explore intelligent routing and canary releases.
-
-In this article, you learn how to:
-
-> [!div class="checklist"]
-> * Deploy the application
-> * Update the application
-> * Roll out a canary release of the application
-> * Finalize the rollout
-
-## Before you begin
-
-> [!NOTE]
-> This scenario has been tested against Istio version `1.3.2`.
-
-The steps detailed in this article assume you've created an AKS cluster (Kubernetes `1.13` and above, with Kubernetes RBAC enabled) and have established a `kubectl` connection with the cluster. You'll also need Istio installed in your cluster.
-
-If you need help with any of these items, then see the [AKS quickstart][aks-quickstart] and [Install Istio in AKS][istio-install] guidance.
-
-## About this application scenario
-
-The sample AKS voting app provides two voting options (**Cats** or **Dogs**) to users. There is a storage component that persists the number of votes for each option. Additionally, there is an analytics component that provides details around the votes cast for each option.
-
-In this application scenario, you start by deploying version `1.0` of the voting app and version `1.0` of the analytics component. The analytics component provides simple counts for the number of votes. The voting app and analytics component interact with version `1.0` of the storage component, which is backed by Redis.
-
-You upgrade the analytics component to version `1.1`, which provides counts, and now totals and percentages.
-
-A subset of users test version `2.0` of the app via a canary release. This new version uses a storage component that is backed by a MySQL database.
-
-Once you're confident that version `2.0` works as expected on your subset of users, you roll out version `2.0` to all your users.
-
-## Deploy the application
-
-Let's start by deploying the application into your Azure Kubernetes Service (AKS) cluster. The following diagram shows what runs by the end of this section - version `1.0` of all components with inbound requests serviced via the Istio ingress gateway:
-
-![Diagram that shows version 1.0 of all components with inbound requests serviced via the Istio ingress gateway.](media/servicemesh/istio/scenario-routing-components-01.png)
-
-The artifacts you need to follow along with this article are available in the [Azure-Samples/aks-voting-app][github-azure-sample] GitHub repo. You can either download the artifacts or clone the repo as follows:
-
-```console
-git clone https://github.com/Azure-Samples/aks-voting-app.git
-```
-
-Change to the following folder in the downloaded / cloned repo and run all subsequent steps from this folder:
-
-```console
-cd aks-voting-app/scenarios/intelligent-routing-with-istio
-```
-
-First, create a namespace in your AKS cluster for the sample AKS voting app named `voting` as follows:
-
-```console
-kubectl create namespace voting
-```
-
-Label the namespace with `istio-injection=enabled`. This label instructs Istio to automatically inject the istio-proxies as sidecars into all of your pods in this namespace.
-
-```console
-kubectl label namespace voting istio-injection=enabled
-```
-
-Now let's create the components for the AKS Voting app. Create these components in the `voting` namespace created in a previous step.
-
-```console
-kubectl apply -f kubernetes/step-1-create-voting-app.yaml --namespace voting
-```
-
-The following example output shows the resources being created:
-
-```output
-deployment.apps/voting-storage-1-0 created
-service/voting-storage created
-deployment.apps/voting-analytics-1-0 created
-service/voting-analytics created
-deployment.apps/voting-app-1-0 created
-service/voting-app created
-```
-
-> [!NOTE]
-> Istio has some specific requirements around pods and services. For more information, see the [Istio Requirements for Pods and Services documentation][istio-requirements-pods-and-services].
-
-To see the pods that have been created, use the [kubectl get pods][kubectl-get] command as follows:
-
-```console
-kubectl get pods -n voting --show-labels
-```
-
-The following example output shows there are three instances of the `voting-app` pod and a single instance of both the `voting-analytics` and `voting-storage` pods. Each of the pods has two containers. One of these containers is the component, and the other is the `istio-proxy`:
-
-```output
-NAME READY STATUS RESTARTS AGE LABELS
-voting-analytics-1-0-57c7fccb44-ng7dl 2/2 Running 0 39s app=voting-analytics,pod-template-hash=57c7fccb44,version=1.0
-voting-app-1-0-956756fd-d5w7z 2/2 Running 0 39s app=voting-app,pod-template-hash=956756fd,version=1.0
-voting-app-1-0-956756fd-f6h69 2/2 Running 0 39s app=voting-app,pod-template-hash=956756fd,version=1.0
-voting-app-1-0-956756fd-wsxvt 2/2 Running 0 39s app=voting-app,pod-template-hash=956756fd,version=1.0
-voting-storage-1-0-5d8fcc89c4-2jhms 2/2 Running 0 39s app=voting-storage,pod-template-hash=5d8fcc89c4,version=1.0
-```
-
-To see information about the pod, we'll use the [kubectl describe pod][kubectl-describe] command with label selectors to select the `voting-analytics` pod. We'll filter the output to show the details of the two containers present in the pod:
----------
-You can't connect to the voting app until you create the Istio [Gateway][istio-reference-gateway] and [Virtual Service][istio-reference-virtualservice]. These Istio resources route traffic from the default Istio ingress gateway to our application.
-
-> [!NOTE]
-> A **Gateway** is a component at the edge of the service mesh that receives inbound or outbound HTTP and TCP traffic.
->
-> A **Virtual Service** defines a set of routing rules for one or more destination services.
-
-Use the `kubectl apply` command to deploy the Gateway and Virtual Service yaml. Remember to specify the namespace that these resources are deployed into.
-
-```console
-kubectl apply -f istio/step-1-create-voting-app-gateway.yaml --namespace voting
-```
-
-The following example output shows the new Gateway and Virtual Service being created:
-
-```output
-virtualservice.networking.istio.io/voting-app created
-gateway.networking.istio.io/voting-app-gateway created
-```
-
-Obtain the IP address of the Istio Ingress Gateway using the following command:
-
-```output
-kubectl get service istio-ingressgateway --namespace istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
-```
-
-The following example output shows the IP address of the Ingress Gateway:
-
-```output
-20.188.211.19
-```
-
-Open up a browser and paste in the IP address. The sample AKS voting app is displayed.
-
-![The AKS Voting app running in our Istio enabled AKS cluster.](media/servicemesh/istio/scenario-routing-deploy-app-01.png)
-
-The information at the bottom of the screen shows that the app uses version `1.0` of `voting-app` and version `1.0` of `voting-storage` (Redis).
-
-## Update the application
-
-Let's deploy a new version of the analytics component. This new version `1.1` displays totals and percentages in addition to the count for each category.
-
-The following diagram shows what will be running at the end of this section - only version `1.1` of our `voting-analytics` component has traffic routed from the `voting-app` component. Even though version `1.0` of our `voting-analytics` component continues to run and is referenced by the `voting-analytics` service, the Istio proxies disallow traffic to and from it.
-
-![Diagram that shows only version 1.1 of the voting-analytics component has traffic routed from the voting-app component.](media/servicemesh/istio/scenario-routing-components-02.png)
-
-Let's deploy version `1.1` of the `voting-analytics` component. Create this component in the `voting` namespace:
-
-```console
-kubectl apply -f kubernetes/step-2-update-voting-analytics-to-1.1.yaml --namespace voting
-```
-
-The following example output shows the resources being created:
-
-```output
-deployment.apps/voting-analytics-1-1 created
-```
-
-Open the sample AKS voting app in a browser again, using the IP address of the Istio Ingress Gateway obtained in the previous step.
-
-Your browser alternates between the two views shown below. Since you are using a Kubernetes [Service][kubernetes-service] for the `voting-analytics` component with only a single label selector (`app: voting-analytics`), Kubernetes uses the default behavior of round-robin between the pods that match that selector. In this case, it is both version `1.0` and `1.1` of your `voting-analytics` pods.
-
-![Version 1.0 of the analytics component running in our AKS Voting app.](media/servicemesh/istio/scenario-routing-deploy-app-01.png)
-
-![Version 1.1 of the analytics component running in our AKS Voting app.](media/servicemesh/istio/scenario-routing-update-app-01.png)
-
-You can visualize the switching between the two versions of the `voting-analytics` component as follows. Remember to use the IP address of your own Istio Ingress Gateway.
----------
-The following example output shows the relevant part of the returned web site as the site switches between versions:
-
-```output
- <div id="results"> Cats: 2 | Dogs: 4 </div>
- <div id="results"> Cats: 2 | Dogs: 4 </div>
- <div id="results"> Cats: 2/6 (33%) | Dogs: 4/6 (67%) </div>
- <div id="results"> Cats: 2 | Dogs: 4 </div>
- <div id="results"> Cats: 2/6 (33%) | Dogs: 4/6 (67%) </div>
-```
-
-### Lock down traffic to version 1.1 of the application
-
-Now let's lock down traffic to only version `1.1` of the `voting-analytics` component and to version `1.0` of the `voting-storage` component. You then define routing rules for all of the other components.
-
-> * A **Virtual Service** defines a set of routing rules for one or more destination services.
-> * A **Destination Rule** defines traffic policies and version specific policies.
-> * A **Policy** defines what authentication methods can be accepted on workload(s).
-
-Use the `kubectl apply` command to replace the Virtual Service definition on your `voting-app` and add [Destination Rules][istio-reference-destinationrule] and [Virtual Services][istio-reference-virtualservice] for the other components. You will add a [Policy][istio-reference-policy] to the `voting` namespace to ensure that all communicate between services is secured using mutual TLS and client certificates.
-
-* The Policy has `peers.mtls.mode` set to `STRICT` to ensure that mutual TLS is enforced between your services within the `voting` namespace.
-* We also set the `trafficPolicy.tls.mode` to `ISTIO_MUTUAL` in all our Destination Rules. Istio provides services with strong identities and secures communications between services using mutual TLS and client certificates that Istio transparently manages.
-
-```console
-kubectl apply -f istio/step-2-update-and-add-routing-for-all-components.yaml --namespace voting
-```
-
-The following example output shows the new Policy, Destination Rules, and Virtual Services being updated/created:
-
-```output
-virtualservice.networking.istio.io/voting-app configured
-policy.authentication.istio.io/default created
-destinationrule.networking.istio.io/voting-app created
-destinationrule.networking.istio.io/voting-analytics created
-virtualservice.networking.istio.io/voting-analytics created
-destinationrule.networking.istio.io/voting-storage created
-virtualservice.networking.istio.io/voting-storage created
-```
-
-If you open the AKS Voting app in a browser again, only the new version `1.1` of the `voting-analytics` component is used by the `voting-app` component.
-
-![Version 1.1 of the analytics component running in our AKS Voting app.](media/servicemesh/istio/scenario-routing-update-app-01.png)
-
-You can visualize that you are now only routed to version `1.1` of your `voting-analytics` component as follows. Remember to use the IP address of your own Istio Ingress Gateway:
----------
-The following example output shows the relevant part of the returned web site:
-
-```output
- <div id="results"> Cats: 2/6 (33%) | Dogs: 4/6 (67%) </div>
- <div id="results"> Cats: 2/6 (33%) | Dogs: 4/6 (67%) </div>
- <div id="results"> Cats: 2/6 (33%) | Dogs: 4/6 (67%) </div>
- <div id="results"> Cats: 2/6 (33%) | Dogs: 4/6 (67%) </div>
- <div id="results"> Cats: 2/6 (33%) | Dogs: 4/6 (67%) </div>
-```
-
-Let's now confirm that Istio is using mutual TLS to secure communications between each of our services. For this we will use the [authn tls-check][istioctl-authn-tls-check] command on the `istioctl` client binary, which takes the following form.
-
-```console
-istioctl authn tls-check <pod-name[.namespace]> [<service>]
-```
-
-This set of commands provide information about the access to the specified services, from all pods that are in a namespace and match a set of labels:
----------
-This following example output shows that mutual TLS is enforced for each of our queries above. The output also shows the Policy and Destination Rules that enforces the mutual TLS:
-
-```output
-# mTLS configuration between istio ingress pods and the voting-app service
-HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
-voting-app.voting.svc.cluster.local:8080 OK mTLS mTLS default/voting voting-app/voting
-
-# mTLS configuration between each of the voting-app pods and the voting-analytics service
-HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
-voting-analytics.voting.svc.cluster.local:8080 OK mTLS mTLS default/voting voting-analytics/voting
-HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
-voting-analytics.voting.svc.cluster.local:8080 OK mTLS mTLS default/voting voting-analytics/voting
-HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
-voting-analytics.voting.svc.cluster.local:8080 OK mTLS mTLS default/voting voting-analytics/voting
-
-# mTLS configuration between each of the voting-app pods and the voting-storage service
-HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
-voting-storage.voting.svc.cluster.local:6379 OK mTLS mTLS default/voting voting-storage/voting
-HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
-voting-storage.voting.svc.cluster.local:6379 OK mTLS mTLS default/voting voting-storage/voting
-HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
-voting-storage.voting.svc.cluster.local:6379 OK mTLS mTLS default/voting voting-storage/voting
-
-# mTLS configuration between each of the voting-analytics version 1.1 pods and the voting-storage service
-HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
-voting-storage.voting.svc.cluster.local:6379 OK mTLS mTLS default/voting voting-storage/voting
-```
-
-## Roll out a canary release of the application
-
-Now let's deploy a new version `2.0` of the `voting-app`, `voting-analytics`, and `voting-storage` components. The new `voting-storage` component use MySQL instead of Redis, and the `voting-app` and `voting-analytics` components are updated to allow them to use this new `voting-storage` component.
-
-The `voting-app` component now supports feature flag functionality. This feature flag allows you to test the canary release capability of Istio for a subset of users.
-
-The following diagram shows what you will have running at the end of this section.
-
-* Version `1.0` of the `voting-app` component, version `1.1` of the `voting-analytics` component and version `1.0` of the `voting-storage` component are able to communicate with each other.
-* Version `2.0` of the `voting-app` component, version `2.0` of the `voting-analytics` component and version `2.0` of the `voting-storage` component are able to communicate with each other.
-* Version `2.0` of the `voting-app` component are only accessible to users that have a specific feature flag set. This change is managed using a feature flag via a cookie.
-
-![Diagram that shows what you'll have running at the end of this section.](media/servicemesh/istio/scenario-routing-components-03.png)
-
-First, update the Istio Destination Rules and Virtual Services to cater for these new components. These updates ensure that you don't route traffic incorrectly to the new components and users don't get unexpected access:
-
-```console
-kubectl apply -f istio/step-3-add-routing-for-2.0-components.yaml --namespace voting
-```
-
-The following example output shows the Destination Rules and Virtual Services being updated:
-
-```output
-destinationrule.networking.istio.io/voting-app configured
-virtualservice.networking.istio.io/voting-app configured
-destinationrule.networking.istio.io/voting-analytics configured
-virtualservice.networking.istio.io/voting-analytics configured
-destinationrule.networking.istio.io/voting-storage configured
-virtualservice.networking.istio.io/voting-storage configured
-```
-
-Next, let's add the Kubernetes objects for the new version `2.0` components. You also update the `voting-storage` service to include the `3306` port for MySQL:
-
-```console
-kubectl apply -f kubernetes/step-3-update-voting-app-with-new-storage.yaml --namespace voting
-```
-
-The following example output shows the Kubernetes objects are successfully updated or created:
-
-```output
-service/voting-storage configured
-secret/voting-storage-secret created
-deployment.apps/voting-storage-2-0 created
-persistentvolumeclaim/mysql-pv-claim created
-deployment.apps/voting-analytics-2-0 created
-deployment.apps/voting-app-2-0 created
-```
-
-Wait until all the version `2.0` pods are running. Use the [kubectl get pods][kubectl-get] command with the `-w` watch switch to watch for changes on all pods in the `voting` namespace:
-
-```console
-kubectl get pods --namespace voting -w
-```
-
-You should now be able to switch between the version `1.0` and version `2.0` (canary) of the voting application. The feature flag toggle at the bottom of the screen sets a cookie. This cookie is used by the `voting-app` Virtual Service to route users to the new version `2.0`.
-
-![Version 1.0 of the AKS Voting app - feature flag IS NOT set.](media/servicemesh/istio/scenario-routing-canary-release-01.png)
-
-![Version 2.0 of the AKS Voting app - feature flag IS set.](media/servicemesh/istio/scenario-routing-canary-release-02.png)
-
-The vote counts are different between the versions of the app. This difference highlights that you are using two different storage backends.
-
-## Finalize the rollout
-
-Once you've successfully tested the canary release, update the `voting-app` Virtual Service to route all traffic to version `2.0` of the `voting-app` component. All users then see version `2.0` of the application, regardless of whether the feature flag is set or not:
-
-![Diagram that shows that users see version 2.0 of the application, regardless of whether the feature flag is set or not.](media/servicemesh/istio/scenario-routing-components-04.png)
-
-Update all the Destination Rules to remove the versions of the components you no longer want active. Then, update all the Virtual Services to stop referencing those versions.
-
-Since there's no longer any traffic to any of the older versions of the components, you can now safely delete all the deployments for those components.
-
-![The AKS Voting app components and routing.](media/servicemesh/istio/scenario-routing-components-05.png)
-
-You have now successfully rolled out a new version of the AKS Voting App.
-
-## Clean up
-
-You can remove the AKS voting app we used in this scenario from your AKS cluster by deleting the `voting` namespace as follows:
-
-```console
-kubectl delete namespace voting
-```
-
-The following example output shows that all the components of the AKS voting app have been removed from your AKS cluster.
-
-```output
-namespace "voting" deleted
-```
-
-## Next steps
-
-You can explore additional scenarios using the [Istio Bookinfo Application example][istio-bookinfo-example].
-
-<!-- LINKS - external -->
-[github-azure-sample]: https://github.com/Azure-Samples/aks-voting-app
-[istio-github]: https://github.com/istio/istio
-
-[istio]: https://istio.io
-[istio-docs-concepts]: https://istio.io/docs/concepts/what-is-istio/
-[istio-requirements-pods-and-services]: https://istio.io/docs/setup/kubernetes/prepare/requirements/
-[istio-reference-gateway]: https://istio.io/docs/reference/config/networking/v1alpha3/gateway/
-[istio-reference-policy]: https://istio.io/docs/reference/config/istio.mesh.v1alpha1/#AuthenticationPolicy
-[istio-reference-virtualservice]: https://istio.io/docs/reference/config/networking/v1alpha3/virtual-service/
-[istio-reference-destinationrule]: https://istio.io/docs/reference/config/networking/v1alpha3/destination-rule/
-[istio-bookinfo-example]: https://istio.io/docs/examples/bookinfo/
-[istioctl-authn-tls-check]: https://istio.io/docs/reference/commands/istioctl/#istioctl-authn-tls-check
-
-[kubernetes-service]: https://kubernetes.io/docs/concepts/services-networking/service/
-[kubectl-get]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get
-[kubectl-describe]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe
-
-<!-- LINKS - internal -->
-[aks-quickstart]: ./kubernetes-walkthrough.md
-[istio-install]: ./servicemesh-istio-install.md
aks Servicemesh Linkerd About https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/aks/servicemesh-linkerd-about.md
- Title: Overview of Linkerd
-description: Obtain an overview of Linkerd
-- Previously updated : 10/09/2019---
-# Linkerd
-
-## Overview
-
-[Linkerd][linkerd] is an easy to use and lightweight service mesh.
-
-## Architecture
-
-Linkerd provides a data plane that is composed of ultralight [Linkerd][linkerd-proxy] specialised proxy sidecars. These intelligent proxies control all network traffic in and out of your meshed apps and workloads. The proxies also expose metrics via [Prometheus][prometheus] metrics endpoints.
-
-The control plane manages the configuration and aggregated telemetry via the following [components][linkerd-architecture]:
--- **Controller** - Provides api that drives the Linkerd CLI and Dashboard. Provides configuration for proxies.--- **Tap** - Establish real-time watches on requests and responses.--- **Identity** - Provides identity and security capabilities that allow for mTLS between services.--- **Web** - Provides the Linkerd dashboard.--
-The following architecture diagram demonstrates how the various components within the data plane and control plane interact.
--
-![Overview of Linkerd components and architecture.](media/servicemesh/linkerd/about-architecture.png)
--
-## Selection criteria
-
-It's important to understand and consider the following areas when evaluating Linkerd for your workloads:
--- [Design Principles](#design-principles)-- [Capabilities](#capabilities)-- [Scenarios](#scenarios)--
-### Design principles
-
-The following design principles [guide][design-principles] the Linkerd project:
--- **Keep it Simple** - Must be easy to use and understand.--- **Minimize Resource Requirements** - Impose minimal performance and resource cost.--- **Just Work** - Don't break existing applications and don't require complex configuration.--
-### Capabilities
-
-Linkerd provides the following set of capabilities:
--- **Mesh** ΓÇô built in debugging option--- **Traffic Management** ΓÇô splitting, timeouts, retries, ingress--- **Security** ΓÇô encryption (mTLS), certificates autorotated every 24 hours--- **Observability** ΓÇô golden metrics, tap, tracing, service profiles and per route metrics, web dashboard with topology graphs, prometheus, grafana--
-### Scenarios
-
-Linkerd is well suited to and suggested for the following scenarios:
--- Simple to use with just the essential set of capability requirements--- Low latency, low overhead, with focus on observability and simple traffic management--
-## Next steps
-
-The following documentation describes how you can install Linkerd on Azure Kubernetes Service (AKS):
-
-> [!div class="nextstepaction"]
-> [Install Linkerd in Azure Kubernetes Service (AKS)][linkerd-install]
-
-You can also further explore Linkerd features and architecture:
--- [Linkerd Features][linkerd-features]-- [Linkerd Architecture][linkerd-architecture]-
-<!-- LINKS - external -->
-[linkerd]: https://linkerd.io/2/overview/
-[linkerd-architecture]: https://linkerd.io/2/reference/architecture/
-[linkerd-features]: https://linkerd.io/2/features/
-[design-principles]: https://linkerd.io/2/design-principles/
-[linkerd-proxy]: https://github.com/linkerd/linkerd2-proxy
-
-[grafana]: https://grafana.com/
-[prometheus]: https://prometheus.io/
-
-<!-- LINKS - internal -->
-[linkerd-install]: ./servicemesh-linkerd-install.md
aks Servicemesh Linkerd Install https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/aks/servicemesh-linkerd-install.md
- Title: Install Linkerd in Azure Kubernetes Service (AKS)
-description: Learn how to install and use Linkerd to create a service mesh in an Azure Kubernetes Service (AKS) cluster
-- Previously updated : 10/09/2019-
-zone_pivot_groups: client-operating-system
--
-# Install Linkerd in Azure Kubernetes Service (AKS)
-
-[Linkerd][linkerd-github] is an open-source service mesh and [CNCF incubating project][linkerd-cncf]. Linkerd is an ultralight service mesh that provides features that include traffic management, service identity and security, reliability, and observability. For more information about Linkerd, see the official [Linkerd FAQ][linkerd-faq] and [Linkerd Architecture][linkerd-architecture] documentation.
-
-This article shows you how to install Linkerd. The Linkerd `linkerd` client binary is installed onto your client machine and the Linkerd components are installed into a Kubernetes cluster on AKS.
-
-> [!NOTE]
-> These instructions reference Linkerd version `stable-2.6.0`.
->
-> The Linkerd `stable-2.6.x` can be run against Kubernetes versions `1.13+`. You can find additional stable and edge Linkerd versions at [GitHub - Linkerd Releases][linkerd-github-releases].
-
-In this article, you learn how to:
-
-> [!div class="checklist"]
-> * Download and install the Linkerd linkerd client binary
-> * Install Linkerd on AKS
-> * Validate the Linkerd installation
-> * Access the Dashboard
-> * Uninstall Linkerd from AKS
-
-## Before you begin
-
-The steps detailed in this article assume that you've created an AKS cluster (Kubernetes `1.13` and above, with Kubernetes RBAC enabled) and have established a `kubectl` connection with the cluster. If you need help with any of these items, then see the [AKS quickstart][aks-quickstart].
-
-All Linkerd pods must be scheduled to run on Linux nodes - this setup is the default in the installation method detailed below and requires no additional configuration.
-
-This article separates the Linkerd installation guidance into several discrete steps. The result is the same in structure as the official Linkerd getting started [guidance][linkerd-getting-started].
----------
-## Install Linkerd on AKS
-
-Before we install Linkerd, we'll run pre-installation checks to determine if the control plane can be installed on our AKS cluster:
-
-```console
-linkerd check --pre
-```
-
-You should see something like the following to indicate that your AKS cluster is a valid installation target for Linkerd:
-
-```console
-kubernetes-api
-√ can initialize the client
-√ can query the Kubernetes API
-
-kubernetes-version
-
-√ is running the minimum Kubernetes API version
-√ is running the minimum kubectl version
-
-pre-kubernetes-setup
-√ control plane namespace does not already exist
-√ can create Namespaces
-√ can create ClusterRoles
-√ can create ClusterRoleBindings
-√ can create CustomResourceDefinitions
-√ can create PodSecurityPolicies
-√ can create ServiceAccounts
-√ can create Services
-√ can create Deployments
-√ can create CronJobs
-√ can create ConfigMaps
-√ no clock skew detected
-
-pre-kubernetes-capability
--
-√ has NET_ADMIN capability
-√ has NET_RAW capability
-
-pre-linkerd-global-resources
--
-√ no ClusterRoles exist
-√ no ClusterRoleBindings exist
-√ no CustomResourceDefinitions exist
-√ no MutatingWebhookConfigurations exist
-√ no ValidatingWebhookConfigurations exist
-√ no PodSecurityPolicies exist
-
-linkerd-version
-
-√ can determine the latest version
-√ cli is up-to-date
-
-Status check results are √
-```
-
-Now it's time to install the Linkerd components. Use the `linkerd` and `kubectl` binaries to install the Linkerd components into your AKS cluster. A `linkerd` namespace will be automatically created, and the components will be installed into this namespace.
-
-```console
-linkerd install | kubectl apply -f -
-```
-
-Linkerd deploys a number of objects. You'll see the list from the output of your `linkerd install` command above. The deployment of the Linkerd components should take around 1 minute to complete, depending on your cluster environment.
-
-At this point, you've deployed Linkerd to your AKS cluster. To ensure we have a successful deployment of Linkerd, let's move on to the next section to [Validate the Linkerd installation](#validate-the-linkerd-installation).
-
-## Validate the Linkerd installation
-
-Confirm that the resources have been successfully created. Use the [kubectl get svc][kubectl-get] and [kubectl get pod][kubectl-get] commands to query the `linkerd` namespace, where the Linkerd components were installed by the `linkerd install` command:
-
-```console
-kubectl get svc --namespace linkerd --output wide
-kubectl get pod --namespace linkerd --output wide
-```
-
-The following example output shows the services and pods (scheduled on Linux nodes) that should now be running:
-
-```console
-NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
-linkerd-controller-api ClusterIP 10.0.110.67 <none> 8085/TCP 66s linkerd.io/control-plane-component=controller
-linkerd-destination ClusterIP 10.0.224.29 <none> 8086/TCP 66s linkerd.io/control-plane-component=controller
-linkerd-dst ClusterIP 10.0.225.148 <none> 8086/TCP 66s linkerd.io/control-plane-component=destination
-linkerd-grafana ClusterIP 10.0.61.124 <none> 3000/TCP 65s linkerd.io/control-plane-component=grafana
-linkerd-identity ClusterIP 10.0.6.104 <none> 8080/TCP 67s linkerd.io/control-plane-component=identity
-linkerd-prometheus ClusterIP 10.0.27.168 <none> 9090/TCP 65s linkerd.io/control-plane-component=prometheus
-linkerd-proxy-injector ClusterIP 10.0.100.133 <none> 443/TCP 64s linkerd.io/control-plane-component=proxy-injector
-linkerd-sp-validator ClusterIP 10.0.221.5 <none> 443/TCP 64s linkerd.io/control-plane-component=sp-validator
-linkerd-tap ClusterIP 10.0.18.14 <none> 8088/TCP,443/TCP 64s linkerd.io/control-plane-component=tap
-linkerd-web ClusterIP 10.0.37.108 <none> 8084/TCP,9994/TCP 66s linkerd.io/control-plane-component=web
-
-NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
-linkerd-controller-66ddc9f94f-cm9kt 3/3 Running 0 66s 10.240.0.50 aks-linux-16165125-vmss000001 <none> <none>
-linkerd-destination-c94bc454-qpkng 2/2 Running 0 66s 10.240.0.78 aks-linux-16165125-vmss000002 <none> <none>
-linkerd-grafana-6868fdcb66-4cmq2 2/2 Running 0 65s 10.240.0.69 aks-linux-16165125-vmss000002 <none> <none>
-linkerd-identity-74d8df4b85-tqq8f 2/2 Running 0 66s 10.240.0.48 aks-linux-16165125-vmss000001 <none> <none>
-linkerd-prometheus-699587cf8-k8ghg 2/2 Running 0 65s 10.240.0.41 aks-linux-16165125-vmss000001 <none> <none>
-linkerd-proxy-injector-6556447f64-n29wr 2/2 Running 0 64s 10.240.0.32 aks-linux-16165125-vmss000000 <none> <none>
-linkerd-sp-validator-56745cd567-v4x7h 2/2 Running 0 64s 10.240.0.6 aks-linux-16165125-vmss000000 <none> <none>
-linkerd-tap-5cd9fc566-ct988 2/2 Running 0 64s 10.240.0.15 aks-linux-16165125-vmss000000 <none> <none>
-linkerd-web-774c79b6d5-dhhwf 2/2 Running 0 65s 10.240.0.70 aks-linux-16165125-vmss000002 <none> <none>
-```
-
-Linkerd provides a command via the `linkerd` client binary to validate that the Linkerd control plane was successfully installed and configured.
-
-```console
-linkerd check
-```
-
-You should see something like the following to indicate that your installation was successful:
-
-```console
-kubernetes-api
-√ can initialize the client
-√ can query the Kubernetes API
-
-kubernetes-version
-
-√ is running the minimum Kubernetes API version
-√ is running the minimum kubectl version
-
-linkerd-config
-√ control plane Namespace exists
-√ control plane ClusterRoles exist
-√ control plane ClusterRoleBindings exist
-√ control plane ServiceAccounts exist
-√ control plane CustomResourceDefinitions exist
-√ control plane MutatingWebhookConfigurations exist
-√ control plane ValidatingWebhookConfigurations exist
-√ control plane PodSecurityPolicies exist
-
-linkerd-existence
-√ 'linkerd-config' config map exists
-√ heartbeat ServiceAccount exist
-√ control plane replica sets are ready
-√ no unschedulable pods
-√ controller pod is running
-√ can initialize the client
-√ can query the control plane API
-
-linkerd-api
-√ control plane pods are ready
-√ control plane self-check
-√ [kubernetes] control plane can talk to Kubernetes
-√ [prometheus] control plane can talk to Prometheus
-√ no invalid service profiles
-
-linkerd-version
-
-√ can determine the latest version
-√ cli is up-to-date
-
-control-plane-version
-
-√ control plane is up-to-date
-√ control plane and cli versions match
-
-Status check results are √
-```
-
-## Access the dashboard
-
-Linkerd comes with a dashboard that provides insight into the service mesh and workloads. To access the dashboard, use the `linkerd dashboard` command. This command leverages [kubectl port-forward][kubectl-port-forward] to create a secure connection between your client machine and the relevant pods in your AKS cluster. It will then automatically open the dashboard in your default browser.
-
-```console
-linkerd dashboard
-```
-
-The command will also create a port-forward and return a link for the Grafana dashboards.
-
-```console
-Linkerd dashboard available at:
-http://127.0.0.1:50750
-Grafana dashboard available at:
-http://127.0.0.1:50750/grafana
-Opening Linkerd dashboard in the default browser
-```
-
-## Uninstall Linkerd from AKS
-
-> [!WARNING]
-> Deleting Linkerd from a running system may result in traffic related issues between your services. Ensure that you have made provisions for your system to still operate correctly without Linkerd before proceeding.
-
-First you'll need to remove the data plane proxies. Remove any Automatic Proxy Injection [annotations][linkerd-automatic-proxy-injection] from workload namespaces and roll out your workload deployments. Your workloads should no longer have any associated data plane components.
-
-Finally, remove the control plane as follows:
-
-```console
-linkerd install --ignore-cluster | kubectl delete -f -
-```
-
-## Next steps
-
-To explore more installation and configuration options for Linkerd, see the following official Linkerd guidance:
--- [Linkerd - Helm installation][linkerd-install-with-helm]-- [Linkerd - Multi-stage installation to cater for role privileges][linkerd-multi-stage-installation]-
-You can also follow additional scenarios using:
--- [Linkerd emojivoto demo][linkerd-demo-emojivoto]-- [Linkerd books demo][linkerd-demo-books]-
-<!-- LINKS - external -->
-
-[linkerd]: https://linkerd.io/
-[linkerd-cncf]: https://landscape.cncf.io/?selected=linkerd
-[linkerd-faq]: https://linkerd.io/2/faq/
-[linkerd-architecture]: https://linkerd.io/2/reference/architecture/
-[linkerd-getting-started]: https://linkerd.io/2/getting-started/
-[linkerd-overview]: https://linkerd.io/2/overview/
-[linkerd-github]: https://github.com/linkerd/linkerd2
-[linkerd-github-releases]: https://github.com/linkerd/linkerd2/releases/
-
-[linkerd-install-with-helm]: https://linkerd.io/2/tasks/install-helm/
-[linkerd-multi-stage-installation]: https://linkerd.io/2/tasks/install/#multi-stage-install
-[linkerd-automatic-proxy-injection]: https://linkerd.io/2/features/proxy-injection/
-
-[linkerd-demo-emojivoto]: https://linkerd.io/2/getting-started/#step-5-install-the-demo-app
-[linkerd-demo-books]: https://linkerd.io/2/tasks/books/
-
-[helm]: https://helm.sh
-
-[kubectl-get]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get
-[kubectl-port-forward]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward
-
-<!-- LINKS - internal -->
-[aks-quickstart]: ./kubernetes-walkthrough.md
app-service Manage Create Arc Environment https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/app-service/manage-create-arc-environment.md
az extension add --yes --source "https://aka.ms/appsvc/appservice_kube-latest-py
1. Create a cluster in Azure Kubernetes Service with a public IP address. Replace `<group-name>` with the resource group name you want.
+ # [bash](#tab/bash)
+ ```azurecli-interactive aksClusterGroupName="<group-name>" # Name of resource group for the AKS cluster aksName="${aksClusterGroupName}-aks" # Name of the AKS cluster
az extension add --yes --source "https://aka.ms/appsvc/appservice_kube-latest-py
az network public-ip create --resource-group $infra_rg --name MyPublicIP --sku STANDARD staticIp=$(az network public-ip show --resource-group $infra_rg --name MyPublicIP --output tsv --query ipAddress) ```+
+ # [PowerShell](#tab/powershell)
+
+ ```powershell
+ $aksClusterGroupName="<group-name>" # Name of resource group for the AKS cluster
+ $aksName="${aksClusterGroupName}-aks" # Name of the AKS cluster
+ $resourceLocation="eastus" # "eastus" or "westeurope"
+
+ az group create -g $aksClusterGroupName -l $resourceLocation
+ az aks create --resource-group $aksClusterGroupName --name $aksName --enable-aad --generate-ssh-keys
+ $infra_rg=$(az aks show --resource-group $aksClusterGroupName --name $aksName --output tsv --query nodeResourceGroup)
+ az network public-ip create --resource-group $infra_rg --name MyPublicIP --sku STANDARD
+ $staticIp=$(az network public-ip show --resource-group $infra_rg --name MyPublicIP --output tsv --query ipAddress)
+ ```
+
+
2. Get the [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) file and test your connection to the cluster. By default, the kubeconfig file is saved to `~/.kube/config`.
az extension add --yes --source "https://aka.ms/appsvc/appservice_kube-latest-py
3. Create a resource group to contain your Azure Arc resources. Replace `<group-name>` with the resource group name you want.
+ # [bash](#tab/bash)
+ ```azurecli-interactive groupName="<group-name>" # Name of resource group for the connected cluster az group create -g $groupName -l $resourceLocation ```+
+ # [PowerShell](#tab/powershell)
+
+ ```powershell
+ $groupName="<group-name>" # Name of resource group for the connected cluster
+
+ az group create -g $groupName -l $resourceLocation
+ ```
+
+
4. Connect the cluster you created to Azure Arc.
+ # [bash](#tab/bash)
+ ```azurecli-interactive clusterName="${groupName}-cluster" # Name of the connected cluster resource az connectedk8s connect --resource-group $groupName --name $clusterName ```
+ # [PowerShell](#tab/powershell)
++
+ ```powershell
+ $clusterName="${groupName}-cluster" # Name of the connected cluster resource
+
+ az connectedk8s connect --resource-group $groupName --name $clusterName
+ ```
+
+
+
5. Validate the connection with the following command. It should show the `provisioningState` property as `Succeeded`. If not, run the command again after a minute. ```azurecli-interactive
While a [Log Analytic workspace](../azure-monitor/logs/quick-create-workspace.md
1. For simplicity, create the workspace now.
+ # [bash](#tab/bash)
+ ```azurecli-interactive workspaceName="$groupName-workspace" # Name of the Log Analytics workspace
While a [Log Analytic workspace](../azure-monitor/logs/quick-create-workspace.md
--resource-group $groupName \ --workspace-name $workspaceName ```+
+ # [PowerShell](#tab/powershell)
+
+ ```powershell
+ $workspaceName="$groupName-workspace"
+
+ az monitor log-analytics workspace create `
+ --resource-group $groupName `
+ --workspace-name $workspaceName
+ ```
+
+
2. Run the following commands to get the encoded workspace ID and shared key for an existing Log Analytics workspace. You need them in the next step.
+ # [bash](#tab/bash)
+ ```azurecli-interactive logAnalyticsWorkspaceId=$(az monitor log-analytics workspace show \ --resource-group $groupName \
While a [Log Analytic workspace](../azure-monitor/logs/quick-create-workspace.md
logAnalyticsKeyEncWithSpace=$(printf %s $logAnalyticsKey | base64) logAnalyticsKeyEnc=$(echo -n "${logAnalyticsKeyEncWithSpace//[[:space:]]/}") # Needed for the next step ```+
+ # [PowerShell](#tab/powershell)
+
+ ```powershell
+ $logAnalyticsWorkspaceId=$(az monitor log-analytics workspace show `
+ --resource-group $groupName `
+ --workspace-name $workspaceName `
+ --query customerId `
+ --output tsv)
+ $logAnalyticsWorkspaceIdEnc=[Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($logAnalyticsWorkspaceId))# Needed for the next step
+ $logAnalyticsKey=$(az monitor log-analytics workspace get-shared-keys `
+ --resource-group $groupName `
+ --workspace-name $workspaceName `
+ --query primarySharedKey `
+ --output tsv)
+ $logAnalyticsKeyEncWithSpace=[Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($logAnalyticsKey))
+ $logAnalyticsKeyEnc=$(echo -n "${logAnalyticsKeyEncWithSpace//[[:space:]]/}") # Needed for the next step
+ ```
+
+ ## Install the App Service extension 1. Set the following environment variables for the desired name of the [App Service extension](overview-arc-integration.md), the cluster namespace in which resources should be provisioned, and the name for the App Service Kubernetes environment. Choose a unique name for `<kube-environment-name>`, because it will be part of the domain name for app created in the App Service Kubernetes environment.
+ # [bash](#tab/bash)
+ ```bash extensionName="appservice-ext" # Name of the App Service extension namespace="appservice-ns" # Namespace in your cluster to install the extension and provision resources kubeEnvironmentName="<kube-environment-name>" # Name of the App Service Kubernetes environment resource ```+
+ # [PowerShell](#tab/powershell)
+
+ ```powershell
+ $extensionName="appservice-ext" # Name of the App Service extension
+ $namespace="appservice-ns" # Namespace in your cluster to install the extension and provision resources
+ $kubeEnvironmentName="<kube-environment-name>" # Name of the App Service Kubernetes environment resource
+ ```
+
+
2. Install the App Service extension to your Azure Arc connected cluster, with Log Analytics enabled. Again, while Log Analytics is not required, you can't add it to the extension later, so it's easier to do it now.
+ # [bash](#tab/bash)
+ ```azurecli-interactive az k8s-extension create \ --resource-group $groupName \
While a [Log Analytic workspace](../azure-monitor/logs/quick-create-workspace.md
--configuration-protected-settings "logProcessor.appLogs.logAnalyticsConfig.sharedKey=${logAnalyticsKeyEnc}" ```
+ # [PowerShell](#tab/powershell)
+
+ ```powershell
+ az k8s-extension create `
+ --resource-group $groupName `
+ --name $extensionName `
+ --cluster-type connectedClusters `
+ --cluster-name $clusterName `
+ --extension-type 'Microsoft.Web.Appservice' `
+ --release-train stable `
+ --auto-upgrade-minor-version true `
+ --scope cluster `
+ --release-namespace $namespace `
+ --configuration-settings "Microsoft.CustomLocation.ServiceAccount=default" `
+ --configuration-settings "appsNamespace=${namespace}" `
+ --configuration-settings "clusterName=${kubeEnvironmentName}" `
+ --configuration-settings "loadBalancerIp=${staticIp}" `
+ --configuration-settings "keda.enabled=true" `
+ --configuration-settings "buildService.storageClassName=default" `
+ --configuration-settings "buildService.storageAccessMode=ReadWriteOnce" `
+ --configuration-settings "customConfigMap=${namespace}/kube-environment-config" `
+ --configuration-settings "envoy.annotations.service.beta.kubernetes.io/azure-load-balancer-resource-group=${aksClusterGroupName}" `
+ --configuration-settings "logProcessor.appLogs.destination=log-analytics" `
+ --configuration-protected-settings "logProcessor.appLogs.logAnalyticsConfig.customerId=${logAnalyticsWorkspaceIdEnc}" `
+ --configuration-protected-settings "logProcessor.appLogs.logAnalyticsConfig.sharedKey=${logAnalyticsKeyEnc}"
+ ```
+
+
+ > [!NOTE] > To install the extension without Log Analytics integration, remove the last three `--configuration-settings` parameters from the command. >
While a [Log Analytic workspace](../azure-monitor/logs/quick-create-workspace.md
3. Save the `id` property of the App Service extension for later.
+ # [bash](#tab/bash)
+ ```azurecli-interactive extensionId=$(az k8s-extension show \ --cluster-type connectedClusters \
While a [Log Analytic workspace](../azure-monitor/logs/quick-create-workspace.md
--output tsv) ```
+ # [PowerShell](#tab/powershell)
+
+ ```powershell
+ $extensionId=$(az k8s-extension show `
+ --cluster-type connectedClusters `
+ --cluster-name $clusterName `
+ --resource-group $groupName `
+ --name $extensionName `
+ --query id `
+ --output tsv)
+ ```
+
+
+ 4. Wait for the extension to fully install before proceeding. You can have your terminal session wait until this complete by running the following command: ```azurecli-interactive
The [custom location](../azure-arc/kubernetes/custom-locations.md) in Azure is u
1. Set the following environment variables for the desired name of the custom location and for the ID of the Azure Arc connected cluster.
+ # [bash](#tab/bash)
+ ```bash customLocationName="my-custom-location" # Name of the custom location connectedClusterId=$(az connectedk8s show --resource-group $groupName --name $clusterName --query id --output tsv) ```+
+ # [PowerShell](#tab/powershell)
+
+ ```powershell
+ $customLocationName="my-custom-location" # Name of the custom location
+
+ $connectedClusterId=$(az connectedk8s show --resource-group $groupName --name $clusterName --query id --output tsv)
+ ```
+
+
-3. Create the custom location:
+2. Create the custom location:
+
+ # [bash](#tab/bash)
```azurecli-interactive az customlocation create \
The [custom location](../azure-arc/kubernetes/custom-locations.md) in Azure is u
--namespace $namespace \ --cluster-extension-ids $extensionId ```+
+ # [PowerShell](#tab/powershell)
+
+ ```azurecli-interactive
+ az customlocation create `
+ --resource-group $groupName `
+ --name $customLocationName `
+ --host-resource-id $connectedClusterId `
+ --namespace $namespace `
+ --cluster-extension-ids $extensionId
+ ```
+
+
<!-- --kubeconfig ~/.kube/config # needed for non-Azure -->
-4. Validate that the custom location is successfully created with the following command. The output should show the `provisioningState` property as `Succeeded`. If not, run it again after a minute.
+3. Validate that the custom location is successfully created with the following command. The output should show the `provisioningState` property as `Succeeded`. If not, run it again after a minute.
```azurecli-interactive
- az customlocation show \
- --resource-group $groupName \
- --name $customLocationName
+ az customlocation show --resource-group $groupName --name $customLocationName
```
-5. Save the custom location ID for the next step.
+4. Save the custom location ID for the next step.
+
+ # [bash](#tab/bash)
```azurecli-interactive customLocationId=$(az customlocation show \
The [custom location](../azure-arc/kubernetes/custom-locations.md) in Azure is u
--query id \ --output tsv) ```+
+ # [PowerShell](#tab/powershell)
+
+ ```azurecli-interactive
+ $customLocationId=$(az customlocation show `
+ --resource-group $groupName `
+ --name $customLocationName `
+ --query id `
+ --output tsv)
+ ```
+
+
## Create the App Service Kubernetes environment Before you can start creating apps on the custom location, you need an [App Service Kubernetes environment](overview-arc-integration.md#app-service-kubernetes-environment). 1. Create the App Service Kubernetes environment:
+
+ # [bash](#tab/bash)
```azurecli-interactive az appservice kube create \
Before you can start creating apps on the custom location, you need an [App Serv
--custom-location $customLocationId \ --static-ip $staticIp ```+
+ # [PowerShell](#tab/powershell)
+
+ ```azurecli-interactive
+ az appservice kube create `
+ --resource-group $groupName `
+ --name $kubeEnvironmentName `
+ --custom-location $customLocationId `
+ --static-ip $staticIp
+ ```
+
+
2. Validate that the App Service Kubernetes environment is successfully created with the following command. The output should show the `provisioningState` property as `Succeeded`. If not, run it again after a minute. ```azurecli-interactive
- az appservice kube show \
- --resource-group $groupName \
- --name $kubeEnvironmentName
+ az appservice kube show --resource-group $groupName --name $kubeEnvironmentName
```
automation Enable Managed Identity For Automation https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/automation/enable-managed-identity-for-automation.md
You can enable a system-assigned managed identity for an Azure Automation accoun
$sub = Get-AzSubscription -ErrorAction SilentlyContinue if(-not($sub)) {
- Connect-AzAccount -Subscription
+ Connect-AzAccount -Identity
} # If you have multiple subscriptions, set the one to use
azure-arc Create Data Controller Direct Azure Portal https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-arc/data/create-data-controller-direct-azure-portal.md
# Create Azure Arc data controller from Azure portal - Direct connectivity mode - This article describes how to deploy the Azure Arc data controller in direct connect mode during the current preview of this feature. ## Complete prerequisites
This article describes how to deploy the Azure Arc data controller in direct con
Before you begin, verify that you have completed the prerequisites in [Deploy data controller - direct connect mode - prerequisites](create-data-controller-direct-prerequisites.md). >[!NOTE]
->You first need to deploy an Arc enabled Kubernetes data services extension usign the Azure CLI.
+>You first need to deploy an Arc enabled Kubernetes data services extension using the Azure CLI.
+>
+> To complete this, you will need to identify:
+>
+> - `<connected_cluster_name>` - Name of your cluster.
+> - `<resource_group_name>` - Name of your resource group.
+> - `<namespace>` - The Kubernetes namespace that will contain your data services.
+>
+> Use these values in the following script to create the extension:
> >```azurecli
->az k8s-extension create -c "{connected_cluster_name}" -g "{resource_group_name}" --name "arcdataservices" --cluster-type "connectedClusters" --extension-type "microsoft.arcdataservices" --scope "cluster" --release-namespace {namespace} --config "Microsoft.CustomLocation.ServiceAccount=sa-bootstrapper"
+>az k8s-extension create -c "<connected_cluster_name>" -g "<resource_group_name>" --name "arcdataservices" --cluster-type "connectedClusters" --extension-type "microsoft.arcdataservices" --scope "cluster" --release-namespace "<namespace>" --config "Microsoft.CustomLocation.ServiceAccount=sa-bootstrapper"
>``` - ## Deploy Azure Arc data controller Azure Arc data controller create flow can be launched from the Azure portal in one of the following ways:
azure-arc Upload Logs https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-arc/data/upload-logs.md
SET WORKSPACE_SHARED_KEY=<primarySharedKey>
```console $Env:WORKSPACE_SHARED_KEY='<primarySharedKey>' ```
-```
::: zone-end
azure-arc Upload Metrics And Logs To Azure Monitor https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-arc/data/upload-metrics-and-logs-to-azure-monitor.md
Follow these commands to create your metrics upload service principal:
To create a service principal, update the following example. Replace `<ServicePrincipalName>`, `SubscriptionId` and `resourcegroup` with your values and run the command: ```azurecli
-az ad sp create-for-rbac --name <ServicePrincipalName> --role Contributor --scopes /subscriptions/{SubscriptionId}/resourceGroups/{resourcegroup}
+az ad sp create-for-rbac --name <ServicePrincipalName> --role Contributor --scopes /subscriptions/<SubscriptionId>/resourceGroups/<resourcegroup>
``` If you created the service principal earlier, and just need to get the current credentials, run the following command to reset the credential.
Run this command to assign the service principal to the `Monitoring Metrics Publ
> You need to use double quotes for role names when running from a Windows environment. ```azurecli
-az role assignment create --assignee <appId> --role "Monitoring Metrics Publisher" --scope subscriptions/{SubscriptionID}/resourceGroups/{resourcegroup}
+az role assignment create --assignee <appId> --role "Monitoring Metrics Publisher" --scope subscriptions/<SubscriptionID>/resourceGroups/<resourcegroup>
``` ::: zone-end
az role assignment create --assignee <appId> --role "Monitoring Metrics Publishe
::: zone pivot="client-operating-system-macos-and-linux" ```azurecli
-az role assignment create --assignee <appId> --role 'Monitoring Metrics Publisher' --scope subscriptions/{SubscriptionID}/resourceGroups/{resourcegroup}
+az role assignment create --assignee <appId> --role 'Monitoring Metrics Publisher' --scope subscriptions/<SubscriptionID>/resourceGroups/<resourcegroup>
``` ::: zone-end
az role assignment create --assignee <appId> --role 'Monitoring Metrics Publishe
::: zone pivot="client-operating-system-powershell" ```powershell
-az role assignment create --assignee <appId> --role 'Monitoring Metrics Publisher' --scope subscriptions/{SubscriptionID}/resourceGroups/{resourcegroup}
+az role assignment create --assignee <appId> --role 'Monitoring Metrics Publisher' --scope subscriptions/<SubscriptionID>/resourceGroups/<resourcegroup>
``` ::: zone-end
Example output:
} ```
+## Verify service principal role
+
+```azurecli
+az role assignment list -o table
+```
+ With the service principal assigned to the appropriate role, you can proceed to upload metrics, or user data. ++ ## Upload logs, metrics, or usage data The specific steps for uploading logs, metrics, or usage data vary depending about the type of information you are uploading.
azure-sql Database Export https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-sql/database/database-export.md
Exporting a BACPAC of a database from [Azure SQL Managed Instance](../managed-in
## SQLPackage utility
-To export a database in SQL Database using the [SqlPackage](/sql/tools/sqlpackage) command-line utility, see [Export parameters and properties](/sql/tools/sqlpackage#export-parameters-and-properties). The SQLPackage utility ships with the latest versions of [SQL Server Management Studio](/sql/ssms/download-sql-server-management-studio-ssms) and [SQL Server Data Tools for Visual Studio](/sql/ssdt/download-sql-server-data-tools-ssdt), or you can download the latest version of [SqlPackage](https://www.microsoft.com/download/details.aspx?id=53876) directly from the Microsoft download center.
+To export a database in SQL Database using the [SqlPackage](/sql/tools/sqlpackage) command-line utility, see [Export parameters and properties](/sql/tools/sqlpackage#export-parameters-and-properties). The SQLPackage utility ships with the latest versions of [SQL Server Management Studio](/sql/ssms/download-sql-server-management-studio-ssms) and [SQL Server Data Tools for Visual Studio](/sql/ssdt/download-sql-server-data-tools-ssdt), or you can download the latest version of [SqlPackage](/sql/tools/sqlpackage/sqlpackage-download?view=sql-server-ver15) directly from the Microsoft download center.
We recommend the use of the SQLPackage utility for scale and performance in most production environments. For a SQL Server Customer Advisory Team blog about migrating using BACPAC files, see [Migrating from SQL Server to Azure SQL Database using BACPAC Files](/archive/blogs/sqlcat/migrating-from-sql-server-to-azure-sql-database-using-bacpac-files).
azure-video-analyzer Embed Player In Power Bi https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-video-analyzer/video-analyzer-docs/embed-player-in-power-bi.md
+
+ Title: Embed player widget in Power BI - Azure Video Analyzer
+description: You can use Azure Video Analyzer for continuous video recording or event-based recording. This article talks about how to embed videos in Microsoft Power BI to provide a customizable UI for your users.
++ Last updated : 08/06/2021++
+# Embed player widget in Power BI
+
+Azure Video Analyzer enables you to [record](detect-motion-record-video-clips-cloud.md) video and associated inference metadata to your Video Analyzer cloud resource. Video Analyzer has a [Player Widget](player-widget.md) - an easy-to-embed widget allowing client apps to playback video and inference metadata.
+
+Dashboards are an insightful way to monitor your business and view all your most important metrics at a glance. A Power BI dashboard is a powerful tool to combine video with multiple sources of data including telemetry from IoT Hub. In this tutorial, you will learn how to add one or more player widgets to a dashboard using [Microsoft Power BI](https://powerbi.microsoft.com/) web service.
+
+## Suggested pre-reading
+
+- Azure Video Analyzer [player widget](player-widget.md)
+- Introduction to [Power BI dashboards](https://docs.microsoft.com/power-bi/create-reports/service-dashboards)
+
+## Prerequisites
+
+- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) if you don't already have one.
+- Complete either [Detect motion and record video](detect-motion-record-video-clips-cloud.md) or [Continuous video recording](continuous-video-recording.md) - a pipeline with video sink is required.
+ [!NOTE] Your video analyzer account should have a minimum of one video recorded to proceed. Check for list of videos by logging into your Azure Video Analyzer account > Videos > Video Analyzer section.
+- A [Power BI](https://powerbi.microsoft.com/) account.
+
+## Create a token
+
+1. Follow steps to [create a token](player-widget.md#create-a-token).
+2. Make sure to save values generated for _Issuer, Audience, Key Type, Algorithm, Key Id, RSA Key Modulus, RSA Key Exponent, Token_. You will need these values when creating an access policy below.
+
+## Get embed code for player widget
+
+1. Login to [Azure portal](https://portal.azure.com/) with your credentials. Locate your Video Analyzer account used to complete the prerequisites and open the Video Analyzer account pane.
+2. Follow steps to [Create an access policy](player-widget.md#create-an-access-policy).
+3. Select **Videos** in the **Video Analyzer** section.
+4. Select any video from the list.
+5. Click on **Widget** setup. A pane **Use widget in your application** opens on the right-hand side. Scroll down to **Option 2 ΓÇô using HTML** and copy the code and paste it in a text editor. Click the **Close** button.
+
+ :::image type="content" source="./media/power-bi/widget-code.png" alt-text="Copy widget HTML code":::
+
+6. Edit the HTML code copied in step 5 to replace values for
+ - Token **AVA-API-JWT-TOKEN** - replace with the value of Token that you saved in the ΓÇ£Create a tokenΓÇ¥ step. Ensure to remove the angular brackets.
+ - Optional ΓÇô you can make other UI changes in this code for example - changing the header from ΓÇ£Example Player widgetΓÇ¥ to ΓÇ£Continuous Video RecordingΓÇ¥.
+
+## Add widget in Power BI dashboard
+
+1. Open the [Power BI service](http://app.powerbi.com/) in your browser. From the navigation pane, select **My Workspace**
+
+ :::image type="content" source="./media/power-bi/power-bi-workspace.png" alt-text="Power BI workspace":::
+
+2. Create a new dashboard by clicking **New** > **Dashboard** or open an existing dashboard. Select the **Edit** drop down arrow and then **Add a tile**. Select **Web content** > **Next**.
+3. In **Add web content tile**, enter your **Embed code** from previous section. Click **Apply**.
+
+ :::image type="content" source="./media/power-bi/embed-code.png" alt-text="Embed the html code in tile":::
+
+4. You will see a player widget pinned to the dashboard with a video.
+
+ :::image type="content" source="./media/power-bi/one-player-added.png" alt-text="One video player widget added":::
+
+5. To add more videos from Azure Video Analyzer Videos section, follow the same steps in this section.
+
+> [!NOTE]
+> To add multiple videos from the same Video Analyzer account, a single set of access policy and token is sufficient.
+
+Here is a sample of multiple videos pinned to a single Power BI dashboard.
+
+> [!div class="mx-imgBorder"]
+> :::image type="content" source="./media/power-bi/two-players-added.png" alt-text="Two video player widgets added":::
+
+## Next steps
+
+- Learn more about the [widget API](https://github.com/Azure/video-analyzer/tree/main/widgets)
azure-vmware Configure Vmware Hcx https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-vmware/configure-vmware-hcx.md
After you complete these steps, you'll have a production-ready environment for c
## Add a site pairing
-You can connect or pair the VMware HCX Cloud Manager in Azure VMware Solution with the VMware HCX Connector in your datacenter.
+In your data center, you can connect or pair the VMware HCX Cloud Manager in Azure VMware Solution with the VMware HCX Connector.
> [!IMPORTANT] > Although the VMware Configuration Maximum tool describes site pairs maximum to be 25 between the on-premises HCX Connector and HCX Cloud Manager, licensing limits this to three for HCX Advanced and 10 for HCX Enterprise Edition. 1. Sign in to your on-premises vCenter, and under **Home**, select **HCX**.
-1. Under **Infrastructure**, select **Site Pairing**, and then select the **Connect To Remote Site** option (in the middle of the screen).
+1. Under **Infrastructure**, select **Site Pairing** and select the **Connect To Remote Site** option (in the middle of the screen).
1. Enter the Azure VMware Solution HCX Cloud Manager URL or IP address that you noted earlier `https://x.x.x.9`, the Azure VMware Solution cloudadmin\@vsphere.local username, and the password. Then select **Connect**.
For an end-to-end overview of this procedure, view the [Azure VMware Solution: H
:::image type="content" source="media/tutorial-vmware-hcx/select-uplink-network-profile.png" alt-text="Screenshot that shows the selection of an uplink network profile and the Continue button." lightbox="media/tutorial-vmware-hcx/select-uplink-network-profile.png":::
-1. From **Select vMotion Network Profile**, select the vMotion network profile that you created in prior steps. Then select **Continue**.
+1. From **Select vMotion Network Profile**, select the vMotion network profile that you created in previous steps. Then select **Continue**.
:::image type="content" source="media/tutorial-vmware-hcx/select-vmotion-network-profile.png" alt-text="Screenshot that shows the selection of a vMotion network profile and the Continue button." lightbox="media/tutorial-vmware-hcx/select-vmotion-network-profile.png":::
-1. From **Select vSphere Replication Network Profile**, select the replication network profile that you created in prior steps. Then select **Continue**.
+1. From **Select vSphere Replication Network Profile**, select the replication network profile that you created in previous steps. Then select **Continue**.
:::image type="content" source="media/tutorial-vmware-hcx/select-replication-network-profile.png" alt-text="Screenshot that shows the selection of a replication network profile and the Continue button." lightbox="media/tutorial-vmware-hcx/select-replication-network-profile.png":::
-1. From **Select Distributed Switches for Network Extensions**, select the switches that contain the virtual machines to be migrated to Azure VMware Solution on a layer-2 extended network. Then select **Continue**.
+1. From **Select Distributed Switches for Network Extensions**, select the switches containing the virtual machines to be migrated to Azure VMware Solution on a layer-2 extended network. Then select **Continue**.
> [!NOTE] > If you are not migrating virtual machines on layer-2 (L2) extended networks, you can skip this step.
For an end-to-end overview of this procedure, view the [Azure VMware Solution: C
:::image type="content" source="media/tutorial-vmware-hcx/create-service-mesh.png" alt-text="Screenshot of selections to start creating a service mesh." lightbox="media/tutorial-vmware-hcx/create-service-mesh.png":::
-1. Review the sites that are pre-populated, and then select **Continue**.
+1. Review the pre-populated sites, and then select **Continue**.
> [!NOTE] > If this is your first service mesh configuration, you won't need to modify this screen.
azure-vmware Install Vmware Hcx https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-vmware/install-vmware-hcx.md
Last updated 07/30/2021
VMware HCX Advanced and its associated Cloud Manager are no longer pre-deployed in Azure VMware Solution. Instead, you'll need to install it through the Azure portal as an add-on. The default is HCX Advanced, after which you can still request VMware HCX Enterprise Edition through support if you need the features in the Enterprise edition. You'll still download the HCX Connector OVA and deploy the virtual appliance on your on-premises vCenter.
-HCX Advanced supports up to three site connections (on-premises to cloud or cloud to cloud). If you need more than three site connections, use HCX Enterprise Edition. To activate HCX Enterprise Edition, which is currently in public preview on Azure VMware Solution, open a support request to have it enabled. Once the service is generally available, you'll have 30 days to decide on your next steps. You can also turn off or opt out of the HCX Enterprise Edition service but keep HCX Advanced as it's part of the node cost.
+HCX Advanced supports up to three site connections (on-premises to cloud or cloud to cloud). If you need more than three site connections, use HCX Enterprise Edition. To activate HCX Enterprise Edition, which is currently in public preview on Azure VMware Solution, open a support request to have it enabled. Once the service is generally available, you'll have 30 days to decide on your next steps. You can also turn off or opt-out of the HCX Enterprise Edition service but keep HCX Advanced as it's part of the node cost.
-Downgrading from HCX Enterprise Edition to HCX Advanced is possible without redeploying. First, make sure youΓÇÖve reverted to an HCX Advanced configuration state and not using the Enterprise features. If you plan to downgrade, ensure that no migrations are scheduled, features like RAV and MON aren't in use, and site pairings are three or less.
+Downgrading from HCX Enterprise Edition to HCX Advanced is possible without redeploying. First, ensure youΓÇÖve reverted to an HCX Advanced configuration state and not using the Enterprise features. If you plan to downgrade, ensure that no scheduled migrations, features like RAV and MON aren't in use, and site pairings are three or fewer.
>[!TIP] >You can also [uninstall HCX Advanced](#uninstall-hcx-advanced) through the portal. When you uninstall HCX Advanced, make sure you don't have any active migrations in progress. Removing HCX Advanced returns the resources to your private cloud occupied by the HCX virtual appliances.
After you're finished, follow the recommended next steps at the end to continue
1. Select the **I agree with terms and conditions** checkbox and then select **Install**.
- It will take around 35 minutes to install HCX Advanced and configure the Cloud Manager. Once installed, the HCX Manager URL and the HCX keys needed for the HCX on-premises connector site pairing displays on the **Migration using HCX** tab.
+ It takes around 35 minutes to install HCX Advanced and configure the Cloud Manager. Once installed, the HCX Manager URL and the HCX keys needed for the HCX on-premises connector site pairing display on the **Migration using HCX** tab.
:::image type="content" source="media/tutorial-vmware-hcx/deployed-hcx-migration-using-hcx-tab.png" alt-text="Screenshot showing the Migration using HCX tab under Connectivity."::: ## Download and deploy the VMware HCX Connector OVA
-In this step, you'll download the VMware HCX Connector OVA file and then you'll deploy the VMware HCX Connector to your on-premises vCenter.
+In this step, you'll download the VMware HCX Connector OVA file, and then you'll deploy the VMware HCX Connector to your on-premises vCenter.
1. Open a browser window, sign in to the Azure VMware Solution HCX Manager on `https://x.x.x.9` port 443 with the **cloudadmin\@vsphere.local** user credentials
-1. Under **Administration** > **System Updates** select **Request Download Link**. If the box is greyed, wait a few seconds for it to generate a link.
+1. Under **Administration** > **System Updates**, select **Request Download Link**. If the box is greyed, wait a few seconds for it to generate a link.
-1. Either download or receive a link for the VMware HCX Connector OVA file which you deploy on your local vCenter.
+1. Either download or receive a link for the VMware HCX Connector OVA file you deploy on your local vCenter.
1. In your on-premises vCenter, select an [OVF template](https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.vm_admin.doc/GUID-17BEDA21-43F6-41F4-8FB2-E01D275FE9B4.html) to deploy the VMware HCX Connector to your on-premises vCenter.
In this step, you'll download the VMware HCX Connector OVA file and then you'll
1. Select a name and location, and select a resource or cluster where you're deploying the VMware HCX Connector. Then review the details and required resources and select **Next**.
-1. Review license terms, select the required storage and network, and then select **Next**.
+1. Review license terms, select the required storage and network and then select **Next**.
1. Select the [VMware HCX management network segment](plan-private-cloud-deployment.md#define-vmware-hcx-network-segments) that you defined during the planning state. Then select **Next**.
In this step, you'll download the VMware HCX Connector OVA file and then you'll
## Activate VMware HCX
-After you deploy the VMware HCX Connector OVA on-premises and start the appliance, you're ready to activate it. First, you'll need to get a license key from the Azure VMware Solution portal and then you'll activate it in VMware HCX Manager. You will need a key for each on premises HCX connector that is deployed.
+After deploying the VMware HCX Connector OVA on-premises and starting the appliance, you're ready to activate it. First, you'll need to get a license key from the Azure VMware Solution portal, and then you'll activate it in VMware HCX Manager. Finally, youΓÇÖll need a key for each on-premises HCX connector deployed.
-1. In the Azure VMware Solution portal, go to **Manage** > **Connectivity**, select the **HCX** tab, and then select **Add**.
+1. In the Azure VMware Solution portal, go to **Manage** > **Connectivity**, select the **HCX** tab, and select **Add**.
1. Use the **admin** credentials to sign in to the on-premises VMware HCX Manager at `https://HCXManagerIP:9443`. Make sure to include the `9443` port number with the VMware HCX Manager IP address.
After you deploy the VMware HCX Connector OVA on-premises and start the applianc
>[!TIP] >The vCenter server is where you deployed the VMware HCX Connector in your datacenter.
-1. In **Configure SSO/PSC**, provide the FQDN or IP address of your Platform Services Controller, and then select **Continue**.
+1. 8. In **Configure SSO/PSC**, provide your Platform Services Controller's FQDN or IP address, and select **Continue**.
>[!NOTE] >Typically, it's the same as your vCenter FQDN or IP address.
After the services restart, you'll see vCenter showing as green on the screen th
## Uninstall HCX Advanced
-You can uninstall HCX Advanced through the portal, which will remove the existing pairing and software.
+You can uninstall HCX Advanced through the portal, which removes the existing pairing and software.
>[!NOTE] >It could take approximately 30 minutes to return the resources to your private cloud occupied by the HCX virtual appliances.
You can uninstall HCX Advanced through the portal, which will remove the existin
1. Ensure that L2 extensions are no longer needed or the networks have been "unstretched" to the destination.
-1. For workloads using MON, ensure that the default gateways have been removed. Otherwise, it may result in workloads not being able to communicate or function.
+1. 3. For workloads using MON, ensure that youΓÇÖve removed the default gateways. Otherwise, it may result in workloads not being able to communicate or function.
1. In your Azure VMware Solution private cloud, select **Manage** > **Add-ons** > **Uninstall**.
azure-vmware Tutorial Access Private Cloud https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-vmware/tutorial-access-private-cloud.md
In this tutorial, you learn how to:
## Create a new Windows virtual machine
-1. In the resource group, select **+ Add** then search and select **Microsoft Windows 10**, and then select **Create**.
+1. In the resource group, select **Add**, search for and select **Microsoft Windows 10**, and then select **Create**.
:::image type="content" source="media/tutorial-access-private-cloud/ss8-azure-w10vm-create.png" alt-text="Screenshot of how to add a new Windows 10 VM for a jump box.":::
In this tutorial, you learn how to:
## Next steps
-In this tutorial you learned how to:
+In this tutorial, you learned how to:
> [!div class="checklist"] > * Create a Windows virtual machine to use to connect to vCenter
azure-vmware Tutorial Configure Networking https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-vmware/tutorial-configure-networking.md
You can use the **Azure vNet connect** feature to use an existing vNet or create
### Select an existing vNet
-When you select an existing vNet, the Azure Resource Manager (ARM) template that creates the vNet and other resources gets redeployed. The resources in this case are the public IP, gateway, gateway connection, and ExpressRoute authorization key. If everything is set up, the deployment won't change anything. However, if anything is missing, it gets created automatically. For example, if the GatewaySubnet is missing, then it gets added during the deployment.
+When you select an existing vNet, the Azure Resource Manager (ARM) template that creates the vNet and other resources gets redeployed. The resources, in this case, are the public IP, gateway, gateway connection, and ExpressRoute authorization key. If everything is set up, the deployment won't change anything. However, if anything is missing, it gets created automatically. For example, if the GatewaySubnet is missing, then it gets added during the deployment.
1. In your Azure VMware Solution private cloud, under **Manage**, select **Connectivity**.
When you create a new vNet, the required components needed to connect to Azure V
3. Provide or update the information for the new vNet and then select **OK**.
- At this point, the vNet validates if overlapping IP address spaces between Azure VMware Solution and vNet are detected. If detected, then change the network address of either the private cloud or the vNet so they don't overlap.
+ At this point, the vNet validates if overlapping IP address spaces between Azure VMware Solution and vNet are detected. If detected, change the network address of either the private cloud or the vNet so they don't overlap.
:::image type="content" source="media/networking/create-new-virtual-network.png" alt-text="Screenshot showing the Create virtual network window.":::
Now that you've created a virtual network, you'll create a virtual network gatew
1. In your resource group, select **+ Add** to add a new resource.
-1. In the **Search the Marketplace** text box type, **Virtual network gateway**. Find the Virtual Network resource and select it.
+1. In the **Search the Marketplace** text box, type **Virtual network gateway**. Find the Virtual Network resource and select it.
1. On the **Virtual Network gateway** page, select **Create**.
Now that you've created a virtual network, you'll create a virtual network gatew
:::image type="content" source="./media/tutorial-configure-networking/create-virtual-network-gateway.png" alt-text="Screenshot showing the details for the virtual network gateway." border="true":::
-1. Verify that the details are correct, and select **Create** to start the deployment of your virtual network gateway.
+1. Verify that the details are correct, and select **Create** to start your virtual network gateway deployment.
+ 1. Once the deployment completes, move to the next section to connect your ExpressRoute connection to the virtual network gateway containing your Azure VMware Solution private cloud. ### Connect ExpressRoute to the virtual network gateway
In this tutorial, you learned how to:
> * Connect your ExpressRoute circuit to the gateway
-Continue to the next tutorial to learn how to create the NSX-T network segments that are used for VMs in vCenter.
+Continue to the next tutorial to learn how to create the NSX-T network segments used for VMs in vCenter.
> [!div class="nextstepaction"]
-> [Create an NSX-T network segment](./tutorial-nsx-t-network-segment.md)
+> [Create an NSX-T network segment](./tutorial-nsx-t-network-segment.md)
azure-vmware Tutorial Create Private Cloud https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-vmware/tutorial-create-private-cloud.md
In this tutorial, you'll learn how to:
- Appropriate administrative rights and permission to create a private cloud. You must be at minimum contributor level in the subscription. - Follow the information you gathered in the [planning](plan-private-cloud-deployment.md) tutorial to deploy Azure VMware Solution.-- Ensure you have the appropriate networking configured as described in [Network planning checklist](tutorial-network-checklist.md).-- Hosts have been provisioned and the Microsoft.AVS [resource provider has been registered](deploy-azure-vmware-solution.md#register-the-microsoftavs-resource-provider).
+- Ensure you have the appropriate networking configured as described in the [Network planning checklist](tutorial-network-checklist.md).
+- Hosts provisioned and the Microsoft.AVS [resource provider has been registered](deploy-azure-vmware-solution.md#register-the-microsoftavs-resource-provider).
## Create a private cloud
azure-vmware Tutorial Expressroute Global Reach Private Cloud https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/azure-vmware/tutorial-expressroute-global-reach-private-cloud.md
After you're finished, follow the recommended next steps at the end to continue
- Review the documentation on how to [enable connectivity in different Azure subscriptions](../expressroute/expressroute-howto-set-global-reach-cli.md#enable-connectivity-between-expressroute-circuits-in-different-azure-subscriptions). -- A separate, functioning ExpressRoute circuit used to connect on-premises environments to Azure, which is _circuit 1_ for peering.
+- A separate, functioning ExpressRoute circuit to connect on-premises environments to Azure, which is _circuit 1_ for peering.
- Ensure that all gateways, including the ExpressRoute provider's service, supports 4-byte Autonomous System Number (ASN). Azure VMware Solution uses 4-byte public ASNs for advertising routes.
Now that you've created an authorization key for the private cloud ExpressRoute
## Verify on-premises network connectivity
-You should now see in your **on-premises edge router** where the ExpressRoute connects the NSX-T network segments and the Azure VMware Solution management segments.
+In your **on-premises edge router**, you should now see where the ExpressRoute connects the NSX-T network segments and the Azure VMware Solution management segments.
>[!IMPORTANT] >Everyone has a different environment, and some will need to allow these routes to propagate back into the on-premises network.
batch Batch Cli Templates https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/batch/batch-cli-templates.md
Title: Run jobs end-to-end using templates description: With only CLI commands, you can create a pool, upload input data, create jobs and associated tasks, and download the resulting output data. Previously updated : 06/14/2021 Last updated : 08/06/2021 # Use Azure Batch CLI templates and file transfer
-Using a Batch extension to the Azure CLI, it is possible to run Batch jobs without writing code.
+By using a Batch extension to Azure CLI, users can run Batch jobs without writing code.
-Create and use JSON template files with the Azure CLI to create Batch
-pools, jobs, and tasks. Use CLI extension commands to easily upload job input files to
-the storage account associated with the Batch account, and download job output files.
+Create and use JSON template files with Azure CLI to create Batch pools, jobs, and tasks. Use CLI extension commands to easily upload job input files to the storage account associated with the Batch account, and download job output files.
> [!NOTE] > JSON files don't support the same functionality as [Azure Resource Manager templates](../azure-resource-manager/templates/syntax.md). They are meant to be formatted like the raw REST request body. The CLI extension doesn't change any existing commands, but it does have a similar template option that adds partial Azure Resource Manager template functionality. See [Azure Batch CLI Extensions for Windows, Mac and Linux](https://github.com/Azure/azure-batch-cli-extensions). ## Overview
-An extension to the Azure CLI enables Batch to be used end-to-end by users who
-are not developers. With only CLI commands, you can create a pool, upload input data, create jobs and
-associated tasks, and download the resulting output data. No additional code is
-required. Run the CLI commands directly or integrate them into scripts.
+An extension to the Azure CLI enables Batch to be used end-to-end by users who are not developers. With only CLI commands, you can create a pool, upload input data, create jobs and
+associated tasks, and download the resulting output data. No additional code is required. Run the CLI commands directly or integrate them into scripts.
-Batch templates build on the existing Batch support in the [Azure
-CLI](batch-cli-get-started.md#json-files-for-resource-creation) for JSON files to specify property values when creating pools,
+Batch templates build on the existing Batch support in the [Azure CLI](batch-cli-get-started.md#json-files-for-resource-creation) for JSON files to specify property values when creating pools,
jobs, tasks, and other items. Batch templates add the following capabilities: -- Parameters can be defined. When the template is used, only the parameter
- values are specified to create the item, with other item property values
- specified in the template body. A user who understands Batch and the
- applications to be run by Batch can create templates, specifying pool, job,
- and task property values. A user less familiar with Batch and/or the
- applications only needs to specify the values for the defined
- parameters.
--- Job task factories create one or more tasks associated with a job,
- avoiding the need for many task definitions to be created and significantly
- simplifying job submission.
--
-Jobs typically use input data files and produce output data files. A storage account is associated, by default, with each Batch account. Transfer files to and from this storage account using the
-CLI, with no coding and no storage credentials.
-
-For example, [ffmpeg](https://ffmpeg.org/) is a popular application that
-processes audio and video files. Here are steps with the Azure Batch CLI to invoke
-ffmpeg to transcode source video files to different resolutions.
--- Create a pool template. The user creating the template knows how to call
- the ffmpeg application and its requirements; they specify the appropriate
- OS, VM size, how ffmpeg is installed (from an application package or
- using a package manager, for example), and other pool property values. Parameters are
- created so when the template is used, only the pool ID and number of VMs
- need to be specified.
--- Create a job template. The user creating the template knows how ffmpeg
- needs to be invoked to transcode source video to a different resolution and
- specifies the task command line; they also know that there is a folder
- containing the source video files, with a task required per input file.
--- An end user with a set of video files to transcode first creates a pool
- using the pool template, specifying only the pool ID and number of VMs
- required. They can then upload the source files to transcode. A job can then
- be submitted using the job template, specifying only the pool ID and
- location of the source files uploaded. The Batch job is created, with
- one task per input file being generated. Finally, the transcoded output
- files can be downloaded.
+- Parameters can be defined. When the template is used, only the parameter values are specified to create the item, with other item property values specified in the template body. A user who understands Batch and the applications to be run by Batch can create templates, specifying pool, job, and task property values. A user less familiar with Batch and/or the applications only needs to specify the values for the defined parameters.
+
+- Job task factories create one or more tasks associated with a job, avoiding the need for many task definitions to be created and significantly simplifying job submission.
+
+Jobs typically use input data files and produce output data files. A storage account is associated, by default, with each Batch account. You can transfer files to and from this storage account using Azure CLI, with no coding and no storage credentials.
+
+For example, [ffmpeg](https://ffmpeg.org/) is a popular application that processes audio and video files. Using the Azure Batch CLI extension, you could make it easier for a user to invoke ffmpeg to transcode source video files to different resolutions. The process might look like this:
+
+- Create a pool template. The user creating the template knows how to call the ffmpeg application and its requirements; they specify the appropriate OS, VM size, how ffmpeg is installed (from an application package or using a package manager, for example), and other pool property values. Parameters are created so when the template is used, only the pool ID and number of VMs need to be specified.
+- Create a job template. The user creating the template knows how ffmpeg needs to be invoked to transcode source video to a different resolution and specifies the task command line; they also know that there is a folder containing the source video files, with a task required per input file.
+- An end user with a set of video files to transcode first creates a pool using the pool template, specifying only the pool ID and number of VMs required. They can then upload the source files to transcode. A job can then be submitted using the job template, specifying only the pool ID and location of the source files uploaded. The Batch job is created, with one task per input file being generated. Finally, the transcoded output files can be downloaded.
## Installation To install the Azure Batch CLI extension, first [Install the Azure CLI 2.0](/cli/azure/install-azure-cli), or run the Azure CLI in [Azure Cloud Shell](../cloud-shell/overview.md).
-Install the latest version of the Batch extension using the
-following Azure CLI command:
+Install the latest version of the Batch extension using the following Azure CLI command:
```azurecli az extension add --name azure-batch-cli-extensions
az extension add --name azure-batch-cli-extensions
For more information about the Batch CLI extension and additional installation options, see the [GitHub repo](https://github.com/Azure/azure-batch-cli-extensions). - To use the CLI extension features, you need an Azure Batch account and, for the commands that transfer files to and from storage, a linked storage account. To log into a Batch account with the Azure CLI, see [Manage Batch resources with Azure CLI](batch-cli-get-started.md). ## Templates
-Azure Batch templates are similar to Azure Resource Manager templates, in functionality and
-syntax. They are JSON files that contain item property names and values, but add
-the following main concepts:
--- **Parameters**-
- - Allow property values to be specified in a body section, with only
- parameter values needing to be supplied when the template is used. For
- example, the complete definition for a pool could be placed in the body
- and only one parameter defined for `poolId`; only a pool ID string
- therefore needs to be supplied to create a pool.
-
- - The template body can be authored by someone with knowledge of Batch and
- the applications to be run by Batch; only values for the author-defined
- parameters must be supplied when the template is used. A user without
- the in-depth Batch and/or application knowledge can therefore use the
- templates.
--- **Variables**-
- - Allow simple or complex parameter values to be specified in one place
- and used in one or more places in the template body. Variables can
- simplify and reduce the size of the template, as well as make it more
- maintainable by having one location to change properties.
+Azure Batch templates are similar to Azure Resource Manager templates, in functionality and syntax. They are JSON files that contain item property names and values, but add the following main concepts:
-- **Higher-level constructs**-
- - Some higher-level constructs are available in the template
- that are not yet
- available in the Batch APIs. For example, a task factory can be defined
- in a job template that creates multiple tasks for the job, using a
- common task definition. These constructs avoid the need to code to
- dynamically create multiple JSON files, such as one file per task, as
- well as create script files to install applications via a package
- manager.
-
- - At some point, these constructs may be added to the
- Batch service and available in the Batch APIs, UIs, etc.
+- **Parameters**: Allow property values to be specified in a body section, with only parameter values needing to be supplied when the template is used. For example, the complete definition for a pool could be placed in the body and only one parameter defined for `poolId`; only a pool ID string therefore needs to be supplied to create a pool. The template body can be authored by someone with knowledge of Batch and the applications to be run by Batch; only values for the author-defined parameters must be supplied when the template is used. This lets users without any in-depth Batch and/or application knowledge use the templates.
+- **Variables**: Allow simple or complex parameter values to be specified in one place and used in one or more places in the template body. Variables can simplify and reduce the size of the template, as well as make it more maintainable by having one location to change properties.
+- **Higher-level constructs**: Some higher-level constructs are available in the template that are not yet available in the Batch APIs. For example, a task factory can be defined in a job template that creates multiple tasks for the job, using a common task definition. These constructs avoid the need to code to dynamically create multiple JSON files, such as one file per task, as well as create script files to install applications via a package manager.
### Pool templates
-Pool templates support the standard template capabilities of parameters and variables. They also support the following higher-level construct:
--- **Package references**
+Pool templates support the standard template capabilities of parameters and variables. They also support **package references**, which optionally allow software to be copied to pool nodes by using package managers. The package manager and package ID are specified in the package reference. By declaring one or more packages, you avoid creating a script that gets the required packages, installing the script, and running the script on each pool node.
- - Optionally allows software to be copied to pool nodes by using package
- managers. The package manager and package ID are specified. By declaring one or more packages, you avoid creating a script that
- gets the required packages, installing the script, and running the script on
- each pool node.
-
-The following is an example of a template that creates a pool of Linux VMs with
-ffmpeg installed. To use it, supply only a pool ID string and the number of VMs in the pool:
+The following is an example of a template that creates a pool of Linux VMs with ffmpeg installed. To use it, supply only a pool ID string and the number of VMs in the pool:
```json {
ffmpeg installed. To use it, supply only a pool ID string and the number of VMs
"vmSize": "STANDARD_D3_V2", "targetDedicatedNodes": "[parameters('nodeCount')]", "enableAutoScale": false,
- "maxTasksPerNode": 1,
+ "taskSlotsPerNode": 1,
"packageReferences": [ { "type": "aptPackage",
az batch pool create --template pool-ffmpeg.json --parameters pool-parameters.js
### Job templates
-Job templates support the standard template capabilities of parameters and variables. They also support the following higher-level construct:
--- **Task factory**
+Job templates support the standard template capabilities of parameters and variables. They also support the **task factory** construct, which creates multiple tasks for a job from one task definition. Three types of task factory are supported: parametric sweep, task per file, and task collection.
- - Creates multiple tasks for a job from one task definition. Three
- types of task factory are supported ΓÇô parametric sweep, task per file,
- and task collection.
-
-The following is an example of a template that creates a job to
-transcode MP4 video files with ffmpeg to one of two lower resolutions. It creates one task
-per source video file. See [File groups and file transfer](#file-groups-and-file-transfer) for more about file groups for job input and output.
+The following is an example of a template that creates a job to transcode MP4 video files with ffmpeg to one of two lower resolutions. It creates one task per source video file. See [File groups and file transfer](#file-groups-and-file-transfer) for more about file groups for job input and output.
```json {
As before, the CLI prompts you to provide values for the parameters. You can als
### Use templates in Batch Explorer
-You can upload a Batch CLI template to the [Batch Explorer](https://github.com/Azure/BatchExplorer) desktop application (formerly called BatchLabs) to create a Batch pool or job. You can also select from predefined pool and job templates in the Batch Explorer Gallery.
+You can upload a Batch CLI template to the [Batch Explorer](https://github.com/Azure/BatchExplorer) desktop application to create a Batch pool or job. You can also select from predefined pool and job templates in the Batch Explorer Gallery.
To upload a template: 1. In Batch Explorer, select **Gallery** > **Local templates**.- 2. Select, or drag and drop, a local pool or job template.- 3. Select **Use this template**, and follow the on-screen prompts. ## File groups and file transfer
az batch file download --file-group ffmpeg-output --local-path
c:\output_lowres_videos ```
-Pool and job templates allow files stored in file groups to be specified for
-copy onto pool nodes or off pool nodes back to a file group. For example, in the
-job template specified previously, the file group *ffmpeg-input* is specified
-for the task factory as the location of the source video files copied down to
-the node for transcoding. The file group *ffmpeg-output* is the location
+Pool and job templates allow files stored in file groups to be specified for copy onto pool nodes or off pool nodes back to a file group. For example, in the job template specified previously, the file group *ffmpeg-input* is specified for the task factory as the location of the source video files copied down to the node for transcoding. The file group *ffmpeg-output* is the location
where the transcoded output files are copied from the node running each task. ## Summary
-Template and file transfer support have currently been added only to the Azure CLI. The goal is to expand the audience that can use Batch to users
-who do not need to develop code using the Batch APIs, such as researchers and IT users. Without coding, users with knowledge of Azure, Batch, and the applications to be run by Batch can create templates for pool and job creation. With template parameters, users without detailed knowledge of Batch and the applications can use the templates.
+Template and file transfer support have currently been added only to the Azure CLI. The goal is to expand the audience that can use Batch to users who do not need to develop code using the Batch APIs, such as researchers and IT users. Without coding, users with knowledge of Azure, Batch, and the applications to be run by Batch can create templates for pool and job creation. With template parameters, users without detailed knowledge of Batch and the applications can use the templates.
Try out the Batch extension for the Azure CLI and provide us with any feedback or suggestions, either in the comments for this article or via the [Batch Community repo](https://github.com/Azure/Batch). ## Next steps -- Detailed installation and usage documentation, samples, and source code are
-available in the [Azure GitHub
-repo](https://github.com/Azure/azure-batch-cli-extensions).
--- Learn more about using [Batch Explorer](https://github.com/Azure/BatchExplorer) to create and manage Batch resources.
+- View detailed installation and usage documentation, samples, and source code in the [Azure GitHub repo](https://github.com/Azure/azure-batch-cli-extensions).
+- Learn more about using [Batch Explorer](https://github.com/Azure/BatchExplorer) to create and manage Batch resources.
connectors Connectors Sftp Ssh https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/connectors/connectors-sftp-ssh.md
This section describes possible solutions to common errors or problems.
This error can happen when your logic app can't successfully establish a connection with the SFTP server. There might be different reasons for this problem, so try these troubleshooting options:
-* The connection timeout is 20 seconds. Check that your SFTP server has good performance and intermediate devices, such as firewalls, aren't adding overhead.
+* The connection timeout is 20 seconds. Check that your SFTP server has good performance and intermediate devices, such as firewalls, aren't adding overhead.
-* If you have a firewall set up, make sure that you add the **Managed connector IP** addresses to the approved list. To find the IP addresses for your logic app's region, see [Limits and configuration for Azure Logic Apps](../logic-apps/logic-apps-limits-and-config.md#multi-tenant-azureoutbound-ip-addresses).
+* If you have a firewall set up, make sure that you add the **Managed connector IP** addresses for your region to the approved list. To find the IP addresses for your logic app's region, see [Managed connector outbound IPs - Azure Logic Apps](/connectors/common/outbound-ip-addresses).
* If this error happens intermittently, change the **Retry policy** setting on the SFTP-SSH action to a retry count higher than the default four retries.
cosmos-db Continuous Backup Restore Permissions https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/cosmos-db/continuous-backup-restore-permissions.md
Following permissions are required to perform the different activities pertainin
||||| |`Microsoft.Resources/deployments/validate/action`, `Microsoft.Resources/deployments/write` | These permissions are required for the ARM template deployment to create the restored account. See the sample permission [RestorableAction](#custom-restorable-action) below for how to set this role. | Not applicable | Not applicable | |`Microsoft.DocumentDB/databaseAccounts/write` | This permission is required to restore an account into a resource group | Resource group under which the restored account is created. | Subscription under which the restored account is created |
-|`Microsoft.DocumentDB/locations/restorableDatabaseAccounts/restore/action` </br> You can't choose resource group as the permission scope. |This permission is required on the source restorable database account scope to allow restore actions to be performed on it. | The *RestorableDatabaseAccount* resource belonging to the source account being restored. This value is also given by the `ID` property of the restorable database account resource. An example of restorable account is */subscriptions/subscriptionId/providers/Microsoft.DocumentDB/locations/regionName/restorableDatabaseAccounts/<guid-instanceid>* | The subscription containing the restorable database account. |
-|`Microsoft.DocumentDB/locations/restorableDatabaseAccounts/read` </br> You can't choose resource group as the permission scope. |This permission is required on the source restorable database account scope to list the database accounts that can be restored. | The *RestorableDatabaseAccount* resource belonging to the source account being restored. This value is also given by the `ID` property of the restorable database account resource. An example of restorable account is */subscriptions/subscriptionId/providers/Microsoft.DocumentDB/locations/regionName/restorableDatabaseAccounts/<guid-instanceid>*| The subscription containing the restorable database account. |
-|`Microsoft.DocumentDB/locations/restorableDatabaseAccounts/*/read` </br> You can't choose resource group as the permission scope. | This permission is required on the source restorable account scope to allow reading of restorable resources such as list of databases and containers for a restorable account. | The *RestorableDatabaseAccount* resource belonging to the source account being restored. This value is also given by the `ID` property of the restorable database account resource. An example of restorable account is */subscriptions/subscriptionId/providers/Microsoft.DocumentDB/locations/regionName/restorableDatabaseAccounts/<guid-instanceid>*| The subscription containing the restorable database account. |
+|`Microsoft.DocumentDB/locations/restorableDatabaseAccounts/restore/action` </br> You can't choose resource group as the permission scope. |This permission is required on the source restorable database account scope to allow restore actions to be performed on it. | The *RestorableDatabaseAccount* resource belonging to the source account being restored. This value is also given by the `ID` property of the restorable database account resource. An example of restorable account is */subscriptions/subscriptionId/providers/Microsoft.DocumentDB/locations/regionName/restorableDatabaseAccounts/\<guid-instanceid\>* | The subscription containing the restorable database account. |
+|`Microsoft.DocumentDB/locations/restorableDatabaseAccounts/read` </br> You can't choose resource group as the permission scope. |This permission is required on the source restorable database account scope to list the database accounts that can be restored. | The *RestorableDatabaseAccount* resource belonging to the source account being restored. This value is also given by the `ID` property of the restorable database account resource. An example of restorable account is */subscriptions/subscriptionId/providers/Microsoft.DocumentDB/locations/regionName/restorableDatabaseAccounts/\<guid-instanceid\>*| The subscription containing the restorable database account. |
+|`Microsoft.DocumentDB/locations/restorableDatabaseAccounts/*/read` </br> You can't choose resource group as the permission scope. | This permission is required on the source restorable account scope to allow reading of restorable resources such as list of databases and containers for a restorable account. | The *RestorableDatabaseAccount* resource belonging to the source account being restored. This value is also given by the `ID` property of the restorable database account resource. An example of restorable account is */subscriptions/subscriptionId/providers/Microsoft.DocumentDB/locations/regionName/restorableDatabaseAccounts/\<guid-instanceid\>*| The subscription containing the restorable database account. |
## Azure CLI role assignment scenarios to restore at different scopes
cosmos-db Graph Modeling Tools https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/cosmos-db/graph-modeling-tools.md
+
+ Title: Third-party data modeling tools for Azure Cosmos DB graph data
+description: This article describes various tools to design the Graph data model.
++++ Last updated : 05/25/2021+++
+# Third-party data modeling tools for Azure Cosmos DB graph data
++
+It is important to design the data model and further important to maintain. Here are set of third-party visual design tools which help in designing & maintaining the graph data model.
+
+> [!IMPORTANT]
+> Solutions mentioned in this article are for information purpose only, the ownership lies to individual solution owner. We recommend users to do thorough evaluation and then select most suitable to you.
+
+## Hackolade
+
+Hackolade is a data modeling and schema design tool for NoSQL databases. It has a data modeling Studio, which helps in management of schemas for data-at-rest and data-in-motion.
+
+### How it works
+This tool provides the data modeling of vertices / edges and their respective properties. It supports several use cases, some of them are:
+- Start from a blank page and think through different options to graphically build your Cosmos DB Gremlin model. Then forward-engineer the model to your Azure instance to evaluate the result and continue the evolution. All such goodies without writing single line of code.
+- Reverse-engineer an existing graph on Azure to clearly understand its structure, so you could effectively query your graph too. Then enrich the data model with descriptions, metadata, and constraints to produce documentation. It supports HTML, Markdown or PDF format, and feeds to corporate data governance or dictionary systems.
+- Migrate from relational database to NoSQL through the de-normalization of data structures.
+- Integrate with a CI/CD pipeline via a Command-Line Interface
+- Collaboration and versioning using Git
+- And much more…
+
+### Sample
+
+The animation at Figure-2 provides a demonstration of reverse engineering, extraction of entities from RDBMS then Hackolade will discover relations from foreign key relationships then modifications.
+
+Sample DDL for source as SQL Server available at [here](https://github.com/Azure-Samples/northwind-ddl-sample/nw.sql)
++
+**Figure-1:** Graph Diagram (extracted the graph data model)
+
+After modification of data model, the tool can generate the gremlin script, which may include custom Cosmos DB index script to ensure optimal indexes are created, refer Figure-2 for full flow.
+
+The following image demonstrates reverse engineering from RDBMS & Hackolade in action:
+
+**Figure-2:** Hackolade in action (demonstrating SQL to Gremlin data model conversion)
+### Useful links
+- [Download a 14-day free trial](https://hackolade.com/download.html)
+- [Schedule a demo](https://c.x.ai/pdesmarets)
+- [Get more data models](https://hackolade.com/samplemodels.html#cosmosdb).
+- [Documentation of Hackolade](https://hackolade.com/help/CosmosDBGremlin.html)
+
+## Next steps
+- [Visualizing the data](/graph-visualization)
cosmos-db Graph Modeling https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/cosmos-db/graph-modeling.md
The more specific the label that the traverser will use to filter the edges, the
## Next steps: * Check out the list of supported [Gremlin steps](gremlin-support.md). * Learn about [graph database partitioning](graph-partitioning.md) to deal with large-scale graphs.
-* Evaluate your Gremlin queries using the [Execution Profile step](graph-execution-profile.md).
+* Evaluate your Gremlin queries using the [Execution Profile step](graph-execution-profile.md).
+* Third-party Graph [design data model](graph-modeling-tools.md)
cosmos-db Linux Emulator https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/cosmos-db/linux-emulator.md
Since the Azure Cosmos DB Emulator provides an emulated environment that runs on
- The Linux emulator does not offer [multi-region replication](distribute-data-globally.md). -- Because the copy of your Azure Cosmos DB Linux Emulator might not always be up to date with the most recent changes in the Azure Cosmos DB service, you should always refer to the [Azure Cosmos DB capacity planner](estimate-ru-with-capacity-planner.md) to accurately estimate the throughput (RUs) needs of your application. <add link>
+- Because the copy of your Azure Cosmos DB Linux Emulator might not always be up to date with the most recent changes in the Azure Cosmos DB service, you should always refer to the [Azure Cosmos DB capacity planner](estimate-ru-with-capacity-planner.md) to accurately estimate the throughput (RUs) needs of your application.
- The Linux emulator supports a maximum ID property size of 254 characters.
cosmos-db Monitoring Solutions https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/cosmos-db/monitoring-solutions.md
+
+ Title: Monitoring Azure Cosmos DB using third-party monitoring tools
+description: This article will describe monitoring third-party tools helps monitoring Cosmos DB.
++++ Last updated : 07/28/2021++
+# Monitoring Azure Cosmos DB using third-party solutions
+
+Apart from Azure Monitor, you can use third party monitoring solutions to monitor your Cosmos DB instances.
+
+> [!IMPORTANT]
+> Solutions mentioned in this article are for information purpose only, the ownership lies to individual solution owner. We recommend users to do thorough evaluation and then select most suitable to you.
+
+## Datadog
+{Supports: SQL API, Azure Cosmos DB API for MongoDB, Gremlin API, Cassandra API & Table API}
+
+[Datadog](https://www.datadoghq.com/) is a fully unified platform encompassing infrastructure monitoring, application performance monitoring, log management, user-experience monitoring, and more. By bringing together data from every tool and service in your companyΓÇÖs stack, Datadog provides a single source of truth for troubleshooting, optimizing performance, and cross-team collaboration.
+Everything in Datadog is organized under the same set of tags, so all the data relevant to a particular issue is automatically correlated. By eliminating the blind spots, Datadog reduces the risk of overlooked errors, mitigates the burden of ongoing service maintenance, and accelerates digital transformations.
+
+Datadog collects over 40 different gauge and count metrics from CosmosDB, including the total available storage per region, the number of SQL databases created, and more. These metrics are collected through the Datadog Azure integration, and appear in the platform 40% faster than the rest of the industry. Datadog also provides an out-of-the-box dashboard for CosmosDB, which provides immediate insight into the performance of CosmosDB instances. Users can visualize platform-level metrics, such as total request units consumed, also API-level metrics, such as the number of Cassandra keyspaces created to better understand their CosmosDB usage.
+
+Datadog is being used by various Cosmos DB customers, which include
+- Maersk
+- PWC
+- PayScale
+- AllScripts
+- Hearst
+++
+**Figure:** Datadog in action
+
+Useful links:
+- [Pricing details](https://www.datadoghq.com/pricing/)
+- [Get started with 14 days trial](https://www.datadoghq.com/free-datadog-trial/)
++
+## Dynatrace
+{Supports: SQL API & Azure Cosmos DB API for MongoDB}
+
+[Dynatrace](https://www.dynatrace.com/platform/) delivers software intelligence for the cloud to tame cloud complexity and accelerate digital transformation. With automatic and intelligent observability at scale, the Dynatrace all-in-one Software Intelligence Platform delivers precise answers about the performance and security of applications, the underlying infrastructure, and the experience of all users, so teams can automate cloud operations, release better software faster, and deliver unrivaled digital experiences.
+Using the Mongo API, Dynatrace collects and delivers CosmosDB metrics, which includes the numbers of calls and response timesΓÇöall visualized according to aggregation, commands, read-, and write operations. It also tells you exact database statements executed in your environment. Lastly with the power of [Davis AI Engine](https://www.dynatrace.com/davis), it can detect exactly which database statement is the root cause of degradation and can see the database identified as the root cause.
+
+**Figure:** Dynatrace in Action
+
+### Useful links
+
+- [Try Dynatrace with 15 days free trial](https://www.dynatrace.com/trial)
+- [Launch from Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/dynatrace.dynatrace-managed)
+- [Documentation on how to Cosmos DB with Azure Monitor](https://www.dynatrace.com/support/help/technology-support/cloud-platforms/microsoft-azure-services/set-up-integration-with-azure-monitor/?_ga=2.184080354.559899881.1623174355-748416177.1603817475)
+- [Cosmos DB - Dynatrace Integration details](https://www.dynatrace.com/news/blog/azure-services-explained-part-4-azure-cosmos-db/?_ga=2.185016301.559899881.1623174355-748416177.1603817475)
+- [Dynatrace Monitoring for Azure databases](https://www.dynatrace.com/technologies/azure-monitoring/azure-database-performance/)
+- [Dynatrace for Azure solution overview](https://www.dynatrace.com/technologies/azure-monitoring/)
+- [Solution Partners](https://www.dynatrace.com/partners/solution-partners/)
+
+## Next steps
+- [Monitoring Cosmos DB data reference](./monitor-cosmos-db-reference.md)
cosmos-db Sql Api Sdk Java Spring V3 https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/cosmos-db/sql-api-sdk-java-spring-v3.md
You can use Spring Data Azure Cosmos DB in your [Azure Spring Cloud](https://azu
| Content | Link | ||| |**SDK download**| [Maven](https://mvnrepository.com/artifact/com.azure/azure-spring-data-cosmos) |
-|**API documentation** | [Java API reference documentation](/java/api/overview/azure/spring-data-cosmos-readme?view=azure-java-stable) |
+|**API documentation** | [Java API reference documentation](/java/api/overview/azure/spring-data-cosmos-readme?view=azure-java-stable&preserve-view=true) |
|**Contribute to SDK** | [Azure SDK for Java Central Repo on GitHub](https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/cosmos/azure-spring-data-cosmos) | |**Get started** | [Quickstart: Build a Spring Data Azure Cosmos DB app to manage Azure Cosmos DB SQL API data](./create-sql-api-spring-data.md) <br> [GitHub repo with quickstart code](https://github.com/Azure-Samples/azure-spring-data-cosmos-java-sql-api-getting-started) | |**Basic code samples** | [Azure Cosmos DB: Spring Data Azure Cosmos DB examples for the SQL API](sql-api-spring-data-sdk-samples.md) <br> [GitHub repo with sample code](https://github.com/Azure-Samples/azure-spring-data-cosmos-java-sql-api-samples)|
cosmos-db Sql Query Linq To Sql https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/cosmos-db/sql-query-linq-to-sql.md
Previously updated : 11/11/2020 Last updated : 08/06/2021
The Azure Cosmos DB query provider performs a best effort mapping from a LINQ query into a Cosmos DB SQL query. If you want to get the SQL query that is translated from LINQ, use the `ToString()` method on the generated `IQueryable`object. The following description assumes a basic familiarity with [LINQ](/dotnet/csharp/programming-guide/concepts/linq/introduction-to-linq-queries). In addition to LINQ, Azure Cosmos DB also supports [Entity Framework Core](/ef/core/providers/cosmos/?tabs=dotnet-core-cli) which works with SQL API.
+> [!NOTE]
+> We recommend using the latest [.NET SDK version](https://www.nuget.org/packages/Microsoft.Azure.Cosmos/3.20.1)
+ The query provider type system supports only the JSON primitive types: numeric, Boolean, string, and null. The query provider supports the following scalar expressions:
data-factory Concepts Data Flow Expression Builder https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/concepts-data-flow-expression-builder.md
Title: Expression builder in mapping data flow
-description: Build expressions by using Expression Builder in mapping data flows in Azure Data Factory
+description: Build expressions by using Expression Builder in mapping data flows in Azure Data Factory and Azure Synapse Analytics
Last updated 04/29/2021
In mapping data flow, many transformation properties are entered as expressions. These expressions are composed of column values, parameters, functions, operators, and literals that evaluate to a Spark data type at run time. Mapping data flows has a dedicated experience aimed to aid you in building these expressions called the **Expression Builder**. Utilizing [IntelliSense](/visualstudio/ide/using-intellisense) code completion for highlighting, syntax checking, and autocompleting, the expression builder is designed to make building data flows easy. This article explains how to use the expression builder to effectively build your business logic.
-![Expression Builder](media/data-flow/expresion-builder.png "Expression Builder")
## Open Expression Builder
toLong(
### Data flow time evaluation Dataflow processes till milliseconds. For *2018-07-31T20:00:00.2170000*, you will see *2018-07-31T20:00:00.217* in output.
-In ADF portal, timestamp is being shown in the **current browser setting**, which can eliminate 217, but when you will run the data flow end to end, 217 (milliseconds part will be processed as well). You can use toString(myDateTimeColumn) as expression and see full precision data in preview. Process datetime as datetime rather than string for all practical purposes.
+In the portal for the service, timestamp is being shown in the **current browser setting**, which can eliminate 217, but when you will run the data flow end to end, 217 (milliseconds part will be processed as well). You can use toString(myDateTimeColumn) as expression and see full precision data in preview. Process datetime as datetime rather than string for all practical purposes.
## Next steps
data-factory Concepts Data Flow Performance https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/concepts-data-flow-performance.md
Title: Mapping data flow performance and tuning guide
-description: Learn about key factors that affect the performance of mapping data flows in Azure Data Factory.
+description: Learn about key factors that affect the performance of mapping data flows in Azure Data Factory and Azure Synapse Analytics pipelines.
Last updated 06/07/2021
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-Mapping data flows in Azure Data Factory provide a code-free interface to design and run data transformations at scale. If you're not familiar with mapping data flows, see the [Mapping Data Flow Overview](concepts-data-flow-overview.md). This article highlights various ways to tune and optimize your data flows so that they meet your performance benchmarks.
+Mapping data flows in Azure Data Factory and Synapse pipelines provide a code-free interface to design and run data transformations at scale. If you're not familiar with mapping data flows, see the [Mapping Data Flow Overview](concepts-data-flow-overview.md). This article highlights various ways to tune and optimize your data flows so that they meet your performance benchmarks.
Watch the below video to see shows some sample timings transforming data with data flows.
Watch the below video to see shows some sample timings transforming data with da
## Testing data flow logic
-When designing and testing data flows from the ADF UX, debug mode allows you to interactively test against a live Spark cluster. This allows you to preview data and execute your data flows without waiting for a cluster to warm up. For more information, see [Debug Mode](concepts-data-flow-debug-mode.md).
+When designing and testing data flows from UI, debug mode allows you to interactively test against a live Spark cluster. This allows you to preview data and execute your data flows without waiting for a cluster to warm up. For more information, see [Debug Mode](concepts-data-flow-debug-mode.md).
## Monitoring data flow performance
-Once you verify your transformation logic using debug mode, run your data flow end-to-end as an activity in a pipeline. Data flows are operationalized in a pipeline using the [execute data flow activity](control-flow-execute-data-flow-activity.md). The data flow activity has a unique monitoring experience compared to other Azure Data Factory activities that displays a detailed execution plan and performance profile of the transformation logic. To view detailed monitoring information of a data flow, click on the eyeglasses icon in the activity run output of a pipeline. For more information, see [Monitoring mapping data flows](concepts-data-flow-monitoring.md).
+Once you verify your transformation logic using debug mode, run your data flow end-to-end as an activity in a pipeline. Data flows are operationalized in a pipeline using the [execute data flow activity](control-flow-execute-data-flow-activity.md). The data flow activity has a unique monitoring experience compared to other activities that displays a detailed execution plan and performance profile of the transformation logic. To view detailed monitoring information of a data flow, click on the eyeglasses icon in the activity run output of a pipeline. For more information, see [Monitoring mapping data flows](concepts-data-flow-monitoring.md).
![Data Flow Monitor](media/data-flow/monitoring-details.png "Data Flow Monitor 2")
The **Optimize** tab contains settings to configure the partitioning scheme of t
![Screenshot shows the Optimize tab, which includes Partition option, Partition type, and Number of partitions.](media/data-flow/optimize.png)
-By default, *Use current partitioning* is selected which instructs Azure Data Factory keep the current output partitioning of the transformation. As repartitioning data takes time, *Use current partitioning* is recommended in most scenarios. Scenarios where you may want to repartition your data include after aggregates and joins that significantly skew your data or when using Source partitioning on a SQL DB.
+By default, *Use current partitioning* is selected which instructs the service keep the current output partitioning of the transformation. As repartitioning data takes time, *Use current partitioning* is recommended in most scenarios. Scenarios where you may want to repartition your data include after aggregates and joins that significantly skew your data or when using Source partitioning on a SQL DB.
To change the partitioning on any transformation, select the **Optimize** tab and select the **Set Partitioning** radio button. You are presented with a series of options for partitioning. The best method of partitioning differs based on your data volumes, candidate keys, null values, and cardinality. > [!IMPORTANT]
-> Single partition combines all the distributed data into a single partition. This is a very slow operation that also significantly affects all downstream transformation and writes. The Azure Data Factory highly recommends against using this option unless there is an explicit business reason to do so.
+> Single partition combines all the distributed data into a single partition. This is a very slow operation that also significantly affects all downstream transformation and writes. This option is strongly discouraged unless there is an explicit business reason to use it.
The following partitioning options are available in every transformation:
Round robin distributes data equally across partitions. Use round-robin when you
### Hash
-Azure Data Factory produces a hash of columns to produce uniform partitions such that rows with similar values fall in the same partition. When you use the Hash option, test for possible partition skew. You can set the number of physical partitions.
+The service produces a hash of columns to produce uniform partitions such that rows with similar values fall in the same partition. When you use the Hash option, test for possible partition skew. You can set the number of physical partitions.
### Dynamic range
If you have a good understanding of the cardinality of your data, key partitioni
## Logging level
-If you do not require every pipeline execution of your data flow activities to fully log all verbose telemetry logs, you can optionally set your logging level to "Basic" or "None". When executing your data flows in "Verbose" mode (default), you are requesting ADF to fully log activity at each individual partition level during your data transformation. This can be an expensive operation, so only enabling verbose when troubleshooting can improve your overall data flow and pipeline performance. "Basic" mode will only log transformation durations while "None" will only provide a summary of durations.
+If you do not require every pipeline execution of your data flow activities to fully log all verbose telemetry logs, you can optionally set your logging level to "Basic" or "None". When executing your data flows in "Verbose" mode (default), you are requesting the service to fully log activity at each individual partition level during your data transformation. This can be an expensive operation, so only enabling verbose when troubleshooting can improve your overall data flow and pipeline performance. "Basic" mode will only log transformation durations while "None" will only provide a summary of durations.
![Logging level](media/data-flow/logging.png "Set logging level")
If you do not require every pipeline execution of your data flow activities to f
Data flows run on Spark clusters that are spun up at run-time. The configuration for the cluster used is defined in the integration runtime (IR) of the activity. There are three performance considerations to make when defining your integration runtime: cluster type, cluster size, and time to live.
-For more information how to create an Integration Runtime, see [Integration Runtime in Azure Data Factory](concepts-integration-runtime.md).
+For more information how to create an Integration Runtime, see [Integration Runtime](concepts-integration-runtime.md).
### Cluster type
There are three available options for the type of Spark cluster spun up: general
If your data flow has many joins and lookups, you may want to use a **memory optimized** cluster. Memory optimized clusters can store more data in memory and will minimize any out-of-memory errors you may get. Memory optimized have the highest price-point per core, but also tend to result in more successful pipelines. If you experience any out of memory errors when executing data flows, switch to a memory optimized Azure IR configuration.
-**Compute optimized** aren't ideal for ETL workflows and aren't recommended by the Azure Data Factory team for most production workloads. For simpler, non-memory intensive data transformations such as filtering data or adding derived columns, compute-optimized clusters can be used at a cheaper price per core.
+**Compute optimized** aren't ideal for ETL workflows and aren't recommended for most production workloads. For simpler, non-memory intensive data transformations such as filtering data or adding derived columns, compute-optimized clusters can be used at a cheaper price per core.
### Cluster size
A best practice is to start small and scale up to meet your performance needs.
By default, every data flow activity spins up a new Spark cluster based upon the Azure IR configuration. Cold cluster start-up time takes a few minutes and data processing can't start until it is complete. If your pipelines contain multiple **sequential** data flows, you can enable a time to live (TTL) value. Specifying a time to live value keeps a cluster alive for a certain period of time after its execution completes. If a new job starts using the IR during the TTL time, it will reuse the existing cluster and start up time will greatly reduced. After the second job completes, the cluster will again stay alive for the TTL time.
-You can additionally minimize the startup time of warm clusters by setting the "Quick re-use" option in the Azure Integration runtime under Data Flow Properties. Setting this to true will tell ADF to not teardown the existing cluster after each job and instead re-use the existing cluster, essentially keeping the compute environment you've set in your Azure IR alive for up to the period of time specified in your TTL. This option makes for the shortest start-up time of your data flow activities when executing from a pipeline.
+You can additionally minimize the startup time of warm clusters by setting the "Quick re-use" option in the Azure Integration runtime under Data Flow Properties. Setting this to true will tell the service to not teardown the existing cluster after each job and instead re-use the existing cluster, essentially keeping the compute environment you've set in your Azure IR alive for up to the period of time specified in your TTL. This option makes for the shortest start-up time of your data flow activities when executing from a pipeline.
However, if most of your data flows execute in parallel, it is not recommended that you enable TTL for the IR that you use for those activities. Only one job can run on a single cluster at a time. If there is an available cluster, but two data flows start, only one will use the live cluster. The second job will spin up its own isolated cluster.
You can read from Azure SQL Database using a table or a SQL query. If you are ex
### Azure Synapse Analytics sources
-When using Azure Synapse Analytics, a setting called **Enable staging** exists in the source options. This allows ADF to read from Synapse using ```Staging```, which greatly improves read performance. Enabling ```Staging``` requires you to specify an Azure Blob Storage or Azure Data Lake Storage gen2 staging location in the data flow activity settings.
+When using Azure Synapse Analytics, a setting called **Enable staging** exists in the source options. This allows the service to read from Azure Synapse using ```Staging```, which greatly improves read performance. Enabling ```Staging``` requires you to specify an Azure Blob Storage or Azure Data Lake Storage gen2 staging location in the data flow activity settings.
![Enable staging](media/data-flow/enable-staging.png "Enable staging") ### File-based sources
-While data flows support a variety of file types, the Azure Data Factory recommends using the Spark-native Parquet format for optimal read and write times.
+While data flows support a variety of file types, the Spark-native Parquet format is recommended for optimal read and write times.
If you're running the same data flow on a set of files, we recommend reading from a folder, using wildcard paths or reading from a list of files. A single data flow activity run can process all of your files in batch. More information on how to set these settings can be found in the connector documentation such as [Azure Blob Storage](connector-azure-blob-storage.md#source-transformation).
With Azure SQL Database, the default partitioning should work in most cases. The
#### Impact of error row handling to performance
-When you enable error row handling ("continue on error") in the sink transformation, ADF will take an additional step before writing the compatible rows to your destination table. This additional step will have a small performance penalty that can be in the range of 5% added for this step with an additional small performance hit also added if you set the option to also with the incompatible rows to a log file.
+When you enable error row handling ("continue on error") in the sink transformation, the service will take an additional step before writing the compatible rows to your destination table. This additional step will have a small performance penalty that can be in the range of 5% added for this step with an additional small performance hit also added if you set the option to also with the incompatible rows to a log file.
#### Disabling indexes using a SQL Script
Schedule a resizing of your source and sink Azure SQL DB and DW before your pipe
### Azure Synapse Analytics sinks
-When writing to Azure Synapse Analytics, make sure that **Enable staging** is set to true. This enables ADF to write using [SQL Copy Command](/sql/t-sql/statements/copy-into-transact-sql) which effectively loads the data in bulk. You will need to reference an Azure Data Lake Storage gen2 or Azure Blob Storage account for staging of the data when using Staging.
+When writing to Azure Synapse Analytics, make sure that **Enable staging** is set to true. This enables the service to write using [SQL Copy Command](/sql/t-sql/statements/copy-into-transact-sql) which effectively loads the data in bulk. You will need to reference an Azure Data Lake Storage gen2 or Azure Blob Storage account for staging of the data when using Staging.
Other than Staging, the same best practices apply to Azure Synapse Analytics as Azure SQL Database. ### File-based sinks
-While data flows support a variety of file types, the Azure Data Factory recommends using the Spark-native Parquet format for optimal read and write times.
+While data flows support a variety of file types, the Spark-native Parquet format is recommended for optimal read and write times.
If the data is evenly distributed, **Use current partitioning** will be the fastest partitioning option for writing files.
Setting a naming **Pattern** will rename each partition file to a more user-frie
If a column corresponds to how you wish to output the data, you can select **As data in column**. This reshuffles the data and can impact performance if the columns are not evenly distributed.
-**Output to single file** combines all the data into a single partition. This leads to long write times, especially for large datasets. The Azure Data Factory team highly recommends **not** choosing this option unless there is an explicit business reason to do so.
+**Output to single file** combines all the data into a single partition. This leads to long write times, especially for large datasets. This option is strongly discouraged unless there is an explicit business reason to use it.
### CosmosDB sinks
If you use literal values in your join conditions or have multiple matches on bo
#### Sorting before joins
-Unlike merge join in tools like SSIS, the join transformation isn't a mandatory merge join operation. The join keys don't require sorting prior to the transformation. The Azure Data Factory team doesn't recommend using Sort transformations in mapping data flows.
+Unlike merge join in tools like SSIS, the join transformation isn't a mandatory merge join operation. The join keys don't require sorting prior to the transformation. Using Sort transformations in mapping data flows is not recommended.
### Window transformation performance
When building complex pipelines with multiple data flows, your logical flow can
### Executing data flows in parallel
-If you execute multiple data flows in parallel, ADF spins up separate Spark clusters for each activity. This allows for each job to be isolated and run in parallel, but will lead to multiple clusters running at the same time.
+If you execute multiple data flows in parallel, the service spins up separate Spark clusters for each activity. This allows for each job to be isolated and run in parallel, but will lead to multiple clusters running at the same time.
If your data flows execute in parallel, its recommended to not enable the Azure IR time to live property as it will lead to multiple unused warm pools.
If your data flows execute in parallel, its recommended to not enable the Azure
### Execute data flows sequentially
-If you execute your data flow activities in sequence, it is recommended that you set a TTL in the Azure IR configuration. ADF will reuse the compute resources resulting in a faster cluster start up time. Each activity will still be isolated receive a new Spark context for each execution. To reduce the time between sequential activities even more, set the "quick re-use" checkbox on the Azure IR to tell ADF to re-use the existing cluster.
+If you execute your data flow activities in sequence, it is recommended that you set a TTL in the Azure IR configuration. The service will reuse the compute resources resulting in a faster cluster start up time. Each activity will still be isolated receive a new Spark context for each execution. To reduce the time between sequential activities even more, set the "quick re-use" checkbox on the Azure IR to tell the service to re-use the existing cluster.
### Overloading a single data flow
-If you put all of your logic inside of a single data flow, ADF will execute the entire job on a single Spark instance. While this may seem like a way to reduce costs, it mixes together different logical flows and can be difficult to monitor and debug. If one component fails, all other parts of the job will fail as well. The Azure Data Factory team recommends organizing data flows by independent flows of business logic. If your data flow becomes too large, splitting it into separates components will make monitoring and debugging easier. While there is no hard limit on the number of transformations in a data flow, having too many will make the job complex.
+If you put all of your logic inside of a single data flow, the service will execute the entire job on a single Spark instance. While this may seem like a way to reduce costs, it mixes together different logical flows and can be difficult to monitor and debug. If one component fails, all other parts of the job will fail as well. Organizing data flows by independent flows of business logic is recommended. If your data flow becomes too large, splitting it into separates components will make monitoring and debugging easier. While there is no hard limit on the number of transformations in a data flow, having too many will make the job complex.
### Execute sinks in parallel The default behavior of data flow sinks is to execute each sink sequentially, in a serial manner, and to fail the data flow when an error is encountered in the sink. Additionally, all sinks are defaulted to the same group unless you go into the data flow properties and set different priorities for the sinks.
-Data flows allow you to group sinks together into groups from the data flow properties tab in the UI designer. You can both set the order of execution of your sinks as well as to group sinks together using the same group number. To help manage groups, you can ask ADF to run sinks in the same group, to run in parallel.
+Data flows allow you to group sinks together into groups from the data flow properties tab in the UI designer. You can both set the order of execution of your sinks as well as to group sinks together using the same group number. To help manage groups, you can ask the service to run sinks in the same group, to run in parallel.
On the pipeline execute data flow activity under the "Sink Properties" section is an option to turn on parallel sink loading. When you enable "run in parallel", you are instructing data flows write to connected sinks at the same time rather than in a sequential manner. In order to utilize the parallel option, the sinks must be group together and connected to the same stream via a New Branch or Conditional Split.
data-factory Concepts Datasets Linked Services https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/concepts-datasets-linked-services.md
Title: Datasets
-description: 'Learn about datasets in Data Factory. Datasets represent input/output data.'
+description: Learn about datasets in Azure Data Factory and Azure Synapse Analytics pipelines. Datasets represent input/output data.
Last updated 08/24/2020
-# Datasets in Azure Data Factory
+# Datasets in Azure Data Factory and Azure Synapse Analytics
> [!div class="op_single_selector" title1="Select the version of Data Factory service you are using:"] > * [Version 1](v1/data-factory-create-datasets.md) > * [Current version](concepts-datasets-linked-services.md)
Last updated 08/24/2020
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article describes what datasets are, how they are defined in JSON format, and how they are used in Azure Data Factory pipelines.
+This article describes what datasets are, how they are defined in JSON format, and how they are used in Azure Data Factory and Synapse pipelines.
-If you are new to Data Factory, see [Introduction to Azure Data Factory](introduction.md) for an overview.
+If you are new to Data Factory, see [Introduction to Azure Data Factory](introduction.md) for an overview. For more information about Azure Synapse see [What is Azure Synapse](../synapse-analytics/overview-what-is.md)
## Overview
-A data factory can have one or more pipelines. A **pipeline** is a logical grouping of **activities** that together perform a task. The activities in a pipeline define actions to perform on your data. Now, a **dataset** is a named view of data that simply points or references the data you want to use in your **activities** as inputs and outputs. Datasets identify data within different data stores, such as tables, files, folders, and documents. For example, an Azure Blob dataset specifies the blob container and folder in Blob storage from which the activity should read the data.
+A data factory or Synapse workspace can have one or more pipelines. A **pipeline** is a logical grouping of **activities** that together perform a task. The activities in a pipeline define actions to perform on your data. Now, a **dataset** is a named view of data that simply points or references the data you want to use in your **activities** as inputs and outputs. Datasets identify data within different data stores, such as tables, files, folders, and documents. For example, an Azure Blob dataset specifies the blob container and folder in Blob storage from which the activity should read the data.
-Before you create a dataset, you must create a [**linked service**](concepts-linked-services.md) to link your data store to the data factory. Linked services are much like connection strings, which define the connection information needed for Data Factory to connect to external resources. Think of it this way; the dataset represents the structure of the data within the linked data stores, and the linked service defines the connection to the data source. For example, an Azure Storage linked service links a storage account to the data factory. An Azure Blob dataset represents the blob container and the folder within that Azure Storage account that contains the input blobs to be processed.
+Before you create a dataset, you must create a [**linked service**](concepts-linked-services.md) to link your data store to the service. Linked services are much like connection strings, which define the connection information needed for the service to connect to external resources. Think of it this way; the dataset represents the structure of the data within the linked data stores, and the linked service defines the connection to the data source. For example, an Azure Storage linked service links a storage account. An Azure Blob dataset represents the blob container and the folder within that Azure Storage account that contains the input blobs to be processed.
-Here is a sample scenario. To copy data from Blob storage to a SQL Database, you create two linked
+Here is a sample scenario. To copy data from Blob storage to a SQL Database, you create two linked
-The following diagram shows the relationships among pipeline, activity, dataset, and linked service in Data Factory:
+The following diagram shows the relationships among pipeline, activity, dataset, and linked
![Relationship between pipeline, activity, dataset, linked services](media/concepts-datasets-linked-services/relationship-between-data-factory-entities.png) ## Dataset JSON
-A dataset in Data Factory is defined in the following JSON format:
+A dataset is defined in the following JSON format:
```json {
The following table describes properties in the above JSON:
Property | Description | Required | -- | -- | -- |
-name | Name of the dataset. See [Azure Data Factory - Naming rules](naming-rules.md). | Yes |
+name | Name of the dataset. See [Naming rules](naming-rules.md). | Yes |
type | Type of the dataset. Specify one of the types supported by Data Factory (for example: DelimitedText, AzureSqlTable). <br/><br/>For details, see [Dataset types](#dataset-type). | Yes | schema | Schema of the dataset, represents the physical data type and shape. | No | typeProperties | The type properties are different for each type. For details on the supported types and their properties, see [Dataset type](#dataset-type). | Yes |
-When you import the schema of dataset, select the **Import Schema** button and choose to import from the source or from a local file. In most cases, you'll import the schema directly from the source. But if you already have a local schema file (a Parquet file or CSV with headers), you can direct Data Factory to base the schema on that file.
+When you import the schema of dataset, select the **Import Schema** button and choose to import from the source or from a local file. In most cases, you'll import the schema directly from the source. But if you already have a local schema file (a Parquet file or CSV with headers), you can direct the service to base the schema on that file.
In copy activity, datasets are used in source and sink. Schema defined in dataset is optional as reference. If you want to apply column/field mapping between source and sink, refer to [Schema and type mapping](copy-activity-schema-and-type-mapping.md).
In Data Flow, datasets are used in source and sink transformations. The datasets
## Dataset type
-Azure Data Factory supports many different types of datasets, depending on the data stores you use. You can find the list of data stores supported by Data Factory from [Connector overview](connector-overview.md) article. Click a data store to learn how to create a linked service and a dataset for it.
+The service supports many different types of datasets, depending on the data stores you use. You can find the list of supported data stores from [Connector overview](connector-overview.md) article. Click a data store to learn how to create a linked service and a dataset for it.
For example, for a Delimited Text dataset, the dataset type is set to **DelimitedText** as shown in the following JSON sample:
You can create datasets by using one of these tools or SDKs: [.NET API](quicksta
## Current version vs. version 1 datasets
-Here are some differences between Data Factory and Data Factory version 1 datasets:
+Here are some differences between datasets in Data Factory current version (and Azure Synapse), and the legacy Data Factory version 1:
- The external property is not supported in the current version. It's replaced by a [trigger](concepts-pipeline-execution-triggers.md). - The policy and availability properties are not supported in the current version. The start time for a pipeline depends on [triggers](concepts-pipeline-execution-triggers.md).
data-factory Connect Data Factory To Azure Purview https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connect-data-factory-to-azure-purview.md
-+ Last updated 12/3/2020
For how to collect lineage from Azure Data Factory, see [data factory lineage](.
## Next steps [Catalog lineage user guide](../purview/catalog-lineage-user-guide.md)
-[Tutorial: Push Data Factory lineage data to Azure Purview](turorial-push-lineage-to-purview.md)
+[Tutorial: Push Data Factory lineage data to Azure Purview](turorial-push-lineage-to-purview.md)
data-factory Connector Azure Blob Storage https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-azure-blob-storage.md
Title: Copy and transform data in Azure Blob storage
-description: Learn how to copy data to and from Blob storage, and transform data in Blob storage by using Data Factory or Azure Synapse Analytics.
+description: Learn how to copy data to and from Blob storage, and transform data in Blob storage using Azure Data Factory or Azure Synapse Analytics.
To learn details about the properties, check [Delete activity](delete-activity.m
## Next steps
-For a list of data stores that the Copy activity supports as sources and sinks, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of data stores that the Copy activity supports as sources and sinks, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Azure Cosmos Db https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-azure-cosmos-db.md
Title: Copy and transform data in Azure Cosmos DB (SQL API)
-description: Learn how to copy data to and from Azure Cosmos DB (SQL API), and transform data in Azure Cosmos DB (SQL API) by using Data Factory.
+description: Learn how to copy data to and from Azure Cosmos DB (SQL API), and transform data in Azure Cosmos DB (SQL API) using Azure Data Factory and Azure Synapse Analytics.
Last updated 05/18/2021
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to use Copy Activity in Azure Data Factory to copy data from and to Azure Cosmos DB (SQL API), and use Data Flow to transform data in Azure Cosmos DB (SQL API). To learn about Azure Data Factory, read the [introductory article](introduction.md).
--
+This article outlines how to use Copy Activity in Azure Data Factory to copy data from and to Azure Cosmos DB (SQL API), and use Data Flow to transform data in Azure Cosmos DB (SQL API). To learn more, read the introductory articles for [Azure Data Factory](introduction.md) and [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md).
>[!NOTE] >This connector only support Cosmos DB SQL API. For MongoDB API, refer to [connector for Azure Cosmos DB's API for MongoDB](connector-azure-cosmos-db-mongodb-api.md). Other API types are not supported now.
For Copy activity, this Azure Cosmos DB (SQL API) connector supports:
- Write to Azure Cosmos DB as **insert** or **upsert**. - Import and export JSON documents as-is, or copy data from or to a tabular dataset. Examples include a SQL database and a CSV file. To copy documents as-is to or from JSON files or to or from another Azure Cosmos DB collection, see [Import and export JSON documents](#import-and-export-json-documents).
-Data Factory integrates with the [Azure Cosmos DB bulk executor library](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started) to provide the best performance when you write to Azure Cosmos DB.
+Data Factory and Synapse pipelines integrate with the [Azure Cosmos DB bulk executor library](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started) to provide the best performance when you write to Azure Cosmos DB.
> [!TIP] > The [Data Migration video](https://youtu.be/5-SRNiC_qOU) walks you through the steps of copying data from Azure Blob storage to Azure Cosmos DB. The video also describes performance-tuning considerations for ingesting data to Azure Cosmos DB in general.
Data Factory integrates with the [Azure Cosmos DB bulk executor library](https:/
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide details about properties you can use to define Data Factory entities that are specific to Azure Cosmos DB (SQL API).
+The following sections provide details about properties you can use to define entities that are specific to Azure Cosmos DB (SQL API).
## Linked service properties
These properties are supported for the linked service:
| database | Specify the name of the database. | Yes | | servicePrincipalId | Specify the application's client ID. | Yes | | servicePrincipalCredentialType | The credential type to use for service principal authentication. Allowed values are **ServicePrincipalKey** and **ServicePrincipalCert**. | Yes |
-| servicePrincipalCredential | The service principal credential. <br/> When you use **ServicePrincipalKey** as the credential type, specify the the application's key. Mark this field as **SecureString** to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). <br/> When you use **ServicePrincipalCert** as the credential, reference a certificate in Azure Key Vault. | Yes |
+| servicePrincipalCredential | The service principal credential. <br/> When you use **ServicePrincipalKey** as the credential type, specify the application's key. Mark this field as **SecureString** to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). <br/> When you use **ServicePrincipalCert** as the credential, reference a certificate in Azure Key Vault. | Yes |
| tenant | Specify the tenant information (domain name or tenant ID) under which your application resides. Retrieve it by hovering the mouse in the upper-right corner of the Azure portal. | Yes |
-| azureCloudType | For service principal authentication, specify the type of Azure cloud environment to which your Azure Active Directory application is registered. <br/> Allowed values are **AzurePublic**, **AzureChina**, **AzureUsGovernment**, and **AzureGermany**. By default, the data factory's cloud environment is used. | No |
+| azureCloudType | For service principal authentication, specify the type of Azure cloud environment to which your Azure Active Directory application is registered. <br/> Allowed values are **AzurePublic**, **AzureChina**, **AzureUsGovernment**, and **AzureGermany**. By default, the service's cloud environment is used. | No |
| connectVia | The [integration runtime](concepts-integration-runtime.md) to be used to connect to the data store. You can use the Azure integration runtime or a self-hosted integration runtime if your data store is in a private network. If not specified, the default Azure integration runtime is used. |No | **Example: using service principal key authentication**
You can also store service principal key in Azure Key Vault.
>[!NOTE] >Currently, the managed identity authentication is not supported in data flow.
-A data factory can be associated with a [managed identity for Azure resources](data-factory-service-identity.md), which represents this specific data factory. You can directly use this managed identity for Cosmos DB authentication, similar to using your own service principal. It allows this designated factory to access and copy data to or from your Cosmos DB.
+A data factory or Synapse pipeline can be associated with a [managed identity for Azure resources](data-factory-service-identity.md), which represents this specific service instance. You can directly use this managed identity for Cosmos DB authentication, similar to using your own service principal. It allows this designated resource to access and copy data to or from your Cosmos DB.
To use managed identities for Azure resource authentication, follow these steps.
-1. [Retrieve the Data Factory managed identity information](data-factory-service-identity.md#retrieve-managed-identity) by copying the value of the **managed identity object ID** generated along with your factory.
+1. [Retrieve the managed identity information](data-factory-service-identity.md#retrieve-managed-identity) by copying the value of the **managed identity object ID** generated along with your service.
2. Grant the managed identity proper permission. See examples on how permission works in Cosmos DB from [Access control lists on files and directories](../cosmos-db/how-to-setup-rbac.md). More specifically, create a role definition, and assign the role to the managed identity.
If you use "DocumentDbCollectionSource" type source, it is still supported as-is
] ```
-When copy data from Cosmos DB, unless you want to [export JSON documents as-is](#import-and-export-json-documents), the best practice is to specify the mapping in copy activity. Data Factory honors the mapping you specified on the activity - if a row doesn't contain a value for a column, a null value is provided for the column value. If you don't specify a mapping, Data Factory infers the schema by using the first row in the data. If the first row doesn't contain the full schema, some columns will be missing in the result of the activity operation.
+When copying data from Cosmos DB, unless you want to [export JSON documents as-is](#import-and-export-json-documents), the best practice is to specify the mapping in copy activity. The service honors the mapping you specified on the activity - if a row doesn't contain a value for a column, a null value is provided for the column value. If you don't specify a mapping, the service infers the schema by using the first row in the data. If the first row doesn't contain the full schema, some columns will be missing in the result of the activity operation.
### Azure Cosmos DB (SQL API) as sink
The following properties are supported in the Copy Activity **sink** section:
| Property | Description | Required | |: |: |: | | type | The **type** property of the Copy Activity sink must be set to **CosmosDbSqlApiSink**. |Yes |
-| writeBehavior |Describes how to write data to Azure Cosmos DB. Allowed values: **insert** and **upsert**.<br/><br/>The behavior of **upsert** is to replace the document if a document with the same ID already exists; otherwise, insert the document.<br /><br />**Note**: Data Factory automatically generates an ID for a document if an ID isn't specified either in the original document or by column mapping. This means that you must ensure that, for **upsert** to work as expected, your document has an ID. |No<br />(the default is **insert**) |
-| writeBatchSize | Data Factory uses the [Azure Cosmos DB bulk executor library](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started) to write data to Azure Cosmos DB. The **writeBatchSize** property controls the size of documents that ADF provides to the library. You can try increasing the value for **writeBatchSize** to improve performance and decreasing the value if your document size being large - see below tips. |No<br />(the default is **10,000**) |
-| disableMetricsCollection | Data Factory collects metrics such as Cosmos DB RUs for copy performance optimization and recommendations. If you are concerned with this behavior, specify `true` to turn it off. | No (default is `false`) |
+| writeBehavior |Describes how to write data to Azure Cosmos DB. Allowed values: **insert** and **upsert**.<br/><br/>The behavior of **upsert** is to replace the document if a document with the same ID already exists; otherwise, insert the document.<br /><br />**Note**: The service automatically generates an ID for a document if an ID isn't specified either in the original document or by column mapping. This means that you must ensure that, for **upsert** to work as expected, your document has an ID. |No<br />(the default is **insert**) |
+| writeBatchSize | The service uses the [Azure Cosmos DB bulk executor library](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started) to write data to Azure Cosmos DB. The **writeBatchSize** property controls the size of documents the service provides to the library. You can try increasing the value for **writeBatchSize** to improve performance and decreasing the value if your document size being large - see below tips. |No<br />(the default is **10,000**) |
+| disableMetricsCollection | The service collects metrics such as Cosmos DB RUs for copy performance optimization and recommendations. If you are concerned with this behavior, specify `true` to turn it off. | No (default is `false`) |
| maxConcurrentConnections |The upper limit of concurrent connections established to the data store during the activity run. Specify a value only when you want to limit concurrent connections.| No |
When transforming data in mapping data flow, you can read and write to collectio
Settings specific to Azure Cosmos DB are available in the **Source Options** tab of the source transformation.
-**Include system columns:** If true, ```id```, ```_ts```, and other system columns will be included in your data flow metadata from CosmosDB. When updating collections, it is important to include this so that you can grab the existing row id.
+**Include system columns:** If true, ```id```, ```_ts```, and other system columns will be included in your data flow metadata from CosmosDB. When updating collections, it is important to include this so that you can grab the existing row ID.
**Page size:** The number of documents per page of the query result. Default is "-1" which uses the service dynamic page up to 1000.
Settings specific to Azure Cosmos DB are available in the **Source Options** tab
#### JSON Settings
-**Single document:** Select this option if ADF is to treat the entire file as a single JSON doc.
+**Single document:** Select this option if the service is to treat the entire file as a single JSON doc.
**Unquoted column names:** Select this option if column names in the JSON as not quoted.
Settings specific to Azure Cosmos DB are available in the **Settings** tab of th
**Batch size**: An integer that represents how many objects are being written to Cosmos DB collection in each batch. Usually, starting with the default batch size is sufficient. To further tune this value, note: - Cosmos DB limits single request's size to 2MB. The formula is "Request Size = Single Document Size * Batch Size". If you hit error saying "Request size is too large", reduce the batch size value.-- The larger the batch size, the better throughput ADF can achieve, while make sure you allocate enough RUs to empower your workload.
+- The larger the batch size, the better throughput the service can achieve, while make sure you allocate enough RUs to empower your workload.
**Partition key:** Enter a string that represents the partition key for your collection. Example: ```/movies/title```
To learn details about the properties, check [Lookup activity](control-flow-look
You can use this Azure Cosmos DB (SQL API) connector to easily: * Copy documents between two Azure Cosmos DB collections as-is.
-* Import JSON documents from various sources to Azure Cosmos DB, including from Azure Blob storage, Azure Data Lake Store, and other file-based stores that Azure Data Factory supports.
+* Import JSON documents from various sources to Azure Cosmos DB, including from Azure Blob storage, Azure Data Lake Store, and other file-based stores that the service supports.
* Export JSON documents from an Azure Cosmos DB collection to various file-based stores. To achieve schema-agnostic copy:
To achieve schema-agnostic copy:
## Migrate from relational database to Cosmos DB
-When migrating from a relational database e.g. SQL Server to Azure Cosmos DB, copy activity can easily map tabular data from source to flatten JSON documents in Cosmos DB. In some cases, you may want to redesign the data model to optimize it for the NoSQL use-cases according to [Data modeling in Azure Cosmos DB](../cosmos-db/modeling-data.md), for example, to denormalize the data by embedding all of the related sub-items within one JSON document. For such case, refer to [this article](../cosmos-db/migrate-relational-to-cosmos-db-sql-api.md) with a walkthrough on how to achieve it using Azure Data Factory copy activity.
+When migrating from a relational database e.g. SQL Server to Azure Cosmos DB, copy activity can easily map tabular data from source to flatten JSON documents in Cosmos DB. In some cases, you may want to redesign the data model to optimize it for the NoSQL use-cases according to [Data modeling in Azure Cosmos DB](../cosmos-db/modeling-data.md), for example, to de-normalize the data by embedding all of the related sub-items within one JSON document. For such case, refer to [this article](../cosmos-db/migrate-relational-to-cosmos-db-sql-api.md) with a walk-through on how to achieve it using the copy activity.
## Next steps
-For a list of data stores that Copy Activity supports as sources and sinks in Azure Data Factory, see [supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of data stores that Copy Activity supports as sources and sinks, see [supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Azure Data Lake Store https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-azure-data-lake-store.md
Title: Copy data to or from Azure Data Lake Storage Gen1
-description: Learn how to copy data from supported source data stores to Azure Data Lake Store, or from Data Lake Store to supported sink stores, by using Data Factory.
+description: Learn how to copy data from supported source data stores to Azure Data Lake Store, or from Data Lake Store to supported sink stores, using Azure Data Factory or Azure Synapse Analytics pipelines.
Last updated 07/19/2021
-# Copy data to or from Azure Data Lake Storage Gen1 using Azure Data Factory
+# Copy data to or from Azure Data Lake Storage Gen1 using Azure Data Factory or Azure Synapse Analytics
> [!div class="op_single_selector" title1="Select the version of Azure Data Factory that you're using:"] >
Last updated 07/19/2021
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to copy data to and from Azure Data Lake Storage Gen1. To learn about Azure Data Factory, read the [introductory article](introduction.md).
+This article outlines how to copy data to and from Azure Data Lake Storage Gen1. To learn more, read the introductory article for [Azure Data Factory](introduction.md) or [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md).
## Supported capabilities
Specifically, with this connector you can:
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide information about properties that are used to define Data Factory entities specific to Azure Data Lake Store.
+The following sections provide information about properties that are used to define entities specific to Azure Data Lake Store.
## Linked service properties
The following properties are supported:
| Property | Description | Required | |: |: |: | | servicePrincipalId | Specify the application's client ID. | Yes |
-| servicePrincipalKey | Specify the application's key. Mark this field as a `SecureString` to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
+| servicePrincipalKey | Specify the application's key. Mark this field as a `SecureString` to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
| tenant | Specify the tenant information, such as domain name or tenant ID, under which your application resides. You can retrieve it by hovering the mouse in the upper-right corner of the Azure portal. | Yes |
-| azureCloudType | For service principal authentication, specify the type of Azure cloud environment to which your Azure Active Directory application is registered. <br/> Allowed values are **AzurePublic**, **AzureChina**, **AzureUsGovernment**, and **AzureGermany**. By default, the data factory's cloud environment is used. | No |
+| azureCloudType | For service principal authentication, specify the type of Azure cloud environment to which your Azure Active Directory application is registered. <br/> Allowed values are **AzurePublic**, **AzureChina**, **AzureUsGovernment**, and **AzureGermany**. By default, the the service's cloud environment is used. | No |
**Example:**
The following properties are supported:
### <a name="managed-identity"></a> Use system-assigned managed identity authentication
-A data factory can be associated with a [system-assigned managed identity](data-factory-service-identity.md), which represents this specific data factory. You can directly use this system-assigned managed identity for Data Lake Store authentication, similar to using your own service principal. It allows this designated factory to access and copy data to or from Data Lake Store.
+A data factory or Synapse workspace can be associated with a [system-assigned managed identity](data-factory-service-identity.md), which represents the service for authentication. You can directly use this system-assigned managed identity for Data Lake Store authentication, similar to using your own service principal. It allows this designated resource to access and copy data to or from Data Lake Store.
To use system-assigned managed identity authentication, follow these steps.
-1. [Retrieve the data factory system-assigned managed identity information](data-factory-service-identity.md#retrieve-managed-identity) by copying the value of the "Service Identity Application ID" generated along with your factory.
+1. [Retrieve the system-assigned managed identity information](data-factory-service-identity.md#retrieve-managed-identity) by copying the value of the "Service Identity Application ID" generated along with your factory or Synapse workspace.
2. Grant the system-assigned managed identity access to Data Lake Store. See examples on how permission works in Data Lake Storage Gen1 from [Access control in Azure Data Lake Storage Gen1](../data-lake-store/data-lake-store-access-control.md#common-scenarios-related-to-permissions). - **As source**: In **Data explorer** > **Access**, grant at least **Execute** permission for ALL upstream folders including the root, along with **Read** permission for the files to copy. You can choose to add to **This folder and all children** for recursive, and add as **an access permission and a default permission entry**. There's no requirement on account-level access control (IAM). - **As sink**: In **Data explorer** > **Access**, grant at least **Execute** permission for ALL upstream folders including the root, along with **Write** permission for the sink folder. You can choose to add to **This folder and all children** for recursive, and add as **an access permission and a default permission entry**.
-In Azure Data Factory, you don't need to specify any properties besides the general Data Lake Store information in the linked service.
+You don't need to specify any properties other than the general Data Lake Store information in the linked service.
**Example:**
The following properties are supported for Azure Data Lake Store Gen1 under `sto
| type | The type property under `storeSettings` must be set to **AzureDataLakeStoreReadSettings**. | Yes | | ***Locate the files to copy:*** | | | | OPTION 1: static path<br> | Copy from the given folder/file path specified in the dataset. If you want to copy all files from a folder, additionally specify `wildcardFileName` as `*`. | |
-| OPTION 2: name range<br>- listAfter | Retrieve the folders/files whose name is after this value alphabetically (exclusive). It utilizes the service-side filter for ADLS Gen1, which provides better performance than a wildcard filter. <br/>Data factory applies this filter to the path defined in dataset, and only one entity level is supported. See more examples in [Name range filter examples](#name-range-filter-examples). | No |
-| OPTION 2: name range<br/>- listBefore | Retrieve the folders/files whose name is before this value alphabetically (inclusive). It utilizes the service-side filter for ADLS Gen1, which provides better performance than a wildcard filter.<br>Data factory applies this filter to the path defined in dataset, and only one entity level is supported. See more examples in [Name range filter examples](#name-range-filter-examples). | No |
+| OPTION 2: name range<br>- listAfter | Retrieve the folders/files whose name is after this value alphabetically (exclusive). It utilizes the service-side filter for ADLS Gen1, which provides better performance than a wildcard filter. <br/>The service applies this filter to the path defined in dataset, and only one entity level is supported. See more examples in [Name range filter examples](#name-range-filter-examples). | No |
+| OPTION 2: name range<br/>- listBefore | Retrieve the folders/files whose name is before this value alphabetically (inclusive). It utilizes the service-side filter for ADLS Gen1, which provides better performance than a wildcard filter.<br>The service applies this filter to the path defined in dataset, and only one entity level is supported. See more examples in [Name range filter examples](#name-range-filter-examples). | No |
| OPTION 3: wildcard<br>- wildcardFolderPath | The folder path with wildcard characters to filter source folders. <br>Allowed wildcards are: `*` (matches zero or more characters) and `?` (matches zero or single character); use `^` to escape if your actual folder name has wildcard or this escape char inside. <br>See more examples in [Folder and file filter examples](#folder-and-file-filter-examples). | No | | OPTION 3: wildcard<br>- wildcardFileName | The file name with wildcard characters under the given folderPath/wildcardFolderPath to filter source files. <br>Allowed wildcards are: `*` (matches zero or more characters) and `?` (matches zero or single character); use `^` to escape if your actual file name has wildcard or this escape char inside. See more examples in [Folder and file filter examples](#folder-and-file-filter-examples). | Yes | | OPTION 4: a list of files<br>- fileListPath | Indicates to copy a given file set. Point to a text file that includes a list of files you want to copy, one file per line, which is the relative path to the path configured in the dataset.<br/>When using this option, do not specify file name in dataset. See more examples in [File list examples](#file-list-examples). |No |
The following properties are supported for Azure Data Lake Store Gen1 under `sto
This section describes the resulting behavior of name range filters.
-| Sample source structure | ADF configuration | Result |
+| Sample source structure | Configuration | Result |
|: |: |: | |root<br/>&nbsp;&nbsp;&nbsp;&nbsp;a<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;ax<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file2.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;ax.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;b<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file3.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;bx.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;c<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file4.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;cx.csv| **In dataset:**<br>- Folder path: `root`<br><br>**In copy activity source:**<br>- List after: `a`<br>- List before: `b`| Then the following files will be copied:<br><br>root<br/>&nbsp;&nbsp;&nbsp;&nbsp;ax<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file2.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;ax.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;b<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file3.csv |
This section describes the resulting behavior of using file list path in copy ac
Assuming you have the following source folder structure and want to copy the files in bold:
-| Sample source structure | Content in FileListToCopy.txt | ADF configuration |
+| Sample source structure | Content in FileListToCopy.txt | Configuration |
| | | | | root<br/>&nbsp;&nbsp;&nbsp;&nbsp;FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File2.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File3.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File5.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;Metadata<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;FileListToCopy.txt | File1.csv<br>Subfolder1/File3.csv<br>Subfolder1/File5.csv | **In dataset:**<br>- Folder path: `root/FolderA`<br><br>**In copy activity source:**<br>- File list path: `root/Metadata/FileListToCopy.txt` <br><br>The file list path points to a text file in the same data store that includes a list of files you want to copy, one file per line with the relative path to the path configured in the dataset. |
This section describes the resulting behavior of the copy operation for differen
## Preserve ACLs to Data Lake Storage Gen2 >[!TIP]
->To copy data from Azure Data Lake Storage Gen1 into Gen2 in general, see [Copy data from Azure Data Lake Storage Gen1 to Gen2 with Azure Data Factory](load-azure-data-lake-storage-gen2-from-gen1.md) for a walk-through and best practices.
+>To copy data from Azure Data Lake Storage Gen1 into Gen2 in general, see [Copy data from Azure Data Lake Storage Gen1 to Gen2](load-azure-data-lake-storage-gen2-from-gen1.md) for a walk-through and best practices.
If you want to replicate the access control lists (ACLs) along with data files when you upgrade from Data Lake Storage Gen1 to Data Lake Storage Gen2, see [Preserve ACLs from Data Lake Storage Gen1](copy-activity-preserve-metadata.md#preserve-acls).
In the source transformation, you can read from a container, folder, or individu
![Source options](media/data-flow/sourceOptions1.png "Source options")
-**Wildcard path:** Using a wildcard pattern will instruct ADF to loop through each matching folder and file in a single Source transformation. This is an effective way to process multiple files within a single flow. Add multiple wildcard matching patterns with the + sign that appears when hovering over your existing wildcard pattern.
+**Wildcard path:** Using a wildcard pattern will instruct the service to loop through each matching folder and file in a single Source transformation. This is an effective way to process multiple files within a single flow. Add multiple wildcard matching patterns with the + sign that appears when hovering over your existing wildcard pattern.
From your source container, choose a series of files that match a pattern. Only container can be specified in the dataset. Your wildcard path must therefore also include your folder path from the root folder.
First, set a wildcard to include all paths that are the partitioned folders plus
![Partition source file settings](media/data-flow/partfile2.png "Partition file setting")
-Use the Partition Root Path setting to define what the top level of the folder structure is. When you view the contents of your data via a data preview, you'll see that ADF will add the resolved partitions found in each of your folder levels.
+Use the Partition Root Path setting to define what the top level of the folder structure is. When you view the contents of your data via a data preview, you'll see that the service will add the resolved partitions found in each of your folder levels.
![Partition root path](media/data-flow/partfile1.png "Partition root path preview")
To learn details about the properties, check [Delete activity](delete-activity.m
## Legacy models >[!NOTE]
->The following models are still supported as-is for backward compatibility. You are suggested to use the new model mentioned in above sections going forward, and the ADF authoring UI has switched to generating the new model.
+>The following models are still supported as-is for backward compatibility. You are suggested to use the new model mentioned in above sections going forward, and the authoring UI has switched to generating the new model.
### Legacy dataset model
To learn details about the properties, check [Delete activity](delete-activity.m
## Next steps
-For a list of data stores supported as sources and sinks by the copy activity in Azure Data Factory, see [supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of data stores supported as sources and sinks by the copy activity, see [supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Azure Databricks Delta Lake https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-azure-databricks-delta-lake.md
Title: Copy data to and from Azure Databricks Delta Lake
-description: Learn how to copy data to and from Azure Databricks Delta Lake by using a copy activity in an Azure Data Factory pipeline.
+description: Learn how to copy data to and from Azure Databricks Delta Lake by using a copy activity in an Azure Data Factory or Azure Synapse Analytics pipeline.
Last updated 06/16/2021
-# Copy data to and from Azure Databricks Delta Lake by using Azure Data Factory
+# Copy data to and from Azure Databricks Delta Lake using Azure Data Factory or Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to use the Copy activity in Azure Data Factory to copy data to and from Azure Databricks Delta Lake. It builds on the [Copy activity in Azure Data Factory](copy-activity-overview.md) article, which presents a general overview of copy activity.
+This article outlines how to use the Copy activity in Azure Data Factory and Azure Synapse to copy data to and from Azure Databricks Delta Lake. It builds on the [Copy activity](copy-activity-overview.md) article, which presents a general overview of copy activity.
## Supported capabilities
This Azure Databricks Delta Lake connector is supported for the following activi
- [Copy activity](copy-activity-overview.md) with a [supported source/sink matrix](copy-activity-overview.md) table - [Lookup activity](control-flow-lookup-activity.md)
-In general, Azure Data Factory supports Delta Lake with the following capabilities to meet your various needs.
+In general, the service supports Delta Lake with the following capabilities to meet your various needs.
- Copy activity supports Azure Databricks Delta Lake connector to copy data from any supported source data store to Azure Databricks delta lake table, and from delta lake table to any supported sink data store. It leverages your Databricks cluster to perform the data movement, see details in [Prerequisites section](#prerequisites). - [Mapping Data Flow](concepts-data-flow-overview.md) supports generic [Delta format](format-delta.md) on Azure Storage as source and sink to read and write Delta files for code-free ETL, and runs on managed Azure Integration Runtime.
In general, Azure Data Factory supports Delta Lake with the following capabiliti
To use this Azure Databricks Delta Lake connector, you need to set up a cluster in Azure Databricks. -- To copy data to delta lake, Copy activity invokes Azure Databricks cluster to read data from an Azure Storage, which is either your original source or a staging area to where Data Factory firstly writes the source data via built-in staged copy. Learn more from [Delta lake as the sink](#delta-lake-as-sink).-- Similarly, to copy data from delta lake, Copy activity invokes Azure Databricks cluster to write data to an Azure Storage, which is either your original sink or a staging area from where Data Factory continues to write data to final sink via built-in staged copy. Learn more from [Delta lake as the source](#delta-lake-as-source).
+- To copy data to delta lake, Copy activity invokes Azure Databricks cluster to read data from an Azure Storage, which is either your original source or a staging area to where the service firstly writes the source data via built-in staged copy. Learn more from [Delta lake as the sink](#delta-lake-as-sink).
+- Similarly, to copy data from delta lake, Copy activity invokes Azure Databricks cluster to write data to an Azure Storage, which is either your original sink or a staging area from where the service continues to write data to final sink via built-in staged copy. Learn more from [Delta lake as the source](#delta-lake-as-source).
The Databricks cluster needs to have access to Azure Blob or Azure Data Lake Storage Gen2 account, both the storage container/file system used for source/sink/staging and the container/file system where you want to write the Delta Lake tables.
The Databricks cluster needs to have access to Azure Blob or Azure Data Lake Sto
- To use **Azure Blob storage**, you can configure a **storage account access key** or **SAS token** on the Databricks cluster as part of the Apache Spark configuration. Follow the steps in [Access Azure Blob storage using the RDD API](/azure/databricks/data/data-sources/azure/azure-storage#access-azure-blob-storage-using-the-rdd-api).
-During copy activity execution, if the cluster you configured has been terminated, Data Factory automatically starts it. If you author pipeline using Data Factory authoring UI, for operations like data preview, you need to have a live cluster, Data Factory won't start the cluster on your behalf.
+During copy activity execution, if the cluster you configured has been terminated, the service automatically starts it. If you author pipeline using authoring UI, for operations like data preview, you need to have a live cluster, the service won't start the cluster on your behalf.
#### Specify the cluster configuration
For cluster configuration details, see [Configure clusters](/azure/databricks/cl
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide details about properties that define Data Factory entities specific to an Azure Databricks Delta Lake connector.
+The following sections provide details about properties that define entities specific to an Azure Databricks Delta Lake connector.
## Linked service properties
The following properties are supported for an Azure Databricks Delta Lake linked
| type | The type property must be set to **AzureDatabricksDeltaLake**. | Yes | | domain | Specify the Azure Databricks workspace URL, e.g. `https://adb-xxxxxxxxx.xx.azuredatabricks.net`. | | | clusterId | Specify the cluster ID of an existing cluster. It should be an already created Interactive Cluster. <br>You can find the Cluster ID of an Interactive Cluster on Databricks workspace -> Clusters -> Interactive Cluster Name -> Configuration -> Tags. [Learn more](/azure/databricks/clusters/configure#cluster-tags). | |
-| accessToken | Access token is required for Data Factory to authenticate to Azure Databricks. Access token needs to be generated from the databricks workspace. More detailed steps to find the access token can be found [here](/azure/databricks/dev-tools/api/latest/authentication#generate-token). | |
+| accessToken | Access token is required for the service to authenticate to Azure Databricks. Access token needs to be generated from the databricks workspace. More detailed steps to find the access token can be found [here](/azure/databricks/dev-tools/api/latest/authentication#generate-token). | |
| connectVia | The [integration runtime](concepts-integration-runtime.md) that is used to connect to the data store. You can use the Azure integration runtime or a self-hosted integration runtime (if your data store is located in a private network). If not specified, it uses the default Azure integration runtime. | No | **Example:**
To copy data from Azure Databricks Delta Lake, the following properties are supp
#### Direct copy from delta lake
-If your sink data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from Azure Databricks Delta table to sink. Data Factory checks the settings and fails the Copy activity run if the following criteria is not met:
+If your sink data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from Azure Databricks Delta table to sink. The service checks the settings and fails the Copy activity run if the following criteria is not met:
- The **sink linked service** is [Azure Blob storage](connector-azure-blob-storage.md) or [Azure Data Lake Storage Gen2](connector-azure-data-lake-storage.md). The account credential should be pre-configured in Azure Databricks cluster configuration, learn more from [Prerequisites](#prerequisites).
If your sink data store and format meet the criteria described in this section,
#### Staged copy from delta lake
-When your sink data store or format does not match the direct copy criteria, as mentioned in the last section, enable the built-in staged copy using an interim Azure storage instance. The staged copy feature also provides you better throughput. Data Factory exports data from Azure Databricks Delta Lake into staging storage, then copies the data to sink, and finally cleans up your temporary data from the staging storage. See [Staged copy](copy-activity-performance-features.md#staged-copy) for details about copying data by using staging.
+When your sink data store or format does not match the direct copy criteria, as mentioned in the last section, enable the built-in staged copy using an interim Azure storage instance. The staged copy feature also provides you better throughput. The service exports data from Azure Databricks Delta Lake into staging storage, then copies the data to sink, and finally cleans up your temporary data from the staging storage. See [Staged copy](copy-activity-performance-features.md#staged-copy) for details about copying data by using staging.
To use this feature, create an [Azure Blob storage linked service](connector-azure-blob-storage.md#linked-service-properties) or [Azure Data Lake Storage Gen2 linked service](connector-azure-data-lake-storage.md#linked-service-properties) that refers to the storage account as the interim staging. Then specify the `enableStaging` and `stagingSettings` properties in the Copy activity.
To copy data to Azure Databricks Delta Lake, the following properties are suppor
#### Direct copy to delta lake
-If your source data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from source to Azure Databricks Delta Lake. Azure Data Factory checks the settings and fails the Copy activity run if the following criteria is not met:
+If your source data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from source to Azure Databricks Delta Lake. The service checks the settings and fails the Copy activity run if the following criteria is not met:
- The **source linked service** is [Azure Blob storage](connector-azure-blob-storage.md) or [Azure Data Lake Storage Gen2](connector-azure-data-lake-storage.md). The account credential should be pre-configured in Azure Databricks cluster configuration, learn more from [Prerequisites](#prerequisites).
If your source data store and format meet the criteria described in this section
#### Staged copy to delta lake
-When your source data store or format does not match the direct copy criteria, as mentioned in the last section, enable the built-in staged copy using an interim Azure storage instance. The staged copy feature also provides you better throughput. Data Factory automatically converts the data to meet the data format requirements into staging storage, then load data into delta lake from there. Finally, it cleans up your temporary data from the storage. See [Staged copy](copy-activity-performance-features.md#staged-copy) for details about copying data using staging.
+When your source data store or format does not match the direct copy criteria, as mentioned in the last section, enable the built-in staged copy using an interim Azure storage instance. The staged copy feature also provides you better throughput. The service automatically converts the data to meet the data format requirements into staging storage, then load data into delta lake from there. Finally, it cleans up your temporary data from the storage. See [Staged copy](copy-activity-performance-features.md#staged-copy) for details about copying data using staging.
To use this feature, create an [Azure Blob storage linked service](connector-azure-blob-storage.md#linked-service-properties) or [Azure Data Lake Storage Gen2 linked service](connector-azure-data-lake-storage.md#linked-service-properties) that refers to the storage account as the interim staging. Then specify the `enableStaging` and `stagingSettings` properties in the Copy activity.
To use this feature, create an [Azure Blob storage linked service](connector-azu
## Monitoring
-Azure Data Factory provides the same [copy activity monitoring experience](copy-activity-monitoring.md) as other connectors. In addition, because loading data from/to delta lake is running on your Azure Databricks cluster, you can further [view detailed cluster logs](/azure/databricks/clusters/clusters-manage#--view-cluster-logs) and [monitor performance](/azure/databricks/clusters/clusters-manage#--monitor-performance).
+The same [copy activity monitoring experience](copy-activity-monitoring.md) is provided as for other connectors. In addition, because loading data from/to delta lake is running on your Azure Databricks cluster, you can further [view detailed cluster logs](/azure/databricks/clusters/clusters-manage#--view-cluster-logs) and [monitor performance](/azure/databricks/clusters/clusters-manage#--monitor-performance).
## Lookup activity properties
For more information about the properties, see [Lookup activity](control-flow-lo
## Next steps
-For a list of data stores supported as sources and sinks by Copy activity in Data Factory, see [supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of data stores supported as sources and sinks by Copy activity, see [supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Azure Sql Data Warehouse https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-azure-sql-data-warehouse.md
Best practices to load data with partition option:
2. If the table has built-in partition, use partition option "Physical partitions of table" to get better performance. 3. If you use Azure Integration Runtime to copy data, you can set larger "[Data Integration Units (DIU)](copy-activity-performance-features.md#data-integration-units)" (>4) to utilize more computing resource. Check the applicable scenarios there. 4. "[Degree of copy parallelism](copy-activity-performance-features.md#parallel-copy)" control the partition numbers, setting this number too large sometime hurts the performance, recommend setting this number as (DIU or number of Self-hosted IR nodes) * (2 to 4).
-5. Note Azure Synapse Analytics can execute a maximum of 32 queries at a moment, setting "Degree of copy parallelism" too large may cause an Azure Synapse throttling issue.
+5. Note Azure Synapse Analytics can execute a maximum of 32 queries at a moment, setting "Degree of copy parallelism" too large may cause a Synapse throttling issue.
**Example: full load from large table with physical partitions**
data-factory Connector Dynamics Crm Office 365 https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-dynamics-crm-office-365.md
Title: Copy data in Dynamics (Microsoft Dataverse)
-description: Learn how to copy data from Microsoft Dynamics CRM or Microsoft Dynamics 365 (Microsoft Dataverse) to supported sink data stores or from supported source data stores to Dynamics CRM or Dynamics 365 by using a copy activity in a data factory pipeline.
+description: Learn how to copy data from Microsoft Dynamics CRM or Microsoft Dynamics 365 (Microsoft Dataverse) to supported sink data stores or from supported source data stores to Dynamics CRM or Dynamics 365 by using a copy activity in an Azure Data Factory or Azure Synapse Analytics pipeline.
Last updated 03/17/2021
-# Copy data from and to Dynamics 365 (Microsoft Dataverse) or Dynamics CRM by using Azure Data Factory
+# Copy data from and to Dynamics 365 (Microsoft Dataverse) or Dynamics CRM
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to use a copy activity in Azure Data Factory to copy data from and to Microsoft Dynamics 365 and Microsoft Dynamics CRM. It builds on the [copy activity overview](copy-activity-overview.md) article that presents a general overview of a copy activity.
+This article outlines how to use a copy activity in Azure Data Factory or Synapse pipelines to copy data from and to Microsoft Dynamics 365 and Microsoft Dynamics CRM. It builds on the [copy activity overview](copy-activity-overview.md) article that presents a general overview of a copy activity.
## Supported capabilities
Refer to the following table of supported authentication types and configuration
| Dynamics 365 on-premises with internet-facing deployment (IFD) <br/><br/> Dynamics CRM 2016 on-premises with IFD <br/><br/> Dynamics CRM 2015 on-premises with IFD | IFD | [Dynamics on-premises with IFD and IFD authentication](#dynamics-365-and-dynamics-crm-on-premises-with-ifd) | >[!NOTE]
->With the [deprecation of regional Discovery Service](/power-platform/important-changes-coming#regional-discovery-service-is-deprecated), Azure Data Factory has upgraded to leverage [global Discovery Service](/powerapps/developer/data-platform/webapi/discover-url-organization-web-api#global-discovery-service) while using Office 365 Authentication.
+>With the [deprecation of regional Discovery Service](/power-platform/important-changes-coming#regional-discovery-service-is-deprecated), the service has upgraded to leverage [global Discovery Service](/powerapps/developer/data-platform/webapi/discover-url-organization-web-api#global-discovery-service) while using Office 365 Authentication.
> [!IMPORTANT] >If your tenant and user is configured in Azure Active Directory for [conditional access](../active-directory/conditional-access/overview.md) and/or Multi-Factor Authentication is required, you will not be able to use Office 365 Authentication type. For those situations, you must use a Azure Active Directory (Azure AD) service principal authentication.
To use this connector with Azure AD service-principal authentication, you must s
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide details about properties that are used to define Data Factory entities specific to Dynamics.
+The following sections provide details about properties that are used to define entities specific to Dynamics.
## Linked service properties
The following properties are supported for the Dynamics linked service.
| authenticationType | The authentication type to connect to a Dynamics server. Valid values are "AADServicePrincipal" and "Office365". | Yes | | servicePrincipalId | The client ID of the Azure AD application. | Yes when authentication is "AADServicePrincipal" | | servicePrincipalCredentialType | The credential type to use for service-principal authentication. Valid values are "ServicePrincipalKey" and "ServicePrincipalCert". | Yes when authentication is "AADServicePrincipal" |
-| servicePrincipalCredential | The service-principal credential. <br/><br/>When you use "ServicePrincipalKey" as the credential type, `servicePrincipalCredential` can be a string that Azure Data Factory encrypts upon linked service deployment. Or it can be a reference to a secret in Azure Key Vault. <br/><br/>When you use "ServicePrincipalCert" as the credential, `servicePrincipalCredential` must be a reference to a certificate in Azure Key Vault. | Yes when authentication is "AADServicePrincipal" |
+| servicePrincipalCredential | The service-principal credential. <br/><br/>When you use "ServicePrincipalKey" as the credential type, `servicePrincipalCredential` can be a string that the service encrypts upon linked service deployment. Or it can be a reference to a secret in Azure Key Vault. <br/><br/>When you use "ServicePrincipalCert" as the credential, `servicePrincipalCredential` must be a reference to a certificate in Azure Key Vault. | Yes when authentication is "AADServicePrincipal" |
| username | The username to connect to Dynamics. | Yes when authentication is "Office365" |
-| password | The password for the user account you specified as the username. Mark this field with "SecureString" to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes when authentication is "Office365" |
+| password | The password for the user account you specified as the username. Mark this field with "SecureString" to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes when authentication is "Office365" |
| connectVia | The [integration runtime](concepts-integration-runtime.md) to be used to connect to the data store. If no value is specified, the property uses the default Azure integration runtime. | No | >[!NOTE]
Additional properties that compare to Dynamics online are **hostName** and **por
| organizationName | The organization name of the Dynamics instance. | Yes. | | authenticationType | The authentication type to connect to the Dynamics server. Specify "Ifd" for Dynamics on-premises with IFD. | Yes. | | username | The username to connect to Dynamics. | Yes. |
-| password | The password for the user account you specified for the username. You can mark this field with "SecureString" to store it securely in Data Factory. Or you can store a password in Key Vault and let the copy activity pull from there when it does data copy. Learn more from [Store credentials in Key Vault](store-credentials-in-key-vault.md). | Yes. |
+| password | The password for the user account you specified for the username. You can mark this field with "SecureString" to store it securely. Or you can store a password in Key Vault and let the copy activity pull from there when it does data copy. Learn more from [Store credentials in Key Vault](store-credentials-in-key-vault.md). | Yes. |
| connectVia | The [integration runtime](concepts-integration-runtime.md) to be used to connect to the data store. If no value is specified, the property uses the default Azure integration runtime. | No | #### Example: Dynamics on-premises with IFD using IFD authentication
To copy data from Dynamics, the copy activity **source** section supports the fo
> [!IMPORTANT] >- When you copy data from Dynamics, explicit column mapping from Dynamics to sink is optional. But we highly recommend the mapping to ensure a deterministic copy result.
->- When Data Factory imports a schema in the authoring UI, it infers the schema. It does so by sampling the top rows from the Dynamics query result to initialize the source column list. In that case, columns with no values in the top rows are omitted. The same behavior applies to copy executions if there is no explicit mapping. You can review and add more columns into the mapping, which are honored during copy runtime.
+>- When the service imports a schema in the authoring UI, it infers the schema. It does so by sampling the top rows from the Dynamics query result to initialize the source column list. In that case, columns with no values in the top rows are omitted. The same behavior applies to copy executions if there is no explicit mapping. You can review and add more columns into the mapping, which are honored during copy runtime.
#### Example
You can also add filters to filter the views. For example, add the following fil
## Data type mapping for Dynamics
-When you copy data from Dynamics, the following table shows mappings from Dynamics data types to Data Factory interim data types. To learn how a copy activity maps to a source schema and a data type maps to a sink, see [Schema and data type mappings](copy-activity-schema-and-type-mapping.md).
+When you copy data from Dynamics, the following table shows mappings from Dynamics data types to interim data types within the service. To learn how a copy activity maps to a source schema and a data type maps to a sink, see [Schema and data type mappings](copy-activity-schema-and-type-mapping.md).
-Configure the corresponding Data Factory data type in a dataset structure that is based on your source Dynamics data type by using the following mapping table:
+Configure the corresponding interim data type in a dataset structure that is based on your source Dynamics data type by using the following mapping table:
-| Dynamics data type | Data Factory interim data type | Supported as source | Supported as sink |
+| Dynamics data type | Service interim data type | Supported as source | Supported as sink |
|: |: |: |: | | AttributeTypeCode.BigInt | Long | Γ£ô | Γ£ô | | AttributeTypeCode.Boolean | Boolean | Γ£ô | Γ£ô |
To learn details about the properties, see [Lookup activity](control-flow-lookup
## Next steps
-For a list of data stores the copy activity in Data Factory supports as sources and sinks, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of supported data stores the copy activity as sources and sinks, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Http https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-http.md
Title: Copy data from an HTTP source by using Azure Data Factory
+ Title: Copy data from an HTTP source
-description: Learn how to copy data from a cloud or on-premises HTTP source to supported sink data stores by using a copy activity in an Azure Data Factory pipeline.
+description: Learn how to copy data from a cloud or on-premises HTTP source to supported sink data stores by using a copy activity in an Azure Data Factory or Azure Synapse Analytics pipeline.
Last updated 03/17/2021
-# Copy data from an HTTP endpoint by using Azure Data Factory
+# Copy data from an HTTP endpoint by using Azure Data Factory or Azure Synapse Analytics
> [!div class="op_single_selector" title1="Select the version of Data Factory service you are using:"] > * [Version 1](v1/data-factory-http-connector.md)
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to use Copy Activity in Azure Data Factory to copy data from an HTTP endpoint. The article builds on [Copy Activity in Azure Data Factory](copy-activity-overview.md), which presents a general overview of Copy Activity.
+This article outlines how to use Copy Activity in Azure Data Factory and Azure Synapse to copy data from an HTTP endpoint. The article builds on [Copy Activity](copy-activity-overview.md), which presents a general overview of Copy Activity.
The difference among this HTTP connector, the [REST connector](connector-rest.md) and the [Web table connector](connector-web-table.md) are:
You can use this HTTP connector to:
- Copy the HTTP response as-is or parse it by using [supported file formats and compression codecs](supported-file-formats-and-compression-codecs.md). > [!TIP]
-> To test an HTTP request for data retrieval before you configure the HTTP connector in Data Factory, learn about the API specification for header and body requirements. You can use tools like Postman or a web browser to validate.
+> To test an HTTP request for data retrieval before you configure the HTTP connector, learn about the API specification for header and body requirements. You can use tools like Postman or a web browser to validate.
## Prerequisites
You can use this HTTP connector to:
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide details about properties you can use to define Data Factory entities that are specific to the HTTP connector.
+The following sections provide details about properties you can use to define entities that are specific to the HTTP connector.
## Linked service properties
Set the **authenticationType** property to **Basic**, **Digest**, or **Windows**
| Property | Description | Required | |: |: |: | | userName | The user name to use to access the HTTP endpoint. | Yes |
-| password | The password for the user (the **userName** value). Mark this field as a **SecureString** type to store it securely in Data Factory. You can also [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
+| password | The password for the user (the **userName** value). Mark this field as a **SecureString** type to store it securely. You can also [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
**Example**
To use ClientCertificate authentication, set the **authenticationType** property
|: |: |: | | embeddedCertData | Base64-encoded certificate data. | Specify either **embeddedCertData** or **certThumbprint**. | | certThumbprint | The thumbprint of the certificate that's installed on your self-hosted Integration Runtime machine's cert store. Applies only when the self-hosted type of Integration Runtime is specified in the **connectVia** property. | Specify either **embeddedCertData** or **certThumbprint**. |
-| password | The password that's associated with the certificate. Mark this field as a **SecureString** type to store it securely in Data Factory. You can also [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | No |
+| password | The password that's associated with the certificate. Mark this field as a **SecureString** type to store it securely. You can also [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | No |
If you use **certThumbprint** for authentication and the certificate is installed in the personal store of the local computer, grant read permissions to the self-hosted Integration Runtime:
To learn details about the properties, check [Lookup activity](control-flow-look
## Legacy models >[!NOTE]
->The following models are still supported as-is for backward compatibility. You are suggested to use the new model mentioned in above sections going forward, and the ADF authoring UI has switched to generating the new model.
+>The following models are still supported as-is for backward compatibility. You are suggested to use the new model mentioned in above sections going forward, and the authoring UI has switched to generating the new model.
### Legacy dataset model
To learn details about the properties, check [Lookup activity](control-flow-look
## Next steps
-For a list of data stores that Copy Activity supports as sources and sinks in Azure Data Factory, see [Supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of data stores that Copy Activity supports as sources and sinks, see [Supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Oracle https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-oracle.md
Title: Copy data to and from Oracle by using Azure Data Factory
+ Title: Copy data to and from Oracle
-description: Learn how to copy data from supported source stores to an Oracle database, or from Oracle to supported sink stores, by using Data Factory.
+description: Learn how to copy data from supported source stores to an Oracle database, or from Oracle to supported sink stores, using Data Factory or Azure Synapse Analytics pipelines.
Last updated 03/17/2021
-# Copy data from and to Oracle by using Azure Data Factory
+# Copy data from and to Oracle by using Azure Data Factory or Azure Synapse Analytics
> [!div class="op_single_selector" title1="Select the version of Data Factory service you are using:"] > * [Version 1](v1/data-factory-onprem-oracle-connector.md)
The integration runtime provides a built-in Oracle driver. Therefore, you don't
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide details about properties that are used to define Data Factory entities specific to the Oracle connector.
+The following sections provide details about properties that are used to define entities specific to the Oracle connector.
## Linked service properties
The Oracle linked service supports the following properties:
| Property | Description | Required | |: |: |: | | type | The type property must be set to **Oracle**. | Yes |
-| connectionString | Specifies the information needed to connect to the Oracle Database instance. <br/>You can also put a password in Azure Key Vault, and pull the `password` configuration out of the connection string. Refer to the following samples and [Store credentials in Azure Key Vault](store-credentials-in-key-vault.md) with more details. <br><br>**Supported connection type**: You can use **Oracle SID** or **Oracle Service Name** to identify your database:<br>- If you use SID: `Host=<host>;Port=<port>;Sid=<sid>;User Id=<username>;Password=<password>;`<br>- If you use Service Name: `Host=<host>;Port=<port>;ServiceName=<servicename>;User Id=<username>;Password=<password>;`<br>For advanced Oracle native connection options, you can choose to add an entry in [TNSNAMES.ORA](http://www.orafaq.com/wiki/Tnsnames.ora) file on the Oracle server, and in ADF Oracle linked service, choose to use Oracle Service Name connection type and configure the corresponding service name. | Yes |
+| connectionString | Specifies the information needed to connect to the Oracle Database instance. <br/>You can also put a password in Azure Key Vault, and pull the `password` configuration out of the connection string. Refer to the following samples and [Store credentials in Azure Key Vault](store-credentials-in-key-vault.md) with more details. <br><br>**Supported connection type**: You can use **Oracle SID** or **Oracle Service Name** to identify your database:<br>- If you use SID: `Host=<host>;Port=<port>;Sid=<sid>;User Id=<username>;Password=<password>;`<br>- If you use Service Name: `Host=<host>;Port=<port>;ServiceName=<servicename>;User Id=<username>;Password=<password>;`<br>For advanced Oracle native connection options, you can choose to add an entry in [TNSNAMES.ORA](http://www.orafaq.com/wiki/Tnsnames.ora) file on the Oracle server, and in Oracle linked service, choose to use Oracle Service Name connection type and configure the corresponding service name. | Yes |
| connectVia | The [integration runtime](concepts-integration-runtime.md) to be used to connect to the data store. Learn more from [Prerequisites](#prerequisites) section. If not specified, the default Azure Integration Runtime is used. |No | >[!TIP]
To enable encryption on Oracle connection, you have two options:
``` 3. Place the `truststore` file on the self-hosted IR machine. For example, place the file at C:\MyTrustStoreFile.
- 4. In Azure Data Factory, configure the Oracle connection string with `EncryptionMethod=1` and the corresponding `TrustStore`/`TrustStorePassword`value. For example, `Host=<host>;Port=<port>;Sid=<sid>;User Id=<username>;Password=<password>;EncryptionMethod=1;TrustStore=C:\\MyTrustStoreFile;TrustStorePassword=<trust_store_password>`.
+ 4. In the service, configure the Oracle connection string with `EncryptionMethod=1` and the corresponding `TrustStore`/`TrustStorePassword`value. For example, `Host=<host>;Port=<port>;Sid=<sid>;User Id=<username>;Password=<password>;EncryptionMethod=1;TrustStore=C:\\MyTrustStoreFile;TrustStorePassword=<trust_store_password>`.
**Example:**
To copy data to Oracle, set the sink type in the copy activity to `OracleSink`.
## Parallel copy from Oracle
-The Data Factory Oracle connector provides built-in data partitioning to copy data from Oracle in parallel. You can find data partitioning options on the **Source** tab of the copy activity.
+The Oracle connector provides built-in data partitioning to copy data from Oracle in parallel. You can find data partitioning options on the **Source** tab of the copy activity.
![Screenshot of partition options](./media/connector-oracle/connector-oracle-partition-options.png)
-When you enable partitioned copy, Data Factory runs parallel queries against your Oracle source to load data by partitions. The parallel degree is controlled by the [`parallelCopies`](copy-activity-performance-features.md#parallel-copy) setting on the copy activity. For example, if you set `parallelCopies` to four, Data Factory concurrently generates and runs four queries based on your specified partition option and settings, and each query retrieves a portion of data from your Oracle database.
+When you enable partitioned copy, the service runs parallel queries against your Oracle source to load data by partitions. The parallel degree is controlled by the [`parallelCopies`](copy-activity-performance-features.md#parallel-copy) setting on the copy activity. For example, if you set `parallelCopies` to four, the service concurrently generates and runs four queries based on your specified partition option and settings, and each query retrieves a portion of data from your Oracle database.
You are suggested to enable parallel copy with data partitioning especially when you load large amount of data from your Oracle database. The following are suggested configurations for different scenarios. When copying data into file-based data store, it's recommanded to write to a folder as multiple files (only specify folder name), in which case the performance is better than writing to a single file. | Scenario | Suggested settings | | | |
-| Full load from large table, with physical partitions. | **Partition option**: Physical partitions of table. <br><br/>During execution, Data Factory automatically detects the physical partitions, and copies data by partitions. |
+| Full load from large table, with physical partitions. | **Partition option**: Physical partitions of table. <br><br/>During execution, the service automatically detects the physical partitions, and copies data by partitions. |
| Full load from large table, without physical partitions, while with an integer column for data partitioning. | **Partition options**: Dynamic range partition.<br>**Partition column**: Specify the column used to partition data. If not specified, the primary key column is used. |
-| Load a large amount of data by using a custom query, with physical partitions. | **Partition option**: Physical partitions of table.<br>**Query**: `SELECT * FROM <TABLENAME> PARTITION("?AdfTabularPartitionName") WHERE <your_additional_where_clause>`.<br>**Partition name**: Specify the partition name(s) to copy data from. If not specified, Data Factory automatically detects the physical partitions on the table you specified in the Oracle dataset.<br><br>During execution, Data Factory replaces `?AdfTabularPartitionName` with the actual partition name, and sends to Oracle. |
-| Load a large amount of data by using a custom query, without physical partitions, while with an integer column for data partitioning. | **Partition options**: Dynamic range partition.<br>**Query**: `SELECT * FROM <TABLENAME> WHERE ?AdfRangePartitionColumnName <= ?AdfRangePartitionUpbound AND ?AdfRangePartitionColumnName >= ?AdfRangePartitionLowbound AND <your_additional_where_clause>`.<br>**Partition column**: Specify the column used to partition data. You can partition against the column with integer data type.<br>**Partition upper bound** and **partition lower bound**: Specify if you want to filter against partition column to retrieve data only between the lower and upper range.<br><br>During execution, Data Factory replaces `?AdfRangePartitionColumnName`, `?AdfRangePartitionUpbound`, and `?AdfRangePartitionLowbound` with the actual column name and value ranges for each partition, and sends to Oracle. <br>For example, if your partition column "ID" is set with the lower bound as 1 and the upper bound as 80, with parallel copy set as 4, Data Factory retrieves data by 4 partitions. Their IDs are between [1,20], [21, 40], [41, 60], and [61, 80], respectively. |
+| Load a large amount of data by using a custom query, with physical partitions. | **Partition option**: Physical partitions of table.<br>**Query**: `SELECT * FROM <TABLENAME> PARTITION("?AdfTabularPartitionName") WHERE <your_additional_where_clause>`.<br>**Partition name**: Specify the partition name(s) to copy data from. If not specified, the service automatically detects the physical partitions on the table you specified in the Oracle dataset.<br><br>During execution, the service replaces `?AdfTabularPartitionName` with the actual partition name, and sends to Oracle. |
+| Load a large amount of data by using a custom query, without physical partitions, while with an integer column for data partitioning. | **Partition options**: Dynamic range partition.<br>**Query**: `SELECT * FROM <TABLENAME> WHERE ?AdfRangePartitionColumnName <= ?AdfRangePartitionUpbound AND ?AdfRangePartitionColumnName >= ?AdfRangePartitionLowbound AND <your_additional_where_clause>`.<br>**Partition column**: Specify the column used to partition data. You can partition against the column with integer data type.<br>**Partition upper bound** and **partition lower bound**: Specify if you want to filter against partition column to retrieve data only between the lower and upper range.<br><br>During execution, the service replaces `?AdfRangePartitionColumnName`, `?AdfRangePartitionUpbound`, and `?AdfRangePartitionLowbound` with the actual column name and value ranges for each partition, and sends to Oracle. <br>For example, if your partition column "ID" is set with the lower bound as 1 and the upper bound as 80, with parallel copy set as 4, the service retrieves data by 4 partitions. Their IDs are between [1,20], [21, 40], [41, 60], and [61, 80], respectively. |
> [!TIP] > When copying data from a non-partitioned table, you can use "Dynamic range" partition option to partition against an integer column. If your source data doesn't have such type of column, you can leverage [ORA_HASH]( https://docs.oracle.com/database/121/SQLRF/functions136.htm) function in source query to generate a column and use it as partition column.
You are suggested to enable parallel copy with data partitioning especially when
## Data type mapping for Oracle
-When you copy data from and to Oracle, the following mappings apply. To learn about how the copy activity maps the source schema and data type to the sink, see [Schema and data type mappings](copy-activity-schema-and-type-mapping.md).
+When you copy data from and to Oracle, the following interim data type mappings are used within the service. To learn about how the copy activity maps the source schema and data type to the sink, see [Schema and data type mappings](copy-activity-schema-and-type-mapping.md).
-| Oracle data type | Data Factory interim data type |
+| Oracle data type | Interim data type |
|: |: | | BFILE |Byte[] | | BLOB |Byte[]<br/>(only supported on Oracle 10g and higher) |
When you copy data from and to Oracle, the following mappings apply. To learn ab
To learn details about the properties, check [Lookup activity](control-flow-lookup-activity.md). ## Next steps
-For a list of data stores supported as sources and sinks by the copy activity in Data Factory, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of data stores supported as sources and sinks by the copy activity, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Salesforce https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-salesforce.md
Title: Copy data from and to Salesforce
-description: Learn how to copy data from Salesforce to supported sink data stores or from supported source data stores to Salesforce by using a copy activity in a data factory pipeline.
+description: Learn how to copy data from Salesforce to supported sink data stores or from supported source data stores to Salesforce by using a copy activity in an Azure Data Factory or Azure Synapse Analytics pipeline.
Last updated 03/17/2021
-# Copy data from and to Salesforce by using Azure Data Factory
+# Copy data from and to Salesforce using Azure Data Factory or Azure Synapse Analytics
> [!div class="op_single_selector" title1="Select the version of Data Factory service you are using:"] > * [Version 1](v1/data-factory-salesforce-connector.md)
Last updated 03/17/2021
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to use Copy Activity in Azure Data Factory to copy data from and to Salesforce. It builds on the [Copy Activity overview](copy-activity-overview.md) article that presents a general overview of the copy activity.
+This article outlines how to use Copy Activity in Azure Data Factory and Azure Synapse pipelines to copy data from and to Salesforce. It builds on the [Copy Activity overview](copy-activity-overview.md) article that presents a general overview of the copy activity.
## Supported capabilities
You might also receive the "REQUEST_LIMIT_EXCEEDED" error message in both scenar
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide details about properties that are used to define Data Factory entities specific to the Salesforce connector.
+The following sections provide details about properties that are used to define entities specific to the Salesforce connector.
## Linked service properties
The following properties are supported for the Salesforce linked service.
| type |The type property must be set to **Salesforce**. |Yes | | environmentUrl | Specify the URL of the Salesforce instance. <br> - Default is `"https://login.salesforce.com"`. <br> - To copy data from sandbox, specify `"https://test.salesforce.com"`. <br> - To copy data from custom domain, specify, for example, `"https://[domain].my.salesforce.com"`. |No | | username |Specify a user name for the user account. |Yes |
-| password |Specify a password for the user account.<br/><br/>Mark this field as a SecureString to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). |Yes |
-| securityToken |Specify a security token for the user account. <br/><br/>To learn about security tokens in general, see [Security and the API](https://developer.salesforce.com/docs/atlas.en-us.api.met). |No |
+| password |Specify a password for the user account.<br/><br/>Mark this field as a SecureString to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). |Yes |
+| securityToken |Specify a security token for the user account. <br/><br/>To learn about security tokens in general, see [Security and the API](https://developer.salesforce.com/docs/atlas.en-us.api.met). |No |
| apiVersion | Specify the Salesforce REST/Bulk API version to use, e.g. `48.0`. By default, the connector uses [v45](https://developer.salesforce.com/docs/atlas.en-us.218.0.api_rest.meta/api_rest/dome_versions.htm) to copy data from Salesforce, and uses [v40](https://developer.salesforce.com/docs/atlas.en-us.208.0.api_asynch.meta/api_asynch/asynch_api_intro.htm) to copy data to Salesforce. | No | | connectVia | The [integration runtime](concepts-integration-runtime.md) to be used to connect to the data store. If not specified, it uses the default Azure Integration Runtime. | No |
-**Example: Store credentials in Data Factory**
+**Example: Store credentials**
```json {
To copy data from and to Salesforce, set the type property of the dataset to **S
> [!IMPORTANT] > The "__c" part of **API Name** is needed for any custom object.
-![Data Factory Salesforce connection API Name](media/copy-data-from-salesforce/data-factory-salesforce-api-name.png)
+![Salesforce connection API Name](media/copy-data-from-salesforce/data-factory-salesforce-api-name.png)
**Example:**
To copy data from Salesforce, set the source type in the copy activity to **Sale
> [!IMPORTANT] > The "__c" part of **API Name** is needed for any custom object.
-![Data Factory Salesforce connection API Name list](media/copy-data-from-salesforce/data-factory-salesforce-api-name-2.png)
+![Salesforce connection API Name list](media/copy-data-from-salesforce/data-factory-salesforce-api-name-2.png)
**Example:**
If you hit error of "MALFORMED_QUERY: Truncated", normally it's due to you have
## Data type mapping for Salesforce
-When you copy data from Salesforce, the following mappings are used from Salesforce data types to Data Factory interim data types. To learn about how the copy activity maps the source schema and data type to the sink, see [Schema and data type mappings](copy-activity-schema-and-type-mapping.md).
+When you copy data from Salesforce, the following mappings are used from Salesforce data types to interim data types within the service internally. To learn about how the copy activity maps the source schema and data type to the sink, see [Schema and data type mappings](copy-activity-schema-and-type-mapping.md).
-| Salesforce data type | Data Factory interim data type |
+| Salesforce data type | Service interim data type |
|: |: | | Auto Number |String | | Checkbox |Boolean |
To learn details about the properties, check [Lookup activity](control-flow-look
## Next steps
-For a list of data stores supported as sources and sinks by the copy activity in Data Factory, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of data stores supported as sources and sinks by the copy activity, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Sap Table https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-sap-table.md
Title: Copy data from an SAP table
-description: Learn how to copy data from an SAP table to supported sink data stores by using a copy activity in an Azure Data Factory pipeline.
+description: Learn how to copy data from an SAP table to supported sink data stores by using a copy activity in an Azure Data Factory or Azure Synapse Analytics pipeline.
Last updated 07/30/2021
-# Copy data from an SAP table by using Azure Data Factory
+# Copy data from an SAP table using Azure Data Factory or Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to use the copy activity in Azure Data Factory to copy data from an SAP table. For more information, see [Copy activity overview](copy-activity-overview.md).
+This article outlines how to use the copy activity in Azure Data Factory and Azure Synapse Analytics pipelines to copy data from an SAP table. For more information, see [Copy activity overview](copy-activity-overview.md).
>[!TIP]
->To learn ADF's overall support on SAP data integration scenario, see [SAP data integration using Azure Data Factory whitepaper](https://github.com/Azure/Azure-DataFactory/blob/master/whitepaper/SAP%20Data%20Integration%20using%20Azure%20Data%20Factory.pdf) with detailed introduction on each SAP connector, comparsion and guidance.
+>To learn the overall support on SAP data integration scenario, see [SAP data integration using Azure Data Factory whitepaper](https://github.com/Azure/Azure-DataFactory/blob/master/whitepaper/SAP%20Data%20Integration%20using%20Azure%20Data%20Factory.pdf) with detailed introduction on each SAP connector, comparsion and guidance.
## Supported capabilities
To use this SAP table connector, you need to:
![Install SAP Connector for .NET](./media/connector-sap-business-warehouse-open-hub/install-sap-dotnet-connector.png) -- The SAP user who's being used in the Data Factory SAP table connector must have the following permissions:
+- The SAP user who's being used in the SAP table connector must have the following permissions:
- Authorization for using Remote Function Call (RFC) destinations. - Permissions to the Execute activity of the S_SDSAUTH authorization object. You can refer to SAP Note 460089 on the majority authorization objects. Certain RFCs are required by the underlying NCo connector, for example RFC_FUNCTION_SEARCH.
To use this SAP table connector, you need to:
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide details about properties that are used to define the Data Factory entities specific to the SAP table connector.
+The following sections provide details about properties that are used to define the entities specific to the SAP table connector.
## Linked service properties
The following properties are supported for the SAP BW Open Hub linked service:
| `clientId` | The ID of the client in the SAP system.<br/>Allowed value: A three-digit decimal number represented as a string. | Yes | | `language` | The language that the SAP system uses.<br/>Default value is `EN`.| No | | `userName` | The name of the user who has access to the SAP server. | Yes |
-| `password` | The password for the user. Mark this field with the `SecureString` type to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
+| `password` | The password for the user. Mark this field with the `SecureString` type to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
| `sncMode` | The SNC activation indicator to access the SAP server where the table is located.<br/>Use if you want to use SNC to connect to the SAP server.<br/>Allowed values are `0` (off, the default) or `1` (on). | No | | `sncMyName` | The initiator's SNC name to access the SAP server where the table is located.<br/>Applies when `sncMode` is on. | No | | `sncPartnerName` | The communication partner's SNC name to access the SAP server where the table is located.<br/>Applies when `sncMode` is on. | No |
To copy data from an SAP table, the following properties are supported:
| `rowCount` | The number of rows to be retrieved. | No | | `rfcTableFields` | The fields (columns) to copy from the SAP table. For example, `column0, column1`. | No | | `rfcTableOptions` | The options to filter the rows in an SAP table. For example, `COLUMN0 EQ 'SOMEVALUE'`. See also the SAP query operator table later in this article. | No |
-| `customRfcReadTableFunctionModule` | A custom RFC function module that can be used to read data from an SAP table.<br>You can use a custom RFC function module to define how the data is retrieved from your SAP system and returned to Data Factory. The custom function module must have an interface implemented (import, export, tables) that's similar to `/SAPDS/RFC_READ_TABLE2`, which is the default interface used by Data Factory.<br>Data Factory | No |
+| `customRfcReadTableFunctionModule` | A custom RFC function module that can be used to read data from an SAP table.<br>You can use a custom RFC function module to define how the data is retrieved from your SAP system and returned to the service. The custom function module must have an interface implemented (import, export, tables) that's similar to `/SAPDS/RFC_READ_TABLE2`, which is the default interface used by the service.| No |
| `partitionOption` | The partition mechanism to read from an SAP table. The supported options include: <ul><li>`None`</li><li>`PartitionOnInt` (normal integer or integer values with zero padding on the left, such as `0000012345`)</li><li>`PartitionOnCalendarYear` (4 digits in the format "YYYY")</li><li>`PartitionOnCalendarMonth` (6 digits in the format "YYYYMM")</li><li>`PartitionOnCalendarDate` (8 digits in the format "YYYYMMDD")</li><li>`PartitionOntime` (6 digits in the format "HHMMSS", such as `235959`)</li></ul> | No | | `partitionColumnName` | The name of the column used to partition the data. | No | | `partitionUpperBound` | The maximum value of the column specified in `partitionColumnName` that will be used to continue with partitioning. | No |
To copy data from an SAP table, the following properties are supported:
<br/> >Taking `partitionOption` as `partitionOnInt` as an example, the number of rows in each partition is calculated with this formula: (total rows falling between `partitionUpperBound` and `partitionLowerBound`)/`maxPartitionsNumber`.<br/> <br/>
->To load data partitions in parallel to speed up copy, the parallel degree is controlled by the [`parallelCopies`](copy-activity-performance-features.md#parallel-copy) setting on the copy activity. For example, if you set `parallelCopies` to four, Data Factory concurrently generates and runs four queries based on your specified partition option and settings, and each query retrieves a portion of data from your SAP table. We strongly recommend making `maxPartitionsNumber` a multiple of the value of the `parallelCopies` property. When copying data into file-based data store, it's also recommanded to write to a folder as multiple files (only specify folder name), in which case the performance is better than writing to a single file.
+>To load data partitions in parallel to speed up copy, the parallel degree is controlled by the [`parallelCopies`](copy-activity-performance-features.md#parallel-copy) setting on the copy activity. For example, if you set `parallelCopies` to four, the service concurrently generates and runs four queries based on your specified partition option and settings, and each query retrieves a portion of data from your SAP table. We strongly recommend making `maxPartitionsNumber` a multiple of the value of the `parallelCopies` property. When copying data into file-based data store, it's also recommanded to write to a folder as multiple files (only specify folder name), in which case the performance is better than writing to a single file.
>[!TIP]
-> The `BASXML` is enabled by default for this SAP Table connector on Azure Data Factory side.
+> The `BASXML` is enabled by default for this SAP Table connector within the service.
In `rfcTableOptions`, you can use the following common SAP query operators to filter the rows:
Below are illustrations of how SAP table connector works with custom function mo
1. Invoke "Custom function module" with the parameters set as below:
- - QUERY_TABLE: the table name you set in the ADF SAP Table dataset;
- - Delimiter: the delimiter you set in the ADF SAP Table Source;
- - ROWCOUNT/Option/Fields: the Rowcount/Aggregated Option/Fields you set in the ADF Table source.
+ - QUERY_TABLE: the table name you set in the SAP Table dataset;
+ - Delimiter: the delimiter you set in the SAP Table Source;
+ - ROWCOUNT/Option/Fields: the Rowcount/Aggregated Option/Fields you set in the Table source.
1. Get the result and parse the data in below ways:
Below are illustrations of how SAP table connector works with custom function mo
## Data type mappings for an SAP table
-When you're copying data from an SAP table, the following mappings are used from the SAP table data types to the Azure Data Factory interim data types. To learn how the copy activity maps the source schema and data type to the sink, see [Schema and data type mappings](copy-activity-schema-and-type-mapping.md).
+When you're copying data from an SAP table, the following mappings are used from the SAP table data types to interim data types used within the service. To learn how the copy activity maps the source schema and data type to the sink, see [Schema and data type mappings](copy-activity-schema-and-type-mapping.md).
-| SAP ABAP Type | Data Factory interim data type |
+| SAP ABAP Type | Service interim data type |
|: |: | | `C` (String) | `String` | | `I` (Integer) | `Int32` |
To learn details about the properties, check [Lookup activity](control-flow-look
## Next steps
-For a list of the data stores supported as sources and sinks by the copy activity in Azure Data Factory, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of the data stores supported as sources and sinks by the copy activity, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Sftp https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-sftp.md
Title: Copy data from and to SFTP server
-description: Learn how to copy data from and to SFTP server by using Azure Data Factory.
+description: Learn how to copy data from and to SFTP server by using Azure Data Factory and Azure Synapse Analytics pipelines.
Last updated 03/17/2021
-# Copy data from and to the SFTP server by using Azure Data Factory
+# Copy data from and to the SFTP server using Azure Data Factory or Azure Synapse Analytics
> [!div class="op_single_selector" title1="Select the version of the Data Factory service that you are using:"] > * [Version 1](v1/data-factory-sftp-connector.md) > * [Current version](connector-sftp.md) [!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to copy data from and to the secure FTP (SFTP) server. To learn about Azure Data Factory, read the [introductory article](introduction.md).
+This article outlines how to copy data from and to the secure FTP (SFTP) server. To learn more read the introductory article for [Azure Data Factory](introduction.md) or [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md).
## Supported capabilities
Specifically, the SFTP connector supports:
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide details about properties that are used to define Data Factory entities specific to SFTP.
+The following sections provide details about properties that are used to define entities specific to SFTP.
## Linked service properties
To use basic authentication, set the *authenticationType* property to *Basic*, a
| Property | Description | Required | |: |: |: | | userName | The user who has access to the SFTP server. |Yes |
-| password | The password for the user (userName). Mark this field as a SecureString to store it securely in your data factory, or [reference a secret stored in an Azure key vault](store-credentials-in-key-vault.md). | Yes |
+| password | The password for the user (userName). Mark this field as a SecureString to store it securely, or [reference a secret stored in an Azure key vault](store-credentials-in-key-vault.md). | Yes |
**Example:**
To use SSH public key authentication, set "authenticationType" property as **Ssh
|: |: |: | | userName | The user who has access to the SFTP server. |Yes | | privateKeyPath | Specify the absolute path to the private key file that the integration runtime can access. This applies only when the self-hosted type of integration runtime is specified in "connectVia." | Specify either `privateKeyPath` or `privateKeyContent`. |
-| privateKeyContent | Base64 encoded SSH private key content. SSH private key should be OpenSSH format. Mark this field as a SecureString to store it securely in your data factory, or [reference a secret stored in an Azure key vault](store-credentials-in-key-vault.md). | Specify either `privateKeyPath` or `privateKeyContent`. |
-| passPhrase | Specify the pass phrase or password to decrypt the private key if the key file or the key content is protected by a pass phrase. Mark this field as a SecureString to store it securely in your data factory, or [reference a secret stored in an Azure key vault](store-credentials-in-key-vault.md). | Yes, if the private key file or the key content is protected by a pass phrase. |
+| privateKeyContent | Base64 encoded SSH private key content. SSH private key should be OpenSSH format. Mark this field as a SecureString to store it securely, or [reference a secret stored in an Azure key vault](store-credentials-in-key-vault.md). | Specify either `privateKeyPath` or `privateKeyContent`. |
+| passPhrase | Specify the pass phrase or password to decrypt the private key if the key file or the key content is protected by a pass phrase. Mark this field as a SecureString to store it securely, or [reference a secret stored in an Azure key vault](store-credentials-in-key-vault.md). | Yes, if the private key file or the key content is protected by a pass phrase. |
> [!NOTE] > The SFTP connector supports an RSA/DSA OpenSSH key. Make sure that your key file content starts with "--BEGIN [RSA/DSA] PRIVATE KEY--". If the private key file is a PPK-format file, use the PuTTY tool to convert from PPK to OpenSSH format.
The following properties are supported for SFTP under `storeSettings` settings i
| type | The *type* property under `storeSettings` must be set to *SftpWriteSettings*. | Yes | | copyBehavior | Defines the copy behavior when the source is files from a file-based data store.<br/><br/>Allowed values are:<br/><b>- PreserveHierarchy (default)</b>: Preserves the file hierarchy in the target folder. The relative path of the source file to the source folder is identical to the relative path of the target file to the target folder.<br/><b>- FlattenHierarchy</b>: All files from the source folder are in the first level of the target folder. The target files have autogenerated names. <br/><b>- MergeFiles</b>: Merges all files from the source folder to one file. If the file name is specified, the merged file name is the specified name. Otherwise, it's an autogenerated file name. | No | | maxConcurrentConnections | The upper limit of concurrent connections established to the data store during the activity run. Specify a value only when you want to limit concurrent connections. | No |
-| useTempFileRename | Indicate whether to upload to temporary files and rename them, or directly write to the target folder or file location. By default, Azure Data Factory first writes to temporary files and then renames them when the upload is finished. This sequence helps to (1) avoid conflicts that might result in a corrupted file if you have other processes writing to the same file, and (2) ensure that the original version of the file exists during the transfer. If your SFTP server doesn't support a rename operation, disable this option and make sure that you don't have a concurrent write to the target file. For more information, see the troubleshooting tip at the end of this table. | No. Default value is *true*. |
+| useTempFileRename | Indicate whether to upload to temporary files and rename them, or directly write to the target folder or file location. By default, the service first writes to temporary files and then renames them when the upload is finished. This sequence helps to (1) avoid conflicts that might result in a corrupted file if you have other processes writing to the same file, and (2) ensure that the original version of the file exists during the transfer. If your SFTP server doesn't support a rename operation, disable this option and make sure that you don't have a concurrent write to the target file. For more information, see the troubleshooting tip at the end of this table. | No. Default value is *true*. |
| operationTimeout | The wait time before each write request to SFTP server times out. Default value is 60 min (01:00:00).|No | >[!TIP]
This table describes the behavior that results from using a file list path in th
## Lookup activity properties
-For information about Lookup activity properties, see [Lookup activity in Azure Data Factory](control-flow-lookup-activity.md).
+For information about Lookup activity properties, see [Lookup activity](control-flow-lookup-activity.md).
## GetMetadata activity properties
-For information about GetMetadata activity properties, see [GetMetadata activity in Azure Data Factory](control-flow-get-metadata-activity.md).
+For information about GetMetadata activity properties, see [GetMetadata activity](control-flow-get-metadata-activity.md).
## Delete activity properties
-For information about Delete activity properties, see [Delete activity in Azure Data Factory](delete-activity.md).
+For information about Delete activity properties, see [Delete activity](delete-activity.md).
## Legacy models >[!NOTE]
->The following models are still supported as is for backward compatibility. We recommend that you use the previously discussed new model, because the Azure Data Factory authoring UI has switched to generating the new model.
+>The following models are still supported as is for backward compatibility. We recommend that you use the previously discussed new model, because the authoring UI has switched to generating the new model.
### Legacy dataset model
For information about Delete activity properties, see [Delete activity in Azure
``` ## Next steps
-For a list of data stores that are supported as sources and sinks by the Copy activity in Azure Data Factory, see [supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of data stores that are supported as sources and sinks by the Copy activity, see [supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Sharepoint Online List https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-sharepoint-online-list.md
Title: Copy data from SharePoint Online List by using Azure Data Factory
+ Title: Copy data from SharePoint Online List
-description: Learn how to copy data from SharePoint Online List to supported sink data stores by using a copy activity in an Azure Data Factory pipeline.
+description: Learn how to copy data from SharePoint Online List to supported sink data stores by using a copy activity in an Azure Data Factory or Azure Synapse Analytics pipeline.
Last updated 05/19/2020
-# Copy data from SharePoint Online List by using Azure Data Factory
+# Copy data from SharePoint Online List by using Azure Data Factory or Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to use Copy Activity in Azure Data Factory to copy data from SharePoint Online List. The article builds on [Copy Activity in Azure Data Factory](copy-activity-overview.md), which presents a general overview of Copy Activity.
+This article outlines how to use Copy Activity in Azure Data Factory and Azure Synapse pipelines to copy data from SharePoint Online List. The article builds on [Copy Activity](copy-activity-overview.md), which presents a general overview of Copy Activity.
## Supported capabilities
The SharePoint List Online connector uses service principal authentication to co
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide details about properties you can use to define Data Factory entities that are specific to SharePoint Online List connector.
+The following sections provide details about properties you can use to define entities that are specific to SharePoint Online List connector.
## Linked service properties
The following properties are supported for an SharePoint Online List linked serv
| type | The type property must be set to:ΓÇ»**SharePointOnlineList**. | Yes | | siteUrl | The SharePoint Online site url, e.g. `https://contoso.sharepoint.com/sites/siteName`. | Yes | | servicePrincipalId | The Application (client) ID of the application registered in Azure Active Directory. | Yes |
-| servicePrincipalKey | The application's key. Mark this field as a **SecureString** to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
+| servicePrincipalKey | The application's key. Mark this field as a **SecureString** to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
| tenantId | The tenant ID under which your application resides. | Yes | | connectVia | The [Integration Runtime](concepts-integration-runtime.md) to use to connect to the data store. Learn more from [Prerequisites](#prerequisites), earlier in this article. If not specified, the default Azure Integration Runtime is used. | No |
To copy data from SharePoint Online List, the following properties are supported
``` > [!NOTE]
-> In Azure Data Factory, you can't select more than one *choice* data type for a SharePoint Online List source.
+> It isn't possible to select more than one *choice* data type for a SharePoint Online List source.
## Data type mapping for SharePoint Online List
-When you copy data from SharePoint Online List, the following mappings are used between SharePoint Online List data types and Azure Data Factory interim data types.
+When you copy data from SharePoint Online List, the following mappings are used between SharePoint Online List data types and interim data types used by the service internally.
-| **SharePoint Online data type** | **OData data type** | **Azure Data Factory interim data type** |
+| **SharePoint Online data type** | **OData data type** | **Interim data type** |
| -- | - | - | | Single line of text | Edm.String | String | | Multiple lines of text | Edm.String | String |
To learn details about the properties, check [Lookup activity](control-flow-look
## Next steps
-For a list of data stores that Copy Activity supports as sources and sinks in Azure Data Factory, see [Supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of data stores that Copy Activity supports as sources and sinks, see [Supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Connector Snowflake https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/connector-snowflake.md
Title: Copy and transform data in Snowflake
-description: Learn how to copy and transform data in Snowflake by using Data Factory.
+description: Learn how to copy and transform data in Snowflake using Data Factory or Azure Synapse Analytics.
Last updated 03/16/2021
-# Copy and transform data in Snowflake by using Azure Data Factory
+# Copy and transform data in Snowflake using Azure Data Factory or Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to use the Copy activity in Azure Data Factory to copy data from and to Snowflake, and use Data Flow to transform data in Snowflake. For more information about Data Factory, see the [introductory article](introduction.md).
+This article outlines how to use the Copy activity in Azure Data Factory and Azure Synapse pipelines to copy data from and to Snowflake, and use Data Flow to transform data in Snowflake. For more information, see the introductory article for [Data Factory](introduction.md) or [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md).
## Supported capabilities
For the Copy activity, this Snowflake connector supports the following functions
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
-The following sections provide details about properties that define Data Factory entities specific to a Snowflake connector.
+The following sections provide details about properties that define entities specific to a Snowflake connector.
## Linked service properties
The following properties are supported for the Snowflake dataset.
| Property | Description | Required | | :-- | :-- | :-- | | type | The type property of the dataset must be set to **SnowflakeTable**. | Yes |
-| schema | Name of the schema. Note the schema name is case-sensitive in ADF. |No for source, yes for sink |
-| table | Name of the table/view. Note the table name is case-sensitive in ADF. |No for source, yes for sink |
+| schema | Name of the schema. Note the schema name is case-sensitive. |No for source, yes for sink |
+| table | Name of the table/view. Note the table name is case-sensitive. |No for source, yes for sink |
**Example:**
To copy data from Snowflake, the following properties are supported in the Copy
| : | :-- | :- | | type | The type property of the Copy activity source must be set to **SnowflakeSource**. | Yes | | query | Specifies the SQL query to read data from Snowflake. If the names of the schema, table and columns contain lower case, quote the object identifier in query e.g. `select * from "schema"."myTable"`.<br>Executing stored procedure is not supported. | No |
-| exportSettings | Advanced settings used to retrieve data from Snowflake. You can configure the ones supported by the COPY into command that Data Factory will pass through when you invoke the statement. | No |
+| exportSettings | Advanced settings used to retrieve data from Snowflake. You can configure the ones supported by the COPY into command that the service will pass through when you invoke the statement. | No |
| ***Under `exportSettings`:*** | | | | type | The type of export command, set to **SnowflakeExportCopyCommand**. | Yes | | additionalCopyOptions | Additional copy options, provided as a dictionary of key-value pairs. Examples: MAX_FILE_SIZE, OVERWRITE. For more information, see [Snowflake Copy Options](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location.html#copy-options-copyoptions). | No |
To copy data from Snowflake, the following properties are supported in the Copy
#### Direct copy from Snowflake
-If your sink data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from Snowflake to sink. Data Factory checks the settings and fails the Copy activity run if the following criteria is not met:
+If your sink data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from Snowflake to sink. The service checks the settings and fails the Copy activity run if the following criteria is not met:
- The **sink linked service** is [**Azure Blob storage**](connector-azure-blob-storage.md) with **shared access signature** authentication. If you want to directly copy data to Azure Data Lake Storage Gen2 in the following supported format, you can create an Azure Blob linked service with SAS authentication against your ADLS Gen2 account, to avoid using [staged copy from Snowflake](#staged-copy-from-snowflake).
If your sink data store and format meet the criteria described in this section,
#### Staged copy from Snowflake
-When your sink data store or format is not natively compatible with the Snowflake COPY command, as mentioned in the last section, enable the built-in staged copy using an interim Azure Blob storage instance. The staged copy feature also provides you better throughput. Data Factory exports data from Snowflake into staging storage, then copies the data to sink, and finally cleans up your temporary data from the staging storage. See [Staged copy](copy-activity-performance-features.md#staged-copy) for details about copying data by using staging.
+When your sink data store or format is not natively compatible with the Snowflake COPY command, as mentioned in the last section, enable the built-in staged copy using an interim Azure Blob storage instance. The staged copy feature also provides you better throughput. The service exports data from Snowflake into staging storage, then copies the data to sink, and finally cleans up your temporary data from the staging storage. See [Staged copy](copy-activity-performance-features.md#staged-copy) for details about copying data by using staging.
To use this feature, create an [Azure Blob storage linked service](connector-azure-blob-storage.md#linked-service-properties) that refers to the Azure storage account as the interim staging. Then specify the `enableStaging` and `stagingSettings` properties in the Copy activity.
To copy data to Snowflake, the following properties are supported in the Copy ac
| :- | :-- | :-- | | type | The type property of the Copy activity sink, set to **SnowflakeSink**. | Yes | | preCopyScript | Specify a SQL query for the Copy activity to run before writing data into Snowflake in each run. Use this property to clean up the preloaded data. | No |
-| importSettings | Advanced settings used to write data into Snowflake. You can configure the ones supported by the COPY into command that Data Factory will pass through when you invoke the statement. | No |
+| importSettings | Advanced settings used to write data into Snowflake. You can configure the ones supported by the COPY into command that the service will pass through when you invoke the statement. | No |
| ***Under `importSettings`:*** | | | | type | The type of import command, set to **SnowflakeImportCopyCommand**. | Yes | | additionalCopyOptions | Additional copy options, provided as a dictionary of key-value pairs. Examples: ON_ERROR, FORCE, LOAD_UNCERTAIN_FILES. For more information, see [Snowflake Copy Options](https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#copy-options-copyoptions). | No |
To copy data to Snowflake, the following properties are supported in the Copy ac
#### Direct copy to Snowflake
-If your source data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from source to Snowflake. Azure Data Factory checks the settings and fails the Copy activity run if the following criteria is not met:
+If your source data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from source to Snowflake. The service checks the settings and fails the Copy activity run if the following criteria is not met:
- The **source linked service** is [**Azure Blob storage**](connector-azure-blob-storage.md) with **shared access signature** authentication. If you want to directly copy data from Azure Data Lake Storage Gen2 in the following supported format, you can create an Azure Blob linked service with SAS authentication against your ADLS Gen2 account, to avoid using [staged copy to Snowflake](#staged-copy-to-snowflake)..
If your source data store and format meet the criteria described in this section
#### Staged copy to Snowflake
-When your source data store or format is not natively compatible with the Snowflake COPY command, as mentioned in the last section, enable the built-in staged copy using an interim Azure Blob storage instance. The staged copy feature also provides you better throughput. Data Factory automatically converts the data to meet the data format requirements of Snowflake. It then invokes the COPY command to load data into Snowflake. Finally, it cleans up your temporary data from the blob storage. See [Staged copy](copy-activity-performance-features.md#staged-copy) for details about copying data using staging.
+When your source data store or format is not natively compatible with the Snowflake COPY command, as mentioned in the last section, enable the built-in staged copy using an interim Azure Blob storage instance. The staged copy feature also provides you better throughput. The service automatically converts the data to meet the data format requirements of Snowflake. It then invokes the COPY command to load data into Snowflake. Finally, it cleans up your temporary data from the blob storage. See [Staged copy](copy-activity-performance-features.md#staged-copy) for details about copying data using staging.
To use this feature, create an [Azure Blob storage linked service](connector-azure-blob-storage.md#linked-service-properties) that refers to the Azure storage account as the interim staging. Then specify the `enableStaging` and `stagingSettings` properties in the Copy activity.
For more information about the properties, see [Lookup activity](control-flow-lo
## Next steps
-For a list of data stores supported as sources and sinks by Copy activity in Data Factory, see [supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).
+For a list of data stores supported as sources and sinks by Copy activity, see [supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).
data-factory Control Flow Azure Function Activity https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/control-flow-azure-function-activity.md
Title: Azure Function Activity in Azure Data Factory
+ Title: Azure Function Activity
-description: Learn how to use the Azure Function activity to run an Azure Function in a Data Factory pipeline
+description: Learn how to use the Azure Function activity to run an Azure Function in an Azure Data Factory or Azure Synapse Analytics pipeline
Last updated 07/30/2021
# Azure Function activity in Azure Data Factory [!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-The Azure Function activity allows you to run [Azure Functions](../azure-functions/functions-overview.md) in a Data Factory pipeline. To run an Azure Function, you need to create a linked service connection and an activity that specifies the Azure Function that you plan to execute.
+The Azure Function activity allows you to run [Azure Functions](../azure-functions/functions-overview.md) in an Azure Data Factory or Synapse pipeline. To run an Azure Function, you need to create a linked service connection and an activity that specifies the Azure Function that you plan to execute.
For an eight-minute introduction and demonstration of this feature, watch the following video:
Learn more about Durable Functions in [this article](../azure-functions/durable/
## Sample
-You can find a sample of a Data Factory that uses an Azure Function to extract the content of a tar file [here](https://github.com/Azure/Azure-DataFactory/tree/master/SamplesV2/UntarAzureFilesWithAzureFunction).
+You can find a sample that uses an Azure Function to extract the content of a tar file [here](https://github.com/Azure/Azure-DataFactory/tree/master/SamplesV2/UntarAzureFilesWithAzureFunction).
## Next steps
-Learn more about activities in Data Factory in [Pipelines and activities in Azure Data Factory](concepts-pipelines-activities.md).
+Learn more about supported activities in [Pipelines and activities](concepts-pipelines-activities.md).
data-factory Control Flow Execute Data Flow Activity https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/control-flow-execute-data-flow-activity.md
Title: Data Flow activity
-description: How to execute data flows from inside a data factory pipeline.
+description: How to execute data flows from inside an Azure Data Factory or Azure Synapse Analytics pipeline.
Last updated 05/20/2021
-# Data Flow activity in Azure Data Factory
+# Data Flow activity in Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
traceLevel | Set logging level of your data flow activity execution | Fine, Coar
The Core Count and Compute Type properties can be set dynamically to adjust to the size of your incoming source data at runtime. Use pipeline activities like Lookup or Get Metadata in order to find the size of the source dataset data. Then, use Add Dynamic Content in the Data Flow activity properties. > [!NOTE]
-> When choosing driver and worker node cores in Synapse Data Flows, a minimum of 3 nodes will always be utilized.
+> When choosing driver and worker node cores in Azure Synapse Data Flows, a minimum of 3 nodes will always be utilized.
![Dynamic Data Flow](media/data-flow/dyna1.png "Dynamic data flow")
The Core Count and Compute Type properties can be set dynamically to adjust to t
### Data Flow integration runtime
-Choose which Integration Runtime to use for your Data Flow activity execution. By default, Data Factory will use the auto-resolve Azure Integration runtime with four worker cores. This IR has a general purpose compute type and runs in the same region as your factory. For operationalized pipelines, it is highly recommended that you create your own Azure Integration Runtimes that define specific regions, compute type, core counts, and TTL for your data flow activity execution.
+Choose which Integration Runtime to use for your Data Flow activity execution. By default, the service will use the auto-resolve Azure Integration runtime with four worker cores. This IR has a general purpose compute type and runs in the same region as your service instance. For operationalized pipelines, it is highly recommended that you create your own Azure Integration Runtimes that define specific regions, compute type, core counts, and TTL for your data flow activity execution.
A minimum compute type of General Purpose (compute optimized is not recommended for large workloads) with an 8+8 (16 total v-cores) configuration and a 10-minute is the minimum recommendation for most production workloads. By setting a small TTL, the Azure IR can maintain a warm cluster that will not incur the several minutes of start time for a cold cluster. You can speed up the execution of your data flows even more by select "Quick re-use" on the Azure IR data flow configurations. For more information, see [Azure integration runtime](concepts-integration-runtime.md).
If you're using an Azure Synapse Analytics as a sink or source, you must choose
## Logging level
-If you do not require every pipeline execution of your data flow activities to fully log all verbose telemetry logs, you can optionally set your logging level to "Basic" or "None". When executing your data flows in "Verbose" mode (default), you are requesting ADF to fully log activity at each individual partition level during your data transformation. This can be an expensive operation, so only enabling verbose when troubleshooting can improve your overall data flow and pipeline performance. "Basic" mode will only log transformation durations while "None" will only provide a summary of durations.
+If you do not require every pipeline execution of your data flow activities to fully log all verbose telemetry logs, you can optionally set your logging level to "Basic" or "None". When executing your data flows in "Verbose" mode (default), you are requesting the service to fully log activity at each individual partition level during your data transformation. This can be an expensive operation, so only enabling verbose when troubleshooting can improve your overall data flow and pipeline performance. "Basic" mode will only log transformation durations while "None" will only provide a summary of durations.
![Logging level](media/data-flow/logging.png "Set logging level") ## Sink properties
-The grouping feature in data flows allow you to both set the order of execution of your sinks as well as to group sinks together using the same group number. To help manage groups, you can ask ADF to run sinks, in the same group, in parallel. You can also set the sink group to continue even after one of the sinks encounters an error.
+The grouping feature in data flows allow you to both set the order of execution of your sinks as well as to group sinks together using the same group number. To help manage groups, you can ask the service to run sinks, in the same group, in parallel. You can also set the sink group to continue even after one of the sinks encounters an error.
The default behavior of data flow sinks is to execute each sink sequentially, in a serial manner, and to fail the data flow when an error is encountered in the sink. Additionally, all sinks are defaulted to the same group unless you go into the data flow properties and set different priorities for the sinks.
If your data flow uses parameterized datasets, set the parameter values in the *
### Parameterized data flows
-If your data flow is parameterized, set the dynamic values of the data flow parameters in the **Parameters** tab. You can use either the ADF pipeline expression language or the data flow expression language to assign dynamic or literal parameter values. For more information, see [Data Flow Parameters](parameters-data-flow.md).
+If your data flow is parameterized, set the dynamic values of the data flow parameters in the **Parameters** tab. You can use either the pipeline expression language or the data flow expression language to assign dynamic or literal parameter values. For more information, see [Data Flow Parameters](parameters-data-flow.md).
### Parameterized compute properties.
To get the number of rows read from a source named 'source1' that was used in th
## Next steps
-See control flow activities supported by Data Factory:
+See supported control flow activities:
- [If Condition Activity](control-flow-if-condition-activity.md) - [Execute Pipeline Activity](control-flow-execute-pipeline-activity.md)
data-factory Control Flow For Each Activity https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/control-flow-for-each-activity.md
Title: ForEach activity in Azure Data Factory
+ Title: ForEach activity
-description: The For Each Activity defines a repeating control flow in your pipeline. It is used for iterating over a collection and execute specified activities.
+description: The For Each Activity defines a repeating control flow in an Azure Data Factory or Azure Synapse Analytics pipeline. The For Each Activity is used for iterating over a collection to execute actions on each item in the collection individually.
Last updated 01/23/2019
-# ForEach activity in Azure Data Factory
+# ForEach activity in Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-The ForEach Activity defines a repeating control flow in your pipeline. This activity is used to iterate over a collection and executes specified activities in a loop. The loop implementation of this activity is similar to Foreach looping structure in programming languages.
+The ForEach Activity defines a repeating control flow in an Azure Data Factory or Synapse pipeline. This activity is used to iterate over a collection and executes specified activities in a loop. The loop implementation of this activity is similar to Foreach looping structure in programming languages.
## Syntax The properties are described later in this article. The items property is the collection and each item in the collection is referred to by using the `@item()` as shown in the following syntax:
Here are some limitations of the ForEach activity and suggested workarounds.
| | | ## Next steps
-See other control flow activities supported by Data Factory:
+See other supported control flow activities:
- [Execute Pipeline Activity](control-flow-execute-pipeline-activity.md) - [Get Metadata Activity](control-flow-get-metadata-activity.md)
data-factory Control Flow Set Variable Activity https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/control-flow-set-variable-activity.md
Title: Set Variable Activity in Azure Data Factory
+ Title: Set Variable Activity
-description: Learn how to use the Set Variable activity to set the value of an existing variable defined in a Data Factory pipeline
+description: Learn how to use the Set Variable activity to set the value of an existing variable defined in an Azure Data Factory or Azure Synapse Analytics pipeline.
-# Set Variable Activity in Azure Data Factory
+# Set Variable Activity in Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-Use the Set Variable activity to set the value of an existing variable of type String, Bool, or Array defined in a Data Factory pipeline.
+Use the Set Variable activity to set the value of an existing variable of type String, Bool, or Array defined in a Data Factory or Synapse pipeline.
## Type properties
variableName | Name of the variable that is set by this activity | yes
## Incrementing a variable
-A common scenario involving variables in Azure Data Factory is using a variable as an iterator within an until or foreach activity. In a set variable activity you cannot reference the variable being set in the `value` field. To workaround this limitation, set a temporary variable and then create a second set variable activity. The second set variable activity sets the value of the iterator to the temporary variable.
+A common scenario involving variables is using a variable as an iterator within an until or foreach activity. In a set variable activity you cannot reference the variable being set in the `value` field. To workaround this limitation, set a temporary variable and then create a second set variable activity. The second set variable activity sets the value of the iterator to the temporary variable.
Below is an example of this pattern:
Below is an example of this pattern:
Variables are currently scoped at the pipeline level. This means that they are not thread safe and can cause unexpected and undesired behavior if they are accessed from within a parallel iteration activity such as a foreach loop, especially when the value is also being modified within that foreach activity. ## Next steps
-Learn about a related control flow activity supported by Data Factory:
+Learn about another related control flow activity:
- [Append Variable Activity](control-flow-append-variable-activity.md)
data-factory Control Flow System Variables https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/control-flow-system-variables.md
Title: System variables in Azure Data Factory
+ Title: System variables
-description: This article describes system variables supported by Azure Data Factory. You can use these variables in expressions when defining Data Factory entities.
+description: This article describes system variables supported by Azure Data Factory and Azure Synapse Analytics. You can use these variables in expressions when defining entities within either service.
Last updated 06/12/2018
-# System variables supported by Azure Data Factory
+# System variables supported by Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article describes system variables supported by Azure Data Factory. You can use these variables in expressions when defining Data Factory entities.
+This article describes system variables supported by Azure Data Factory and Azure Synapse. You can use these variables in expressions when defining entities within either service.
## Pipeline scope
These system variables can be referenced anywhere in the pipeline JSON.
| Variable Name | Description | | | |
-| @pipeline().DataFactory |Name of the data factory the pipeline run is running in |
+| @pipeline().DataFactory |Name of the data or Synapse workspace the pipeline run is running in |
| @pipeline().Pipeline |Name of the pipeline | | @pipeline().RunId |ID of the specific pipeline run |
-| @pipeline().TriggerType |The type of trigger that invoked the pipeline (for example, `ScheduleTrigger`, `BlobEventsTrigger`). For a list of supported trigger types, see [Pipeline execution and triggers in Azure Data Factory](concepts-pipeline-execution-triggers.md). A trigger type of `Manual` indicates that the pipeline was triggered manually. |
+| @pipeline().TriggerType |The type of trigger that invoked the pipeline (for example, `ScheduleTrigger`, `BlobEventsTrigger`). For a list of supported trigger types, see [Pipeline execution and triggers](concepts-pipeline-execution-triggers.md). A trigger type of `Manual` indicates that the pipeline was triggered manually. |
| @pipeline().TriggerId|ID of the trigger that invoked the pipeline | | @pipeline().TriggerName|Name of the trigger that invoked the pipeline | | @pipeline().TriggerTime|Time of the trigger run that invoked the pipeline. This is the time at which the trigger **actually** fired to invoke the pipeline run, and it may differ slightly from the trigger's scheduled time. | | @pipeline().GroupId | ID of the group to which pipeline run belongs. |
-| @pipeline()?.TriggeredByPipelineName | Name of the pipeline that trigger the pipeline run. Applicable when the pipeline run is triggered by an ExecutePipeline activity. Evaluate to _Null_ when used in other circumstances. Note the question mark after @pipeline() |
-| @pipeline()?.TriggeredByPipelineRunId | Run id of the pipeline that trigger the pipeline run. Applicable when the pipeline run is triggered by an ExecutePipeline activity. Evaluate to _Null_ when used in other circumstances. Note the question mark after @pipeline() |
+| @pipeline()?TriggeredByPipelineName | Name of the pipeline that triggers the pipeline run. Applicable when the pipeline run is triggered by an ExecutePipeline activity. Evaluate to _Null_ when used in other circumstances. Note the question mark after @pipeline() |
+| @pipeline()?TriggeredByPipelineRunId | Run ID of the pipeline that triggers the pipeline run. Applicable when the pipeline run is triggered by an ExecutePipeline activity. Evaluate to _Null_ when used in other circumstances. Note the question mark after @pipeline() |
>[!NOTE] >Trigger-related date/time system variables (in both pipeline and trigger scopes) return UTC dates in ISO 8601 format, for example, `2017-06-01T22:20:00.4061448Z`.
These system variables can be referenced anywhere in the trigger JSON for trigge
These system variables can be referenced anywhere in the trigger JSON for triggers of type [CustomEventsTrigger](concepts-pipeline-execution-triggers.md#event-based-trigger). >[!NOTE]
->Azure Data Factory expects custom event to be formatted with [Azure Event Grid event schema](../event-grid/event-schema.md).
+>The service expects custom events to be formatted with [Azure Event Grid event schema](../event-grid/event-schema.md).
| Variable Name | Description | | |
data-factory Control Flow Web Activity https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/control-flow-web-activity.md
Title: Web Activity in Azure Data Factory
+ Title: Web Activity
-description: Learn how you can use Web Activity, one of the control flow activities supported by Data Factory, to invoke a REST endpoint from a pipeline.
+description: Learn how you can use Web Activity, one of the control flow activities supported by Azure Data Factory and Azure Synapse Analytics, to invoke a REST endpoint from a pipeline.
Last updated 12/19/2018
-# Web activity in Azure Data Factory
+# Web activity in Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-Web Activity can be used to call a custom REST endpoint from a Data Factory pipeline. You can pass datasets and linked services to be consumed and accessed by the activity.
+Web Activity can be used to call a custom REST endpoint from an Azure Data Factory or Synapse pipeline. You can pass datasets and linked services to be consumed and accessed by the activity.
> [!NOTE] > Web Activity is supported for invoking URLs that are hosted in a private virtual network as well by leveraging self-hosted integration runtime. The integration runtime should have a line of sight to the URL endpoint.
Specify base64-encoded contents of a PFX file and the password.
### Managed Identity
-Specify the resource uri for which the access token will be requested using the managed identity for the data factory. To call the Azure Resource Management API, use `https://management.azure.com/`. For more information about how managed identities works see the [managed identities for Azure resources overview page](../active-directory/managed-identities-azure-resources/overview.md).
+Specify the resource uri for which the access token will be requested using the managed identity for the data factory or Synapse workspace instance. To call the Azure Resource Management API, use `https://management.azure.com/`. For more information about how managed identities works see the [managed identities for Azure resources overview page](../active-directory/managed-identities-azure-resources/overview.md).
```json "authentication": {
Specify the resource uri for which the access token will be requested using the
``` > [!NOTE]
-> If your data factory is configured with a git repository, you must store your credentials in Azure Key Vault to use basic or client certificate authentication. Azure Data Factory doesn't store passwords in git.
+> If your data factory or Synapse workspace is configured with a git repository, you must store your credentials in Azure Key Vault to use basic or client certificate authentication. The service does not store passwords in git.
## Request payload schema When you use the POST/PUT method, the body property represents the payload that is sent to the endpoint. You can pass linked services and datasets as part of the payload. Here is the schema for the payload:
public HttpResponseMessage Execute(JObject payload)
``` ## Next steps
-See other control flow activities supported by Data Factory:
+See other supported control flow activities:
- [Execute Pipeline Activity](control-flow-execute-pipeline-activity.md) - [For Each Activity](control-flow-for-each-activity.md)
data-factory Copy Activity Monitoring https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/copy-activity-monitoring.md
Title: Monitor copy activity
-description: Learn about how to monitor the copy activity execution in Azure Data Factory.
+description: Learn about how to monitor the copy activity execution in Azure Data Factory and Azure Synapse Analytics.
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines how to monitor the copy activity execution in Azure Data Factory. It builds on the [copy activity overview](copy-activity-overview.md) article that presents a general overview of copy activity.
+This article outlines how to monitor the copy activity execution in Azure Data Factory and Synapse pipelines. It builds on the [copy activity overview](copy-activity-overview.md) article that presents a general overview of copy activity.
## Monitor visually
-Once you've created and published a pipeline in Azure Data Factory, you can associate it with a trigger or manually kick off an ad hoc run. You can monitor all of your pipeline runs natively in the Azure Data Factory user experience. Learn about Azure Data Factory monitoring in general from [Visually monitor Azure Data Factory](monitor-visually.md).
+Once you've created and published a pipeline, you can associate it with a trigger or manually kick off an ad hoc run. You can monitor all of your pipeline runs natively in the user experience. Learn about monitoring in general from [Visually monitor Azure Data Factory and Synapse pipelines](monitor-visually.md).
-To monitor the Copy activity run, go to your data factory **Author & Monitor** UI. On the **Monitor** tab, you see a list of pipeline runs, click the **pipeline name** link to access the list of activity runs in the pipeline run.
+To monitor the Copy activity run, go to the **Data Factory Studio** or **Azure Synapse Studio** UI for your service instance. On the **Monitor** tab, you see a list of pipeline runs, click the **pipeline name** link to access the list of activity runs in the pipeline run.
+
+# [Azure Data Factory](#tab/data-factory)
![Monitor pipeline run](./media/copy-activity-overview/monitor-pipeline-run.png)
+# [Azure Synapse](#tab/synapse-analytics)
+
+![Monitor pipeline run](./media/copy-activity-overview/monitor-pipeline-run-synapse.png)
+++ At this level, you can see links to copy activity input, output, and errors (if the Copy activity run fails), as well as statistics like duration/status. Clicking the **Details** button (eyeglasses) next to the copy activity name will give you deep details on your copy activity execution. ![Monitor copy activity run](./media/copy-activity-overview/monitor-copy-activity-run.png)
-In this graphical monitoring view, Azure Data Factory presents you the copy activity execution information, including data read/written volume, number of files/rows of data copied from source to sink, throughput, the configurations applied for your copy scenario, steps the copy activity goes through with corresponding durations and details, and more. Refer to [this table](#monitor-programmatically) on each possible metric and its detailed description.
+In this graphical monitoring view, the service presents you the copy activity execution information, including data read/written volume, number of files/rows of data copied from source to sink, throughput, the configurations applied for your copy scenario, steps the copy activity goes through with corresponding durations and details, and more. Refer to [this table](#monitor-programmatically) on each possible metric and its detailed description.
-In some scenarios, when you run a Copy activity in Data Factory, you'll see **"Performance tuning tips"** at the top of the copy activity monitoring view as shown in the example. The tips tell you the bottleneck identified by ADF for the specific copy run, along with suggestion on what to change to boost copy throughput. Learn more about [auto performance tuning tips](copy-activity-performance-troubleshooting.md#performance-tuning-tips).
+In some scenarios, when you run a Copy activity, you'll see **"Performance tuning tips"** at the top of the copy activity monitoring view as shown in the example. The tips tell you the bottleneck identified by the service for the specific copy run, along with suggestion on what to change to boost copy throughput. Learn more about [auto performance tuning tips](copy-activity-performance-troubleshooting.md#performance-tuning-tips).
The bottom **execution details and durations** describes the key steps your copy activity goes through, which is especially useful for troubleshooting the copy performance. The bottleneck of your copy run is the one with the longest duration. Refer to [Troubleshoot copy activity performance](copy-activity-performance-troubleshooting.md) on for what each stage represents and the detailed troubleshooting guidance.
The bottom **execution details and durations** describes the key steps your copy
## Monitor programmatically
-Copy activity execution details and performance characteristics are also returned in the **Copy Activity run result** > **Output** section, which is used to render the UI monitoring view. Following is a complete list of properties that might be returned. You'll see only the properties that are applicable to your copy scenario. For information about how to monitor activity runs programmatically in general, see [Programmatically monitor an Azure data factory](monitor-programmatically.md).
+Copy activity execution details and performance characteristics are also returned in the **Copy Activity run result** > **Output** section, which is used to render the UI monitoring view. Following is a complete list of properties that might be returned. You'll see only the properties that are applicable to your copy scenario. For information about how to monitor activity runs programmatically in general, see [Programmatically monitor an Azure Data Factory or Synapse pipeline](monitor-programmatically.md).
| Property name | Description | Unit in output | |: |: |: |
data-factory Copy Activity Performance Features https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/copy-activity-performance-features.md
Title: Copy activity performance optimization features
-description: Learn about the key features that help you optimize the copy activity performance in Azure Data Factory。
+description: Learn about the key features that help you optimize the copy activity performance in Azure Data Factory and Azure Synapse Analytics pipelines.
Last updated 09/24/2020
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-This article outlines the copy activity performance optimization features that you can leverage in Azure Data Factory.
+This article outlines the copy activity performance optimization features that you can leverage in Azure Data Factory and Synapse pipelines.
## Data Integration Units
-A Data Integration Unit is a measure that represents the power (a combination of CPU, memory, and network resource allocation) of a single unit in Azure Data Factory. Data Integration Unit only applies to [Azure integration runtime](concepts-integration-runtime.md#azure-integration-runtime), but not [self-hosted integration runtime](concepts-integration-runtime.md#self-hosted-integration-runtime).
+A Data Integration Unit is a measure that represents the power (a combination of CPU, memory, and network resource allocation) of a single unit within the service. Data Integration Unit only applies to [Azure integration runtime](concepts-integration-runtime.md#azure-integration-runtime), but not [self-hosted integration runtime](concepts-integration-runtime.md#self-hosted-integration-runtime).
-The allowed DIUs to empower a copy activity run is **between 2 and 256**. If not specified or you choose "Auto" on the UI, Data Factory dynamically applies the optimal DIU setting based on your source-sink pair and data pattern. The following table lists the supported DIU ranges and default behavior in different copy scenarios:
+The allowed DIUs to empower a copy activity run is **between 2 and 256**. If not specified or you choose "Auto" on the UI, the service dynamically applies the optimal DIU setting based on your source-sink pair and data pattern. The following table lists the supported DIU ranges and default behavior in different copy scenarios:
| Copy scenario | Supported DIU range | Default DIUs determined by service | |: |: |- |
You can set parallel copy (`parallelCopies` property) on copy activity to indica
The parallel copy is orthogonal to [Data Integration Units](#data-integration-units) or [Self-hosted IR nodes](#self-hosted-integration-runtime-scalability). It is counted across all the DIUs or Self-hosted IR nodes.
-For each copy activity run, by default Azure Data Factory dynamically applies the optimal parallel copy setting based on your source-sink pair and data pattern.
+For each copy activity run, by default the service dynamically applies the optimal parallel copy setting based on your source-sink pair and data pattern.
> [!TIP]
-> The default behavior of parallel copy usually gives you the best throughput, which is auto-determined by ADF based on your source-sink pair, data pattern and number of DIUs or the Self-hosted IR's CPU/memory/node count. Refer to [Troubleshoot copy activity performance](copy-activity-performance-troubleshooting.md) on when to tune parallel copy.
+> The default behavior of parallel copy usually gives you the best throughput, which is auto-determined by the service based on your source-sink pair, data pattern and number of DIUs or the Self-hosted IR's CPU/memory/node count. Refer to [Troubleshoot copy activity performance](copy-activity-performance-troubleshooting.md) on when to tune parallel copy.
The following table lists the parallel copy behavior:
When you copy data from a source data store to a sink data store, you might choo
### How staged copy works
-When you activate the staging feature, first the data is copied from the source data store to the staging storage (bring your own Azure Blob or Azure Data Lake Storage Gen2). Next, the data is copied from the staging to the sink data store. Azure Data Factory copy activity automatically manages the two-stage flow for you, and also cleans up temporary data from the staging storage after the data movement is complete.
+When you activate the staging feature, first the data is copied from the source data store to the staging storage (bring your own Azure Blob or Azure Data Lake Storage Gen2). Next, the data is copied from the staging to the sink data store. The copy activity automatically manages the two-stage flow for you, and also cleans up temporary data from the staging storage after the data movement is complete.
![Staged copy](media/copy-activity-performance/staged-copy.png)
data-factory Copy Activity Performance Troubleshooting https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/copy-activity-performance-troubleshooting.md
Title: Troubleshoot copy activity performance
-description: Learn about how to troubleshoot copy activity performance in Azure Data Factory.
+description: Learn about how to troubleshoot copy activity performance in Azure Data Factory and Azure Synapse Analytics.
After you run a copy activity, you can collect the run result and performance st
## Performance tuning tips
-In some scenarios, when you run a copy activity in Data Factory, you'll see **"Performance tuning tips"** at the top as shown in the above example. The tips tell you the bottleneck identified by ADF for this particular copy run, along with suggestion on how to boost copy throughput. Try making the recommanded change, then run the copy again.
+In some scenarios, when you run a copy activity, you'll see **"Performance tuning tips"** at the top as shown in the above example. The tips tell you the bottleneck identified by the service for this particular copy run, along with suggestion on how to boost copy throughput. Try making the recommended change, then run the copy again.
As a reference, currently the performance tuning tips provide suggestions for the following cases:
When the copy activity performance doesn't meet your expectation, to troubleshoo
- Consider to split single large data set into several smaller data sets, and let those copy jobs run concurrently each tackles portion of data. You can do this with Lookup/GetMetadata + ForEach + Copy. Refer to [Copy files from multiple containers](solution-template-copy-files-multiple-containers.md) or [Migrate data from Amazon S3 to ADLS Gen2](solution-template-migration-s3-azure.md) solution templates as general example.
- - Check if ADF reports any throttling error on source or if your data store is under high utilization state. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
+ - Check if the service reports any throttling error on source or if your data store is under high utilization state. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
- Use Azure IR in the same or close to your source data store region.
When the copy activity performance doesn't meet your expectation, to troubleshoo
- Adopt connector-specific data loading best practice if applies. For example, when copying data from [Amazon Redshift](connector-amazon-redshift.md), configure to use Redshift UNLOAD.
- - Check if ADF reports any throttling error on source or if your data store is under high utilization. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
+ - Check if the service reports any throttling error on source or if your data store is under high utilization. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
- Check your copy source and sink pattern:
When the copy activity performance doesn't meet your expectation, to troubleshoo
- Adopt connector-specific data loading best practice if applies. For example, when copying data into [Azure Synapse Analytics](connector-azure-sql-data-warehouse.md), use PolyBase or COPY statement.
- - Check if ADF reports any throttling error on sink or if your data store is under high utilization. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
+ - Check if the service reports any throttling error on sink or if your data store is under high utilization. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
- Check your copy source and sink pattern:
When the copy performance doesn't meet your expectation, to troubleshoot single
- Consider to split single large data set into several smaller data sets, and let those copy jobs run concurrently each tackles portion of data. You can do this with Lookup/GetMetadata + ForEach + Copy. Refer to [Copy files from multiple containers](solution-template-copy-files-multiple-containers.md) or [Migrate data from Amazon S3 to ADLS Gen2](solution-template-migration-s3-azure.md) solution templates as general example.
- - Check if ADF reports any throttling error on source or if your data store is under high utilization state. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
+ - Check if the service reports any throttling error on source or if your data store is under high utilization state. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
- **"Transfer - reading from source" experienced long working duration**:
When the copy performance doesn't meet your expectation, to troubleshoot single
- Check if the Self-hosted IR machine has enough inbound bandwidth to read and transfer the data efficiently. If your source data store is in Azure, you can use [this tool](https://www.azurespeed.com/Azure/Download) to check the download speed.
- - Check the Self-hosted IR's CPU and memory usage trend in Azure portal -> your data factory -> overview page. Consider to [scale up/out IR](create-self-hosted-integration-runtime.md#high-availability-and-scalability) if the CPU usage is high or available memory is low.
+ - Check the Self-hosted IR's CPU and memory usage trend in Azure portal -> your data factory or Synapse workspace -> overview page. Consider to [scale up/out IR](create-self-hosted-integration-runtime.md#high-availability-and-scalability) if the CPU usage is high or available memory is low.
- Adopt connector-specific data loading best practice if applies. For example:
When the copy performance doesn't meet your expectation, to troubleshoot single
- When copying data from [Amazon Redshift](connector-amazon-redshift.md), configure to use Redshift UNLOAD.
- - Check if ADF reports any throttling error on source or if your data store is under high utilization. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
+ - Check if the service reports any throttling error on source or if your data store is under high utilization. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
- Check your copy source and sink pattern:
When the copy performance doesn't meet your expectation, to troubleshoot single
- Check if the Self-hosted IR machine has enough outbound bandwidth to transfer and write the data efficiently. If your sink data store is in Azure, you can use [this tool](https://www.azurespeed.com/Azure/UploadLargeFile) to check the upload speed.
- - Check if the Self-hosted IR's CPU and memory usage trend in Azure portal -> your data factory -> overview page. Consider to [scale up/out IR](create-self-hosted-integration-runtime.md#high-availability-and-scalability) if the CPU usage is high or available memory is low.
+ - Check if the Self-hosted IR's CPU and memory usage trend in Azure portal -> your data factory or Synapse workspace -> overview page. Consider to [scale up/out IR](create-self-hosted-integration-runtime.md#high-availability-and-scalability) if the CPU usage is high or available memory is low.
- - Check if ADF reports any throttling error on sink or if your data store is under high utilization. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
+ - Check if the service reports any throttling error on sink or if your data store is under high utilization. If so, either reduce your workloads on the data store, or try contacting your data store administrator to increase the throttling limit or available resource.
- Consider to gradually tune the [parallel copies](copy-activity-performance-features.md), note that too many parallel copies may even hurt the performance.
Activity execution time varies when the dataset is based on different Integratio
- **Symptoms**: Simply toggling the Linked Service dropdown in the dataset performs the same pipeline activities, but has drastically different run-times. When the dataset is based on the Managed Virtual Network Integration Runtime, it takes more time on average than the run when based on the Default Integration Runtime. -- **Cause**: Checking the details of pipeline runs, you can see that the slow pipeline is running on Managed VNet (Virtual Network) IR while the normal one is running on Azure IR. By design, Managed VNet IR takes longer queue time than Azure IR as we are not reserving one compute node per data factory, so there is a warm up for each copy activity to start, and it occurs primarily on VNet join rather than Azure IR. --
+- **Cause**: Checking the details of pipeline runs, you can see that the slow pipeline is running on Managed VNet (Virtual Network) IR while the normal one is running on Azure IR. By design, Managed VNet IR takes longer queue time than Azure IR as we are not reserving one compute node per service instance, so there is a warm up for each copy activity to start, and it occurs primarily on VNet join rather than Azure IR.
### Low performance when loading data into Azure SQL Database
Activity execution time varies when the dataset is based on different Integratio
- For operations like importing schema, previewing data, and listing worksheets on excel dataset, the timeout is 100 s and static. For large Excel file, these operations may not finish within the timeout value.
- - ADF copy activity reads the whole Excel file into memory then locate the specified worksheet and cells to read data. This behavior is due to the underlying SDK ADF uses.
+ - The copy activity reads the whole Excel file into memory then locate the specified worksheet and cells to read data. This behavior is due to the underlying SDK the service uses.
- **Resolution**:
See the other copy activity articles:
- [Copy activity overview](copy-activity-overview.md) - [Copy activity performance and scalability guide](copy-activity-performance.md) - [Copy activity performance optimization features](copy-activity-performance-features.md)-- [Use Azure Data Factory to migrate data from your data lake or data warehouse to Azure](data-migration-guidance-overview.md)
+- [Migrate data from your data lake or data warehouse to Azure](data-migration-guidance-overview.md)
- [Migrate data from Amazon S3 to Azure Storage](data-migration-guidance-s3-azure-storage.md)
data-factory Create Azure Integration Runtime https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/create-azure-integration-runtime.md
Title: Create Azure integration runtime in Azure Data Factory
+ Title: Create Azure integration runtime
-description: Learn how to create Azure integration runtime in Azure Data Factory, which is used to copy data and dispatch transform activities.
+description: Learn how to create Azure integration runtime in Azure Data Factory and Azure Synapse Analytics, which is used to copy data and dispatch transform activities.
# How to create and configure Azure Integration Runtime [!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-The Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory to provide data integration capabilities across different network environments. For more information about IR, see [Integration runtime](concepts-integration-runtime.md).
+The Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory and Synapse pipelines to provide data integration capabilities across different network environments. For more information about IR, see [Integration runtime](concepts-integration-runtime.md).
Azure IR provides a fully managed compute to natively perform data movement and dispatch data transformation activities to compute services like HDInsight. It is hosted in Azure environment and supports connecting to resources in public network environment with public accessible endpoints.
This document introduces how you can create and configure Azure Integration Runt
[!INCLUDE [updated-for-az](../../includes/updated-for-az.md)] ## Default Azure IR
-By default, each data factory has an Azure IR in the backend that supports operations on cloud data stores and compute services in public network. The location of that Azure IR is autoresolve. If **connectVia** property is not specified in the linked service definition, the default Azure IR is used. You only need to explicitly create an Azure IR when you would like to explicitly define the location of the IR, or if you would like to virtually group the activity executions on different IRs for management purpose.
+By default, each data factory or Synapse workspace has an Azure IR in the backend that supports operations on cloud data stores and compute services in public network. The location of that Azure IR is autoresolve. If **connectVia** property is not specified in the linked service definition, the default Azure IR is used. You only need to explicitly create an Azure IR when you would like to explicitly define the location of the IR, or if you would like to virtually group the activity executions on different IRs for management purpose.
## Create Azure IR
For Azure IR, the type must be set to **Managed**. You do not need to specify co
You can configure an existing Azure IR to change its location using the Set-AzDataFactoryV2IntegrationRuntime PowerShell cmdlet. For more information about the location of an Azure IR, see [Introduction to integration runtime](concepts-integration-runtime.md).
-### Create an Azure IR via Azure Data Factory UI
-Use the following steps to create an Azure IR using Azure Data Factory UI.
+### Create an Azure IR via UI
+Use the following steps to create an Azure IR using UI.
-1. On the home page of Azure Data Factory UI, select the [Manage tab](./author-management-hub.md) from the leftmost pane.
+1. On the home page for the service, select the [Manage tab](./author-management-hub.md) from the leftmost pane.
- ![The home page Manage button](media/doc-common-process/get-started-page-manage-button.png)
+ # [Azure Data Factory](#tab/data-factory)
+
+ :::image type="content" source="media/doc-common-process/get-started-page-manage-button.png" alt-text="The home page Manage button":::
-1. Select **Integration runtimes** on the left pane, and then select **+New**.
+ # [Azure Synapse](#tab/synapse-analytics)
- ![Screenshot that highlights Integration runtimes in the left pane and the +New button.](media/doc-common-process/manage-new-integration-runtime.png)
+ :::image type="content" source="media/doc-common-process/get-started-page-manage-button-synapse.png" alt-text="The home page Manage button":::
-1. On the **Integration runtime setup** page, select **Azure, Self-Hosted**, and then select **Continue**.
++
+2. Select **Integration runtimes** on the left pane, and then select **+New**.
+
+ # [Azure Data Factory](#tab/data-factory)
+
+ :::image type="content" source="media/doc-common-process/manage-new-integration-runtime.png" alt-text="Screenshot that highlights Integration runtimes in the left pane and the +New button.":::
+
+ # [Azure Synapse](#tab/synapse-analytics)
+
+ :::image type="content" source="media/doc-common-process/manage-new-integration-runtime-synapse.png" alt-text="Screenshot that highlights Integration runtimes in the left pane and the +New button.":::
+++
+3. On the **Integration runtime setup** page, select **Azure, Self-Hosted**, and then select **Continue**.
1. On the following page, select **Azure** to create an Azure IR, and then select **Continue**. ![Create an integration runtime](media/create-azure-integration-runtime/new-azure-integration-runtime.png)
data-factory Data Factory Private Link https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/data-factory-private-link.md
Enabling the Private Link service for each of the preceding communication channe
> For functionality that's not currently supported, you still need to configure the previously mentioned domain and port in the virtual network or your corporate firewall. > [!NOTE]
- > Connecting to Azure Data Factory via private endpoint is only applicable to self-hosted integration runtime in data factory. It's not supported in Synapse.
+ > Connecting to Azure Data Factory via private endpoint is only applicable to self-hosted integration runtime in data factory. It is not supported for Azure Synapse.
> [!WARNING] > If you enable Private Link in Azure Data Factory and block public access at the same time, make sure when you create a linked service, your credentials are stored in an Azure key vault. Otherwise, the credentials won't work.
data-factory Data Flow Alter Row https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/data-flow-alter-row.md
Title: Alter row transformation in mapping data flow
-description: How to update database target using the alter row transformation in mapping data flow
+description: How to update database target using the alter row transformation in the mapping data flow in Azure Data Factory and Azure Synapse Analytics pipelines.
The default behavior is to only allow inserts. To allow updates, upserts, or del
The sink transformation requires either a single key or a series of keys for unique row identification in your target database. For SQL sinks, set the keys in the sink settings tab. For CosmosDB, set the partition key in the settings and also set the CosmosDB system field "id" in your sink mapping. For CosmosDB, it is mandatory to include the system column "id" for updates, upserts, and deletes.
-## Merges and upserts with Azure SQL Database and Synapse
+## Merges and upserts with Azure SQL Database and Azure Synapse
-ADF Data Flows supports merges against Azure SQL Database and Synapse database pool (data warehouse) with the upsert option.
+Data Flows support merges against Azure SQL Database and Azure Synapse database pool (data warehouse) with the upsert option.
-However, you may run into scenarios where your target database schema utilized the identity property of key columns. ADF requires you to identify the keys that you will use to match the row values for updates and upserts. But if the target column has the identity property set and you are using the upsert policy, the target database will not allow you to write to the column. You may also run into errors when you try to upsert against a distributed table's distribution column.
+However, you may run into scenarios where your target database schema utilized the identity property of key columns. The service requires you to identify the keys that you will use to match the row values for updates and upserts. But if the target column has the identity property set and you are using the upsert policy, the target database will not allow you to write to the column. You may also run into errors when you try to upsert against a distributed table's distribution column.
Here are ways to fix that:
-1. Go to the Sink transformation Settings and set "Skip writing key columns". This will tell ADF to not write the column that you have selected as the key value for your mapping.
+1. Go to the Sink transformation Settings and set "Skip writing key columns". This will tell the service to not write the column that you have selected as the key value for your mapping.
2. If that key column is not the column that is causing the issue for identity columns, then you can use the Sink transformation pre-processing SQL option: ```SET IDENTITY_INSERT tbl_content ON```. Then, turn it off with the post-processing SQL property: ```SET IDENTITY_INSERT tbl_content OFF```.
Here are ways to fix that:
The below example is an alter row transformation named `CleanData` that takes an incoming stream `SpecifyUpsertConditions` and creates three alter row conditions. In the previous transformation, a column named `alterRowCondition` is calculated that determines whether or not a row is inserted, updated, or deleted in the database. If the value of the column has a string value that matches the alter row rule, it is assigned that policy.
-In the Data Factory UX, this transformation looks like the below image:
+In the UI, this transformation looks like the below image:
![Alter row example](media/data-flow/alter-row4.png "Alter row example")
data-factory Data Flow Derived Column https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/data-flow-derived-column.md
Title: Derived column transformation in mapping data flow
+ Title: Derived column transformation in mapping data flows
-description: Learn how to transform data at scale in Azure Data Factory with the mapping data flow Derived Column transformation.
+description: Learn how to transform data at scale in Azure Data Factory and Azure Synapse Analytics with the mapping data flow Derived Column transformation.
To reference a local in a derived column, either click on the local from the **E
The below example is a derived column named `CleanData` that takes an incoming stream `MoviesYear` and creates two derived columns. The first derived column replaces column `Rating` with Rating's value as an integer type. The second derived column is a pattern that matches each column whose name starts with 'movies'. For each matched column, it creates a column `movie` that is equal to the value of the matched column prefixed with 'movie_'.
-In the Data Factory UX, this transformation looks like the below image:
+In the UI, this transformation looks like the below image:
![Derive example](media/data-flow/derive-script.png "Derive example")
data-factory Data Flow Sink https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/data-flow-sink.md
Last updated 07/20/2021
After you finish transforming your data, write it into a destination store by using the sink transformation. Every data flow requires at least one sink transformation, but you can write to as many sinks as necessary to complete your transformation flow. To write to additional sinks, create new streams via new branches and conditional splits.
-Each sink transformation is associated with exactly one Azure Data Factory dataset object or linked service. The sink transformation determines the shape and location of the data you want to write to.
+Each sink transformation is associated with exactly one dataset object or linked service. The sink transformation determines the shape and location of the data you want to write to.
## Inline datasets
Mapping data flow follows an extract, load, and transform (ELT) approach and wor
Settings specific to these connectors are located on the **Settings** tab. Information and data flow script examples on these settings are located in the connector documentation.
-Azure Data Factory has access to more than [90 native connectors](connector-overview.md). To write data to those other sources from your data flow, use the Copy Activity to load that data from a supported sink.
+The service has access to more than [90 native connectors](connector-overview.md). To write data to those other sources from your data flow, use the Copy Activity to load that data from a supported sink.
## Sink settings
The following video explains a number of different sink options for text-delimit
![Screenshot that shows Sink settings.](media/data-flow/sink-settings.png "Screenshot that shows Sink settings.")
-**Schema drift**: [Schema drift](concepts-data-flow-schema-drift.md) is the ability of Data Factory to natively handle flexible schemas in your data flows without needing to explicitly define column changes. Enable **Allow schema drift** to write additional columns on top of what's defined in the sink data schema.
+**Schema drift**: [Schema drift](concepts-data-flow-schema-drift.md) is the ability of the service to natively handle flexible schemas in your data flows without needing to explicitly define column changes. Enable **Allow schema drift** to write additional columns on top of what's defined in the sink data schema.
**Validate schema**: If validate schema is selected, the data flow will fail if any column of the incoming source schema isn't found in the source projection, or if the data types don't match. Use this setting to enforce that the source data meets the contract of your defined projection. It's useful in database source scenarios to signal that column names or types have changed.
By default, data is written to multiple sinks in a nondeterministic order. The e
### Sink groups
-You can group sinks together by applying the same order number for a series of sinks. ADF will treat those sinks as groups that can execute in parallel. Options for parallel execution will surface in the pipeline data flow activity.
+You can group sinks together by applying the same order number for a series of sinks. The service will treat those sinks as groups that can execute in parallel. Options for parallel execution will surface in the pipeline data flow activity.
## Error row handling
-When writing to databases, certain rows of data may fail due to constraints set by the destination. By default, a data flow run will fail on the first error it gets. In certain connectors, you can choose to **Continue on error** that allows your data flow to complete even if individual rows have errors. Currently, this capability is only available in Azure SQL Database and Synapse. For more information, see [error row handling in Azure SQL DB](connector-azure-sql-database.md#error-row-handling).
+When writing to databases, certain rows of data may fail due to constraints set by the destination. By default, a data flow run will fail on the first error it gets. In certain connectors, you can choose to **Continue on error** that allows your data flow to complete even if individual rows have errors. Currently, this capability is only available in Azure SQL Database and Azure Synapse. For more information, see [error row handling in Azure SQL DB](connector-azure-sql-database.md#error-row-handling).
Below is a video tutorial on how to use database error row handling automatically in your sink transformation.
data-factory Data Flow Source https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/data-flow-source.md
Title: Source transformation in mapping data flow
-description: Learn how to set up a source transformation in mapping data flow.
+description: Learn how to set up a source transformation in a mapping data flow in Azure Data Factory or Azure Synapse Analytics pipelines.
Mapping data flow follows an extract, load, and transform (ELT) approach and wor
Settings specific to these connectors are located on the **Source options** tab. Information and data flow script examples on these settings are located in the connector documentation.
-Azure Data Factory has access to more than [90 native connectors](connector-overview.md). To include data from those other sources in your data flow, use the Copy Activity to load that data into one of the supported staging areas.
+Azure Data Factory and Synapse pipelines have access to more than [90 native connectors](connector-overview.md). To include data from those other sources in your data flow, use the Copy Activity to load that data into one of the supported staging areas.
## Source settings
Development values for dataset parameters can be configured in [debug settings](
**Test connection**: Test whether or not the data flow's Spark service can successfully connect to the linked service used in your source dataset. Debug mode must be on for this feature to be enabled.
-**Schema drift**: [Schema drift](concepts-data-flow-schema-drift.md) is the ability of Data Factory to natively handle flexible schemas in your data flows without needing to explicitly define column changes.
+**Schema drift**: [Schema drift](concepts-data-flow-schema-drift.md) is the ability of the service to natively handle flexible schemas in your data flows without needing to explicitly define column changes.
* Select the **Allow schema drift** check box if the source columns will change often. This setting allows all incoming source fields to flow through the transformations to the sink.
-* Selecting **Infer drifted column types** instructs Data Factory to detect and define data types for each new column discovered. With this feature turned off, all drifted columns will be of type string.
+* Selecting **Infer drifted column types** instructs the service to detect and define data types for each new column discovered. With this feature turned off, all drifted columns will be of type string.
**Validate schema:** If **Validate schema** is selected, the data flow will fail to run if the incoming source data doesn't match the defined schema of the dataset.
Like schemas in datasets, the projection in a source defines the data columns, t
![Screenshot that shows settings on the Projection tab.](media/data-flow/source3.png "Screenshot that shows settings on the Projection tab.")
-If your text file has no defined schema, select **Detect data type** so that Data Factory will sample and infer the data types. Select **Define default format** to autodetect the default data formats.
+If your text file has no defined schema, select **Detect data type** so that the service will sample and infer the data types. Select **Define default format** to autodetect the default data formats.
**Reset schema** resets the projection to what is defined in the referenced dataset.
Importing schema is useful in datasets like Avro and Azure Cosmos DB that suppor
The **Optimize** tab allows for editing of partition information at each transformation step. In most cases, **Use current partitioning** will optimize for the ideal partitioning structure for a source.
-If you're reading from an Azure SQL Database source, custom **Source** partitioning will likely read data the fastest. Data Factory will read large queries by making connections to your database in parallel. This source partitioning can be done on a column or by using a query.
+If you're reading from an Azure SQL Database source, custom **Source** partitioning will likely read data the fastest. The service will read large queries by making connections to your database in parallel. This source partitioning can be done on a column or by using a query.
![Screenshot that shows the Source partition settings.](media/data-flow/sourcepart3.png "Screenshot that shows the Source partition settings.")
data-factory Delete Activity https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/delete-activity.md
Title: Delete Activity in Azure Data Factory
-description: Learn how to delete files in various file stores with the Delete Activity in Azure Data Factory.
+description: Learn how to delete files in various file stores with the Delete Activity in Azure Data Factory and Azure Synapse Analytics.
Last updated 08/12/2020
-# Delete Activity in Azure Data Factory
+# Delete Activity in Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)] - You can use the Delete Activity in Azure Data Factory to delete files or folders from on-premises storage stores or cloud storage stores. Use this activity to clean up or archive files when they are no longer needed. > [!WARNING]
Here are some recommendations for using the Delete activity:
- Back up your files before deleting them with the Delete activity in case you need to restore them in the future. -- Make sure that Data Factory has write permissions to delete folders or files from the storage store.
+- Make sure that the service has write permissions to delete folders or files from the storage store.
- Make sure you are not deleting files that are being written at the same time.
Now you are using the Delete activity to delete folder or files by the combinati
### Periodically clean up the time-partitioned folder or files
-You can create a pipeline to periodically clean up the time partitioned folder or files. For example, the folder structure is similar as: `/mycontainer/2018/12/14/*.csv`. You can leverage ADF system variable from schedule trigger to identify which folder or files should be deleted in each pipeline run.
+You can create a pipeline to periodically clean up the time partitioned folder or files. For example, the folder structure is similar as: `/mycontainer/2018/12/14/*.csv`. You can leverage the service system variable from schedule trigger to identify which folder or files should be deleted in each pipeline run.
#### Sample pipeline
You can also get the template to move files from [here](solution-template-move-f
## Next steps
-Learn more about moving files in Azure Data Factory.
+Learn more about moving files in Azure Data Factory and Synapse pipelines.
-- [Copy Data tool in Azure Data Factory](copy-data-tool.md)
+- [Copy Data tool](copy-data-tool.md)
data-factory Format Common Data Model https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/format-common-data-model.md
When mapping data flow columns to entity properties in the Sink transformation,
2. Find the partitions.Location property 3. Change "blob.core.windows.net" to "dfs.core.windows.net" 4. Fix any "%2F" encoding in the URL to "/"
-5. If using ADF Data Flows, Special characters in the partition file path must be replaced with alpha-numeric values, or switch to Synapse Data Flows
+5. If using ADF Data Flows, Special characters in the partition file path must be replaced with alpha-numeric values, or switch to Azure Synapse Data Flows
### CDM source data flow script example
data-factory Format Delimited Text https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/format-delimited-text.md
Title: Delimited text format in Azure Data Factory
-description: 'This topic describes how to deal with delimited text format in Azure Data Factory.'
+description: This topic describes how to deal with delimited text format in Azure Data Factory and Azure Synapse Analytics.
Last updated 03/23/2021
-# Delimited text format in Azure Data Factory
+# Delimited text format in Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
Supported **delimited text read settings** under `formatSettings`:
| type | The type of formatSettings must be set to **DelimitedTextReadSettings**. | Yes | | skipLineCount | Indicates the number of **non-empty** rows to skip when reading data from input files. <br>If both skipLineCount and firstRowAsHeader are specified, the lines are skipped first and then the header information is read from the input file. | No | | compressionProperties | A group of properties on how to decompress data for a given compression codec. | No |
-| preserveZipFileNameAsFolder<br>(*under `compressionProperties`->`type` as `ZipDeflateReadSettings`*) | Applies when input dataset is configured with **ZipDeflate** compression. Indicates whether to preserve the source zip file name as folder structure during copy.<br>- When set to **true (default)**, Data Factory writes unzipped files to `<path specified in dataset>/<folder named as source zip file>/`.<br>- When set to **false**, Data Factory writes unzipped files directly to `<path specified in dataset>`. Make sure you don't have duplicated file names in different source zip files to avoid racing or unexpected behavior. | No |
-| preserveCompressionFileNameAsFolder<br>(*under `compressionProperties`->`type` as `TarGZipReadSettings` or `TarReadSettings`*) | Applies when input dataset is configured with **TarGzip**/**Tar** compression. Indicates whether to preserve the source compressed file name as folder structure during copy.<br>- When set to **true (default)**, Data Factory writes decompressed files to `<path specified in dataset>/<folder named as source compressed file>/`. <br>- When set to **false**, Data Factory writes decompressed files directly to `<path specified in dataset>`. Make sure you don't have duplicated file names in different source files to avoid racing or unexpected behavior. | No |
+| preserveZipFileNameAsFolder<br>(*under `compressionProperties`->`type` as `ZipDeflateReadSettings`*) | Applies when input dataset is configured with **ZipDeflate** compression. Indicates whether to preserve the source zip file name as folder structure during copy.<br>- When set to **true (default)**, the service writes unzipped files to `<path specified in dataset>/<folder named as source zip file>/`.<br>- When set to **false**, the service writes unzipped files directly to `<path specified in dataset>`. Make sure you don't have duplicated file names in different source zip files to avoid racing or unexpected behavior. | No |
+| preserveCompressionFileNameAsFolder<br>(*under `compressionProperties`->`type` as `TarGZipReadSettings` or `TarReadSettings`*) | Applies when input dataset is configured with **TarGzip**/**Tar** compression. Indicates whether to preserve the source compressed file name as folder structure during copy.<br>- When set to **true (default)**, the service writes decompressed files to `<path specified in dataset>/<folder named as source compressed file>/`. <br>- When set to **false**, the service writes decompressed files directly to `<path specified in dataset>`. Make sure you don't have duplicated file names in different source files to avoid racing or unexpected behavior. | No |
```json "activities": [
data-factory Format Excel https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/format-excel.md
Title: Excel format in Azure Data Factory
-description: 'This topic describes how to deal with Excel format in Azure Data Factory.'
+description: This topic describes how to deal with Excel format in Azure Data Factory and Azure Synapse Analytics.
Last updated 12/08/2020
-# Excel format in Azure Data Factory
+# Excel file format in Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-Follow this article when you want to **parse the Excel files**. Azure Data Factory supports both ".xls" and ".xlsx".
+Follow this article when you want to **parse the Excel files**. The service supports both ".xls" and ".xlsx".
Excel format is supported for the following connectors: [Amazon S3](connector-amazon-simple-storage-service.md), [Amazon S3 Compatible Storage](connector-amazon-s3-compatible-storage.md), [Azure Blob](connector-azure-blob-storage.md), [Azure Data Lake Storage Gen1](connector-azure-data-lake-store.md), [Azure Data Lake Storage Gen2](connector-azure-data-lake-storage.md), [Azure File Storage](connector-azure-file-storage.md), [File System](connector-file-system.md), [FTP](connector-ftp.md), [Google Cloud Storage](connector-google-cloud-storage.md), [HDFS](connector-hdfs.md), [HTTP](connector-http.md), [Oracle Cloud Storage](connector-oracle-cloud-storage.md) and [SFTP](connector-sftp.md). It is supported as source but not sink.
data-factory Format Json https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/format-json.md
Title: JSON format in Azure Data Factory
+ Title: JSON format
-description: 'This topic describes how to deal with JSON format in Azure Data Factory.'
+description: This topic describes how to deal with JSON format in Azure Data Factory and Azure Synapse Analytics pipelines.
Last updated 10/29/2020
-# JSON format in Azure Data Factory
+# JSON format in Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
Supported **JSON read settings** under `formatSettings`:
| - | | -- | | type | The type of formatSettings must be set to **JsonReadSettings**. | Yes | | compressionProperties | A group of properties on how to decompress data for a given compression codec. | No |
-| preserveZipFileNameAsFolder<br>(*under `compressionProperties`->`type` as `ZipDeflateReadSettings`*) | Applies when input dataset is configured with **ZipDeflate** compression. Indicates whether to preserve the source zip file name as folder structure during copy.<br>- When set to **true (default)**, Data Factory writes unzipped files to `<path specified in dataset>/<folder named as source zip file>/`.<br>- When set to **false**, Data Factory writes unzipped files directly to `<path specified in dataset>`. Make sure you don't have duplicated file names in different source zip files to avoid racing or unexpected behavior. | No |
-| preserveCompressionFileNameAsFolder<br>(*under `compressionProperties`->`type` as `TarGZipReadSettings` or `TarReadSettings`*) | Applies when input dataset is configured with **TarGzip**/**Tar** compression. Indicates whether to preserve the source compressed file name as folder structure during copy.<br>- When set to **true (default)**, Data Factory writes decompressed files to `<path specified in dataset>/<folder named as source compressed file>/`. <br>- When set to **false**, Data Factory writes decompressed files directly to `<path specified in dataset>`. Make sure you don't have duplicated file names in different source files to avoid racing or unexpected behavior. | No |
+| preserveZipFileNameAsFolder<br>(*under `compressionProperties`->`type` as `ZipDeflateReadSettings`*) | Applies when input dataset is configured with **ZipDeflate** compression. Indicates whether to preserve the source zip file name as folder structure during copy.<br>- When set to **true (default)**, the service writes unzipped files to `<path specified in dataset>/<folder named as source zip file>/`.<br>- When set to **false**, the service writes unzipped files directly to `<path specified in dataset>`. Make sure you don't have duplicated file names in different source zip files to avoid racing or unexpected behavior. | No |
+| preserveCompressionFileNameAsFolder<br>(*under `compressionProperties`->`type` as `TarGZipReadSettings` or `TarReadSettings`*) | Applies when input dataset is configured with **TarGzip**/**Tar** compression. Indicates whether to preserve the source compressed file name as folder structure during copy.<br>- When set to **true (default)**, the service writes decompressed files to `<path specified in dataset>/<folder named as source compressed file>/`. <br>- When set to **false**, the service writes decompressed files directly to `<path specified in dataset>`. Make sure you don't have duplicated file names in different source files to avoid racing or unexpected behavior. | No |
### JSON as sink
data-factory Format Parquet https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/format-parquet.md
Title: Parquet format in Azure Data Factory
+ Title: Parquet format
-description: 'This topic describes how to deal with Parquet format in Azure Data Factory.'
+description: This topic describes how to deal with Parquet format in Azure Data Factory and Azure Synapse Analytics pipelines.
Last updated 09/27/2020
-# Parquet format in Azure Data Factory
+# Parquet format in Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)] Follow this article when you want to **parse the Parquet files or write the data into Parquet format**.
Parquet complex data types (e.g. MAP, LIST, STRUCT) are currently supported only
> [!IMPORTANT] > For copy empowered by Self-hosted Integration Runtime e.g. between on-premises and cloud data stores, if you are not copying Parquet files **as-is**, you need to install the **64-bit JRE 8 (Java Runtime Environment) or OpenJDK** and **Microsoft Visual C++ 2010 Redistributable Package** on your IR machine. Check the following paragraph with more details.
-For copy running on Self-hosted IR with Parquet file serialization/deserialization, ADF locates the Java runtime by firstly checking the registry *`(SOFTWARE\JavaSoft\Java Runtime Environment\{Current Version}\JavaHome)`* for JRE, if not found, secondly checking system variable *`JAVA_HOME`* for OpenJDK.
+For copy running on Self-hosted IR with Parquet file serialization/deserialization, the service locates the Java runtime by firstly checking the registry *`(SOFTWARE\JavaSoft\Java Runtime Environment\{Current Version}\JavaHome)`* for JRE, if not found, secondly checking system variable *`JAVA_HOME`* for OpenJDK.
- **To use JRE**: The 64-bit IR requires 64-bit JRE. You can find it from [here](https://go.microsoft.com/fwlink/?LinkId=808605). - **To use OpenJDK**: It's supported since IR version 3.13. Package the jvm.dll with all other required assemblies of OpenJDK into Self-hosted IR machine, and set system environment variable JAVA_HOME accordingly.
For copy running on Self-hosted IR with Parquet file serialization/deserializati
![Set JVM heap size on Self-hosted IR](./media/supported-file-formats-and-compression-codecs/set-jvm-heap-size-on-selfhosted-ir.png)
-Example: set variable `_JAVA_OPTIONS` with value `-Xms256m -Xmx16g`. The flag `Xms` specifies the initial memory allocation pool for a Java Virtual Machine (JVM), while `Xmx` specifies the maximum memory allocation pool. This means that JVM will be started with `Xms` amount of memory and will be able to use a maximum of `Xmx` amount of memory. By default, ADF use min 64 MB and max 1G.
+Example: set variable `_JAVA_OPTIONS` with value `-Xms256m -Xmx16g`. The flag `Xms` specifies the initial memory allocation pool for a Java Virtual Machine (JVM), while `Xmx` specifies the maximum memory allocation pool. This means that JVM will be started with `Xms` amount of memory and will be able to use a maximum of `Xmx` amount of memory. By default, the service uses min 64 MB and max 1G.
## Next steps
data-factory How To Create Schedule Trigger https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/how-to-create-schedule-trigger.md
Title: Create schedule triggers in Azure Data Factory
+ Title: Create schedule triggers
-description: Learn how to create a trigger in Azure Data Factory that runs a pipeline on a schedule.
+description: Learn how to create a trigger in Azure Data Factory or Azure Synapse Analytics that runs a pipeline on a schedule.
When creating a schedule trigger, you specify a schedule (start date, recurrence
The following sections provide steps to create a schedule trigger in different ways.
-## Data Factory UI
+## UI Experience
You can create a **schedule trigger** to schedule a pipeline to run periodically (hourly, daily, etc.). > [!NOTE] > For a complete walkthrough of creating a pipeline and a schedule trigger, which associates the trigger with the pipeline, and runs and monitors the pipeline, see [Quickstart: create a data factory using Data Factory UI](quickstart-create-data-factory-portal.md).
-1. Switch to the **Edit** tab, shown with a pencil symbol.
+1. Switch to the **Edit** tab in Data Factory or the Integrate tab in Azure Synapse.
+ # [Azure Data Factory](#tab/data-factory)
![Switch to Edit tab](./media/how-to-create-schedule-trigger/switch-edit-tab.png)
-1. Select **Trigger** on the menu, then select **New/Edit**.
+ # [Azure Synapse](#tab/synapse-analytics)
+ ![Switch to Edit tab](./media/how-to-create-schedule-trigger/switch-edit-tab-synapse.png)
++
+
+2. Select **Trigger** on the menu, then select **New/Edit**.
![New trigger menu](./media/how-to-create-schedule-trigger/new-trigger-menu.png)
You can create a **schedule trigger** to schedule a pipeline to run periodically
1. Specify **Recurrence** for the trigger. Select one of the values from the drop-down list (Every minute, Hourly, Daily, Weekly, and Monthly). Enter the multiplier in the text box. For example, if you want the trigger to run once for every 15 minutes, you select **Every Minute**, and enter **15** in the text box. 1. In the **Recurrence**, if you choose "Day(s), Week(s) or Month(s)" from the drop-down, you can find "Advanced recurrence options". :::image type="content" source="./media/how-to-create-schedule-trigger/advanced.png" alt-text="Advanced recurrence options of Day(s), Week(s) or Month(s)":::
- 1. To specify an end date time, select **Specify an End Date**, and specify _Ends On_, then select **OK**. There is a cost associated with each pipeline run. If you are testing, you may want to ensure that the pipeline is triggered only a couple of times. However, ensure that there is enough time for the pipeline to run between the publish time and the end time. The trigger comes into effect only after you publish the solution to Data Factory, not when you save the trigger in the UI.
+ 1. To specify an end date time, select **Specify an End Date**, and specify _Ends On_, then select **OK**. There is a cost associated with each pipeline run. If you are testing, you may want to ensure that the pipeline is triggered only a couple of times. However, ensure that there is enough time for the pipeline to run between the publish time and the end time. The trigger comes into effect only after you publish the solution, not when you save the trigger in the UI.
![Trigger settings](./media/how-to-create-schedule-trigger/trigger-settings-01.png)
You can create a **schedule trigger** to schedule a pipeline to run periodically
![Trigger settings - Finish button](./media/how-to-create-schedule-trigger/new-trigger-finish.png)
-1. Select **Publish all** to publish the changes to Data Factory. Until you publish the changes to Data Factory, the trigger doesn't start triggering the pipeline runs.
+1. Select **Publish all** to publish the changes. Until you publish the changes, the trigger doesn't start triggering the pipeline runs.
![Publish button](./media/how-to-create-schedule-trigger/publish-2.png) 1. Switch to the **Pipeline runs** tab on the left, then select **Refresh** to refresh the list. You will see the pipeline runs triggered by the scheduled trigger. Notice the values in the **Triggered By** column. If you use the **Trigger Now** option, you will see the manual trigger run in the list.
+ # [Azure Data Factory](#tab/data-factory)
+ ![Monitor triggered runs](./media/how-to-create-schedule-trigger/monitor-triggered-runs.png)
-1. Switch to the **Trigger Runs** \ **Schedule** view.
+ # [Azure Synapse](#tab/synapse-analytics)
+ ![Monitor triggered runs](./media/how-to-create-schedule-trigger/monitor-triggered-runs-synapse.png)
+
++
+9. Switch to the **Trigger Runs** \ **Schedule** view.
+
+ # [Azure Data Factory](#tab/data-factory)
![Monitor trigger runs](./media/how-to-create-schedule-trigger/monitor-trigger-runs.png)
+ # [Azure Synapse](#tab/synapse-analytics)
+ ![Monitor trigger runs](./media/how-to-create-schedule-trigger/monitor-trigger-runs-synapse.png)
+
++ ## Azure PowerShell [!INCLUDE [updated-for-az](../../includes/updated-for-az.md)]
You can use an Azure Resource Manager template to create a trigger. For step-by-
## Pass the trigger start time to a pipeline
-Azure Data Factory version 1 supports reading or writing partitioned data by using the system variables: **SliceStart**, **SliceEnd**, **WindowStart**, and **WindowEnd**. In the current version of Azure Data Factory, you can achieve this behavior by using a pipeline parameter. The start time and scheduled time for the trigger are set as the value for the pipeline parameter. In the following example, the scheduled time for the trigger is passed as a value to the pipeline **scheduledRunTime** parameter:
+Azure Data Factory version 1 supports reading or writing partitioned data by using the system variables: **SliceStart**, **SliceEnd**, **WindowStart**, and **WindowEnd**. In the current version of Azure Data Factory and Synapse pipelines, you can achieve this behavior by using a pipeline parameter. The start time and scheduled time for the trigger are set as the value for the pipeline parameter. In the following example, the scheduled time for the trigger is passed as a value to the pipeline **scheduledRunTime** parameter:
```json "parameters": {
Here are some of time zones supported for Schedule triggers:
| India Standard Time (IST) | +5:30 | `India Standard Time` | No | `'yyyy-MM-ddTHH:mm:ss'` | | China Standard Time | +8 | `China Standard Time` | No | `'yyyy-MM-ddTHH:mm:ss'` |
-This list is incomplete. For complete list of time zone options, explore in Data Factory portal [Trigger creation page](#data-factory-ui)
+This list is incomplete. For complete list of time zone options, explore in the portal [Trigger creation page](#ui-experience)
### startTime property The following table shows you how the **startTime** property controls a trigger run:
data-factory How To Create Tumbling Window Trigger https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/how-to-create-tumbling-window-trigger.md
Title: Create tumbling window triggers in Azure Data Factory
+ Title: Create tumbling window triggers
-description: Learn how to create a trigger in Azure Data Factory that runs a pipeline on a tumbling window.
+description: Learn how to create a trigger in Azure Data Factory or Azure Synapse Analytics that runs a pipeline on a tumbling window.
Previously updated : 10/25/2020 Last updated : 07/26/2021 # Create a trigger that runs a pipeline on a tumbling window
This article provides steps to create, start, and monitor a tumbling window trig
Tumbling window triggers are a type of trigger that fires at a periodic time interval from a specified start time, while retaining state. Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals. A tumbling window trigger has a one-to-one relationship with a pipeline and can only reference a singular pipeline. Tumbling window trigger is a more heavy weight alternative for schedule trigger offering a suite of features for complex scenarios([dependency on other tumbling window triggers](#tumbling-window-trigger-dependency), [rerunning a failed job](tumbling-window-trigger-dependency.md#monitor-dependencies) and [set user retry for pipelines](#user-assigned-retries-of-pipelines)). To further understand the difference between schedule trigger and tumbling window trigger, please visit [here](concepts-pipeline-execution-triggers.md#trigger-type-comparison).
-## Data Factory UI
+## UI Experience
-1. To create a tumbling window trigger in the Data Factory UI, select the **Triggers** tab, and then select **New**.
+1. To create a tumbling window trigger in the UI, select the **Triggers** tab, and then select **New**.
1. After the trigger configuration pane opens, select **Tumbling Window**, and then define your tumbling window trigger properties. 1. When you're done, select **Save**.
-![Create a tumbling window trigger in the Azure portal](media/how-to-create-tumbling-window-trigger/create-tumbling-window-trigger.png)
+# [Azure Data Factory](#tab/data-factory)
+
+# [Azure Synapse](#tab/synapse-analytics)
++ ## Tumbling window trigger type properties
You can cancel runs for a tumbling window trigger, if the specific window is in
* If the window is in **Running** state, cancel the associated _Pipeline Run_, and the trigger run will be marked as _Canceled_ afterwards * If the window is in **Waiting** or **Waiting on Dependency** state, you can cancel the window from Monitoring:
-![Cancel a tumbling window trigger from Monitoring page](media/how-to-create-tumbling-window-trigger/cancel-tumbling-window-trigger.png)
+# [Azure Data Factory](#tab/data-factory)
++
+# [Azure Synapse](#tab/synapse-analytics)
+++ You can also rerun a canceled window. The rerun will take the _latest_ published definitions of the trigger, and dependencies for the specified window will be _re-evaluated_ upon rerun
-![Rerun a tumbling window trigger for previously canceled runs](media/how-to-create-tumbling-window-trigger/rerun-tumbling-window-trigger.png)
+# [Azure Data Factory](#tab/data-factory)
++
+# [Azure Synapse](#tab/synapse-analytics)
+++ ## Sample for Azure PowerShell
data-factory How To Expression Language Functions https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/how-to-expression-language-functions.md
In this document, we will primarily focus on learning fundamental concepts with
## Azure data factory UI and parameters
-If you are new to Azure data factory parameter usage in ADF user interface, please review [Data factory UI for linked services with parameters](./parameterize-linked-services.md#data-factory-ui) and [Data factory UI for metadata driven pipeline with parameters](./how-to-use-trigger-parameterization.md#data-factory-ui) for visual explanation.
+If you are new to Azure data factory parameter usage in ADF user interface, please review [Data factory UI for linked services with parameters](./parameterize-linked-services.md#ui-experience) and [Data factory UI for metadata driven pipeline with parameters](./how-to-use-trigger-parameterization.md#data-factory-ui) for visual explanation.
## Parameter and expression concepts
data-factory Load Azure Data Lake Storage Gen2 From Gen1 https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/load-azure-data-lake-storage-gen2-from-gen1.md
This article shows you how to use the Data Factory copy data tool to copy data f
To assess upgrading from Azure Data Lake Storage Gen1 to Azure Data Lake Storage Gen2 in general, see [Upgrade your big data analytics solutions from Azure Data Lake Storage Gen1 to Azure Data Lake Storage Gen2](../storage/blobs/data-lake-storage-migrate-gen1-to-gen2.md). The following sections introduce best practices for using Data Factory for a data upgrade from Data Lake Storage Gen1 to Data Lake Storage Gen2.
-### Data partition for historical data copy
+### Historical data copy
-- If your total data size in Data Lake Storage Gen1 is less than 30 TB and the number of files is less than 1 million, you can copy all data in a single copy activity run.-- If you have a larger amount of data to copy, or you want the flexibility to manage data migration in batches and make each of them complete within a specific time frame, partition the data. Partitioning also reduces the risk of any unexpected issue.
+#### Performance tuning by proof-of-concept
Use a proof of concept to verify the end-to-end solution and test the copy throughput in your environment. Major proof-of-concept steps:
-1. Create one Data Factory pipeline with a single copy activity to copy several TBs of data from Data Lake Storage Gen1 to Data Lake Storage Gen2 to get a copy performance baseline. Start with [data integration units (DIUs)](copy-activity-performance-features.md#data-integration-units) as 128.
-2. Based on the copy throughput you get in step 1, calculate the estimated time that's required for the entire data migration.
-3. (Optional) Create a control table and define the file filter to partition the files to be migrated. The way to partition the files is to:
+1. Create one Data Factory pipeline with a single copy activity to copy several TBs of data from Data Lake Storage Gen1 to Data Lake Storage Gen2 to get a copy performance baseline. Start with [data integration units (DIUs)](copy-activity-performance-features.md#data-integration-units) as 128. The [Parallel copy](copy-activity-performance-features.md#parallel-copy) is suggested to be set as **empty (default)**.
+2. Based on the copy throughput you get in step 1, calculate the estimated time that's required for the entire data migration. If the copy throughput is not good for you, identify and resolve the performance bottlenecks by following the [performance tuning steps](copy-activity-performance.md#performance-tuning-steps).
+3. If you have maximized the performance of a single copy activity, but have not yet achieved the throughput upper limits of your environment, you can run multiple copy activities in parallel. Each copy activity can be configured to copy one partition at a time, so that multiple copy activities can copy data from single Data Lake Storage Gen1 account cocurrently. The way to partition the files is to use **name range- listAfter/listBefore** in [copy activity property](connector-azure-data-lake-store.md#copy-activity-properties).
- - Partition by folder name or folder name with a wildcard filter. We recommend this method.
- - Partition by a file's last modified time.
+If your total data size in Data Lake Storage Gen1 is less than 30 TB and the number of files is less than 1 million, you can copy all data in a single copy activity run. If you have a larger amount of data to copy, or you want the flexibility to manage data migration in batches and make each of them complete within a specific time frame, partition the data. Partitioning also reduces the risk of any unexpected issue.
-### Network bandwidth and storage I/O
-You can control the concurrency of Data Factory copy jobs that read data from Data Lake Storage Gen1 and write data to Data Lake Storage Gen2. In this way, you can manage the use on that storage I/O to avoid affecting the normal business work on Data Lake Storage Gen1 during the migration.
+#### Network bandwidth and storage I/O
-### Permissions
-
-In Data Factory, the [Data Lake Storage Gen1 connector](connector-azure-data-lake-store.md) supports service principal and managed identity for Azure resource authentications. The [Data Lake Storage Gen2 connector](connector-azure-data-lake-storage.md) supports account key, service principal, and managed identity for Azure resource authentications. To make Data Factory able to navigate and copy all the files or access control lists (ACLs) you need, grant high enough permissions for the account you provide to access, read, or write all files and set ACLs if you choose to. Grant it a super-user or owner role during the migration period.
+If you see significant number of throttling errors from [copy activity monitoring](copy-activity-monitoring.md#monitor-visually), it indicates you have reached the capacity limit of your storage account. ADF will retry automatically to overcome each throttling error to make sure there will not be any data lost, but too many retries impact your copy throughput as well. In such case, you are encouraged to reduce the number of copy activities running cocurrently to avoid significant amounts of throttling errors. If you have been using single copy activity to copy data, then you are encouraged to reduce the number of [data integration units (DIUs)](copy-activity-performance-features.md#data-integration-units).
-### Preserve ACLs from Data Lake Storage Gen1
-
-If you want to replicate the ACLs along with data files when you upgrade from Data Lake Storage Gen1 to Data Lake Storage Gen2, see [Preserve ACLs from Data Lake Storage Gen1](connector-azure-data-lake-storage.md#preserve-acls).
### Incremental copy You can use several approaches to load only the new or updated files from Data Lake Storage Gen1: - Load new or updated files by time partitioned folder or file name. An example is /2019/05/13/*.-- Load new or updated files by LastModifiedDate.
+- Load new or updated files by LastModifiedDate. If you are copying large amounts of files, do partitions first in order to avoid low copy throughput result from single copy activity scanning your entire Data Lake Storage Gen1 account to identify new files.
- Identify new or updated files by any third-party tool or solution. Then pass the file or folder name to the Data Factory pipeline via parameter or a table or file. The proper frequency to do incremental load depends on the total number of files in Azure Data Lake Storage Gen1 and the volume of new or updated files to be loaded every time. +
+### Preserve ACLs
+
+If you want to replicate the ACLs along with data files when you upgrade from Data Lake Storage Gen1 to Data Lake Storage Gen2, see [Preserve ACLs from Data Lake Storage Gen1](connector-azure-data-lake-storage.md#preserve-acls).
+
+### Permissions
+
+In Data Factory, the [Data Lake Storage Gen1 connector](connector-azure-data-lake-store.md) supports service principal and managed identity for Azure resource authentications. The [Data Lake Storage Gen2 connector](connector-azure-data-lake-storage.md) supports account key, service principal, and managed identity for Azure resource authentications. To make Data Factory able to navigate and copy all the files or access control lists (ACLs) you need, grant high enough permissions for the account you provide to access, read, or write all files and set ACLs if you choose to. Grant it a super-user or owner role during the migration period.
++ ## Next steps > [!div class="nextstepaction"]
data-factory Load Azure Sql Data Warehouse https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/load-azure-sql-data-warehouse.md
Title: Load data into Azure Synapse Analytics
-description: Use Azure Data Factory to copy data into Azure Synapse Analytics
+description: Use Azure Data Factory or a Synapse pipeline to copy data into Azure Synapse Analytics.
Last updated 07/28/2021
-# Load data into Azure Synapse Analytics by using Azure Data Factory
+# Load data into Azure Synapse Analytics using Azure Data Factory or a Synapse pipeline
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)] [Azure Synapse Analytics](../synapse-analytics/sql-data-warehouse/sql-data-warehouse-overview-what-is.md) is a cloud-based, scale-out database that's capable of processing massive volumes of data, both relational and non-relational. Azure Synapse Analytics is built on the massively parallel processing (MPP) architecture that's optimized for enterprise data warehouse workloads. It offers cloud elasticity with the flexibility to scale storage and compute independently.
-Getting started with Azure Synapse Analytics is now easier than ever when you use Azure Data Factory. Azure Data Factory is a fully managed cloud-based data integration service. You can use the service to populate an Azure Synapse Analytics with data from your existing system and save time when building your analytics solutions.
+Getting started with Azure Synapse Analytics is now easier than ever. Azure Data Factory and its equivalent pipelines feature within Azure Synapse itself provide a fully managed cloud-based data integration service. You can use the service to populate an Azure Synapse Analytics with data from your existing system and save time when building your analytics solutions.
-Azure Data Factory offers the following benefits for loading data into Azure Synapse Analytics:
+Azure Data Factory and Synapse pipelines offer the following benefits for loading data into Azure Synapse Analytics:
* **Easy to set up**: An intuitive 5-step wizard with no scripting required. * **Rich data store support**: Built-in support for a rich set of on-premises and cloud-based data stores. For a detailed list, see the table of [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats). * **Secure and compliant**: Data is transferred over HTTPS or ExpressRoute. The global service presence ensures that your data never leaves the geographical boundary. * **Unparalleled performance by using PolyBase**: Polybase is the most efficient way to move data into Azure Synapse Analytics. Use the staging blob feature to achieve high load speeds from all types of data stores, including Azure Blob storage and Data Lake Store. (Polybase supports Azure Blob storage and Azure Data Lake Store by default.) For details, see [Copy activity performance](copy-activity-performance.md).
-This article shows you how to use the Data Factory Copy Data tool to _load data from Azure SQL Database into Azure Synapse Analytics_. You can follow similar steps to copy data from other types of data stores.
+This article shows you how to use the Copy Data tool to _load data from Azure SQL Database into Azure Synapse Analytics_. You can follow similar steps to copy data from other types of data stores.
> [!NOTE]
-> For more information, see [Copy data to or from Azure Synapse Analytics by using Azure Data Factory](connector-azure-sql-data-warehouse.md).
+> For more information, see [Copy data to or from Azure Synapse Analytics](connector-azure-sql-data-warehouse.md).
## Prerequisites
This article shows you how to use the Data Factory Copy Data tool to _load data
## Create a data factory
+> [!NOTE]
+> You can skip the creation of a new data factory if you wish to use the pipelines feature within your existing Synapse workspace to load the data. Azure Synapse embeds the functionality of Azure Data Factory within its pipelines feature.
+ 1. On the left menu, select **Create a resource** > **Data + Analytics** > **Data Factory**: 2. On the **New data factory** page, provide values for following items:
This article shows you how to use the Data Factory Copy Data tool to _load data
## Load data into Azure Synapse Analytics
-1. In the home page of Azure Data Factory, select the **Ingest** tile to launch the Copy Data tool.
+1. In the home page of Azure Data Factory or Azure Synapse workspace, select the **Ingest** tile to launch the Copy Data tool. Then choose the **Built-in copy task**.
2. In the **Properties** page, choose **Built-in copy task** under **Task type**, then select **Next**.
This article shows you how to use the Data Factory Copy Data tool to _load data
1. In the **Destination data store** page, select the newly created connection as sink in the **Connection** section.
- 1. In the section of table mapping, review the content, and select **Next**. An intelligent table mapping displays. The source tables are mapped to the destination tables based on the table names. If a source table doesn't exist in the destination, Azure Data Factory creates a destination table with the same name by default. You can also map a source table to an existing destination table.
+6. In the **Table mapping** section, review the content and select **Next**. An intelligent table mapping displays. The source tables are mapped to the destination tables based on the table names. If a source table doesn't exist in the destination, the service creates a destination table with the same name by default. You can also map a source table to an existing destination table.
![Screenshot showing the configuration of 'Destination data store' page.](./media/load-azure-sql-data-warehouse/destination-data-store-page.png)
-6. In the **Column mapping** page, review the content, and select **Next**. The intelligent table mapping is based on the column name. If you let Data Factory automatically create the tables, data type conversion can occur when there are incompatibilities between the source and destination stores. If there's an unsupported data type conversion between the source and destination column, you see an error message next to the corresponding table.
+1. In the **Column mapping** page, review the content, and select **Next**. The intelligent table mapping is based on the column name. If you let the service automatically create the tables, data type conversion can occur when there are incompatibilities between the source and destination stores. If there's an unsupported data type conversion between the source and destination column, you see an error message next to the corresponding table.
![Column mapping page](./media/load-azure-sql-data-warehouse/schema-mapping.png)
-7. In the **Settings** page, complete the following steps:
+1. In the **Settings** page, complete the following steps:
1. Specify **CopyFromSQLToSQLDW** for the **Task name** field. 1. In **Staging settings** section, select **+ New** to new a staging storage. The storage is used for staging the data before it loads into Azure Synapse Analytics by using PolyBase. After the copy is complete, the interim data in Azure Blob Storage is automatically cleaned up.
This article shows you how to use the Data Factory Copy Data tool to _load data
10. Notice that the **Monitor** tab on the left is automatically selected. When the pipeline run completes successfully, select the **CopyFromSQLToSQLDW** link under the **Pipeline name** column to view activity run details or to rerun the pipeline.
- [![Monitor pipeline runs](./media/load-azure-sql-data-warehouse/pipeline-monitoring.png)](./media/load-azure-sql-data-warehouse/pipeline-monitoring.png#lightbox)
+ # [Azure Data Factory](#tab/data-factory)
+ :::image type="content" source="./media/load-azure-sql-data-warehouse/pipeline-monitoring.png" alt-text="Monitor pipeline runs":::
+
+ # [Azure Synapse](#tab/synapse-analytics)
+ :::image type="content" source="./media/load-azure-sql-data-warehouse/pipeline-monitoring-synapse.png" alt-text="Monitor pipeline runs":::
+
+
-11. To switch back to the pipeline runs view, select the **All pipeline runs** link at the top. Select **Refresh** to refresh the list.
+12. To switch back to the pipeline runs view, select the **All pipeline runs** link at the top. Select **Refresh** to refresh the list.
![Monitor activity runs](./media/load-azure-sql-data-warehouse/activity-monitoring.png)
-12. To monitor the execution details for each copy activity, select the **Details** link (eyeglasses icon) under **Activity name** in the activity runs view. You can monitor details like the volume of data copied from the source to the sink, data throughput, execution steps with corresponding duration, and used configurations.
+1. To monitor the execution details for each copy activity, select the **Details** link (eyeglasses icon) under **Activity name** in the activity runs view. You can monitor details like the volume of data copied from the source to the sink, data throughput, execution steps with corresponding duration, and used configurations.
![Monitor activity run details first](./media/load-azure-sql-data-warehouse/monitor-activity-run-details-1.png)
data-factory Managed Virtual Network Private Endpoint https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/managed-virtual-network-private-endpoint.md
-+ Last updated 07/20/2021
data-factory Parameterize Linked Services https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/parameterize-linked-services.md
Title: Parameterize linked services in Azure Data Factory
+ Title: Parameterize linked services
-description: Learn how to parameterize linked services in Azure Data Factory and pass dynamic values at run time.
+description: Learn how to parameterize linked services in Azure Data Factory and Azure Synapse Analytics pipelines, and pass dynamic values at run time.
-# Parameterize linked services in Azure Data Factory
+# Parameterize linked services in Azure Data Factory and Azure Synapse Analytics
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)] You can now parameterize a linked service and pass dynamic values at run time. For example, if you want to connect to different databases on the same logical SQL server, you can now parameterize the database name in the linked service definition. This prevents you from having to create a linked service for each database on the logical SQL server. You can parameterize other properties in the linked service definition as well - for example, *User name.*
-You can use the Data Factory UI in the Azure portal or a programming interface to parameterize linked services.
+You can use the UI in the Azure portal or a programming interface to parameterize linked services.
> [!TIP] > We recommend not to parameterize passwords or secrets. Store all secrets in Azure Key Vault instead, and parameterize the *Secret Name*.
For a seven-minute introduction and demonstration of this feature, watch the fol
All the linked service types are supported for parameterization.
-**Natively supported on ADF UI:** When authoring linked service on UI, Data Factory provides built-in parameterization experience for the following types of linked services. In linked service creation/edit blade, you can find options to new parameters and add dynamic content. Refer to [Data Factory UI experience](#data-factory-ui).
+**Natively supported in UI:** When authoring linked service on UI, the service provides built-in parameterization experience for the following types of linked services. In linked service creation/edit blade, you can find options to new parameters and add dynamic content. Refer to [UI experience](#ui-experience).
- Amazon Redshift - Amazon S3
All the linked service types are supported for parameterization.
- In linked service creation/edit blade -> expand "Advanced" at the bottom -> check "Specify dynamic contents in JSON format" checkbox -> specify the linked service JSON payload. - Or, after you create a linked service without parameterization, in [Management hub](author-visually.md#management-hub) -> Linked services -> find the specific linked service -> click "Code" (button "{}") to edit the JSON.
-Refer to the [JSON sample](#json) to add ` parameters` section to define parameters and reference the parameter using ` @{linkedService().paraName} `.
+Refer to the [JSON sample](#json) to add ` parameters` section to define parameters and reference the parameter using ` @{linkedService().paramName} `.
-## Data Factory UI
+## UI Experience
-![Add dynamic content to the Linked Service definition](media/parameterize-linked-services/parameterize-linked-services-image1.png)
+# [Azure Data Factory](#tab/data-factory)
-![Create a new parameter](media/parameterize-linked-services/parameterize-linked-services-image2.png)
+![Add dynamic content to the Linked Service definition](media/parameterize-linked-services/parameterize-linked-services-image-1.png)
+
+![Create a new parameter](media/parameterize-linked-services/parameterize-linked-services-image-2.png)
+
+# [Azure Synapse](#tab/synapse-analytics)
+
+![Add dynamic content to the Linked Service definition](media/parameterize-linked-services/parameterize-linked-services-image-1-synapse.png)
+
+![Create a new parameter](media/parameterize-linked-services/parameterize-linked-services-image-2-synapse.png)
++ ## JSON
data-factory Parameters Data Flow https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/parameters-data-flow.md
Title: Parameterizing mapping data flows
-description: Learn how to parameterize a mapping data flow from data factory pipelines
+description: Learn how to parameterize a mapping data flow from Azure Data Factory and Azure Synapse Analytics pipelines
Last updated 04/19/2021
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-Mapping data flows in Azure Data Factory and Azure Synapse Analytics support the use of parameters. Define parameters inside of your data flow definition and use them throughout your expressions. The parameter values are set by the calling pipeline via the Execute Data Flow activity. You have three options for setting the values in the data flow activity expressions:
+Mapping data flows in Azure Data Factory and Synapse pipelines support the use of parameters. Define parameters inside of your data flow definition and use them throughout your expressions. The parameter values are set by the calling pipeline via the Execute Data Flow activity. You have three options for setting the values in the data flow activity expressions:
* Use the pipeline control flow expression language to set a dynamic value * Use the data flow expression language to set a dynamic value
data-factory Pipeline Trigger Troubleshoot Guide https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/pipeline-trigger-troubleshoot-guide.md
The expression builder can fail to load due to network or cache problems with th
**Resolution** + Upgrade the web browser to the latest version of a supported browser, clear cookies for the site, and refresh the page. ### "Code":"BadRequest","message":"ErrorCode=FlowRunSizeLimitExceeded
data-factory Policy Reference https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/policy-reference.md
-+ Last updated 07/16/2021
data-factory Quickstart Create Data Factory Azure Cli https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/quickstart-create-data-factory-azure-cli.md
Last updated 03/24/2021-
- - template-quickstart
- - devx-track-azurecli
+ # Quickstart: Create an Azure Data Factory using Azure CLI
data-factory Security And Access Control Troubleshoot Guide https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/security-and-access-control-troubleshoot-guide.md
You can reassign access to data factory following permissions: **Get, Unwrap Ke
* Customer deleted Key Vault / CMK before deleting ADF. CMK in ADF should have "Soft Delete" enabled and "Purge Protect" enabled which has default retention policy of 90 days. You can restore the deleted key.
-Please review [Recover deleted Key](../key-vault/general/key-vault-recovery.md?tabs=azure-portal#list-recover-or-purge-soft-deleted-secrets-keys-and-certificates ) and [Deleted Key Value](../key-vault/general/key-vault-recovery.md?tabs=azure-portal#list-recover-or-purge-a-soft-deleted-key-vault)
+Please review [Recover deleted Key](../key-vault/general/key-vault-recovery.md?tabs=azure-portal#list-recover-or-purge-soft-deleted-secrets-keys-and-certificates) and [Deleted Key Value](../key-vault/general/key-vault-recovery.md?tabs=azure-portal#list-recover-or-purge-a-soft-deleted-key-vault)
* User Assigned Managed Identity (UA-MI) was deleted before ADF. You can recover from this by using REST API calls, you can do this in an http client of your choice in any programming language. If you have not anything already set up for REST API calls with Azure authentication, the easiest way to do this would be by using POSTMAN/Fiddler. Please follow following steps.
data-factory Supported File Formats And Compression Codecs https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/supported-file-formats-and-compression-codecs.md
Title: Supported file formats by copy activity in Azure Data Factory
-description: 'This topic describes the file formats and compression codes that are supported by copy activity in Azure Data Factory.'
+description: This topic describes the file formats and compression codes that are supported by copy activity in Azure Data Factory and Azure Synapse Analytics.
Last updated 07/16/2020
-# Supported file formats and compression codecs by copy activity in Azure Data Factory
+# Supported file formats and compression codecs by copy activity in Azure Data Factory and Azure Synapse pipelines
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)] *This article applies to the following connectors: [Amazon S3](connector-amazon-simple-storage-service.md), [Amazon S3 Compatible Storage](connector-amazon-s3-compatible-storage.md), [Azure Blob](connector-azure-blob-storage.md), [Azure Data Lake Storage Gen1](connector-azure-data-lake-store.md), [Azure Data Lake Storage Gen2](connector-azure-data-lake-storage.md), [Azure File Storage](connector-azure-file-storage.md), [File System](connector-file-system.md), [FTP](connector-ftp.md), [Google Cloud Storage](connector-google-cloud-storage.md), [HDFS](connector-hdfs.md), [HTTP](connector-http.md), [Oracle Cloud Storage](connector-oracle-cloud-storage.md) and [SFTP](connector-sftp.md).*
data-factory Transform Data Using Dotnet Custom Activity https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/transform-data-using-dotnet-custom-activity.md
Title: Use custom activities in a pipeline
-description: Learn how to create custom activities by using .NET, and then use the activities in an Azure Data Factory pipeline.
+description: Learn how to create custom activities by using .NET, and then use the activities in an Azure Data Factory or Azure Synapse Analytics pipeline.
Last updated 11/26/2018
-# Use custom activities in an Azure Data Factory pipeline
+# Use custom activities in an Azure Data Factory or Azure Synapse Analytics pipeline
> [!div class="op_single_selector" title1="Select the version of Data Factory service you are using:"] > * [Version 1](v1/data-factory-use-custom-activities.md) > * [Current version](transform-data-using-dotnet-custom-activity.md) [!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
-There are two types of activities that you can use in an Azure Data Factory pipeline.
+There are two types of activities that you can use in an Azure Data Factory or Synapse pipeline.
- [Data movement activities](copy-activity-overview.md) to move data between [supported source and sink data stores](copy-activity-overview.md#supported-data-stores-and-formats). - [Data transformation activities](transform-data.md) to transform data using compute services such as Azure HDInsight, Azure Batch, and Azure Machine Learning.
-To move data to/from a data store that Data Factory does not support, or to transform/process data in a way that isn't supported by Data Factory, you can create a **Custom activity** with your own data movement or transformation logic and use the activity in a pipeline. The custom activity runs your customized code logic on an **Azure Batch** pool of virtual machines.
+To move data to/from a data store that the service does not support, or to transform/process data in a way that isn't supported by the service, you can create a **Custom activity** with your own data movement or transformation logic and use the activity in a pipeline. The custom activity runs your customized code logic on an **Azure Batch** pool of virtual machines.
[!INCLUDE [updated-for-az](../../includes/updated-for-az.md)]
See following articles if you are new to Azure Batch service:
## Azure Batch linked service
-The following JSON defines a sample Azure Batch linked service. For details, see [Compute environments supported by Azure Data Factory](compute-linked-services.md)
+The following JSON defines a sample Azure Batch linked service. For details, see [Supported compute environments](compute-linked-services.md)
```json {
The following table describes names and descriptions of properties that are spec
| command | Command of the custom application to be executed. If the application is already available on the Azure Batch Pool Node, the resourceLinkedService and folderPath can be skipped. For example, you can specify the command to be `cmd /c dir`, which is natively supported by the Windows Batch Pool node. | Yes | | resourceLinkedService | Azure Storage Linked Service to the Storage account where the custom application is stored | No &#42; | | folderPath | Path to the folder of the custom application and all its dependencies<br/><br/>If you have dependencies stored in subfolders - that is, in a hierarchical folder structure under *folderPath* - the folder structure is currently flattened when the files are copied to Azure Batch. That is, all files are copied into a single folder with no subfolders. To work around this behavior, consider compressing the files, copying the compressed file, and then unzipping it with custom code in the desired location. | No &#42; |
-| referenceObjects | An array of existing Linked Services and Datasets. The referenced Linked Services and Datasets are passed to the custom application in JSON format so your custom code can reference resources of the Data Factory | No |
+| referenceObjects | An array of existing Linked Services and Datasets. The referenced Linked Services and Datasets are passed to the custom application in JSON format so your custom code can reference resources of the service | No |
| extendedProperties | User-defined properties that can be passed to the custom application in JSON format so your custom code can reference additional properties | No | | retentionTimeInDays | The retention time for the files submitted for custom activity. Default value is 30 days. | No |
You can directly execute a command using Custom Activity. The following example
## Passing objects and properties
-This sample shows how you can use the referenceObjects and extendedProperties to pass Data Factory objects and user-defined properties to your custom application.
+This sample shows how you can use the referenceObjects and extendedProperties to pass objects and user-defined properties from the service to your custom application.
```json {
If you would like to consume the content of stdout.txt in downstream activities,
## Pass outputs to another activity
-You can send custom values from your code in a Custom Activity back to Azure Data Factory. You can do so by writing them into `outputs.json` from your application. Data Factory copies the content of `outputs.json` and appends it into the Activity Output as the value of the `customOutput` property. (The size limit is 2MB.) If you want to consume the content of `outputs.json` in downstream activities, you can get the value by using the expression `@activity('<MyCustomActivity>').output.customOutput`.
+You can send custom values from your code in a Custom Activity back to the service. You can do so by writing them into `outputs.json` from your application. The service copies the content of `outputs.json` and appends it into the Activity Output as the value of the `customOutput` property. (The size limit is 2MB.) If you want to consume the content of `outputs.json` in downstream activities, you can get the value by using the expression `@activity('<MyCustomActivity>').output.customOutput`.
## Retrieve SecureString outputs
-Sensitive property values designated as type *SecureString*, as shown in some of the examples in this article, are masked out in the Monitoring tab in the Data Factory user interface. In actual pipeline execution, however, a *SecureString* property is serialized as JSON within the `activity.json` file as plain text. For example:
+Sensitive property values designated as type *SecureString*, as shown in some of the examples in this article, are masked out in the Monitoring tab in the user interface. In actual pipeline execution, however, a *SecureString* property is serialized as JSON within the `activity.json` file as plain text. For example:
```json "extendedProperties": {
Sensitive property values designated as type *SecureString*, as shown in some of
} ```
-This serialization is not truly secure, and is not intended to be secure. The intent is to hint to Data Factory to mask the value in the Monitoring tab.
+This serialization is not truly secure, and is not intended to be secure. The intent is a hint to the service to mask the value in the Monitoring tab.
To access properties of type *SecureString* from a custom activity, read the `activity.json` file, which is placed in the same folder as your .EXE, deserialize the JSON, and then access the JSON property (extendedProperties => [propertyName] => value).
To access properties of type *SecureString* from a custom activity, read the `ac
In Azure Data Factory version 1, you implement a (Custom) DotNet Activity by creating a .NET Class Library project with a class that implements the `Execute` method of the `IDotNetActivity` interface. The Linked Services, Datasets, and Extended Properties in the JSON payload of a (Custom) DotNet Activity are passed to the execution method as strongly-typed objects. For details about the version 1 behavior, see [(Custom) DotNet in version 1](v1/data-factory-use-custom-activities.md). Because of this implementation, your version 1 DotNet Activity code has to target .NET Framework 4.5.2. The version 1 DotNet Activity also has to be executed on Windows-based Azure Batch Pool nodes.
-In the Azure Data Factory V2 Custom Activity, you are not required to implement a .NET interface. You can now directly run commands, scripts, and your own custom code, compiled as an executable. To configure this implementation, you specify the `Command` property together with the `folderPath` property. The Custom Activity uploads the executable and its dependencies to `folderpath` and executes the command for you.
+In the Azure Data Factory V2 and Synapse pipelines Custom Activity, you are not required to implement a .NET interface. You can now directly run commands, scripts, and your own custom code, compiled as an executable. To configure this implementation, you specify the `Command` property together with the `folderPath` property. The Custom Activity uploads the executable and its dependencies to `folderpath` and executes the command for you.
-The Linked Services, Datasets (defined in referenceObjects), and Extended Properties defined in the JSON payload of a Data Factory v2 Custom Activity can be accessed by your executable as JSON files. You can access the required properties using a JSON serializer as shown in the preceding SampleApp.exe code sample.
+The Linked Services, Datasets (defined in referenceObjects), and Extended Properties defined in the JSON payload of a Data Factory v2 or Synapse pipeline Custom Activity can be accessed by your executable as JSON files. You can access the required properties using a JSON serializer as shown in the preceding SampleApp.exe code sample.
-With the changes introduced in the Data Factory V2 Custom Activity, you can write your custom code logic in your preferred language and execute it on Windows and Linux Operation Systems supported by Azure Batch.
+With the changes introduced in the Data Factory V2 and Synapse pipeline Custom Activity, you can write your custom code logic in your preferred language and execute it on Windows and Linux Operation Systems supported by Azure Batch.
-The following table describes the differences between the Data Factory V2 Custom Activity and the Data Factory version 1 (Custom) DotNet Activity:
+The following table describes the differences between the Data Factory V2 and Synapse pipeline Custom Activity and the Data Factory version 1 (Custom) DotNet Activity:
|Differences | Custom Activity | version 1 (Custom) DotNet Activity | | - | - | - |
If you have existing .NET code written for a version 1 (Custom) DotNet Activity,
- The Microsoft.Azure.Management.DataFactories NuGet package is no longer required. - Compile your code, upload the executable and its dependencies to Azure Storage, and define the path in the `folderPath` property.
-For a complete sample of how the end-to-end DLL and pipeline sample described in the Data Factory version 1 article [Use custom activities in an Azure Data Factory pipeline](./v1/data-factory-use-custom-activities.md) can be rewritten as a Data Factory Custom Activity, see [Data Factory Custom Activity sample](https://github.com/Azure/Azure-DataFactory/tree/master/SamplesV1/ADFv2CustomActivitySample).
+For a complete sample of how the end-to-end DLL and pipeline sample described in the Data Factory version 1 article [Use custom activities in an Azure Data Factory pipeline](./v1/data-factory-use-custom-activities.md) can be rewritten as a Custom Activity for Data Factory v2 and Synapse pipelines, see [Custom Activity sample](https://github.com/Azure/Azure-DataFactory/tree/master/SamplesV1/ADFv2CustomActivitySample).
## Auto-scaling of Azure Batch
-You can also create an Azure Batch pool with **autoscale** feature. For example, you could create an azure batch pool with 0 dedicated VMs and an autoscale formula based on the number of pending tasks.
+You can also create an Azure Batch pool with **autoscale** feature. For example, you could create an Azure batch pool with 0 dedicated VMs and an autoscale formula based on the number of pending tasks.
The sample formula here achieves the following behavior: When the pool is initially created, it starts with 1 VM. $PendingTasks metric defines the number of tasks in running + active (queued) state. The formula finds the average number of pending tasks in the last 180 seconds and sets TargetDedicated accordingly. It ensures that TargetDedicated never goes beyond 25 VMs. So, as new tasks are submitted, pool automatically grows and as tasks complete, VMs become free one by one and the autoscaling shrinks those VMs. startingNumberOfVMs and maxNumberofVMs can be adjusted to your needs.
data-factory Transform Data https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/transform-data.md
Title: Transform data
-description: Transform data or process data in Azure Data Factory using Hadoop, Azure Machine Learning Studio (classic), or Azure Data Lake Analytics.
+description: Transform data or process data in Azure Data Factory or Azure Synapse Analytics using Hadoop, Azure Machine Learning Studio (classic), or Azure Data Lake Analytics.
Last updated 07/31/2018
-# Transform data in Azure Data Factory
+# Transform data in Azure Data Factory and Azure Synapse Analytics
> [!div class="op_single_selector"] > * [Mapping data flow](data-flow-create.md)
Last updated 07/31/2018
> * [Azure Machine Learning Studio (classic)](transform-data-using-machine-learning.md) > * [Stored Procedure](transform-data-using-stored-procedure.md) > * [Data Lake Analytics U-SQL](transform-data-using-data-lake-analytics.md)
-> * [Synapse notebook](../synapse-analytics/synapse-notebook-activity.md)
+> * [Azure Synapse notebook](../synapse-analytics/synapse-notebook-activity.md)
> * [Databricks notebook](transform-data-databricks-notebook.md) > * [Databricks Jar](transform-data-databricks-jar.md) > * [Databricks Python](transform-data-databricks-python.md)
Last updated 07/31/2018
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)] ## Overview
-This article explains data transformation activities in Azure Data Factory that you can use to transform and process your raw data into predictions and insights at scale. A transformation activity executes in a computing environment such as Azure Databricks or Azure HDInsight. It provides links to articles with detailed information on each transformation activity.
+This article explains data transformation activities in Azure Data Factory and Synapse pipelines that you can use to transform and process your raw data into predictions and insights at scale. A transformation activity executes in a computing environment such as Azure Databricks or Azure HDInsight. It provides links to articles with detailed information on each transformation activity.
-Data Factory supports the following data transformation activities that can be added to [pipelines](concepts-pipelines-activities.md) either individually or chained with another activity.
+The service supports the following data transformation activities that can be added to [pipelines](concepts-pipelines-activities.md) either individually or chained with another activity.
-## Transform natively in Azure Data Factory with data flows
+## Transform natively in Azure Data Factory and Azure Synapse Analytics with data flows
### Mapping data flows
-Mapping data flows are visually designed data transformations in Azure Data Factory. Data flows allow data engineers to develop graphical data transformation logic without writing code. The resulting data flows are executed as activities within Azure Data Factory pipelines that use scaled-out Spark clusters. Data flow activities can be operationalized via existing Data Factory scheduling, control, flow, and monitoring capabilities. For more information, see [mapping data flows](concepts-data-flow-overview.md).
+Mapping data flows are visually designed data transformations in Azure Data Factory and Azure Synapse. Data flows allow data engineers to develop graphical data transformation logic without writing code. The resulting data flows are executed as activities within pipelines that use scaled-out Spark clusters. Data flow activities can be operationalized via existing scheduling, control, flow, and monitoring capabilities within the service. For more information, see [mapping data flows](concepts-data-flow-overview.md).
### Data wrangling
-Power Query in Azure Data Factory enables cloud-scale data wrangling, which allows you to do code-free data preparation at cloud scale iteratively. Data wrangling integrates with [Power Query Online](/power-query/) and makes Power Query M functions available for data wrangling at cloud scale via spark execution. For more information, see [data wrangling in ADF](wrangling-overview.md).
+Power Query in Azure Data Factory enables cloud-scale data wrangling, which allows you to do code-free data preparation at cloud scale iteratively. Data wrangling integrates with [Power Query Online](/power-query/) and makes Power Query M functions available for data wrangling at cloud scale via spark execution. For more information, see [data wrangling in Azure Data Factory](wrangling-overview.md).
+
+> [!NOTE]
+> Power Query is currently only supported in Azure Data Factory, and not in Azure Synapse. For a list of specific features supported in each service, see [Available features in Azure Data Factory & Azure Synapse Analytics pipelines](../synapse-analytics/data-integration/concepts-data-factory-differences.md).
## External transformations Optionally, you can hand-code transformations and manage the external compute environment yourself. ### HDInsight Hive activity
-The HDInsight Hive activity in a Data Factory pipeline executes Hive queries on your own or on-demand Windows/Linux-based HDInsight cluster. See [Hive activity](transform-data-using-hadoop-hive.md) article for details about this activity.
+The HDInsight Hive activity in a pipeline executes Hive queries on your own or on-demand Windows/Linux-based HDInsight cluster. See [Hive activity](transform-data-using-hadoop-hive.md) article for details about this activity.
### HDInsight Pig activity
-The HDInsight Pig activity in a Data Factory pipeline executes Pig queries on your own or on-demand Windows/Linux-based HDInsight cluster. See [Pig activity](transform-data-using-hadoop-pig.md) article for details about this activity.
+The HDInsight Pig activity in a pipeline executes Pig queries on your own or on-demand Windows/Linux-based HDInsight cluster. See [Pig activity](transform-data-using-hadoop-pig.md) article for details about this activity.
### HDInsight MapReduce activity
-The HDInsight MapReduce activity in a Data Factory pipeline executes MapReduce programs on your own or on-demand Windows/Linux-based HDInsight cluster. See [MapReduce activity](transform-data-using-hadoop-map-reduce.md) article for details about this activity.
+The HDInsight MapReduce activity in a pipeline executes MapReduce programs on your own or on-demand Windows/Linux-based HDInsight cluster. See [MapReduce activity](transform-data-using-hadoop-map-reduce.md) article for details about this activity.
### HDInsight Streaming activity
-The HDInsight Streaming activity in a Data Factory pipeline executes Hadoop Streaming programs on your own or on-demand Windows/Linux-based HDInsight cluster. See [HDInsight Streaming activity](transform-data-using-hadoop-streaming.md) for details about this activity.
+The HDInsight Streaming activity in a pipeline executes Hadoop Streaming programs on your own or on-demand Windows/Linux-based HDInsight cluster. See [HDInsight Streaming activity](transform-data-using-hadoop-streaming.md) for details about this activity.
### HDInsight Spark activity
-The HDInsight Spark activity in a Data Factory pipeline executes Spark programs on your own HDInsight cluster. For details, see [Invoke Spark programs from Azure Data Factory](transform-data-using-spark.md).
+The HDInsight Spark activity in a pipeline executes Spark programs on your own HDInsight cluster. For details, see [Invoke Spark programs with Azure Data Factory or Azure Synapse Analytics](transform-data-using-spark.md).
### Azure Machine Learning Studio (classic) activities
-Azure Data Factory enables you to easily create pipelines that use a published Azure Machine Learning Studio (classic) web service for predictive analytics. Using the [Batch Execution activity](transform-data-using-machine-learning.md) in an Azure Data Factory pipeline, you can invoke a Studio (classic) web service to make predictions on the data in batch.
+The service enables you to easily create pipelines that use a published Azure Machine Learning Studio (classic) web service for predictive analytics. Using the [Batch Execution activity](transform-data-using-machine-learning.md) in a pipeline, you can invoke a Studio (classic) web service to make predictions on the data in batch.
Over time, the predictive models in the Studio (classic) scoring experiments need to be retrained using new input datasets. After you are done with retraining, you want to update the scoring web service with the retrained machine learning model. You can use the [Update Resource activity](update-machine-learning-models.md) to update the web service with the newly trained model.
You can use the SQL Server Stored Procedure activity in a Data Factory pipeline
### Data Lake Analytics U-SQL activity Data Lake Analytics U-SQL activity runs a U-SQL script on an Azure Data Lake Analytics cluster. See [Data Analytics U-SQL activity](transform-data-using-data-lake-analytics.md) article for details.
-### Synapse Notebook activity
+### Azure Synapse Notebook activity
-The Azure Synapse Notebook Activity in a Synapse pipeline runs a Synapse notebook in your Azure Synapse workspace. See [Transform data by running a Synapse notebook](../synapse-analytics/synapse-notebook-activity.md).
+The Azure Synapse Notebook Activity in a Synapse pipeline runs a Synapse notebook in your Azure Synapse workspace. See [Transform data by running an Azure Synapse notebook](../synapse-analytics/synapse-notebook-activity.md).
### Databricks Notebook activity
-The Azure Databricks Notebook Activity in a Data Factory pipeline runs a Databricks notebook in your Azure Databricks workspace. Azure Databricks is a managed platform for running Apache Spark. See [Transform data by running a Databricks notebook](transform-data-databricks-notebook.md).
+The Azure Databricks Notebook Activity in a pipeline runs a Databricks notebook in your Azure Databricks workspace. Azure Databricks is a managed platform for running Apache Spark. See [Transform data by running a Databricks notebook](transform-data-databricks-notebook.md).
### Databricks Jar activity
-The Azure Databricks Jar Activity in a Data Factory pipeline runs a Spark Jar in your Azure Databricks cluster. Azure Databricks is a managed platform for running Apache Spark. See [Transform data by running a Jar activity in Azure Databricks](transform-data-databricks-jar.md).
+The Azure Databricks Jar Activity in a pipeline runs a Spark Jar in your Azure Databricks cluster. Azure Databricks is a managed platform for running Apache Spark. See [Transform data by running a Jar activity in Azure Databricks](transform-data-databricks-jar.md).
### Databricks Python activity
-The Azure Databricks Python Activity in a Data Factory pipeline runs a Python file in your Azure Databricks cluster. Azure Databricks is a managed platform for running Apache Spark. See [Transform data by running a Python activity in Azure Databricks](transform-data-databricks-python.md).
+The Azure Databricks Python Activity in a pipeline runs a Python file in your Azure Databricks cluster. Azure Databricks is a managed platform for running Apache Spark. See [Transform data by running a Python activity in Azure Databricks](transform-data-databricks-python.md).
### Custom activity If you need to transform data in a way that is not supported by Data Factory, you can create a custom activity with your own data processing logic and use the activity in the pipeline. You can configure the custom .NET activity to run using either an Azure Batch service or an Azure HDInsight cluster. See [Use custom activities](transform-data-using-dotnet-custom-activity.md) article for details.
-You can create a custom activity to run R scripts on your HDInsight cluster with R installed. See [Run R Script using Azure Data Factory](https://github.com/Azure/Azure-DataFactory/tree/master/SamplesV1/RunRScriptUsingADFSample).
+You can create a custom activity to run R scripts on your HDInsight cluster with R installed. See [Run R Script using Azure Data Factory and Synapse pipelines](https://github.com/Azure/Azure-DataFactory/tree/master/SamplesV1/RunRScriptUsingADFSample).
### Compute environments
-You create a linked service for the compute environment and then use the linked service when defining a transformation activity. There are two types of compute environments supported by Data Factory.
+You create a linked service for the compute environment and then use the linked service when defining a transformation activity. There are two supported types of compute environments.
-- **On-Demand**: In this case, the computing environment is fully managed by Data Factory. It is automatically created by the Data Factory service before a job is submitted to process data and removed when the job is completed. You can configure and control granular settings of the on-demand compute environment for job execution, cluster management, and bootstrapping actions. -- **Bring Your Own**: In this case, you can register your own computing environment (for example HDInsight cluster) as a linked service in Data Factory. The computing environment is managed by you and the Data Factory service uses it to execute the activities.
+- **On-Demand**: In this case, the computing environment is fully managed by the service. It is automatically created by the service before a job is submitted to process data and removed when the job is completed. You can configure and control granular settings of the on-demand compute environment for job execution, cluster management, and bootstrapping actions.
+- **Bring Your Own**: In this case, you can register your own computing environment (for example HDInsight cluster) as a linked service. The computing environment is managed by you and the service uses it to execute the activities.
-See [Compute Linked Services](compute-linked-services.md) article to learn about compute services supported by Data Factory.
+See [Compute Linked Services](compute-linked-services.md) article to learn about supported compute services.
## Next steps
-See the following tutorial for an example of using a transformation activity: [Tutorial: transform data using Spark](tutorial-transform-data-spark-powershell.md)
+See the following tutorial for an example of using a transformation activity: [Tutorial: transform data using Spark](tutorial-transform-data-spark-powershell.md)
data-factory Turorial Push Lineage To Purview https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/data-factory/turorial-push-lineage-to-purview.md
You also can see lineage data for Execute SSIS Package activity.
## Next steps [Catalog lineage user guide](../purview/catalog-lineage-user-guide.md)
-[Connect Data Factory to Azure Purview](connect-data-factory-to-azure-purview.md)
+[Connect Data Factory to Azure Purview](connect-data-factory-to-azure-purview.md)
defender-for-iot Concept Agent Portfolio Overview Os Support https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/defender-for-iot/device-builders/concept-agent-portfolio-overview-os-support.md
Title: Agent portfolio overview and OS support (Preview) description: Azure Defender for IoT provides a large portfolio of agents based on the device type. Previously updated : 05/02/2021 Last updated : 08/08/2021 # Agent portfolio overview and OS support (Preview)
-Azure Defender for IoT provides a large portfolio of agents based on the device type.
+Azure Defender for IoT provides a large portfolio of agents based on the device type.
## Standalone agent
-The standalone agent covers most of the Linux Operating Systems (OS), which can be deployed as a binary package or as a source code that can be incorporated as part of the firmware and allow modification and customization based on customer needs. The following are some examples of supported OS:
+The standalone agent covers most of the Linux Operating Systems (OS), which can be deployed as a binary package or as a source code that can be incorporated as part of the firmware and allow modification and customization based on customer needs. The following are some examples of supported OS:
-| Operating system | AMD64 | ARM32v7 |
-|--|--|--|
-| Debian 9 | Γ£ô | Γ£ô |
-| Ubuntu 18.04 | Γ£ô | |
-| Ubuntu 20.04 | Γ£ô | |
+| Operating system | AMD64 | ARM32v7 | ARM64 |
+|--|--|--|--|
+| Debian 9 | Γ£ô | Γ£ô | |
+| Ubuntu 18.04 | Γ£ô | | Γ£ô |
+| Ubuntu 20.04 | Γ£ô | | |
-For more information, supported operating systems, or to request access to the source code so you can incorporate it as a part of the device's firmware, contact your account manager, or send an email to <defender_micro_agent@microsoft.com>.
+For more information, supported operating systems, or to request access to the source code so you can incorporate it as a part of the device's firmware, contact your account manager, or send an email to <defender_micro_agent@microsoft.com>.
## Azure RTOS micro agent
defender-for-iot How To Manage Individual Sensors https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/defender-for-iot/organizations/how-to-manage-individual-sensors.md
Azure Defender for IoT uses SSL/TLS certificates to:
- Meet specific certificate and encryption requirements requested by your organization by uploading the CA-signed certificate. -- Allow validation between the management console and connected sensors, and between a management console and a High Availability management console. Validations is evaluated against a Certificate Revocation List, and the certificate expiration date. *If validation fails, communication between the management console and the sensor is halted and a validation error is presented in the console*. This option is enabled by default after installation.
+- Allow validation between the management console and connected sensors, and between a management console and a High Availability management console. Validations is evaluated against a Certificate Revocation List (CRL), and the certificate expiration date. *If validation fails, communication between the management console and the sensor is halted and a validation error is presented in the console*. This option is enabled by default after installation.
-- Third party Forwarding rules, for example alert information sent to SYSLOG, Splunk or ServiceNow; or communications with Active Directory are not validated.
+- Third party Forwarding rules, for example alert information sent to SYSLOG, Splunk or ServiceNow; or communications with Active Directory are validated.
### About CRL servers When validation is on, the appliance should be able to establish connection to the CRL server defined by the certificate. By default, the certificate will reference the CRL URL on HTTP port 80. Some organizational security policies may block access to this port. If your organization does not have access to port 80, you can: 1. Define another URL and a specific port in the certificate. -- The URL should be defined as http://<URL>:<Port> instead of http://<URL>.
+- The URL should be defined as http:// rather than https://.
- Verify that the destination CRL server can listen on the port you defined. 1. Use a proxy server that will access the CRL on port 80. 1. Not carry out CRL validation. In this case, remove the CRL URL reference in the certificate.
+### About SSL/TLS certificates
-### About SSL certificates
+The Defender for IoT sensor and on-premises management console use SSL and TLS certificates for the following functions:
-The Defender for IoT sensor, and on-premises management console use SSL, and TLS certificates for the following functions:
-
+ - Secure communications between users, and the web console of the appliance.
- Secure communications to the REST API on the sensor and on-premises management console.
+ - Secure communications between the sensors and an on-premises management console.
Once installed, the appliance generates a local self-signed certificate to allow preliminary access to the web console.
digital-twins Concepts Route Events https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/digital-twins/concepts-route-events.md
Alternatively, the event message also contains the ID of the source twin that se
The compute resource also needs to establish security and access permissions independently.
-To walk through the process of setting up an Azure function to process digital twin events, see [Set up an Azure function for processing data](how-to-create-azure-function.md).
+To walk through the process of setting up an Azure function to process digital twin events, see [Set up twin-to-twin event handling](how-to-send-twin-to-twin-events.md).
## Create an endpoint
See how to set up and manage an event route:
* [Manage endpoints and routes](how-to-manage-routes.md) Or, see how to use Azure Functions to route events within Azure Digital Twins:
-* [Set up an Azure function for processing data](how-to-create-azure-function.md)
+* [Set up twin-to-twin event handling](how-to-send-twin-to-twin-events.md).
digital-twins How To Authenticate Client https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/digital-twins/how-to-authenticate-client.md
Here is a code sample to add a `DefaultAzureCredential` to your project:
### ManagedIdentityCredential method
-The [ManagedIdentityCredential](/dotnet/api/azure.identity.managedidentitycredential?view=azure-dotnet&preserve-view=true) method works great in cases where you need [managed identities (MSI)](../active-directory/managed-identities-azure-resources/overview.md)ΓÇöfor example, when working with Azure Functions.
+The [ManagedIdentityCredential](/dotnet/api/azure.identity.managedidentitycredential?view=azure-dotnet&preserve-view=true) method works great in cases where you need [managed identities (MSI)](../active-directory/managed-identities-azure-resources/overview.md)ΓÇöfor example, when [authenticating with Azure Functions](#authenticate-azure-functions).
This means that you may use `ManagedIdentityCredential` in the same project as `DefaultAzureCredential` or `InteractiveBrowserCredential`, to authenticate a different part of the project.
Here is an example of the code to create an authenticated SDK client using `Inte
## Authenticate Azure Functions
-See [Set up an Azure function for processing data](how-to-create-azure-function.md) for a more complete example that explains some of the important configuration choices in the context of functions.
+This section contains some of the important configuration choices in the context of authenticating with Azure Functions. First, you'll read about recommended class-level variables and authentication code that will allow the function to access Azure Digital Twins. Then, you'll read about some final configuration steps to complete for your function after its code is published to Azure.
-Also, to use authentication in a function, remember to:
-* [Enable managed identity](../app-service/overview-managed-identity.md?tabs=dotnet)
-* Use [environment variables](/sandbox/functions-recipes/environment-variables?tabs=csharp) as appropriate
-* Assign permissions to the functions app that enable it to access the Digital Twins APIs. For more information on Azure Functions processes, see [Set up an Azure function for processing data](how-to-create-azure-function.md).
+### Write application code
+
+When writing the Azure function, consider adding these variables and code to your function:
+
+* **Code to read the Azure Digital Twins service URL as an environment variable or configuration setting.** It's a good practice to read the service URL from an [application setting/environment variable](../azure-functions/functions-how-to-use-azure-function-app-settings.md?tabs=portal), rather than hard-coding it in the function. In an Azure function, that code to read the environment variable might look like this:
+
+ :::code language="csharp" source="~/digital-twins-docs-samples/sdks/csharp/adtIngestFunctionSample.cs" id="ADT_service_URL":::
+
+ Later, after publishing the function, you'll create and set the value of the environment variable for this code to read. For instructions on how to do this, skip ahead to [Configure application settings](#configure-application-settings).
+
+* **A static variable to hold an HttpClient instance.** HttpClient is relatively expensive to create, so you'll probably want to create it once with the authentication code to avoid creating it for every function invocation.
+
+ :::code language="csharp" source="~/digital-twins-docs-samples/sdks/csharp/adtIngestFunctionSample.cs" id="HTTP_client":::
+
+* **Managed identity credentials.** Create a managed identity credential that your function will use to access Azure Digital Twins.
+ :::code language="csharp" source="~/digital-twins-docs-samples/sdks/csharp/adtIngestFunctionSample.cs" id="ManagedIdentityCredential":::
+
+ Later, after publishing the function, you'll make sure the function's identity has permission to access the Azure Digital Twins APIs. For instructions on how to do this, skip ahead to [Assign an access role](#assign-an-access-role).
+
+* **A local variable _DigitalTwinsClient_.** Add the variable inside your function to hold your Azure Digital Twins client instance. Do *not* make this variable static inside your class.
+ :::code language="csharp" source="~/digital-twins-docs-samples/sdks/csharp/adtIngestFunctionSample.cs" id="DigitalTwinsClient":::
+
+* **A null check for _adtInstanceUrl_.** Add the null check and then wrap your function logic in a try/catch block to catch any exceptions.
+
+After these are added to a function, your function code might look like the following example.
++
+When you're finished with your function code, including adding authentication and the function's logic, [publish the app to Azure](../azure-functions/functions-develop-vs.md#publish-to-azure)
+
+### Configure published app
+
+Finally, complete the following configuration steps for a published Azure function to make sure it can access your Azure Digital Twins instance.
+ ## Authenticate across tenants
-Azure Digital Twins is a service that only supports one [Azure Active Directory (Azure AD) tenant](../active-directory/develop/quickstart-create-new-tenant.md): the main tenant from the subscription where the Azure Digital Twins instance is located.
+Azure Digital Twins is a service that only supports one [Azure Active Directory (Azure AD) tenant](../active-directory/develop/quickstart-create-new-tenant.md): the main tenant from the subscription where the Azure Digital Twins instance is located.
[!INCLUDE [digital-twins-tenant-limitation](../../includes/digital-twins-tenant-limitation.md)]
digital-twins How To Create Azure Function https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/digital-twins/how-to-create-azure-function.md
-
-# Mandatory fields.
Title: Set up a function in Azure to process data-
-description: See how to create a function in Azure that can access and be triggered by digital twins.
-- Previously updated : 7/14/2021---
-# Optional fields. Don't forget to remove # if you need a field.
-#
-#
-#
--
-# Connect function apps in Azure for processing data
-
-Digital twins can be updated based on data by using [event routes](concepts-route-events.md) through compute resources. For example, a function that's made by using [Azure Functions](../azure-functions/functions-overview.md) can update a digital twin in response to:
-* Device telemetry data from Azure IoT Hub.
-* A property change or other data from another digital twin within the twin graph.
-
-This article shows you how to create a function in Azure for use with Azure Digital Twins. To create a function, you'll follow these basic steps:
-
-1. Create an Azure Functions project in Visual Studio.
-2. Write a function that has an [Azure Event Grid](../event-grid/overview.md) trigger.
-3. Add authentication code to the function so you can access Azure Digital Twins.
-4. Publish the function app to Azure.
-5. Set up [security](concepts-security.md) for the function app.
-
-## Prerequisite: Set up Azure Digital Twins
--
-## Create a function app in Visual Studio
-
-For instructions on how to create a function app using Visual Studio, see [Develop Azure Functions using Visual Studio](../azure-functions/functions-develop-vs.md#publish-to-azure).
-
-## Write a function that has an Event Grid trigger
-
-You can write a function by adding an SDK to your function app. The function app interacts with Azure Digital Twins by using the [Azure Digital Twins SDK for .NET (C#)](/dotnet/api/overview/azure/digitaltwins/client?view=azure-dotnet&preserve-view=true).
-
-To use the SDK, you'll need to include the following packages in your project. Install the packages by using the Visual Studio NuGet package manager. Or add the packages by using `dotnet` in a command-line tool.
-
-* [Azure.DigitalTwins.Core](https://www.nuget.org/packages/Azure.DigitalTwins.Core/)
-* [Azure.Identity](https://www.nuget.org/packages/Azure.Identity/)
-* [System.Net.Http](https://www.nuget.org/packages/System.Net.Http/)
-* [Azure.Core](https://www.nuget.org/packages/Azure.Core/)
-
-Next, in Visual Studio Solution Explorer, open the _.cs_ file that includes your sample code. Add the following `using` statements for the packages.
--
-## Add authentication code to the function
-
-Now declare class-level variables and add authentication code that will allow the function to access Azure Digital Twins. Add the variables and code to your function.
-
-* **Code to read the Azure Digital Twins service URL as an environment variable.** It's a good practice to read the service URL from an environment variable rather than hard-coding it in the function. You'll set the value of this environment variable [later in this article](#set-up-security-access-for-the-function-app). For more information about environment variables, see [Manage your function app](../azure-functions/functions-how-to-use-azure-function-app-settings.md?tabs=portal).
-
- :::code language="csharp" source="~/digital-twins-docs-samples/sdks/csharp/adtIngestFunctionSample.cs" id="ADT_service_URL":::
-
-* **A static variable to hold an HttpClient instance.** HttpClient is relatively expensive to create, so we want to avoid creating it for every function invocation.
-
- :::code language="csharp" source="~/digital-twins-docs-samples/sdks/csharp/adtIngestFunctionSample.cs" id="HTTP_client":::
-
-* **Managed identity credentials.**
- :::code language="csharp" source="~/digital-twins-docs-samples/sdks/csharp/adtIngestFunctionSample.cs" id="ManagedIdentityCredential":::
-
-* **A local variable _DigitalTwinsClient_.** Add the variable inside your function to hold your Azure Digital Twins client instance. Do *not* make this variable static inside your class.
- :::code language="csharp" source="~/digital-twins-docs-samples/sdks/csharp/adtIngestFunctionSample.cs" id="DigitalTwinsClient":::
-
-* **A null check for _adtInstanceUrl_.** Add the null check and then wrap your function logic in a try/catch block to catch any exceptions.
-
-After these changes, your function code will look like the following example.
--
-Now that your application is written, you can publish it to Azure.
-
-## Publish the function app to Azure
-
-For instructions on how to publish a function app, see [Develop Azure Functions using Visual Studio](../azure-functions/functions-develop-vs.md#publish-to-azure).
-
-### Verify the publication of your function
-
-1. Sign in by using your credentials in the [Azure portal](https://portal.azure.com/).
-2. In the search box at the top of the window, search for your function app name and then select it.
-
- :::image type="content" source="media/how-to-create-azure-function/search-function-app.png" alt-text="Screenshot showing the Azure portal. In the search field, enter the function app name." lightbox="media/how-to-create-azure-function/search-function-app.png":::
-
-3. On the **Function app** page that opens, in the menu on the left, choose **Functions**. If your function is successfully published, its name appears in the list.
-
- > [!Note]
- > You might have to wait a few minutes or refresh the page couple of times before your function appears in the list of published functions.
-
- :::image type="content" source="media/how-to-create-azure-function/view-published-functions.png" alt-text="Screenshot showing published functions in the Azure portal." lightbox="media/how-to-create-azure-function/view-published-functions.png":::
-
-To access Azure Digital Twins, your function app needs a system-managed identity with permissions to access your Azure Digital Twins instance. You'll set that up next.
-
-## Set up security access for the function app
--
-## Next steps
-
-In this article, you set up a function app in Azure for use with Azure Digital Twins. Next, see how to build on your basic function to [ingest IoT Hub data into Azure Digital Twins](how-to-ingest-iot-hub-data.md).
digital-twins How To Ingest Iot Hub Data https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/digital-twins/how-to-ingest-iot-hub-data.md
Before continuing with this example, you'll need to set up the following resourc
This article also uses **Visual Studio**. You can download the latest version from [Visual Studio Downloads](https://visualstudio.microsoft.com/downloads/).
-### Example telemetry scenario
+## Example telemetry scenario
This how-to outlines how to send messages from IoT Hub to Azure Digital Twins, using a function in Azure. There are many possible configurations and matching strategies you can use for sending messages, but the example for this article contains the following parts: * A thermostat device in IoT Hub, with a known device ID
In this section, you'll create an Azure function to access Azure Digital Twins a
4. Publish the project with the *IoTHubtoTwins.cs* function to a function app in Azure. For instructions on how to do this, see [Develop Azure Functions using Visual Studio](../azure-functions/functions-develop-vs.md#publish-to-azure). +
+To access Azure Digital Twins, your function app needs a system-managed identity with permissions to access your Azure Digital Twins instance. You'll set that up next.
+ ### Configure the function app Next, **assign an access role** for the function and **configure the application settings** so that it can access your Azure Digital Twins instance.
digital-twins How To Provision Using Device Provisioning Service https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/digital-twins/how-to-provision-using-device-provisioning-service.md
Start by opening the function app project in Visual Studio on your machine and f
4. Publish the project with the *DpsAdtAllocationFunc.cs* function to a function app in Azure. For instructions on how to do this, see [Develop Azure Functions using Visual Studio](../azure-functions/functions-develop-vs.md#publish-to-azure). > [!IMPORTANT]
-> When creating the function app for the first time in the [Prerequisites section](#prerequisites), you may have already assigned an access role for the function and configured the application settings for it to access your Azure Digital Twins instance. These need to be done once for the entire function app, so verify they've been completed in your app before continuing. You can find instructions in the [Set up security access for the function app](how-to-create-azure-function.md#set-up-security-access-for-the-function-app) section of the *How-to: Set up a function in Azure to process data* article.
+> When creating the function app for the first time in the [Prerequisites section](#prerequisites), you may have already assigned an access role for the function and configured the application settings for it to access your Azure Digital Twins instance. These need to be done once for the entire function app, so verify they've been completed in your app before continuing. You can find instructions in the [Configure published app](how-to-authenticate-client.md#configure-published-app) section of the *Write app authentication code* article.
### Create Device Provisioning enrollment
Start by opening the function app project in Visual Studio on your machine and f
4. Publish the project with the *DeleteDeviceInTwinFunc.cs* function to a function app in Azure. For instructions on how to do this, see [Develop Azure Functions using Visual Studio](../azure-functions/functions-develop-vs.md#publish-to-azure). > [!IMPORTANT]
-> When creating the function app for the first time in the [Prerequisites section](#prerequisites), you may have already assigned an access role for the function and configured the application settings for it to access your Azure Digital Twins instance. These need to be done once for the entire function app, so verify they've been completed in your app before continuing. You can find instructions in the [Set up security access for the function app](how-to-create-azure-function.md#set-up-security-access-for-the-function-app) section of the *How-to: Set up a function in Azure to process data* article.
+> When creating the function app for the first time in the [Prerequisites section](#prerequisites), you may have already assigned an access role for the function and configured the application settings for it to access your Azure Digital Twins instance. These need to be done once for the entire function app, so verify they've been completed in your app before continuing. You can find instructions in the [Configure published app](how-to-authenticate-client.md#configure-published-app) section of the *Write app authentication code* article.
### Create an IoT Hub route for lifecycle events
digital-twins How To Send Twin To Twin Events https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/digital-twins/how-to-send-twin-to-twin-events.md
+
+# Mandatory fields.
+ Title: Set up twin-to-twin event handling
+
+description: See how to create a function in Azure for propagating events through the twin graph.
++ Last updated : 8/5/2021+++
+# Optional fields. Don't forget to remove # if you need a field.
+#
+#
+#
++
+# Set up twin-to-twin event handling with Azure Functions
+
+A fully-connected Azure Digital Twins graph is driven by event propagation. Data arrives into Azure Digital Twins from external sources like IoT Hub, and then is propagated through the Azure Digital Twins graph, updating relevant twins as appropriate.
+
+For example, consider a graph representing Floors and Rooms in a building, where each Floor contains multiple Rooms. You may want to set up a twin-to-twin data flow such that every time the temperature property on a Room twin is updated, a new average temperature is calculated for all the Rooms on the same Floor, and the temperature property of the Floor twin is updated to reflect the new average temperature across all the Rooms it contains (including the one that was updated).
+
+In this article, you'll see how to send events from twin to twin, allowing you to update twins in response to property changes or other data from another twin in the graph. Currently, twin-to-twin updates are handled by setting up an [Azure function](../azure-functions/functions-overview.md) that watches for twin life cycle events that should affect other areas of the graph, and makes changes to other twins accordingly.
+
+## Prerequisites
+
+This article uses **Visual Studio**. You can download the latest version from [Visual Studio Downloads](https://visualstudio.microsoft.com/downloads/).
+
+To set up twin-to-twin handling, you'll need an **Azure Digital Twins instance** to work with. For instructions on how to create an instance, see [Set up an Azure Digital Twins instance and authentication](./how-to-set-up-instance-portal.md). The instance should contain at least **two twins** that you want to send data between.
+
+Optionally, you may want to set up [automatic telemetry ingestion through IoT Hub](how-to-ingest-iot-hub-data.md) for your twins as well. This is not required in order to send data from twin to twin, but it's an important piece of a complete solution where the twin graph is driven by live telemetry.
+
+## Set up endpoint and route
+
+To set up twin-to-twin event handling, start by creating an **endpoint** in Azure Digital Twins and a **route** to that endpoint. Twins undergoing an update will use the route to send information about their update events to the endpoint (where Event Grid can pick them up later and pass them to an Azure function for processing).
++
+## Create the Azure function
+
+Next, create an Azure function that will listen on the endpoint and receive twin events that are sent there via the route.
+
+1. First, create an Azure Functions project in Visual Studio on your machine. For instructions on how to do this, see [Develop Azure Functions using Visual Studio](../azure-functions/functions-develop-vs.md#create-an-azure-functions-project).
+
+2. Add the following packages to your project (you can use the Visual Studio NuGet package manager or `dotnet` commands in a command-line tool).
+
+ * [Azure.DigitalTwins.Core](https://www.nuget.org/packages/Azure.DigitalTwins.Core/)
+ * [Azure.Identity](https://www.nuget.org/packages/Azure.Identity/)
+ * [Microsoft.Azure.WebJobs.Extensions.EventGrid](https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.EventGrid)
+
+3. Fill in the logic of your function. You can view sample function code for several scenarios in the [azure-digital-twins-getting-started](https://github.com/Azure-Samples/azure-digital-twins-getting-started/tree/main/azure-functions) repository to help you get started.
+
+5. Publish the function app to Azure. For instructions on how to publish a function app, see [Develop Azure Functions using Visual Studio](../azure-functions/functions-develop-vs.md#publish-to-azure).
++
+### Configure the function app
+
+Before your function can access Azure Digital Twins, it needs some information about the instance and permission to access it. In this section, you'll **assign an access role** for the function and **configure the application settings** so that it can find and access the instance.
++
+## Connect the function to Event Grid
+
+Next, subscribe your Azure function to the event grid topic you created earlier. This will ensure that data can flow from an updated twin through the event grid topic to the function.
+
+To do this, you'll create an **Event Grid subscription** that sends data from the event grid topic that you created earlier to your Azure function.
+
+Use the following CLI command, filling in placeholders for your subscription ID, resource group, function app, and function name.
+
+```azurecli-interactive
+az eventgrid event-subscription create --name <name-for-your-event-subscription> --source-resource-id /subscriptions/<subscription-ID>/resourceGroups/<your-resource-group>/providers/Microsoft.EventGrid/topics/<your-event-grid-topic> \ --endpoint-type azurefunction --endpoint /subscriptions/<subscription-ID>/resourceGroups/<your-resource-group>/providers/Microsoft.Web/sites/<your-function-app-name>/functions/<function-name>
+```
+
+Now, your function can receive events through your event grid topic. The data flow setup is complete.
+
+## Test and verify results
+
+The last step is to verify that the flow is working, by updating a twin and checking that related twins are updated according to the logic in your Azure function.
+
+To kick off the process, update the twin that's the source of the event flow. You can use the [Azure CLI](/cli/azure/dt/twin?view=azure-cli-latest&preserve-view=true#az_dt_twin_update), [Azure Digital Twins SDK](how-to-manage-twin.md#update-a-digital-twin), or [Azure Digital Twins REST APIs](how-to-use-postman.md?tabs=data-plane) to make the update.
+
+Next, query your Azure Digital Twins instance for the related twin. You can use the [Azure CLI](/cli/azure/dt/twin?view=azure-cli-latest&preserve-view=true#az_dt_twin_query), or the [Azure Digital Twins REST APIs and SDK](how-to-query-graph.md#run-queries-with-the-api). Verify that the twin received the data and updated as expected.
+
+## Next steps
+
+In this article, you set up twin-to-twin event handling in Azure Digital Twins. Next, set up an Azure function to trigger this flow automatically based on incoming telemetry from IoT Hub devices: [Ingest telemetry from IoT Hub](how-to-ingest-iot-hub-data.md).
digital-twins Resources Compare Original Release https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/digital-twins/resources-compare-original-release.md
The chart below provides a side-by-side view of concepts that have changed betwe
| | | | | | **Modeling**<br>*More flexible* | The original release was designed around smart spaces, so it came with a built-in vocabulary for buildings. | The current Azure Digital Twins is domain-agnostic. You can define your own custom vocabulary and custom models for your solution, to represent more kinds of environments in more flexible ways.<br><br>Learn more in [Custom models](concepts-models.md). | | **Topology**<br>*More flexible*| The original release supported a tree data structure, tailored to smart spaces. Digital twins were connected with hierarchical relationships. | With the current release, your digital twins can be connected into arbitrary graph topologies, organized however you want. This gives you more flexibility to express the complex relationships of the real world.<br><br>Learn more in [Digital twins and the twin graph](concepts-twins-graph.md). |
-| **Compute**<br>*Richer, more flexible* | In the original release, logic for processing events and telemetry was defined in JavaScript user-defined functions (UDFs). Debugging with UDFs was limited. | The current release has an open compute model: you provide custom logic by attaching external compute resources like [Azure Functions](../azure-functions/functions-overview.md). This lets you use a programming language of your choice, access custom code libraries without restriction, and take advantage of development and debugging resources that the external service may have.<br><br>Learn more in [Set up an Azure function for processing data](how-to-create-azure-function.md). |
+| **Compute**<br>*Richer, more flexible* | In the original release, logic for processing events and telemetry was defined in JavaScript user-defined functions (UDFs). Debugging with UDFs was limited. | The current release has an open compute model: you provide custom logic by attaching external compute resources like [Azure Functions](../azure-functions/functions-overview.md). This lets you use a programming language of your choice, access custom code libraries without restriction, and take advantage of development and debugging resources that the external service may have.<br><br>To see an end-to-end scenario driven by data flow through Azure functions, see [Connect an end-to-end solution](tutorial-end-to-end.md). |
| **Device management with IoT Hub**<br>*More accessible* | The original release managed devices with an instance of [IoT Hub](../iot-hub/about-iot-hub.md) that was internal to the Azure Digital Twins service. This integrated hub was not fully accessible to developers. | In the current release, you "bring your own" IoT hub, by attaching an independently-created IoT Hub instance (along with any devices it already manages). This gives you full access to IoT Hub's capabilities and puts you in control of device management.<br><br>Learn more in [Ingest telemetry from IoT Hub](how-to-ingest-iot-hub-data.md). | | **Security**<br>*More standard* | The original release had pre-defined roles that you could use to manage access to your instance. | The current release integrates with the same [Azure role-based access control (Azure RBAC)](../role-based-access-control/overview.md) back-end service that other Azure services use. This may make it simpler to authenticate between other Azure services in your solution, like IoT Hub, Azure Functions, Event Grid, and more.<br>With RBAC, you can still use pre-defined roles, or you can build and configure custom roles.<br><br>Learn more in [Security for Azure Digital Twins solutions](concepts-security.md). | | **Scalability**<br>*Greater* | The original release had scale limitations for devices, messages, graphs, and scale units. Only one instance of Azure Digital Twins was supported per subscription. | The current release relies on a new architecture with improved scalability, and has greater compute power. It also supports 10 instances per region, per subscription.<br><br>See [Azure Digital Twins service limits](reference-service-limits.md) for details of the limits in the current release. |
digital-twins Tutorial End To End https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/digital-twins/tutorial-end-to-end.md
To do this, you'll use the *ProcessDTRoutedData* Azure function to update a Room
:::image type="content" source="media/tutorial-end-to-end/building-scenario-c.png" alt-text="Diagram of an excerpt from the full building scenario diagram highlighting the section that shows the elements after Azure Digital Twins."::: Here are the actions you will complete to set up this data flow:
-1. Create an Event Grid endpoint in Azure Digital Twins that connects the instance to Event Grid
-2. Set up a route within Azure Digital Twins to send twin property change events to the endpoint
-3. Deploy an Azure Functions app that listens (through [Event Grid](../event-grid/overview.md)) on the endpoint, and updates other twins accordingly
-4. Run the simulated device and query Azure Digital Twins to see the live results
+1. [Create an event grid topic](#create-the-event-grid-topic) to facilitate movement of data between Azure services
+1. [Create an endpoint](#create-the-endpoint) in Azure Digital Twins that connects the instance to the event grid topic
+1. [Set up a route](#create-the-route) within Azure Digital Twins that sends twin property change events to the endpoint
+1. [Set up an Azure function](#connect-the-azure-function) that listens on the event grid topic at the endpoint, receives the twin property change events that are sent there, and updates other twins in the graph accordingly
-### Set up endpoint
-[Event Grid](../event-grid/overview.md) is an Azure service that helps you route and deliver events coming from Azure Services to other places within Azure. You can create an [event grid topic](../event-grid/concepts.md) to collect certain events from a source, and then subscribers can listen on the topic to receive the events as they come through.
-
-In this section, you create an event grid topic, and then create an endpoint within Azure Digital Twins that points (sends events) to that topic.
-
-In Azure Cloud Shell, run the following command to create an event grid topic:
-
-```azurecli-interactive
-az eventgrid topic create --resource-group <your-resource-group> --name <name-for-your-event-grid-topic> --location <region>
-```
-
-> [!TIP]
-> To output a list of Azure region names that can be passed into commands in the Azure CLI, run this command:
-> ```azurecli-interactive
-> az account list-locations --output table
-> ```
-
-The output from this command is information about the event grid topic you've created.
-
-Next, create an Event Grid endpoint in Azure Digital Twins, which will connect your instance to your event grid topic. Use the command below, filling in the placeholder fields as necessary:
-
-```azurecli-interactive
-az dt endpoint create eventgrid --dt-name <your-Azure-Digital-Twins-instance> --eventgrid-resource-group <your-resource-group> --eventgrid-topic <your-event-grid-topic> --endpoint-name <name-for-your-Azure-Digital-Twins-endpoint>
-```
-
-The output from this command is information about the endpoint you've created.
-
-You can also verify that the endpoint creation succeeded by running the following command to query your Azure Digital Twins instance for this endpoint:
-
-```azurecli-interactive
-az dt endpoint show --dt-name <your-Azure-Digital-Twins-instance> --endpoint-name <your-Azure-Digital-Twins-endpoint>
-```
-
-Look for the `provisioningState` field in the output, and check that the value is "Succeeded". It may also say "Provisioning", meaning that the endpoint is still being created. In this case, wait a few seconds and run the command again to check that it has completed successfully.
--
-Save the names that you gave to your **event grid topic** and your Event Grid **endpoint** in Azure Digital Twins. You will use them later.
-
-### Set up route
-
-Next, create an Azure Digital Twins route that sends events to the Event Grid endpoint you just created.
-
-```azurecli-interactive
-az dt route create --dt-name <your-Azure-Digital-Twins-instance> --endpoint-name <your-Azure-Digital-Twins-endpoint> --route-name <name-for-your-Azure-Digital-Twins-route>
-```
-
-The output from this command is some information about the route you've created.
-
->[!NOTE]
->Endpoints (from the previous step) must be finished provisioning before you can set up an event route that uses them. If the route creation fails because the endpoints aren't ready, wait a few minutes and then try again.
-
-#### Connect the function to Event Grid
+### Connect the Azure function
Next, subscribe the *ProcessDTRoutedData* Azure function to the event grid topic you created earlier, so that telemetry data can flow from the thermostat67 twin through the event grid topic to the function, which goes back into Azure Digital Twins and updates the room21 twin accordingly.
On the *Create Event Subscription* page, fill in the fields as follows (fields f
Back on the *Create Event Subscription* page, select **Create**.
-### Run the simulation and see the results
+## Run the simulation and see the results
+
+Now, events should be able to flow from the simulated device into Azure Digital Twins, and through the Azure Digital Twins graph to update twins as appropriate. In this section, you'll run the device simulator again to kick off the full event flow you've set up, and query Azure Digital Twins to see the live results
-Now you can run the device simulator to kick off the new event flow you've set up. Go to your Visual Studio window where the _**DeviceSimulator**_ project is open, and run the project.
+Go to your Visual Studio window where the _**DeviceSimulator**_ project is open, and run the project.
Like when you ran the device simulator earlier, a console window will open and display simulated temperature telemetry messages. These events are going through the flow you set up earlier to update the thermostat67 twin, and then going through the flow you set up recently to update the room21 twin to match.
iot-hub Iot Hub Configure File Upload Cli https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/iot-hub/iot-hub-configure-file-upload-cli.md
The configuration requires the following values:
* **File notification maximum delivery count**: The number of times the IoT Hub attempts to deliver a file upload notification. Set to 10 by default.
+* **File notification lock duration**: The lock duration for the file notification queue. Set to 60 seconds by default.
+ * **Authentication type**: The type of authentication for IoT Hub to use with Azure Storage. This setting determines how your IoT hub authenticates and authorizes with Azure Storage. The default is key-based authentication; however, system-assigned and user-assigned managed identities can also be used. Managed identities provide Azure services with an automatically managed identity in Azure AD in a secure manner. To learn how to configure managed identities on your IoT hub and Azure Storage account, see [IoT Hub support for managed identities](./iot-hub-managed-identity.md). Once configured, you can set one of your managed identities to use for authentication with Azure storage. > [!NOTE]
az iot hub update --name {your iot hub name} \
--fileupload-sas-ttl 1 ```
-The following command enables file notifications and sets the file notification properties to their default values. (The file upload notification time to live is set to one hour.)
+The following command enables file notifications and sets the file notification properties to their default values. (The file upload notification time to live is set to one hour and the lock duration is set to 60 seconds.)
```azurecli az iot hub update --name {your iot hub name} \ --fileupload-notifications true \ --fileupload-notification-max-delivery-count 10 \ --fileupload-notification-ttl 1 \
- --set properties.messagingEndpoints.fileNotifications.lockDurationAsIso8601=PT0H1M0S
+ --fileupload-notification-lock-duration 60
```
-> [!NOTE]
-> The lock duration can only be set by using the `--set` parameter. There is not currently a named parameter available.
The following command configures key-based authentication:
logic-apps Logic Apps Limits And Config https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/logic-apps/logic-apps-limits-and-config.md
The following table lists the message size limits that apply to B2B protocols:
## Firewall configuration: IP addresses and service tags
-If your environment has strict network requirements or firewalls that limit traffic to specific IP addresses, your environment or firewall needs to allow access for *both* the [inbound](#inbound) and [outbound](#outbound) IP addresses used by the Azure Logic Apps service or runtime in the Azure region where your logic app resource exists. *All* logic apps in the same region use the same IP address ranges.
+If your environment has strict network requirements or firewalls that limit traffic to specific IP addresses, your environment or firewall needs to allow access for *both* the [inbound](#inbound) and [outbound](#outbound) IP addresses used by the Azure Logic Apps service or runtime in the Azure region where your logic app resource exists. To set up this access, you can create Azure Firewall [rules](/firewall/rule-processing). *All* logic apps in the same region use the same IP address ranges.
-For example, suppose your logic apps are deployed in the West US region. To support calls that your logic apps send or receive through built-in triggers and actions, such as the [HTTP trigger or action](../connectors/connectors-native-http.md), your firewall needs to allow access for *all* the Azure Logic Apps service inbound IP addresses *and* outbound IP addresses that exist in the West US region.
-
-If your workflow also uses [managed connectors](../connectors/managed.md), such as the Office 365 Outlook connector or SQL connector, or uses [custom connectors](/connectors/custom-connectors/), the firewall also needs to allow access for *all* the [managed connector outbound IP addresses](#outbound) in your logic app's Azure region.
+> [!NOTE]
+> If you're using [Power Automate](/power-automate/getting-started), some actions, such as **HTTP** and **HTTP + OpenAPI**,
+> go directly through the Azure Logic Apps service and come from the IP addresses that are listed here. For more information
+> about the IP addresses used by Power Automate, see [Limits and configuration for Power Automate](/flow/limits-and-config#ip-address-configuration).
-If you use custom connectors that access on-premises resources through the [on-premises data gateway resource in Azure](logic-apps-gateway-connection.md), you need to set up the gateway installation to allow access for the corresponding *managed connectors [outbound IP addresses](#outbound)*.
+For example, suppose your logic apps are deployed in the West US region. To support calls that your logic apps send or receive through built-in triggers and actions, such as the [HTTP trigger or action](../connectors/connectors-native-http.md), your firewall needs to allow access for *all* the Azure Logic Apps service inbound IP addresses *and* outbound IP addresses that exist in the West US region.
-For more information about setting up communication settings on the gateway, see these topics:
+If your workflow uses [managed connectors](../connectors/managed.md), such as the Office 365 Outlook connector or SQL connector, or uses [custom connectors](/connectors/custom-connectors/), the firewall also needs to allow access for *all* the [managed connector outbound IP addresses](/connectors/common/outbound-ip-addresses) in your logic app's Azure region. If your workflow uses custom connectors that access on-premises resources through the [on-premises data gateway resource in Azure](logic-apps-gateway-connection.md), you need to set up the gateway installation to allow access for the corresponding [*managed connector* outbound IP addresses](/connectors/common/outbound-ip-addresses). For more information about setting up communication settings on the gateway, review these topics:
* [Adjust communication settings for the on-premises data gateway](/data-integration/gateway/service-gateway-communication) * [Configure proxy settings for the on-premises data gateway](/data-integration/gateway/service-gateway-proxy)
+> [!IMPORTANT]
+> If you're using [Microsoft Azure operated by 21Vianet](/azure/china/), managed connectors and custom connectors don't have reserved or fixed IP addresses.
+> So, you can't set up firewall rules for logic apps that use these connectors in this cloud. For the Azure Logic Apps service IPs, review the
+> [documentation version for Azure operated by 21Vianet](https://docs.azure.cn/en-us/logic-apps/logic-apps-limits-and-config#firewall-ip-configuration).
+ <a name="ip-setup-considerations"></a> ### Firewall IP configuration considerations Before you set up your firewall with IP addresses, review these considerations:
-* If you're using [Power Automate](/power-automate/getting-started), some actions, such as **HTTP** and **HTTP + OpenAPI**, go directly through the Azure Logic Apps service and come from the IP addresses that are listed here. For more information about the IP addresses used by Power Automate, see [Limits and configuration for Power Automate](/flow/limits-and-config#ip-address-configuration).
-
-* For [Microsoft Azure operated by 21Vianet](/azure/china/), review the [documentation version for Azure operated by 21Vianet](https://docs.azure.cn/en-us/logic-apps/logic-apps-limits-and-config#firewall-ip-configuration).
- * If your logic app workflows run in single-tenant Azure Logic Apps, you need to find the fully qualified domain names (FQDNs) for your connections. For more information, review the corresponding sections in these topics: * [Firewall permissions for single tenant logic apps - Azure portal](create-single-tenant-workflows-azure-portal.md#firewall-setup)
Before you set up your firewall with IP addresses, review these considerations:
### Inbound IP addresses
-This section lists the inbound IP addresses for the Azure Logic Apps service only. If you have Azure Government, see [Azure Government - Inbound IP addresses](#azure-government-inbound).
+This section lists the inbound IP addresses for the Azure Logic Apps service only. If you're using Azure Government, see [Azure Government - Inbound IP addresses](#azure-government-inbound).
> [!TIP]
-> To help reduce complexity when you create security rules, you can optionally use the
-> [service tag](../virtual-network/service-tags-overview.md), **LogicAppsManagement**,
-> rather than specify inbound Logic Apps IP address prefixes for each region. Optionally,
-> you can also use the **AzureConnectors** service tag for managed connectors that make
-> inbound webhook callbacks to the Logic Apps service, rather than specify inbound managed
-> connector IP address prefixes for each region. These tags work across the regions where
-> the Logic Apps service is available.
+> To help reduce complexity when you create security rules, you can optionally use the [service tag](../virtual-network/service-tags-overview.md),
+> **LogicAppsManagement**, rather than specify inbound Logic Apps IP address prefixes for each region.
+>
+> Some managed connectors make inbound webhook callbacks to the Azure Logic Apps service. For these managed connectors, you can optionally use the
+> **AzureConnectors** service tag for these managed connectors, rather than specify inbound managed connector IP address prefixes for each region.
+> These tags work across the regions where the Logic Apps service is available.
> > The following connectors make inbound webhook callbacks to the Logic Apps service: >
-> Adobe Creative Cloud, Adobe Sign, Adobe Sign Demo, Adobe Sign Preview, Adobe Sign Stage,
-> Azure Sentinel, Business Central, Calendly, Common Data Service, DocuSign, DocuSign Demo,
-> Dynamics 365 for Fin & Ops, LiveChat, Office 365 Outlook, Outlook.com, Parserr, SAP*,
+> Adobe Creative Cloud, Adobe Sign, Adobe Sign Demo, Adobe Sign Preview, Adobe Sign Stage, Azure Sentinel, Business Central, Calendly,
+> Common Data Service, DocuSign, DocuSign Demo, Dynamics 365 for Fin & Ops, LiveChat, Office 365 Outlook, Outlook.com, Parserr, SAP*,
> Shifts for Microsoft Teams, Teamwork Projects, Typeform >
-> \* **SAP**: The return caller depends on whether the deployment environment is either
-> multi-tenant Azure or ISE. In the multi-tenant environment, the on-premises data gateway
-> makes the call back to the Logic Apps service. In an ISE, the SAP connector makes the
-> call back to the Logic Apps service.
+> \* **SAP**: The return caller depends on whether the deployment environment is either multi-tenant Azure or ISE. In the
+> multi-tenant environment, the on-premises data gateway makes the call back to the Logic Apps service. In an ISE, the SAP
+> connector makes the call back to the Logic Apps service.
<a name="multi-tenant-inbound"></a>
This section lists the inbound IP addresses for the Azure Logic Apps service onl
### Outbound IP addresses
-This section lists the outbound IP addresses for the Azure Logic Apps service and managed connectors. If you have Azure Government, see [Azure Government - Outbound IP addresses](#azure-government-outbound).
+This section lists the outbound IP addresses for the Azure Logic Apps service. If you're using Azure Government, see [Azure Government - Outbound IP addresses](#azure-government-outbound).
> [!TIP]
-> To help reduce complexity when you create security rules, you can optionally use the
-> [service tag](../virtual-network/service-tags-overview.md), **LogicApps**, rather than
-> specify outbound Logic Apps IP address prefixes for each region. Optionally, you can
-> also use the **AzureConnectors** service tag for managed connectors that make outbound
-> calls to their respective services, such as Azure Storage or Azure Event Hubs, rather than
-> specify outbound managed connector IP address prefixes for each region. These tags work
-> across the regions where the Logic Apps service is available.
+> To help reduce complexity when you create security rules, you can optionally use the [service tag](../virtual-network/service-tags-overview.md),
+> **LogicApps**, rather than specify outbound Logic Apps IP address prefixes for each region. Optionally, you can also use the **AzureConnectors**
+> service tag for managed connectors that make outbound calls to their respective services, such as Azure Storage or Azure Event Hubs, rather than
+> specify outbound managed connector IP address prefixes for each region. These tags work across the regions where the Logic Apps service is available.
<a name="multi-tenant-outbound"></a>
-#### Multi-tenant Azure - Outbound IP addresses
-
-| Multi-tenant region | Logic Apps IP | Managed connectors IP |
-|||--|
-| Australia East | 13.75.149.4, 104.210.91.55, 104.210.90.241, 52.187.227.245, 52.187.226.96, 52.187.231.184, 52.187.229.130, 52.187.226.139 | 52.237.214.72, 13.72.243.10, 13.70.72.192 - 13.70.72.207, 13.70.78.224 - 13.70.78.255, 20.70.220.192 - 20.70.220.223, 20.70.220.224 - 20.70.220.239 |
-| Australia Southeast | 13.73.114.207, 13.77.3.139, 13.70.159.205, 52.189.222.77, 13.77.56.167, 13.77.58.136, 52.189.214.42, 52.189.220.75 | 52.255.48.202, 13.70.136.174, 13.77.50.240 - 13.77.50.255, 13.77.55.160 - 13.77.55.191, 20.92.3.64 - 20.92.3.95, 20.92.3.96 - 20.92.3.111 |
-| Brazil South | 191.235.82.221, 191.235.91.7, 191.234.182.26, 191.237.255.116, 191.234.161.168, 191.234.162.178, 191.234.161.28, 191.234.162.131 | 191.232.191.157, 104.41.59.51, 191.233.203.192 - 191.233.203.207, 191.233.207.160 - 191.233.207.191, 191.238.76.112 - 191.238.76.127, 191.238.76.128 - 191.238.76.159 |
-| Brazil Southeast | 20.40.32.81, 20.40.32.19, 20.40.32.85, 20.40.32.60, 20.40.32.116, 20.40.32.87, 20.40.32.61, 20.40.32.113 | 23.97.120.109, 23.97.121.26, 20.206.0.0 - 20.206.0.63, 191.233.51.0 - 191.233.51.63 |
-| Canada Central | 52.233.29.92, 52.228.39.244, 40.85.250.135, 40.85.250.212, 13.71.186.1, 40.85.252.47, 13.71.184.150 | 52.237.32.212, 52.237.24.126, 13.71.170.208 - 13.71.170.223, 13.71.175.160 - 13.71.175.191, 20.48.200.192 - 20.48.200.223, 20.48.200.224 - 20.48.200.239 |
-| Canada East | 52.232.128.155, 52.229.120.45, 52.229.126.25, 40.86.203.228, 40.86.228.93, 40.86.216.241, 40.86.226.149, 40.86.217.241 | 52.242.30.112, 52.242.35.152, 40.69.106.240 - 40.69.106.255, 40.69.111.0 - 40.69.111.31, 52.139.111.0 - 52.139.111.31, 52.139.111.32 - 52.139.111.47 |
-| Central India | 52.172.154.168, 52.172.186.159, 52.172.185.79, 104.211.101.108, 104.211.102.62, 104.211.90.169, 104.211.90.162, 104.211.74.145 | 52.172.212.129, 52.172.211.12, 20.43.123.0 - 20.43.123.31, 104.211.81.192 - 104.211.81.207, 20.192.168.64 - 20.192.168.95, 20.192.168.96 - 20.192.168.111 |
-| Central US | 13.67.236.125, 104.208.25.27, 40.122.170.198, 40.113.218.230, 23.100.86.139, 23.100.87.24, 23.100.87.56, 23.100.82.16 | 52.173.241.27, 52.173.245.164, 13.89.171.80 - 13.89.171.95, 13.89.178.64 - 13.89.178.95, 40.77.68.110, 20.98.144.224 - 20.98.144.255, 20.98.145.0 - 20.98.145.15 |
-| East Asia | 13.75.94.173, 40.83.127.19, 52.175.33.254, 40.83.73.39, 65.52.175.34, 40.83.77.208, 40.83.100.69, 40.83.75.165 | 13.75.110.131, 52.175.23.169, 13.75.36.64 - 13.75.36.79, 104.214.164.0 - 104.214.164.31, 20.205.67.48 - 20.205.67.63, 20.205.67.64 - 20.205.67.95, 104.214.165.128 - 104.214.165.191 |
-| East US | 13.92.98.111, 40.121.91.41, 40.114.82.191, 23.101.139.153, 23.100.29.190, 23.101.136.201, 104.45.153.81, 23.101.132.208 | 40.71.249.139, 40.71.249.205, 40.114.40.132, 40.71.11.80 - 40.71.11.95, 40.71.15.160 - 40.71.15.191, 52.188.157.160, 20.88.153.176 - 20.88.153.191, 20.88.153.192 - 20.88.153.223 |
-| East US 2 | 40.84.30.147, 104.208.155.200, 104.208.158.174, 104.208.140.40, 40.70.131.151, 40.70.29.214, 40.70.26.154, 40.70.27.236 | 52.225.129.144, 52.232.188.154, 104.209.247.23, 40.70.146.208 - 40.70.146.223, 40.70.151.96 - 40.70.151.127, 40.65.220.25, 20.98.192.80 - 20.98.192.95, 20.98.192.96 - 20.98.192.127 |
-| France Central | 52.143.164.80, 52.143.164.15, 40.89.186.30, 20.188.39.105, 40.89.191.161, 40.89.188.169, 40.89.186.28, 40.89.190.104 | 40.89.186.239, 40.89.135.2, 40.79.130.208 - 40.79.130.223, 40.79.148.96 - 40.79.148.127, 51.138.215.48 - 51.138.215.63, 51.138.215.64 - 51.138.215.95 |
-| France South | 52.136.132.40, 52.136.129.89, 52.136.131.155, 52.136.133.62, 52.136.139.225, 52.136.130.144, 52.136.140.226, 52.136.129.51 | 52.136.142.154, 52.136.133.184, 40.79.178.240 - 40.79.178.255, 40.79.180.224 - 40.79.180.255, 52.136.189.16 - 52.136.189.31, 52.136.189.32 - 52.136.189.63 |
-| Germany North | 51.116.211.168, 51.116.208.165, 51.116.208.175, 51.116.208.192, 51.116.208.200, 51.116.208.222, 51.116.208.217, 51.116.208.51 | 51.116.60.192, 51.116.211.212, 51.116.59.16 - 51.116.59.31, 51.116.60.192 - 51.116.60.223, 51.116.55.240 - 51.116.55.255, 51.116.74.32 - 51.116.74.63 |
-| Germany West Central | 51.116.233.35, 51.116.171.49, 51.116.233.33, 51.116.233.22, 51.116.168.104, 51.116.175.17, 51.116.233.87, 51.116.175.51 | 51.116.158.97, 51.116.236.78, 51.116.155.80 - 51.116.155.95, 51.116.158.96 - 51.116.158.127, 20.52.93.80 - 20.52.93.95, 20.52.93.96 - 20.52.93.127 |
-| Japan East | 13.71.158.3, 13.73.4.207, 13.71.158.120, 13.78.18.168, 13.78.35.229, 13.78.42.223, 13.78.21.155, 13.78.20.232 | 13.73.21.230, 13.71.153.19, 13.78.108.0 - 13.78.108.15, 40.79.189.64 - 40.79.189.95, 20.89.11.48 - 20.89.11.63, 20.89.11.64 - 20.89.11.95 |
-| Japan West | 40.74.140.4, 104.214.137.243, 138.91.26.45, 40.74.64.207, 40.74.76.213, 40.74.77.205, 40.74.74.21, 40.74.68.85 | 104.215.27.24, 104.215.61.248, 40.74.100.224 - 40.74.100.239, 40.80.180.64 - 40.80.180.95, 20.189.192.144 - 20.189.192.159, 20.189.192.160 - 20.189.192.191 |
-| Korea Central | 52.231.14.11, 52.231.14.219, 52.231.15.6, 52.231.10.111, 52.231.14.223, 52.231.77.107, 52.231.8.175, 52.231.9.39 | 52.141.1.104, 52.141.36.214, 20.44.29.64 - 20.44.29.95, 52.231.18.208 - 52.231.18.223, 20.200.194.160 - 20.200.194.191, 20.200.194.192 - 20.200.194.207 |
-| Korea South | 52.231.204.74, 52.231.188.115, 52.231.189.221, 52.231.203.118, 52.231.166.28, 52.231.153.89, 52.231.155.206, 52.231.164.23 | 52.231.201.173, 52.231.163.10, 52.231.147.0 - 52.231.147.15, 52.231.148.224 - 52.231.148.255, 52.147.117.32 - 52.147.117.63, 52.147.117.64 - 52.147.117.79 |
-| North Central US | 168.62.248.37, 157.55.210.61, 157.55.212.238, 52.162.208.216, 52.162.213.231, 65.52.10.183, 65.52.9.96, 65.52.8.225 | 52.162.126.4, 52.162.242.161, 52.162.107.160 - 52.162.107.175, 52.162.111.192 - 52.162.111.223, 20.51.4.192 - 20.51.4.223, 20.51.4.224 - 20.51.4.239 |
-| North Europe | 40.113.12.95, 52.178.165.215, 52.178.166.21, 40.112.92.104, 40.112.95.216, 40.113.4.18, 40.113.3.202, 40.113.1.181 | 52.169.28.181, 52.178.150.68, 94.245.91.93, 13.69.227.208 - 13.69.227.223, 13.69.231.192 - 13.69.231.223, 40.115.108.29, 20.82.246.112 - 20.82.246.127, 52.146.138.32 - 52.146.138.63 |
-| Norway East | 51.120.88.52, 51.120.88.51, 51.13.65.206, 51.13.66.248, 51.13.65.90, 51.13.65.63, 51.13.68.140, 51.120.91.248 | 51.120.100.192, 51.120.92.27, 51.120.98.224 - 51.120.98.239, 51.120.100.192 - 51.120.100.223, 20.100.0.96 - 20.100.0.127, 20.100.0.128 - 20.100.0.143 |
-| South Africa North | 102.133.231.188, 102.133.231.117, 102.133.230.4, 102.133.227.103, 102.133.228.6, 102.133.230.82, 102.133.231.9, 102.133.231.51 | 102.133.168.167, 40.127.2.94, 102.133.155.0 - 102.133.155.15, 102.133.253.0 - 102.133.253.31, 102.37.166.80 - 102.37.166.95, 102.37.166.96 - 102.37.166.127 |
-| South Africa West | 102.133.72.98, 102.133.72.113, 102.133.75.169, 102.133.72.179, 102.133.72.37, 102.133.72.183, 102.133.72.132, 102.133.75.191 | 102.133.72.85, 102.133.75.194, 102.37.64.0 - 102.37.64.31, 102.133.27.0 - 102.133.27.15, 102.37.84.128 - 102.37.84.159, 102.37.84.160 - 102.37.84.175 |
-| South Central US | 104.210.144.48, 13.65.82.17, 13.66.52.232, 23.100.124.84, 70.37.54.122, 70.37.50.6, 23.100.127.172, 23.101.183.225 | 52.171.130.92, 13.65.86.57, 13.73.244.224 - 13.73.244.255, 104.214.19.48 - 104.214.19.63, 20.97.33.48 - 20.97.33.63, 20.97.33.64 - 20.97.33.95, 104.214.70.191 |
-| South India | 52.172.50.24, 52.172.55.231, 52.172.52.0, 104.211.229.115, 104.211.230.129, 104.211.230.126, 104.211.231.39, 104.211.227.229 | 13.71.127.26, 13.71.125.22, 20.192.184.32 - 20.192.184.63, 40.78.194.240 - 40.78.194.255, 20.192.152.64 - 20.192.152.95, 20.192.152.96 - 20.192.152.111, 52.172.80.0 - 52.172.80.63 |
-| Southeast Asia | 13.76.133.155, 52.163.228.93, 52.163.230.166, 13.76.4.194, 13.67.110.109, 13.67.91.135, 13.76.5.96, 13.67.107.128 | 52.187.115.69, 52.187.68.19, 13.67.8.240 - 13.67.8.255, 13.67.15.32 - 13.67.15.63, 20.195.82.240 - 20.195.82.255, 20.195.83.0 - 20.195.83.31 |
-| Switzerland North | 51.103.137.79, 51.103.135.51, 51.103.139.122, 51.103.134.69, 51.103.138.96, 51.103.138.28, 51.103.136.37, 51.103.136.210 | 51.103.142.22, 51.107.86.217, 51.107.59.16 - 51.107.59.31, 51.107.60.224 - 51.107.60.255, 51.107.246.112 - 51.107.246.127, 51.107.246.128 - 51.107.246.159 |
-| Switzerland West | 51.107.239.66, 51.107.231.86, 51.107.239.112, 51.107.239.123, 51.107.225.190, 51.107.225.179, 51.107.225.186, 51.107.225.151, 51.107.239.83 | 51.107.156.224, 51.107.231.190, 51.107.155.16 - 51.107.155.31, 51.107.156.224 - 51.107.156.255, 51.107.254.32 - 51.107.254.63, 51.107.254.64 - 51.107.254.79 |
-| UAE Central | 20.45.75.200, 20.45.72.72, 20.45.75.236, 20.45.79.239, 20.45.67.170, 20.45.72.54, 20.45.67.134, 20.45.67.135 | 20.45.67.45, 20.45.67.28, 20.37.74.192 - 20.37.74.207, 40.120.8.0 - 40.120.8.31, 20.45.90.208 - 20.45.90.223, 20.45.90.224 - 20.45.90.255 |
-| UAE North | 40.123.230.45, 40.123.231.179, 40.123.231.186, 40.119.166.152, 40.123.228.182, 40.123.217.165, 40.123.216.73, 40.123.212.104 | 65.52.250.208, 40.123.224.120, 40.120.64.64 - 40.120.64.95, 65.52.250.208 - 65.52.250.223, 40.120.86.16 - 40.120.86.31, 40.120.86.32 - 40.120.86.63 |
-| UK South | 51.140.74.14, 51.140.73.85, 51.140.78.44, 51.140.137.190, 51.140.153.135, 51.140.28.225, 51.140.142.28, 51.140.158.24 | 51.140.74.150, 51.140.80.51, 51.140.61.124, 51.105.77.96 - 51.105.77.127, 51.140.148.0 - 51.140.148.15, 20.90.129.0 - 20.90.129.31, 20.90.129.32 - 20.90.129.47 |
-| UK West | 51.141.54.185, 51.141.45.238, 51.141.47.136, 51.141.114.77, 51.141.112.112, 51.141.113.36, 51.141.118.119, 51.141.119.63 | 51.141.52.185, 51.141.47.105, 51.141.124.13, 51.140.211.0 - 51.140.211.15, 51.140.212.224 - 51.140.212.255, 20.58.70.192 - 20.58.70.223, 20.58.70.224 - 20.58.70.239 |
-| West Central US | 52.161.27.190, 52.161.18.218, 52.161.9.108, 13.78.151.161, 13.78.137.179, 13.78.148.140, 13.78.129.20, 13.78.141.75 | 52.161.101.204, 52.161.102.22, 13.78.132.82, 13.71.195.32 - 13.71.195.47, 13.71.199.192 - 13.71.199.223, 20.69.4.0 - 20.69.4.31, 20.69.4.32 - 20.69.4.47 |
-| West Europe | 40.68.222.65, 40.68.209.23, 13.95.147.65, 23.97.218.130, 51.144.182.201, 23.97.211.179, 104.45.9.52, 23.97.210.126, 13.69.71.160, 13.69.71.161, 13.69.71.162, 13.69.71.163, 13.69.71.164, 13.69.71.165, 13.69.71.166, 13.69.71.167 | 52.166.78.89, 52.174.88.118, 40.91.208.65, 13.69.64.208 - 13.69.64.223, 13.69.71.192 - 13.69.71.223, 13.93.36.78, 20.86.93.32 - 20.86.93.63, 20.86.93.64 - 20.86.93.79 |
-| West India | 104.211.164.80, 104.211.162.205, 104.211.164.136, 104.211.158.127, 104.211.156.153, 104.211.158.123, 104.211.154.59, 104.211.154.7 | 104.211.189.124, 104.211.189.218, 20.38.128.224 - 20.38.128.255, 104.211.146.224 - 104.211.146.239, 20.192.82.48 - 20.192.82.63, 20.192.82.64 - 20.192.82.95 |
-| West US | 52.160.92.112, 40.118.244.241, 40.118.241.243, 157.56.162.53, 157.56.167.147, 104.42.49.145, 40.83.164.80, 104.42.38.32, 13.86.223.0, 13.86.223.1, 13.86.223.2, 13.86.223.3, 13.86.223.4, 13.86.223.5 | 13.93.148.62, 104.42.122.49, 40.112.195.87, 13.86.223.32 - 13.86.223.63, 40.112.243.160 - 40.112.243.175, 20.59.77.0 - 20.59.77.31, 20.66.6.112 - 20.66.6.127 |
-| West US 2 | 13.66.210.167, 52.183.30.169, 52.183.29.132, 13.66.210.167, 13.66.201.169, 13.77.149.159, 52.175.198.132, 13.66.246.219 | 52.191.164.250, 52.183.78.157, 13.66.140.128 - 13.66.140.143, 13.66.145.96 - 13.66.145.127, 13.66.164.219, 20.83.220.208 - 20.83.220.223, 20.83.220.224 - 20.83.220.255 |
-||||
+#### Multi-tenant & single-tenant - Outbound IP addresses
+
+| Region | Logic Apps IP |
+|--||
+| Australia East | 13.75.149.4, 104.210.91.55, 104.210.90.241, 52.187.227.245, 52.187.226.96, 52.187.231.184, 52.187.229.130, 52.187.226.139 |
+| Australia Southeast | 13.73.114.207, 13.77.3.139, 13.70.159.205, 52.189.222.77, 13.77.56.167, 13.77.58.136, 52.189.214.42, 52.189.220.75 |
+| Brazil South | 191.235.82.221, 191.235.91.7, 191.234.182.26, 191.237.255.116, 191.234.161.168, 191.234.162.178, 191.234.161.28, 191.234.162.131 |
+| Brazil Southeast | 20.40.32.81, 20.40.32.19, 20.40.32.85, 20.40.32.60, 20.40.32.116, 20.40.32.87, 20.40.32.61, 20.40.32.113 |
+| Canada Central | 52.233.29.92, 52.228.39.244, 40.85.250.135, 40.85.250.212, 13.71.186.1, 40.85.252.47, 13.71.184.150 |
+| Canada East | 52.232.128.155, 52.229.120.45, 52.229.126.25, 40.86.203.228, 40.86.228.93, 40.86.216.241, 40.86.226.149, 40.86.217.241 |
+| Central India | 52.172.154.168, 52.172.186.159, 52.172.185.79, 104.211.101.108, 104.211.102.62, 104.211.90.169, 104.211.90.162, 104.211.74.145 |
+| Central US | 13.67.236.125, 104.208.25.27, 40.122.170.198, 40.113.218.230, 23.100.86.139, 23.100.87.24, 23.100.87.56, 23.100.82.16 |
+| East Asia | 13.75.94.173, 40.83.127.19, 52.175.33.254, 40.83.73.39, 65.52.175.34, 40.83.77.208, 40.83.100.69, 40.83.75.165 |
+| East US | 13.92.98.111, 40.121.91.41, 40.114.82.191, 23.101.139.153, 23.100.29.190, 23.101.136.201, 104.45.153.81, 23.101.132.208 |
+| East US 2 | 40.84.30.147, 104.208.155.200, 104.208.158.174, 104.208.140.40, 40.70.131.151, 40.70.29.214, 40.70.26.154, 40.70.27.236 |
+| France Central | 52.143.164.80, 52.143.164.15, 40.89.186.30, 20.188.39.105, 40.89.191.161, 40.89.188.169, 40.89.186.28, 40.89.190.104 |
+| France South | 52.136.132.40, 52.136.129.89, 52.136.131.155, 52.136.133.62, 52.136.139.225, 52.136.130.144, 52.136.140.226, 52.136.129.51 |
+| Germany North | 51.116.211.168, 51.116.208.165, 51.116.208.175, 51.116.208.192, 51.116.208.200, 51.116.208.222, 51.116.208.217, 51.116.208.51 |
+| Germany West Central | 51.116.233.35, 51.116.171.49, 51.116.233.33, 51.116.233.22, 51.116.168.104, 51.116.175.17, 51.116.233.87, 51.116.175.51 |
+| Japan East | 13.71.158.3, 13.73.4.207, 13.71.158.120, 13.78.18.168, 13.78.35.229, 13.78.42.223, 13.78.21.155, 13.78.20.232 |
+| Japan West | 40.74.140.4, 104.214.137.243, 138.91.26.45, 40.74.64.207, 40.74.76.213, 40.74.77.205, 40.74.74.21, 40.74.68.85 |
+| Korea Central | 52.231.14.11, 52.231.14.219, 52.231.15.6, 52.231.10.111, 52.231.14.223, 52.231.77.107, 52.231.8.175, 52.231.9.39 |
+| Korea South | 52.231.204.74, 52.231.188.115, 52.231.189.221, 52.231.203.118, 52.231.166.28, 52.231.153.89, 52.231.155.206, 52.231.164.23 |
+| North Central US | 168.62.248.37, 157.55.210.61, 157.55.212.238, 52.162.208.216, 52.162.213.231, 65.52.10.183, 65.52.9.96, 65.52.8.225 |
+| North Europe | 40.113.12.95, 52.178.165.215, 52.178.166.21, 40.112.92.104, 40.112.95.216, 40.113.4.18, 40.113.3.202, 40.113.1.181 |
+| Norway East | 51.120.88.52, 51.120.88.51, 51.13.65.206, 51.13.66.248, 51.13.65.90, 51.13.65.63, 51.13.68.140, 51.120.91.248 |
+| South Africa North | 102.133.231.188, 102.133.231.117, 102.133.230.4, 102.133.227.103, 102.133.228.6, 102.133.230.82, 102.133.231.9, 102.133.231.51 |
+| South Africa West | 102.133.72.98, 102.133.72.113, 102.133.75.169, 102.133.72.179, 102.133.72.37, 102.133.72.183, 102.133.72.132, 102.133.75.191 |
+| South Central US | 104.210.144.48, 13.65.82.17, 13.66.52.232, 23.100.124.84, 70.37.54.122, 70.37.50.6, 23.100.127.172, 23.101.183.225 |
+| South India | 52.172.50.24, 52.172.55.231, 52.172.52.0, 104.211.229.115, 104.211.230.129, 104.211.230.126, 104.211.231.39, 104.211.227.229 |
+| Southeast Asia | 13.76.133.155, 52.163.228.93, 52.163.230.166, 13.76.4.194, 13.67.110.109, 13.67.91.135, 13.76.5.96, 13.67.107.128 |
+| Switzerland North | 51.103.137.79, 51.103.135.51, 51.103.139.122, 51.103.134.69, 51.103.138.96, 51.103.138.28, 51.103.136.37, 51.103.136.210 |
+| Switzerland West | 51.107.239.66, 51.107.231.86, 51.107.239.112, 51.107.239.123, 51.107.225.190, 51.107.225.179, 51.107.225.186, 51.107.225.151, 51.107.239.83 |
+| UAE Central | 20.45.75.200, 20.45.72.72, 20.45.75.236, 20.45.79.239, 20.45.67.170, 20.45.72.54, 20.45.67.134, 20.45.67.135 |
+| UAE North | 40.123.230.45, 40.123.231.179, 40.123.231.186, 40.119.166.152, 40.123.228.182, 40.123.217.165, 40.123.216.73, 40.123.212.104 |
+| UK South | 51.140.74.14, 51.140.73.85, 51.140.78.44, 51.140.137.190, 51.140.153.135, 51.140.28.225, 51.140.142.28, 51.140.158.24 |
+| UK West | 51.141.54.185, 51.141.45.238, 51.141.47.136, 51.141.114.77, 51.141.112.112, 51.141.113.36, 51.141.118.119, 51.141.119.63 |
+| West Central US | 52.161.27.190, 52.161.18.218, 52.161.9.108, 13.78.151.161, 13.78.137.179, 13.78.148.140, 13.78.129.20, 13.78.141.75 |
+| West Europe | 40.68.222.65, 40.68.209.23, 13.95.147.65, 23.97.218.130, 51.144.182.201, 23.97.211.179, 104.45.9.52, 23.97.210.126, 13.69.71.160, 13.69.71.161, 13.69.71.162, 13.69.71.163, 13.69.71.164, 13.69.71.165, 13.69.71.166, 13.69.71.167 |
+| West India | 104.211.164.80, 104.211.162.205, 104.211.164.136, 104.211.158.127, 104.211.156.153, 104.211.158.123, 104.211.154.59, 104.211.154.7 |
+| West US | 52.160.92.112, 40.118.244.241, 40.118.241.243, 157.56.162.53, 157.56.167.147, 104.42.49.145, 40.83.164.80, 104.42.38.32, 13.86.223.0, 13.86.223.1, 13.86.223.2, 13.86.223.3, 13.86.223.4, 13.86.223.5 |
+| West US 2 | 13.66.210.167, 52.183.30.169, 52.183.29.132, 13.66.210.167, 13.66.201.169, 13.77.149.159, 52.175.198.132, 13.66.246.219 |
+|||
<a name="azure-government-outbound"></a> #### Azure Government - Outbound IP addresses
-| Region | Logic Apps IP | Managed connectors IP |
-|--||--|
-| US DoD Central | 52.182.48.215, 52.182.92.143 | 52.127.58.160 - 52.127.58.175, 52.182.54.8, 52.182.48.136, 52.127.61.192 - 52.127.61.223, 52.245.153.80 - 52.245.153.95, 52.245.153.96 - 52.245.153.127 |
-| US Gov Arizona | 52.244.67.143, 52.244.65.66, 52.244.65.190 | 52.127.2.160 - 52.127.2.175, 52.244.69.0, 52.244.64.91, 52.127.5.224 - 52.127.5.255, 20.141.9.240 - 20.141.9.255, 20.141.10.0 - 20.141.10.31 |
-| US Gov Texas | 52.238.114.217, 52.238.115.245, 52.238.117.119 | 52.127.34.160 - 52.127.34.175, 40.112.40.25, 52.238.161.225, 20.140.137.128 - 20.140.137.159, 20.140.146.192 - 20.140.146.223, 20.140.146.224 - 20.140.146.239 |
-| US Gov Virginia | 13.72.54.205, 52.227.138.30, 52.227.152.44 | 52.127.42.128 - 52.127.42.143, 52.227.143.61, 52.227.162.91, 20.140.94.192 - 20.140.94.223, 52.235.252.144 - 52.235.252.159, 52.235.252.160 - 52.235.252.191 |
-||||
+| Region | Logic Apps IP |
+|--||
+| US DoD Central | 52.182.48.215, 52.182.92.143 |
+| US Gov Arizona | 52.244.67.143, 52.244.65.66, 52.244.65.190 |
+| US Gov Texas | 52.238.114.217, 52.238.115.245, 52.238.117.119 |
+| US Gov Virginia | 13.72.54.205, 52.227.138.30, 52.227.152.44 |
+|||
## Next steps
logic-apps Parameterize Workflow App https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/logic-apps/parameterize-workflow-app.md
Title: Create parameters for workflows in single-tenant Azure Logic Apps
-description: Define parameters for values that differ in workflows across deployment environments in single-tenant Azure Logic Apps.
+ Title: Create parameters for values in single-tenant workflows
+description: Define parameters for values in workflows that differ across deployment environments for single-tenant Azure Logic Apps.
ms.suite: integration Previously updated : 06/08/2021 Last updated : 07/30/2021
-# Create parameters for values that change in workflows across environments for single-tenant Azure Logic Apps
+# Create parameters to use in workflows across environments in single-tenant Azure Logic Apps
-In Azure Logic Apps, you can use parameters to abstract values that might change between environments. By defining parameters to use in your workflows, you can first focus on designing your workflows, and then insert your environment-specific variables later.
+In Azure Logic Apps, you can abstract values that might change in your workflows across your development, test, and production environments by defining *parameters*. When you use parameters instead, you can focus more on designing your workflows, and insert your environment-specific variables later.
-In *multi-tenant* Azure Logic Apps, you can create and reference parameters in the workflow designer, and then set the variables in your Azure Resource Manager (ARM) template and parameters files. Parameters are defined and set at deployment. So, even if you need to only change one variable, you have to redeploy your logic app's ARM template.
+This article introduces how parameters work in single-tenant Azure Logic Apps and how to edit, reference, and manage environment variables using the single-tenant parameters experience.
-In *single-tenant* Azure Logic Apps, you can work with environment variables both at runtime and deployment time by using parameters and app settings. This article shows how to edit, call, and reference environment variables with the new single-tenant parameters experience.
+<a name="parameters-introduction"></a>
+
+## Parameters in single-tenant versus multi-tenant
+
+If you've worked with *multi-tenant* Azure Logic Apps, you might already be familiar with parameters. Parameterizing your workflow inputs in single-tenant Azure Logic Apps works similarly to multi-tenant Azure Logic Apps but with a major difference.
+
+For example, in both the single-tenant and multi-tenant service, you can define parameters when you're working in workflow designer. After you define the parameter, you can reference that parameter from any workflow or connection that's in the *same* logic app resource. However, in the *multi-tenant* service, after you create and use parameters in the designer, you define and set the environment variables in your Azure Resource Manager template (ARM template) and template parameters files. In this scenario, you have to define and set the parameters *at deployment*, which means that even if you only have to change one variable, you have to redeploy your logic app's ARM template.
+
+By comparison, with the *single-tenant* service, you can work with environment variables both at runtime and deployment by using parameters *and* app settings. In single-tenant Azure Logic Apps, app settings contain global configuration options for *all the workflows* in the same logic app. For more information, review [Edit host and app settings for single-tenant based logic apps](edit-app-settings-host-settings.md).
+
+For example, you can use app settings to integrate with Azure Key Vault and [directly reference secure strings](../app-service/app-service-key-vault-references.md), such as connection strings and keys. Similar to ARM templates, where you can define environment variables at deployment time, you can define app settings within your [logic app workflow definition](/azure/templates/microsoft.logic/workflows). You can then capture dynamically generated infrastructure values, such as connection endpoints, storage strings, and more.
+
+However, app settings have size limits and can't be referenced from certain areas in Azure Logic Apps. Parameters offers a wider range of use cases than app settings, such as support for large value sizes and complex objects.
+
+For example, if you use Visual Studio Code as your local development tool to run workflows locally, in your logic app project, you can define parameters using the **parameters.json** file. You can then reference any parameter in this parameters file from any workflow in your project's **workflow.json** file or from any connection object in your project's **connections.json** file. The following list describes a couple common use cases:
+
+* Have a test parameters file that includes all the values that you use during testing. At deployment, you can replace your test parameters file with your production parameters file.
+
+* Parameterize different parts of your **connections.json** file. You can then check your **connections.json** file into source control, and then manage any connections through your **parameters.json** file.
+
+* Parameterize complex objects, such as the `authentication` JSON object. For example, you can replace the `authentication` object value with a string that holds a single parameters expression, such as `@parameters('api-auth')`.
+
+* Review and edit the app settings in your project's **local.settings.json** file. You can then reference these app settings in your parameters.
+
+> [!NOTE]
+> As a general recommendation, consider using parameters as the default way to parameterize values,
+> not app settings. That way, when you need to store secure keys or strings, you can follow the
+> recommendation to reference app settings from your parameters. If you want, you can use both
+> options in your solution by using parameters to reference app settings.
## Prerequisites -- An Azure account and subscription. If you don't have a subscription, [sign up for a free Azure account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).
+* An Azure account and subscription. If you don't have a subscription, [sign up for a free Azure account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).
-- A [logic app workflow hosted in single-tenant Azure Logic Apps](single-tenant-overview-compare.md).
+* A [logic app workflow hosted in single-tenant Azure Logic Apps](single-tenant-overview-compare.md).
If you don't have a logic app, [create your logic app (Standard) in the Azure portal](create-single-tenant-workflows-azure-portal.md) or [in Visual Studio Code](create-single-tenant-workflows-visual-studio-code.md).
-## Parameters versus app settings
+## Define, use, and edit parameters
-Before you decide where to store your environment variables, review the following information.
+### Azure portal
-If you already use Azure Functions or Azure Web Apps, you might be familiar with app settings. In Azure Logic Apps, app settings integrate with Azure Key Vault. You can [directly reference secure strings](../app-service/app-service-key-vault-references.md), such as connection strings and keys. Similar to ARM templates, where you can define environment variables at deployment time, you can define app settings within your [logic app workflow definition](/azure/templates/microsoft.logic/workflows). You can then capture dynamically generated infrastructure values, such as connection endpoints, storage strings, and more. However, app settings have size limitations and can't be referenced from certain areas in Azure Logic Apps.
+1. In the [Azure portal](https://portal.azure.com), open your single-tenant based logic app. Under **Workflows**, select and open your workflow in the designer.
-If you're familiar with workflows in multi-tenant Azure Logic Apps, you might also be familiar with parameters. You can use parameters in a wider range of use cases than app settings, such as supporting complex objects and values of large sizes. If you use Visual Studio Code as your local development tool, you can also reference parameters in your logic app project's **workflow.json** and **connections.json** files. If you want to use both options in your solution, you can also reference app settings using parameters.
+1. From the designer toolbar, select **Parameters**.
-> [!NOTE]
-> For development, if you parameterize the **connections.json** file, the designer experience becomes restricted, both locally and in the Azure portal.
-> If you need to use the designer for development, use a non-parameterized **connections.json** file. Then, in your deployment pipelines, replace with
-> the parameterized file. The runtime still works with parameterization. Designer improvements are in development.
+ ![Screenshot showing Azure portal, workflow designer, and "Parameters" on designer toolbar selected.](./media/parameterize-workflow-app/portal-designer-select-parameters.png)
-Consider the recommendation to use parameters as the default mechanism for parameterization. That way, when you need to store secure keys or strings, you can follow the recommendation to reference app settings from your parameters.
+1. On the **Parameters** pane, select **Create parameter**.
-## What is parameterization?
+1. Provide the following information about the parameter to create:
-If you use Visual Studio Code, in your logic app project, you can define parameters in the **parameters.json** file. You can reference any parameter in **parameters.json** file from any workflow or connection object in your logic app. Parameterizing your workflow inputs in single-tenant Azure Logic Apps works similarly to multi-tenant Azure Logic Apps.
+ | Property | Required | Description |
+ |-|-|-|
+ | **Name** | Yes | The name for the parameter to create. |
+ | **Type** | Yes | The data type for the parameter, such as **Array**, **Bool**, **Float**, **Int**, **Object**, and **String**. <p><p>**Note**: In single-tenant based workflows, secure data types, such as `securestring` and `secureobject` aren't supported. |
+ | **Value** | Yes | The value for the parameter. <p><p>In single-tenant Azure Logic Apps, you have to specify the parameter value because the the workflow logic, connection information, and parameter values don't exist in a single location. The designer must be able to resolve parameter value before loading. |
+ |||
-To reference parameters in your trigger or action inputs, use the expression `@parameters('<parameter-name>')`.
+ The following example shows a definition for a string parameter:
-> [!IMPORTANT]
-> Make sure that you also include any parameters that you reference in your **parameters.json** file.
+ ![Screenshot showing Azure portal, workflow designer, and the "Parameters" pane with an example parameter definition.](./media/parameterize-workflow-app/define-parameter.png)
-In *single-tenant* Azure Logic Apps, you can parameterize different parts of your **connections.json** file. You can then check your **connections.json** file into source control, and then manage any connections through your **parameters.json** file. To parameterize your **connections.json** file, replace the values for literals, such as `ConnectionRuntimeUrl`, with a single `parameters()` expression, for example, `@parameters('api-runtimeUrl')`.
+1. When you're done, close the **Parameters** pane, but make sure to save your workflow to save your new parameter definition.
-You can also parameterize complex objects, such as the `authentication` JSON object. For example, replace the `authentication` object value with a string that holds a single parameters expression, such as `@parameters('api-auth')`.
+1. To reference the parameter from a trigger or action that's in any workflow within the same logic app, follow these steps:
-> [!NOTE]
-> The only valid expression types in the **connections.json** file are `@parameters` and `@appsetting`.
+ 1. In the designer, open the workflow that you want, and expand the trigger or action.
-## Define parameters
+ 1. In the property where you want to use the parameter, click inside that property's edit box.
-In single-tenant based workflows, you need to put all parameter values in a root-level JSON file named **parameters.json**. This file contains an object that contains key-value pairs. The keys are the names of each parameter, and the values are the structures for each parameter. Each structure needs to include both a `type` and `value` declaration.
+ 1. From the dynamic content list that opens, under **Parameters**, select you previously created parameter, for example:
-> [!NOTE]
-> The only valid expression type in the **parameters.json** file is `@appsetting`.
+ ![Screenshot showing expanded example action with the cursor inside property edit box, the expanded dynamic content list, and the previously created parameter selected.](./media/parameterize-workflow-app/reference-parameter.png)
+
+1. To view or edit parameters in the same logic app, follow either step:
+
+ * Open any workflow in that logic app. On the workflow menu, select **Designer**. On the designer toolbar, select **Parameters**.
+
+ The **Parameters** pane opens and displays all the parameters that you defined from workflows in that logic app.
+
+ * To view or edit in bulk JSON, on your logic app's main menu, select **Parameters**.
-The following example shows a basic parameters file:
+ The **Parameters** JSON view opens and displays all the parameters that you defined from workflows in that logic app.
+
+### Visual Studio Code
+
+1. In a project root-level JSON file named **parameters.json**, define *all* the parameters and their values. This file has an object that includes *key-value* pairs. Each *key* is the name for each parameter, while each *value* is the structure for each parameter. Each structure needs to include both a `type` and `value` declaration.
+
+ > [!IMPORTANT]
+ > Your **parameters.json** file must define and include all the parameters and their values that you
+ > reference or use elsewhere in your project, for example, in workflow definitions or connections.
+
+ The following example shows a basic parameters file:
+
+ ```json
+ {
+ "responseString":ΓÇ»{
+ "type":ΓÇ»"string",
+ "value":ΓÇ»"hello"
+ },
+ "functionAuth":ΓÇ»{
+ "type":ΓÇ»"object",
+ "value":ΓÇ»{
+ "type":ΓÇ»"QueryString",
+ "name":ΓÇ»"Code",
+ "value":ΓÇ»"@appsetting('<AzureFunctionsOperation-FunctionAppKey>')"
+ }
+ }
+ }
+ ```
+
+ > [!NOTE]
+ > In the **parameters.json** file, `@appsetting` is the only valid expression type.
+
+1. To reference parameters in your trigger or action inputs, use the expression `@parameters('<parameter-name>')`.
+
+#### Parameterize connections file
+
+To parameterize your **connections.json** file, replace the values for literals, such as `ConnectionRuntimeUrl`, with a single `parameters()` expression, for example, `@parameters('api-runtimeUrl')`. In the **connections.json** file, the only valid expression types are `@parameters` and `@appsetting`.
+
+> [!IMPORTANT]
+>
+> If you parameterize the **connections.json** file during development, the designer experience becomes restricted,
+> both locally and in the Azure portal. If you need to use the designer for development, use a non-parameterized
+> **connections.json** file instead. Then, in your deployment pipelines, replace with the parameterized file.
+> The runtime still works with parameterization. Designer improvements are in development.
+
+The following example shows a parameterized **connections.json** file that uses both app settings and parameters. Although you want to use parameters where possible, this scenario is an exception or edge case where you'd use app settings over parameters because app settings are generated during deployment and are easier to dynamically populate in a development pipeline. This sample file uses a parameter for the complex `blob_auth` authentication object and app settings for the other values. In this case, you can use a parameter for the authentication object as you're unlikely to reference the parameter in your workflow:
```json
-{
- "responseString":ΓÇ»{
- "type":ΓÇ»"string",
- "value":ΓÇ»"hello"
- },
- "functionAuth":ΓÇ»{
- "type":ΓÇ»"object",
- "value":ΓÇ»{
- "type":ΓÇ»"QueryString",
- "name":ΓÇ»"Code",
- "value":ΓÇ»"@appsetting('<AzureFunctionsOperation-FunctionAppKey')"
- }
- }
-}
+{
+ "serviceProviderConnections": {
+ "serviceBus": {
+ "parameterValues": {
+ "connectionString": "@appsetting('serviceBus_connectionString')"
+ },
+ "serviceProvider": {
+ "id": "/serviceProviders/serviceBus"
+ },
+ "displayName": "servicebus"
+ }
+ },
+ "managedApiConnections": {
+ "azureblob": {
+ "api": {
+ "id": "/subscriptions/@appsetting('WORKFLOWS_SUBSCRIPTION_ID')/providers/Microsoft.Web/locations/@appsetting('WORKFLOWS_LOCATION_NAME')/managedApis/azureblob"
+ },
+ "connection": {
+ "id": "/subscriptions/@appsetting('WORKFLOWS_SUBSCRIPTION_ID')/resourceGroups/@appsetting('WORKFLOWS_RESOURCE_GROUP_NAME')/providers/Microsoft.Web/connections/azureblob"
+ },
+ "connectionRuntimeUrl": "@appsetting('BLOB_CONNECTION_RUNTIMEURL')",
+ "authentication": "@parameters('blob_auth')"
+ }
+ }
+}
```
-Typically, you need to manage multiple versions of parameter files. You might have targeted values for different deployment environments, such as development, testing, and production. Managing these parameter files often works like managing ARM template parameter files. When you deploy to a specific environment, you promote the corresponding parameter file, generally through a pipeline for DevOps.
+## Manage parameters files
+
+Typically, you need to manage multiple versions of parameters files. You might have targeted values for different deployment environments, such as development, testing, and production. Managing these parameters files often works like managing ARM template parameters files. When you deploy to a specific environment, you promote the corresponding parameters file, generally through a pipeline for DevOps.
-To replace parameter files dynamically using the Azure CLI, run the following command:
+To dynamically replace parameters files using the Azure CLI, run the following command:
```azurecli az functionapp deploy --resource-group MyResourceGroup --name MyLogicApp --src-path C:\parameters.json --type static --target-path parameters.json ```
-If you have a NuGet-based Logic App project, you have to update your project file (**&lt;logic-app-name&gt;.csproj**) to include the parameters file in the build output, for example:
-
+If you have a NuGet-based logic app project, you have to update your project file (**&lt;logic-app-name&gt;.csproj**) to include the parameters file in the build output, for example:
+ ```csproj <ItemGroup> <None Update="parameters.json">
If you have a NuGet-based Logic App project, you have to update your project fil
``` > [!NOTE]
-> Currently, the capability to dynamically replace parameter files is not yet available in the Azure portal or the workflow designer.
+> Currently, the capability to dynamically replace parameters files is not yet available in the Azure portal or the workflow designer.
For more information about setting up your logic apps for DevOps deployments, review the following documentation: -- [DevOps deployment overview for single-tenant based logic apps](devops-deployment-single-tenant-azure-logic-apps.md)-- [Set up DevOps deployment for single-tenant based logic apps](set-up-devops-deployment-single-tenant-azure-logic-apps.md)
+* [DevOps deployment overview for single-tenant based logic apps](devops-deployment-single-tenant-azure-logic-apps.md)
+* [Set up DevOps deployment for single-tenant based logic apps](set-up-devops-deployment-single-tenant-azure-logic-apps.md)
## Manage app settings
-In single-tenant Azure Logic Apps, app settings contain global configuration options for *all the workflows* in the same logic app. When you run workflows locally in Visual Studio Code, these settings are accessible as local environment variables in the **local.settings.json** file. You can then reference these app settings in your parameters.
+In single-tenant Azure Logic Apps, app settings contain global configuration options for *all the workflows* in the same logic app. When you run workflows locally in Visual Studio Code, you can access these app settings as local environment variables in the **local.settings.json** file. You can then reference these app settings in your parameters.
To add, update, or delete app settings, select and review the following sections for Visual Studio Code, Azure portal, Azure CLI, or ARM (Bicep) template.
To add, update, or delete app settings, select and review the following sections
To review the app settings for your logic app in the Azure portal, follow these steps:
-1. In the [Azure portal](https://portal.azure.com/) search box, find and open your single-tenant based logic app.
+1. In the [Azure portal](https://portal.azure.com/), open your single-tenant based logic app.
+ 1. On your logic app menu, under **Settings**, select **Configuration**.+ 1. On the **Configuration** page, on the **Application settings** tab, review the app settings for your logic app.+ 1. To view all values, select **Show Values**. Or, to view a single value, select that value. To add a new setting, follow these steps: 1. On the **Application settings** tab, under **Application settings**, select **New application setting**.+ 1. For **Name**, enter the *key* or name for your new setting.+ 1. For **Value**, enter the value for your new setting.+ 1. When you're ready to create your new *key-value* pair, select **OK**. :::image type="content" source="./media/parameterize-workflow-app/portal-app-settings-values.png" alt-text="Screenshot showing the Azure portal and the configuration pane with the app settings and values for a single-tenant based logic app." lightbox="./media/parameterize-workflow-app/portal-app-settings-values.png":::
This example shows file settings for either ARM templates or Bicep templates:
## Next steps > [!div class="nextstepaction"]
-> [Single-tenant vs. multi-tenant and integration service environment for Azure Logic Apps](single-tenant-overview-compare.md)
+> [Single-tenant versus multi-tenant and integration service environment for Azure Logic Apps](single-tenant-overview-compare.md)
machine-learning Concept Compute Instance https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/machine-learning/concept-compute-instance.md
For production grade model training, use an [Azure Machine Learning compute clus
For compute instance Jupyter functionality to work, ensure that web socket communication is not disabled. Please ensure your network allows websocket connections to *.instances.azureml.net and *.instances.azureml.ms.
+> [!IMPORTANT]
+> Items marked (preview) in this article are currently in public preview.
+> The preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.
+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
+ ## Why use a compute instance? A compute instance is a fully managed cloud-based workstation optimized for your machine learning development environment. It provides the following benefits:
A compute instance is a fully managed cloud-based workstation optimized for your
|Preconfigured&nbsp;for&nbsp;ML|Save time on setup tasks with pre-configured and up-to-date ML packages, deep learning frameworks, GPU drivers.| |Fully customizable|Broad support for Azure VM types including GPUs and persisted low-level customization such as installing packages and drivers makes advanced scenarios a breeze. |
-You can [create a compute instance](how-to-create-manage-compute-instance.md?tabs=python#create) yourself, or an administrator can **[create a compute instance on your behalf](how-to-create-manage-compute-instance.md?tabs=python#on-behalf)**.
-
-You can also **[use a setup script (preview)](how-to-create-manage-compute-instance.md#setup-script)** for an automated way to customize and configure the compute instance as per your needs.
-
-Compute instance is also a secure training compute target similar to compute clusters but it is single node.
+* The compute instance is also a secure training compute target similar to compute clusters, but it is single node.
+* You can [create a compute instance](how-to-create-manage-compute-instance.md?tabs=python#create) yourself, or an administrator can **[create a compute instance on your behalf](how-to-create-manage-compute-instance.md?tabs=python#on-behalf)**.
+* You can also **[use a setup script (preview)](how-to-create-manage-compute-instance.md#setup-script)** for an automated way to customize and configure the compute instance as per your needs.
+* To save on costs, **[create a schedule (preview)](how-to-create-manage-compute-instance.md#schedule)** to automatically start and stop the compute instance (preview).
## <a name="contents"></a>Tools and environments
-> [!IMPORTANT]
-> Items marked (preview) in this article are currently in public preview.
-> The preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.
-> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
- Azure Machine Learning compute instance enables you to author, train, and deploy models in a fully integrated notebook experience in your workspace. You can run Jupyter notebooks in [VS Code](https://techcommunity.microsoft.com/t5/azure-ai/power-your-vs-code-notebooks-with-azml-compute-instances/ba-p/1629630) using compute instance as the remote server with no SSH needed. You can also enable VS Code integration through [remote SSH extension](https://devblogs.microsoft.com/python/enhance-your-azure-machine-learning-experience-with-the-vs-code-extension/).
Writing small files can be slower on network drives than writing to the compute
Do not store training data on the notebooks file share. You can use the `/tmp` directory on the compute instance for your temporary data. However, do not write very large files of data on the OS disk of the compute instance. OS disk on compute instance has 128 GB capacity. You can also store temporary training data on temporary disk mounted on /mnt. Temporary disk size is configurable based on the VM size chosen and can store larger amounts of data if a higher size VM is chosen. You can also mount [datastores and datasets](concept-azure-machine-learning-architecture.md#datasets-and-datastores). Any software packages you install are saved on the OS disk of compute instance. Please note customer managed key encryption is currently not supported for OS disk. The OS disk for compute instance is encrypted with Microsoft-managed keys. --
-## Managing a compute instance
-
-In your workspace in Azure Machine Learning studio, select **Compute**, then select **Compute Instance** on the top.
-
-![Manage a compute instance](./media/concept-compute-instance/manage-compute-instance.png)
-
-For more about managing the compute instance, see [Create and manage an Azure Machine Learning compute instance](how-to-create-manage-compute-instance.md).
- ### <a name="create"></a>Create a compute instance As an administrator, you can **[create a compute instance for others in the workspace (preview)](how-to-create-manage-compute-instance.md#on-behalf)**. You can also **[use a setup script (preview)](how-to-create-manage-compute-instance.md#setup-script)** for an automated way to customize and configure the compute instance.
-To create your a compute instance for yourself, use your workspace in Azure Machine Learning studio, [create a new compute instance](how-to-create-attach-compute-studio.md#compute-instance) from either the **Compute** section or in the **Notebooks** section when you are ready to run one of your notebooks.
+To create your a compute instance for yourself, use your workspace in Azure Machine Learning studio, [create a new compute instance](how-to-create-manage-compute-instance.md?tabs=azure-studio#create) from either the **Compute** section or in the **Notebooks** section when you are ready to run one of your notebooks.
You can also create an instance * Directly from the [integrated notebooks experience](tutorial-train-models-with-aml.md#azure)
machine-learning Concept Plan Manage Cost https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/machine-learning/concept-plan-manage-cost.md
For more information, see [Azure Machine Learning pricing](https://azure.microso
Azure Machine Learning runs on Azure infrastructure that accrues costs along with Azure Machine Learning when you deploy the new resource. It's important to understand that additional infrastructure might accrue cost. You need to manage that cost when you make changes to deployed resources. ---- ### Costs that typically accrue with Azure Machine Learning When you create resources for an Azure Machine Learning workspace, resources for other Azure services are also created. They are:
When you create resources for an Azure Machine Learning workspace, resources for
* [Azure Block Blob Storage](https://azure.microsoft.com/pricing/details/storage/blobs?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) (general purpose v1) * [Key Vault](https://azure.microsoft.com/pricing/details/key-vault?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) * [Application Insights](https://azure.microsoft.com/en-us/pricing/details/monitor?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn)+
+When you create a [compute instance](concept-compute-instance.md), the VM stays on so it is available for your work. [Set up a schedule](how-to-create-manage-compute-instance.md#schedule) to automatically start and stop the compute instance (preview) to save cost when you aren't planning to use it.
### Costs might accrue before resource deletion
Use the following tips to help you manage and optimize your compute resource cos
- Set quotas on your subscription and workspaces - Set termination policies on your training run - Use low-priority virtual machines (VM)
+- Schedule compute instances to shut down and start up automatically
- Use an Azure Reserved VM Instance - Train locally - Parallelize training
machine-learning How To Create Attach Compute Studio https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/machine-learning/how-to-create-attach-compute-studio.md
In this article, learn how to create and manage compute targets in Azure Machine
* [Other compute resources](how-to-attach-compute-targets.md) * The [VS Code extension](how-to-manage-resources-vscode.md#compute-clusters) for Azure Machine Learning.
+> [!IMPORTANT]
+> Items marked (preview) in this article are currently in public preview.
+> The preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.
+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
## Prerequisites
Follow the previous steps to view the list of compute targets. Then use these st
1. Fill out the form for your compute type:
- * [Compute instance](#compute-instance)
+ * [Compute instance](how-to-create-manage-compute-instance.md?tabs=azure-studio#create)
* [Compute clusters](#amlcompute) * [Inference clusters](#inference-clusters) * [Attached compute](#attached-compute)
Follow the previous steps to view the list of compute targets. Then use these st
:::image type="content" source="media/how-to-create-attach-studio/view-list.png" alt-text="View compute status from a list":::
+Follow the steps in [Create and manage an Azure Machine Learning compute instance](how-to-create-manage-compute-instance.md?tabs=azure-studio#create).
-## <a name="compute-instance"></a> Create compute instance
-
-Use the [steps above](#portal-create) to start creation of the compute instance. Then fill out the form as follows:
-
-|Field |Description |
-|||
-|Compute name | <li>Name is required and must be between 3 to 24 characters long.</li><li>Valid characters are upper and lower case letters, digits, and the **-** character.</li><li>Name must start with a letter</li><li>Name needs to be unique across all existing computes within an Azure region. You will see an alert if the name you choose is not unique</li><li>If **-** character is used, then it needs to be followed by at least one letter later in the name</li> |
-|Virtual machine type | Choose CPU or GPU. This type cannot be changed after creation |
-|Virtual machine size | Supported virtual machine sizes might be restricted in your region. Check the [availability list](https://azure.microsoft.com/global-infrastructure/services/?products=virtual-machines) |
-
-Select **Create** unless you want to configure advanced settings for the compute instance.
-
-### Advanced settings
-
-Select **Next: Advanced Settings** if you want to:
-
-* Enable SSH access. Follow the [detailed instructions](#enable-ssh) below.
-* Enable virtual network. Specify the **Resource group**, **Virtual network**, and **Subnet** to create the compute instance inside an Azure Virtual Network (vnet). For more information, see these [network requirements](./how-to-secure-training-vnet.md) for vnet.
-* Assign the computer to another user. For more about assigning to other users, see [Create on behalf of](how-to-create-manage-compute-instance.md#on-behalf).
-* Provision with a setup script - for more details about how to create and use a setup script, see [Customize the compute instance with a script](how-to-create-manage-compute-instance.md#setup-script).
-
-### <a name="enable-ssh"></a> Enable SSH access
-
-SSH access is disabled by default. SSH access cannot be changed after creation. Make sure to enable access if you plan to debug interactively with [VS Code Remote](how-to-set-up-vs-code-remote.md).
-
-After you have selected **Next: Advanced Settings**:
-
-1. Turn on **Enable SSH access**.
-1. In the **SSH public key source**, select one of the options from the dropdown:
- * If you **Generate new key pair**:
- 1. Enter a name for the key in **Key pair name**.
- 1. Select **Create**.
- 1. Select **Download private key and create compute**. The key is usually downloaded into the **Downloads** folder.
- * If you select **Use existing public key stored in Azure**, search for and select the key in **Stored key**.
- * If you select **Use existing public key**, provide an RSA public key in the single-line format (starting with "ssh-rsa") or the multi-line PEM format. You can generate SSH keys using ssh-keygen on Linux and OS X, or PuTTYGen on Windows.
-
-Once the compute instance is created and running, see [Connect with SSH access](#ssh-access).
## <a name="amlcompute"></a> Create compute clusters
Select **Next** to proceed to **Advanced Settings** and fill out the form as fol
| Enable SSH access | Use the same instructions as [Enable SSH access](#enable-ssh) for a compute instance (above). | |Advanced settings | Optional. Configure a virtual network. Specify the **Resource group**, **Virtual network**, and **Subnet** to create the compute instance inside an Azure Virtual Network (vnet). For more information, see these [network requirements](./how-to-secure-training-vnet.md) for vnet. Also attach [managed identities](#managed-identity) to grant access to resources |
+### <a name="enable-ssh"></a> Enable SSH access
+
+SSH access is disabled by default. SSH access cannot be changed after creation. Make sure to enable access if you plan to debug interactively with [VS Code Remote](how-to-set-up-vs-code-remote.md).
++
+Once the compute cluster is created and running, see [Connect with SSH access](#ssh-access).
+ ### <a name="managed-identity"></a> Set up managed identity [!INCLUDE [aml-clone-in-azure-notebook](../../includes/aml-managed-identity-intro.md)]
myvm = ComputeTarget(workspace=ws, name='my-vm-name')
* [Tutorial: Train a model](tutorial-train-models-with-aml.md) uses a managed compute target to train a model. * Learn how to [efficiently tune hyperparameters](how-to-tune-hyperparameters.md) to build better models. * Once you have a trained model, learn [how and where to deploy models](how-to-deploy-and-where.md).
-* [Use Azure Machine Learning with Azure Virtual Networks](./how-to-network-security-overview.md)
+* [Use Azure Machine Learning with Azure Virtual Networks](./how-to-network-security-overview.md)
machine-learning How To Create Manage Compute Instance https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/machine-learning/how-to-create-manage-compute-instance.md
Previously updated : 07/16/2021 Last updated : 08/06/2021 # Create and manage an Azure Machine Learning compute instance
Use a compute instance as your fully configured and managed development environm
In this article, you learn how to:
-* Create a compute instance
-* Manage (start, stop, restart, delete) a compute instance
-* Access the terminal window
-* Install R or Python packages
-* Create new environments or Jupyter kernels
+* [Create](#create) a compute instance
+* [Manage](#manage) (start, stop, restart, delete) a compute instance
+* [Create a schedule](#schedule) to automatically start and stop the compute instance (preview)
+* [Use a setup script](#setup-script) to customize and configure the compute instance
Compute instances can run jobs securely in a [virtual network environment](how-to-secure-training-vnet.md), without requiring enterprises to open up SSH ports. The job executes in a containerized environment and packages your model dependencies in a Docker container.
Compute instances can run jobs securely in a [virtual network environment](how-t
**Time estimate**: Approximately 5 minutes.
-Creating a compute instance is a one time process for your workspace. You can reuse the compute as a development workstation or as a compute target for training. You can have multiple compute instances attached to your workspace.
+Creating a compute instance is a one time process for your workspace. You can reuse the compute as a development workstation or as a compute target for training. You can have multiple compute instances attached to your workspace.
-The dedicated cores per region per VM family quota and total regional quota, which applies to compute instance creation, is unified and shared with Azure Machine Learning training compute cluster quota. Stopping the compute instance does not release quota to ensure you will be able to restart the compute instance. Note it is not possible to change the virtual machine size of compute instance once it is created.
+The dedicated cores per region per VM family quota and total regional quota, which applies to compute instance creation, is unified and shared with Azure Machine Learning training compute cluster quota. Stopping the compute instance does not release quota to ensure you will be able to restart the compute instance. It is not possible to change the virtual machine size of compute instance once it is created.
-The following example demonstrates how to create a compute instance:
+<a name="create-instance"></a> The following example demonstrates how to create a compute instance:
# [Python](#tab/python)
For more information, see the [az ml computetarget create computeinstance](/cli/
# [Studio](#tab/azure-studio)
-In your workspace in Azure Machine Learning studio, create a new compute instance from either the **Compute** section or in the **Notebooks** section when you are ready to run one of your notebooks.
+1. Navigate to [Azure Machine Learning studio](https://ml.azure.com).
+1. Under __Manage__, select __Compute__.
+1. Select **Compute instance** at the top.
+1. If you have no compute instances, select **Create** in the middle of the page.
+
+ :::image type="content" source="media/how-to-create-attach-studio/create-compute-target.png" alt-text="Create compute target":::
-For information on creating a compute instance in the studio, see [Create compute targets in Azure Machine Learning studio](how-to-create-attach-compute-studio.md#compute-instance).
+1. If you see a list of compute resources, select **+New** above the list.
+
+ :::image type="content" source="media/how-to-create-attach-studio/select-new.png" alt-text="Select new":::
+1. Fill out the form:
+
+ |Field |Description |
+ |||
+ |Compute name | <ul><li>Name is required and must be between 3 to 24 characters long.</li><li>Valid characters are upper and lower case letters, digits, and the **-** character.</li><li>Name must start with a letter</li><li>Name needs to be unique across all existing computes within an Azure region. You will see an alert if the name you choose is not unique</li><li>If **-** character is used, then it needs to be followed by at least one letter later in the name</li></ul> |
+ |Virtual machine type | Choose CPU or GPU. This type cannot be changed after creation |
+ |Virtual machine size | Supported virtual machine sizes might be restricted in your region. Check the [availability list](https://azure.microsoft.com/global-infrastructure/services/?products=virtual-machines) |
+
+1. Select **Create** unless you want to configure advanced settings for the compute instance.
+1. <a name="advanced-settings"></a> Select **Next: Advanced Settings** if you want to:
+
+ * Enable SSH access. Follow the [detailed SSH access instructions](#enable-ssh) below.
+ * Enable virtual network. Specify the **Resource group**, **Virtual network**, and **Subnet** to create the compute instance inside an Azure Virtual Network (vnet). For more information, see these [network requirements](./how-to-secure-training-vnet.md) for vnet.
+ * Assign the computer to another user. For more about assigning to other users, see [Create on behalf of](#on-behalf).
+ * Provision with a setup script (preview) - for more details about how to create and use a setup script, see [Customize the compute instance with a script](#setup-script).
+ * Add schedule (preview). Schedule times for the compute instance to automatically start and/or shutdown. See [schedule details](#schedule) below.
You can also create a compute instance with an [Azure Resource Manager template](https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.machinelearningservices/machine-learning-compute-create-computeinstance).
+## <a name="enable-ssh"></a> Enable SSH access
+SSH access is disabled by default. SSH access cannot be changed after creation. Make sure to enable access if you plan to debug interactively with [VS Code Remote](how-to-set-up-vs-code-remote.md).
++
+Once the compute instance is created and running, see [Connect with SSH access](how-to-create-attach-compute-studio.md#ssh-access).
## <a name="on-behalf"></a> Create on behalf of (preview) As an administrator, you can create a compute instance on behalf of a data scientist and assign the instance to them with:
+* Studio, using the [Advanced settings](?tabs=azure-studio#advanced-settings)
+ * [Azure Resource Manager template](https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.machinelearningservices/machine-learning-compute-create-computeinstance). For details on how to find the TenantID and ObjectID needed in this template, see [Find identity object IDs for authentication configuration](../healthcare-apis/azure-api-for-fhir/find-identity-object-ids.md). You can also find these values in the Azure Active Directory portal. * REST API The data scientist you create the compute instance for needs the following be [Azure role-based access control (Azure RBAC)](../role-based-access-control/overview.md) permissions:+ * *Microsoft.MachineLearningServices/workspaces/computes/start/action* * *Microsoft.MachineLearningServices/workspaces/computes/stop/action* * *Microsoft.MachineLearningServices/workspaces/computes/restart/action* * *Microsoft.MachineLearningServices/workspaces/computes/applicationaccess/action*
+* *Microsoft.MachineLearningServices/workspaces/computes/updateSchedules/action*
The data scientist can start, stop, and restart the compute instance. They can use the compute instance for: * Jupyter
The data scientist can start, stop, and restart the compute instance. They can u
* RStudio * Integrated notebooks
+## <a name="schedule"></a> Schedule automatic start and stop (preview)
+
+Define multiple schedules for auto-shutdown and auto-start. For instance, create a schedule to start at 9 AM and stop at 6 PM from Monday-Thursday, and a second schedule to start at 9 AM and stop at 4 PM for Friday. You can create a total of four schedules per compute instance.
+
+Schedules can also be defined for [create on behalf of](#on-behalf) compute instances. You can create schedule to create a compute instance in a stopped state. This is particularly useful when a user creates a compute instance on behalf of another user.
+
+### Create a schedule in studio
+
+1. [Fill out the form](?tabs=azure-studio#create-instance).
+1. On the second page of the form, open **Show advanced settings**.
+1. Select **Add schedule** to add a new schedule.
+
+ :::image type="content" source="media/how-to-create-attach-studio/create-schedule.png" alt-text="Screenshot: Add schedule in advanced settings.":::
+
+1. Select **Start compute instance** or **Stop compute instance**.
+1. Select the **Time zone**.
+1. Select the **Startup time** or **Shutdown time**.
+1. Select the days when this schedule is active.
+
+ :::image type="content" source="media/how-to-create-attach-studio/stop-compute-schedule.png" alt-text="Screenshot: schedule a compute instance to shut down.":::
+
+1. Select **Add schedule** again if you want to create another schedule.
+
+Once the compute instance is created, you can view, edit, or add new schedules from the compute instance details section.
+
+### Create a schedule with a Resource Manager template
+
+You can schedule the automatic start and stop of a compute instance by using a Resource Manager template. In a Resource Manager template, use either cron or LogicApps expressions to define a schedule to start or stop the instance.
+
+```json
+"schedules": {
+ "value": {
+ "computeStartStop": [
+ {
+ "TriggerType": "Cron",
+ "Cron": {
+ "StartTime": "2021-03-10T21:21:07",
+ "TimeZone": "Pacific Standard Time",
+ "Expression": "0 18 * * *"
+ },
+ "Action": "Stop",
+ "Status": "Enabled"
+ },
+ {
+ "TriggerType": "Cron",
+ "Cron": {
+ "StartTime": "2021-03-10T21:21:07",
+ "TimeZone": "Pacific Standard Time",
+ "Expression": "0 8 * * *"
+ },
+ "Action": "Start",
+ "Status": "Enabled"
+ },
+ {
+ "triggerType":ΓÇ»"Recurrence",
+ "recurrence":ΓÇ»{
+ "frequency":ΓÇ»"Day",
+ "interval": 1,
+ "timeZone": "Pacific Standard Time",
+   "schedule": {
+ "hours":ΓÇ»[18],
+     "minutes": [0],
+ "weekDays":ΓÇ»[
+ "Saturday",
+ "Sunday"
+ ]
+ }
+ },
+ "Action":ΓÇ»"Stop",
+ "Status":ΓÇ»"Enabled"
+ }
+ ]
+```
+
+* Action can have value of ΓÇ£StartΓÇ¥ or ΓÇ£StopΓÇ¥.
+* For trigger type of `Recurrence` use the same syntax as logic app, with this [recurrence schema](../logic-apps/logic-apps-workflow-actions-triggers.md#recurrence-trigger).
+* For trigger type of `cron`, use standard cron syntax:
+
+ ```cron
+ // Crontab expression format:
+ //
+ // * * * * *
+ //
+ // | | | | |
+ // | | | | +-- day of week (0 - 6) (Sunday=0)
+ // | | | +- month (1 - 12)
+ // | | + day of month (1 - 31)
+ // | +-- hour (0 - 23)
+ // +- min (0 - 59)
+ //
+ // Star (*) in the value field above means all legal values as in
+ // braces for that column. The value column can have a * or a list
+ // of elements separated by commas. An element is either a number in
+ // the ranges shown above or two numbers in the range separated by a
+ // hyphen (meaning an inclusive range).
+ ```
+
+Use Azure policy to enforce a shutdown schedule exists for every compute instance in a subscription or default to a schedule if nothing exists.
+ ## <a name="setup-script"></a> Customize the compute instance with a script (preview) Use a setup script for an automated way to customize and configure the compute instance at provisioning time. As an administrator, you can write a customization script to be used to provision all compute instances in the workspace according to your requirements.
Once you store the script, specify it during creation of your compute instance:
1. Sign into the [studio](https://ml.azure.com/) and select your workspace. 1. On the left, select **Compute**. 1. Select **+New** to create a new compute instance.
-1. [Fill out the form](how-to-create-attach-compute-studio.md#compute-instance).
-1. On the second page of the form, open **Show advanced settings**
-1. Turn on **Provision with setup script**
+1. [Fill out the form](?tabs=azure-studio#create-instance).
+1. On the second page of the form, open **Show advanced settings**.
+1. Turn on **Provision with setup script**.
1. Browse to the shell script you saved. Or upload a script from your computer. 1. Add command arguments as needed.
For example, specify a base64 encoded command string for `scriptData`:
Logs from the setup script execution appear in the logs folder in the compute instance details page. Logs are stored back to your notebooks file share under the Logs\<compute instance name> folder. Script file and command arguments for a particular compute instance are shown in the details page. + ## Manage Start, stop, restart, and delete a compute instance. A compute instance does not automatically scale down, so make sure to stop the resource to prevent ongoing charges. Stopping a compute instance deallocates it. Then start it again when you need it. While stopping the compute instance stops the billing for compute hours, you will still be billed for disk, public IP, and standard load balancer.
+You can [create a schedule](#schedule) for the compute instance to automatically start and stop based on a time and day of week.
+ > [!TIP] > The compute instance has 120GB OS disk. If you run out of disk space, [use the terminal](how-to-access-terminal.md) to clear at least 1-2 GB before you stop or restart the compute instance. Please do not stop the compute instance by issuing sudo shutdown from the terminal.
In the examples below, the name of the compute instance is **instance**
For more information, see [az ml computetarget delete computeinstance](/cli/azure/ml(v1)/computetarget#az_ml_computetarget_delete). # [Studio](#tab/azure-studio)
+<a name="schedule"></a>
In your workspace in Azure Machine Learning studio, select **Compute**, then select **Compute Instance** on the top.
You can perform the following actions:
* Create a new compute instance * Refresh the compute instances tab.
-* Start, stop, and restart a compute instance. You do pay for the instance whenever it is running. Stop the compute instance when you are not using it to reduce cost. Stopping a compute instance deallocates it. Then start it again when you need it.
+* Start, stop, and restart a compute instance. You do pay for the instance whenever it is running. Stop the compute instance when you are not using it to reduce cost. Stopping a compute instance deallocates it. Then start it again when you need it. You can also schedule a time for the compute instance to start and stop.
* Delete a compute instance. * Filter the list of compute instances to show only those you have created.
-For each compute instance in your workspace that you created (or that was created for you), you can:
+For each compute instance in a workspace that you created (or that was created for you), you can:
-* Access Jupyter, JupyterLab, RStudio on the compute instance
+* Access Jupyter, JupyterLab, RStudio on the compute instance.
* SSH into compute instance. SSH access is disabled by default but can be enabled at compute instance creation time. SSH access is through public/private key mechanism. The tab will give you details for SSH connection such as IP address, username, and port number. In a virtual network deployment, disabling SSH prevents SSH access from public internet, you can still SSH from within virtual network using private IP address of compute instance node and port 22.
-* Get details about a specific compute instance such as IP address, and region.
+* Select the compute name to:
+ * View details about a specific compute instance such as IP address, and region.
+ * Create or modify the schedule for starting and stopping the compute instance (preview). Scroll down to the bottom of the page to edit the schedule.
These actions can be controlled by Azure RBAC:
* *Microsoft.MachineLearningServices/workspaces/computes/start/action* * *Microsoft.MachineLearningServices/workspaces/computes/stop/action* * *Microsoft.MachineLearningServices/workspaces/computes/restart/action*
+* *Microsoft.MachineLearningServices/workspaces/computes/updateSchedules/action*
To create a compute instance you'll need permissions for the following actions: * *Microsoft.MachineLearningServices/workspaces/computes/write*
machine-learning How To Create Workspace Template https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/machine-learning/how-to-create-workspace-template.md
New-AzResourceGroupDeployment `
## Use the Azure portal
-1. Follow the steps in [Deploy resources from custom template](../azure-resource-manager/templates/deploy-portal.md#deploy-resources-from-custom-template). When you arrive at the __Select a template__ screen, choose the **quickstarts** entry. When it appears, select the link labled "Click here to open template repository". This link takes you to the `quickstarts` directory in the Azure quickstart templates repository.
-1. In the list of quickstart templates, select `microsoft.machinelearningservices'. Finally, select `Deploy to Azure`.
+1. Follow the steps in [Deploy resources from custom template](../azure-resource-manager/templates/deploy-portal.md#deploy-resources-from-custom-template). When you arrive at the __Select a template__ screen, choose the **quickstarts** entry. When it appears, select the link labeled "Click here to open template repository". This link takes you to the `quickstarts` directory in the Azure quickstart templates repository.
+1. In the list of quickstart templates, select `microsoft.machinelearningservices`. Finally, select `Deploy to Azure`.
1. When the template appears, provide the following required information and any other parameters depending on your deployment scenario. * Subscription: Select the Azure subscription to use for these resources.
machine-learning How To Manage Optimize Cost https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/machine-learning/how-to-manage-optimize-cost.md
Use the following tips to help you manage and optimize your compute resource cos
- Set quotas on your subscription and workspaces - Set termination policies on your training run - Use low-priority virtual machines (VM)
+- Schedule compute instances to shut down and start up automatically
- Use an Azure Reserved VM Instance - Train locally - Parallelize training
Low-Priority VMs have a single quota separate from the dedicated quota value, wh
Low-Priority VMs don't work for compute instances, since they need to support interactive notebook experiences.
+## Schedule compute instances
+
+When you create a [compute instance](concept-compute-instance.md), the VM stays on so it is available for your work. [Set up a schedule](how-to-create-manage-compute-instance.md#schedule) to automatically start and stop the compute instance (preview) to save cost when you aren't planning to use it.
+ ## Use reserved instances Another way to save money on compute resources is Azure Reserved VM Instance. With this offering, you commit to one-year or three-year terms. These discounts range up to 72% of the pay-as-you-go prices and are applied directly to your monthly Azure bill.
marketplace What Is New https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/marketplace/what-is-new.md
Previously updated : 06/01/2021 Last updated : 08/06/2021 # What's new in the Microsoft commercial marketplace
Learn about important updates in the commercial marketplace program of Partner C
| Category | Description | Date | | | - | - |
+| Offers | Additional properties at the plan level are now available for Azure Virtual Machine offers. See the [virtual machine technical configuration properties](azure-vm-create-plans.md#properties) article for more information. | 2021-07-26 |
+| Fees | Microsoft has changed its fees for certain services. See [Commercial marketplace transact capabilities](marketplace-commercial-transaction-capabilities-and-considerations.md#examples-of-pricing-and-store-fees) and Common questions about payouts and taxes, "[How do I find the current Store Service Fee and the payout rate?](/partner-center/payout-faq)". | 2021-07-14 |
| Offers | Publishers can publish a virtual machine (VM) that they have built on premises. To learn more, see [Create a virtual machine using your own image](./azure-vm-create-using-own-image.md). | 2021-06-23 | | Customer engagement | Publishers can now respond to user reviews on Azure Marketplace or AppSource. To learn more, see [Ratings & Reviews analytics dashboard in Partner Center](./ratings-reviews.md). | 2021-06- 03 | | Offers | Publishers now have a simpler and faster way to prepare and publish their Azure Virtual Machine-based offers in Partner Center. To learn more, see [How to create a virtual machine using an approved base](azure-vm-create-using-approved-base.md). | 2021-03-22 | | Analytics | Developers can use new report APIs to programmatically access commercial marketplace analytics data. You can schedule custom reports and download your marketplace data into your internal analytics systems. To learn more, see [Get started with programmatic access to analytics data](analytics-get-started.md). | 2021-03-08 |
-| Grow your business | Publishers can more easily mitigate the risk of their customers receiving an incorrect bill for metered billing usage. To learn more, see [Manage metered billing anomalies in Partner Center](anomaly-detection.md). | 2021-02-18 |
-|||
+|
## Tax updates
Learn about important updates in the commercial marketplace program of Partner C
| Taxation | Updated [tax details page](/partner-center/tax-details-marketplace) country list to include the following: <ul><li>Argentina</li><li>Bulgaria</li><li>Hong Kong SAR</li><li>Korea (South)</li><li>Pakistan</li><li>Palestinian Authority</li><li>Panama</li><li>Paraguay</li><li>Peru</li><li>Philippines</li><li>Saint Kitts and Nevis</li><li>Senegal</li><li>Sri Lanka</li><li>Tajikistan</li><li>Tanzania</li><li>Thailand</li><li>Trinidad and Tobago</li><li>Tunisia</li><li>Turkmenistan</li><li>Uganda</li><li>Uzbekistan</li><li>Zimbabwe</li></ul> | 2021-07-01 | | Taxation | Nigeria moved from the "shared publisher/developer-managed countries" list to the ΓÇ£end-customer taxation with differences in Marketplaces". | 2021-07-01 | | Payouts | Added commercial marketplace payout scenarios to the [tax details page](/partner-center/tax-details-marketplace). | 2021-04-26 |
-|||
+|
## Documentation updates | Category | Description | Date | | | - | - |
+| Policy | Updated [certification](/legal/marketplace/certification-policies?context=/azure/marketplace/context/context) policy; see [change history](/legal/marketplace/offer-policies-change-history). | 2021-08-06 |
| Co-sell | Information added for the MACC program including, requirements, how often we update MACC status, and definitions for Enrolled, and not Enrolled. To learn more, see [Azure Consumption Commitment enrollment](./azure-consumption-commitment-enrollment.md), or [Co-sell with Microsoft sales teams and partners overview](./co-sell-overview.md). | 2021-06-03 | | Offers | Additional information regarding VM pricing options and descriptions. To learn more see [How to plan a SaaS offer for the commercial marketplace](./plan-saas-offer.md). | 2021-05-25| | API guidance | This topic gives publishers a single convenient location to find recent changes to the commercial marketplace, including updates to certification policy, changes to tax rules, and more. To learn more, see [Align your business with our e-commerce platform](./marketplace-apis-guide.md). | 2021-05-13 |
+| Policy | Updated [certification](/legal/marketplace/certification-policies?context=/azure/marketplace/context/context) policy; see [change history](/legal/marketplace/offer-policies-change-history). | 2021-05-07 |
| Policy | The [Microsoft Publisher Agreement](/legal/marketplace/msft-publisher-agreement) has been updated. To see whatΓÇÖs changed, see [Change history for Microsoft Publisher Agreement](/legal/marketplace/mpa-change-history). | 2021-04-16 | | Offers | Microsoft 365 independent software vendors (ISVs) can now link their software as a service (SaaS) offer to their related Teams apps, Office add-ins (WXPO), and SharePoint SPFx solutions in Partner Center. SaaS ISVs can also declare if their SaaS offer is integrated with Microsoft Graph API. To learn more, see [Test and deploy Microsoft 365 Apps by partners in the Integrated apps portal](/microsoft-365/admin/manage/test-and-deploy-microsoft-365-apps). | 2021-04-08 |
-|||
+| Policy | Updated [certification](/legal/marketplace/certification-policies?context=/azure/marketplace/context/context) policy; see [change history](/legal/marketplace/offer-policies-change-history). | 2021-04-02 |
+| Policy | Updated [certification](/legal/marketplace/certification-policies?context=/azure/marketplace/context/context) policy; see [change history](/legal/marketplace/offer-policies-change-history). | 2021-03-05 |
+|
postgresql Concepts Networking https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/postgresql/flexible-server/concepts-networking.md
Here are some concepts to be familiar with when you're using virtual networks wi
At this time, we don't support NSGs where an ASG is part of the rule with Azure Database for PostgreSQL - Flexible Server. We currently advise using [IP-based source or destination filtering](../../virtual-network/network-security-groups-overview.md#security-rules) in an NSG.
+ > [!IMPORTANT]
+ > Features of Azure Database for PostgreSQL - Flexible Server require ability to send outbound traffic to destination ports 5432, 6432. If you create Network Security Groups (NSG) to deny outbound traffic from your Azure Database for PostgreSQL - Flexible Server, please make sure to allow traffic to these destination ports.
+ * **Private DNS zone integration**. Azure private DNS zone integration allows you to resolve the private DNS within the current virtual network or any in-region peered virtual network where the private DNS zone is linked. ### Using a private DNS zone
postgresql Howto Configure Server Parameters Using Portal https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/postgresql/flexible-server/howto-configure-server-parameters-using-portal.md
To step through this how-to guide you need:
## Working with time zone parameters If you plan to work with date and time data in PostgreSQL, youΓÇÖll want to ensure that youΓÇÖve set the correct time zone for your location. All timezone-aware dates and times are stored internally in Postgres in UTC. They are converted to local time in the zone specified by the **TimeZone** server parameter before being displayed to the client. This parameter can be edited on **Server parameters** page as explained above. PostgreSQL allows you to specify time zones in three different forms:
-1. A full time zone name, for example America/New_York. The recognized time zone names are listed in the **pg_timezone_names** view.
-2. A time zone abbreviation, for example PST. Such a specification merely defines a particular offset from UTC, in contrast to full time zone names which can imply a set of daylight savings transition-date rules as well. The recognized abbreviations are listed in the **pg_timezone_abbrevs view**
+1. A full time zone name, for example America/New_York. The recognized time zone names are listed in the [**pg_timezone_names**](https://www.postgresql.org/docs/9.2/view-pg-timezone-names.html) view.
+ Example to query this view in psql and get list of time zone names:
+ <pre>select name FROM pg_timezone_names LIMIT 20;</pre>
+
+ You should see result set like:
+
+ <pre>
+ name
+ --
+ GMT0
+ Iceland
+ Factory
+ NZ-CHAT
+ America/Panama
+ America/Fort_Nelson
+ America/Pangnirtung
+ America/Belem
+ America/Coral_Harbour
+ America/Guayaquil
+ America/Marigot
+ America/Barbados
+ America/Porto_Velho
+ America/Bogota
+ America/Menominee
+ America/Martinique
+ America/Asuncion
+ America/Toronto
+ America/Tortola
+ America/Managua
+ (20 rows)
+ </pre>
+
+2. A time zone abbreviation, for example PST. Such a specification merely defines a particular offset from UTC, in contrast to full time zone names which can imply a set of daylight savings transition-date rules as well. The recognized abbreviations are listed in the [**pg_timezone_abbrevs view**](https://www.postgresql.org/docs/9.4/view-pg-timezone-abbrevs.html)
+ Example to query this view in psql and get list of time zone abbreviations:
+
+ <pre> select abbrev from pg_timezone_abbrevs limit 20;</pre>
+
+ You should see result set like:
+
+ <pre>
+ abbrev|
+ +
+ ACDT |
+ ACSST |
+ ACST |
+ ACT |
+ ACWST |
+ ADT |
+ AEDT |
+ AESST |
+ AEST |
+ AFT |
+ AKDT |
+ AKST |
+ ALMST |
+ ALMT |
+ AMST |
+ AMT |
+ ANAST |
+ ANAT |
+ ARST |
+ ART |
+ </pre>
+ 3. In addition to the timezone names and abbreviations, PostgreSQL will accept POSIX-style time zone specifications of the form STDoffset or STDoffsetDST, where STD is a zone abbreviation, offset is a numeric offset in hours west from UTC, and DST is an optional daylight-savings zone abbreviation, assumed to stand for one hour ahead of the given offset.
purview Purview Connector Overview https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/purview/purview-connector-overview.md
details.
||[Azure SQL Database Managed Instance](register-scan-azure-sql-database-managed-instance.md)|Yes| Yes| No| Yes| Yes| Yes| ||[Azure Dedicated SQL pool (formerly SQL DW)](register-scan-azure-synapse-analytics.md)|Yes| Yes| No| Yes| Yes| Yes| ||[Azure Synapse Analytics (Workspace)](register-scan-synapse-workspace.md)|Yes| Yes| No| Yes| Yes| Yes|
-|Database|[Google BigQuery](register-scan-google-bigquery-source.md)|Yes| Yes| No| No| No| Yes|
+|Database|[Cassandra](register-scan-cassandra-source.md)|Yes| Yes| No| No| No| Yes|
+||[Google BigQuery](register-scan-google-bigquery-source.md)|Yes| Yes| No| No| No| Yes|
||[Hive Metastore DB](register-scan-oracle-source.md)|Yes| Yes| No| No| No| Yes| ||[Oracle DB](register-scan-oracle-source.md)|Yes| Yes| No| No| No| Yes| ||[SQL Server](register-scan-on-premises-sql-server.md)|Yes| Yes| No| Yes| Yes| Yes|
purview Register Scan Cassandra Source https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/purview/register-scan-cassandra-source.md
+
+ Title: Register Cassandra as a source and setup scans in Azure Purview
+description: This article outlines how to register Cassandra server in Azure Purview and set up a scan.
+++++ Last updated : 8/06/2021+
+# Register and Scan a Cassandra source
+
+This article outlines how to register a Cassandra server in Purview and set up a scan.
+
+## Supported capabilities
+
+The Cassandra source supports Full scan to extract metadata from a
+Cassandra server and fetches Lineage between data assets.
+
+## Prerequisites
+
+1. Set up the latest [self-hosted integration
+ runtime](https://www.microsoft.com/download/details.aspx?id=39717).
+ For more information, see 
+ [Create and configure a self-hosted integration runtime](../data-factory/create-self-hosted-integration-runtime.md).
+
+2. Make sure [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html)
+ is installed on your virtual machine where self-hosted integration
+ runtime is installed.
+
+3. Make sure \"Visual C++ Redistributable 2012 Update 4\" is installed
+ on the self-hosted integration runtime machine. If you don\'t yet
+ have it installed, download it from
+ [here](https://www.microsoft.com/download/details.aspx?id=30679).
+
+4. Supported Cassandra server versions are 3.x to 4.x
+
+## Register a Cassandra server
+
+To register a new Cassandra server in your data catalog, do the
+following:
+
+1. Navigate to your Purview account.
+2. Select **Data Map** on the left navigation.
+3. Select **Register.**
+4. On Register sources, select **Cassandra** . Select **Continue.**
+ :::image type="content" source="media/register-scan-cassandra-source/register-sources.png" alt-text="register Cassandra source" border="true":::
+
+On the Register sources (Cassandra) screen, do the following:
+
+1. Enter a **Name** that the data source will be listed within the
+ Catalog.
+
+2. Enter the server address where Cassandra server is running in the **Host** field. For example, 20.190.193.10
+
+3. Enter the port used by Cassandra server in the **Port** field.
+4. Select a collection or create a new one (Optional)
+
+5. Click on **Register**.
+ :::image type="content" source="media/register-scan-cassandra-source/configure-sources.png" alt-text="configure Cassandra source" border="true":::
+
+## Creating and running a scan
+
+To create and run a new scan, do the following:
+
+1. In the Management Center, click on Integration runtimes. Make sure a
+ self-hosted integration runtime is set up. If it is not set up, use
+ the steps mentioned
+ [here](./manage-integration-runtimes.md)
+ to setup a self-hosted integration runtime
+
+2. Navigate to **Sources**.
+
+3. Select the registered **Cassandra** server.
+
+4. Select **+ New scan**.
+
+5. Provide the below details:
+
+ a. **Name**: The name of the scan
+
+ b. **Connect via integration runtime**: Select the configured
+ self-hosted integration runtime
+
+ c. **Credential**: While configuring Cassandra credential, make sure
+ to:
+
+ - Select **Basic Authentication** as the Authentication method
+ - Provide the username on who's behalf the connection is being made in the User name field.
+ - Save Cassandra user's password on whose behalf the connection is being made in the key vault's secret
+
+ To understand more on credentials, refer to the link [here](manage-credentials.md).
+
+ d. **Keyspaces**: Specify a list of Cassandra keyspaces to be imported. Multiple keypsaces must be semicolon separated. For example, keyspace1; keyspace2. When the list is empty, all available keyspaces are imported.
+ Acceptable keyspace name patterns using SQL LIKE expressions syntax include using %,
+
+ e.g. A%; %B; %C%; D
+ - start with A or
+ - end with B or
+ - contain C or
+ - equal D
+Usage of NOT and special characters are not acceptable.
+
+ f. **Use Secure Sockets Layer(SSL)** : Select True or False to Notify
+ if Secure Sockets Layer (SSL) must be used when connecting to the
+ Cassandra server. By default, this value is set to False.
+
+ g. **Maximum memory available**: Maximum memory (in GB) available on customer's VM to be used by scanning processes. This is dependent on the size of Cassandra server to be scanned.
+ :::image type="content" source="media/register-scan-cassandra-source/scan.png" alt-text="scan Cassandra source" border="true":::
+
+6. Click on **Test connection.**
+
+7. Click on **Continue**.
+
+8. Choose your **scan trigger**. You can set up a schedule or ran the
+ scan once.
+
+9. Review your scan and click on **Save and Run**.
+
+## Viewing your scans and scan runs
+
+1. Navigate to the management center. Select **Data sources** under the **Sources and scanning** section.
+
+2. Select the desired data source. You will see a list of existing scans on that data source.
+
+3. Select the scan whose results you are interested to view.
+
+4. This page will show you all of the previous scan runs along with metrics and status for each scan run. It will also display whether your scan was scheduled or manual, how many assets had classifications applied, how many total assets were discovered, the start and end time of the scan, and the total scan duration.
+
+## Manage your scans
+
+To manage or delete a scan, do the following:
+
+1. Navigate to the management center. Select **Data sources** under the **Sources and scanning** section then select on the desired data source.
+
+2. Select the scan you would like to manage. You can edit the scan by selecting **Edit**.
+
+3. You can delete your scan by selecting **Delete**.
+
+## Next steps
+
+- [Browse the Azure Purview Data catalog](how-to-browse-catalog.md)
+- [Search the Azure Purview Data Catalog](how-to-search-catalog.md)
purview Register Scan Google Bigquery Source https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/purview/register-scan-google-bigquery-source.md
To create and run a new scan, do the following:
to: - Select **Basic Authentication** as the Authentication method
- - Provide the email ID of the service account in the User name field. For example,\xyz\@developer.gserviceaccount.com
+ - Provide the email ID of the service account in the User name field. For example, xyz\@developer.gserviceaccount.com
- Save your Private key file of the service account in the JSON format in the key vault's secret To create a new private key from Google's cloud platform, in the
search Cognitive Search Concept Intro https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/search/cognitive-search-concept-intro.md
Previously updated : 07/30/2021 Last updated : 08/10/2021 # AI enrichment in Azure Cognitive Search
-In Azure Cognitive Search, AI enrichment is about using AI to create new structures and information from raw content in external data sources. AI enrichment is defined by a [skillset](cognitive-search-working-with-skillsets.md) that's attached to an [indexer](search-indexer-overview.md). The indexer will extract and set up the content, while the skillset identifies, analyzes, and creates new information and structures from images, blobs, and other unstructured data sources. The purpose of enrichment is make content more accessible in output objects, which is either a [search index](search-what-is-an-index.md) or a [knowledge store](knowledge-store-concept-intro.md).
+In Azure Cognitive Search, AI enrichment refers to built-in cognitive skills and custom skills that add content transformation and generation during indexing. Enrichments create new information where none previously existed: extracting information from images, detecting sentiment, key phrases, and entities from text, to name a few. Enrichments also add structure to undifferentiated text. All of these processes result in documents that make full text search more effective. In many instances, enriched documents are useful for scenarios other than search, such as for knowledge mining.
+
+Enrichment is defined by a [skillset](cognitive-search-working-with-skillsets.md) that's attached to an [indexer](search-indexer-overview.md). The indexer will extract and set up the content, while the skillset identifies, analyzes, and creates new information and structures from images, blobs, and other unstructured data sources. The output of enrichment is either a [search index](search-what-is-an-index.md) or a [knowledge store](knowledge-store-concept-intro.md).
A skillset can contain built-in skills from Cognitive Search or embed external processing that you provide in a [*custom skill*](cognitive-search-create-custom-skill-example.md). Examples of a custom skill might be a custom entity module or document classifier targeting a specific domain such as finance, scientific publications, or medicine.
-Built-in skills fall into these categories:
+Built-in skills fall into these categories:
+ **Natural language processing** skills include [entity recognition](cognitive-search-skill-entity-recognition-v3.md), [language detection](cognitive-search-skill-language-detection.md), [key phrase extraction](cognitive-search-skill-keyphrases.md), text manipulation, [sentiment detection (including opinion mining)](cognitive-search-skill-sentiment-v3.md), and [PII detection](cognitive-search-skill-pii-detection.md). With these skills, unstructured text is mapped as searchable and filterable fields in an index.
Additionally, you might consider adding a custom skill if you have open-source,
### Use-cases for built-in skills
-A [skillset](cognitive-search-defining-skillset.md) that's assembled using built-in skills is well suited for the following application scenarios:
+A [skillset](cognitive-search-defining-skillset.md) that's assembled using built-in skills is well-suited for the following application scenarios:
-+ Scanned documents (JPEG) that you want to make full-text searchable. You can attach an optical character recognition (OCR) skill to identify, extract, and ingest text from JPEG files.
++ [Optical Character Recognition (OCR)](cognitive-search-skill-ocr.md) that recognizes typeface and handwritten text in scanned documents (JPEG) is perhaps the most commonly used skill. Attaching the OCR skill will identify, extract, and ingest text from JPEG files.
-+ PDFs with combined image and text. Text in PDFs can be extracted during indexing without the use of enrichment steps, but the addition of image and natural language processing can often produce a better outcome than a standard indexing provides.
++ [Text translation](cognitive-search-skill-text-translation.md) of multilingual content is another commonly used skill. Language detection is built into Text Translation, but you can also run [Language Detection](cognitive-search-skill-language-detection.md) independently if you just want the language codes of the content in your corpus.
-+ Multi-lingual content against which you want to apply language detection and possibly text translation.
++ PDFs with combined image and text. Text in PDFs can be extracted during indexing without the use of enrichment steps, but the addition of image and natural language processing can often produce a better outcome than a standard indexing provides. + Unstructured or semi-structured documents containing content that has inherent meaning or context that is hidden in the larger document.
- Blobs in particular often contain a large body of content that is packed into a single "field". By attaching image and natural language processing skills to an indexer, you can create new information that is extant in the raw content, but not otherwise surfaced as distinct fields. Some ready-to-use built-in cognitive skills that can help: key phrase extraction, sentiment analysis, and entity recognition (people, organizations, and locations).
+ Blobs in particular often contain a large body of content that is packed into a single "field". By attaching image and natural language processing skills to an indexer, you can create new information that is extant in the raw content, but not otherwise surfaced as distinct fields. Some ready-to-use built-in cognitive skills that can help: [Key Phrase Extraction](cognitive-search-skill-keyphrases.md) and [Entity Recognition](cognitive-search-skill-entity-recognition-v3.md) (people, organizations, and locations to name a few).
Additionally, built-in skills can also be used restructure content through text split, merge, and shape operations.
Custom skills can support more complex scenarios, such as recognizing forms, or
## Enrichment steps <a name="enrichment-steps"></a>
-An enrichment pipeline consists of [*indexers*](search-indexer-overview.md) that have [*skillsets*](cognitive-search-working-with-skillsets.md). A skillset defines the enrichment steps, and the indexer drives the skillset. When configuring an indexer, you can include properties like output field mappings that send enriched content to a search index or knowledge store.
+An enrichment pipeline consists of [*indexers*](search-indexer-overview.md) that have [*skillsets*](cognitive-search-working-with-skillsets.md). A skillset defines the enrichment steps, and the indexer drives the skillset. When configuring an indexer, you can include properties like output field mappings that send enriched content to a [search index](search-what-is-an-index.md) or a [knowledge store](knowledge-store-concept-intro.md).
Post-indexing, you can access content via search requests through all [query types supported by Azure Cognitive Search](search-query-overview.md). ### Step 1: Connection and document cracking phase
-Indexers connect to external sources using information provided in an indexer data source. When the indexer connects to the resource, it will ["crack documents"](search-indexer-overview.md#document-cracking) to extract text and images. Image content can be routed to skills that specify image processing, while text content is queued for text processing.
+Indexers connect to external sources using information provided in an indexer data source. When the indexer connects to the resource, it will ["crack documents"](search-indexer-overview.md#document-cracking) to extract text and images. Image content can be routed to skills that perform image processing, while text content is queued for text processing.
![Document cracking phase](./media/cognitive-search-intro/document-cracking-phase-blowup.png "document cracking")
A skillset defines the atomic operations that are performed on each document. Fo
![Enrichment phase](./media/cognitive-search-intro/enrichment-phase-blowup.png "enrichment phase")
-Skillset composition can be [built-in skills](cognitive-search-predefined-skills.md), [custom skills](cognitive-search-create-custom-skill-example.md) that you create, or both. A skillset can be minimal or highly complex, and determines not only the type of processing, but also the order of operations. Most skillsets contain about three to five skills.
+ skillset can be minimal or highly complex, and determines not only the type of processing, but also the order of operations. Most skillsets contain about three to five skills.
-A skillset, plus the output field mappings defined as part of an indexer, fully specifies the enrichment pipeline. For more information about pulling all of these pieces together, see [Define a skillset](cognitive-search-defining-skillset.md).
+A skillset, plus the [output field mappings](cognitive-search-output-field-mapping.md) defined as part of an indexer, fully specifies the enrichment pipeline. For more information about pulling all of these pieces together, see [Define a skillset](cognitive-search-defining-skillset.md).
Internally, the pipeline generates a collection of enriched documents. You can decide which parts of the enriched documents should be mapped to indexable fields in your search index. For example, if you applied the key phrase extraction and the entity recognition skills, those new fields would become part of the enriched document, and can be mapped to fields on your index. See [Annotations](cognitive-search-concept-annotations-syntax.md) to learn more about input/output formations. ### Step 3: Indexing
-Indexing is the process wherein raw and enriched content is ingested into a search index. A search index has fields, and those fields contain values used in queries, filters, expressions, and potentially in projections in a knowledge store.
+Indexing is the process wherein raw and enriched content is ingested as fields in a search index, and as [projections](knowledge-store-projection-overview.md) if you are also creating a knowledge store. The same enriched content can appear in both, using implicit or explicit field mappings to send the content to the correct fields.
-Enriched content is generated during skillset execution, and is temporary unless you save it. In order for enriched content to appear in a search index, the indexer must have mapping information so that it can send enriched content to a field in a search index. [Output field mappings](cognitive-search-output-field-mapping.md) are the mechanism by which this association is made.
+Enriched content is generated during skillset execution, and is temporary unless you save it. In order for enriched content to appear in a search index, the indexer must have mapping information so that it can send enriched content to a field in a search index. [Output field mappings](cognitive-search-output-field-mapping.md) set up these associations.
## Saving enriched output
-In Azure Cognitive Search, an indexer saves the output it creates. One of the outputs is always a [searchable index](search-what-is-an-index.md). Specifying an index is a required component of an indexer, and when you attach a skillset, the output of the skillset, plus any fields that are imported directly from the source, are used to populate the index. Usually, the outputs of specific skills, such as key phrases or sentiment scores, are ingested into the index in a field created for that purpose.
+In Azure Cognitive Search, an indexer saves the output it creates. One of the outputs is always a [searchable index](search-what-is-an-index.md). Specifying an index is a required component of an indexer, and when you attach a skillset, the output of the skillset, plus any fields that are imported directly from the source, are used to populate the index. Usually, the outputs of specific skills, such as key phrases or sentiment scores, are ingested into the index in fields created for that purpose.
+
+Optionally, an indexer can also send the output to a [knowledge store](knowledge-store-concept-intro.md) for downstream knowledge mining. A knowledge store is defined within a skillset. Its definition determines whether your enriched documents are projected as tables or objects (files or blobs). Tabular projections are well suited for interactive analysis in tools like Power BI, whereas files and blobs are typically used in data science or similar processes.
-Optionally, an indexer can also send the output to a [knowledge store](knowledge-store-concept-intro.md) for consumption in other tools or processes. A knowledge store is defined as part of the skillset. Its definition determines whether your enriched documents are projected as tables or objects (files or blobs). Tabular projections are well suited for interactive analysis in tools like Power BI, whereas files and blobs are typically used in data science or similar processes.
+Finally, an indexer can [cache enriched documents](cognitive-search-incremental-indexing-conceptual.md) in Azure Blob Storage for potential reuse in subsequent skillset executions. Cached enrichments are consumable by the same skillset that you rerun at a later date. Caching is helpful if your skillset include image analysis or OCR, and you want to avoid the time and expense of reprocessing image files.
-Finally, an indexer can [cache enriched documents](cognitive-search-incremental-indexing-conceptual.md) in Azure Blob Storage for potential reuse in subsequent skillset executions. Cached enrichments are consumable by the same skillset that you rerun at a later date. Caching is particularly helpful if your skillset include image analysis or OCR, and you want to avoid the time and expense of re-processing image files.
+Indexes and knowledge stores are fully independent of each other. While you must attach an index to satisfy indexer requirements, if your sole objective is a knowledge store, you can ignore the index after it's populated. Avoid deleting it though. If you want to rerun the indexer and skillset, you'll need the index in order for the indexer to run.
-## Accessing your content
+## Using enriched content
When processing is finished, you have a [search index](search-what-is-an-index.md) consisting of enriched documents, fully text-searchable in Azure Cognitive Search. [Querying the index](search-query-overview.md) is how developers and users access the enriched content generated by the pipeline. You might also have a [knowledge store](knowledge-store-concept-intro.md).
-![Index with search icon](./media/cognitive-search-intro/search-phase-blowup.png "Index with search icon")
+![Enrichment with search index and knowledge store](./media/cognitive-search-intro/cogsearch-arch-kstore.png "Enrichment with search index and knowledge store")
-The index is like any other you might create for Azure Cognitive Search: you can supplement with custom analyzers, invoke fuzzy search queries, add filtered search, or experiment with scoring profiles to reshape the search results.
+The index is like any other you might create for Azure Cognitive Search: you can supplement text analysis with custom analyzers, invoke fuzzy search queries, add filters, or experiment with scoring profiles to tune search relevance.
-Indexes are generated from an index schema that defines the fields, attributes, and other constructs attached to a specific index, such as scoring profiles and synonym maps. Once an index is defined and populated, you can index incrementally to pick up new and updated source documents. Certain modifications require a full rebuild. You should use a small data set until the schema design is stable. For more information, see [How to rebuild an index](search-howto-reindex.md).
+The knowledge store contains data that can be consumed in knowledge mining scenarios like analytics or machine learning.
## Checklist: A typical workflow
-1. Understand the data you are working with so that you can anticipate which skills to use, and have an idea of what you want to achieve (index or knowledge store). When beginning a project, it's helpful to work with a subset of data. Indexer and skillset design is an iterative process, and you'll iterate more quickly if you're working with a small, representative data set.
-
-1. Subset your Azure source data into a representative sample. Indexing takes time so start with a small, representative data set and then build it up incrementally as your solution matures.
-
-1. Create a [data source object](/rest/api/searchservice/create-data-source) in Azure Cognitive Search to provide a connection string for data retrieval.
-
-1. Create a [skillset](/rest/api/searchservice/create-skillset) with enrichment steps.
+1. When beginning a project, it's helpful to work with a subset of data. Indexer and skillset design is an iterative process, and you'll iterate more quickly if you're working with a small representative data set.
-1. Define the [index schema](/rest/api/searchservice/create-index). The *Fields* collection includes fields from source data. You should also stub out additional fields to hold generated values for content created during enrichment.
+1. Create a [data source](/rest/api/searchservice/create-data-source) that specifies a connection to your data.
-1. Define the [indexer](/rest/api/searchservice/create-indexer) referencing the data source, skillset, and index.
+1. Create a [skillset](/rest/api/searchservice/create-skillset) to add enrichment.
- Within the indexer, add *outputFieldMappings*. This section maps output from the skillset to the inputs fields in the index schema.
+1. Create an [index schema](/rest/api/searchservice/create-index) that defines a search index.
-1. Send *Create Indexer* request you just created (a POST request with an indexer definition in the request body) to express the indexer in Azure Cognitive Search. This step is how you run the indexer, invoking the pipeline.
+1. Create an [indexer](/rest/api/searchservice/create-indexer) to bring all of the above components together. Creating or running indexer retrieves the data, runs the skillset, and loads the index.
1. Run queries to evaluate results and modify code to update skillsets, schema, or indexer configuration.
-1. [Reset the indexer](search-howto-reindex.md) before rebuilding the pipeline, or delete and recreate the objects on each run (recommended if you are using the free tier).
+To iterate over the above steps, [reset the indexer](search-howto-reindex.md) before rebuilding the pipeline, or delete and recreate the objects on each run (recommended if you are using the free tier). You should also [enable enrichment caching](cognitive-search-incremental-indexing-conceptual.md) to reuse existing enrichments wherever possible.
## Next steps
-+ [AI enrichment documentation links](cognitive-search-resources-documentation.md)
-+ [Example: Creating a custom skill for AI enrichment (C#)](cognitive-search-create-custom-skill-example.md)
+ [Quickstart: Try AI enrichment in a portal walk-through](cognitive-search-quickstart-blob.md)
-+ [Tutorial: Learn about the AI enrichment APIs](cognitive-search-tutorial-blob.md)
++ [Tutorial: Learn about the AI enrichment REST APIs](cognitive-search-tutorial-blob.md) + [Knowledge store](knowledge-store-concept-intro.md) + [Create a knowledge store in REST](knowledge-store-create-rest.md)
-+ [Troubleshooting tips](cognitive-search-concept-troubleshooting.md)
search Cognitive Search Output Field Mapping https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/search/cognitive-search-output-field-mapping.md
Previously updated : 11/04/2019 Last updated : 08/10/2021 # How to map AI-enriched fields to a searchable index
Examples of output field mappings:
* You donΓÇÖt have a skillset but are indexing a complex type from a Cosmos DB database. You would like to get to a node on that complex type and map it into a field in your index. > [!NOTE]
-> We recently enabled the functionality of mapping functions on output field mappings. For more details on mapping functions, see [Field mapping functions](./search-indexer-field-mappings.md#field-mapping-functions)
+> Output field mappings apply to search indexes only. For indexers that create [knowledge stores](knowledge-store-concept-intro.md), output field mappings are ignored.
## Use outputFieldMappings
The body of the request is structured as follows:
} ```
-For each output field mapping, set the location of the data in the enriched document tree (sourceFieldName), and the name of the field as referenced in the index (targetFieldName).
+For each output field mapping, set the location of the data in the enriched document tree (sourceFieldName), and the name of the field as referenced in the index (targetFieldName). Assign any [mapping functions](search-indexer-field-mappings.md#field-mapping-functions) that you require to transform the content of a field before it's stored in the index.
## Flattening Information from Complex Types
search Cognitive Search Resources Documentation https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/search/cognitive-search-resources-documentation.md
The following is a consolidated list of the documentation for AI enrichment.
+ [Create a knowledge store using REST and Postman](knowledge-store-create-rest.md) + [View a knowledge store with Storage Explorer](knowledge-store-view-storage-explorer.md) + [Connect a knowledge store with Power BI](knowledge-store-connect-power-bi.md)
-+ [Projection examples (how to shape and export enrichments)](knowledge-store-projections-examples.md)
++ [Define projections in a knowledge store](knowledge-store-projections-examples.md) ## Custom skills (advanced)
search Knowledge Store Connect Power Bi https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/search/knowledge-store-connect-power-bi.md
Previously updated : 06/30/2020 Last updated : 08/10/2021 # Connect a knowledge store with Power BI In this article, learn how to connect to and explore a knowledge store using Power Query in the Power BI Desktop app. You can get started faster with templates, or build a custom dashboard from scratch. This brief video below demonstrates how you can enrich your experience with your data by using Azure Cognitive Search in combination with Power BI. -
-> [!VIDEO https://www.youtube.com/embed/XWzLBP8iWqg?version=3&start=593&end=663]
--- + Follow the steps in [Create a knowledge store in the Azure portal](knowledge-store-create-portal.md) or [Create an Azure Cognitive Search knowledge store by using REST](knowledge-store-create-rest.md) to create the sample knowledge store used in this walkthrough. You will also need the name of the Azure Storage account that you used to create the knowledge store, along with its access key from the Azure portal. + [Install Power BI Desktop](https://powerbi.microsoft.com/downloads/)
+> [!VIDEO https://www.youtube.com/embed/XWzLBP8iWqg?version=3&start=593&end=663]
+ ## Sample Power BI template - Azure portal only When creating a [knowledge store using the Azure portal](knowledge-store-create-portal.md), you have the option of downloading a [Power BI template](https://github.com/Azure-Samples/cognitive-search-templates) on the second page of the **Import data** wizard. This template gives you several visualizations, such as WordCloud and Network Navigator, for text-based content.
search Knowledge Store Create Portal https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/search/knowledge-store-create-portal.md
Previously updated : 11/17/2020 Last updated : 08/10/2021 # Quickstart: Create an Azure Cognitive Search knowledge store in the Azure portal
-Knowledge store is a feature of Azure Cognitive Search that persists output from a content processing pipeline for subsequent analyses or downstream processing.
+Knowledge store is a feature of Azure Cognitive Search that persists output from an [AI enrichment pipeline](cognitive-search-concept-intro.md) for subsequent analyses or downstream processing.
-A pipeline accepts unstructured text and image content, applies AI powered by Cognitive Services (such as OCR and natural language processing), and outputs new structures and information that didn't previously exist. One of the physical artifacts created by a pipeline is a [knowledge store](knowledge-store-concept-intro.md), which you can access through tools to analyze and explore content.
+A pipeline accepts unstructured text and image content, applies AI powered by Cognitive Services (such as sentiment analysis and text translation), and outputs new structures and information that didn't previously exist. One of the physical artifacts created by a pipeline is a [knowledge store](knowledge-store-concept-intro.md), which you can access through tools that analyze and explore content in Azure Storage.
In this quickstart, you'll combine services and data in the Azure cloud to create a knowledge store. Once everything is in place, you'll run the **Import data** wizard in the portal to pull it all together. The end result is original text content plus AI-generated content that you can view in the portal ([Storage Explorer](knowledge-store-view-storage-explorer.md)).
Now that you have enriched your data using Cognitive Services and projected the
You can view content in Storage Explorer, or take it a step further with Power BI to gain insights through visualization.
-> [!div class="nextstepaction"]
-> [View with Storage Explorer](knowledge-store-view-storage-explorer.md)
-> [Connect with Power BI](knowledge-store-connect-power-bi.md)
++ [View with Storage Explorer](knowledge-store-view-storage-explorer.md)+++[Connect with Power BI](knowledge-store-connect-power-bi.md) > [!Tip] > If you want to repeat this exercise or try a different AI enrichment walkthrough, delete the *hotel-reviews-idxr* indexer. Deleting the indexer resets the free daily transaction counter back to zero for Cognitive Services processing.
search Knowledge Store Create Rest https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/search/knowledge-store-create-rest.md
Title: Create a knowledge store using REST
-description: Use the REST API and Postman to create an Azure Cognitive Search knowledge store for persisting enrichments from an AI enrichment pipeline.
+description: Use the REST API and Postman to create an Azure Cognitive Search knowledge store for persisting AI enrichments from skillset.
Previously updated : 11/18/2020 Last updated : 08/10/2021 # Create a knowledge store using REST and Postman
-A knowledge store contains output from an Azure Cognitive Search enrichment pipeline for later analysis or other downstream processing. An AI-enriched pipeline accepts image files or unstructured text files, indexes them by using Azure Cognitive Search, applies AI enrichments from Cognitive Services (such as image analysis and natural language processing), and then saves the results to a knowledge store in Azure Storage. You can use tools like Power BI or Storage Explorer in the Azure portal to explore the knowledge store.
+A knowledge store contains output from an Azure Cognitive Search enrichment pipeline for later analysis or other downstream processing. An AI-enriched pipeline accepts image files or unstructured text files, applies AI enrichments from Cognitive Services (such as image analysis and natural language processing), and then saves the output to a knowledge store in Azure Storage. You can use tools like Power BI or Storage Explorer in the Azure portal to explore the knowledge store.
-In this article, you use the REST API interface to ingest, index, and apply AI enrichments to a set of hotel reviews. The hotel reviews are imported into Azure Blob Storage. The results are saved as a knowledge store in Azure Table Storage.
+In this article, you use the REST API to ingest, enrich, and explore a set of customer reviews of hotel stays. To make the initial data set available, the hotel reviews are first imported into Azure Blob Storage. Post-processing, the results are saved as a knowledge store in Azure Table Storage.
-After you create the knowledge store, you can learn about how to access the knowledge store by using [Storage Explorer](knowledge-store-view-storage-explorer.md) or [Power BI](knowledge-store-connect-power-bi.md).
+After you create the knowledge store, explore its content using [Storage Explorer](knowledge-store-view-storage-explorer.md) or [Power BI](knowledge-store-connect-power-bi.md).
If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.
If you don't have an Azure subscription, create a [free account](https://azure.m
## Create services and load data
-This quickstart uses Azure Cognitive Search, Azure Blob Storage, and [Azure Cognitive Services](https://azure.microsoft.com/services/cognitive-services/) for the AI.
+This exercise uses Azure Cognitive Search, Azure Blob Storage, and [Azure Cognitive Services](https://azure.microsoft.com/services/cognitive-services/) for the AI.
-Because the workload is so small, Cognitive Services is tapped behind the scenes to provide free processing for up to 20 transactions daily. Because the data set is so small, you can skip creating or attaching a Cognitive Services resource.
+Because the workload is so small, Cognitive Services is tapped behind the scenes to provide free processing for up to 20 transactions daily. A small workload means that you can skip creating or attaching a Cognitive Services resource.
1. [Download HotelReviews_Free.csv](https://knowledgestoredemo.blob.core.windows.net/hotel-reviews/HotelReviews_Free.csv?sp=r&st=2019-11-04T01:23:53Z&se=2025-11-04T16:00:00Z&spr=https&sv=2019-02-02&sr=b&sig=siQgWOnI%2FDamhwOgxmj11qwBqqtKMaztQKFNqWx00AY%3D). This data is hotel review data saved in a CSV file (originates from Kaggle.com) and contains 19 pieces of customer feedback about a single hotel.
To get the value for `admin-key`, go to the Azure Cognitive Search service and s
### Review the request collection in Postman
+Knowledge stores are defined in skillsets, which are in turn attached to indexers. Creating a knowledge store requires that you create all of the upstream objects, including an index, data source, skillset, and indexer. Although an index is unrelated to a knowledge store, an indexer requires it for execution, so you will create one as an indexer prerequisite.
+ When you create a knowledge store, you must issue four HTTP requests: -- **PUT request to create the index**: This index holds the data that Azure Cognitive Search uses and returns.-- **POST request to create the datasource**: This datasource connects your Azure Cognitive Search behavior to the data and knowledge store's storage account.
+- **PUT request to create the index**: This index holds the data that Azure Cognitive Search uses and returns in query requests.
+- **POST request to create the datasource**: This datasource connects to your Azure Storage account.
- **PUT request to create the skillset**: The skillset specifies the enrichments that are applied to your data and the structure of the knowledge store. - **PUT request to create the indexer**: Running the indexer reads the data, applies the skillset, and stores the results. You must run this request last.
The [source code](https://github.com/Azure-Samples/azure-search-postman-samples/
![Screenshot showing Postman's interface for headers](media/knowledge-store-create-rest/postman-headers-ui.png) > [!Note]
-> You must set `api-key` and `Content-type` headers in all your requests. If Postman recognizes a variable, the variable appears in
-> orange text, as with `{{admin-key}}` in the preceding screenshot. If the variable is misspelled, it appears in red text.
+> All of the requests in the collection set `api-key` and `Content-type` headers, which are required. If Postman recognizes a variable, the variable appears in orange text, as with `{{admin-key}}` in the preceding screenshot. If the variable is misspelled, it appears in red text.
> ## Create an Azure Cognitive Search index
Set the structure of your Azure Cognitive Search index in the body of the reques
```
-This index definition is a combination of data that you'd like to present to the user (the name of the hotel, review content, the date), search metadata, and AI enhancement data (Sentiment, Keyphrases, and Language).
+This index definition is a combination of data that you'd like to present to the user (the name of the hotel, review content, the date), search metadata, and AI enhancement data (Sentiment, Key Phrases, and Language).
Select **Send** to issue the PUT request. You should see the status `201 - Created`. If you see a different status, in the **Body** pane, look for a JSON response that contains an error message.
+The index is created but not loaded. Importing documents occurs later when you run the indexer.
+ ## Create the datasource Next, connect Azure Cognitive Search to the hotel data you stored in Blob storage. To create the datasource, send a POST request to `https://{{search-service-name}}.search.windows.net/datasources?api-version={{api-version}}`. You must set the `api-key` and `Content-Type` headers as discussed earlier.
To generate the skillset, select the **Send** button in Postman to PUT the reque
The final step is to create the indexer. The indexer reads the data and activates the skillset. In Postman, select the **Create Indexer** request, and then review the body. The definition of the indexer refers to several other resources that you already created: the datasource, the index, and the skillset.
-The `parameters/configuration` object controls how the indexer ingests the data. In this case, the input data is in a single document that has a header line and comma-separated values. The document key is a unique identifier for the document. Before encoding, the document key is the URL of the source document. Finally, the skillset output values, like language code, sentiment, and key phrases, are mapped to their locations in the document. Although there's a single value for `Language`, `Sentiment` is applied to each element in the array of `pages`. `Keyphrases` is an array that's also applied to each element in the `pages` array.
+The `parameters/configuration` object controls how the indexer ingests the data. In this case, the input data is in a single CSV file that has a header line and comma-separated values. The document key is a unique identifier for the document. Before encoding, the document key is the URL of the source document. Finally, the skillset output values, like language code, sentiment, and key phrases, are mapped to their locations in the document. Although there's a single value for `Language`, `Sentiment` is applied to each element in the array of `pages`. `Keyphrases` is an array that's also applied to each element in the `pages` array.
After you set the `api-key` and `Content-type` headers and confirm that the body of the request is similar to the following source code, select **Send** in Postman. Postman sends a PUT request to `https://{{search-service-name}}.search.windows.net/indexers/{{indexer-name}}?api-version={{api-version}}`. Azure Cognitive Search creates and runs the indexer.
After you set the `api-key` and `Content-type` headers and confirm that the body
In the Azure portal, go to the Azure Cognitive Search service's **Overview** page. Select the **Indexers** tab, and then select **hotels-reviews-ixr**. If the indexer hasn't already run, select **Run**. The indexing task might raise some warnings related to language recognition. The data includes some reviews that are written in languages that aren't yet supported by the cognitive skills.
+## Clean up
+
+When you're working in your own subscription, it's a good idea at the end of a project to identify whether you still need the resources you created. Resources left running can cost you money. You can delete resources individually or delete the resource group to delete the entire set of resources.
+
+You can find and manage resources in the portal, using the **All resources** or **Resource groups** link in the left-navigation pane.
+
+If you are using a free service, remember that you are limited to three indexes, indexers, and data sources. You can delete individual items in the portal to stay under the limit.
+ ## Next steps Now that you've enriched your data by using Cognitive Services and projected the results to a knowledge store, you can use Storage Explorer or Power BI to explore your enriched data set.
search Knowledge Store View Storage Explorer https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/search/knowledge-store-view-storage-explorer.md
Previously updated : 06/30/2020 Last updated : 08/10/2021 # View a knowledge store with Storage Explorer
-In this article, you'll learn by example how to connect to and explore a knowledge store using Storage Explorer in the Azure portal.
+A [knowledge store](knowledge-store-concept-intro.md) is created by a skillset and saved to Azure Storage. In this article, you'll learn how to view the content of a knowledge store using Storage Explorer in the Azure portal.
## Prerequisites
-+ Follow the steps in [Create a knowledge store in Azure portal](knowledge-store-create-portal.md) to create the sample knowledge store used in this walkthrough.
++ Create a knowledge store in [Azure portal](knowledge-store-create-portal.md) or [Postman and the REST APIs](knowledge-store-create-rest.md).
-+ You will also need the name of the Azure storage account that you used to create the knowledge store, along with its access key from the Azure portal.
++ You will also need the name of the Azure Storage account that has the knowledge store, along with its access key from the Azure portal.
-## View, edit, and query a knowledge store in Storage Explorer
+## Start Storage Explorer
1. In the Azure portal, [open the Storage account](https://ms.portal.azure.com/#blade/HubsExtension/BrowseResourceBlade/resourceType/Microsoft.Storage%2storageAccounts/) that you used to create the knowledge store. 1. In the storage account's left navigation pane, click **Storage Explorer**.
-1. Expand the **TABLES** list to show a list of Azure table projections that were created when you ran the **Import Data** wizard on your hotel reviews sample data.
+## View, edit, and query tables
-Select any table to view the enriched data, including key phrases and sentiment scores.
+Both the portal and REST walkthroughs create a knowledge store in Table Storage.
+
+1. Expand the **TABLES** list to show a list of Azure table projections that were created when you created the knowledge store. The tables should contain content related to hotel reviews.
+
+1. Select any table to view the enriched data, including key phrases and sentiment scores.
![View tables in Storage Explorer](media/knowledge-store-view-storage-explorer/storage-explorer-tables.png "View tables in Storage Explorer")
-To change the data type for any table value or to change individual values in your table, click **Edit**. When you change the data type for any column in one table row, it will be applied to all rows.
+1. To change the data type for any table value or to change individual values in your table, click **Edit**. When you change the data type for any column in one table row, it will be applied to all rows.
![Edit table in Storage Explorer](media/knowledge-store-view-storage-explorer/storage-explorer-edit-table.png "Edit table in Storage Explorer")
-To run queries, click **Query** on the command bar and enter your conditions.
+1. To run queries, click **Query** on the command bar and enter your conditions.
![Query table in Storage Explorer](media/knowledge-store-view-storage-explorer/storage-explorer-query-table.png "Query table in Storage Explorer")
-## Clean up
-
-When you're working in your own subscription, it's a good idea at the end of a project to identify whether you still need the resources you created. Resources left running can cost you money. You can delete resources individually or delete the resource group to delete the entire set of resources.
-
-You can find and manage resources in the portal, using the **All resources** or **Resource groups** link in the left-navigation pane.
-
-If you are using a free service, remember that you are limited to three indexes, indexers, and data sources. You can delete individual items in the portal to stay under the limit.
- ## Next steps Connect this knowledge store to Power BI for deeper analysis, or move forward with code, using the REST API and Postman to create a different knowledge store. > [!div class="nextstepaction"] > [Connect with Power BI](knowledge-store-connect-power-bi.md)
+> [Create a knowledge store in Azure portal](knowledge-store-create-portal.md)
> [Create a knowledge store in REST](knowledge-store-create-rest.md)
search Samples Rest https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/search/samples-rest.md
Code samples from the Cognitive Search team demonstrate features and workflows.
| [Debug-sessions](https://github.com/Azure-Samples/azure-search-postman-samples/tree/master/Debug-sessions) | Source code for [Tutorial: Diagnose, repair, and commit changes to your skillset](cognitive-search-tutorial-debug-sessions.md). This article shows you how to use a skillset debug session in the Azure portal. REST is used to create the objects used during debug.| | [custom-analyzers](https://github.com/Azure-Samples/azure-search-postman-samples/tree/master/custom-analyzers) | Source code for [Tutorial: Create a custom analyzer for phone numbers](tutorial-create-custom-analyzer.md). This article explains how to use analyzers to preserve patterns and special characters in searchable content.| | [knowledge-store](https://github.com/Azure-Samples/azure-search-postman-samples/tree/master/knowledge-store) | Source code for [Create a knowledge store using REST and Postman](knowledge-store-create-rest.md). This article explains the necessary steps for populating a knowledge store used for knowledge mining workflows. |
-| [projections](https://github.com/Azure-Samples/azure-search-postman-samples/tree/master/projections) | Source code for [How to shape and export enrichments](knowledge-store-projections-examples.md). This article explains how to specify the physical data structures in a knowledge store.|
+| [projections](https://github.com/Azure-Samples/azure-search-postman-samples/tree/master/projections) | Source code for [Define projections in a knowledge store](knowledge-store-projections-examples.md). This article explains how to specify the physical data structures in a knowledge store.|
| [index-encrypted-blobs](https://github.com/Azure-Samples/azure-search-postman-samples/commit/f5ebb141f1ff98f571ab84ac59dcd6fd06a46718) | Source code for [How to index encrypted blobs using blob indexers and skillsets](search-howto-index-encrypted-blobs.md). This article shows how to index documents in Azure Blob Storage that have been previously encrypted using Azure Key Vault. | > [!Tip]
search Search Indexer Field Mappings https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/search/search-indexer-field-mappings.md
Previously updated : 01/28/2021 Last updated : 08/10/2021 # Field mappings and transformations using Azure Cognitive Search indexers
Some situations where field mappings are useful:
* You want to populate an index field with data from more than one data source, and the data sources each use different field names. * You need to Base64 encode or decode your data. Field mappings support several **mapping functions**, including functions for Base64 encoding and decoding.
+Field mappings in indexers are a simple way to map data fields to index fields, with some ability for light-weight data conversion. More complex data might require pre-processing to reshape it into a form that's conducive to indexing. One option you might consider is [Azure Data Factory](../data-factory/index.yml).
+ > [!NOTE]
-> Field mappings in indexers are a simple way to map data fields to index fields, with some ability for light-weight data conversion. More complex data might require pre-processing to reshape it into a form that's conducive to indexing. One option you might consider is [Azure Data Factory](../data-factory/index.yml).
+> Field mappings apply to search indexes only. For indexers that create [knowledge stores](knowledge-store-concept-intro.md), field mappings are ignored.
## Set up field mappings
security-center Azure Defender Dashboard https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/azure-defender-dashboard.md
The dashboard includes the following sections:
In this article, you learned about the Azure Defender dashboard.
-For more on Azure Defender, see [Introduction to Azure Defender](azure-defender.md)
- > [!div class="nextstepaction"]
-> [Enable Azure Defender](enable-azure-defender.md)
+> [Enable Azure Defender](enable-azure-defender.md)
+
+For more on Azure Defender, see [Introduction to Azure Defender](azure-defender.md)
security-center Defender For App Service Introduction https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/defender-for-app-service-introduction.md
For a full list of the Azure App Service alerts, see the [Reference table of ale
In this article, you learned about Azure Defender for App Service.
+> [!div class="nextstepaction"]
+> [Enable Azure Defender](enable-azure-defender.md)
+ For related material, see the following articles: - To export your alerts to Azure Sentinel, any third-party SIEM, or any other external tool, follow the instructions in [Stream alerts to a SIEM, SOAR, or IT Service Management solution](export-to-siem.md). - For a list of the Azure Defender for App Service alerts, see the [Reference table of alerts](alerts-reference.md#alerts-azureappserv).-- For more information on App Service plans, see [App Service plans](https://azure.microsoft.com/pricing/details/app-service/plans/).
-> [!div class="nextstepaction"]
-> [Enable Azure Defender](enable-azure-defender.md)
+- For more information on App Service plans, see [App Service plans](https://azure.microsoft.com/pricing/details/app-service/plans/).
security-center Defender For Dns Introduction https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/defender-for-dns-introduction.md
To protect your DNS layer, enable Azure Defender for DNS for each of your subscr
## Next steps
-In this article, you learned about Azure Defender for DNS. For related material, see the following article:
+In this article, you learned about Azure Defender for DNS.
-- Security alerts might be generated by Security Center or received by Security Center from different security products. To export all of these alerts to Azure Sentinel, any third-party SIEM, or any other external tool, follow the instructions in [Exporting alerts to a SIEM](continuous-export.md).
+> [!div class="nextstepaction"]
+> [Enable Azure Defender](enable-azure-defender.md)
-- > [!div class="nextstepaction"]
- > [Enable Azure Defender](enable-azure-defender.md)
+For related material, see the following article:
+
+- Security alerts might be generated by Security Center or received by Security Center from different security products. To export all of these alerts to Azure Sentinel, any third-party SIEM, or any other external tool, follow the instructions in [Exporting alerts to a SIEM](continuous-export.md).
security-center Defender For Resource Manager Introduction https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/defender-for-resource-manager-introduction.md
To investigate security alerts from Azure Defender for Resource
## Next steps
-In this article, you learned about Azure Defender for Resource Manager. For related material, see the following article:
+In this article, you learned about Azure Defender for Resource Manager.
-- Security alerts might be generated by Security Center or received by Security Center from different security products. To export all of these alerts to Azure Sentinel, any third-party SIEM, or any other external tool, follow the instructions in [Exporting alerts to a SIEM](continuous-export.md).
+> [!div class="nextstepaction"]
+> [Enable Azure Defender](enable-azure-defender.md)
-- > [!div class="nextstepaction"]
- > [Enable Azure Defender](enable-azure-defender.md)
+For related material, see the following article:
+
+- Security alerts might be generated by Security Center or received by Security Center from different security products. To export all of these alerts to Azure Sentinel, any third-party SIEM, or any other external tool, follow the instructions in [Exporting alerts to a SIEM](continuous-export.md).
security-center Defender For Servers Introduction https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/defender-for-servers-introduction.md
Azure Defender for servers adds threat detection and advanced defenses for your
For Windows, Azure Defender integrates with Azure services to monitor and protect your Windows-based machines. Security Center presents the alerts and remediation suggestions from all of these services in an easy-to-use format.
-For Linux, Azure Defender collects audit records from Linux machines by using **auditd**, one of the most common Linux auditing frameworks. auditd lives in the mainline kernel.
+For Linux, Azure Defender collects audit records from Linux machines by using auditd, one of the most common Linux auditing frameworks. auditd lives in the mainline kernel.
## What are the benefits of Azure Defender for servers?
You can simulate alerts by downloading one of the following playbooks:
In this article, you learned about Azure Defender for servers.
-For related material, see the following articles:
+> [!div class="nextstepaction"]
+> [Enable Azure Defender](enable-azure-defender.md)
-- Whether an alert is generated by Security Center, or received by Security Center from a different security product, you can export it. To export your alerts to Azure Sentinel, any third-party SIEM, or any other external tool, follow the instructions in [Exporting alerts to a SIEM](continuous-export.md).
+For related material, see the following:
-- > [!div class="nextstepaction"]
- > [Enable Azure Defender](enable-azure-defender.md)
+- Whether an alert is generated by Security Center, or received by Security Center from a different security product, you can export it. To export your alerts to Azure Sentinel, any third-party SIEM, or any other external tool, follow the instructions in [Exporting alerts to a SIEM](continuous-export.md).
security-center Deploy Vulnerability Assessment Vm https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/deploy-vulnerability-assessment-vm.md
Within 48 hrs of the disclosure of a critical vulnerability, Qualys incorporates
> [!div class="nextstepaction"] > [Remediate the findings from your vulnerability assessment solution](remediate-vulnerability-findings-vm.md) - Security Center also offers vulnerability analysis for your: - SQL databases - see [Explore vulnerability assessment reports in the vulnerability assessment dashboard](defender-for-sql-on-machines-vulnerability-assessment.md#explore-vulnerability-assessment-reports)
security-center Just In Time Explained https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/just-in-time-explained.md
If you want to create custom roles that can work with JIT, you'll need the detai
## Next steps
-This page explained _why_ just-in-time (JIT) virtual machine (VM) access should be used.
-
-Advance to the how-to article to learn about enabling JIT and requesting access to your JIT-enabled VMs:
+This page explained _why_ just-in-time (JIT) virtual machine (VM) access should be used. To learn about _how_ to enable JIT and request access to your JIT-enabled VMs, see the following:
> [!div class="nextstepaction"] > [How to secure your management ports with JIT](security-center-just-in-time.md)
security-center Release Notes https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/release-notes.md
Previously updated : 08/04/2021 Last updated : 08/08/2021
Updates in August include:
- [Microsoft Defender for Endpoint for Linux now supported by Azure Defender for servers (in preview)](#microsoft-defender-for-endpoint-for-linux-now-supported-by-azure-defender-for-servers-in-preview) - [Two new recommendations for managing endpoint protection solutions (in preview)](#two-new-recommendations-for-managing-endpoint-protection-solutions-in-preview)
+- [Built-in troubleshooting and guidance for solving common issues](#built-in-troubleshooting-and-guidance-for-solving-common-issues)
### Microsoft Defender for Endpoint for Linux now supported by Azure Defender for servers (in preview)
We've added two **preview** recommendations to deploy and maintain the endpoint
> :::image type="content" source="media/release-notes/freshness-interval.png" alt-text="Freshness interval indicator for these two new Security Center recommendations"::: - [Microsoft Defender for Endpoint for Linux now supported by Azure Defender for servers (in preview)](#microsoft-defender-for-endpoint-for-linux-now-supported-by-azure-defender-for-servers-in-preview)
+### Built-in troubleshooting and guidance for solving common issues
+
+A new, dedicated area of the Security Center pages in the Azure portal provides a collated, ever-growing set of self-help materials for solving common challenges with Security Center and Azure Defender.
+
+When you're facing an issue, or are seeking advice from our support team, **Diagnose and solve problems** is another tool to help you find the solution:
+
+
## July 2021
security-center Secure Score Security Controls https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/secure-score-security-controls.md
In some cases, you'll see a control max score greater than zero, but the impact
## Next steps
-This article described the secure score and the security controls it introduces. For related material, see the following articles:
+This article described the secure score and the included security controls.
-- [Learn about the different elements of a recommendation](security-center-recommendations.md)-- [Learn how to remediate recommendations](security-center-remediate-recommendations.md)-- [View the GitHub-based tools for working programmatically with secure score](https://github.com/Azure/Azure-Security-Center/tree/master/Secure%20Score)
+> [!div class="nextstepaction"]
+> [Access and track your secure score](secure-score-access-and-track.md)
+For related material, see the following articles:
-> [!div class="nextstepaction"]
-> [Access and track your secure score](secure-score-access-and-track.md)
+- [Learn about the different elements of a recommendation](security-center-recommendations.md)
+- [Learn how to remediate recommendations](security-center-remediate-recommendations.md)
+- [View the GitHub-based tools for working programmatically with secure score](https://github.com/Azure/Azure-Security-Center/tree/master/Secure%20Score)
security-center Security Center Just In Time https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/security-center-just-in-time.md
You can gain insights into VM activities using log search. To view the logs:
## Next steps
-In this article, you learned how to set up and use just-in-time VM access. To learn why JIT should be used, read the concept article explaining the threats it's defending against:
+In this article, you learned _how_ to configure and use just-in-time VM access. To learn _why_ JIT should be used, read the concept article explaining the threats it defends against:
> [!div class="nextstepaction"] > [JIT explained](just-in-time-explained.md)
security-center Tutorial Protect Resources https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/security-center/tutorial-protect-resources.md
Title: Access & application controls tutorial - Azure Security Center description: This tutorial shows you how to configure a just-in-time VM access policy and an application control policy. - Last updated 12/03/2018
In this tutorial, you learned how to limit your exposure to threats by:
Advance to the next tutorial to learn about responding to security incidents. > [!div class="nextstepaction"]
-> [Tutorial: Respond to security incidents](tutorial-security-incident.md)
-
-<!--Image references-->
-[1]: ./media/tutorial-protect-resources/just-in-time-vm-access.png
-[2]: ./media/tutorial-protect-resources/add-port.png
-[3]: ./media/tutorial-protect-resources/adaptive-application-control-options.png
-[4]: ./media/tutorial-protect-resources/recommended-resource-groups.png
+> [Tutorial: Respond to security incidents](tutorial-security-incident.md)
sentinel Connect Windows Firewall https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/sentinel/connect-windows-firewall.md
The solution collects Windows firewall events from the Windows machines on which
## Validate connectivity
-Because Windows Firewall logs are sent to Azure Sentinel only when the local log file reaches capacity, leaving the log at its default size of 4096 KB will most likely result in high collection latency. You can lower the latency by lowering the log file size. For more information, see [configure the Windows Firewall log](/windows/security/threat-protection/windows-firewall/configure-the-windows-firewall-log).
+Because Windows Firewall logs are sent to Azure Sentinel only when the local log file reaches capacity, leaving the log at its default size of 4096 KB will most likely result in high collection latency. You can lower the latency by lowering the log file size. For more information, see [configure the Windows Firewall log](/windows/security/threat-protection/windows-firewall/configure-the-windows-firewall-log).
-> [!TIP]
-> The data collection configuration requires a minimum of 1000 new lines in the log file before data is collected, and therefore the log file size should not be set lower than 100 KB. While smaller log sizes will reduce collection latency, they might also negatively impact the local machine's performance.
+> [!NOTE]
+>
+> - While smaller log sizes will reduce collection latency, they might also negatively impact the local machine's performance.
+>
+> - The data collection configuration requires a minimum of 1000 new lines in the log file before data is collected. Therefore, the log file size should be set to no less than 100 (one hundred) KB, as this will ensure the accumulation of 1000 lines.
## Next steps In this document, you learned how to connect Windows firewall to Azure Sentinel. To learn more about Azure Sentinel, see the following articles:
spring-cloud Quotas https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/spring-cloud/quotas.md
Azure Spring Cloud service instances | per region per subscription | 10 | 10
Total app instances | per Azure Spring Cloud service instance | 25 | 500 Custom Domains | per Azure Spring Cloud service instance | 0 | 25 Persistent volumes | per Azure Spring Cloud service instance | 1 GB/app x 10 apps | 50 GB/app x 10 apps
-Public IPs | 5 | 5
+Inbound Public Endpoints | per Azure Spring Cloud service instance| 10 <sup>1</sup> | 10 <sup>1</sup>
+Outbound Public IPs | per Azure Spring Cloud service instance| 1 <sup>2</sup> | 2 <sup>2</sup> <br> 1 if using VNet<sup>2</sup>
+
+<sup>1</sup> You can increase this limit via support request to a maximum of 1 per app.
+
+<sup>2</sup> You can increase this limit via support request to a maximum of 10.
> [!TIP] > Limits listed for Total app instances per service instance apply for apps and deployments in any state, including stopped state. Please delete apps or deployments that are not in use.
static-web-apps Private Endpoint https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/static-web-apps/private-endpoint.md
Last updated 7/28/2021
You can use a private endpoint (also called private link) to restrict access to your static web app so that it is only accessible from your private network.
+> [!NOTE]
+> Private endpoints support in Static Web Apps is currently in preview.
+ ## How it works An Azure Virtual Network (VNet) is a network just like you might have in a traditional data center, but resources within the VNet talk to each other securely on the Microsoft backbone network.
In this section, you create a private endpoint for your static web app.
## Testing your private endpoint
-Since your application is no longer publicly available, the only way to access it is from inside of your virtual network. To test, setup a virtual machine inside of your virtual network and navigate to your site.
+Since your application is no longer publicly available, the only way to access it is from inside of your virtual network. To test, set up a virtual machine inside of your virtual network and navigate to your site.
## Next steps
storage Files Smb Protocol https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/storage/files/files-smb-protocol.md
You can disable encryption in transit for an Azure storage account. When encrypt
## SMB protocol settings Azure Files offers multiple settings for the SMB protocol. -- **SMB Multichannel**: Enable/disable SMB multichannel (premium file shares only). To learn how to enable SMB Multichannel, see [Enable SMB Multichannel on a FileStorage storage account](storage-files-enable-smb-multichannel.md).
+- **SMB Multichannel**: Enable/disable SMB Multichannel (premium file shares only). To learn how to enable SMB Multichannel, see [Enable SMB Multichannel on a FileStorage storage account](storage-files-enable-smb-multichannel.md). The default value for SMB Multichannel is disabled.
- **SMB versions**: Which versions of SMB are allowed. Supported protocol versions are SMB 3.1.1, SMB 3.0, and SMB 2.1. By default, all SMB versions are allowed, although SMB 2.1 is disallowed if "require secure transit" is enabled, since SMB 2.1 does not support encryption in transit. - **Authentication methods**: Which SMB authentication methods are allowed. Supported authentication methods are NTLMv2 and Kerberos. By default, all authentication methods are allowed. Removing NTLMv2 disallows using the storage account key to mount the Azure file share. - **Kerberos ticket encryption**: Which encryption algorithms are allowed. Supported encryption algorithms are RC4-HMAC and AES-256.
Azure Files offers multiple settings for the SMB protocol.
SMB protocol settings can be toggled via the Azure PowerShell module.
-To changing the SMB protocol settings, you must [install the 3.7.1-preview version](https://www.powershellgallery.com/packages/Az.Storage/3.7.1-preview) of the Azure Storage PowerShell module.
+# [Portal](#tab/azure-portal)
+The SMB protocol settings can be viewed and changed using PowerShell or CLI. Please select the desired tab to see the steps on how to get and set the SMB protocol settings.
+
+# [PowerShell](#tab/azure-powershell)
+To get or set the SMB protocol settings, you must provide a reference to a storage account, either directly by providing the resource group and storage account names, or by providing a storage account object you've acquired from the `Get-AzStorageAccount` cmdlet. The following examples use the latter method to get and set the SMB protocol settings.
Remember to replace `<resource-group>` and `<storage-account>` with the appropriate values for your environment before running these PowerShell commands.
Remember to replace `<resource-group>` and `<storage-account>` with the appropri
$resourceGroupName = "<resource-group>" $storageAccountName = "<storage-account>"
+$storageAccount = Get-AzStorageAccount `
+ -ResourceGroupName $resourceGroupName `
+ -StorageAccountName $storageAccountName
+```
+
+To get the SMB protocol settings, use the `Get-AzStorageFileServiceProperty` cmdlet. If you've never modified the SMB protocol settings, the values returned by the cmdlet will be null. Null returned values should be interpreted as "default settings are in effect". To make this more user-friendly, the following PowerShell commands replace null values with the human-readable default values.
+
+```PowerShell
+# Replacement values for null parameters. If you copy this into your own
+# scripts, you will need to ensure that you keep these variables up-to-date with any new
+# options we may add to these parameters in the future.
+$smbMultichannelEnabled = $false
+$smbProtocolVersions = "SMB2.1", "SMB3.0", "SMB3.1.1"
+$smbAuthenticationMethods = "NTLMv2", "Kerberos"
+$smbKerberosTicketEncryption = "RC4-HMAC", "AES-256"
+$smbChannelEncryption = "AES-128-CCM", "AES-128-GCM", "AES-256-GCM"
+
+Get-AzStorageFileServiceProperty -StorageAccount $storageAccount | `
+ Select-Object -Property `
+ ResourceGroupName, `
+ StorageAccountName, `
+ @{
+ Name = "SmbMultichannelEnabled";
+ Expression = {
+ if ($null -eq $_.ProtocolSettings.Smb.Multichannel.Enabled) {
+ $smbMultichannelEnabled
+ } else {
+ $_.ProtocolSettings.Smb.Multichannel.Enabled
+ }
+ }
+ },
+ @{
+ Name = "SmbProtocolVersions";
+ Expression = {
+ if ($null -eq $_.ProtocolSettings.Smb.Versions) {
+ [String]::Join(", ", $smbProtocolVersions)
+ } else {
+ [String]::Join(", ", $_.ProtocolSettings.Smb.Versions)
+ }
+ }
+ },
+ @{
+ Name = "SmbChannelEncryption";
+ Expression = {
+ if ($null -eq $_.ProtocolSettings.Smb.ChannelEncryption) {
+ [String]::Join(", ", $smbChannelEncryption)
+ } else {
+ [String]::Join(", ", $_.ProtocolSettings.Smb.ChannelEncryption)
+ }
+ }
+ },
+ @{
+ Name = "SmbAuthenticationMethods";
+ Expression = {
+ if ($null -eq $_.ProtocolSettings.Smb.AuthenticationMethods) {
+ [String]::Join(", ", $smbAuthenticationMethods)
+ } else {
+ [String]::Join(", ", $_.ProtocolSettings.Smb.AuthenticationMethods)
+ }
+ }
+ },
+ @{
+ Name = "SmbKerberosTicketEncryption";
+ Expression = {
+ if ($null -eq $_.ProtocolSettings.Smb.KerberosTicketEncryption) {
+ [String]::Join(", ", $smbKerberosTicketEncryption)
+ } else {
+ [String]::Join(", ", $_.ProtocolSettings.Smb.KerberosTicketEncryption)
+ }
+ }
+ }
+```
+
+Depending on your organization's security, performance, and compatibility requirements, you may wish to modify the SMB protocol settings. The following PowerShell command restricts your SMB file shares to only the most secure options.
+
+> [!Important]
+> Restricting SMB Azure file shares to only the most secure options may result in some clients not being able to connect if they do not meet the requirements. For example, AES-256-GCM was introduced as an option for SMB channel encryption starting in Windows 10, version 21H1. This means that older clients that do not support AES-256-GCM will not be able to connect.
+
+```PowerShell
Update-AzStorageFileServiceProperty ` -ResourceGroupName $resourceGroupName ` -StorageAccountName $storageAccountName `
Update-AzStorageFileServiceProperty `
-SmbProtocolVersion "SMB3.1.1" ```
+# [Azure CLI](#tab/azure-cli)
+To get the SMB protocol settings, use the `az storage account file-service-properties show` command. If you've never modified the SMB protocol settings, the values returned by the command will be null. Null returned values should be interpreted as "default settings are in effect". To make this more user-friendly, the following Bash commands replace null values with the human-readable default values.
+
+```bash
+resourceGroupName="<resource-group>"
+storageAccountName="<storage-account>"
+
+# Values to be replaced
+replaceSmbMultichannel="\"smbMultichannelEnabled\": null"
+replaceSmbProtocolVersion="\"smbProtocolVersions\": null"
+replaceSmbChannelEncryption="\"smbChannelEncryption\": null"
+replaceSmbAuthenticationMethods="\"smbAuthenticationMethods\": null"
+replaceSmbKerberosTicketEncryption="\"smbKerberosTicketEncryption\": null"
+
+# Replacement values for null parameters. If you copy this into your own
+# scripts, you will need to ensure that you keep these variables up-to-date with any new
+# options we may add to these parameters in the future.
+defaultSmbMultichannelEnabled="\"smbMultichannelEnabled\": false"
+defaultSmbProtocolVersions="\"smbProtocolVersions\": \"SMB2.1;SMB3.0;SMB3.1.1\""
+defaultSmbChannelEncryption="\"smbChannelEncryption\": \"AES-128-CCM;AES-128-GCM;AES-256-GCM\""
+defaultSmbAuthenticationMethods="\"smbAuthenticationMethods\": \"NTLMv2;Kerberos\""
+defaultSmbKerberosTicketEncryption="\"smbKerberosTicketEncryption\": \"RC4-HMAC;AES-256\""
+
+# Build JMESPath query string
+query="{"
+query="${query}smbMultichannelEnabled: protocolSettings.smb.multichannel.enabled,"
+query="${query}smbProtocolVersions: protocolSettings.smb.versions,"
+query="${query}smbChannelEncryption: protocolSettings.smb.channelEncryption,"
+query="${query}smbAuthenticationMethods: protocolSettings.smb.authenticationMethods,"
+query="${query}smbKerberosTicketEncryption: protocolSettings.smb.kerberosTicketEncryption"
+query="${query}}"
+
+# Get protocol settings from the Azure Files FileService object
+protocolSettings=$(az storage account file-service-properties show \
+ --resource-group $resourceGroupName \
+ --account-name $storageAccountName \
+ --query "${query}")
+
+# Replace returned values if null with default values
+protocolSettings="${protocolSettings/$replaceSmbMultichannel/$defaultSmbMultichannelEnabled}"
+protocolSettings="${protocolSettings/$replaceSmbProtocolVersion/$defaultSmbProtocolVersion}"
+protocolSettings="${protocolSettings/$replaceSmbChannelEncryption/$defaultSmbChannelEncryption}"
+protocolSettings="${protocolSettings/$replaceSmbAuthenticationMethods/$defaultSmbAuthenticationMethods}"
+protocolSettings="${protocolSettings/$replaceSmbKerberosTicketEncryption/$defaultSmbKerberosTicketEncryption}"
+
+# Print returned settings
+echo $protocolSettings
+```
+
+Depending on your organizations security, performance, and compatibility requirements, you may wish to modify the SMB protocol settings. The following Azure CLI command restricts your SMB file shares to only the most secure options.
+
+> [!Important]
+> Restricting SMB Azure file shares to only the most secure options may result in some clients not being able to connect if they do not meet the requirements. For example, AES-256-GCM was introduced as an option for SMB channel encryption starting in Windows 10, version 21H1. This means that older clients that do not support AES-256-GCM will not be able to connect.
+
+```bash
+az storage account file-service-properties update \
+ --resource-group $resourceGroupName \
+ --account-name $storageAccountName \
+ --versions "SMB3.1.1" \
+ --channel-encryption "AES-256-GCM" \
+ --auth-methods "Kerberos" \
+ --kerb-ticket-encryption "AES-256"
+```
++ ## Limitations SMB file shares in Azure Files support a subset of features supported by SMB protocol and the NTFS file system. Although most use cases and applications do not require these features, some applications may not work properly with Azure Files if they rely on unsupported features. The following features are not supported:
virtual-machines Cli Ps Findimage https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/virtual-machines/linux/cli-ps-findimage.md
To deploy this image, you need to accept the terms and provide the purchase plan
## Accept the terms
-To view and accept the license terms, use the [az vm image accept-terms](/cli/azure/vm/image/terms) command. When you accept the terms, you enable programmatic deployment in your subscription. You only need to accept terms once per subscription for the image. For example:
+To view and accept the license terms, use the [az vm image terms](/cli/azure/vm/image/terms) command. When you accept the terms, you enable programmatic deployment in your subscription. You only need to accept terms once per subscription for the image. For example:
```azurecli az vm image terms show --urn bitnami:rabbitmq:rabbitmq:latest
virtual-wan Create Bgp Peering Hub Portal https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/virtual-wan/create-bgp-peering-hub-portal.md
# How to create BGP peering with virtual hub (Preview) - Azure portal
-This article helps you configure an Azure Virtual WAN hub router to peer with a Network Virtual Appliance (NVA) in your virtual network using the Azure portal. The hub router learns routes from the NVA in a spoke VNet that is connected to a virtual WAN hub. The hub router also advertises the virtual network routes to the NVA. For more information, see [About BGP peering with a virtual hub](scenario-bgp-peering-hub.md).
+This article helps you configure an Azure Virtual WAN hub router to peer with a Network Virtual Appliance (NVA) in your virtual network using the Azure portal. The virtual hub router learns routes from the NVA in a spoke VNet that is connected to a virtual WAN hub. The virtual hub router also advertises the virtual network routes to the NVA. For more information, see [Scenario: BGP peering with a virtual hub](scenario-bgp-peering-hub.md).
[!INCLUDE [Gated public preview SLA link](../../includes/virtual-wan-gated-public-preview-sla.md)]
This article helps you configure an Azure Virtual WAN hub router to peer with a
## Prerequisites > [!IMPORTANT]
-> The BGP peering with Virtual WAN hub router feature is currently in gated public preview. If you are interested in trying this feature, please email **previewbgpwithvhub@microsoft.com** along with the Resource ID of your Virtual WAN resource.
+> The BGP peering with Virtual WAN hub feature is currently in gated public preview. If you are interested in trying this feature, please email **previewbgpwithvhub@microsoft.com** along with the Resource ID of your Virtual WAN resource.
> > To locate the Resource ID, open the Azure portal, navigate to your Virtual WAN resource, and click **Settings > Properties > Resource ID.**<br> Example: `/subscriptions/<subscriptionID>/resourceGroups/<resourceGroupName>/providers/Microsoft.Network/virtualWans/<virtualWANname>` >
In this section, you create a connection between your hub and VNet.
## Next steps
-* For more information about BGP scenarios, see [About BGP peering with a virtual hub](scenario-bgp-peering-hub.md).
+* For more information about BGP scenarios, see [Scenario: BGP peering with a virtual hub](scenario-bgp-peering-hub.md).
virtual-wan Scenario Bgp Peering Hub https://github.com/MicrosoftDocs/azure-docs/commits/master/articles/virtual-wan/scenario-bgp-peering-hub.md
Last updated 08/06/2021
-# About BGP peering with a virtual hub (Preview)
+# Scenario: BGP peering with a virtual hub (Preview)
-In this scenario, the Azure Virtual WAN virtual hub router acts as a route manager and provides simplification in routing operation within and across virtual hubs. The virtual hub's router, also called the virtual router, does the following:
+Azure Virtual WAN hub router, also called as virtual hub router, acts as a route manager and provides simplification in routing operation within and across virtual hubs. In other words, a virtual hub router does the following:
-* Simplifies route management by being the route manager with VPN, ExpressRoute, and other entities or gateways in the virtual hub.
+* Simplifies routing management by being the central routing engine talking to gateways such as VPN, ExpressRoute, P2S, and Network Virtual Appliances (NVA).
* Enables advance routing scenarios of custom route tables, association, and propagation of routes. * Acts as the router for traffic transiting between/to virtual networks connected to a virtual hub.
-The virtual router now also exposes the ability to peer with it, thereby exchanging routing information directly through Border Gateway Protocol (BGP) routing protocol. Network Virtual Appliances (NVAs) or a BGP end point provisioned in a virtual network connected to a virtual hub, can directly peer with the virtual hub router if it supports the BGP routing protocol and ensures that ASN on the NVA is set up to be different from the virtual hub ASN.
+The virtual hub router now also exposes the ability to peer with it, thereby exchanging routing information directly through Border Gateway Protocol (BGP) routing protocol. NVA or a BGP end point provisioned in a virtual network connected to a virtual hub, can directly peer with the virtual hub router if it supports the BGP routing protocol and ensures that ASN on the NVA is set up to be different from the virtual hub ASN.
## Benefits and considerations
The virtual router now also exposes the ability to peer with it, thereby exchang
* The virtual hub router only supports 16-bit (2 bytes) ASN. * The virtual network connection that has the NVA BGP connection endpoint must always be associated and propagating to defaultRouteTable. Custom route tables are not supported at this time. * The virtual hub router supports transit connectivity between virtual networks connected to virtual hubs. This has nothing to do with this feature for BGP peering capability as Virtual WAN already supports transit connectivity. Examples:
- * VNET1: NVA1 connected to Virtual Hub 1 -> (transit connectivity) -> VNET2: NVA2 connected to Virtual Hub 1.
- * VNET1: NVA1 connected to Virtual Hub 1 -> (transit connectivity) -> VNET2: NVA2 connected to Virtual Hub 2.
+ * VNET1: NVA1 connected to Virtual Hub 1 -> (transit connectivity) -> VNET2: NVA2 connected to Virtual Hub 1.
+ * VNET1: NVA1 connected to Virtual Hub 1 -> (transit connectivity) -> VNET2: NVA2 connected to Virtual Hub 2.
* You can use your own public ASNs or private ASNs in your network virtual appliance. You can't use the ranges reserved by Azure or IANA. The following ASNs are reserved by Azure or IANA: * ASNs reserved by Azure: * Public ASNs: 8074, 8075, 12076
The virtual router now also exposes the ability to peer with it, thereby exchang
||| | Number of routes each BGP peer can advertise to the virtual hub.| The hub can only accept a maximum number of 10,000 routes (total) from its connected resources. For example, if a virtual hub has a total of 6000 routes from the connected virtual networks, branches, virtual hubs etc., then when a new BGP peering is configured with an NVA, the NVA can only advertise up to 4000 routes. |
-## About hub routing
+## BGP peering scenarios
-A virtual hub route table can contain one or more routes. A route includes its name, a label, a destination type, a list of destination prefixes, and next hop information for a packet to be routed. A connection typically will have a routing configuration that associates or propagates to a route table.
+This section describes scenarios where BGP peering feature can be utilized to configure routing.
-## <a name="vnet-vnet"></a>VNet-to-VNet scenario
+## <a name="vnet-vnet"></a>Transit VNet connectivity
:::image type="content" source="./media/scenario-bgp-peering-hub/vnet-vnet.png" alt-text="Graphic with VNet-to-VNet routing.":::
-In this example, the virtual hub named "Hub 1" is connected to several virtual networks.
+In this scenario, the virtual hub named "Hub 1" is connected to several virtual networks. The goal is to establish routing between virtual networks VNET1 and VNET5.
### Configuration steps without BGP peering
-To establish routing between virtual networks VNET1 and VNET5, the following steps are required when BGP peering is not used on the virtual hub:
+The following steps are required when BGP peering is not used on the virtual hub:
Virtual hub configuration
Virtual network configuration
### Configuration steps with BGP peering
-The maintenance of these static routes and UDR can become complex if the VNET5 configuration changes frequently. To address this challenge, the BGP peering with virtual WAN hub feature can be used and the routing configuration must be changed to the following steps:
+In the previous configuration, the maintenance of the static routes and UDR can become complex if the VNET5 configuration changes frequently. To address this challenge, the BGP peering with a virtual hub feature can be used and the routing configuration must be changed to the following steps:
Virtual hub configuration
The table below shows few entries from Hub 1's effective routes in the defaultRo
Configuring routing in this manner using the feature eliminates the need for static route entries on the virtual hub. Therefore, the configuration is simpler and route tables are updated dynamically when the configuration in connected virtual networks (like VNET5) changes.
-## <a name="branch-vnet"></a>Branch-to-VNet scenario
+## <a name="branch-vnet"></a>Branch VNet connectivity
:::image type="content" source="./media/scenario-bgp-peering-hub/branch-vnet.png" alt-text="Graphic with Branch-to-VNet routing.":::
-In this scenario, the on-premises site named "NVA Branch 1" has a VPN configured to terminate on the VNET2 NVA.
+In this scenario, the on-premises site named "NVA Branch 1" has a VPN configured to terminate on the VNET2 NVA. The goal is to configure routing between NVA Branch 1 and virtual network VNET1.
### Configuration steps without BGP peering
-To configure routing between NVA Branch 1 and virtual network VNET1, the following steps are required when BGP peering is not used on the virtual hub:
+The following steps are required when BGP peering is not used on the virtual hub:
Virtual hub configuration
Virtual network configuration
### Configuration steps with BGP peering
-Over time, the destination prefixes in NVA Branch 1 may change, or there may be many sites like NVA Branch 1, which need connectivity to VNET1. This would result in needing updates to the static routes on the Hub 1 and the VNET2 connection, which can get cumbersome. In such cases, we can use the "BGP peer with virtual hub" feature and the configuration steps for routing connecting would be as given below.
+Over time, the destination prefixes in NVA Branch 1 may change, or there may be many sites like NVA Branch 1, which need connectivity to VNET1. This would result in needing updates to the static routes on the Hub 1 and the VNET2 connection, which can get cumbersome. In such cases, we can use the BGP peering with a virtual hub feature and the configuration steps for routing connectivity would be as given below.
Virtual hub configuration