+Within the NPS extension, you can designate an Active Directory attribute to be used as the UPN for Azure AD Multi-Factor Authentication. This enables you to protect your on-premises resources with two-step verification without modifying your on-premises UPNs.

+| LDAP_ALTERNATE_LOGINID_ATTRIBUTE | string | Empty | Designate the name of Active Directory attribute that you want to use as the UPN. This attribute is used as the AlternateLoginId attribute. If this registry value is set to a [valid Active Directory attribute](/windows/win32/adschema/attributes-all) (for example, mail or displayName), then the attribute's value is used as the user's UPN for authentication. If this registry value is empty or not configured, then AlternateLoginId is disabled and the user's UPN is used for authentication. |

+- [Using authentication context with Microsoft Purview Information Protection and SharePoint](/microsoft-365/compliance/sensitivity-labels-teams-groups-sites#more-information-about-the-dependencies-for-the-authentication-context-option)

+Configuration parameters can be loaded from many sources, like a JavaScript file or from environment variables. Below, an *authConfig.js* file is used.

+Import the configuration object from *authConfig.js* file. MSAL Node can be initialized minimally as below. See the available [configuration options](https://github.com/AzureAD/microsoft-authentication-library-for-js/blob/dev/lib/msal-node/docs/configuration.md).

+const { msalConfig } = require('./authConfig')

+/**

+* Initialize a public client application. For more information, visit:

+* https://github.com/AzureAD/microsoft-authentication-library-for-js/blob/dev/lib/msal-node/docs/initialize-public-client-application.md

+*/

+clientApplication = new PublicClientApplication(msalConfig);

+- If you build a Node.js Electron app, use a custom string protocol instead of a regular web (https://) redirect URI in order to handle the redirection step of the authorization flow, for instance `msal{Your_Application/Client_Id}://auth` (e.g. *msalfa29b4c9-7675-4b61-8a0a-bf7b2b4fda91://auth*). The custom string protocol name shouldn't be obvious to guess and should follow the suggestions in the [OAuth2.0 specification for Native Apps](https://tools.ietf.org/html/rfc8252#section-7.1).

+>

+- Redirect URI: `msal{Your_Application/Client_Id}://auth`

+ npm install --save-dev babel electron@18.2.3 webpack

+ :::code language="html" source="~/ms-identity-JavaScript-nodejs-desktop/App/https://docsupdatetracker.net/index.html":::

+ :::code language="js" source="~/ms-identity-JavaScript-nodejs-desktop/App/main.js":::

+In the code snippet above, we initialize an Electron main window object and create some event handlers for interactions with the Electron window. We also import configuration parameters, instantiate *authProvider* class for handling sign-in, sign-out and token acquisition, and call the Microsoft Graph API.

+4. In the same folder (*App*), create another file named *renderer.js* and add the following code:

+ :::code language="js" source="~/ms-identity-JavaScript-nodejs-desktop/App/renderer.js":::

+The renderer methods are exposed by the preload script found in the *preload.js* file in order to give the renderer access to the `Node API` in a secure and controlled way

+5. Then, create a new file *preload.js* and add the following code:

+ :::code language="js" source="~/ms-identity-JavaScript-nodejs-desktop/App/preload.js":::

+This preload script exposes a renderer methods to give the renderer process controlled access to some `Node APIs` by applying IPC channels that have been configured for communication between the main and renderer processes.

+6. Next, create *UIManager.js* class inside the *App* folder and add the following code:

+ :::code language="js" source="~/ms-identity-JavaScript-nodejs-desktop/App/UIManager.js":::

+7. After that, create *CustomProtocolListener.js* class and add the following code there:

+ :::code language="js" source="~/ms-identity-JavaScript-nodejs-desktop/App/CustomProtocolListener.js":::

+*CustomProtocolListener* class can be instantiated in order to register and unregister a custom typed protocol on which MSAL Node can listen for Auth Code responses.

+8. Finally, create a file named *constants.js* that will store the strings constants for describing the application **events**:

+ :::code language="js" source="~/ms-identity-JavaScript-nodejs-desktop/App/constants.js":::

+│   ├── AuthProvider.js

+│   ├── CustomProtocolListener.js

+| Γö£ΓöÇΓöÇ main.js

+| Γö£ΓöÇΓöÇ preload.js

+| Γö£ΓöÇΓöÇ renderer.js

+│   ├── UIManager.js

+│   ├── authConfig.js

+In *App* folder, create a file named *AuthProvider.js*. The *AuthProvider.js* file will contain an authentication provider class that will handle login, logout, token acquisition, account selection and related authentication tasks using MSAL Node. Add the following code there:

+Finally, create an environment file to store the app registration details that will be used when acquiring tokens. To do so, create a file named *authConfig.js* inside the root folder of the sample (*ElectronDesktopApp*), and add the following code:

+- `Enter_the_Redirect_Uri_Here`: The Redirect Uri of the application you registered `msal{Your_Application/Client_Id}:///auth`.

+After consent, you'll view the messages returned in the response from the call to the Microsoft Graph API:

+When a user selects the **Sign In** button for the first time, get `getTokenInteractive` method of *AuthProvider.js* is called. This method redirects the user to sign-in with the Microsoft identity platform endpoint and validates the user's credentials, and then obtains an **authorization code**. This code is then exchanged for an access token using `acquireTokenByCode` public API of MSAL Node.

+The Microsoft Graph API requires the *user.read* scope to read a user's profile. By default, this scope is automatically added in every application that's registered in the Azure portal. Other APIs for Microsoft Graph, and custom APIs for your back-end server, might require extra scopes. For example, the Microsoft Graph API requires the *Mail.Read* scope in order to list the user's email.

+As you add scopes, your users might be prompted to provide another consent for the added scopes.

+## May 2022

+### Updated articles

+- [Developer guide to Conditional Access authentication context](developer-guide-conditional-access-authentication-context.md)

+- [Migrate confidential client applications from ADAL.NET to MSAL.NET](msal-net-migration-confidential-client.md)

+- [Protected web API: App registration](scenario-protected-web-api-app-registration.md)

+- [Quickstart: Sign in users and call the Microsoft Graph API from an Android app](mobile-app-quickstart-portal-android.md)

+- [Quickstart: Sign in users and call the Microsoft Graph API from an iOS or macOS app](mobile-app-quickstart-portal-ios.md)

+- [Set up your application's Azure AD test environment](test-setup-environment.md)

+- [Single Sign-On SAML protocol](single-sign-on-saml-protocol.md)

+- [Single sign-on with MSAL.js](msal-js-sso.md)

+- [Tutorial: Sign in users and acquire a token for Microsoft Graph in a Node.js & Express web app](tutorial-v2-nodejs-webapp-msal.md)

+- [What's new for authentication?](reference-breaking-changes.md)

+In Azure Active Directory (Azure AD), you can create attribute-based rules to enable dynamic membership for a group. Dynamic group membership adds and removes group members automatically using membership rules based on member attributes. This article details the properties and syntax to create dynamic membership rules for users or devices. You can set up a rule for dynamic membership on security groups or Microsoft 365 groups.

+- You can't create a device group based on the user attributes of the device owner. Device membership rules can reference only device attributes.

+A rule such as this one can be used to group all users for whom a Microsoft 365 or other Microsoft Online Service capability is enabled. You could then apply with a set of policies to the group.

+Extension attributes and custom extension properties are supported as string properties in dynamic membership rules. [Extension attributes](/graph/api/resources/onpremisesextensionattributes) are synced from on-premises Window Server Active Directory and take the format of "ExtensionAttributeX", where X equals 1 - 15. Here's an example of a rule that uses an extension attribute as a property:

+[Custom extension properties](../hybrid/how-to-connect-sync-feature-directory-extensions.md) are synced from on-premises Windows Server Active Directory or from a connected SaaS application and are of the format of `user.extension_[GUID]_[Attribute]`, where:

+ Title: Group membership for Azure AD dynamic groups with memberOf - Azure AD | Microsoft Docs

+description: How to create a dynamic membership group that can contain members of other groups in Azure Active Directory.

+documentationcenter: ''

+# Group membership in a dynamic group (preview) in Azure Active Directory

+This feature preview enables admins to create dynamic groups in Azure Active Directory (Azure AD) that populate by adding members of other groups using the memberOf attribute. Apps that couldn't read group-based membership previously in Azure AD can now read the entire membership of these new memberOf groups. Not only can these groups be used for apps, they can also be used for licensing assignment and role-based access control. The following diagram illustrates how you could create Dynamic-Group-A with members of Security-Group-X and Security-Group-Y. Members of the groups inside of Security-Group-X and Security-Group-Y don't become members of Dynamic-Group-A.

+

+With this preview, admins can configure dynamic groups with the memberOf attribute in the Azure portal, Microsoft Graph, and PowerShell. Security groups, Microsoft 365 groups, groups that are synced from on-premises Active Directory can all be added as members of these dynamic groups, and can all be added to a single group. For example, the dynamic group could be a security group, but you can use Microsoft 365 groups, security groups, and groups that are synced from on-premises to define its membership.

+## Prerequisites

+Only administrators in the Global Administrator, Intune Administrator, or User Administrator role can use the memberOf attribute to create an Azure AD dynamic group. You must have an Azure AD Premium license for the Azure AD tenant.

+## Preview limitations

+- Each Azure AD tenant is limited to 500 dynamic groups using the memberOf attribute. memberOf groups do count towards the total dynamic group member quota of 5,000.

+- Each dynamic group can have up to 50 member groups.

+- When adding members of security groups to memberOf dynamic groups, only direct members of the security group become members of the dynamic group.

+- You can't use one memberOf dynamic group to define the membership of another memberOf dynamic groups. For example, Dynamic Group A, with members of group B and C in it, can't be a member of Dynamic Group D).

+- MemberOf can't be used with other rules. For example, a rule that states dynamic group A should contain members of group B and also should contain only users located in Redmond will fail.

+- Dynamic group rule builder and validate feature can't be used for memberOf at this time.

+- MemberOf can't be used with other operators. For example, you can't create a rule that states ΓÇ£Members Of group A can't be in Dynamic group B.ΓÇ¥

+## Getting started

+This feature can be used in the Azure AD portal, Microsoft Graph, and in PowerShell. Because memberOf isn't yet supported in the rule builder, you must enter your rule in the rule editor.

+### Steps to create a memberOf dynamic group

+1. Sign in to the Azure portal with an account that has Global Administrator, Intune Administrator, or User Administrator role permissions.

+1. Select **Azure Active Directory** > **Groups**, and then select **New group**.

+1. Fill in group details. The group type can be Security or Microsoft 365, and the membership type can be set to **Dynamic User** or **Dynamic Device**.

+1. Select **Add dynamic query**.

+1. MemberOf isn't yet supported in the rule builder. Select **Edit** to write the rule in the **Rule syntax** box.

+ 1. Example user rule: `user.memberof -any (group.objectId -in ['groupId', 'groupId'])`

+ 1. Example device rule: `device.memberof -any (group.objectId -in ['groupId', 'groupId'])`

+1. Select **OK**.

+1. Select **Create group**.

+## Next steps

+To report an issue, contact us in the [Teams channel](https://teams.microsoft.com/l/channel/19%3a39Q7HFuexXXE3Vh90woJRNQQBbZl1YyesJHIEquuQCw1%40thread.tacv2/General?groupId=bfd3bfb8-e0db-4e9e-9008-5d7ba8c996b0&tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47).

+| <ul><li>EnableMIPLabels<li>Type: Boolean<li>Default: "False" |The flag indicating whether sensitivity labels published in Microsoft Purview compliance portal can be applied to Microsoft 365 groups. For more information, see [Assign Sensitivity Labels for Microsoft 365 groups](groups-assign-sensitivity-labels.md). |

+Members of your compliance team who will create sensitivity labels need permissions to the Microsoft 365 Defender portal, Microsoft Purview compliance portal, or Office 365 Security & Compliance Center.

+- **SAML** - Choose SAML whenever possible for existing applications that do not use OpenID Connect or OAuth. For more information, see [Single Sign-On SAML protocol](../develop/single-sign-on-saml-protocol.md).

+- Consider completing the single sign-on training in [Enable single sign-on for applications by using Azure Active Directory](/learn/modules/enable-single-sign-on).

+When you add a new application from the gallery and configure a SAML-based sign-on (by selecting **Single sign-on** > **SAML** from the application overview page), Azure AD generates a self-signed certificate for the application that is valid for three years. To download the active certificate as a security certificate (**.cer**) file, return to that page (**SAML-based sign-on**) and select a download link in the **SAML Signing Certificate** heading. You can choose between the raw (binary) certificate or the Base64 (base 64-encoded text) certificate. For gallery applications, this section might also show a link to download the certificate as federation metadata XML (an **.xml** file), depending on the requirement of the application.

+Consider implementing SSO in your application. You can manually configure most applications for SSO. The most popular options in Azure AD are [SAML-based SSO and OpenID Connect-based SSO](../develop/active-directory-v2-protocols.md). Before you start, make sure that you understand the requirements for SSO and how to [plan for deployment](plan-sso-deployment.md). For training related to configuring SAML-based SSO for an enterprise application in your Azure AD tenant, see [Enable single sign-on for an application by using Azure Active Directory](/learn/modules/enable-single-sign-on).

+If included in your subscription, [assign groups to an application](assign-user-or-group-access-portal.md) so that you can delegate ongoing access management to the group owner.

+- [Plan for single sign-on deployment](plan-sso-deployment.md)

+### What happens to tokens after a managed identity is deleted?

+When a managed identity is deleted, an Azure resource that was previously associated with that identity can no longer request new tokens for that identity. Tokens that were issued before the identity was deleted will still be valid until their original expiry. Some target endpoints' authorization systems may carry out additional checks in the directory for the identity, in which case the request will fail as the object can't be found. However some systems, like Azure RBAC, will continue to accept requests from that token until it expires.

+Users with this role have permissions to manage compliance-related features in the Microsoft Purview compliance portal, Microsoft 365 admin center, Azure, and Office 365 Security & Compliance Center. Assignees can also manage all features within the Exchange admin center and Teams & Skype for Business admin centers and create support tickets for Azure and Microsoft 365. More information is available at [About Microsoft 365 admin roles](https://support.office.com/article/About-Office-365-admin-roles-da585eea-f576-4f55-a1e0-87090b6aaa9d).

+[Microsoft Purview compliance portal](https://protection.office.com) | Protect and manage your organization's data across Microsoft 365 services<br>Manage compliance alerts

+Users with this role have permissions to track data in the Microsoft Purview compliance portal, Microsoft 365 admin center, and Azure. Users can also track compliance data within the Exchange admin center, Compliance Manager, and Teams & Skype for Business admin center and create support tickets for Azure and Microsoft 365. [This documentation](/microsoft-365/security/office-365-security/permissions-in-the-security-and-compliance-center#permissions-needed-to-use-features-in-the-security--compliance-center) has details on differences between Compliance Administrator and Compliance Data Administrator.

+[Microsoft Purview compliance portal](https://protection.office.com) | Monitor compliance-related policies across Microsoft 365 services<br>Manage compliance alerts

+Users with this role have access to all administrative features in Azure Active Directory, as well as services that use Azure Active Directory identities like the Microsoft 365 Defender portal, the Microsoft Purview compliance portal, Exchange Online, SharePoint Online, and Skype for Business Online. Furthermore, Global Administrators can [elevate their access](../../role-based-access-control/elevate-access-global-admin.md) to manage all Azure subscriptions and management groups. This allows Global Administrators to get full access to all Azure resources using the respective Azure AD Tenant. The person who signs up for the Azure AD organization becomes a Global Administrator. There can be more than one Global Administrator at your company. Global Administrators can reset the password for any user and all other administrators.

+ Title: 'Tutorial: Azure AD SSO integration with Capriza Platform'

+# Tutorial: Azure AD SSO integration with Capriza Platform

+In this tutorial, you'll learn how to integrate Capriza Platform with Azure Active Directory (Azure AD). When you integrate Capriza Platform with Azure AD, you can:

+* Control in Azure AD who has access to Capriza Platform.

+* Enable your users to be automatically signed-in to Capriza Platform with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+To get started, you need the following items:

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* Capriza Platform single sign-on (SSO) enabled subscription.

+* Along with Cloud Application Administrator, Application Administrator can also add or manage applications in Azure AD.

+For more information, see [Azure built-in roles](../roles/permissions-reference.md).

+* Capriza Platform supports **SP** initiated SSO.

+* Capriza Platform supports **Just In Time** user provisioning.

+> [!NOTE]

+> Identifier of this application is a fixed string value so only one instance can be configured in one tenant.

+## Add Capriza Platform from the gallery

+To configure the integration of Capriza Platform into Azure AD, you need to add Capriza Platform from the gallery to your list of managed SaaS apps.

+1. Sign in to the Azure portal using either a work or school account, or a personal Microsoft account.

+1. On the left navigation pane, select the **Azure Active Directory** service.

+1. Navigate to **Enterprise Applications** and then select **All Applications**.

+1. To add new application, select **New application**.

+1. In the **Add from the gallery** section, type **Capriza Platform** in the search box.

+1. Select **Capriza Platform** from results panel and then add the app. Wait a few seconds while the app is added to your tenant.

+## Configure and test Azure AD SSO for Capriza Platform

+Configure and test Azure AD SSO with Capriza Platform using a test user called **B.Simon**. For SSO to work, you need to establish a link relationship between an Azure AD user and the related user in Capriza Platform.

+To configure and test Azure AD SSO with Capriza Platform, perform the following steps:

+1. **[Configure Azure AD SSO](#configure-azure-ad-sso)** - to enable your users to use this feature.

+ 1. **[Create an Azure AD test user](#create-an-azure-ad-test-user)** - to test Azure AD single sign-on with B.Simon.

+ 1. **[Assign the Azure AD test user](#assign-the-azure-ad-test-user)** - to enable B.Simon to use Azure AD single sign-on.

+1. **[Configure Capriza Platform SSO](#configure-capriza-platform-sso)** - to configure the single sign-on settings on application side.

+ 1. **[Create Capriza Platform test user](#create-capriza-platform-test-user)** - to have a counterpart of B.Simon in Capriza Platform that is linked to the Azure AD representation of user.

+1. **[Test SSO](#test-sso)** - to verify whether the configuration works.

+## Configure Azure AD SSO

+Follow these steps to enable Azure AD SSO in the Azure portal.

+1. In the Azure portal, on the **Capriza Platform** application integration page, find the **Manage** section and select **single sign-on**.

+2. On the **Select a single sign-on method** page, select **SAML**.

+3. On the **Set up single sign-on with SAML** page, click the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+4. On the **Basic SAML Configuration** section, perform the following step:

+ 

+ 

+In this section, you'll create a test user in the Azure portal called B.Simon.

+1. From the left pane in the Azure portal, select **Azure Active Directory**, select **Users**, and then select **All users**.

+1. Select **New user** at the top of the screen.

+1. In the **User** properties, follow these steps:

+ 1. In the **Name** field, enter `B.Simon`.

+ 1. In the **User name** field, enter the username@companydomain.extension. For example, `B.Simon@contoso.com`.

+ 1. Select the **Show password** check box, and then write down the value that's displayed in the **Password** box.

+ 1. Click **Create**.

+In this section, you'll enable B.Simon to use Azure single sign-on by granting access to Capriza Platform.

+1. In the Azure portal, select **Enterprise Applications**, and then select **All applications**.

+1. In the applications list, select **Capriza Platform**.

+1. In the app's overview page, find the **Manage** section and select **Users and groups**.

+1. Select **Add user**, then select **Users and groups** in the **Add Assignment** dialog.

+1. In the **Users and groups** dialog, select **B.Simon** from the Users list, then click the **Select** button at the bottom of the screen.

+1. If you are expecting a role to be assigned to the users, you can select it from the **Select a role** dropdown. If no role has been set up for this app, you see "Default Access" role selected.

+1. In the **Add Assignment** dialog, click the **Assign** button.

+## Configure Capriza Platform SSO

+To configure single sign-on on **Capriza Platform** side, you need to send the downloaded **Certificate (Base64)** and appropriate copied URLs from Azure portal to [Capriza Platform support team](mailto:support@capriza.com). They set this setting to have the SAML SSO connection set properly on both sides.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+* Click on **Test this application** in Azure portal. This will redirect to Capriza Platform Sign-on URL where you can initiate the login flow.

+* Go to Capriza Platform Sign-on URL directly and initiate the login flow from there.

+* You can use Microsoft My Apps. When you click the Capriza Platform tile in the My Apps, this will redirect to Capriza Platform Sign-on URL. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Next steps

+Once you configure Capriza Platform you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+ Title: 'Tutorial: Azure AD SSO integration with CompetencyIQ'

+# Tutorial: Azure AD SSO integration with CompetencyIQ

+In this tutorial, you'll learn how to integrate CompetencyIQ with Azure Active Directory (Azure AD). When you integrate CompetencyIQ with Azure AD, you can:

+* Control in Azure AD who has access to CompetencyIQ.

+* Enable your users to be automatically signed-in to CompetencyIQ with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+To get started, you need the following items:

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* CompetencyIQ single sign-on (SSO) enabled subscription.

+* Along with Cloud Application Administrator, Application Administrator can also add or manage applications in Azure AD.

+For more information, see [Azure built-in roles](../roles/permissions-reference.md).

+* CompetencyIQ supports **SP** initiated SSO.

+> [!NOTE]

+> Identifier of this application is a fixed string value so only one instance can be configured in one tenant.

+## Add CompetencyIQ from the gallery

+To configure the integration of CompetencyIQ into Azure AD, you need to add CompetencyIQ from the gallery to your list of managed SaaS apps.

+1. Sign in to the Azure portal using either a work or school account, or a personal Microsoft account.

+1. On the left navigation pane, select the **Azure Active Directory** service.

+1. Navigate to **Enterprise Applications** and then select **All Applications**.

+1. To add new application, select **New application**.

+1. In the **Add from the gallery** section, type **CompetencyIQ** in the search box.

+1. Select **CompetencyIQ** from results panel and then add the app. Wait a few seconds while the app is added to your tenant.

+## Configure and test Azure AD SSO for CompetencyIQ

+Configure and test Azure AD SSO with CompetencyIQ using a test user called **B.Simon**. For SSO to work, you need to establish a link relationship between an Azure AD user and the related user in CompetencyIQ.

+To configure and test Azure AD SSO with CompetencyIQ, perform the following steps:

+1. **[Configure Azure AD SSO](#configure-azure-ad-sso)** - to enable your users to use this feature.

+ 1. **[Create an Azure AD test user](#create-an-azure-ad-test-user)** - to test Azure AD single sign-on with B.Simon.

+ 1. **[Assign the Azure AD test user](#assign-the-azure-ad-test-user)** - to enable B.Simon to use Azure AD single sign-on.

+1. **[Configure CompetencyIQ SSO](#configure-competencyiq-sso)** - to configure the single sign-on settings on application side.

+ 1. **[Create CompetencyIQ test user](#create-competencyiq-test-user)** - to have a counterpart of B.Simon in CompetencyIQ that is linked to the Azure AD representation of user.

+1. **[Test SSO](#test-sso)** - to verify whether the configuration works.

+## Configure Azure AD SSO

+Follow these steps to enable Azure AD SSO in the Azure portal.

+1. In the Azure portal, on the **CompetencyIQ** application integration page, find the **Manage** section and select **single sign-on**.

+2. On the **Select a single sign-on method** page, select **SAML**.

+3. On the **Set up single sign-on with SAML** page, click the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+ a. In the **Identifier (Entity ID)** text box, type the URL:

+ `https://www.competencyiq.com/`

+ b. In the **Sign on URL** text box, type a URL using the following pattern:

+ 

+ 

+In this section, you'll create a test user in the Azure portal called B.Simon.

+1. From the left pane in the Azure portal, select **Azure Active Directory**, select **Users**, and then select **All users**.

+1. Select **New user** at the top of the screen.

+1. In the **User** properties, follow these steps:

+ 1. In the **Name** field, enter `B.Simon`.

+ 1. In the **User name** field, enter the username@companydomain.extension. For example, `B.Simon@contoso.com`.

+ 1. Select the **Show password** check box, and then write down the value that's displayed in the **Password** box.

+ 1. Click **Create**.

+In this section, you'll enable B.Simon to use Azure single sign-on by granting access to CompetencyIQ.

+1. In the Azure portal, select **Enterprise Applications**, and then select **All applications**.

+1. In the applications list, select **CompetencyIQ**.

+1. In the app's overview page, find the **Manage** section and select **Users and groups**.

+1. Select **Add user**, then select **Users and groups** in the **Add Assignment** dialog.

+1. In the **Users and groups** dialog, select **B.Simon** from the Users list, then click the **Select** button at the bottom of the screen.

+1. If you are expecting a role to be assigned to the users, you can select it from the **Select a role** dropdown. If no role has been set up for this app, you see "Default Access" role selected.

+1. In the **Add Assignment** dialog, click the **Assign** button.

+## Configure CompetencyIQ SSO

+To configure single sign-on on **CompetencyIQ** side, you need to send the downloaded **Federation Metadata XML** and appropriate copied URLs from Azure portal to [CompetencyIQ support team](https://www.competencyiq.com/). They set this setting to have the SAML SSO connection set properly on both sides.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+* Click on **Test this application** in Azure portal. This will redirect to CompetencyIQ Sign-on URL where you can initiate the login flow.

+* Go to CompetencyIQ Sign-on URL directly and initiate the login flow from there.

+* You can use Microsoft My Apps. When you click the CompetencyIQ tile in the My Apps, this will redirect to CompetencyIQ Sign-on URL. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Next steps

+Once you configure CompetencyIQ you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+ Title: 'Tutorial: Azure AD SSO integration with Edcor'

+# Tutorial: Azure AD SSO integration with Edcor

+In this tutorial, you'll learn how to integrate Edcor with Azure Active Directory (Azure AD). When you integrate Edcor with Azure AD, you can:

+* Control in Azure AD who has access to Edcor.

+* Enable your users to be automatically signed-in to Edcor with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+To get started, you need the following items:

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* Edcor single sign-on (SSO) enabled subscription.

+* Along with Cloud Application Administrator, Application Administrator can also add or manage applications in Azure AD.

+For more information, see [Azure built-in roles](../roles/permissions-reference.md).

+* Edcor supports **IDP** initiated SSO.

+> [!NOTE]

+> Identifier of this application is a fixed string value so only one instance can be configured in one tenant.

+## Add Edcor from the gallery

+1. Sign in to the Azure portal using either a work or school account, or a personal Microsoft account.

+1. On the left navigation pane, select the **Azure Active Directory** service.

+1. Navigate to **Enterprise Applications** and then select **All Applications**.

+1. To add new application, select **New application**.

+1. In the **Add from the gallery** section, type **Edcor** in the search box.

+1. Select **Edcor** from results panel and then add the app. Wait a few seconds while the app is added to your tenant.

+## Configure and test Azure AD SSO for Edcor

+Configure and test Azure AD SSO with Edcor using a test user called **B.Simon**. For SSO to work, you need to establish a link relationship between an Azure AD user and the related user in Edcor.

+To configure and test Azure AD SSO with Edcor, perform the following steps:

+1. **[Configure Azure AD SSO](#configure-azure-ad-sso)** - to enable your users to use this feature.

+ 1. **[Create an Azure AD test user](#create-an-azure-ad-test-user)** - to test Azure AD single sign-on with B.Simon.

+ 1. **[Assign the Azure AD test user](#assign-the-azure-ad-test-user)** - to enable B.Simon to use Azure AD single sign-on.

+1. **[Configure Edcor SSO](#configure-edcor-sso)** - to configure the single sign-on settings on application side.

+ 1. **[Create Edcor test user](#create-edcor-test-user)** - to have a counterpart of B.Simon in Edcor that is linked to the Azure AD representation of user.

+1. **[Test SSO](#test-sso)** - to verify whether the configuration works.

+## Configure Azure AD SSO

+Follow these steps to enable Azure AD SSO in the Azure portal.

+1. In the Azure portal, on the **Edcor** application integration page, find the **Manage** section and select **single sign-on**.

+2. On the **Select a single sign-on method** page, select **SAML**.

+3. On the **Set up single sign-on with SAML** page, click the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+4. On the **Basic SAML Configuration** section, perform the following step:

+ In the **Identifier** text box, type the URL:

+ 

+ 

+In this section, you'll create a test user in the Azure portal called B.Simon.

+1. From the left pane in the Azure portal, select **Azure Active Directory**, select **Users**, and then select **All users**.

+1. Select **New user** at the top of the screen.

+1. In the **User** properties, follow these steps:

+ 1. In the **Name** field, enter `B.Simon`.

+ 1. In the **User name** field, enter the username@companydomain.extension. For example, `B.Simon@contoso.com`.

+ 1. Select the **Show password** check box, and then write down the value that's displayed in the **Password** box.

+ 1. Click **Create**.

+In this section, you'll enable B.Simon to use Azure single sign-on by granting access to Edcor.

+1. In the Azure portal, select **Enterprise Applications**, and then select **All applications**.

+1. In the applications list, select **Edcor**.

+1. In the app's overview page, find the **Manage** section and select **Users and groups**.

+1. Select **Add user**, then select **Users and groups** in the **Add Assignment** dialog.

+1. In the **Users and groups** dialog, select **B.Simon** from the Users list, then click the **Select** button at the bottom of the screen.

+1. If you are expecting a role to be assigned to the users, you can select it from the **Select a role** dropdown. If no role has been set up for this app, you see "Default Access" role selected.

+1. In the **Add Assignment** dialog, click the **Assign** button.

+## Configure Edcor SSO

+To configure single sign-on on **Edcor** side, you need to send the downloaded **Federation Metadata XML** and appropriate copied URLs from Azure portal to [Edcor support team](https://www.edcor.com/contact-us/). They set this setting to have the SAML SSO connection set properly on both sides.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+* Click on Test this application in Azure portal and you should be automatically signed in to the Edcor for which you set up the SSO.

+* You can use Microsoft My Apps. When you click the Edcor tile in the My Apps, you should be automatically signed in to the Edcor for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Next steps

+Once you configure Edcor you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+ Title: 'Tutorial: Azure AD SSO integration with eLuminate'

+# Tutorial: Azure AD SSO integration with eLuminate

+In this tutorial, you'll learn how to integrate eLuminate with Azure Active Directory (Azure AD). When you integrate eLuminate with Azure AD, you can:

+* Control in Azure AD who has access to eLuminate.

+* Enable your users to be automatically signed-in to eLuminate with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+To get started, you need the following items:

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* eLuminate single sign-on (SSO) enabled subscription.

+* Along with Cloud Application Administrator, Application Administrator can also add or manage applications in Azure AD.

+For more information, see [Azure built-in roles](../roles/permissions-reference.md).

+* eLuminate supports **SP** initiated SSO.

+> [!NOTE]

+> Identifier of this application is a fixed string value so only one instance can be configured in one tenant.

+## Add eLuminate from the gallery

+To configure the integration of eLuminate into Azure AD, you need to add eLuminate from the gallery to your list of managed SaaS apps.

+1. Sign in to the Azure portal using either a work or school account, or a personal Microsoft account.

+1. On the left navigation pane, select the **Azure Active Directory** service.

+1. Navigate to **Enterprise Applications** and then select **All Applications**.

+1. To add new application, select **New application**.

+1. In the **Add from the gallery** section, type **eLuminate** in the search box.

+1. Select **eLuminate** from results panel and then add the app. Wait a few seconds while the app is added to your tenant.

+## Configure and test Azure AD SSO for eLuminate

+Configure and test Azure AD SSO with eLuminate using a test user called **B.Simon**. For SSO to work, you need to establish a link relationship between an Azure AD user and the related user in eLuminate.

+To configure and test Azure AD SSO with eLuminate, perform the following steps:

+1. **[Configure Azure AD SSO](#configure-azure-ad-sso)** - to enable your users to use this feature.

+ 1. **[Create an Azure AD test user](#create-an-azure-ad-test-user)** - to test Azure AD single sign-on with B.Simon.

+ 1. **[Assign the Azure AD test user](#assign-the-azure-ad-test-user)** - to enable B.Simon to use Azure AD single sign-on.

+1. **[Configure eLuminate SSO](#configure-eluminate-sso)** - to configure the single sign-on settings on application side.

+ 1. **[Create eLuminate test user](#create-eluminate-test-user)** - to have a counterpart of B.Simon in eLuminate that is linked to the Azure AD representation of user.

+1. **[Test SSO](#test-sso)** - to verify whether the configuration works.

+## Configure Azure AD SSO

+Follow these steps to enable Azure AD SSO in the Azure portal.

+1. In the Azure portal, on the **eLuminate** application integration page, find the **Manage** section and select **single sign-on**.

+2. On the **Select a single sign-on method** page, select **SAML**.

+3. On the **Set up single sign-on with SAML** page, click the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+ a. In the **Identifier (Entity ID)** text box, type the value:

+ `Eluminate/ClientShortName`

+ b. In the **Sign on URL** text box, type the URL:

+ > These values are not real. Update these values with the actual Identifier and Sign on URL. Contact [eLuminate Client support team](mailto:support@intellimedia.ca) to get these values. You can also refer to the patterns shown in the **Basic SAML Configuration** section in the Azure portal.

+5. On the **Set up Single Sign-On with SAML** page, In the **SAML Signing Certificate** section, click copy button to copy **App Federation Metadata Url** and save it on your computer.

+ 

+In this section, you'll create a test user in the Azure portal called B.Simon.

+1. From the left pane in the Azure portal, select **Azure Active Directory**, select **Users**, and then select **All users**.

+1. Select **New user** at the top of the screen.

+1. In the **User** properties, follow these steps:

+ 1. In the **Name** field, enter `B.Simon`.

+ 1. In the **User name** field, enter the username@companydomain.extension. For example, `B.Simon@contoso.com`.

+ 1. Select the **Show password** check box, and then write down the value that's displayed in the **Password** box.

+ 1. Click **Create**.

+In this section, you'll enable B.Simon to use Azure single sign-on by granting access to eLuminate.

+1. In the Azure portal, select **Enterprise Applications**, and then select **All applications**.

+1. In the applications list, select **eLuminate**.

+1. In the app's overview page, find the **Manage** section and select **Users and groups**.

+1. Select **Add user**, then select **Users and groups** in the **Add Assignment** dialog.

+1. In the **Users and groups** dialog, select **B.Simon** from the Users list, then click the **Select** button at the bottom of the screen.

+1. If you are expecting a role to be assigned to the users, you can select it from the **Select a role** dropdown. If no role has been set up for this app, you see "Default Access" role selected.

+1. In the **Add Assignment** dialog, click the **Assign** button.

+## Configure eLuminate SSO

+To configure single sign-on on **eLuminate** side, you need to send the **App Federation Metadata Url** to [eLuminate support team](mailto:support@intellimedia.ca). They set this setting to have the SAML SSO connection set properly on both sides.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+* Click on **Test this application** in Azure portal. This will redirect to eLuminate Sign-on URL where you can initiate the login flow.

+* Go to eLuminate Sign-on URL directly and initiate the login flow from there.

+* You can use Microsoft My Apps. When you click the eLuminate tile in the My Apps, this will redirect to eLuminate Sign-on URL. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Next steps

+Once you configure eLuminate you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+ > These values are not real. Update these values with the actual Identifier,Reply URL and Sign-On URL. Contact [EthicsPoint Incident Management (EPIM) Client support team](https://www.navex.com/en-us/products/navex-ethics-compliance/ethicspoint-hotline-incident-management/) to get these values. You can also refer to the patterns shown in the **Basic SAML Configuration** section in the Azure portal.

+To configure single sign-on on **EthicsPoint Incident Management (EPIM)** side, you need to send the downloaded **Federation Metadata XML** and appropriate copied URLs from Azure portal to [EthicsPoint Incident Management (EPIM) support team](https://www.navex.com/en-us/products/navex-ethics-compliance/ethicspoint-hotline-incident-management/). They set this setting to have the SAML SSO connection set properly on both sides.

+In this section, you create a user called Britta Simon in EthicsPoint Incident Management (EPIM). Work with [EthicsPoint Incident Management (EPIM) support team](https://www.navex.com/en-us/products/navex-ethics-compliance/ethicspoint-hotline-incident-management/) to add the users in the EthicsPoint Incident Management (EPIM) platform. Users must be created and activated before you use single sign-on.

+1. OpenLearning Identity Authentication application expects the SAML assertions in a specific format, which requires you to add custom attribute mappings to your SAML token attributes configuration. The following screenshot shows the list of default attributes.

+ 

+1. In addition to above, OpenLearning Identity Authentication application expects few more attributes to be passed back in SAML response which are shown below. These attributes are also pre populated but you can review them as per your requirements.

+ | Name | Source Attribute|

+ | | |

+ | urn:oid:0.9.2342.19200300.100.1.3 | user.mail |

+ | urn:oid:2.16.840.1.113730.3.1.241 | user.displayname |

+ | urn:oid:1.3.6.1.4.1.5923.1.1.1.9 | user.extensionattribute1 |

+ | urn:oid:1.3.6.1.4.1.5923.1.1.1.6 | user.objectid |

+ | urn:oid:2.5.4.10 | user.companyname |

+In this section, a user called Britta Simon is created in OpenLearning. OpenLearning supports just-in-time user provisioning, which is enabled by default. There is no action item for you in this section. If a user doesn't already exist in OpenLearning, a new one is created after authentication.

+ Title: 'Tutorial: Azure AD SSO integration with Seculio'

+description: Learn how to configure single sign-on between Azure Active Directory and Seculio.

+# Tutorial: Azure AD SSO integration with Seculio

+In this tutorial, you'll learn how to integrate Seculio with Azure Active Directory (Azure AD). When you integrate Seculio with Azure AD, you can:

+* Control in Azure AD who has access to Seculio.

+* Enable your users to be automatically signed-in to Seculio with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+## Prerequisites

+To get started, you need the following items:

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* Seculio single sign-on (SSO) enabled subscription.

+* Along with Cloud Application Administrator, Application Administrator can also add or manage applications in Azure AD.

+For more information, see [Azure built-in roles](../roles/permissions-reference.md).

+## Scenario description

+In this tutorial, you configure and test Azure AD SSO in a test environment.

+* Seculio supports **SP** and **IDP** initiated SSO.

+## Add Seculio from the gallery

+To configure the integration of Seculio into Azure AD, you need to add Seculio from the gallery to your list of managed SaaS apps.

+1. Sign in to the Azure portal using either a work or school account, or a personal Microsoft account.

+1. On the left navigation pane, select the **Azure Active Directory** service.

+1. Navigate to **Enterprise Applications** and then select **All Applications**.

+1. To add new application, select **New application**.

+1. In the **Add from the gallery** section, type **Seculio** in the search box.

+1. Select **Seculio** from results panel and then add the app. Wait a few seconds while the app is added to your tenant.

+## Configure and test Azure AD SSO for Seculio

+Configure and test Azure AD SSO with Seculio using a test user called **B.Simon**. For SSO to work, you need to establish a link relationship between an Azure AD user and the related user in Seculio.

+To configure and test Azure AD SSO with Seculio, perform the following steps:

+1. **[Configure Azure AD SSO](#configure-azure-ad-sso)** - to enable your users to use this feature.

+ 1. **[Create an Azure AD test user](#create-an-azure-ad-test-user)** - to test Azure AD single sign-on with B.Simon.

+ 1. **[Assign the Azure AD test user](#assign-the-azure-ad-test-user)** - to enable B.Simon to use Azure AD single sign-on.

+1. **[Configure Seculio SSO](#configure-seculio-sso)** - to configure the single sign-on settings on application side.

+ 1. **[Create Seculio test user](#create-seculio-test-user)** - to have a counterpart of B.Simon in Seculio that is linked to the Azure AD representation of user.

+1. **[Test SSO](#test-sso)** - to verify whether the configuration works.

+## Configure Azure AD SSO

+Follow these steps to enable Azure AD SSO in the Azure portal.

+1. In the Azure portal, on the **Seculio** application integration page, find the **Manage** section and select **single sign-on**.

+1. On the **Select a single sign-on method** page, select **SAML**.

+1. On the **Set up single sign-on with SAML** page, click the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+1. On the **Basic SAML Configuration** section, perform the following steps:

+ a. In the **Identifier** textbox, type a URL using the following pattern:

+ `https://seculio.com/saml/<ID>`

+ b. In the **Reply URL** textbox, type a URL using the following pattern:

+ `https://seculio.com/saml/acs/<ID>`

+1. Click **Set additional URLs** and perform the following step if you wish to configure the application in **SP** initiated mode:

+ In the **Sign-on URL** text box, type the URL:

+ `https://seculio.com/`

+ > [!Note]

+ > These values are not real. Update these values with the actual Identifier and Reply URL. Contact [Seculio support team](mailto:seculio@lrm.jp) to get these values. You can also refer to the patterns shown in the **Basic SAML Configuration** section in the Azure portal.

+1. On the **Set-up single sign-on with SAML** page, in the **SAML Signing Certificate** section, find **Federation Metadata XML** and select **Download** to download the certificate and save it on your computer.

+ 

+1. On the **Set up Seculio** section, copy the appropriate URL(s) based on your requirement.

+ 

+### Create an Azure AD test user

+In this section, you'll create a test user in the Azure portal called B.Simon.

+1. From the left pane in the Azure portal, select **Azure Active Directory**, select **Users**, and then select **All users**.

+1. Select **New user** at the top of the screen.

+1. In the **User** properties, follow these steps:

+ 1. In the **Name** field, enter `B.Simon`.

+ 1. In the **User name** field, enter the username@companydomain.extension. For example, `B.Simon@contoso.com`.

+ 1. Select the **Show password** check box, and then write down the value that's displayed in the **Password** box.

+ 1. Click **Create**.

+### Assign the Azure AD test user

+In this section, you'll enable B.Simon to use Azure single sign-on by granting access to Seculio.

+1. In the Azure portal, select **Enterprise Applications**, and then select **All applications**.

+1. In the applications list, select **Seculio**.

+1. In the app's overview page, find the **Manage** section and select **Users and groups**.

+1. Select **Add user**, then select **Users and groups** in the **Add Assignment** dialog.

+1. In the **Users and groups** dialog, select **B.Simon** from the Users list, then click the **Select** button at the bottom of the screen.

+1. If you are expecting a role to be assigned to the users, you can select it from the **Select a role** dropdown. If no role has been set up for this app, you see "Default Access" role selected.

+1. In the **Add Assignment** dialog, click the **Assign** button.

+## Configure Seculio SSO

+To configure single sign-on on **Seculio** side, you need to send the downloaded **Federation Metadata XML** and appropriate copied URLs from Azure portal to [Seculio support team](mailto:seculio@lrm.jp). They set this setting to have the SAML SSO connection set properly on both sides.

+### Create Seculio test user

+In this section, you create a user called Britta Simon in Seculio. Work with [Seculio support team](mailto:seculio@lrm.jp) to add the users in the Seculio platform. Users must be created and activated before you use single sign-on.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+#### SP initiated:

+* Click on **Test this application** in Azure portal. This will redirect to Seculio Sign on URL where you can initiate the login flow.

+* Go to Seculio Sign-on URL directly and initiate the login flow from there.

+#### IDP initiated:

+* Click on **Test this application** in Azure portal and you should be automatically signed in to the Seculio for which you set up the SSO.

+You can also use Microsoft My Apps to test the application in any mode. When you click the Seculio tile in the My Apps, if configured in SP mode you would be redirected to the application sign on page for initiating the login flow and if configured in IDP mode, you should be automatically signed in to the Seculio for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Next steps

+Once you configure Seculio you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+1. Your TVU Service application expects the SAML assertions in a specific format, which requires you to add custom attribute mappings to your SAML token attributes configuration. The following screenshot shows an example for this. The default value of **Unique User Identifier** is **user.userprincipalname** but TVU Service expects this to be mapped with the user's email address. For that you can use **user.mail** attribute from the list or use the appropriate attribute value based on your organization configuration.

+ Title: 'Tutorial: Azure AD SSO integration with xCarrier®'

+description: Learn how to configure single sign-on between Azure Active Directory and xCarrier®.

+# Tutorial: Azure AD SSO integration with xCarrier®

+In this tutorial, you'll learn how to integrate xCarrier® with Azure Active Directory (Azure AD). When you integrate xCarrier® with Azure AD, you can:

+* Control in Azure AD who has access to xCarrier®.

+* Enable your users to be automatically signed-in to xCarrier® with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+## Prerequisites

+To get started, you need the following items:

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* xCarrier® single sign-on (SSO) enabled subscription.

+* Along with Cloud Application Administrator, Application Administrator can also add or manage applications in Azure AD.

+For more information, see [Azure built-in roles](../roles/permissions-reference.md).

+## Scenario description

+In this tutorial, you configure and test Azure AD SSO in a test environment.

+* xCarrier® supports **SP** and **IDP** initiated SSO.

+> [!NOTE]

+> Identifier of this application is a fixed string value so only one instance can be configured in one tenant.

+## Add xCarrier® from the gallery

+To configure the integration of xCarrier® into Azure AD, you need to add xCarrier® from the gallery to your list of managed SaaS apps.

+1. Sign in to the Azure portal using either a work or school account, or a personal Microsoft account.

+1. On the left navigation pane, select the **Azure Active Directory** service.

+1. Navigate to **Enterprise Applications** and then select **All Applications**.

+1. To add new application, select **New application**.

+1. In the **Add from the gallery** section, type **xCarrier®** in the search box.

+1. Select **xCarrier®** from results panel and then add the app. Wait a few seconds while the app is added to your tenant.

+## Configure and test Azure AD SSO for xCarrier®

+Configure and test Azure AD SSO with xCarrier® using a test user called **B.Simon**. For SSO to work, you need to establish a link relationship between an Azure AD user and the related user in xCarrier®.

+To configure and test Azure AD SSO with xCarrier®, perform the following steps:

+1. **[Configure Azure AD SSO](#configure-azure-ad-sso)** - to enable your users to use this feature.

+ 1. **[Create an Azure AD test user](#create-an-azure-ad-test-user)** - to test Azure AD single sign-on with B.Simon.

+ 1. **[Assign the Azure AD test user](#assign-the-azure-ad-test-user)** - to enable B.Simon to use Azure AD single sign-on.

+1. **[Configure xCarrier® SSO](#configure-xcarrier-sso)** - to configure the single sign-on settings on application side.

+ 1. **[Create xCarrier® test user](#create-xcarrier-test-user)** - to have a counterpart of B.Simon in xCarrier® that is linked to the Azure AD representation of user.

+1. **[Test SSO](#test-sso)** - to verify whether the configuration works.

+## Configure Azure AD SSO

+Follow these steps to enable Azure AD SSO in the Azure portal.

+1. In the Azure portal, on the **xCarrier®** application integration page, find the **Manage** section and select **single sign-on**.

+1. On the **Select a single sign-on method** page, select **SAML**.

+1. On the **Set up single sign-on with SAML** page, click the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+1. On the **Basic SAML Configuration** section, the user does not have to perform any step as the app is already pre-integrated with Azure.

+1. Click **Set additional URLs** and perform the following step if you wish to configure the application in **SP** initiated mode:

+ In the **Sign-on URL** text box, type the URL:

+ `https://msdev.myxcarrier.com/Home/Index`

+1. On the **Set-up single sign-on with SAML** page, in the **SAML Signing Certificate** section, find **Federation Metadata XML** and select **Download** to download the certificate and save it on your computer.

+ 

+1. On the **Set up xCarrier®** section, copy the appropriate URL(s) based on your requirement.

+ 

+### Create an Azure AD test user

+In this section, you'll create a test user in the Azure portal called B.Simon.

+1. From the left pane in the Azure portal, select **Azure Active Directory**, select **Users**, and then select **All users**.

+1. Select **New user** at the top of the screen.

+1. In the **User** properties, follow these steps:

+ 1. In the **Name** field, enter `B.Simon`.

+ 1. In the **User name** field, enter the username@companydomain.extension. For example, `B.Simon@contoso.com`.

+ 1. Select the **Show password** check box, and then write down the value that's displayed in the **Password** box.

+ 1. Click **Create**.

+### Assign the Azure AD test user

+In this section, you'll enable B.Simon to use Azure single sign-on by granting access to xCarrier®.

+1. In the Azure portal, select **Enterprise Applications**, and then select **All applications**.

+1. In the applications list, select **xCarrier®**.

+1. In the app's overview page, find the **Manage** section and select **Users and groups**.

+1. Select **Add user**, then select **Users and groups** in the **Add Assignment** dialog.

+1. In the **Users and groups** dialog, select **B.Simon** from the Users list, then click the **Select** button at the bottom of the screen.

+1. If you are expecting a role to be assigned to the users, you can select it from the **Select a role** dropdown. If no role has been set up for this app, you see "Default Access" role selected.

+1. In the **Add Assignment** dialog, click the **Assign** button.

+## Configure xCarrier® SSO

+To configure single sign-on on **xCarrier®** side, you need to send the downloaded **Federation Metadata XML** and appropriate copied URLs from Azure portal to [xCarrier® support team](mailto:pw_support@elemica.com). They set this setting to have the SAML SSO connection set properly on both sides.

+### Create xCarrier® test user

+In this section, a user called B.Simon is created in xCarrier®. xCarrier® supports just-in-time user provisioning, which is enabled by default. There's no action item for you in this section. If a user doesn't already exist in xCarrier®, a new one is created after authentication.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+#### SP initiated:

+* Click on **Test this application** in Azure portal. This will redirect to xCarrier® Sign on URL where you can initiate the login flow.

+* Go to xCarrier® Sign-on URL directly and initiate the login flow from there.

+#### IDP initiated:

+* Click on **Test this application** in Azure portal and you should be automatically signed in to the xCarrier® for which you set up the SSO.

+You can also use Microsoft My Apps to test the application in any mode. When you click the xCarrier® tile in the My Apps, if configured in SP mode you would be redirected to the application sign on page for initiating the login flow and if configured in IDP mode, you should be automatically signed in to the xCarrier® for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Next steps

+Once you configure xCarrier® you can enforce session control, which protects exfiltration and infiltration of your organization’s sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+* [Microsoft Purview compliance portal](/microsoft-365/compliance/microsoft-365-compliance-center)

+* The cluster identity used by the AKS cluster must have at least [Network Contributor](../role-based-access-control/built-in-roles.md#network-contributor) role on the subnet within your virtual network. CLI helps do the role assignment automatically. If you are using ARM template or other clients, the role assignment needs to be done manually. You must also have the appropriate permissions, such as the subscription owner, to create a cluster identity and assign it permissions. If you wish to define a [custom role](../role-based-access-control/custom-roles.md) instead of using the built-in Network Contributor role, the following permissions are required:

+> [!NOTE]

+> If you are using CLI, you can skip this step. With ARM template or other clients, you need to do the below role assignment.

+- If you don't have one already, you need to create an [AKS cluster][deploy-cluster] and an Azure Container Registry instance.

+| keda | Event-driven autoscaling for the applications on your AKS cluster. | [Simplified application autoscaling with Kubernetes Event-driven Autoscaling (KEDA) add-on][keda]|

+| [Grafana][grafana] | An open-source dashboard for observability. | [Deploy Grafana on Kubernetes][grafana-install] or use [Managed Grafana][managed-grafana]|

+[managed-grafana]: ../managed-grafan

+[keda]: keda-about.md

+[web-app-routing]: web-app-routing.md

+> [!NOTE]

+> AKS will create a kubelet MI in the Node resource group if you do not bring your own kubelet MI.

+> For creating and using your own VNet, static IP address, or attached Azure disk where the resources are outside of the worker node resource group, CLI will add the role assignement automatically. If you are using ARM template or other clients, you need to use the PrincipalID of the cluster System Assigned Managed Identity to perform a role assignment. For more information on role assignment, see [Delegate access to other Azure resources](kubernetes-service-principal.md#delegate-access-to-other-azure-resources).

+Azure CLI will automatically add required role assignment for control plane MI. If you are using ARM template or other clients, you need to create the role assignment manually.

+az role assignment create --assignee <control-plane-identity-object-id> --role "Managed Identity Operator" --scope <kubelet-identity-resource-id>

+Now you can use the following command to create your cluster with your existing identities. Provide the control plane identity resource ID via `assign-identity` and the kubelet managed identity via `assign-kubelet-identity`:

+ --assign-identity <identity-resource-id> \

+ --assign-kubelet-identity <kubelet-identity-resource-id>

+Now you can use the following command to update your cluster with your existing identities. Provide the control plane identity resource ID via `assign-identity` and the kubelet managed identity via `assign-kubelet-identity`:

+ --assign-identity <identity-resource-id> \

+ --assign-kubelet-identity <kubelet-identity-resource-id>

+The Premium V3 and Isolated V2 App Service Plan types can optionally be distributed across Availability Zones to improve resiliency and reliability for your business-critical workloads. This architecture is also known as [zone redundancy](how-to-zone-redundancy.md). The JBoss EAP clustering feature is compatabile with the zone redundancy feature.

+ The `LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN` environment variable enables the [Cleartext plugin](https://dev.mysql.com/doc/refman/8.0/en/cleartext-pluggable-authentication.html) in the MySQL Connector (see [Use Azure Active Directory for authentication with MySQL](../mysql/howto-configure-sign-in-azure-ad-authentication.md#compatibility-with-application-drivers)).

+> [Tutorial: Isolate back-end communication with Virtual Network integration](tutorial-networking-isolate-vnet.md)

+Next, we must connect the App hosted in our App Service to our database using a Connection String. You can use [Service Connector](../service-connector/overview.md) to create the connection.

+| [!INCLUDE [Connect Service step 1](<./includes/tutorial-dotnetcore-sqldb-app/azure-portal-connect-database-01.md>)] | :::image type="content" source="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-01-240px.png" alt-text="A screenshot showing how to locate the app service in the Azure portal." lightbox="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-01.png"::: |

+| [!INCLUDE [Connect Service step 2](<./includes/tutorial-dotnetcore-sqldb-app/azure-portal-connect-database-02.md>)] | :::image type="content" source="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-02-240px.png" alt-text="A screenshot showing how to locate Service Connector from the Azure portal." lightbox="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-02.png"::: |

+| [!INCLUDE [Connect Service step 3](<./includes/tutorial-dotnetcore-sqldb-app/azure-portal-connect-database-03.md>)] | :::image type="content" source="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-03-240px.png" alt-text="A screenshot showing how to create a connection to the SQL database for the app in the Azure portal." lightbox="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-03.png"::: |

+| [!INCLUDE [Connect Service step 4](<./includes/tutorial-dotnetcore-sqldb-app/azure-portal-connect-database-04.md>)] | :::image type="content" source="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-04-240px.png" alt-text="A screenshot showing how to enter username and password of SQL Database during service connection in the Azure portal." lightbox="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-04.png"::: |

+| [!INCLUDE [Connect Service step 5](<./includes/tutorial-dotnetcore-sqldb-app/azure-portal-connect-database-05.md>)] | :::image type="content" source="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-05-240px.png" alt-text="A screenshot showing how to review and create the connection in the Azure portal." lightbox="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-05.png"::: |

+| [!INCLUDE [Connect Service step 6](<./includes/tutorial-dotnetcore-sqldb-app/azure-portal-connect-database-06.md>)] | :::image type="content" source="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-06-240px.png" alt-text="A screenshot showing how to get the connection string for a service connector in the Azure portal." lightbox="./media/tutorial-dotnetcore-sqldb-app/azure-portal-connect-sql-db-06.png"::: |

+Configure the connection between your app and the SQL database by using the [az webapp connection create sql](/cli/azure/webapp/connection/create#az-webapp-connection-create-sql) command.

+az webapp connection create sql \

+ --resource-group msdocs-core-sql \

+ --name <your-app-service-name> \

+ --target-resource-group msdocs-core-sql \

+ --server <server-name> \

+ --database coreDB \

+ --query configurations

+When prompted, provide the administrator username and password for the SQL database.

+> [!NOTE]

+> The CLI command does everything the app needs to successfully connect to the database, including:

+>

+> - In your App Service app, adds a connection string with the name `AZURE_SQL_CONNECTIONSTRING`, which your code can use for its database connection. If the connection string is already in use, `AZURE_SQL_<connection-name>_CONNECTIONSTRING` is used for the name instead.

+> - In your SQL database server, allows Azure services to access the SQL database server.

+Copy this connection string value from the output for later.

+To see the entirety of the command output, drop the `--query` in the command.

+To generate our database schema, set up a firewall rule on the SQL database server. This rule lets your local computer connect to Azure. For this step, you'll need to know your local computer's IP address. For more information about how to find the IP address, [see here](https://whatismyipaddress.com/).

+Next, update the *appsettings.json* file in the sample project with the [connection string Azure SQL Database](#5connect-the-app-to-the-database). The update allows us to run migrations locally against our database hosted in Azure. Replace the username and password placeholders with the values you chose when creating your database.

+"AZURE_SQL_CONNECTIONSTRING": "Data Source=<your-server-name>.database.windows.net,1433;Initial Catalog=coreDb;User ID=<username>;Password=<password>"

+```

+Next, update the *Startup.cs* file the sample project by updating the existing connection string name `MyDbConnection` to `AZURE_SQL_CONNECTIONSTRING`:

+```csharp

+services.AddDbContext<MyDatabaseContext>(options =>

+ options.UseSqlServer(Configuration.GetConnectionString("AZURE_SQL_CONNECTIONSTRING")));

+If you receive the error `Client with IP address xxx.xxx.xxx.xxx is not allowed to access the server`, that means the IP address you entered into your Azure firewall rule is incorrect. To fix this issue, update the Azure firewall rule with the IP address provided in the error message.

+ document_analysis_client = DocumentAnalysisClient(endpoint=endpoint, credential=AzureKeyCredential(key))

+Use the Azure portal to create a script that automates the agent download and installation and establishes the connection with Azure Arc.

+<!--1. Launch the Azure Arc service in the Azure portal by searching for and selecting **Servers - Azure Arc**.

+1. On the **Servers - Azure Arc** page, select **Add** near the upper left.-->

+1. [Go to the Azure portal page for adding servers with Azure Arc](https://ms.portal.azure.com/#view/Microsoft_Azure_HybridCompute/HybridVmAddBlade). Select the **Add a single server** tile, then select **Generate script**.

+ :::image type="content" source="media/quick-enable-hybrid-vm/add-single-server.png" alt-text="Screenshot of Azure portal's add server page." lightbox="media/quick-enable-hybrid-vm/add-single-server-expanded.png":::

+ > [!NOTE]

+ > In the portal, you can also reach the page for adding servers by searching for and selecting "Servers - Azure Arc" and then selecting **+Add**.

+ Title: Automatic extension upgrade (preview) for Azure Arc-enabled servers

+description: Learn how to enable the automatic extension upgrades for your Azure Arc-enabled servers.

+# Automatic extension upgrade (preview) for Azure Arc-enabled servers

+Automatic extension upgrade (preview) is available for Azure Arc-enabled servers that have supported VM extensions installed. When automatic extension upgrade is enabled on a machine, the extension is upgraded automatically whenever the extension publisher releases a new version for that extension.

+ Automatic extension upgrade has the following features:

+> In this release, it is only possible to configure automatic extension upgrade with the Azure CLI and Azure PowerShell module.

+## How does automatic extension upgrade work?

+The extension upgrade process replaces the existing Azure VM extension version supported by Azure Arc-enabled servers with a new version of the same extension when published by the extension publisher. This feature is enabled by default for all extensions you deploy the Azure Arc-enabled servers unless you explicitly opt-out of automatic upgrades.

+- Geo-paired regions aren't applicable.

+- Availability Zones aren't applicable.

+- Machines are batched on a best effort basis to avoid concurrent updates for all machines registered with Arc-enabled servers in a subscription.

+### Automatic rollback and retries

+If an extension upgrade fails, Azure will try to repair the extension by performing the following actions:

+1. The Azure Connected Machine agent will automatically reinstall the last known good version of the extension to attempt to restore functionality.

+1. If the rollback is successful, the extension status will show as **Succeeded** and the extension will be added to the automatic upgrade queue again. The next upgrade attempt can be as soon as the next hour and will continue until the upgrade is successful.

+1. If the rollback fails, the extension status will show as **Failed** and the extension will no longer function as intended. You'll need to [remove](manage-vm-extensions-cli.md#remove-extensions) and [reinstall](manage-vm-extensions-cli.md#enable-extension) the extension to restore functionality.

+If you continue to have trouble upgrading an extension, you can [disable automatic extension upgrade](#disable-automatic-extension-upgrade) to prevent the system from trying again while you troubleshoot the issue. You can [enable automatic extension upgrade](#enable-automatic-extension-upgrade) again when you're ready.

+Automatic extension upgrade supports the following extensions (and more are added periodically):

+## Enable automatic extension upgrade

+Automatic extension upgrade is enabled by default when you install extensions on Azure Arc-enabled servers. To enable automatic extension upgrade for an existing extension, you can use Azure CLI or Azure PowerShell to set the `enableAutomaticUpgrade` property on the extension to `true`. You'll need to repeat this process for every extension where you'd like to enable automatic upgrades.

+Use the [az connectedmachine extension update](/cli/azure/connectedmachine/extension) command to enable automatic upgrade on an extension:

+To check the status of automatic extension upgrade for all extensions on an Arc-enabled server, run the following command:

+To enable automatic extension upgrade for an extension using Azure PowerShell, use the [Update-AzConnectedMachineExtension](/powershell/module/az.connectedmachine/update-azconnectedmachineextension) cmdlet:

+To check the status of automatic extension upgrade for all extensions on an Arc-enabled server, run the following command:

+If multiple extension upgrades are available for a machine, the upgrades may be batched together, but each extension upgrade is applied individually on a machine. A failure on one extension doesn't impact the other extension(s) to be upgraded. For example, if two extensions are scheduled for an upgrade, and the first extension upgrade fails, the second extension will still be upgraded.

+## Disable automatic extension upgrade

+To disable automatic extension upgrade for an extension, set the `enable-auto-upgrade` property to `false`.

+With Azure CLI, use the [az connectedmachine extension update](/cli/azure/connectedmachine/extension) command to disable automatic upgrade on an extension:

+With Azure PowerShell, use the [Update-AzConnectedMachineExtension](/powershell/module/az.connectedmachine/update-azconnectedmachineextension) cmdlet:

+## Check automatic extension upgrade history

+You can use the Azure Activity Log to identify extensions that were automatically upgraded. You can find the Activity Log tab on individual Azure Arc-enabled server resources, resource groups, and subscriptions. Extension upgrades are identified by the `Upgrade Extensions on Azure Arc machines (Microsoft.HybridCompute/machines/upgradeExtensions/action)` operation.

+To view automatic extension upgrade history, search for the **Azure Activity Log** in the Azure Portal. Select **Add filter** and choose the Operation filter. For the filter criteria, search for "Upgrade Extensions on Azure Arc machines" and select that option. You can optionally add a second filter for **Event initiated by** and set "Azure Regional Service Manager" as the filter criteria to only see automatic upgrade attempts and exclude upgrades manually initiated by users.

+ "apiVersion": "2022-03-10",

+ "enableAutomaticUpgrade": true,

+ "apiVersion": "2022-03-10",

+ "enableAutomaticUpgrade": true,

+ "apiVersion": "2022-03-10",

+ "apiVersion": "2022-03-10",

+ "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",

+ "contentVersion": "1.0.0.0",

+ "apiVersion": "2022-03-10",

+ "enableAutomaticUpgrade": true

+ "outputs": {

+ }

+ "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",

+ "contentVersion": "1.0.0.0",

+ "apiVersion": "2022-03-10",

+ "enableAutomaticUpgrade": true

+ "outputs": {

+ }

+ "apiVersion": "2022-03-10",

+ "enableAutomaticUpgrade": true,

+ "msiEndpoint": "http://localhost:40342/metadata/identity"

+ "apiVersion": "2022-03-10",

+ "enableAutomaticUpgrade": true,

+ "requireInitialSync": <initial synchronization of certificates e.g.: true>,

+ "msiEndpoint": "http://localhost:40342/metadata/identity"

+```yaml

+ vars:

+ azure:

+ service_principal_id: 'INSERT-SERVICE-PRINCIPAL-CLIENT-ID'

+ service_principal_secret: 'INSERT-SERVICE-PRINCIPAL-SECRET'

+ resource_group: 'INSERT-RESOURCE-GROUP'

+ tenant_id: 'INSERT-TENANT-ID'

+ subscription_id: 'INSERT-SUBSCRIPTION-ID'

+ location: 'INSERT-LOCATION'

+ - name: Download the Connected Machine Agent on Linux servers

+ - name: Download the Connected Machine Agent on Windows servers

+ win_get_url:

+ url: https://aka.ms/AzureConnectedMachineAgent

+ dest: C:\AzureConnectedMachineAgent.msi

+ win_package:

+ path: C:\AzureConnectedMachineAgent.msi

+ shell: sudo azcmagent connect --service-principal-id {{ azure.service_principal_id }} --service-principal-secret {{ azure.service_principal_secret }} --resource-group {{ azure.resource_group }} --tenant-id {{ azure.tenant_id }} --location {{ azure.location }} --subscription-id {{ azure.subscription_id }}

+ win_shell: '& $env:ProgramFiles\AzureConnectedMachineAgent\azcmagent.exe connect --service-principal-id "{{ azure.service_principal_id }}" --service-principal-secret "{{ azure.service_principal_secret }}" --resource-group "{{ azure.resource_group }}" --tenant-id "{{ azure.tenant_id }}" --location "{{ azure.location }}" --subscription-id "{{ azure.subscription_id }}"'

+1. Within the Ansible playbook, modify the variables under the **vars section** with the service principal and Azure details collected earlier:

+1. Enter the correct hosts field capturing the target servers for onboarding to Azure Arc. You can employ [Ansible patterns](https://docs.ansible.com/ansible/latest/user_guide/intro_patterns.html#common-patterns) to selectively target which hybrid machines to onboard.

+**&ast;&ast;** Azure Information Protection (AIP) is part of the Microsoft Purview Information Protection solution - it extends the labeling and classification functionality provided by Microsoft 365. Before AIP can be used for DoD workloads at a given impact level (IL), the corresponding Microsoft 365 services must be authorized at the same IL.

+|[Dell Marketing LP](https://www.dell.com/)|

+|Doublehorn, LLC|

+The **recommendation** is to enable automatic update of the agent by enabling the [Automatic Extension Upgrade (preview)](../../azure-arc/servers/manage-automatic-vm-extension-upgrade.md#enable-automatic-extension-upgrade) feature, using the following PowerShell commands.

+The **recommendation** is to enable automatic update of the agent by enabling the [Automatic Extension Upgrade (preview)](../../azure-arc/servers/manage-automatic-vm-extension-upgrade.md#enable-automatic-extension-upgrade) feature, using the following PowerShell commands.

+ms.reviwer: casocha

+ms.reviwer: cithomas

+ // Hence instrumentation key/ connection string and any changes to default logging level must be specified here.

+ services.AddApplicationInsightsTelemetryWorkerService("instrumentation key here");

+ // To pass a connection string

+ // - aiserviceoptions must be created

+ // - set connectionstring on it

+ // - pass it to AddApplicationInsightsTelemetryWorkerService()

+| | [Enable for a new OpenShift cluster by using the Azure CLI](/azure/openshift/#az-openshift-create) |

+|WestCentralUS¹|EastUS |

+Activity logs you send to a [Log Analytics workspace](/azure/azure-monitor/logs/log-analytics-workspace-overview) are stored in a table called AzureActivity.

+Activity log insights are a curated [Log Analytics workbook](/azure/azure-monitor/visualize/workbooks-overview) with dashboards that visualize the data in the AzureActivity table. For example, which administrators deleted, updated or created resources, and whether the activities failed or succeeded.

+ * **Azure Activity Log Entries** shows the count of Activity log records in each [activity log category](/azure/azure-monitor/essentials/activity-log#categories).

+|AvailableStorage|No|(deprecated) Available Storage|Bytes|Total|"Available Storage"will be removed from Azure Monitor at the end of September 2023. Cosmos DB collection storage size is now unlimited. The only restriction is that the storage size for each logical partition key is 20GB. You can enable PartitionKeyStatistics in Diagnostic Log to know the storage consumption for top partition keys. For more info about Cosmos DB storage quota, see [Azure Cosmos DB service quotas](../../cosmos-db/concepts-limits.md). After deprecation, the remaining alert rules still defined on the deprecated metric will be automatically disabled post the deprecation date.|CollectionName, DatabaseName, Region|

+The following diagram details the steps taken by information from the database engine and Azure resource logs, and how they can be surfaced. For a more detailed diagram of Azure SQL logging, see [Monitoring and diagnostic telemetry](/azure/azure-sql/database/monitor-tune-overview#monitoring-and-diagnostic-telemetry).

+- [Monitoring and performance tuning in Azure SQL Database and Azure SQL Managed Instance](/azure/azure-sql/database/monitor-tune-overview)

+### Focus navigation

+Choose whether or not to enable focus navigation.

+If enabled, only one screen at a time will be visible as you step through a process in the portal. If disabled, as you move through the steps of a process, you'll be able to move between them through a horizontal scroll bar.

+- **Select a directory**: Choose this option to select one of your directories. You'll start in that directory every time you sign in to the Azure portal, even if you had been working in a different directory last time.

+Notifications are system messages related to your current session. They provide information such as showing your current credit balance, confirming your last action, or letting you know when resources you created become available. When pop-up notifications are turned on, the messages briefly display in the top corner of your screen.

+You must have the appropriate access to a subscription before you can create a support request for it. This means you must have the [Owner](../../role-based-access-control/built-in-roles.md#owner), [Contributor](../../role-based-access-control/built-in-roles.md#contributor), or [Support Request Contributor](../../role-based-access-control/built-in-roles.md#support-request-contributor) role, or a custom role with [Microsoft.Support/*](../../role-based-access-control/resource-provider-operations.md#microsoftsupport), at the subscription level.

+> In most cases, you'll need to specify a subscription. Be sure to choose the subscription where you are experiencing the problem. The support engineer assigned to your case will only be able to access resources in the subscription you specify. The access requirement serves as a point of confirmation that the support engineer is sharing information to the right audience, which is a key factor for ensuring the security and privacy of customer data. For details on how Azure treats customer data, see [Data Privacy in the Trusted Cloud](https://azure.microsoft.com/overview/trusted-cloud/privacy/).

+>

+> If the issue applies to multiple subscriptions, you can mention additional subscriptions in your description, or by [sending a message](how-to-manage-azure-support-request.md#send-a-message) later. However, the support engineer will only be able to work on [subscriptions to which you have access](#azure-role-based-access-control). If you don't have the required access for a subscription, we won't be able to work on it as part of your request.

+For more details, see [pricing](https://azure.microsoft.com/pricing/details/video-indexer/).

+> [!NOTE]

+> The following sample is intended for Classic accounts only and not compatible with ARM accounts. For an updated sample for ARM please see [this ARM sample repo](https://github.com/Azure-Samples/media-services-video-indexer/blob/master/ApiUsage/ArmBased/Program.cs).

+ Title: Tutorial - Create a serverless chat app using Azure Web PubSub service and Azure Static Web Apps

+description: A tutorial for how to use Azure Web PubSub service and Azure Static Web Apps to build a serverless chat application.

+# Tutorial: Create a serverless chat app using Azure Web PubSub service and Azure Static Web Apps

+The Azure Web PubSub service helps you build real-time messaging web applications using WebSockets. And with Azure Static Web Apps, you can automatically build and deploy full stack web apps to Azure from a code repository conveniently. In this tutorial, you learn how to use Azure Web PubSub service and Azure Static Web Apps to build a serverless real-time messaging application under chat room scenario.

+In this tutorial, you learn how to:

+> [!div class="checklist"]

+> * Build a serverless chat app

+> * Work with Web PubSub function input and output bindings

+> * Work with Static Web Apps

+## Overview

+* GitHub along with DevOps provide source control and continuous delivery. So whenever there's code change to the source repo, Azure DevOps pipeline will soon apply it to Azure Static Web App and present to endpoint user.

+* When a new user is login, Functions `login` API will be triggered and generate Azure Web PubSub service client connection url.

+* When client init the connection request to Azure Web PubSub service, service will send a system `connect` event and Functions `connect` API will be triggered to auth the user.

+* When client send message to Azure Web PubSub service, service will send a user `message` event and Functions `message` API will be triggered and broadcast the message to all the connected clients.

+* Functions `validate` API will be triggered periodically for [CloudEvents Abuse Protection](https://github.com/cloudevents/spec/blob/v1.0/http-webhook.md#4-abuse-protection) purpose, when the events in Azure Web PubSub are configured with predefined parameter `{event}`, that is, https://$STATIC_WEB_APP/api/{event}.

+> [!NOTE]

+> Functions APIs `connect` and `message` will be triggered when Azure Web PubSub service is configured with these 2 events.

+## Prerequisites

+* [GitHub](https://github.com/) account

+* [Azure](https://portal.azure.com/) account

+* [Azure CLI](/cli/azure) (version 2.29.0 or higher) or [Azure Cloud Shell](../cloud-shell/quickstart.md) to manage Azure resources

+## Create a Web PubSub resource

+1. Sign in to the Azure CLI by using the following command.

+ ```azurecli-interactive

+ az login

+ ```

+1. Create a resource group.

+ ```azurecli-interactive

+ az group create \

+ --name my-awps-swa-group \

+ --location "eastus2"

+ ```

+1. Create a Web PubSub resource.

+ ```azurecli-interactive

+ az webpubsub create \

+ --name my-awps-swa \

+ --resource-group my-awps-swa-group \

+ --location "eastus2" \

+ --sku Free_F1

+ ```

+1. Get and hold the access key for later use.

+ ```azurecli-interactive

+ az webpubsub key show \

+ --name my-awps-swa \

+ --resource-group my-awps-swa-group

+ ```

+ ```azurecli-interactive

+ AWPS_ACCESS_KEY=<YOUR_AWPS_ACCESS_KEY>

+ ```

+ Replace the placeholder `<YOUR_AWPS_ACCESS_KEY>` from previous result `primaryConnectionString`.

+## Create a repository

+This article uses a GitHub template repository to make it easy for you to get started. The template features a starter app used to deploy using Azure Static Web Apps.

+1. Navigate to the following template to create a new repository under your repo:

+ 1. [https://github.com/Azure/awps-swa-sample/generate](https://github.com/login?return_to=/Azure/awps-swa-sample/generate)

+1. Name your repository **my-awps-swa-app**

+Select **`Create repository from template`**.

+## Create a static web app

+Now that the repository is created, you can create a static web app from the Azure CLI.

+1. Create a variable to hold your GitHub user name.

+ ```azurecli-interactive

+ GITHUB_USER_NAME=<YOUR_GITHUB_USER_NAME>

+ ```

+ Replace the placeholder `<YOUR_GITHUB_USER_NAME>` with your GitHub user name.

+1. Create a new static web app from your repository. As you execute this command, the CLI starts GitHub interactive login experience. Following the message to complete authorization.

+ ```azurecli-interactive

+ az staticwebapp create \

+ --name my-awps-swa-app \

+ --resource-group my-awps-swa-group \

+ --source https://github.com/$GITHUB_USER_NAME/my-awps-swa-app \

+ --location "eastus2" \

+ --branch main \

+ --app-location "src" \

+ --api-location "api" \

+ --login-with-github

+ ```

+ > [!IMPORTANT]

+ > The URL passed to the `--source` parameter must not include the `.git` suffix.

+1. Navigate to **https://github.com/login/device**.

+1. Enter the user code as displayed your console's message.

+1. Select the **Continue** button.

+1. Select the **Authorize AzureAppServiceCLI** button.

+1. Configure the static web app settings.

+ ```azurecli-interactive

+ az staticwebapp appsettings set \

+ -n my-awps-swa-app \

+ --setting-names WebPubSubConnectionString=$AWPS_ACCESS_KEY WebPubSubHub=sample_swa

+ ```

+## View the website

+There are two aspects to deploying a static app. The first operation creates the underlying Azure resources that make up your app. The second is a GitHub Actions workflow that builds and publishes your application.

+Before you can navigate to your new static site, the deployment build must first finish running.

+1. Return to your console window and run the following command to list the URLs associated with your app.

+ ```azurecli-interactive

+ az staticwebapp show \

+ --name my-awps-swa-app \

+ --query "repositoryUrl"

+ ```

+ The output of this command returns the URL to your GitHub repository.

+1. Copy the **repository URL** and paste it into the browser.

+1. Select the **Actions** tab.

+ At this point, Azure is creating the resources to support your static web app. Wait until the icon next to the running workflow turns into a check mark with green background ✅. This operation may take a few minutes to complete.

+ Once the success icon appears, the workflow is complete and you can return back to your console window.

+2. Run the following command to query for your website's URL.

+ ```azurecli-interactive

+ az staticwebapp show \

+ --name my-awps-swa-app \

+ --query "defaultHostname"

+ ```

+ Hold the url to set in the Web PubSub event handler.

+ ```azurecli-interactive

+ STATIC_WEB_APP=<YOUR_STATIC_WEB_APP>

+ ```

+## Configure the Web PubSub event handler

+Now you're very close to complete. The last step is to configure Web PubSub transfer client requests to your function APIs.

+1. Run command to configure Web PubSub service events. It's mapping to some functions under the `api` folder in your repo.

+ ```azurecli-interactive

+ az webpubsub hub create \

+ -n "my-awps-swa" \

+ -g "my-awps-swa-group" \

+ --hub-name "sample_swa" \

+ --event-handler url-template=https://$STATIC_WEB_APP/api/{event} user-event-pattern="*" \

+ --event-handler url-template=https://$STATIC_WEB_APP/api/{event} system-event="connect"

+ ```

+Now you're ready to play with your website **<YOUR_STATIC_WEB_APP>**. Copy it to browser and click continue to start chatting with your friends.

+## Clean up resources

+If you're not going to continue to use this application, you can delete the resource group and the static web app by running the following command.

+```azurecli-interactive

+az group delete --name my-awps-swa-group

+```

+## Next steps

+In this quickstart, you learned how to run a serverless chat application. Now, you could start to build your own application.

+> [!div class="nextstepaction"]

+> [Tutorial: Client streaming using subprotocol](tutorial-subprotocol.md)

+> [!div class="nextstepaction"]

+> [Azure Web PubSub bindings for Azure Functions](reference-functions-bindings.md)

+> [!div class="nextstepaction"]

+> [Explore more Azure Web PubSub samples](https://github.com/Azure/azure-webpubsub/tree/main/samples)

+Select the right locale that matches your training data to train a custom neural voice model. For example, if the recording data is spoken in English with a British accent, select `en-GB`.

+With the cross-lingual feature (preview), you can transfer your custom neural voice model to speak a second language. For example, with the `zh-CN` data, you can create a voice that speaks `en-AU` or any of the languages marked with "Yes" in the Cross-lingual column in the following table.

+There are two Custom Neural Voice (CNV) project types: CNV Pro and CNV Lite (preview). In the following table, all the languages are supported by CNV Pro, and the languages marked with "Yes" in the Custom Neural Voice Lite column are supported by CNV Lite.

+| Language | Locale | Cross-lingual (preview) |Custom Neural Voice Lite (preview)|

+|--|--|--|--|

+| Arabic (Egypt) | `ar-EG` | No |No|

+| Arabic (Saudi Arabia) | `ar-SA` | No |No|

+| Bulgarian (Bulgaria) | `bg-BG` | No |No|

+| Catalan (Spain) | `ca-ES` | No |No|

+| Chinese (Cantonese, Traditional) | `zh-HK` | No |No|

+| Chinese (Mandarin, Simplified) | `zh-CN` | Yes |Yes|

+| Chinese (Mandarin, Simplified), English bilingual | `zh-CN` bilingual | Yes |No|

+| Chinese (Taiwanese Mandarin) | `zh-TW` | No |No|

+| Croatian (Croatia) | `hr-HR` | No |No|

+| Czech (Czech) | `cs-CZ` | No |No|

+| Danish (Denmark) | `da-DK` | No |No|

+| Dutch (Netherlands) | `nl-NL` | No |No|

+| English (Australia) | `en-AU` | Yes |No|

+| English (Canada) | `en-CA` | No |Yes|

+| English (India) | `en-IN` | No |No|

+| English (Ireland) | `en-IE` | No |No|

+| English (United Kingdom) | `en-GB` | Yes |Yes|

+| English (United States) | `en-US` | Yes |Yes|

+| Finnish (Finland) | `fi-FI` | No |No|

+| French (Canada) | `fr-CA` | Yes |No|

+| French (France) | `fr-FR` | Yes |Yes|

+| French (Switzerland) | `fr-CH` | No |No|

+| German (Austria) | `de-AT` | No |No|

+| German (Germany) | `de-DE` | Yes |Yes|

+| German (Switzerland) | `de-CH` | No |No|

+| Greek (Greece) | `el-GR` | No |No|

+| Hebrew (Israel) | `he-IL` | No |No|

+| Hindi (India) | `hi-IN` | No |No|

+| Hungarian (Hungary) | `hu-HU` | No |No|

+| Indonesian (Indonesia) | `id-ID` | No |No|

+| Italian (Italy) | `it-IT` | Yes |Yes|

+| Japanese (Japan) | `ja-JP` | Yes |No|

+| Korean (Korea) | `ko-KR` | Yes |Yes|

+| Malay (Malaysia) | `ms-MY` | No |No|

+| Norwegian (Bokmål, Norway) | `nb-NO` | No |No|

+| Polish (Poland) | `pl-PL` | No |No|

+| Portuguese (Brazil) | `pt-BR` | Yes |Yes|

+| Portuguese (Portugal) | `pt-PT` | No |No|

+| Romanian (Romania) | `ro-RO` | No |No|

+| Russian (Russia) | `ru-RU` | Yes |No|

+| Slovak (Slovakia) | `sk-SK` | No |No|

+| Slovenian (Slovenia) | `sl-SI` | No |No|

+| Spanish (Mexico) | `es-MX` | Yes |Yes|

+| Spanish (Spain) | `es-ES` | Yes |No|

+| Swedish (Sweden) | `sv-SE` | No |No|

+| Tamil (India) | `ta-IN` | No |No |

+| Telugu (India) | `te-IN` | No |No |

+| Thai (Thailand) | `th-TH` | No |No |

+| Turkish (Turkey) | `tr-TR` | No |No|

+| Vietnamese (Vietnam) | `vi-VN` | No |No|

+Source file types will be preserved during the document translation with the following **exceptions**:

+| Document summarization | 25 |

+You also will need an Azure storage account where you will upload your `.txt` documents that will be used to train a model to extract entities.

+> [!Note]

+> You shouldn't move the storage account to a different resource group or subscription once it's linked with the Language resource.

+> [!Note]

+> You shouldn't move the storage account to a different resource group or subscription once it's linked with the Language resource.

+> The conversation summarization feature is a preview capability provided ΓÇ£AS ISΓÇ¥ and ΓÇ£WITH ALL FAULTS.ΓÇ¥ As such, Conversation Summarization (preview) should not be implemented or deployed in any production use. The customer is solely responsible for any use of conversation summarization.

+ Title: Summarization language support

+# [Conversation summarization (preview)](#tab/conversation-summarization)

+## Languages supported by conversation summarization (preview)

+Summarization is one of the features offered by [Azure Cognitive Service for Language](../overview.md), a collection of machine learning and AI algorithms in the cloud for developing intelligent applications that involve written language. Use this article to learn more about this feature, and how to use it in your applications.

+* Summarization works with a variety of written languages. See [language support](language-support.md?tabs=document-summarization) for more information.

+* Conversation summarization accepts text in English. See [language support](language-support.md?tabs=conversation-summarization) for more information.

+The endpoints below should be reachable for U.S. Government GCC High customers only

+| Category | IP ranges or FQDN | Ports |

+| :-- | :-- | :-- |

+| Media traffic | 52.127.88.0/21, 52.238.114.160/32, 52.238.115.146/32, 52.238.117.171/32, 52.238.118.132/32, 52.247.167.192/32, 52.247.169.1/32, 52.247.172.50/32, 52.247.172.103/32, 104.212.44.0/22, 195.134.228.0/22 | UDP 3478 through 3481, TCP ports 443 |

+| Signaling, telemetry, registration| *.gov.teams.microsoft.us, *.infra.gov.skypeforbusiness.us, *.online.gov.skypeforbusiness.us, gov.teams.microsoft.us | TCP 443, 80 |

+| **Consumption** | Integration service environment (ISE) | [Managed connector - Standard class](managed.md) and ISE version. For more information, review the [SQL Server managed connector reference](/connectors/sql). <br><br>**Note**: The ISE version uses the [ISE message limits](../logic-apps/logic-apps-limits-and-config.md#message-size-limits), not the managed version's message limits. |

+ * In an ISE, you don't need the on-premises data gateway for SQL Server Authentication and non-Windows Authentication connections, and you can use the ISE-versioned SQL Server connector. For Windows Authentication, you need the [on-premises data gateway](../logic-apps/logic-apps-gateway-install.md) on a local computer and a [data gateway resource that's already created in Azure](../logic-apps/logic-apps-gateway-connection.md). The ISE-version connector doesn't support Windows Authentication, so you have to use the regular SQL Server managed connector.

+1. In the [Azure portal](https://portal.azure.com), open your blank logic app workflow in the designer.

+1. In the [Azure portal](https://portal.azure.com), open your blank logic app workflow in the designer.

+* Consumption logic app workflows: [Visual Studio](../logic-apps/quickstart-create-logic-apps-with-visual-studio.md) or [Visual Studio Code](../logic-apps/quickstart-create-logic-apps-visual-studio-code.md)

+* Standard logic app workflows: [Visual Studio Code](../logic-apps/create-single-tenant-workflows-visual-studio-code.md)

+1. In the [Azure portal](https://portal.azure.com), open your logic app workflow in the designer.

+1. In the [Azure portal](https://portal.azure.com), open your logic app workflow in the designer.

+After you provide this information, continue with the following steps based on your target database:

+In the connection information box, complete the following steps:

+ | **Service principal (Azure AD application)** | - Supported with the SQL Server managed connector. <br><br>- Requires an Azure AD application and service principal. For more information, see [Create an Azure AD application and service principal that can access resources using the Azure portal](../active-directory/develop/howto-create-service-principal-portal.md). |

+ | **Logic Apps Managed Identity** | - Supported with the SQL Server managed connector and ISE-versioned connector. <br><br>- Requires the following items: <br><br> A valid managed identity that's [enabled on your logic app resource](../logic-apps/create-managed-service-identity.md) and has access to your database. <br><br> **SQL DB Contributor** role access to the SQL Server resource <br><br> **Contributor** access to the resource group that includes the SQL Server resource. <br><br>For more information, see [SQL - Server-Level Roles](/sql/relational-databases/security/authentication-access/server-level-roles). |

+ | [**Azure AD Integrated**](/azure/azure-sql/database/authentication-aad-overview) | - Supported with the SQL Server managed connector and ISE-versioned connector. <br><br>- Requires a valid managed identity in Azure Active Directory (Azure AD) that's [enabled on your logic app resource](../logic-apps/create-managed-service-identity.md) and has access to your database. For more information, see these topics: <br><br>- [Azure SQL Security Overview - Authentication](/azure/azure-sql/database/security-overview#authentication) <br>- [Authorize database access to Azure SQL - Authentication and authorization](/azure/azure-sql/database/logins-create-manage#authentication-and-authorization) <br>- [Azure SQL - Azure AD Integrated authentication](/azure/azure-sql/database/authentication-aad-overview) |

+ | [**SQL Server Authentication**](/sql/relational-databases/security/choose-an-authentication-mode#connecting-through-sql-server-authentication) | - Supported with the SQL Server managed connector and ISE-versioned connector. <br><br>- Requires the following items: <br><br> A data gateway resource that's previously created in Azure for your connection, regardless whether your logic app is in multi-tenant Azure Logic Apps or an ISE. <br><br> A valid user name and strong password that are created and stored in your SQL Server database. For more information, see the following topics: <br><br>- [Azure SQL Security Overview - Authentication](/azure/azure-sql/database/security-overview#authentication) <br>- [Authorize database access to Azure SQL - Authentication and authorization](/azure/azure-sql/database/logins-create-manage#authentication-and-authorization) |

+ The following examples show how the connection information box might appear if you select **Azure AD Integrated** authentication.

+In the connection information box, complete the following steps:

+ | [**SQL Server Authentication**](/sql/relational-databases/security/choose-an-authentication-mode#connecting-through-sql-server-authentication) | - Supported with the SQL Server managed connector and ISE-versioned connector. <br><br>- Requires the following items: <br><br> A data gateway resource that's previously created in Azure for your connection, regardless whether your logic app is in multi-tenant Azure Logic Apps or an ISE. <br><br> A valid user name and strong password that are created and stored in your SQL Server. <br><br>For more information, see [SQL Server Authentication](/sql/relational-databases/security/choose-an-authentication-mode#connecting-through-sql-server-authentication). |

+ | [**Windows Authentication**](/sql/relational-databases/security/choose-an-authentication-mode#connecting-through-windows-authentication) | - Supported with the SQL Server managed connector. <br><br>- Requires the following items: <br><br> A data gateway resource that's previously created in Azure for your connection, regardless whether your logic app is in multi-tenant Azure Logic Apps or an ISE. <br><br> A valid Windows user name and password to confirm your identity through your Windows account. <br><br>For more information, see [Windows Authentication](/sql/relational-databases/security/choose-an-authentication-mode#connecting-through-windows-authentication). |

+ The following examples show how the connection information box might appear if you select **Windows** authentication.

+* Learn about other [managed connectors for Azure Logic Apps](../connectors/apis-list.md)

+The following is an example of the `containers` array in the [`properties.template`](azure-resource-manager-api-spec.md#propertiestemplate) section of a container app resource template. The excerpt shows the available configuration options when setting up a container.

+With the registry information set up, the saved credentials can be used to pull a container image from the private registry when your app is deployed.

+### Managed identity with Azure Container Registry

+You can use an Azure managed identity to authenticate with Azure Container Registry instead of using a username and password. To use a managed identity:

+- Assign a system-assigned or user-assigned managed identity to your container app.

+- Specify the managed identity you want to use for each registry.

+When assigned a managed identity to a registry, use the managed identity resource ID for a user-assigned identity, or "system" for the system-assigned identity. For more information about using managed identities see, [Managed identities in Azure Container Apps Preview](managed-identity.md).

+```json

+{

+ "identity": {

+ "type": "SystemAssigned,UserAssigned",

+ "userAssignedIdentities": {

+ "<IDENTITY1_RESOURCE_ID>": {}

+ }

+ }

+ "properties": {

+ "configuration": {

+ "registries": [

+ {

+ "server": "myacr1.azurecr.io",

+ "identity": "<IDENTITY1_RESOURCE_ID>"

+ },

+ {

+ "server": "myacr2.azurecr.io",

+ "identity": "system"

+ }]

+ }

+ ...

+ }

+}

+```

+The managed identity must have `AcrPull` access for the Azure Container Registry. For more information about assigning Azure Container Registry permissions to managed identities, see [Authenticate with managed identity](../container-registry/container-registry-authentication-managed-identity.md).

+#### Configure a user-assigned managed identity

+To configure a user-assigned managed identity:

+1. Create the user-assigned identity if it doesn't exist.

+1. Give the user-assigned identity `AcrPull` permission to your private repository.

+1. Add the identity to your container app configuration as shown above.

+For more information about configuring user-assigned identities, see [Add a user-assigned identity](managed-identity.md#add-a-user-assigned-identity).

+#### Configure a system-assigned managed identity

+System-assigned identities are created at the time your container app is created, and therefore, won't have `AcrPull` access to your Azure Container Registry. As a result, the image can't be pulled from your private registry when your app is first deployed.

+To configure a system-assigned identity, you must use one of the following methods.

+- **Option 1**: Use a public registry for the initial deployment:

+ 1. Create your container app using a public image and a system-assigned identity.

+ 1. Give the new system-assigned identity `AcrPull` access to your private Azure Container Registry.

+ 1. Update your container app replacing the public image with the image from your private Azure Container Registry.

+- **Option 2**: Restart your app after assigning permissions:

+ 1. Create your container app using a private image and a system-assigned identity. (The deployment will result in a failure to pull the image.)

+ 1. Give the new system-assigned identity `AcrPull` access to your private Azure Container Registry.

+ 1. Restart your container app revision.

+For more information about configuring system-assigned identities, see [Add a system-assigned identity](managed-identity.md#add-a-system-assigned-identity).

+- You can use managed identity to [authenticate with a private Azure Container Registry](containers.md#container-registries) without a username and password to pull containers for your Container App.

+The identity is only available within a running container, which means you can't use a managed identity in scaling rules or Dapr configuration. To access resources that require a connection string or key, such as storage resources, you'll still need to include the connection string or key in the `secretRef` of the scaling rule.

+To get a token for a resource, make an HTTP GET request to the endpoint, including the following parameters:

+| resource | Query | The Azure AD resource URI of the resource for which a token should be obtained. The resource could be one of the [Azure services that support Azure AD authentication](../active-directory/managed-identities-azure-resources/services-support-managed-identities.md#azure-services-that-support-azure-ad-authentication) or any other resource URI. |

+ ```azurecli-interactive

+ Title: Find request unit charge for a SQL query in Azure Cosmos DB

+description: Find the request unit charge for SQL queries against containers created with Azure Cosmos DB, using the Azure portal, .NET, Java, Python, or Node.js.

+- devx-track-js

+- kr2b-contr-experiment

+# Find the request unit charge for operations in Azure Cosmos DB SQL API

+The cost of all database operations is normalized by Azure Cosmos DB and is expressed by *request units* (RU). *Request charge* is the request units consumed by all your database operations. You can think of RUs as a performance currency abstracting the system resources such as CPU, IOPS, and memory that are required to perform the database operations supported by Azure Cosmos DB. No matter which API you use to interact with your container, costs are always measured in RUs. Whether the database operation is a write, point read, or query, costs are always measured in RUs. To learn more, see [Request Units in Azure Cosmos DB](../request-units.md).

+This article presents the different ways that you can find the request unit consumption for any operation run against a container in Azure Cosmos DB SQL API. If you're using a different API, see [API for MongoDB](../mongodb/find-request-unit-charge-mongodb.md), [Cassandra API](../cassandr).

+Currently, you can measure consumption only by using the Azure portal or by inspecting the response sent from Azure Cosmos DB through one of the SDKs. If you're using the SQL API, you have multiple options for finding the request charge for an operation.

+ :::image type="content" source="../media/find-request-unit-charge/portal-sql-query.png" alt-text="Screenshot of a SQL query request charge in the Azure portal.":::

+* [Request Units in Azure Cosmos DB](../request-units.md)

+* [Introduction to provisioned throughput in Azure Cosmos DB](../set-throughput.md)

+* [Monitor and debug with insights in Azure Cosmos DB](../use-metrics.md)

+Container container = client.GetContainer(databaseName,containerName);

+ItemResponse<SalesOrder> response = await this._container.CreateItemAsync(

+Uri collectionUri = UriFactory.CreateDocumentCollectionUri(databaseName, containerName);

+await client.CreateDocumentAsync(

+ collectionUri,

+ salesOrder,

+ new RequestOptions { PartitionKey = new PartitionKey(salesOrder.AccountNumber) });

+Because the .NET v3 SDK allows users to configure a custom serialization engine, there's no direct replacement for the `Document` type. When using Newtonsoft.Json (default serialization engine), `JObject` can be used to achieve the same functionality. When using a different serialization engine, you can use its base json document type (for example, `JsonDocument` for System.Text.Json). The recommendation is to use a C# type that reflects the schema of your items instead of relying on generic types.

+> The bitwise operators in Azure Cosmos DB SQL API follow the same behavior as bitwise operators in JavaScript. JavaScript stores numbers as 64 bits floating point numbers, but all bitwise operations are performed on 32 bits binary numbers. Before a bitwise operation is performed, JavaScript converts numbers to 32 bits signed integers. After the bitwise operation is performed, the result is converted back to 64 bits JavaScript numbers. For more information about the bitwise operators in JavaScript, see [JavaScript binary bitwise operators at MDN Web Docs](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators#binary_bitwise_operators).

+>- Azure portal for Germany is: `https://portal.microsoftazure.de`

+* Learn more in [Azure Support FAQ](https://azure.microsoft.com/support/faq)

+The anomaly detection model is a univariate time-series, unsupervised prediction and reconstruction-based model that uses 60 days of historical usage for training, then forecasts expected usage for the day. Anomaly detection forecasting uses a deep learning algorithm called [WaveNet](https://www.deepmind.com/blog/wavenet-a-generative-model-for-raw-audio). Note this is different than the Cost Management forecast. The total normalized usage is determined to be anomalous if it falls outside the expected range based on a predetermined confidence interval.

+- Learn about how to [Optimize your cloud investment with Cost Management](../costs/cost-mgt-best-practices.md).

+Whenever you find yourself building the same logic in an expression across multiple mapping data flows this would be a good opportunity to turn that into a user defined function.

+<tr><td><b>User interface</b></td><td>New regional format support</td><td>Choosing your language and the regional format in settings will influence the format of how data such as dates and times appear in the Azure Data Factory Studio monitoring. For example, the time format in Monitoring will appear like "Apr 2, 2022, 3:40:29 pm" when choosing English as the regional format, and "2 Apr 2022, 15:40:29" when choosing French as regional format. These settings affect only the Azure Data Factory Studio user interface and do not change/ modify your actual data and time zone.</td></tr>

+> For details of which Defender for Servers features are relevant for machines running on other cloud environments, see [Supported features for virtual machines and servers](supported-machines-endpoint-solutions-clouds-servers.md?tabs=features-windows#supported-features-for-virtual-machines-and-servers).

+Microsoft Defender for SQL includes two Microsoft Defender plans that extend Microsoft Defender for Cloud's [data security package](/azure/azure-sql/database/azure-defender-for-sql) to protect your SQL estate regardless of where it is located (Azure, multi-cloud or Hybrid environments). Microsoft Defender for SQL includes functions that can be used to discover and mitigate potential database vulnerabilities. Defender for SQL can also detect anomalous activities that may be an indication of a threat to your databases.

+To protect SQL databases in hybrid and multi-cloud environments, Defender for Cloud uses Azure Arc. Azure ARC connects your hybrid and multi-cloud machines. You can check out the following articles for more information:

+- [Connect your non-Azure machines to Microsoft Defender for Cloud](quickstart-onboard-machines.md)

+- [Connect your AWS accounts to Microsoft Defender for Cloud](quickstart-onboard-aws.md)

+- [Connect your GCP project to Microsoft Defender for Cloud](quickstart-onboard-gcp.md)

+

+

+

+ - Multi-cloud SQL servers:

+ - [Connect your AWS accounts to Microsoft Defender for Cloud](quickstart-onboard-aws.md)

+ - [Connect your GCP project to Microsoft Defender for Cloud](quickstart-onboard-gcp.md)

+ - **Microsoft Defender for SQL** brings threat detection and advanced defenses to your SQL Servers running on AWS EC2, AWS RDS Custom for SQL Server. This plan includes the advanced threat protection and vulnerability assessment scanning. You can view the [full list of available features](defender-for-sql-introduction.md).

+|Pricing:|The **CSPM plan** is free.<br>The **[Defender for SQL](defender-for-sql-introduction.md)** plan is billed at the same price as Azure resources.<br>The **[Defender for Containers](defender-for-containers-introduction.md)** plan is free during the preview. After which, it will be billed for AWS at the same price as for Azure resources.<br>For every AWS machine connected to Azure with [Azure Arc-enabled servers](../azure-arc/servers/overview.md), the **Defender for Servers** plan is billed at the same price as the [Microsoft Defender for Servers](defender-for-servers-introduction.md) plan for Azure machines. If an AWS EC2 doesn't have the Azure Arc agent deployed, you won't be charged for that machine.|

+- **To enable the Defender for SQL plan**, you'll need:

+ - Microsoft Defender for SQL enabled on your subscription. Learn how to [enable protection on all of your databases](quickstart-enable-database-protections.md).

+ - An active AWS account, with EC2 instances running SQL server or RDS Custom for SQL Server.

+ - Azure Arc for servers installed on your EC2 instances/RDS Custom for SQL Server.

+ - (Recommended) Use the auto provisioning process to install Azure Arc on all of your existing and future EC2 instances.

+ Auto provisioning is managed by AWS Systems Manager (SSM) using the SSM agent. Some Amazon Machine Images (AMIs) already have the SSM agent pre-installed. If you already have the SSM agent pre-installed, the AMI's are listed in [AMIs with SSM Agent preinstalled](https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent-technical-details.html#ami-preinstalled-agent). If your EC2 instances don't have the SSM Agent, you will need to install it using either of the following relevant instructions from Amazon:

+ - [Install SSM Agent for a hybrid environment (Windows)](https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-managed-win.html)

+ > [!NOTE]

+ > To enable the Azure Arc auto-provisioning, you'll need **Owner** permission on the relevant Azure subscription.

+

+ - Additional extensions should be enabled on the Arc-connected machines.

+ - Log Analytics (LA) agent on Arc machines, and ensure the selected workspace has security solution installed. The LA agent is currently configured in the subscription level. All of your multicloud AWS accounts and GCP projects under the same subscription will inherit the subscription settings.

+

+ Learn how to [configure auto-provisioning on your subscription](enable-data-collection.md#configure-auto-provisioning-for-agents-and-extensions-from-microsoft-defender-for-cloud).

+1. By default the **Databases** plan is set to **On**. This is necessary to extend Defender for SQL's coverage to your AWS EC2 and RDS Custom for SQL Server.

+ - (Optional) Select **Configure**, to edit the configuration as required. We recommend you leave it set to the default configuration.

+ - **Microsoft Defender for SQL** brings threat detection and advanced defenses to your SQL Servers running on GCP compute engine instances. This plan includes the advanced threat protection and vulnerability assessment scanning. You can view the [full list of available features](defender-for-sql-introduction.md).

+|Pricing:|The **CSPM plan** is free.<br>The **[Defender for SQL](defender-for-sql-introduction.md)** plan is billed at the same price as Azure resources.<br> The **Defender for Servers** plan is billed at the same price as the [Microsoft Defender for Servers](defender-for-servers-introduction.md) plan for Azure machines. If a GCP VM instance doesn't have the Azure Arc agent deployed, you won't be charged for that machine. <br>The **[Defender for Containers](defender-for-containers-introduction.md)** plan is free during the preview. After which, it will be billed for GCP at the same price as for Azure resources.|

+(**Servers/SQL only**) When Arc auto-provisioning is enabled, copy the unique numeric ID presented at the end of the Cloud Shell script.

+To locate the unique numeric ID in the GCP portal, Navigate to **IAM & Admin** > **Service Accounts**, in the Name column, locate `Azure-Arc for servers onboarding` and copy the unique numeric ID number (OAuth 2 Client ID).

+1. (**Servers/SQL only**) Select **Azure-Arc for servers onboarding**

+### Configure the Databases plan

+Connect your GCP VM instances to Azure Arc in order to have full visibility to Microsoft Defender for SQL security content.

+Microsoft Defender for SQL brings threat detection and vulnerability assessment to your GCP VM instances.

+To have full visibility to Microsoft Defender for SQL security content, ensure you have the following requirements configured:

+- Microsoft SQL servers on machines plan enabled on your subscription. Learn how to enable plan in the [Enable enhanced security features](quickstart-enable-database-protections.md) article.

+- Azure Arc for servers installed on your VM instances.

+ - **(Recommended) Auto-provisioning** - Auto-provisioning is enabled by default in the onboarding process and requires owner permissions on the subscription. Arc auto-provisioning process is using the OS config agent on GCP end. Learn more about the [OS config agent availability on GCP machines](https://cloud.google.com/compute/docs/images/os-details#vm-manager).

+ > [!NOTE]

+ > The Arc auto-provisioning process leverages the VM manager on your Google Cloud Platform, to enforce policies on the your VMs through the OS config agent. A VM with an [Active OS agent](https://cloud.google.com/compute/docs/manage-os#agent-state), will incur a cost according to GCP. Refer to [GCP's technical documentation](https://cloud.google.com/compute/docs/vm-manager#pricing) to see how this may affect your account.

+ > <br><br> Microsoft Defender for Servers does not install the OS config agent to a VM that does not have it installed. However, Microsoft Defender for Servers will enable communication between the OS config agent and the OS config service if the agent is already installed but not communicating with the service.

+ > <br><br> This can change the OS config agent from `inactive` to `active`, and will lead to additional costs.

+- Additional extensions should be enabled on the Arc-connected machines.

+ - SQL servers on machines. Ensure the plan is enabled on your subscription.

+ - Log Analytics (LA) agent on Arc machines. Ensure the selected workspace has security solution installed.

+ The LA agent and SQL servers on machines plan are currently configured in the subscription level, such that all the multicloud accounts and projects (from both AWS and GCP) under the same subscription will inherit the subscription settings and may result in additional charges.

+ Learn how to [configure auto-provisioning on your subscription](enable-data-collection.md#configure-auto-provisioning-for-agents-and-extensions-from-microsoft-defender-for-cloud).

+ > [!NOTE]

+ > Defender for SQL assigns tags to your GCP resources to manage the auto-provisioning process. You must have these tags properly assigned to your resources so that Defender for Cloud can manage your resources:

+ **Cloud**, **InstanceName**, **MDFCSecurityConnector**, **MachineId**, **ProjectId**, **ProjectNumber**

+- Automatic SQL server discovery and registration. Enable these settings to allow automatic discovery and registration of SQL servers, providing centralized SQL asset inventory and management.

+**To configure the Databases plan**:

+1. Follow the steps to [Connect your GCP project](#connect-your-gcp-project).

+1. On the Select plans screen select **Configure**.

+ :::image type="content" source="media/quickstart-onboard-gcp/view-configuration.png" alt-text="Screenshot showing where to click to configure the Databases plan.":::

+1. On the Auto provisioning screen, toggle the switches on or off depending on your need.

+ :::image type="content" source="media/quickstart-onboard-gcp/auto-provision-databases-screen.png" alt-text="Screenshot showing the toggle switches for the Databases plan.":::

+ > [!Note]

+ > If Azure Arc is toggled **Off**, you will need to follow the manual installation process mentioned above.

+1. Select **Save**.

+1. Continue from step number 8 of the [Connect your GCP projects](#connect-your-gcp-projects) instructions.

+## June 2022

+Updates in June include:

+- [General availability (GA) of Defender for SQL on machines for AWS and GCP environments](#general-availability-ga-of-defender-for-sql-on-machines-for-aws-and-gcp-environments)

+### General availability (GA) of Defender for SQL on machines for AWS and GCP environments

+The database protection capabilities provided by Microsoft Defender for Cloud, has added support for your SQL servers that are hosted in either AWS or GCP environments.

+Using Defender for SQL, enterprises can now protect their entire database estate, hosted in Azure, AWS, GCP and on-premises machines.

+Microsoft Defender for SQL provides a unified multicloud experience to view security recommendations, security alerts and vulnerability assessment findings for both the SQL server and the underlining Windows OS.

+Using the multicloud onboarding experience, you can enable and enforce databases protection for SQL servers running on AWS EC2, RDS Custom for SQL Server and GCP compute engine. After enabling either of these plans, all supported resources that exist within the subscription are protected. Future resources created on the same subscription will also be protected.

+Learn how to protect and connect your [AWS environment](quickstart-onboard-aws.md) and your [GCP organization](quickstart-onboard-gcp.md) with Microsoft Defender for Cloud.

+Microsoft Defender for Cloud has two main goals:

+## How your secure score is calculated

+ The current score for this control.<br>Current score=[Score per resource]*[Number of healthy resources].

+- **Insights** - :::image type="icon" source="media/secure-score-security-controls/insights.png" border="false":::

+ Gives you extra details for each recommendation. Which can be:

+ - :::image type="icon" source="media/secure-score-security-controls/preview-icon.png" border="false"::: Preview recommendation - This recommendation won't affect your secure score until it's GA.

+ - :::image type="icon" source="media/secure-score-security-controls/fix-icon.png" border="false"::: Fix - From within the recommendation details page, you can use 'Fix' to resolve this issue.

+ - :::image type="icon" source="media/secure-score-security-controls/enforce-icon.png" border="false"::: Enforce - From within the recommendation details page, you can automatically deploy a policy to fix this issue whenever someone creates a non-compliant resource.

+ - :::image type="icon" source="media/secure-score-security-controls/deny-icon.png" border="false"::: Deny - From within the recommendation details page, you can prevent new resources from being created with this issue.

+The table below lists the security controls in Microsoft Defender for Cloud. For each control, you can see the maximum number of points you can add to your secure score if you remediate *all* of the recommendations listed in the control, for *all* of your resources.

+The set of security recommendations provided with Defender for Cloud is tailored to the available resources in each organization's environment. You can [disable policies](tutorial-security-policy.md#disable-security-policies-and-disable-recommendations) and [exempt specific resources from a recommendation](exempt-resource.md) to further customize the recommendations.

+> For details about reviewing and editing your initiatives, see [Working with security policies](tutorial-security-policy.md).

+This article described the secure score and the included security controls.

+## Supported features for virtual machines and servers<a name="vm-server-features"></a>

+| [GA support for Arc-enabled Kubernetes clusters](#ga-support-for-arc-enabled-kubernetes-clusters) | July 2022 |

+| [Deprecating the "API App should only be accessible over HTTPS" policy](#deprecating-the-api-app-should-only-be-accessible-over-https-policy)|June 2022|

+### GA support for Arc-enabled Kubernetes clusters

+**Estimated date for change:** July 2022

+Defender for Containers is currently a preview feature for Arc-enabled Kubernetes clusters. In July, Arc-enabled Kubernetes clusters will be charged according to the listing on the [pricing page](https://azure.microsoft.com/pricing/details/defender-for-cloud/). Customers that already have clusters onboarded to Arc (on the subscription level) will incur charges.

+ - [Install endpoint protection solution on virtual machines (key: 83f577bd-a1b6-b7e1-0891-12ca19d1e6df)](https://portal.azure.com/#blade/Microsoft_Azure_Security/RecommendationsBlade/assessmentKey/83f577bd-a1b6-b7e1-0891-12ca19d1e6df)

+ - [Install endpoint protection solution on your machines (key: 383cf3bc-fdf9-4a02-120a-3e7e36c6bfee)](https://portal.azure.com/#blade/Microsoft_Azure_Security/RecommendationsBlade/assessmentKey/383cf3bc-fdf9-4a02-120a-3e7e36c6bfee)

+ - Assessment key for the **preview** recommendation: 37a3689a-818e-4a0e-82ac-b1392b9bb000

+ - Assessment key for the **GA** recommendation: 3bcd234d-c9c7-c2a2-89e0-c01f419c1a8a

+- **Account exemption capability** - Defender for Cloud has many features you can use to customize your experience and ensure that your secure score reflects your organization's security priorities. For example, you can [exempt resources and recommendations from your secure score](exempt-resource.md).

+#### Recommendations rename

+ | Property | Current value | New update's change |

+ |Assessment key | e52064aa-6853-e252-a11e-dffc675689c2 | No change|

+ | Name | [Deprecated accounts with owner permissions should be removed from your subscription](https://ms.portal.azure.com/#blade/Microsoft_Azure_Security/RecommendationsBlade/assessmentKey/e52064aa-6853-e252-a11e-dffc675689c2) |Subscriptions should be purged of accounts that are blocked in Active Directory and have owner permissions.|

+ |Description| User accounts that have been blocked from signing in, should be removed from your subscriptions.|These accounts can be targets for attackers looking to find ways to access your data without being noticed. <br> Learn more about securing the identity perimeter in [Azure Identity Management and access control security best practices](../security/fundamentals/identity-management-best-practices.md).|

+ |Related policy|[Deprecated accounts with owner permissions should be removed from your subscription](https://ms.portal.azure.com/#blade/Microsoft_Azure_Policy/PolicyDetailBlade/definitionId/%2fproviders%2fMicrosoft.Authorization%2fpolicyDefinitions%2febb62a0c-3560-49e1-89ed-27e074e9f8ad) | Subscriptions should be purged of accounts that are blocked in Active Directory and have owner permissions.|

+ |**Second recommendation**| - | - |

+ | Assessment key | 00c6d40b-e990-6acf-d4f3-471e747a27c4 | No change |

+ | Name | [Deprecated accounts should be removed from your subscription](https://ms.portal.azure.com/#blade/Microsoft_Azure_Security/RecommendationsBlade/assessmentKey/00c6d40b-e990-6acf-d4f3-471e747a27c4)|Subscriptions should be purged of accounts that are blocked in Active Directory and have read and write permissions.|

+|Description|User accounts that have been blocked from signing in, should be removed from your subscriptions. <br> These accounts can be targets for attackers looking to find ways to access your data without being noticed.|User accounts that have been blocked from signing into Active Directory, should be removed from your subscriptions.<br> Learn more about securing the identity perimeter in [Azure Identity Management and access control security best practices](../security/fundamentals/identity-management-best-practices.md).|

+ | Related policy | [Deprecated accounts should be removed from your subscription](https://ms.portal.azure.com/#blade/Microsoft_Azure_Policy/PolicyDetailBlade/definitionId/%2fproviders%2fMicrosoft.Authorization%2fpolicyDefinitions%2f6b1cbf55-e8b6-442f-ba4c-7246b6381474) | Subscriptions should be purged of accounts that are blocked in Active Directory and have read and write permissions. |

+| **SSH server is running inside a container** <br>(VM_ContainerSSH) | Machine logs indicate that an SSH server is running inside a Docker container. While this behavior can be intentional, it frequently indicates that a container is misconfigured or breached. | Execution | Medium |

+The policy `API App should only be accessible over HTTPS` is set to be deprecated. This policy will be replaced with `Web Application should only be accessible over HTTPS`, which will be renamed to `App Service apps should only be accessible over HTTPS`.

+In this article, you'll find a list of recommendations, which can be triggered on your IoT devices.

+| Low | Security twin configuration not optimal | Legacy Defender-IoT-micro-agent | Security twin configuration isn't optimal. |

+In this article, you'll find a list of built-in alerts, which can be triggered on your IoT devices.

+| Successful local login | High | Defender-IoT-micro-agent | Successful local sign-in to the device detected | Make sure the signed in user is an authorized party. | IoT_SucessfulLocalLogin |

+| Behavior similar to Fairware ransomware detected | Medium | Defender-IoT-micro-agent | Execution of rm -rf commands applied to suspicious locations detected using analysis of host data. Because rm -rf recursively deletes files, it's normally only used on discrete folders. In this case, it's being used in a location that could remove a large amount of data. Fairware ransomware is known to execute rm -rf commands in this folder. | Review with the user that ran the command this was legitimate activity that you expect to see on the device. If not, escalate the alert to the information security team. | IoT_FairwareMalware |

+| Crypto coin miner container image detected | Medium | Defender-IoT-micro-agent | Container detecting running known digital currency mining images. | 1. If this behavior isn't intended, delete the relevant container image. <br> 2. Make sure that the Docker daemon isn't accessible via an unsafe TCP socket. <br> 3. Escalate the alert to the information security team. | IoT_CryptoMinerContainer |

+1. Select on the device, and then select the **azureiotsecurity** module.

+1. Select **Module Identity Twin**.

+1. Add baseline properties to the Defender-IoT-micro-agent and select **Save**.

+| Custom alert - The number of direct method invokes are outside the allowed range | Low | IoT Hub | The amount of direct method invokes within a specific time window is outside the currently configured and allowable range. | IoT_CA_DirectMethodInvokesNotInAllowedRange |

+| Custom alert - The number of cloud to device messages in HTTP protocol is outside the allowed range | Low | IoT Hub | The amount of cloud to device messages (HTTP protocol) in a time window isn't in the configured allowed range | IoT_CA_HttpC2DMessagesNotInAllowedRange |

+| Custom alert - The number of rejected cloud to device messages in HTTP protocol isn't in the allowed range | Low | IoT Hub | The amount of cloud to device messages (HTTP protocol) within a specific time window is outside the currently configured and allowable range. | IoT_CA_HttpC2DRejectedMessagesNotInAllowedRange |

+| Custom alert - The number of module twin updates is outside the allowed range | Low | IoT Hub | The number of module twin updates within a specific time window is outside the currently configured and allowable range. | IoT_CA_TwinUpdatesNotInAllowedRange |

+| Custom alert - The number of unauthorized operations is outside the allowed range | Low | IoT Hub | The number of unauthorized operations within a specific time window is outside the currently configured and allowable range. | IoT_CA_UnauthorizedOperationsNotInAllowedRange |

+Microsoft Defender for IoT is a separate service, which adds an extra layer of threat protection to the Azure IoT Hub, IoT Edge, and your devices. Defender for IoT may process, and store your data within a different geographic location than your IoT Hub.

+| **nics** | The network interface controller. The full list of properties is listed below. |

+| **Devices** | A list of the network devices separated by a comma. <br><br>For example `eth0,eth1` | Defines the list of network devices (interfaces) that the agent will use to monitor the traffic. <br><br>If a network device isn't listed, the Network Raw events won't be recorded for the missing device.| `eth0` |

+In this tutorial you'll learn how to:

+1. Select a workspace from the drop-down menu. If you don't already have an existing Log Analytics workspace, you can select **Create New Workspace** to create a new one.

+description: In this tutorial, you'll learn how to create a DefenderIotMicroAgent module twin for new devices.

+In this tutorial you'll learn how to:

+In this tutorial you'll learn how to:

+In this tutorial you'll learn how to:

+In this tutorial you'll learn how to:

+- Verify you're running one of the following [operating systems](concept-agent-portfolio-overview-os-support.md#agent-portfolio-overview-and-os-support-preview).

+1. Ensure that you've updated the apt using the following command:

+1. Ensure that the service is stable by making sure it's `active`, and that the uptime of the process is appropriate.

+1. Ensure that you've upgraded the apt. Run:

+1. Ensure that you've upgraded the apt. Run:

+ Title: Learn about devices discovered by all sensors

+# Investigate all sensor detections in the device inventory

+Public internet IP addresses, multicast groups, and broadcast groups aren't considered inventory devices.

+## Integrate data into the device inventory

+Data integration capabilities let you enhance the data in the device inventory with information from other resources. These sources include CMDBs, DNS, firewalls, and Web APIs.

+To integrate data from other entities:

+The Device map provides a graphical representation of network devices detected, and the connections between them. Use the map to:

+A variety of map tools help you gain insight into devices and connections of interest to you.

+- Filtering omits the devices that aren't in the selected group.

+- Highlights display all devices and highlights the selected items in the group in blue.

+| **non-standard ports (default)** | Devices that use non-standard ports or ports that haven't been assigned an alias. |

+| **Not In Active Directory** | All non-PLC devices that aren't communicating with the Active Directory. |

+1. Specific connections between devices are displayed in blue. In addition, you'll see connections that cross various Purdue levels.

+1. Right-click the icon on the map that represents the IT network and select **Expand Network**.

+1. A confirmation box appears, notifying you that the layout change can't be redone.

+| :::image type="content" source="media/how-to-work-with-maps/not-authorized-v2.png" alt-text="Screenshot of the device learning period"::: | A device that was detected after the Learning period and wasn't authorized as a network device. |

+| Scanner or Programming device | **Scanner**: Enable this option if you know that this device is known as a scanner and there's no need to alert you about it. <br /> **Programming Device**: Enable this option if you know that this device is known as a programming device and is used to make programming changes. Identifying it as a programming device will prevent alerts for programming changes originating from this asset. |

+| Firmware | If Backplane information is available, firmware information won't be displayed. |

+You may want to delete a device if the information learned isn't relevant. For example,

+If you don't delete the device, the sensor will continue monitoring it. After 60 days, a notification will appear, recommending that you delete.

+You can't undo a device merge. If you mistakenly merged two devices, delete the device and wait for the sensor to rediscover both.

+During the Learning period, all the devices discovered in the network are identified as authorized devices. The **Authorized** label doesn't appear on these devices in the Device map.

+## Link your forwarding ruleset to the second virtual network

+To apply your forwarding ruleset to the second virtual network, you must create a virtual link.

+1. Search for **DNS forwarding rulesets** in the Azure services list and select your ruleset (ex: **myruleset**).

+2. Select **Virtual Network Links**, select **Add**, choose **myvnet2** and use the default Link Name **myvnet2-link**.

+3. Select **Add** and verify that the link was added successfully. You might need to refresh the page.

+ 

+## Configure a DNS forwarding ruleset

+Add or remove specific rules your DNS forwarding ruleset as desired, such as:

+- A rule to resolve an Azure Private DNS zone linked to your virtual network: azure.contoso.com.

+- A rule to resolve an on-premises zone: internal.contoso.com.

+- A wildcard rule to forward unmatched DNS queries to a protective DNS service.

+### Delete a rule from the forwarding ruleset

+Individual rules can be deleted or disabled. In this example, a rule is deleted.

+1. Search for **Dns Forwarding Rulesets** in the Azure Services list and select it.

+2. Select the ruleset you previously configured (ex: **myruleset**) and then select **Rules**.

+3. Select the **contosocom** sample rule that you previously configured, select **Delete**, and then select **OK**.

+### Add rules to the forwarding ruleset

+Add three new conditional forwarding rules to the ruleset.

+1. On the **myruleset | Rules** page, click **Add**, and enter the following rule data:

+ - Rule Name: **AzurePrivate**

+ - Domain Name: **azure.contoso.com.**

+ - Rule State: **Enabled**

+2. Under **Destination IP address** enter 10.0.0.4, and then click **Add**.

+3. On the **myruleset | Rules** page, click **Add**, and enter the following rule data:

+ - Rule Name: **Internal**

+ - Domain Name: **internal.contoso.com.**

+ - Rule State: **Enabled**

+4. Under **Destination IP address** enter 192.168.1.2, and then click **Add**.

+5. On the **myruleset | Rules** page, click **Add**, and enter the following rule data:

+ - Rule Name: **Wildcard**

+ - Domain Name: **.** (enter only a dot)

+ - Rule State: **Enabled**

+6. Under **Destination IP address** enter 10.5.5.5, and then click **Add**.

+ 

+In this example:

+- 10.0.0.4 is the resolver's inbound endpoint.

+- 192.168.1.2 is an on-premises DNS server.

+- 10.5.5.5 is a protective DNS service.

+- DNS zones in the public internet DNS namespace.

+Azure DNS Private Resolver is a new service currently in public preview. Azure DNS Private Resolver enables you to query Azure DNS private zones from an on-premises environment and vice versa without deploying VM based DNS servers. For more information, including benefits, capabilities, and regional availability, see [What is Azure DNS Private Resolver](dns-private-resolver-overview.md).

+Create an inbound endpoint to enable name resolution from on-premises or another private location using an IP address that is part of your private virtual network address space.

+Create a second virtual network to simulate an on-premises or other environment.

+## Create forwarding rules

+$targetDNS1 = New-AzDnsResolverTargetDnsServerObject -IPAddress 192.168.1.2 -Port 53

+$targetDNS2 = New-AzDnsResolverTargetDnsServerObject -IPAddress 192.168.1.3 -Port 53

+$targetDNS3 = New-AzDnsResolverTargetDnsServerObject -IPAddress 10.0.0.4 -Port 53

+$targetDNS4 = New-AzDnsResolverTargetDnsServerObject -IPAddress 10.5.5.5 -Port 53

+$forwardingrule = New-AzDnsForwardingRulesetForwardingRule -ResourceGroupName myresourcegroup -DnsForwardingRulesetName myruleset -Name "Internal" -DomainName "internal.contoso.com." -ForwardingRuleState "Enabled" -TargetDnsServer @($targetDNS1,$targetDNS2)

+$forwardingrule = New-AzDnsForwardingRulesetForwardingRule -ResourceGroupName myresourcegroup -DnsForwardingRulesetName myruleset -Name "AzurePrivate" -DomainName "." -ForwardingRuleState "Enabled" -TargetDnsServer $targetDNS3

+$forwardingrule = New-AzDnsForwardingRulesetForwardingRule -ResourceGroupName myresourcegroup -DnsForwardingRulesetName myruleset -Name "Wildcard" -DomainName "." -ForwardingRuleState "Enabled" -TargetDnsServer $targetDNS4

+In this example:

+- 10.0.0.4 is the resolver's inbound endpoint.

+- 192.168.1.2 and 192.168.1.3 are on-premises DNS servers.

+- 10.5.5.5 is a protective DNS service.

+- Private DNS zones that are hosted on-premises.

+- An outbound endpoint can't be deleted unless the DNS forwarding ruleset and the virtual network links under it are deleted.

+- IPv6 enabled subnets aren't supported in Public Preview.

+- Currently, rulesets can't be linked across different subscriptions.

+* 25 advanced filters and 25 filter values across all the filters per event grid subscription

+- [Configure WAF policies using Azure Firewall Manager (preview)](../web-application-firewall/shared/manage-policies.md)

+> [!IMPORTANT]

+> If you deploy a Secured Virtual Hub in forced tunnel mode, advertising the default route over Express Route or VPN Gateway is not currently supported. A fix is being investigated.

+The cluster, SQL database, and other objects are created through the Azure portal using an Azure Resource Manager template. The template can be found in [Azure quickstart templates](https://azure.microsoft.com/resources/templates/hdinsight-linux-with-sql-database/). The Resource Manager template calls a bacpac package to deploy the table schemas to a SQL database. If you want to use a private container for the bacpac files, use the following values in the template:

+To use a template that creates two virtual networks in two different regions and the VPN connection between the VNets, select the following **Deploy to Azure** button.

+ To learn how to write this Resource Manager template, see [MSDN: Install an HDInsight application](/rest/api/hdinsight/hdinsight-application).

+* The PowerShell [Az Module](/powershell/azure/install-az-ps) installed.

+5. Copies the sample HiveQL script (**partitionweblogs.hql**) the Blob container. The sample script is already available in another public Blob container. The PowerShell script below makes a copy of these files into the Azure Storage account it creates.

+[Connect an on-premises network to Azure](/azure/architecture/reference-architectures/hybrid-networking)

+Automated key rotation in [Key Vault](../general/overview.md) allows users to configure Key Vault to automatically generate a new key version at a specified frequency. For more information about how keys are versioned, see [Key Vault objects, identifiers, and versioning](../general/about-keys-secrets-certificates.md#objects-identifiers-and-versioning).

+You can use rotation policy to configure rotation for each individual key. Our recommendation is to rotate encryption keys at least every two years to meet cryptographic best practices.

+This feature enables end-to-end zero-touch rotation for encryption at rest for Azure services with customer-managed key (CMK) stored in Azure Key Vault. Please refer to specific Azure service documentation to see if the service covers end-to-end rotation.

+For more information about data encryption in Azure, see:

+- [Azure Encryption at Rest](../../security/fundamentals/encryption-atrest.md#azure-encryption-at-rest-components)

+- [Azure services data encryption support table](../../security/fundamentals/encryption-models.md#supporting-services)

+> [!IMPORTANT]

+> Key rotation generates a new key version of an existing key with new key material. Ensure that your data encryption solution uses versioned key uri to point to the same key material for encrypt/decrypt, wrap/unwrap operations to avoid disruption to your services. All Azure services are currently following that pattern for data encryption.

+Set rotation policy on a key passing previously saved file using Azure CLI [az keyvault key rotation-policy update](/cli/azure/keyvault/key/rotation-policy) command.

+Use Azure CLI [az keyvault key rotate](/cli/azure/keyvault/key#az-keyvault-key-rotate) command to rotate key.

+ Title: Set up a networking lab with GNS3

+ Title: Connect to Azure Lab Services VMs from Mac

+ For example, when you use the `replace` filter, use `Replace`, not `replace`. The same rule applies if you try out examples at [DotLiquid online](http://dotliquidmarkup.org/TryOnline). For more information, see [Shopify Liquid filters](https://shopify.dev/docs/themes/liquid/reference/filters) and [DotLiquid Liquid filters](https://github.com/dotliquid/dotliquid/wiki/DotLiquid-for-Developers#create-your-own-filters). The Shopify specification includes examples for each filter, so for comparison, you can try these examples at [DotLiquid - Try online](http://dotliquidmarkup.org/TryOnline).

+* [DotLiquid - Try online](http://dotliquidmarkup.org/TryOnline)

+ Title: Autoscale online endpoints

+description: Learn to scale up online endpoints. Get more CPU, memory, disk space, and extra features.

+# Autoscale an online endpoint

+Autoscale automatically runs the right amount of resources to handle the load on your application. [Online endpoints](concept-endpoints.md) supports autoscaling through integration with the Azure Monitor autoscale feature.

+* A deployed endpoint. [Deploy and score a machine learning model by using an online endpoint](how-to-deploy-managed-online-endpoints.md).

+ Title: Monitor online endpoints

+description: Monitor online endpoints and create alerts with Application Insights.

+# Monitor online endpoints

+In this article, you learn how to monitor [Azure Machine Learning online endpoints](concept-endpoints.md). Use Application Insights to view metrics and create alerts to stay up to date with your online endpoints.

+> * View metrics for your online endpoint

+- Deploy an Azure Machine Learning online endpoint.

+1. Navigate to the online endpoint or deployment resource.

+ online endpoints and deployments are Azure Resource Manager (ARM) resources that can be found by going to their owning resource group. Look for the resource types **Machine Learning online endpoint** and **Machine Learning online deployment**.

+Depending on the resource that you select, the metrics that you see will be different. Metrics are scoped differently for online endpoints and online deployments.

+Bandwidth will be throttled if the limits are exceeded for _managed_ online endpoints (see managed online endpoints section in [Manage and increase quotas for resources with Azure Machine Learning](how-to-manage-quotas.md#azure-machine-learning-managed-online-endpoints)). To determine if requests are throttled:

+You can create custom dashboards to visualize data from multiple sources in the Azure portal, including the metrics for your online endpoint. For more information, see [Create custom KPI dashboards using Application Insights](../azure-monitor/app/tutorial-app-dashboards.md#add-custom-metric-chart).

+You can also create custom alerts to notify you of important status updates to your online endpoint:

+ Check this [example](https://github.com/Azure/azureml-examples/blob/main/sdk/assets/environment/environment.ipynb) on how to create custom environments.

+ Title: Managed online endpoints VM SKU list

+description: Lists the VM SKUs that can be used for managed online endpoints in Azure Machine Learning.

+# Managed online endpoints SKU list

+This table shows the VM SKUs that are supported for Azure Machine Learning managed online endpoints.

+ > However, usually When you score the data, you won't know the target variable values. For such case, you can remove the target variable column in the inference pipeline using **Select Columns in Dataset** component. Make sure that the output of **Select Columns in Dataset** removing target variable column is connected to the same port as the output of the **Web Service Input** component.

+* [Option A: Train and deploy models by using Jupyter Notebooks](tutorial-power-bi-custom-model.md). This code-first authoring experience uses Jupyter Notebooks that are hosted in Azure Machine Learning studio.

+Sign in to [Azure Machine Learning studio](https://ml.azure.com). In the menu on the left, select **Compute** and then **New**:

+To create the dataset, in the menu on the left, select **Data**. Then select **Create**. You see the following options:

+After you create the compute and datasets, you can use the designer to create the machine learning model. In Azure Machine Learning studio, select **Designer** and then **New pipeline**:

+> Although you may add up to 100 authorizations per Azure region, we recommend you create an Active Directory user group and specify its ID in the "Principal ID." This lets you add more users to the management group after the plan is deployed and reduces the need to update the plan just to add more authorizations.

+> Single Server to Flexible Server migration tool is in public preview.

+Azure Database for PostgreSQL Flexible Server provides zone redundant high availability, control over price, and control over maintenance window. Single to Flexible Server Migration tool enables customers to migrate their databases from Single server to Flexible. See this [documentation](../flexible-server/concepts-compare-single-server-flexible-server.md) to understand the differences between Single and Flexible servers. Customers can initiate migrations for multiple servers and databases in a repeatable fashion using this migration tool. This tool automates most of the steps needed to do the migration and thus making the migration journey across Azure platforms as seamless as possible. The tool is provided free of cost for customers.

+Single to Flexible server migration tool provides an inline experience to migrate databases from Single Server (source) to Flexible Server (target).

+You choose the source server and can select up to **8** databases from it. This limitation is per migration task. The migration tool automates the following steps:

+Following is the flow diagram for Single to Flexible migration tool.

+The migration tool is exposed through **Azure Portal** and via easy-to-use **Azure CLI** commands. It allows you to create migrations, list migrations, display migration details, modify the state of the migration, and delete migrations

+Follow the steps provided in this section before you get started with the single to flexible server migration tool.

+- **Target Server Creation** - You need to create the target PostgreSQL flexible server before using the migration tool. Use the creation [QuickStart guide](../flexible-server/quickstart-create-server-portal.md) to create one.

+- **Azure Active Directory App set up** - It is a critical component of the migration tool. Azure AD App helps with role-based access control as the migration tool needs access to both the source and target servers. See [How to setup and configure Azure AD App](./how-to-setup-azure-ad-app-portal.md) for step-by-step process.

+- Databases of sizes up to 1TB can be migrated using this tool. To migrate larger databases or heavy write workloads, reach out to your account team or reach us @ AskAzureDBforPGS2F@microsoft.com.

+- Single to Flexible Server migration tool uses logical decoding feature of PostgreSQL to perform the online migration and it comes with the following limitations. See PostgreSQL documentation for [logical replication limitations](https://www.postgresql.org/docs/10/logical-replication-restrictions.html).

+- The migration tool migrates only data and schema of the single server databases to flexible server. It does not migrate other features such as server parameters, connection security details, firewall rules, users, roles and permissions. In other words, everything except data and schema must be manually configured in the target flexible server.

+- For failed migrations, there is no option to retry the same migration task. A new migration task with a unique name has to be created.

+- The migration tool does not include assessment of your single server.

+> Single Server to Flexible Server migration tool is in public preview.

+This quick start article shows you how to use Single to Flexible Server migration tool to migrate databases from Azure database for PostgreSQL Single server to Flexible server.

+4. Take care of the pre-requisites listed in this [**document**](./concepts-single-to-flexible.md#pre-requisites) which are necessary to get started with the Single to Flexible migration tool.

+Single to Flexible Server migration tool comes with a list of easy-to-use CLI commands to do migration-related tasks. All the CLI commands start with **az postgres flexible-server migration**. You can use the **help** parameter to help you with understanding the various options associated with a command and in framing the right syntax for the same.

+The migration tool offers online and offline mode of migration. To know more about the migration modes and their differences, visit this [link](./concepts-single-to-flexible.md)

+| **MigrationResourceGroup** | optional | This section consists of two properties. <br> **ResourceID (optional)** : The migration infrastructure and other network infrastructure components are created to migrate data and schema from the source to target. By default, all the components created by this tool are provisioned under the resource group of the target server. If you wish to deploy them under a different resource group, then you can assign the resource ID of that resource group to this property. <br> **SubnetResourceID (optional)** : In case if your source has public access turned OFF or if your target server is deployed inside a VNet, then specify a subnet under which migration infrastructure needs to be created so that it can connect to both source and target servers. |

+| **OverwriteDBsinTarget** | Optional | If the target server happens to have an existing database with the same name as the one you are trying to migrate, the migration will pause until you acknowledge that overwrites in the target DBs are allowed. This pause can be avoided by giving the migration tool, permission to automatically overwrite databases by setting the value of this property to **true** |

+This guide shows you how to use Single to Flexible server migration tool to migrate databases from Azure database for PostgreSQL Single server to Flexible server.

+Take care of the [pre-requisites](./concepts-single-to-flexible.md#pre-requisites) to get started with the migration tool.

+Single to Flexible server migration tool comes with a simple, wizard-based portal experience. Let us get started to know the steps needed to consume the tool from portal.

+4. Click the **Migrate from Single Server** button to start a migration from Single Server to Flexible Server. If this is the first time you are using the migration tool, you will see an empty grid with a prompt to begin your first migration.

+- The **Migration resource group** is where all the migration-related components will be created by this migration tool.

+Once the single server is chosen, the fields such as **Location, PostgreSQL version, Server admin login name** are automatically pre-populated. The server admin login name is the admin username that was used to create the single server. Enter the password for the **server admin login name**. This is required for the migration tool to login into the single server to initiate the dump and migration.

+The final property in the source tab is migration mode. The migration tool offers online and offline mode of migration. The concepts page talks more about the [migration modes and their differences](./concepts-single-to-flexible.md).

+This tab displays metadata of the flexible server like the **Subscription**, **Resource Group**, **Server name**, **Location**, and **PostgreSQL version**. It displays **server admin login name** which is the admin username that was used during the creation of the flexible server.Enter the corresponding password for the admin user. This is required for the migration tool to login into the flexible server to perform restore operations.

+All the fields will be automatically populated with subnet details. This is the subnet in which the migration tool will deploy Azure DMS to move data between the source and target.

+- [Single Server](./overview-single-server.md)

+- [Flexible Server](../flexible-server/overview.md)

+- [Hyperscale (Citus)](../hyperscale/overview.md)

+ Title: Configure an application security group with a private endpoint

+description: Learn how to create a private endpoint with an Application Security Group or apply an ASG to an existing private endpoint.

+# Configure an application security group (ASG) with a private endpoint

+Azure Private endpoints support application security groups for network security. Private endpoints can be associated with an existing ASG in your current infrastructure along side virtual machines and other network resources.

+## Prerequisites

+- An Azure account with an active subscription. If you don't already have an Azure account, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+- An Azure web app with a *PremiumV2-tier* or higher app service plan, deployed in your Azure subscription.

+ - For more information and an example, see [Quickstart: Create an ASP.NET Core web app in Azure](../app-service/quickstart-dotnetcore.md).

+

+ - The example webapp in this article is named **myWebApp1979**. Replace the example with your webapp name.

+- An existing Application Security Group in your subscription. For more information about ASGs, see [Application security groups](../virtual-network/application-security-groups.md).

+

+ - The example ASG used in this article is named **myASG**. Replace the example with your application security group.

+- An existing Azure Virtual Network and subnet in your subscription. For more information about creating a virtual network, see [Quickstart: Create a virtual network using the Azure portal](../virtual-network/quick-create-portal.md).

+ - The example virtual network used in this article is named **myVNet**. Replace the example with your virtual network.

+## Create private endpoint with an ASG

+An ASG can be associated with a private endpoint when it's created. The following procedures demonstrate how to associate an ASG with a private endpoint when it's created.

+1. Sign-in to the [Azure portal](https://portal.azure.com).

+2. In the search box at the top of the portal, enter **Private endpoint**. Select **Private endpoints** in the search results.

+3. Select **+ Create** in **Private endpoints**.

+4. In the **Basics** tab of **Create a private endpoint**, enter or select the following information.

+ | Value | Setting |

+ | -- | - |

+ | **Project details** | |

+ | Subscription | Select your subscription. |

+ | Resource group | Select your resource group. </br> In this example, it's **myResourceGroup**. |

+ | **Instance details** | |

+ | Name | Enter **myPrivateEndpoint**. |

+ | Region | Select **East US**. |

+5. Select **Next: Resource** at the bottom of the page.

+6. In the **Resource** tab, enter or select the following information.

+ | Value | Setting |

+ | -- | - |

+ | Connection method | Select **Connect to an Azure resource in my directory.** |

+ | Subscription | Select your subscription |

+ | Resource type | Select **Microsoft.Web/sites**. |

+ | Resource | Select **mywebapp1979**. |

+ | Target subresource | Select **sites**. |

+7. Select **Next: Virtual Network** at the bottom of the page.

+8. In the **Virtual Network** tab, enter or select the following information.

+ | Value | Setting |

+ | -- | - |

+ | **Networking** | |

+ | Virtual network | Select **myVNet**. |

+ | Subnet | Select your subnet. </br> In this example, it's **myVNet/myBackendSubnet(10.0.0.0/24)**. |

+ | Enable network policies for all private endpoints in this subnet. | Leave the default of checked. |

+ | **Application security group** | |

+ | Application security group | Select **myASG**. |

+ :::image type="content" source="./media/configure-asg-private-endpoint/asg-new-endpoint.png" alt-text="Screenshot of ASG selection when creating a new private endpoint.":::

+9. Select **Next: DNS** at the bottom of the page.

+10. Select **Next: Tags** at the bottom of the page.

+11. Select **Next: Review + create**.

+12. Select **Create**.

+## Associate an ASG with an existing private endpoint

+An ASG can be associated with an existing private endpoint. The following procedures demonstrate how to associate an ASG with an existing private endpoint.

+> [!IMPORTANT]

+> You must have a previously deployed private endpoint to proceed with the steps in this section. The example endpoint used in this section is named **myPrivateEndpoint**. Replace the example with your private endpoint.

+1. Sign-in to the [Azure portal](https://portal.azure.com).

+2. In the search box at the top of the portal, enter **Private endpoint**. Select **Private endpoints** in the search results.

+3. In **Private endpoints**, select **myPrivateEndpoint**.

+4. In **myPrivateEndpoint**, in **Settings**, select **Application security groups**.

+5. In **Application security groups**, select **myASG** in the pull-down box.

+ :::image type="content" source="./media/configure-asg-private-endpoint/asg-existing-endpoint.png" alt-text="Screenshot of ASG selection when associating with an existing private endpoint.":::

+6. Select **Save**.

+## Next steps

+For more information about Azure Private Link, see:

+- [What is Azure Private Link?](private-link-overview.md)

+Annotations that classify and protect an organizationΓÇÖs data. Microsoft Purview integrates with Microsoft Purview Information Protection for creation of sensitivity labels.

+* Stored procedures running remotely from data integration tools like Azure Data Factory is currently not supported

+- [Search Data Catalog](how-to-search-catalog.md)

+- Empty workspaces are skipped.

+1. Make sure Power BI tenant ID is entered correctly during the registration. By default, Power BI tenant ID that exists in the same Azure Active Directory as Microsoft Purview will be populated.

+1. Make sure your [PowerBI Metadata model is up to date by enabling metadata scanning.](/power-bi/admin/service-admin-metadata-scanning-setup#enable-tenant-settings-for-metadata-scanning)

+1. From Azure portal, validate if Microsoft Purview account Network is set to public access.

+1. From Power BI tenant Admin Portal, make sure Power BI tenant is configured to allow public network.

+1. Check your Azure Key Vault to make sure:

+1. Review your credential to validate:

+1. In Power BI Azure AD tenant, validate Power BI admin user settings to make sure:

+1. In Power BI Azure AD tenant, validate App registration settings to make sure:

+1. Make sure Power BI tenant ID is entered correctly during the registration.By default, Power BI tenant ID that exists in the same Azure Active Directory as Microsoft Purview will be populated.

+1. Make sure your [PowerBI Metadata model is up to date by enabling metadata scanning.](/power-bi/admin/service-admin-metadata-scanning-setup#enable-tenant-settings-for-metadata-scanning)

+1. From Azure portal, validate if Microsoft Purview account Network is set to public access.

+1. From Power BI tenant Admin Portal, make sure Power BI tenant is configured to allow public network.

+1. Check your Azure Key Vault to make sure:

+1. Review your credential to validate:

+1. In Power BI Azure AD tenant, validate Power BI admin user settings to make sure:

+1. In Power BI Azure AD tenant, validate App registration settings to make sure:

+1. Validate Self-hosted runtime settings:

+## My schema is not showing up after scanning

+It can take some time for schema to finish the scanning and ingestion process, depending on the size of your Power BI Tenant. Currently if you have a large PowerBI tenant, this process could take a few hours.

+- **Recommendation**: Disable multi-factor authentication requirement and exclude user from conditional access policies. Login with the user to Power BI dashboard to validate if user can successfully login to the application.

+- Empty workspaces are skipped.

+1. Make sure Power BI tenant ID is entered correctly during the registration.

+1. Make sure your [PowerBI Metadata model is up to date by enabling metadata scanning.](/power-bi/admin/service-admin-metadata-scanning-setup#enable-tenant-settings-for-metadata-scanning)

+1. From Azure portal, validate if Microsoft Purview account Network is set to public access.

+1. From Power BI tenant Admin Portal, make sure Power BI tenant is configured to allow public network.

+1. In Azure Active Directory tenant, create a security group.

+1. From Azure Active Directory tenant, make sure [Microsoft Purview account MSI is member of the new security group](#authenticate-to-power-bi-tenant-managed-identity-only).

+1. On the Power BI Tenant Admin portal, validate if [Allow service principals to use read-only Power BI admin APIs](#associate-the-security-group-with-power-bi-tenant) is enabled for the new security group.

+1. Make sure Power BI tenant ID is entered correctly during the registration.

+1. Make sure your [PowerBI Metadata model is up to date by enabling metadata scanning.](/power-bi/admin/service-admin-metadata-scanning-setup#enable-tenant-settings-for-metadata-scanning)

+1. From Azure portal, validate if Microsoft Purview account Network is set to public access.

+1. From Power BI tenant Admin Portal, make sure Power BI tenant is configured to allow public network.

+1. Check your Azure Key Vault to make sure:

+1. Review your credential to validate:

+1. Validate Power BI admin user settings to make sure:

+1. Validate App registration settings to make sure:

+1. Validate Self-hosted runtime settings:

+1. Make sure Power BI tenant ID is entered correctly during the registration.

+1. Make sure your [PowerBI Metadata model is up to date by enabling metadata scanning.](/power-bi/admin/service-admin-metadata-scanning-setup#enable-tenant-settings-for-metadata-scanning)

+1. Check your Azure Key Vault to make sure:

+1. Review your credential to validate:

+1. Validate Power BI admin user to make sure:

+1. Validate Self-hosted runtime settings:

+1. Validate App registration settings to make sure:

+1. Review network configuration and validate if:

+If you haven't already, step through [Quickstart: Create a skillset for AI enrichment](cognitive-search-quickstart-blob.md) for an introduction to enrichment of blob data.

+ Title: "Quickstart: Create a skillset in the Azure portal"

+description: In this portal quickstart, use the Import Data wizard to add cognitive skills to an indexing pipeline to generate searchable text from images and unstructured documents. Skills in this quickstart include OCR, image analysis, and natural language processing.

+# Quickstart: Create an Azure Cognitive Search skillset in the Azure portal

+Learn how AI enrichment in Azure Cognitive Search adds Optical Character Recognition (OCR), image analysis, language detection, text translation, and entity recognition to create searchable content in a search index.

+In this quickstart, you'll run the **Import data** wizard to apply skills that transform and enrich content during indexing. Output is a searchable index containing image text, translated text, and entities. Enriched content is queryable in the portal using [Search explorer](search-explorer.md).

+1. [Download sample data](https://1drv.ms/f/s!As7Oy81M_gVPa-LCb5lC_3hbS-4) consisting of a small file set of different types. Unzip the files.

+1. Sign in to the [Azure portal](https://portal.azure.com/) with your Azure account.

+1. [Create an Azure Storage account](../storage/common/storage-account-create.md?tabs=azure-portal) or [find an existing account](https://ms.portal.azure.com/#blade/HubsExtension/BrowseResourceBlade/resourceType/Microsoft.Storage%2storageAccounts/).

+ + Choose the same region as Azure Cognitive Search to avoid bandwidth charges.

+ + Choose the StorageV2 (general purpose V2).

+1. In Azure portal, open your Azure Storage page and create a container. You can use the default public access level.

+1. In Container, select **Upload** to upload the sample files you downloaded in the first step. Notice that you have a wide range of content types, including images and application files that are not full text searchable in their native formats.

+ :::image type="content" source="media/cognitive-search-quickstart-blob/sample-data.png" alt-text="Screenshot of source files in Azure Blob Storage." border="false":::

+1. [Find your search service](https://ms.portal.azure.com/#blade/HubsExtension/BrowseResourceBlade/resourceType/Microsoft.Storage%2storageAccounts/) and on the Overview page, select **Import data** on the command bar to set up cognitive enrichment in four steps.

+ :::image type="content" source="medi.png" alt-text="Screenshot of the Import data command." border="true":::

+1. In **Connect to your data**, choose **Azure Blob Storage**.

+1. Choose an existing connection to the storage account and select the container you created. Give the data source a name, and use default values for the rest.

+ :::image type="content" source="media/cognitive-search-quickstart-blob/blob-datasource.png" alt-text="Screenshot of the data source definition page." border="true":::

+ Continue to the next page.

+Next, configure AI enrichment to invoke OCR, image analysis, and natural language processing.

+1. For this quickstart, we are using the **Free** Cognitive Services resource. The sample data consists of 14 files, so the free allotment of 20 transaction on Cognitive Services is sufficient for this quickstart.

+ :::image type="content" source="media/cognitive-search-quickstart-blob/cog-search-attach.png" alt-text="Screenshot of the Attach Cognitive Services tab." border="true":::

+1. Expand **Add enrichments** and make six selections.

+ Enable OCR to add image analysis skills to wizard page.

+ Choose entity recognition (people, organizations, locations) and image analysis skills (tags, captions).

+ :::image type="content" source="media/cognitive-search-quickstart-blob/skillset.png" alt-text="Screenshot of the skillset definition page." border="true":::

+ Continue to the next page.

+An index contains your searchable content and the **Import data** wizard can usually create the schema for you by sampling the data source. In this step, review the generated schema and potentially revise any settings. Below is the default schema created for the demo Blob data set.

+ :::image type="content" source="media/cognitive-search-quickstart-blob/index-fields.png" alt-text="Screenshot of the index definition page." border="true":::

+Marking a field as **Retrievable** does not mean that the field *must* be present in the search results. You can control search results composition by using the **$select** query parameter to specify which fields to include.

+

+Continue to the next page.

+The indexer drives the indexing process. It specifies the data source name, a target index, and frequency of execution. The **Import data** wizard creates several objects, including an indexer that you can reset and run repeatedly.

+1. In the **Indexer** page, you can accept the default name and select **Once** to run it immediately.

+ :::image type="content" source="media/cognitive-search-quickstart-blob/indexer-def.png" alt-text="Screenshot of the indexer definition page." border="true":::

+Cognitive skills indexing takes longer to complete than typical text-based indexing, especially OCR and image analysis. To monitor progress, go to the Overview page and select **Indexers** in the middle of page.

+ :::image type="content" source="media/cognitive-search-quickstart-blob/indexer-notification.png" alt-text="Screenshot of the indexer status page." border="true":::

+To check details about execution status, select an indexer from the list, and then select **Success** (or **Failed**) to view execution details.

+In this demo, there is one warning. It tells you that a PNG file in the data source doesn't provide a text input to Entity Recognition. This warning occurs because the upstream OCR skill didn't recognize any text in the image, and thus could not provide a text input to the downstream Entity Recognition skill.

+Warnings are common in skillset execution. As you become familiar with how skills iterate over your data, you'll begin to notice patterns and learn which warnings are safe to ignore.

+After an index is created, run queries in **Search explorer** to return results.

+1. On the search service dashboard page, select **Search explorer** on the command bar.

+1. Enter a search string to query the index, such as `search=Satya Nadella&$select=people,organizations,locations&$count=true`.

+Results are returned as verbose JSON, which can be hard to read, especially in large documents. Some tips for searching in this tool include the following techniques:

+Query strings are case-sensitive so if you get an "unknown field" message, check **Fields** or **Index Definition (JSON)** to verify name and case.

+ :::image type="content" source="media/cognitive-search-quickstart-blob/search-explorer.png" alt-text="Screenshot of the the Search explorer page." border="true":::

+## Takeaways

+You've now created your first skillset and learned important concepts useful for prototyping an enriched search solution using your own data.

+Some key concepts that we hope you picked up include the dependency on Azure data sources. A skillset is bound to an indexer, and indexers are Azure and source-specific. Although this quickstart uses Azure Blob Storage, other Azure data sources are possible. For more information, see [Indexers in Azure Cognitive Search](search-indexer-overview.md).

+Another important concept is that skills operate over content types, and when working with heterogeneous content, some inputs will be skipped. Also, large files or fields might exceed the indexer limits of your service tier. It's normal to see warnings when these events occur.

+Output is directed to a search index, and there is a mapping between name-value pairs created during indexing and individual fields in your index. Internally, the portal sets up [annotations](cognitive-search-concept-annotations-syntax.md) and defines a [skillset](cognitive-search-defining-skillset.md), establishing the order of operations and general flow. These steps are hidden in the portal, but when you start writing code, these concepts become important.

+Finally, you learned that can verify content by querying the index. In the end, what Azure Cognitive Search provides is a searchable index, which you can query using either the [simple](/rest/api/searchservice/simple-query-syntax-in-azure-search) or [fully extended query syntax](/rest/api/searchservice/lucene-query-syntax-in-azure-search). An index containing enriched fields is like any other. If you want to incorporate standard or [custom analyzers](search-analyzers.md), [scoring profiles](/rest/api/searchservice/add-scoring-profiles-to-a-search-index), [synonyms](search-synonyms.md), [faceted navigation](search-faceted-navigation.md), geo-search, or any other Azure Cognitive Search feature, you can certainly do so.

+If you are using a free service, remember that you are limited to three indexes, indexers, and data sources. You can delete individual items in the portal to stay under the limit.

+You can create skillsets using the portal, .NET SDK, or REST API. To further your knowledge, try the REST API using Postman and more sample data.

+> [Tutorial: Extract text and structure from JSON blobs using REST APIs ](cognitive-search-tutorial-blob.md)

+Although you won't use the options in this quickstart, the wizard includes a page for AI enrichment so that you can extract text and structure from image files and unstructured text. For a similar walkthrough that includes AI enrichment, see + [Quickstart: Create a skillset](cognitive-search-quickstart-blob.md).

+### Create an S3HD service

+To create a [S3HD](./search-sku-tier.md#tier-descriptions) service, a combination of `-Sku` and `-HostingMode` is used. Set `-Sku` to `Standard3` and `-HostingMode` to `HighDensity`.

+```azurepowershell-interactive

+New-AzSearchService -ResourceGroupName <resource-group-name> `

+ -Name <search-service-name> `

+ -Sku Standard3 `

+ -Location "West US" `

+ -PartitionCount 1 -ReplicaCount 3 `

+ -HostingMode HighDensity

+```

+- **Get help from inside Microsoft products**, including the Microsoft 365 Defender portal, Microsoft Purview compliance portal, and Office 365 Security & Compliance Center by selecting the **Help** (**?**) button in the top navigation bar.

+6. Provide sudo privileges to current user by making an entry `<USERNAME\> ALL = (ALL) NOPASSWD:ALL` in /etc/sudoers

+* Learn about [Service Fabric support options](service-fabric-support.md)

+1. Navigate to your storage account. Under **Security + networking**, select **Security**.

+ :::image type="content" source="media/azure-defender-storage-configure/enable-azure-defender-portal.png" alt-text="Screenshot showing how to enable a storage account for Microsoft Defender for Storage.":::

+For more information on proactive recall, see [Deploy Azure File Sync](file-sync-deployment-guide.md#optional-proactively-recall-new-and-changed-files-from-an-azure-file-share).

+4. **Optional**: If you intend to use Azure File Sync with a Windows Server Failover Cluster, the **File Server for general use** role must be configured prior to installing the Azure File Sync agent on each node in the cluster. For more information on how to configure the **File Server for general use** role on a Failover Cluster, see [Deploying a two-node clustered file server](/windows-server/failover-clustering/deploy-two-node-clustered-file-server).

+4. **Optional**: If you intend to use Azure File Sync with a Windows Server Failover Cluster, the **File Server for general use** role must be configured prior to installing the Azure File Sync agent on each node in the cluster. For more information on how to configure the **File Server for general use** role on a Failover Cluster, see [Deploying a two-node clustered file server](/windows-server/failover-clustering/deploy-two-node-clustered-file-server).

+4. **Optional**: If you intend to use Azure File Sync with a Windows Server Failover Cluster, the **File Server for general use** role must be configured prior to installing the Azure File Sync agent on each node in the cluster. For more information on how to configure the **File Server for general use** role on a Failover Cluster, see [Deploying a two-node clustered file server](/windows-server/failover-clustering/deploy-two-node-clustered-file-server).

+## Optional: Configure firewall and virtual network settings

+If you'd like to configure Azure File Sync to work with firewall and virtual network settings, do the following:

+## Optional: Self-service restore through Previous Versions and VSS (Volume Shadow Copy Service)

+1. It configures Azure File Sync's cloud tiering on the specified volume to be compatible with previous versions and guarantees that a file can be restored from a previous version, even if it was tiered to the cloud on the server.

+VSS snapshots by default can consume up to 10% of the volume space. To adjust the amount of storage that can be used for VSS snapshots, use the [vssadmin resize shadowstorage](/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/cc788050(v=ws.11)) command.

+## Optional: Proactively recall new and changed files from an Azure file share

+Azure File Sync has a mode that allows globally distributed companies to have the server cache in a remote region pre-populated even before local users are accessing any files. When enabled on a server endpoint, this mode will cause this server to recall files that have been created or changed in the Azure file share.

+## Optional: SMB over QUIC on a server endpoint

+Although the Azure file share (cloud endpoint) is a full SMB endpoint capable of direct access from the cloud or on-premises, customers that desire accessing the file share data cloud-side often deploy an Azure File Sync server endpoint on a Windows Server instance hosted on an Azure VM. The most common reason to have an additional server endpoint rather than accessing the Azure file share directly is that changes made directly on the Azure file share may take up to 24 hours or longer to be discovered by Azure File Sync, while changes made on a server endpoint are discovered nearly immediately and synced to all other server and cloud-endpoints.

+This configuration is extremely common in environments where a substantial portion of users are not on-premises, such as when users are working from home or from the road. Traditionally, accessing any file share with SMB over the public internet, including both file shares hosted on Windows File Server or on Azure Files directly, is very difficult since most organizations and ISPs block port 445. You can work around this limitation with [private endpoints and VPNs](file-sync-networking-overview.md#private-endpoints), however Windows Server 2022 Azure Edition provides an additional access strategy: SMB over the QUIC transport protocol.

+SMB over QUIC communicates over port 443, which most organizations and ISPs have open to support HTTPS traffic. Using SMB over QUIC greatly simplifies the networking required to access a file share hosted on an Azure File Sync server endpoint for clients using Windows 11 or greater. To learn more about how to setup and configure SMB over QUIC on Windows Server Azure Edition, see [SMB over QUIC for Windows File Server](/windows-server/storage/file-server/smb-over-quic).

+## Onboarding with Azure File Sync

+The recommended steps to onboard on Azure File Sync for the first time with zero downtime while preserving full file fidelity and access control list (ACL) are as follows:

+1. Deploy a Storage Sync Service.

+1. Create a sync group.

+1. Install Azure File Sync agent on the server with the full data set.

+1. Register that server and create a server endpoint on the share.

+1. Let sync do the full upload to the Azure file share (cloud endpoint).

+1. After the initial upload is complete, install Azure File Sync agent on each of the remaining servers.

+1. Create new file shares on each of the remaining servers.

+1. Create server endpoints on new file shares with cloud tiering policy, if desired. (This step requires additional storage to be available for the initial setup.)

+1. Let Azure File Sync agent do a rapid restore of the full namespace without the actual data transfer. After the full namespace sync, sync engine will fill the local disk space based on the cloud tiering policy for the server endpoint.

+1. Ensure sync completes and test your topology as desired.

+1. Redirect users and applications to this new share.

+1. You can optionally delete any duplicate shares on the servers.

+If you don't have extra storage for initial onboarding and would like to attach to the existing shares, you can pre-seed the data in the Azure files shares. This approach is suggested, if and only if you can accept downtime and absolutely guarantee no data changes on the server shares during the initial onboarding process.

+1. Ensure that data on any of the servers can't change during the onboarding process.

+1. Pre-seed Azure file shares with the server data using any data transfer tool over the SMB. Robocopy, for example. You can also use AzCopy over REST. Be sure to use AzCopy with the appropriate switches to preserve ACLs timestamps and attributes.

+1. Create Azure File Sync topology with the desired server endpoints pointing to the existing shares.

+1. Let sync finish reconciliation process on all endpoints.

+1. Once reconciliation is complete, you can open shares for changes.

+Currently, pre-seeding approach has a few limitations -

+- Data changes on the server before the sync topology is fully up and running can cause conflicts on the server endpoints.

+- After the cloud endpoint is created, Azure File Sync runs a process to detect the files in the cloud before starting the initial sync. The time taken to complete this process varies depending on the various factors like network speed, available bandwidth, and number of files and folders. For the rough estimation in the preview release, detection process runs approximately at 10 files/sec. Hence, even if pre-seeding runs fast, the overall time to get a fully running system may be significantly longer when data is pre-seeded in the cloud.

+[Volume Shadow Copy Service (VSS) snapshots](file-sync-deployment-guide.md#optional-self-service-restore-through-previous-versions-and-vss-volume-shadow-copy-service) (including the **Previous Versions** tab) are supported on volumes with cloud tiering enabled. This allows you to perform self-service restores instead of relying on an admin to perform restores for you. However, you must enable previous version compatibility through PowerShell, which will increase your snapshot storage costs. VSS snapshots don't protect against disasters on the server endpoint itself, so they should only be used alongside cloud-side backups. For details, see [Self Service restore through Previous Versions and VSS](file-sync-deployment-guide.md#optional-self-service-restore-through-previous-versions-and-vss-volume-shadow-copy-service).

+1. Windows Server Failover Clustering is supported by Azure File Sync for the "File Server for general use" deployment option. For more information on how to configure the "File Server for general use" role on a Failover Cluster, see [Deploying a two-node clustered file server](/windows-server/failover-clustering/deploy-two-node-clustered-file-server).

+> To support Data Deduplication on volumes with cloud tiering enabled on Windows Server 2019, Windows update [KB4520062 - October 2019](https://support.microsoft.com/help/4520062) or a later monthly rollup update must be installed.

+Because antivirus works by scanning files for known malicious code, an antivirus product might cause the recall of tiered files, resulting in high egress charges. Tiered files have the secure Windows attribute FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS set and we recommend consulting with your software vendor to learn how to configure their solution to skip reading files with this attribute set (many do it automatically).

+> VSS snapshots (including Previous Versions tab) are supported on volumes which have cloud tiering enabled. However, you must enable previous version compatibility through PowerShell. [Learn how](file-sync-deployment-guide.md#optional-self-service-restore-through-previous-versions-and-vss-volume-shadow-copy-service).

+| 0x80c86044 | -2134351804 | ECS_E_AZURE_AUTHORIZATION_FAILED | The file cannot be synced because the firewall and virtual network settings on the storage account are enabled and the server does not have access to the storage account. | Add the Server IP address or virtual network by following the steps documented in the [Configure firewall and virtual network settings](file-sync-deployment-guide.md?tabs=azure-portal#optional-configure-firewall-and-virtual-network-settings) section in the deployment guide. |

+4. [Verify the firewall and virtual network settings on the storage account are configured properly (if enabled)](file-sync-deployment-guide.md?tabs=azure-portal#optional-configure-firewall-and-virtual-network-settings)

+3. [Verify the firewall and virtual network settings on the storage account are configured properly (if enabled)](file-sync-deployment-guide.md?tabs=azure-portal#optional-configure-firewall-and-virtual-network-settings)

+3. [Verify the firewall and virtual network settings on the storage account are configured properly (if enabled)](file-sync-deployment-guide.md?tabs=azure-portal#optional-configure-firewall-and-virtual-network-settings)

+2. [Verify the firewall and virtual network settings on the storage account are configured properly (if enabled)](file-sync-deployment-guide.md?tabs=azure-portal#optional-configure-firewall-and-virtual-network-settings)

+This error occurs when the Azure file share is inaccessible because of a storage account firewall or because the storage account belongs to a virtual network. Verify the firewall and virtual network settings on the storage account are configured properly. For more information, see [Configure firewall and virtual network settings](file-sync-deployment-guide.md?tabs=azure-portal#optional-configure-firewall-and-virtual-network-settings).

+This error occurs if the firewall and virtual network settings are enabled on the storage account and the "Allow trusted Microsoft services to access this storage account" exception is not checked. To resolve this issue, follow the steps documented in the [Configure firewall and virtual network settings](file-sync-deployment-guide.md?tabs=azure-portal#optional-configure-firewall-and-virtual-network-settings) section in the deployment guide.

+2. [Verify the firewall and virtual network settings on the storage account are configured properly (if enabled)](file-sync-deployment-guide.md?tabs=azure-portal#optional-configure-firewall-and-virtual-network-settings)

+| 0x80070005 | -2147024891 | ERROR_ACCESS_DENIED | The file failed to recall due to an access denied error. This issue can occur if the firewall and virtual network settings on the storage account are enabled and the server does not have access to the storage account. | To resolve this issue, add the Server IP address or virtual network by following the steps documented in the [Configure firewall and virtual network settings](file-sync-deployment-guide.md?tabs=azure-portal#optional-configure-firewall-and-virtual-network-settings) section in the deployment guide. |

+| Throughput (ingress + egress) for a single file share (MiB/sec) | <ul><li>Up to 300 MiB/sec, with large file share feature enabled<sup>2</sup></li><li>Up to 60 MiB/sec, default</li></ul> | 100 + CEILING(0.04 * ProvisionedStorageGiB) + CEILING(0.06 * ProvisionedStorageGiB) |

+ This video is an interview that discusses the basics of the Azure Files billing model. It covers how to optimize Azure file shares to achieve the lowest costs possible and how to compare Azure Files to other file storage offerings on-premises and in the cloud.

+Check with your operating system vendor if your operating system isn't listed.

+If you're migrating to Azure Files from on-premises or comparing Azure Files to other cloud storage solutions, you should consider the following factors to ensure a fair, apples-to-apples comparison:

+- **How do you pay for storage, IOPS, and bandwidth?** With Azure Files, the billing model you use depends on whether you're deploying [premium](#provisioned-model) or [standard](#pay-as-you-go-model) file shares. Most cloud solutions have models that align with the principles of either provisioned storage, such as price determinism and simplicity, or pay-as-you-go storage, which can optimize costs by only charging you for what you actually use. Of particular interest for provisioned models are minimum provisioned share size, the provisioning unit, and the ability to increase and decrease provisioning.

+- **Are there any methods to optimize storage costs?** With Azure Files, you can use [capacity reservations](#reserve-capacity) to achieve an up to 36% discount on storage. Other solutions may employ strategies like deduplication or compression to optionally optimize storage efficiency. However, these storage optimization strategies often have non-monetary costs, such as reducing performance. Azure Files capacity reservations have no side effects on performance.

+- **How do you achieve storage resiliency and redundancy?** With Azure Files, storage resiliency and redundancy are baked into the product offering. All tiers and redundancy levels ensure that data is highly available and at least three copies of your data are accessible. When considering other file storage options, consider whether storage resiliency and redundancy is built in or something you must assemble yourself.

+- **Term**: Reservations can be purchased for either a one-year or three-year term, with more significant discounts for purchasing a longer reservation term.

+Once you purchase a capacity reservation, it will automatically be consumed by your existing storage utilization. If you use more storage than you have reserved, you'll pay list price for the balance not covered by the capacity reservation. Transaction, bandwidth, data transfer, and metadata storage charges aren't included in the reservation.

+It's possible to decrease the size of your provisioned share below your used GiB. If you do, you won't lose data, but you'll still be billed for the size used and receive the performance of the provisioned share, not the size used.

+When you provision a premium file share, you specify how many GiBs your workload requires. Each GiB that you provision entitles you to more IOPS and throughput on a fixed ratio. In addition to the baseline IOPS for which you are guaranteed, each premium file share supports bursting on a best effort basis. The formulas for IOPS and throughput are as follows:

+| Baseline IOPS formula | `MIN(3000 + 1 * ProvisionedStorageGiB, 100000)` |

+| Burst limit | `MIN(MAX(10000, 3 * ProvisionedStorageGiB), 100000)` |

+| Throughput rate (ingress + egress) (MiB/sec) | `100 + CEILING(0.04 * ProvisionedStorageGiB) + CEILING(0.06 * ProvisionedStorageGiB)` |

+If your workload needs the extra performance to meet peak demand, your share can use burst credits to go above the share's baseline IOPS limit to give the share the performance it needs to meet the demand. Bursting is automated and operates based on a credit system. Bursting works on a best effort basis, and the burst limit isn't a guarantee.

+Credits accumulate in a burst bucket whenever traffic for your file share is below baseline IOPS. Earned credits are used later to enable burst when operations would exceed the baseline IOPS.

+New file shares start with the full number of credits in its burst bucket. Burst credits won't be accrued if the share IOPS fall below baseline IOPS due to throttling by the server.

+Azure Files uses a pay-as-you-go business model for standard file shares. In a pay-as-you-go business model, the amount you pay is determined by how much you actually use, rather than based on a provisioned amount. At a high level, you pay a cost for the amount of logical data stored, and then an additional set of transactions based on your usage of that data. A pay-as-you-go model can be cost-efficient, because you don't need to overprovision to account for future growth or performance requirements. You also don't need to deprovision if your workload and data footprint vary over time. On the other hand, a pay-as-you-go model can also be difficult to plan as part of a budgeting process, because the pay-as-you-go billing model is driven by end-user consumption.

+- Hot is for active workloads that don't involve a large number of transactions, and has a slightly lower data at-rest storage price, but slightly higher transaction prices as compared to transaction optimized. Think of it as the middle ground between the transaction optimized and cool tiers.

+- Cool optimizes the price for workloads that don't have much activity, offering the lowest data at-rest storage price, but the highest transaction prices.

+If you put an infrequently accessed workload in the transaction optimized tier, you'll pay almost nothing for the few times in a month that you make transactions against your share. However, you'll pay a high amount for the data storage costs. If you moved this same share to the cool tier, you'd still pay almost nothing for the transaction costs, simply because you're infrequently making transactions for this workload. However, the cool tier has a much cheaper data storage price. Selecting the appropriate tier for your use case allows you to considerably reduce your costs.

+Similarly, if you put a highly accessed workload in the cool tier, you'll pay a lot more in transaction costs, but less for data storage costs. This can lead to a situation where the increased costs from the transaction prices increase outweigh the savings from the decreased data storage price, leading you to pay more money on cool than you would have on transaction optimized. For some usage levels, it's possible that the hot tier will be the most cost efficient, and the cool tier will be more expensive than transaction optimized.

+Regardless of how you migrate existing data into Azure Files, we recommend initially creating the file share in transaction optimized tier due to the large number of transactions incurred during migration. After your migration is complete and you've operated for a few days or weeks with regular usage, you can plug your transaction counts into the [pricing calculator](https://azure.microsoft.com/pricing/calculator/) to figure out which tier is best suited for your workload.

+There are five basic transaction categories: write, list, read, other, and delete. All operations done via the REST API or SMB are bucketed into one of these categories:

+> NFS 4.1 is only available for premium file shares, which use the provisioned billing model. Transactions don't affect billing for premium file shares.

+- **Provisioned size or quota**: With both premium and standard file shares, you specify the maximum size that the file share is allowed to grow to. In premium file shares, this value is called the provisioned size, and whatever amount you provision is what you pay for, regardless of how much you actually use. In standard file shares, this value is called quota and does not directly affect your bill. Provisioned size is a required field for premium file shares. For standard file shares, if provisioned size isn't directly specified, the share will default to the maximum value supported by the storage account. This is either 5 TiB or 100 TiB, depending on the storage account type and settings.

+- **Logical size**: The logical size of a file share or file relates to how big it is without considering how it's actually stored, where additional optimizations may be applied. One way to think about this is that the logical size of the file is how many KiB/MiB/GiB will be transferred over the wire if you copy it to a different location. In both premium and standard file shares, the total logical size of the file share is what is used for enforcement against provisioned size/quota. In standard file shares, the logical size is the quantity used for the data at-rest usage billing. Logical size is referred to as "size" in the Windows properties dialog for a file/folder and as "content length" by Azure Files metrics.

+- **Physical size**: The physical size of the file relates to the size of the file as encoded on disk. This may align with the file's logical size, or it may be smaller, depending on how the file has been written to by the operating system. A common reason for the logical size and physical size to be different is by using [sparse files](/windows/win32/fileio/sparse-files). The physical size of the files in the share is used for snapshot billing, although allocated ranges are shared between snapshots if they are unchanged (differential storage). To learn more about how snapshots are billed in Azure Files, see [Snapshots](#snapshots).

+Azure Files supports snapshots, which are similar to volume shadow copies (VSS) on Windows File Server. Snapshots are always differential from the live share and from each other, meaning that you're always paying only for what's different in each snapshot. For more information on share snapshots, see [Overview of snapshots for Azure Files](storage-snapshots-files.md).

+Snapshots do not count against file share size limits, although you're limited to a specific number of snapshots. To see the current snapshot limits, see [Azure file share scale targets](storage-files-scale-targets.md#azure-file-share-scale-targets).

+- In premium file shares, snapshots are billed against their own snapshot meter, which has a reduced price over the provisioned storage price. This means that you'll see a separate line item on your bill representing snapshots for premium file shares for each FileStorage storage account on your bill.

+- In standard file shares, snapshots are billed as part of the normal used storage meter, although you're still only billed for the differential cost of the snapshot. This means that you won't see a separate line item on your bill representing snapshots for each standard storage account containing Azure file shares. This also means that differential snapshot usage counts against capacity reservations that are purchased for standard file shares.

+Like on-premises storage solutions that offer first- and third-party features and product integrations to add value to the hosted file shares, Azure Files provides integration points for first- and third-party products to integrate with customer-owned file shares. Although these solutions may provide considerable extra value to Azure Files, you should consider the extra costs that these services add to the total cost of an Azure Files solution.

+Costs are broken down into three buckets:

+- **Licensing costs for the value-added service.** These may come in the form of a fixed cost per customer, end user (sometimes called a "head cost"), Azure file share or storage account. They may also be based on units of storage utilization, such as a fixed cost for every 500 GiB chunk of data in the file share.

+- **Azure Files costs for using a value-added service.** Azure Files does not directly charge customers costs for adding value-added services, but as part of adding value to the Azure file share, the value-added service might increase the costs that you see on your Azure file share. This is easy to see with standard file shares, because standard file shares have a pay-as-you-go model with transaction charges. If the value-added service does transactions against the file share on your behalf, they will show up in your Azure Files transaction bill even though you didn't directly do those transactions yourself. This applies to premium file shares as well, although it may be less noticeable. Additional transactions against premium file shares from value-added services count against your provisioned IOPS numbers, meaning that value-added services may require provisioning more storage to have enough IOPS or throughput available for your workload.

+If you're migrating to Azure File Sync from StorSimple, see [Comparing the costs of StorSimple to Azure File Sync](../file-sync/file-sync-storsimple-cost-comparison.md).

+Azure Backup provides a serverless backup solution for Azure Files that seamlessly integrates with your file shares, and with other value-added services such as Azure File Sync. Azure Backup for Azure Files is a snapshot-based backup solution that provides a scheduling mechanism for automatically taking snapshots on an administrator-defined schedule. It also provides a user-friendly interface for restoring deleted files/folders or the entire share to a particular point in time. To learn more about Azure Backup for Azure Files, see [About Azure file share backup](../../backup/azure-file-share-backup-overview.md?toc=/azure/storage/files/toc.json).

+When considering the costs of using Azure Backup to back up your Azure file shares, consider the following:

+- **Protected instance licensing cost for Azure file share data.** Azure Backup charges a protected instance licensing cost per storage account containing backed up Azure file shares. A protected instance is defined as 250 GiB of Azure file share storage. Storage accounts containing less than 250 GiB of Azure file share storage are subject to a fractional protected instance cost. For more information, see [Azure Backup pricing](https://azure.microsoft.com/pricing/details/backup/). Note that you must select *Azure Files* from the list of services Azure Backup can protect.

+ - **Differential costs from Azure file share snapshots.** Azure Backup automates taking Azure file share snapshots on an administrator-defined schedule. Snapshots are always differential; however, the additional cost added to the total bill depends on the length of time snapshots are kept and the amount of churn on the file share during that time. This dictates how different the snapshot is from the live file share and therefore how much additional data is stored by Azure Files.

+The main cost from Microsoft Defender for Storage is an additional set of transaction costs that the product levies on top of the transactions that are done against the Azure file share. Although these costs are based on the transactions incurred in Azure Files, they aren't part of the billing for Azure Files, but rather are part of the Microsoft Defender pricing. Microsoft Defender for Storage charges a transaction rate even on premium file shares, where Azure Files includes transactions as part of IOPS provisioning. The current transaction rate can be found on [Microsoft Defender for Cloud pricing page](https://azure.microsoft.com/pricing/details/defender-for-cloud/) under the *Microsoft Defender for Storage* table row.

+ Title: Quickstart - Create an Azure Stream Analytics job using Bicep

+description: This quickstart shows how to use Bicep to create an Azure Stream Analytics job.

+# Quickstart: Create an Azure Stream Analytics job using Bicep

+In this quickstart, you use Bicep to create an Azure Stream Analytics job. Once the job is created, you validate the deployment.

+## Prerequisites

+To complete this article, you need to have an Azure subscription. [Create one for free](https://azure.microsoft.com/free/).

+## Review the Bicep file

+The Bicep file used in this quickstart is from [Azure Quickstart Templates](https://azure.microsoft.com/resources/templates/streamanalytics-create/).

+The Azure resource defined in the Bicep file is [Microsoft.StreamAnalytics/StreamingJobs](/azure/templates/microsoft.streamanalytics/streamingjobs): create an Azure Stream Analytics job.

+## Deploy the Bicep file

+1. Save the Bicep file as **main.bicep** to your local computer.

+1. Deploy the Bicep file using either Azure CLI or Azure PowerShell.

+ # [CLI](#tab/CLI)

+ ```azurecli

+ az group create --name exampleRG --location eastus

+ az deployment group create --resource-group exampleRG --template-file main.bicep --parameters streamAnalyticsJobName =<job-name> numberOfStreamingUnits=<int>

+ ```

+ # [PowerShell](#tab/PowerShell)

+ ```azurepowershell

+ New-AzResourceGroup -Name exampleRG -Location eastus

+ New-AzResourceGroupDeployment -ResourceGroupName exampleRG -TemplateFile ./main.bicep -streamAnalyticsJobName "<job-name>" -numberOfStreamingUnits <int>

+ ```

+

+ You need to provide values for the following parameters:

+ - **streamAnalyticsJobName**: Replace **\<job-name\>** with the Stream Analytics job name. The name can contain alphanumeric characters and hyphens, and it must be at least 3-63 characters long.

+ - **numberOfStreamingUnits**: Replace **\<int\>** with the number of Streaming Units. Allowed values include: 1, 3, 6, 12, 18, 24, 30, 36, 42, and 48.

+ > [!NOTE]

+ > When the deployment finishes, you should see a message indicating the deployment succeeded.

+## Review deployed resources

+You can either use the Azure portal to check the Azure Stream Analytics job or use the following Azure CLI or Azure PowerShell script to list the resource.

+### Azure CLI

+Use the Azure portal, Azure CLI, or Azure PowerShell to list the deployed resources in the resource group.

+# [CLI](#tab/CLI)

+```azurecli-interactive

+az resource list --resource-group exampleRG

+```

+# [PowerShell](#tab/PowerShell)

+```azurepowershell-interactive

+Get-AzResource -ResourceGroupName exampleRG

+```

+## Clean up resources

+If you plan to continue on to subsequent tutorials, you may wish to leave these resources in place. When no longer needed, delete the resource group, which deletes the Azure Stream Analytics job. To delete the resource group by using Azure CLI or Azure PowerShell:

+# [CLI](#tab/CLI)

+```azurecli-interactive

+az group delete --name exampleRG

+```

+# [PowerShell](#tab/PowerShell)

+```azurepowershell-interactive

+Remove-AzResourceGroup -Name exampleRG

+```

+## Next steps

+In this quickstart, you created an Azure Stream Analytics job using Bicep and validated the deployment. To learn how to create your own Bicep files using Visual Studio Code, continue on to the following article:

+> [!div class="nextstepaction"]

+> [Quickstart: Create Bicep files with Visual Studio Code](../azure-resource-manager/bicep/quickstart-create-bicep-use-visual-studio-code.md)

+ Title: "Synapse implementation success methodology: Assess environment"

+description: "Learn how to assess your environment to help evaluate the solution design and make informed technology decisions to implement Azure Synapse Analytics."

+# Synapse implementation success methodology: Assess environment

+The first step when implementing Azure Synapse Analytics is to assessment your environment. An assessment provides you with the opportunity to gather all the available information about your existing environment, environmental requirements, project requirements, constraints, timelines, and pain points. This information will form the basis of later evaluations and checkpoint activities. It will prove invaluable when it comes time to validate and compare against the project solution as it's planned, designed, and developed. We recommend that you dedicate a good amount of time to gather all the information and be sure to have necessary discussions with relevant groups. Relevant groups can include project stakeholders, business users, solution designers, and subject matter experts (SMEs) of the existing solution and environment.

+The assessment will become a guide to help you evaluate the solution design and make informed technology recommendations to implement Azure Synapse.

+## Workload assessment

+The workload assessment is concerned with the environment, analytical workload roles, ETL/ELT, networking and security, the Azure environment, and data consumption.

+### Environment

+For the environment, evaluate the following points.

+- Describe your existing analytical workload:

+ - What are the workloads (like data warehouse or big data)?

+ - How is this workload helping the business? What are the use case scenarios?

+ - What is the business driver for this analytical platform and for potential migration?

+ - Gather details about the existing architecture, design, and implementation choices.

+ - Gather details about all existing upstream and downstream dependent components and consumers.

+- Are you migrating an existing data warehouse (like Microsoft SQL Server, Microsoft Analytics Platform System (APS), Netezza, Snowflake, or Teradata)?

+- Are you migrating a big data platform (like Cloudera or Hortonworks)?

+- Gather the architecture and dataflow diagrams for the current analytical environment.

+- Where are the data sources for your planned analytical workloads located (Azure, other cloud providers, or on-premises)?

+- What is the total size of existing datasets (historical and incremental)? What is the current rate of growth of your dataset(s)? What is the projected rate of growth of your datasets for the next 2-5 years?

+- Do you have an existing data lake? Gather as much detail as possible about file types (like Parquet or CSV), file sizes, and security configuration.

+- Do you have semi-structured or unstructured data to process and analyze?

+- Describe the nature of the data processing (batch or real-time processing).

+- Do you need interactive data exploration from relational data, data lake, or other sources?

+- Do you need real-time data analysis and exploration from operational data sources?

+- What are the pain points and limitations in the current environment?

+- What source control and DevOps tools are you using today?

+- Do you have a use case to build a hybrid (cloud and on-premises) analytical solution, cloud only, or multi-cloud?

+- Gather information on the existing cloud environment. Is it a single-cloud provider or a multi-cloud provider?

+- Gather plans about the future cloud environment. Will it be a single-cloud provider or a multi-cloud provider?

+- What are the RPO/RTO/HA/SLA requirements in the existing environment?

+- What are the RPO/RTO/HA/SLA requirements in the planned environment?

+### Analytical workload roles

+For the analytical workload roles, evaluate the following points.

+- Describe the different roles (data scientist, data engineer, data analyst, and others).

+- Describe the analytical platform access control requirement for these roles.

+- Identify the platform owner who's responsible to provision compute resources and grant access.

+- Describe how different data roles currently collaborate.

+- Are there multiple teams collaborating on the same analytical platform? If so, what's the access control and isolation requirements for each of these teams?

+- What are the client tools that end users use to interact with the analytical platform?

+### ETL/ELT, transformation, and orchestration

+For ETL/ELT, transformation, and orchestration, evaluate the following points.

+- What tools are you using today for data ingestion (ETL or ELT)?

+- Where do these tools exist in the existing environment (on-premises or the cloud)?

+- What are your current data load and update requirements (real-time, micro batch, hourly, daily, weekly, or monthly)?

+- Describe the transformation requirements for each layer (big data, data lake, data warehouse).

+- What is the current programming approach for transforming the data (no-code, low-code, programming like SQL, Python, Scala, C#, or other)?

+- What is the preferred planned programming approach to transform the data (no-code, low-code, programming like SQL, Python, Scala, C#, or other)?

+- What tools are currently in use for data orchestration to automate the data-driven process?

+- Where are the data sources for your existing ETL located (Azure, other cloud provider, or on-premises)?

+- What are the existing data consumptions tools (reporting, BI tools, open-source tools) that require integration with the analytical platform?

+- What are the planned data consumptions tools (reporting, BI tools, open-source tools) that will require integration with the analytical platform?

+### Networking and security

+For networking and security, evaluate the following points.

+- What regulatory requirements do you have for your data?

+- If your data contains customer content, payment card industry (PCI), or Health Insurance Portability and Accountability Act of 1996 (HIPAA) data, has your security group certified Azure for this data? If so, for which Azure services?

+- Describe your user authorization and authentication requirements.

+- Are there security issues that could limit access to data during implementation?

+- Is there test data available to use during development and testing?

+- Describe the organizational network security requirements on the analytical compute and storage (private network, public network, or firewall restrictions).

+- Describe the network security requirements for client tools to access analytical compute and storage (peered network, private endpoint, or other).

+- Describe the current network setup between on-premises and Azure (Azure ExpressRoute, site-to-site, or other).

+Use the following checklists of possible requirements to guide your assessment.

+- Data protection:

+ - In-transit encryption

+ - Encryption at rest (default keys or customer-managed keys)

+ - Data discovery and classification

+- Access control:

+ - Object-level security

+ - Row-level security

+ - Column-level security

+ - Dynamic data masking

+- Authentication:

+ - SQL login

+ - Azure Active Directory (Azure AD)

+ - Multi-factor authentication (MFA)

+- Network security:

+ - Virtual networks

+ - Firewall

+ - Azure ExpressRoute

+- Threat protection:

+ - Threat detection

+ - Auditing

+ - Vulnerability assessment

+For more information, see the [Azure Synapse Analytics security white paper](security-white-paper-introduction.md).

+### Azure environment

+For the Azure environment, evaluate the following points.

+- Are you currently using Azure? Is it used for production workloads?

+- If you're using Azure, which services are you using? Which regions are you using?

+- Do you use Azure ExpressRoute? What's its bandwidth?

+- Do you have budget approval to provision the required Azure services?

+- How do you currently provision and manage resources (Azure Resource Manager (ARM) or Terraform)?

+- Is your key team familiar with Synapse Analytics? Is any training required?

+### Data consumption

+For data consumption, evaluate the following points.

+- Describe how and what tools you currently use to perform activities like ingest, explore, prepare, and data visualization.

+- Identify what tools you plan to use to perform activities like ingest, explore, prepare, and data visualization.

+- What applications are planned to interact with the analytical platform (Microsoft Power BI, Microsoft Excel, Microsoft SQL Server Reporting Services, Tableau, or others)?

+- Identify all data consumers.

+- Identify data export and data sharing requirements.

+## Azure Synapse services assessment

+The Azure Synapse services assessment is concerned with the services within Azure Synapse. Azure Synapse has the following components for compute and data movement:

+- **Synapse SQL:** A distributed query system for Transact-SQL (T-SQL) that enables data warehousing and data virtualization scenarios. It also extends T-SQL to address streaming and machine learning (ML) scenarios. Synapse SQL offers both *serverless* and *dedicated* resource models.

+- **Serverless SQL pool:** A distributed data processing system, built for large-scale data and computational functions. There's no infrastructure to set up or clusters to maintain. This service is suited for unplanned or burst workloads. Recommended scenarios include quick data exploration on files directly on the data lake, logical data warehouse, and data transformation of raw data.

+- **Dedicated SQL pool:** Represents a collection of analytic resources that are provisioned when using Synapse SQL. The size of a dedicated SQL pool (formerly SQL DW) is determined by Data Warehousing Units (DWU). This service is suited for a data warehouse with predictable, high performance continuous workloads over data stored in SQL tables. 

+- **Apache Spark pool:** Deeply and seamlessly integrates Apache Spark, which is the most popular open source big data engine used for data preparation, data engineering, ETL, and ML.

+- **Data integration pipelines:** Azure Synapse contains the same data integration engine and experiences as Azure Data Factory (ADF). They allow you to create rich at-scale ETL pipelines without leaving Azure Synapse.

+To help determine the best SQL pool type (dedicated or serverless), evaluate the following points.

+- Do you want to build a traditional relational data warehouse by reserving processing power for data stored in SQL tables?

+- Do your use cases demand predictable performance?

+- Do you want to a build a logical data warehouse on top of a data lake?

+- Do you want to query data directly from a data lake?

+- Do you want to explore data from a data lake?

+The following table compares the two Synapse SQL pool types.

+| **Comparison** | **Dedicated SQL pool** | **Serverless SQL pool** |

+|:-|:-|:-|

+| Value propositions | Fully managed capabilities of a data warehouse. Predictable and high performance for continuous workloads. Optimized for managed (loaded) data. | Easy to get started and explore data lake data. Better total cost of ownership (TCO) for ad hoc and intermittent workloads. Optimized for querying data in a data lake. |

+| Workloads | *Ideal for continuous workloads.* Loading boosts performance, with more complexity. Charging per DWU (when sized well) will be cost-beneficial. | *Ideal for ad hoc or intermittent workloads.* There's no need to load data, so it's easier to start and run. Charging per usage will be cost-beneficial. |

+| Query performance | *Delivers high concurrency and low latency.* Supports rich caching options, including materialized views. There's the ability to choose trade-offs with workload management (WLM). | *Not suited for dashboarding queries.* Millisecond response times aren't expected. It works only on external data. |

+### Dedicated SQL pool assessment

+For the dedicated SQL pool assessment, evaluate the following platform points.

+- What is the current data warehouse platform (Microsoft SQL Server, Netezza, Teradata, Greenplum, or other)?

+- For a migration workload, determine the make and model of your appliance for each environment. Include details of CPUs, GPUs, and memory.

+- For an appliance migration, when was the hardware purchased? Has the appliance been fully depreciated? If not, when will depreciation end? And, how much capital expenditure is left?

+- Are there any hardware and network architecture diagrams?

+- Where are the data sources for your planned data warehouse located (Azure, other cloud provider, or on-premises)?

+- What are the data hosting platforms of the data sources for your data warehouse (Microsoft SQL Server, Azure SQL Database, DB2, Oracle, Azure Blob Storage, AWS, Hadoop, or other)?

+- Are any of the data sources data warehouses? If so, which ones?

+- Identify all ETL, ELT, and data loading scenarios (batch windows, streaming, near real-time). Identify existing service level agreements (SLAs) for each scenario and document the expected SLAs in the new environment.

+- What is the current data warehouse size?

+- What rate of dataset growth is being targeted for the dedicated SQL pool?

+- Describe the environments you're using today (development, test, or production).

+- Which tools are currently in place for data movement (ADF, Microsoft SQL Server Integration Services (SSIS), robocopy, Informatica, SFTP, or others)?

+- Are you planning to load real-time or near real-time data?

+Evaluate the following database points.

+- What is the number of objects in each data warehouse (schemas, tables, views, stored procedures, functions)?

+- Is it a star schema, snowflake schema or other design?

+- What are the largest tables in terms of size and number of records?

+- What are the widest tables in terms of the number of columns?

+- Is there already a data model designed for your data warehouse? Is it a Kimball, Inmon, or star schema design?

+- Are Slowly Changing Dimensions (SCDs) in use? If so, which types?

+- Will a semantic layer be implemented by using relational data marts or Analysis Services (tabular or multidimensional), or another product?

+- What are the HA/RPO/RTO/data archiving requirements?

+- What are the region replication requirements?

+Evaluate the following workload characteristics.

+- What is the estimated number of concurrent users or jobs accessing the data warehouse during *peak hours*?

+- What is the estimated number of concurrent users or jobs accessing the data warehouse during *off peak hours*?

+- Is there a period of time when there will be no users or jobs?

+- What are your query execution performance expectations for interactive queries?

+- What are your data load performance expectations for daily/weekly/monthly data loads or updates?

+- What are your query execution expectations for reporting and analytical queries?

+- How complex will the most commonly executed queries be?

+- What percentage of your total dataset size is your active dataset?

+- Approximately what percentage of the workload is anticipated for loading or updating, batch processing or reporting, interactive query, and analytical processing?

+- Identify the data consuming patterns and platforms:

+ - Current and planned reporting method and tools.

+ - Which application or analytical tools will access the data warehouse?

+ - Number of concurrent queries?

+ - Average number of active queries at any point in time?

+ - What is the nature of data access (interactive, ad hoc, export, or others)?

+ - Data roles and complete description of their data requirements.

+ - Maximum number of concurrent connections.

+- Query performance SLA pattern by:

+ - Dashboard users.

+ - Batch reporting.

+ - ML users.

+ - ETL process.

+- What are the security requirements for the existing environment and for the new environment (row-level security, column-level security, access control, encryption, and others)?

+- Do you have requirements to integrate ML model scoring with T-SQL?

+### Serverless SQL pool assessment

+Synapse Serverless SQL pool supports three major use cases.

+- **Basic discovery and exploration:** Quickly reason about the data in various formats (Parquet, CSV, JSON) in your data lake, so you can plan how to extract insights from it.

+- **Logical data warehouse:** Provide a relational abstraction on top of raw or disparate data without relocating and transforming data, allowing an always-current view of your data.

+- **Data transformation:** Simple, scalable, and performant way to transform data in the lake by using T-SQL, so it can be fed to BI and other tools or loaded into a relational data store (Synapse SQL databases, Azure SQL Database, or others).

+Different data roles can benefit from serverless SQL pool:

+- **Data engineers** can explore the data lake, transform and prepare data by using this service, and simplify their data transformation pipelines.

+- **Data scientists** can quickly reason about the contents and structure of the data in the data lake, thanks to features such as OPENROWSET and automatic schema inference.

+- **Data analysts** can [explore data and Spark external tables](../sql/develop-storage-files-spark-tables.md) created by data scientists or data engineers by using familiar T-SQL statements or their favorite query tools.

+- **BI professionals** can quickly [create Power BI reports on top of data in the data lake](../sql/tutorial-connect-power-bi-desktop.md) and Spark tables.

+> [!NOTE]

+> The T-SQL language is used in both dedicated SQL pool and the serverless SQL pool, however there are some differences in the set of supported features. For more information about T-SQL features supported in Synapse SQL (dedicated and serverless), see [Transact-SQL features supported in Azure Synapse SQL](../sql/overview-features.md).

+For the serverless SQL pool assessment, evaluate the following points.

+- Do you have use cases to discover and explore data from a data lake by using relational queries (T-SQL)?

+- Do you have use cases to build a logical data warehouse on top of a data lake?

+- Identify whether there are use cases to transform data in the data lake without first moving data from the data lake.

+- Is your data already in Azure Data Lake Storage (ADLS) or Azure Blob Storage?

+- If your data is already in ADLS, do you have a good partition strategy in the data lake?

+- Do you have operational data in Azure Cosmos DB? Do you have use cases for real-time analytics on Azure Cosmos DB without impacting transactions?

+- Identify the file types in the data lake.

+- Identify the query performance SLA. Does your use case demand predictable performance and cost?

+- Do you have unplanned or bursty SQL analytical workloads?

+- Identify the data consuming pattern and platforms:

+ - Current and planned reporting method and tools.

+ - Which application or analytical tools will access the serverless SQL pool?

+ - Average number of active queries at any point in time.

+ - What is the nature of data access (interactive, ad hoc, export, or others)?

+ - Data roles and complete description of their data requirements.

+ - Maximum number of concurrent connections.

+ - Query complexity?

+- What are the security requirements (access control, encryption, and others)?

+- What is the required T-SQL functionality (stored procedures or functions)?

+- Identify the number of queries that will be sent to the serverless SQL pool and the result set size of each query.

+> [!TIP]

+> If you're new to serverless SQL pools, we recommend you work through the [Build data analytics solutions using Azure Synapse serverless SQL pools](/learn/paths/build-data-analytics-solutions-using-azure-synapse-serverless-sql-pools/) learning path.

+### Spark pool assessment

+Spark pools in Azure Synapse enable the following key scenarios.

+- **Data engineering/Data preparation:** Apache Spark includes many language features to support preparation and processing of large volumes of data. Preparation and processing can make the data more valuable and allow it to be consumed by other Azure Synapse services. It's enabled through multiple languages (C#, Scala, PySpark, Spark SQL) and by using supplied libraries for processing and connectivity.

+- **Machine learning:** Apache Spark comes with [MLlib](https://spark.apache.org/mllib/), which is an ML library built on top of Spark that you can use from a Spark pool. Spark pools also include Anaconda, which is a Python distribution that comprises various packages for data science including ML. In addition, Apache Spark on Synapse provides pre-installed libraries for [Microsoft Machine Learning](https://mmlspark.blob.core.windows.net/website/https://docsupdatetracker.net/index.html), which is a fault-tolerant, elastic, and RESTful ML framework. When combined with built-in support for notebooks, you have a rich environment for creating ML applications.

+> [!NOTE]

+> For more information, see [Apache Spark in Azure Synapse Analytics](../spark/apache-spark-overview.md).

+>

+> Also, Azure Synapse is compatible with Linux Foundation Delta Lake. Delta Lake is an open-source storage layer that brings ACID (atomicity, consistency, isolation, and durability) transactions to Apache Spark and big data workloads. For more information, see [What is Delta Lake](../spark/apache-spark-what-is-delta-lake.md).

+For the Spark pool assessment, evaluate the following points.

+- Identify the workloads that require data engineering or data preparation.

+- Clearly define the types of transformations.

+- Identify whether you have unstructured data to process.

+- When you're migrating from an existing Spark/Hadoop workload:

+ - What is the existing big data platform (Cloudera, Hortonworks, cloud services, or other)?

+ - If it's a migration from on-premises, has hardware depreciated or licenses expired? If not, when will depreciation or expiry happen?

+ - What is the existing cluster type?

+ - What are the required libraries and Spark versions?

+ - Is it a Hadoop migration to Spark?

+ - What are the current or preferred programming languages?

+ - What is the type of workload (big data, ML, or other)?

+ - What are the existing and planned client tools and reporting platforms?

+ - What are the security requirements?

+ - Are there any current pain points and limitations?

+- Do you plan to use, or are currently using, Delta Lake?

+- How do you manage packages today?

+- Identify the required compute cluster types.

+- Identify whether cluster customization is required.

+> [!TIP]

+> If you're new to Spark pools, we recommend you work through the [Perform data engineering with Azure Synapse Apache Spark Pools](/learn/paths/perform-data-engineering-with-azure-synapse-apache-spark-pools/) learning path.

+## Next steps

+In the [next article](implementation-success-evaluate-workspace-design.md) in the *Azure Synapse success by design* series, learn how to evaluate the Synapse workspace design and validate that it meets guidelines and requirements.

+ Title: "Synapse implementation success methodology: Evaluate data integration design"

+description: "Learn how to evaluate the data integration design and validate that it meets guidelines and requirements."

+# Synapse implementation success methodology: Evaluate data integration design

+Azure Synapse Analytics contains the same data integration engine and experiences as Azure Data Factory (ADF), allowing you to create rich at-scale ETL pipelines without leaving Azure Synapse Analytics.

+This article describes how to evaluate the design of the data integration components for your project. Specifically, it helps you to determine whether Azure Synapse pipelines are the best fit for your data integration requirements. Time invested in evaluating the design prior to solution development can help to eliminate unexpected design changes that may impact on your project timeline or cost.

+## Fit gap analysis

+You should perform a thorough fit gap analysis of your data integration strategy. If you choose Azure Synapse pipelines as the data integration tool, review the following points to ensure they're the best fit for your data integration requirements and orchestration. Even if you choose different data integration tools, you should still review the following points to validate that all key design points have been considered and that your chosen tool will support your solution needs. This information should have been captured during your assessment performed earlier in this methodology.

+- Review your data sources and destinations (targets):

+ - Validate that source and destination stores are [supported data stores](/azure/data-factory/connector-overview).

+ - If they're not supported, check whether you can use the [extensible options](/azure/data-factory/connector-overview#integrate-with-more-data-stores).

+- Review the triggering points of your data integration and the frequency:

+ - Azure Synapse pipelines support schedule, tumbling window, and storage event triggers.

+ - Validate the minimum recurrence interval and supported storage events against your requirements.

+- Review the required modes of data integration:

+ - Scheduled, periodic, and triggered batch processing can be effectively designed in Azure Synapse pipelines.

+ - To implement Change Data Capture (CDC) functionality, use third-party products or create a custom solution.

+ - To support real-time streaming, use [Azure Event Hubs](/azure/event-hubs/event-hubs-about), [Azure Event Hubs from Apache Kafka](/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview), or [Azure IoT Hub](/azure/iot-hub/iot-concepts-and-iot-hub).

+ - To run Microsoft SQL Server Integration Services (SSIS) packages, you can [lift and shift SSIS workloads to the cloud](/sql/integration-services/lift-shift/ssis-azure-lift-shift-ssis-packages-overview?view=sql-server-ver15&preserve-view=true).

+- Review the compute design:

+ - Does the compute required for the pipelines need to be serverless or provisioned?

+ - Azure Synapse pipelines support both modes of integration runtime (IR): serverless or self-hosted on a Windows machine.

+ - Validate [ports and firewalls](/azure/data-factory/create-self-hosted-integration-runtime?tabs=data-factory#ports-and-firewalls) and [proxy setting](/azure/data-factory/create-self-hosted-integration-runtime?tabs=data-factory#proxy-server-considerations) when using the self-hosted IR (provisioned).

+- Review security requirements, networking and firewall configuration of the environment and compare them to the security, networking and firewall configuration design:

+ - Review how the data sources are secured and networked.

+ - Review how the target data stores are secured and networked. Azure Synapse pipelines have different [data access strategies](/azure/data-factory/data-access-strategies) that provide a secure way to connect data stores via private endpoints or virtual networks.

+ - Use [Azure Key Vault](/azure/key-vault/general/basic-concepts) to store credentials whenever applicable.

+ - Use ADF for customer-managed key (CMK) encryption of credentials and store them in the self-hosted IR.

+- Review the design for ongoing monitoring of all data integration components.

+## Architecture considerations

+As you review the data integration design, consider the following recommendations and guidelines to ensure that the data integration components of your solution will provide ongoing operational excellence, performance efficiency, reliability, and security.

+### Operational excellence

+For operational excellence, evaluate the following points.

+- **Environment:** When planning your environments, segregate them by development/test, user acceptance testing (UAT), and production. Use the folder organizational options to organize your pipelines and datasets by business/ETL jobs to support better maintainability. Use [annotations](https://azure.microsoft.com/resources/videos/azure-friday-enhanced-monitoring-capabilities-and-tagsannotations-in-azure-data-factory/) to tag your pipelines so you can easily monitor them. Create reusable pipelines by using parameters, and iteration and conditional activities.

+- **Monitoring and alerting:** Synapse workspaces include the [Monitor Hub](../get-started-monitor.md), which has rich monitoring information of each and every pipeline run. It also integrates with [Log Analytics](/azure/azure-monitor/logs/log-analytics-overview) for further log analysis and alerting. You should implement these features to provide proactive error notifications. Also, use *Upon Failure* paths to implement customized [error handling](https://techcommunity.microsoft.com/t5/azure-data-factory/understanding-pipeline-failures-and-error-handling/ba-p/1630459).

+- **Automated deployment and testing:** Azure Synapse pipelines are built into Synapse workspace, so you can take advantage of workspace automation and deployment. Use [ARM templates](../quickstart-deployment-template-workspaces.md) to minimize manual activities when creating Synapse workspaces. Also, [integrate Synapse workspaces with Azure DevOps](../cicd/continuous-integration-delivery.md#set-up-a-release-pipeline-in-azure-devops) to build code versioning and automate publication.

+### Performance efficiency

+For performance efficiency, evaluate the following points.

+- Follow [performance guidance](/azure/data-factory/copy-activity-performance) and [optimization features](/azure/data-factory/copy-activity-performance-features) when working with the copy activity.

+- Choose optimized connectors for data transfer instead of generic connectors. For example, use PolyBase instead of bulk insert when moving data from Azure Data Lake Storage Gen2 (ALDS Gen2) to a dedicated SQL pool.

+- When creating a new Azure IR, set the region location as [auto-resolve](/azure/data-factory/concepts-integration-runtime#azure-ir-location) or select the same region as the data stores.

+- For self-hosted IR, choose the [Azure virtual machine (VM) size](/azure/data-factory/copy-activity-performance-features#self-hosted-integration-runtime-scalability) based on the integration requirements.

+- Choose a stable network connection, like [Azure ExpressRoute](/azure/expressroute/expressroute-introduction), for fast and consistent bandwidth.

+### Reliability

+When you execute a pipeline by using Azure IR, it's serverless in nature and so it provides resiliency out of the box. There's little for customers to manage. However, when a pipeline runs in a self-hosted IR, we recommend that you run it by using a [high availability configuration](/azure/data-factory/create-self-hosted-integration-runtime?tabs=data-factory#high-availability-and-scalability) in Azure VMs. This configuration ensures integration pipelines aren't broken even when a VM goes offline. Also, we recommend that you use Azure ExpressRoute for a fast and reliable network connection between on-premises and Azure.

+### Security

+A secured data platform is one of the key requirements of every organization. You should thoroughly plan security for the entire platform rather than individual components. Here are some security guidelines for Azure Synapse pipeline solutions.

+- Secure data movement to the cloud by using [Azure Synapse private endpoints](https://techcommunity.microsoft.com/t5/azure-architecture-blog/understanding-azure-synapse-private-endpoints/ba-p/2281463).

+- Use Azure Active Directory (Azure AD) [managed identities](/azure/active-directory/managed-identities-azure-resources/overview) for authentication.

+- Use Azure role-based access control (RBAC) and [Synapse RBAC](../security/synapse-workspace-synapse-rbac.md) for authorization.

+- Store credentials, secrets, and keys in Azure Key Vault rather than in the pipeline. For more information, see [Use Azure Key Vault secrets in pipeline activities](/azure/data-factory/how-to-use-azure-key-vault-secrets-pipeline-activities).

+- Connect to on-premises resources via Azure ExpressRoute or VPN over private endpoints.

+- Enable the **Secure output** and **Secure input** options in pipeline activities when parameters store secrets or passwords.

+## Next steps

+In the [next article](implementation-success-evaluate-dedicated-sql-pool-design.md) in the *Azure Synapse success by design* series, learn how to evaluate your dedicated SQL pool design to identify issues and validate that it meets guidelines and requirements.

+ Title: "Synapse implementation success methodology: Evaluate dedicated SQL pool design"

+description: "Learn how to evaluate your dedicated SQL pool design to identify issues and validate that it meets guidelines and requirements."

+# Synapse implementation success methodology: Evaluate dedicated SQL pool design

+You should evaluate your [dedicated SQL pool](../sql-data-warehouse/sql-data-warehouse-overview-what-is.md) design to identify issues and validate that it meets guidelines and requirements. By evaluating the design *before solution development begins*, you can avoid blockers and unexpected design changes. That way, you protect the project's timeline and budget.

+Synapse SQL has a scale-out architecture that distributes computational data processing across multiple nodes. Compute is separate from storage, which enables you to scale compute independently of the data in your system. For more information, see [Dedicated SQL pool (formerly SQL DW) architecture in Azure Synapse Analytics](../sql-data-warehouse/massively-parallel-processing-mpp-architecture.md).

+## Assessment analysis

+During the [assessment stage](implementation-success-assess-environment.md), you collected information about how the original system was deployed and details of the structures that were implemented. That information can now help you to identify gaps between what's implemented and what needs to be developed. For example, now's the time to consider the impact of designing round-robin tables instead of hash distributed tables, or the performance benefits of correctly using replicated tables.

+## Review the target architecture

+To successfully deploy a dedicated SQL pool, it's important to adopt an architecture that's aligned with business requirements. For more information, see [Data warehousing in Microsoft Azure](/azure/architecture/data-guide/relational-dat).

+## Migration path

+A migration project for Azure Synapse is similar to any other database migration. You should consider that there might be differences between the original system and Azure Synapse.

+Ensure that you have a clear migration path established for:

+- Database objects, scripts, and queries

+- Data transfer (export from source and transit to the cloud)

+- Initial data load into Azure Synapse

+- Logins and users

+- Data access control (row-level security)

+For more information, see [Migrate a data warehouse to a dedicated SQL pool in Azure Synapse Analytics](../migration-guides/migrate-to-synapse-analytics-guide.md).

+## Feature gaps

+Determine whether the original system depends on features that aren't supported by Azure Synapse. Unsupported features in dedicated SQL pools include certain data types, like XML and spatial data types, and cursors.

+For more information, see:

+- [Table data types for dedicated SQL pool (formerly SQL DW) in Azure Synapse Analytics](../sql-data-warehouse/sql-data-warehouse-tables-data-types.md#identify-unsupported-data-types)

+- [Transact-SQL features supported in Azure Synapse SQL](../sql/overview-features.md)

+## Dedicated SQL pool testing

+As with any other project, you should conduct tests to ensure that your dedicated SQL pool delivers the required business needs. It's critical to test data quality, data integration, security, and performance.

+## Next steps

+In the [next article](implementation-success-evaluate-serverless-sql-pool-design.md) in the *Azure Synapse success by design* series, learn how to evaluate your Spark pool design to identify issues and validate that it meets guidelines and requirements.

+ Title: "Synapse implementation success methodology: Evaluate project plan"

+description: "Learn how to evaluate your modern data warehouse project plan before the project starts."

+# Synapse implementation success methodology: Evaluate project plan

+In the lifecycle of the project, the most important and extensive planning is done *before implementation*. This article describes how to conduct a high-level review of your project plan. The aim is to ensure it contains critical artifacts and information to deliver a successful solution. It includes checklists of items that you should complete and approve before the project starts.

+A detailed review should follow the high-level project plan review. The detailed review should focus on the specific Azure Synapse components identified during the [assessment stage](implementation-success-assess-environment.md).

+## Evaluate the project plan

+Work through the following two high-level checklists, taking care to verify that each task aligns with the information gathered during the [assessment stage](implementation-success-assess-environment.md).

+First, ensure that your project plan defines the following points.

+> [!div class="checklist"]

+> - **The core resource team:** Assemble a group of key people that have expertise crucial to the project.

+> - **Scope:** Document how the project scope will be defined, verified, measured, and how the work breakdown will be defined and assigned.

+> - **Schedule:** Define the time duration required to complete the project.

+> - **Cost:** Estimate costs for internal and external resources, including infrastructure, hardware, and software.

+Second, having defined and assigned the work breakdown, prepare the following artifacts.

+> [!div class="checklist"]

+> - **Migration plan:** Document the plan to migrate from your current system to Azure Synapse. Incorporate tasks for executing the migration within the project plan scope and schedule.

+> - **Success criteria:** Define the critical success criteria for stakeholders (or the project sponsor), including go and no-go criteria.

+> - **Quality assurance:** Define how to conduct code reviews, and the development, staging, and production promotion approval processes.

+> - **Test plan:** Define test cases, success criteria for unit, integration, user testing, and metrics to validate all deliverables. Incorporate tasks for developing and executing the test plans within the project plan scope and schedule.

+## Evaluate project plan detailed tasks

+Once the high-level project plan review is complete and approved, the next step is to drill down into each component of the project plan.

+Identify the project plan components that address each aspect of Azure Synapse as it's intended for use in your solution. Also, validate that the project plan accounts for all the effort and resources required to develop, test, deploy, and operate your solution by evaluating:

+- The workspace project plan.

+- The data integration project plan.

+- The dedicated SQL pool project plan.

+- The serverless SQL pool project plan.

+- The Spark pool project plan.

+## Next steps

+In the [next article](implementation-success-evaluate-solution-development-environment-design.md) in the *Azure Synapse success by design* series, learn how to evaluate the environments for your modern data warehouse project to support development, testing, and production.

+ Title: "Synapse implementation success methodology: Evaluate serverless SQL pool design"

+description: "Learn how to evaluate your serverless SQL pool design to identify issues and validate that it meets guidelines and requirements."

+# Synapse implementation success methodology: Evaluate serverless SQL pool design

+You should evaluate your [serverless SQL pool](../sql/on-demand-workspace-overview.md) design to identify issues and validate that it meets guidelines and requirements. By evaluating the design *before solution development begins*, you can avoid blockers and unexpected design changes. That way, you protect the project's timeline and budget.

+The architectural separation of storage and compute for modern data, analytical platforms and services has been a trend and frequently used pattern. It provides cost savings and more flexibility allowing independent on-demand scaling of your storage and compute. Synapse SQL serverless extends this pattern by adding the capability to query your data lake data directly. There's no need to worry about compute management when using self-service types of workloads.

+## Fit gap analysis

+When planning to implement SQL serverless pools within Azure Synapse, you first need to ensure serverless pools are the right fit for your workloads. You should consider operational excellence, performance efficiency, reliability, and security.

+### Operational excellence

+For operational excellence, evaluate the following points.

+- **Solution development environment:** Within this methodology, there's an evaluation of the [solution development environment](implementation-success-evaluate-solution-development-environment-design.md). Identify how the environments (development, test, and production) are designed to support solution development. Commonly, you'll find a production and non-production environments (for development and test). You should find Synapse workspaces in all of the environments. In most cases, you'll be obliged to segregate your production and development/test users and workloads.

+- **Synapse workspace design:** Within this methodology, there's an evaluation of the [Synapse workspace design](implementation-success-evaluate-workspace-design.md). Identify how the workspaces have been designed for your solution. Become familiar with the design and know whether the solution will use a single workspace or whether multiple workspaces form part of the solution. Know why a single or multiple workspace design was chosen. A multi-workspace design is often chosen to enforce strict security boundaries.

+- **Deployment:** SQL serverless is available on-demand with every Synapse workspace, so it doesn't require any special deployment actions. Check regional proximity of the service and that of the Azure Data Lake Storage Gen2 (ADLS Gen2) account that it's connected to.

+- **Monitoring:** Check whether built-in monitoring is sufficient and whether any external services need to be put in place to store historical log data. Log data allows analyzing changes in performance and allows you to define alerting or triggered actions for specific circumstances.

+### Performance efficiency

+Unlike traditional database engines, SQL serverless doesn't rely on its own optimized storage layer. For that reason, its performance is heavily dependent on how data is organized in ADLS Gen2. For performance efficiency, evaluate the following points.

+- **Data ingestion:** Review how data is stored in the data lake. File sizes, the number of files, and folder structure all have an impact on performance. Keep in mind that while some file sizes might work for SQL serverless, they may impose issues for efficient processing or consumption by other engines or applications. You'll need to evaluate the data storage design and validate it against all of the data consumers, including SQL serverless and any other data tools that form part of your solution.

+- **Data placement:** Evaluate whether your design has unified and defined common patterns for data placement. Ensure that directory branching can support your security requirements. There are a few common patterns that can help you keep your time series data organized. Whatever your choice, ensure that it also works with other engines and workloads. Also, validate whether it can help partition auto-discovery for Spark applications and external tables.

+- **Data formats:** In most cases, SQL serverless will offer the best performance and better compatibility feature-wise by using a Parquet format. Verify your performance and compatibility requirements, because while Parquet improves performance - thanks to better compression and reduction of IO (by reading only required columns needed for analysis) - it requires more compute resources. Also, because some source systems don't natively support Parquet as an export format, it could lead to more transformation steps in your pipelines and/or dependencies in your overall architecture.

+- **Exploration:** Every industry is different. In many cases, however, there are common data access patterns found in the most frequent-run queries. Patterns typically involve filtering, and aggregations by dates, categories, or geographic regions. Identify your most common filtering criteria, and relate them to how much data is read/discarded by the most frequent-run queries. Validate whether the information on the data lake is organized to favor your exploration requirements and expectations. For the queries identified in your design and in your assessment, see whether you can eliminate unnecessary partitions in your OPENROWSET path parameter, or - if there are external tables - whether creating more indexes can help.

+### Reliability

+For reliability, evaluate the following points.

+- **Availability:** Validate any availability requirements that were identified during the [assessment stage](implementation-success-assess-environment.md). While there aren't any specific SLAs for SQL serverless, there's a 30-minute timeout for query execution. Identify the longest running queries from your assessment and validate them against your serverless SQL design. A 30-minute timeout could break the expectations for your workload and appear as a service problem.

+- **Consistency:** SQL serverless is designed primarily for read workloads. So, validate whether all consistency checks have been performed during the data lake data provisioning and formation process. Keep abreast of new capabilities, like [Delta Lake](/spark/apache-spark-what-is-delta-lake.md) open-source storage layer, which provides support for ACID (atomicity, consistency, isolation, and durability) guarantees for transactions. This capability allows you to implement effective [lambda or kappa architectures](/azure/architecture/data-guide/big-data/) to support both streaming and batch use cases. Be sure to evaluate your design for opportunities to apply new capabilities but not at the expense of your project's timeline or cost.

+- **Backup:** Review any disaster recovery requirements that were identified during the assessment. Validate them against your SQL serverless design for recovery. SQL serverless itself doesn't have its own storage layer and that would require handling snapshots and backup copies of your data. The data store accessed by serverless SQL is external (ADLS Gen2). Review the recovery design in your project for these datasets.

+### Security

+Organization of your data is important for building flexible security foundations. In most cases, different processes and users will require different permissions and access to specific sub areas of your data lake or logical data warehouse.

+For security, evaluate the following points.

+- **Data storage:** Using the information gathered during the [assessment stage](implementation-success-assess-environment.md), identify whether typical *Raw*, *Stage*, and *Curated* data lake areas need to be placed on the same storage account instead of independent storage accounts. The latter might result in more flexibility in terms of roles and permissions. It can also add more input/output operations per second (IOPS) capacity that might be needed if your architecture must support heavy and simultaneous read/write workloads (like real-time or IoT scenarios). Validate whether you need to segregate further by keeping your sandboxed and master data areas on separate storage accounts. Most users won't need to update or delete data, so they don't need write permissions to the data lake, except for sandboxed and private areas.

+- From your assessment information, identify whether any requirements rely on security features like [Always Encrypted](/sql/relational-databases/security/encryption/always-encrypted-database-engine?view=sql-server-ver15&viewFallbackFrom=azure-sqldw-latest&preserve-view=true), [Dynamic data masking](/azure/azure-sql/database/dynamic-data-masking-overview?view=azuresql&preserve-view=true) or [Row-level security](/sql/relational-databases/security/row-level-security?view=azure-sqldw-latest&preserve-view=true). Validate the availability of these features in specific scenarios, like when used with the OPENROWSET function. Anticipate potential workarounds that may be required.

+- From your assessment information, identify what would be the best authentication methods. Consider Azure Active Directory (Azure AD) service principals, shared access signature (SAS), and when and how authentication pass-through can be used and integrated in the exploration tool of choice of the customer. Evaluate the design and validate that the best authentication method as part of the design.

+### Other considerations

+Review your design and check whether you have put in place [best practices and recommendations](../sql/best-practices-serverless-sql-pool.md). Give special attention to filter optimization and collation to ensure that predicate pushdown works properly.

+## Next steps

+In the [next article](implementation-success-evaluate-spark-pool-design.md) in the *Azure Synapse success by design* series, learn how to evaluate your Spark pool design to identify issues and validate that it meets guidelines and requirements.

+ Title: "Synapse implementation success methodology: Evaluate solution development environment design"

+description: "Learn how to set up multiple environments for your modern data warehouse project to support development, testing, and production."

+# Synapse implementation success methodology: Evaluate solution development environment design

+Solution development and the environment within which it's performed is key to the success of your project. Regardless of your selected project methodology (like waterfall, Agile, or Scrum), you should set up multiple environments to support development, testing, and production. You should also define clear processes for promoting changes between environments.

+Setting up a modern data warehouse environment for both production and pre-production use can be complex. Keep in mind that one of the key design decisions is automation. Automation helps increase productivity while minimizing the risk of errors. Further, your environments should support future agile development, including the addition of new workloads, like data science or real-time. During the design review, produce a solution development environment design that will support your solution not only for the current project but also for ongoing support and development of your solution.

+## Solution development environment design 

+The environment design should include the production environment, which hosts the production solution, and at least one non-production environment. Most environments contain two non-production environments: one for development and another for testing, Quality Assurance (QA), and User Acceptance Testing (UAT). Typically, environments are hosted in separate Azure subscriptions. Consider creating a production subscription, and a non-production subscription. This separation will provide a clear security boundary and delineation between production and non-production.

+Ideally, you should establish three environments.

+- **Development:** The environment within which your data and analytics solutions are built. Determine whether to provide sandboxes for developers. Sandboxes can allow developers to make and test their changes in isolation, while a shared development environment will host integrated changes from the entire development team.

+- **Test/QA/UAT:** The production-like environment for testing deployments prior to their release to production.

+- **Production:** The final production environment.

+### Synapse workspaces

+For each Synapse workspace in your solution, the environment should include a production workspace and at least one non-production workspace for development and test/QA/UAT. Use the same name for all pools and artifacts across environments. Consistent naming will ease the promotion of workspaces to other environments.

+Promoting a workspace to another workspace is a two-part process:

+1. Use an [Azure Resource Manager template (ARM template)](../../azure-resource-manager/templates/overview.md) to create or update workspace resources.

+1. Migrate artifacts like SQL scripts, notebooks, Spark job definitions, pipelines, datasets, and data flows by using [Azure Synapse continuous integration and delivery (CI/CD) tools in Azure DevOps or on GitHub](../cicd/continuous-integration-delivery.md).

+### Azure DevOps or GitHub

+Ensure that integration with Azure DevOps or GitHub is properly set up. Design a repeatable process that releases changes across development, Test/QA/UAT, and production environments. 

+>[!IMPORTANT]

+> We recommend that sensitive configuration data always be stored securely in [Azure Key Vault](/azure/key-vault/general/basic-concepts.md). Use Azure Key Vault to maintain a central, secure location for sensitive configuration data, like database connection strings. That way, appropriate services can access configuration data from within each environment.

+## Next steps

+In the [next article](implementation-success-evaluate-team-skill-sets.md) in the *Azure Synapse success by design* series, learn how to evaluate your team of skilled resources that will implement your Azure Synapse solution.

+ Title: "Synapse implementation success methodology: Evaluate Spark pool design"

+description: "Learn how to evaluate your Spark pool design to identify issues and validate that it meets guidelines and requirements."

+# Synapse implementation success methodology: Evaluate Spark pool design

+You should evaluate your [Apache Spark pool](../spark/apache-spark-overview.md) design to identify issues and validate that it meets guidelines and requirements. By evaluating the design *before solution development begins*, you can avoid blockers and unexpected design changes. That way, you protect the project's timeline and budget.

+Apache Spark in Synapse brings the Apache Spark parallel data processing to Azure Synapse Analytics. This evaluation provides guidance on when Apache Spark in Azure Synapse is - or isn't - the right fit for your workload. It describes points to consider when you're evaluating your solution design elements that incorporate Spark pools.

+## Fit gap analysis

+When planning to implement Spark pools with Azure Synapse, first ensure they're the best fit for your workload.

+Consider the following points.

+- Does your workload require data engineering/data preparation?

+ - Apache Spark works best for workloads that require:

+ - Data cleaning.

+ - Transforming semi-structured data, like XML into relational.

+ - Complex free-text transformation, like fuzzy matching or natural language processing (NLP).

+ - Data preparation for machine learning (ML).

+- Does your workload for data engineering/data preparation involve complex or simple transformations? And, are you looking for a low-code/no-code approach?

+ - For simple transformations, like removing columns, changing column data types, or joining datasets, consider creating an Azure Synapse pipeline by using a data flow activity.

+ - Data flow activities provide a low-code/no-code approach to prepare your data.

+- Does your workload require ML on big data?

+ - Apache Spark works well for large datasets that will be used for ML. If you're using small datasets, consider using [Azure Machine Learning](../../machine-learning/overview-what-is-azure-ml.md) as the compute service.

+- Do you plan to perform data exploration or ad hoc query analysis on big data?

+ - Apache Spark in Azure Synapse provides Python/Scal).

+- Do you have a current Spark/Hadoop workload and do you need a unified big data platform?

+ - Azure Synapse provides a unified analytical platform for working with big data. There are Spark and SQL serverless pools for ad hoc queries, and the dedicated SQL pool for reporting and serving data.

+ - Moving from a Spark/Hadoop workload from on-premises (or another cloud environment) may involve some refactoring that you should take into consideration.

+ - If you're looking for a lift-and-shift approach of your Apache big data environment from on-premises to the cloud, and you need to meet a strict data engineering service level agreement (SLA), consider using [Azure HDInsight](../../hdinsight/hdinsight-overview.md).

+## Architecture considerations

+To ensure that your Apache Spark pool meets your requirements for operational excellence, performance, reliability, and security, there are key areas to validate in your architecture.

+### Operational excellence

+For operational excellence, evaluate the following points.

+- **Environment:** When configuring your environment, design your Spark pool to take advantage of features such as [autoscale and dynamic allocation](../spark/apache-spark-autoscale.md). Also, to reduce costs, consider enabling the [automatic pause](../spark/apache-spark-pool-configurations.md#automatic-pause) feature.

+- **Package management:** Determine whether required Apache Spark libraries will be used at a workspace, pool, or session level. For more information, see [Manage libraries for Apache Spark in Azure Synapse Analytics](../spark/apache-spark-azure-portal-add-libraries.md).

+- **Monitoring:** Apache Spark in Azure Synapse provides built-in monitoring of [Spark pools](../monitoring/how-to-monitor-spark-pools.md) and [applications](../monitoring/apache-spark-applications.md) with the creation of each spark session. Also consider implementing application monitoring with [Azure Log Analytics](../spark/apache-spark-azure-log-analytics.md) or [Prometheus and Grafana](../spark/use-prometheus-grafana-to-monitor-apache-spark-application-level-metrics.md), which you can use to visualize metrics and logs.

+### Performance efficiency

+For performance efficiency, evaluate the following points.

+- **File size and file type:** File size and the number of files have an impact on performance. Design the architecture to ensure that the file types are conducive to native ingestion with Apache Spark. Also, lean toward fewer large files instead of many small files.

+- **Partitioning:** Identify whether partitioning at the folder and/or file level will be implemented for your workload. *Folder partitions* limit the amount of data to search and read. *File partitions* reduce the amount of data to be searched inside the file - but only apply to specific file formats that should be considered in the initial architecture.

+### Reliability

+For reliability, evaluate the following points.

+- **Availability:** Spark pools have a start time of three to four minutes. It could take longer if there are many libraries to install. When designing batch vs. streaming workloads, identify the SLA for executing the job from your assessment information and determine which architecture best meets your needs. Also, take into consideration that each job execution creates a new Spark pool cluster.

+- **Checkpointing:** Apache Spark streaming has a built-in checkpointing mechanism. Checkpointing allows your stream to recover from the last processed entry should there be a failure on a node in your pool.

+### Security

+For security, evaluate the following points.

+- **Data access:** Data access must be considered for the Azure Data Lake Storage (ADLS) account that's attached to the Synapse workspace. In addition, determine the security levels required to access any data that isn't within the Azure Synapse environment. Refer to the information you collected during the [assessment stage](implementation-success-assess-environment.md).

+- **Networking:** Review the networking information and requirements gathered during your assessment. If the design involves a managed virtual network with Azure Synapse, consider the implications this requirement will have on Apache Spark in Azure Synapse. One implication is the inability to use Spark SQL while accessing data.

+## Next steps

+In the [next article](implementation-success-evaluate-project-plan.md) in the *Azure Synapse success by design* series, learn how to evaluate your modern data warehouse project plan before the project starts.

+For more information on best practices, see [Apache Spark for Azure Synapse Guidance](https://azuresynapsestorage.blob.core.windows.net/customersuccess/Guidance%20Video%20Series/EGUI_Synapse_Spark_Guidance.pdf).

+ Title: "Synapse implementation success methodology: Evaluate team skill sets"

+description: "Learn how to evaluate your team of skilled resources that will implement your Azure Synapse solution."

+# Synapse implementation success methodology: Evaluate team skill sets

+Solution development requires a team comprising individuals with many different skills. It's important for the success of your solution that your team has the necessary skills to successfully complete their assigned tasks. This evaluation takes an honest and critical look at the skill level of your project resources, and it provides you with a list of roles that are often required during the implementation of an Azure Synapse solution. Your team needs to possess relevant experience and skills to complete their assigned project tasks within the expected time frame.

+## Microsoft learning level definitions

+This article uses the Microsoft standard level definitions for describing learning levels.

+| Level | Description |

+|:-|:-|

+| 100 | Assumes little or no expertise with the topic, and covers topic concepts, functions, features, and benefits. |

+| 200 | Assumes 100-level knowledge and provides specific details about the topic. |

+| 300 | *Advanced material.* Assumes 200-level knowledge, in-depth understanding of features in a real-world environment, and strong coding skills. Provides a detailed technical overview of a subset of product/technology features, covering architecture, performance, migration, deployment, and development. |

+| 400 | *Expert material.* Assumes a deep level of technical knowledge and experience, and a detailed, thorough understanding of the topic. Provides expert-to-expert interaction and coverage of specialized topics. |

+## Roles, resources, and readiness

+Successfully delivering an Azure Synapse solution involves many different roles and skill sets. This topic describes roles commonly required to implement a successful project. Not all of these roles will be required for all projects, and not all of these roles will be required for the entire duration of the project. However, these roles will be required to complete some critical project tasks. You should evaluate the skill level of the individuals executing tasks to ensure their success in completing their job.

+Refer to your [project plan](implementation-success-evaluate-project-plan.md) and verify that these resources and roles were identified. Also, check to see if your project plan identifies other resources and roles. In many cases, you may find that individuals belong to more than one role. For example, the Azure administrator could also be your Azure network administrator. It's also possible that a role in your organization is split between multiple individuals. For example, the Synapse administrator doesn't get involved in Synapse SQL security. In this case, adjust your evaluation accordingly.

+Evaluate the following points.

+- Identify the roles that will be required by your solution implementation.

+- Identify the specific individuals in your project that will fulfill each role.

+- Identify the specific project tasks that will be performed by each individual.

+- Assign a [learning level](#microsoft-learning-leveldefinitions) to each individual for their tasks and roles.

+Typically, a successful implementation requires that each individual has at least a level-300 proficiency for the tasks they'll perform. It's highly recommended that individuals at level-200 (or below) be provided with guidance and instruction to raise their level of understanding prior to beginning their project tasks. In this case, involve a level-300 (or above) individual to mentor and review. It's recommended that you adjust the project plan timeline and effort estimates to factor in learning new skills.

+> [!NOTE]

+> We recommend you align your roles with the built-in roles. There are two sets of built-in roles: [RBAC roles for Azure Synapse](../security/synapse-workspace-synapse-rbac-roles.md) and [RBAC roles built into Azure](../../role-based-access-control/built-in-roles.md). These two sets of built-in roles and permissions are independent.

+### Azure administrator

+The *Azure administrator* manages administrative aspects of Azure. They're responsible for subscriptions, region identification, resource groups, monitoring, and portal access. They also provision resources, like resource groups, storage accounts, Azure Data Factory (ADF), Microsoft Purview, and more.

+### Security administrator

+The *security administrator* must have local knowledge of the existing security landscape and requirements. This role collaborates with the [Synapse administrator](#synapse-administrator), [Synapse database administrator](#synapse-database-administrator), [Synapse Spark administrator](#synapse-spark-administrator), and other roles to set up security requirements. The security administrator could also be an Azure Active Directory (Azure AD) administrator.

+### Network administrator

+The *network administrator* must have local knowledge of the existing networking landscape and requirements. This role requires Azure networking skills and Synapse networking skills.

+### Synapse administrator

+The *Synapse administrator* is responsible for the administration of the overall Azure Synapse environment. This role is responsible for the availability and scale of workspace resources, data lake administration, analytics runtimes, and workspace administration and monitoring. This role works closely with all other roles to ensure access to Azure Synapse, the availability of analytics services, and sufficient scale. Other responsibilities include:

+- Provision Synapse workspaces.

+- Set up Azure Synapse networking and security requirements.

+- Monitor Synapse workspace activity.

+### Synapse database administrator

+The *Synapse database administrator* is responsible for the design, implementation, maintenance, and operational aspects of the SQL pools (serverless and dedicated). This role is responsible for the overall availability, consistent performance, and optimizations of the SQL pools. This role is also responsible for managing the security of the data in the databases, granting privileges over the data, and granting or denying user access. Other responsibilities include:

+- Perform various dedicated SQL pool administration functions, like provisioning, scale, pauses, resumes, restores, workload management, monitoring, and others.

+- Perform various dedicated SQL pool administration functions, like securing, monitoring, and others.

+- Set up SQL pool database security.

+- Performance tuning and troubleshooting.

+### Synapse Spark administrator

+The *Synapse Spark administrator* is responsible for the design, implementation, maintenance, and operational aspects of the Spark pools. This role is responsible for the overall availability, consistent performance, and optimizations of the Spark pools. This role is also responsible for managing the security of the data, granting privileges over the data, and granting or denying user access. Other responsibilities include:

+- Perform various dedicated Spark pool administration functions, like provisioning, monitoring, and others.

+- Set up Spark pool data security.

+- Notebook troubleshooting and performance.

+- Pipeline Spark execution troubleshooting and performance.

+### Synapse SQL pool database developer

+The *Synapse pool database developer* is responsible for database design and development. For dedicated SQL pools, responsibilities include table structure and indexing, developing database objects, and schema design. For serverless SQL pools, responsibilities include external tables, views, and schema design. Other responsibilities include:

+- Logical and physical database design.

+- Table design, including distribution, indexing, and partitioning.

+- Programming object design and development, including stored procedures and functions.

+- Design and development of other performance optimizations, including materialized views, workload management, and more.

+- Design and implementation of [data protection](security-white-paper-data-protection.md), including data encryption.

+- Design and implementation of [access control](security-white-paper-access-control.md), including object-level security, row-level security, column-level security, dynamic data masking, and Synapse role-based access control.

+- Monitoring, auditing, performance tuning and troubleshooting.

+### Spark developer

+The *Spark developer* is responsible for creating notebooks and executing Spark processing by using Spark pools.

+### Data integration administrator

+The *Data integration administrator* is responsible for setting up and securing data integration by using Synapse pipelines, ADF, or third-party integration tools, and for performing all configuration and security functions to support the data integration tools.

+For Synapse pipelines and ADF, other responsibilities include setting up the integration runtime (IR), self-hosted integration runtime (SHIR), and/or SSIS integration runtime (SSIS-IR). Knowledge of virtual machine provisioning - on-premises or in Azure - may be required.

+### Data integration developer

+The *Data integration developer* is responsible for developing ETL/ELT and other data integration processes by using the solution's selected data integration tools.

+### Data consumption tools administrator

+The *Data consumption tools administrator* is responsible for the data consumption tools. Tools can include [Microsoft Power BI](https://powerbi.microsoft.com/), Microsoft Excel, Tableau, and others. The administrator of each tool will need to set up permissions to grant access to data in Azure Synapse.

+### Data engineer

+The *Data engineer* role is responsible for implementing data-related artifacts, including data ingestion pipelines, cleansing and transformation activities, and data stores for analytical workloads. It involves using a wide range of data platform technologies, including relational and non-relational databases, file stores, and data streams.

+Data engineers are responsible for ensuring that the privacy of data is maintained within the cloud, and spanning from on-premises to the cloud data stores. They also own the management and monitoring of data stores and data pipelines to ensure that data loads perform as expected.

+### Data scientist

+The *Data scientist* derives value and insights from data. Data scientists find innovative ways to work with data and help teams achieve a rapid return on investment (ROI) on analytics efforts. They work with data curation and advanced search, matching, and recommendation algorithms. Data scientists need access to the highest quality data and substantial amounts of computing resources to extract deep insights.

+### Data analyst

+The *Data analyst* enables businesses to maximize the value of their data assets. They transform raw data into relevant insights based on identified business requirements. Data analysts are responsible for designing and building scalable data models, cleaning, and transforming data, and presenting advanced analytics in reports and visualizations.

+### Azure DevOps engineer

+The *Azure DevOps engineer* is responsible for designing and implementing strategies for collaboration, code, infrastructure, source control, security, compliance, continuous integration, testing, delivery, and monitoring of an Azure Synapse project.

+## Learning resources and certifications

+If you're interested to learn about Microsoft Certifications that may help assess your team's readiness, browse the available certifications for [Azure Synapse Analytics](/learn/certifications/browse/?expanded=azure&products=azure-synapse-analytics).

+To complete online, self-paced training, browse the available learning paths and modules for [Azure Synapse Analytics](/learn/browse/?filter-products=synapse&products=azure-synapse-analytics).

+## Next steps

+In the [next article](implementation-success-perform-operational-readiness-review.md) in the *Azure Synapse success by design* series, learn how to perform an operational readiness review to evaluate your solution for its preparedness to provide optimal services to users.

+ Title: "Synapse implementation success methodology: Evaluate workspace design"

+description: "Learn how to evaluate the Synapse workspace design and validate that it meets guidelines and requirements."

+# Synapse implementation success methodology: Evaluate workspace design

+Synapse workspace is a unified graphical user experience that stitches together your analytical and data processing engines, data lakes, databases, tables, datasets, and reporting artifacts along with code and process orchestration. Considering the number of technologies and services that are integrated into Synapse workspace, ensure that the key components are included in your design.

+## Synapse workspace design review

+Identify whether your solution design involves one Synapse workspace or multiple workspaces. Determine the drivers of this design. While there might be different reasons, in most cases the reason for multiple workspaces is either security segregation or billing segregation. When determining the number of workspaces and database boundaries, keep in mind that there's a limit of 20 workspaces per subscription.

+Identify which elements or services within each workspace need to be shared and with which resources. Resources can include data lakes, integration runtimes (IRs), metadata or configurations, and code. Determine why this particular design was chosen in terms of potential synergies. Ask yourself whether these synergies justify the extra cost and management overhead.

+## Data lake design review

+We recommended that the data lake (if part of your solution) be properly tiered. You should divide your data lake into three major areas that relate to *Bronze*, *Silver*, and *Gold* datasets. Bronze - or the raw layer - might reside on its own separate storage account because it has stricter access controls due to unmasked sensitive data that it might store.

+## Security design review

+Review the security design for the workspace and compare it with the information you gathered during the assessment. Ensure all of the requirements are met, and all of the constraints have been taken into account. For ease of management, we recommended that users be organized into groups with appropriate permissions profiling: you can simplify access control by using security groups that align with roles. That way, network administrators can add or remove users from appropriate security groups to manage access.

+Serverless SQL pools and Apache Spark tables store their data in an Azure Data Lake Gen2 (ADLS Gen2) container that's associated with the workspace. User-installed Apache Spark libraries are also managed in this same storage account. To enable these use cases, both users and the workspace managed service identity (MSI) must be added to the **Storage Blob Data Contributor** role of the ADLS Gen2 storage container. Verify this requirement against your security requirements.

+Dedicated SQL pools provide a rich set of security features to encrypt and mask sensitive data. Both dedicated and serverless SQL pools enable the full surface area of SQL Server permissions including built-in roles, user-defined roles, SQL authentication, and Azure Active Directory (Azure AD) authentication. Review the security design for your solution's dedicated SQL pool and serverless SQL pool access and data.

+Review the security plan for your data lake and all the ADLS Gen2 storage accounts (and others) that will form part of your Azure Synapse Analytics solution. ADLS Gen2 storage isn't itself a compute engine and so it doesn't have a built-in ability to selectively mask data attributes. You can apply ADLS Gen2 permissions at the storage account or container level by using role-based access control (RBAC) and/or at the folder or file level by using access control lists (ACLs). Review the design carefully and strive to avoid unnecessary complexity.

+Here are some points to consider for the security design.

+- Make sure Azure AD set up requirements are included in the design.

+- Check for cross-tenant scenarios. Such issues may arise because some data is in another Azure tenant, or it needs to move to another tenant, or it needs to be accessed by users from another tenant. Ensure these scenarios are considered in your design.

+- What are the roles for each workspace? How will they use the workspace?

+- How is the security designed within the workspace?

+ - Who can view all scripts, notebooks, and pipelines?

+ - Who can execute scripts and pipelines?

+ - Who can create/pause/resume SQL and Spark pools?

+ - Who can publish changes to the workspace?

+ - Who can commit changes to source control?

+- Will pipelines access data by using stored credentials or the workspace managed identity?

+- Do users have the appropriate access to the data lake to browse the data in Synapse Studio?

+- Is the data lake properly secured by using the appropriate combination of RBAC and ACLs?

+- Have the SQL pool user permissions been correctly set for each role (data scientist, developer, administrator, business user, and others)?

+## Networking design review

+Here are some points to consider for the network design.

+- Is connectivity designed between all the resources?

+- What is the networking mechanism to be used (Azure ExpressRoute, public Internet, or private endpoints)?

+- Do you need to be able to securely connect to Synapse Studio?

+- Has data exfiltration been taken into consideration?

+- Do you need to connect to on-premises data sources?

+- Do you need to connect to other cloud data sources or compute engines, such as Azure Machine Learning?

+- Have Azure networking components, like network security groups (NSGs), been reviewed for proper connectivity and data movement?

+- Has integration with the private DNS zones been taken into consideration?

+- Do you need to be able to browse the data lake from within Synapse Studio or simply query data in the data lake with serverless SQL or PolyBase?

+Finally, identify all of your data consumers and verify that their connectivity is accounted for in the design. Check that network and security outposts allow your service to access required on-premises sources and that its authentication protocols and mechanisms are supported. In some scenarios, you might need to have more than one self-hosted IR or data gateway for SaaS solutions, like Microsoft Power BI.

+## Monitoring design review

+Review the design of the monitoring of the Azure Synapse components to ensure they meet the requirements and expectations identified during the assessment. Verify that monitoring of resources and data access has been designed, and that it identifies each monitoring requirement. A robust monitoring solution should be put in place as part of the first deployment to production. That way, failures can be identified, diagnosed, and addressed in a timely manner. Aside from the base infrastructure and pipeline runs, data should also be monitored. Depending on the Azure Synapse components in use, identify the monitoring requirements for each component. For example, if Spark pools form part of the solution, monitor the malformed record store. 

+Here are some points to consider for the monitoring design.

+- Who can monitor each resource type (pipelines, pools, and others)?

+- How long do database activity logs need to be retained?

+- Will workspace and database log retention use Log Analytics or Azure Storage?

+- Will alerts be triggered in the event of a pipeline error? If so, who should be notified?

+- What threshold level of a SQL pool should trigger an alert? Who should be notified?

+## Source control design review

+By default, a Synapse workspace applies changes directly to the Synapse service by using the built-in publish functionality. You can enable source control integration, which provides many advantages. Advantages include better collaboration, versioning, approvals, and release pipelines to promote changes through to development, test, and production environments. Azure Synapse allows a single source control repository per workspace, which can be either Azure DevOps Git or GitHub.

+Here are some points to consider for the source control design.

+- If using Azure DevOps Git, is the Synapse workspace and its repository in the same tenant?

+- Who will be able to access source control?

+- What permissions will each user be granted in source control?

+- Has a branching and merging strategy been developed?

+- Will release pipelines be developed for deployment to different environments?

+- Will an approval process be used for merging and for release pipelines?

+> [!NOTE]

+> The design of the development environment is of critical importance to the success of your project. If a development environment has been designed, it will be evaluated in a [separate stage of this methodology](implementation-success-evaluate-solution-development-environment-design.md).

+## Next steps

+In the [next article](implementation-success-evaluate-data-integration-design.md) in the *Azure Synapse success by design* series, learn how to evaluate the data integration design and validate that it meets guidelines and requirements.

+ Title: Azure Synapse implementation success by design

+description: "Learn about the Azure Synapse success series of articles that's designed to help you deliver a successful implementation of Azure Synapse Analytics."

+# Azure Synapse implementation success by design

+The *Azure Synapse implementation success by design* series of articles is designed to help you deliver a successful implementation of Azure Synapse Analytics. It describes a methodology to complement your solution implementation project. It includes suggested checks at strategic points during your project that can help assure a successful implementation. It's important to understand that the methodology shouldn't replace or change your chosen project management methodology (Scrum, Agile, or waterfall). Rather, it suggests validations that can improve the success of your project deployment to a production environment.

+[Azure Synapse](../overview-what-is.md) is an enterprise analytics service that accelerates time to insight across data warehouses and big data systems. It brings together the best of SQL technologies used in enterprise data warehousing, Spark technologies used for big data, pipelines for data integration and ETL/ELT, and deep integration with other Azure services, such as Power BI, Azure Cosmos DB, and Azure Machine Learning.

+The methodology uses a strategic checkpoint approach to assess and monitor the progress of your project. The goals of these checkpoints are:

+- Proactive identification of possible issues and blockers.

+- Continuous validation of the solution's fit to the use cases.

+- Successful deployment to production.

+- Smooth operation and monitoring once in production.

+The checkpoints are invoked at four milestones during the project:

+1. [Project planning](#project-planning-checkpoint)

+1. [Solution development](#solution-development-checkpoint)

+1. [Pre go-live](#pre-go-live-checkpoint)

+1. [Post go-live](#post-go-live-checkpoint)

+## Project planning checkpoint

+The project planning checkpoint includes the solution evaluation, project plan evaluation, the solution development environment design evaluation, and the team skill sets evaluation.

+#### Solution evaluation

+Evaluate your entire solution with a focus on how it intends to use Azure Synapse. An assessment involves gathering data that will identify the required components of Azure Synapse, the interfaces each will have with other products, a review of the data sources, the data consumers, the roles, and use cases. This assessment will also gather data about the existing environment including detailed specifications from existing data warehouses, big data environments, and integration and data consumption tooling. The assessment will identify which Azure Synapse components will be implemented and therefore which evaluations and checkpoints should be made throughout the implementation effort. This assessment will also provide additional information to validate the design and implementation against requirements, constraints, and assumptions.

+Here's a list of tasks you should complete.

+1. [Assess](implementation-success-assess-environment.md) your environment to help evaluate the solution design.

+1. Make informed technology decisions to implement Azure Synapse and identify the solution components to implement.

+1. [Evaluate the workspace design](implementation-success-evaluate-workspace-design.md).

+1. [Evaluate the data integration design](implementation-success-evaluate-data-integration-design.md).

+1. [Evaluate the dedicated SQL pool design](implementation-success-evaluate-dedicated-sql-pool-design.md).

+1. [Evaluate the serverless SQL pool design](implementation-success-evaluate-serverless-sql-pool-design.md).

+1. [Evaluate the Spark pool design](implementation-success-evaluate-spark-pool-design.md).

+1. Review the results of each evaluation and respond accordingly.

+#### Project plan evaluation

+Evaluate the project plan as it relates to the Azure Synapse requirements that need to be developed. This evaluation isn't about producing a project plan. Rather, the evaluation is about identifying any steps that could lead to blockers or that could impact on the project timeline. Once evaluated, you may need to make adjustments to the project plan.

+Here's a list of tasks you should complete.

+1. [Evaluate the project plan](implementation-success-evaluate-project-plan.md).

+1. Evaluate project planning specific to the Azure Synapse components you plan to implement.

+1. Review the results of each evaluation and respond accordingly.

+#### Solution development environment design evaluation

+Evaluate the environment that's to be used to develop the solution. Establish separate development, test, and production environments. Also, it's important to understand that setting up automated deployment and source code control is essential to a successful and smooth development effort.

+Here's a list of tasks you should complete.

+1. [Evaluate the solution development environment design](implementation-success-evaluate-solution-development-environment-design.md).

+1. Review the results of each evaluation and respond accordingly.

+#### Team skill sets evaluation

+Evaluate the project team with a focus on their skill level and readiness to implement the Azure Synapse solution. The success of the project depends on having the correct skill sets and experience. Many different skill sets are required to implement an Azure Synapse solution, so ensure you identify gaps and secure suitable resources that have the required skill sets (or arrange for them to complete training). This evaluation is critical at this stage of your project because a lack of the proper skills can impact on both the timeline and the overall success of the project.

+Here's a list of tasks you should complete.

+1. [Evaluate the team skill sets](implementation-success-evaluate-team-skill-sets.md).

+1. Secure skilled resources, or upskill resources to expand their capabilities.

+1. Review the results of each evaluation and respond accordingly.

+### Solution development checkpoint

+The solution development checkpoint includes periodic quality checks and additional skill building.

+#### Periodic quality checks

+During solution development, you should make periodic checks to validate that the solution is being developed according to recommended practices. Check that the project use cases will be satisfied and that enterprise requirements are being met. For the purposes of this methodology, these checks are called *periodic quality checks*.

+Implement the following quality checks:

+- Quality checks for workspaces.

+- Quality checks for data integration.

+- Quality checks for dedicated SQL pools.

+- Quality checks for serverless SQL pools.

+- Quality checks for Spark pools.

+#### Additional skill building

+As the project progresses, identify whether more skill sets are needed. Take the time to determine whether more skill sets could improve the quality of the solution. Supplementing the team with more skill sets can help to avoid project delays and project timeline impacts.

+### Pre go-live checkpoint

+Before deploying your solution to production, we recommend that you perform reviews to assess the preparedness of the solution.

+The *pre go-live* checklist provides a final readiness check to successfully deploy to production.

+1. [Perform the operational readiness review](implementation-success-perform-operational-readiness-review.md).

+1. [Perform the user readiness and onboarding plan review](implementation-success-perform-user-readiness-and-onboarding-plan-review.md).

+1. Review the results of each review and respond accordingly.

+### Post go-live checkpoint

+After deploying to production, we recommend that you validate that the solution operates as expected.

+The *post go-live* checklist provides a final readiness check to monitor your Azure Synapse solution.

+1. [Perform the monitoring review](implementation-success-perform-monitoring-review.md).

+1. Continually monitor your Azure Synapse solution.

+## Next steps

+In the [next article](implementation-success-assess-environment.md) in the *Azure Synapse implementation success by design* series, learn how to assess your environment to help evaluate the solution design and make informed technology decisions to implement Azure Synapse.

+ Title: "Synapse implementation success methodology: Perform monitoring review"

+description: "Learn how to perform monitoring of your Azure Synapse solution."

+# Synapse implementation success methodology: Perform monitoring review

+Monitoring is a key part of the operationalization of any Azure solution. This article provides guidance on reviewing and configuring the monitoring of your Azure Synapse Analytics environment. Key to this activity is the identification of what needs to be monitored and who needs to review the monitoring results.

+Using your solution requirements and other data collected during the [assessment stage](implementation-success-assess-environment.md) and [solution development](implementation-success-evaluate-solution-development-environment-design.md), build a list of important behaviors and activities that need to be monitored in your production environment. As you build this list, identify the groups of users that will need access to monitoring information and build the procedures they can follow to respond to monitoring results.

+You can use [Azure Monitor](/azure/azure-monitor/overview) to provide base-level infrastructure metrics, alerts, and logs for most Azure services. Azure diagnostic logs are emitted by a resource to provide rich, frequent data about the operation of that resource. Azure Synapse can write diagnostic logs in Azure Monitor.

+For more information, see [Use Azure Monitor with your Azure Synapse Analytics workspace](../monitoring/how-to-monitor-using-azure-monitor.md).

+## Monitor dedicated SQL pools

+You can monitor a dedicated SQL pool by using Azure Monitor, altering, dynamic management views (DMVs), and Log Analytics.

+- **Alerts:** You can set up alerts that send you an email or call a webhook when a certain metric reaches a predefined threshold. For example, you can receive an alert email when the database size grows too large. For more information, see [Create alerts for Azure SQL Database and Azure Synapse Analytics using the Azure portal](/azure/azure-sql/database/alerts-insights-configure-portal).

+- **DMVs:** You can use [DMVs](../sql-data-warehouse/sql-data-warehouse-manage-monitor.md) to monitor workloads to help investigate query executions in SQL pools.

+- **Log Analytics:** [Log Analytics](/azure/azure-monitor/logs/log-analytics-tutorial) is a tool in the Azure portal that you can use to edit and run log queries from data collected by Azure Monitor. For more information, see [Monitor workload - Azure portal](../sql-data-warehouse/sql-data-warehouse-monitor-workload-portal.md).

+## Monitor serverless SQL pools

+You can monitor a serverless SQL pool by [monitoring your SQL requests](../monitoring/how-to-monitor-sql-requests.md) in Synapse Studio. That way, you can keep an eye on the status of running requests and review details of historical requests.

+## Monitor Spark pools

+You can [monitor your Apache Spark applications](../monitoring/apache-spark-applications.md) in Synapse Studio. That way, you can keep an eye on the latest status, issues, and progress.

+You can enable the Synapse Studio connector that's built in to Log Analytics. You can then collect and send Apache Spark application metrics and logs to your Log Analytics workspace. You can also use an Azure Monitor workbook to visualize the metrics and logs. For more information, see [Monitor Apache Spark applications with Azure Log Analytics](../spark/apache-spark-azure-log-analytics.md).

+## Monitor pipelines

+ Azure Synapse allows you to create complex pipelines that automate and integrate your data movement, data transformation, and compute activities. You can author and monitor pipelines by using Synapse Studio to keep an eye on the latest status, issues, and progress of your pipelines. For more information, see [Use Synapse Studio to monitor your workspace pipeline runs](../monitoring/how-to-monitor-pipeline-runs.md).

+## Next steps

+For more information about this article, check out the following resources:

+- [Synapse implementation success methodology](implementation-success-overview.md)

+- [Use Azure Monitor with your Azure Synapse Analytics workspace](../monitoring/how-to-monitor-using-azure-monitor.md)

+ Title: "Synapse implementation success methodology: Perform operational readiness review"

+description: "Learn how to perform an operational readiness review to evaluate your solution for its preparedness to provide optimal services to users."

+# Synapse implementation success methodology: Perform operational readiness review

+Once you build an Azure Synapse Analytics solution and it's ready to deploy, it's important to ensure the operational readiness of that solution. Performing an operational readiness review evaluates the solution for its preparedness to provide optimal services to users. Organizations that invest time and resources in assessing operational readiness before launch have a much higher rate of success. It's also important to conduct an operational readiness review periodically post deployment - perhaps annually - to ensure there isn't any drift from operational expectations.

+## Process and focus areas

+Process and focus areas include service operational goals, solution readiness, security, monitoring, high availability (HA) and disaster recovery (DR).

+### Service operational goals

+ Document service expectations from the customer's point of view, and get buy-in from the business on these service expectations. Make any necessary modifications to meet business goals and objectives of the service.

+The service level agreement (SLA) of each Azure service varies based on the service. For example, Microsoft guarantees a specific monthly uptime percentage. For more information, see [SLA for Azure Synapse Analytics](https://azure.microsoft.com/support/legal/sla/synapse-analytics/). Ensure these SLAs align with your own business SLAs and document any gaps. It's also important to define any operational level agreements (OLAs) between different teams and ensure that they align with the SLAs.

+### Solution readiness

+It's important to review solution readiness by using the following points.

+- Describe the entire solution architecture calling out critical functionalities of different components and how they interact with each other.

+- Document scalability aspects of your solution. Include specific details about the effort involved in scaling and the impact of it on business. Consider whether it can respond to sudden surges of user activity. Bear in mind that Azure Synapse provides functionality for scaling with minimal downtime.

+- Document any single points of failure in your solution, along with how to recover should such failures occur. Include the impact of such failures on dependent services in order to minimize the impact.

+- Document all dependent services on the solution and their impact.

+### Security

+Data security and privacy are non-negotiable. Azure Synapse implements a multi-layered security architecture for end-to-end protection of your data. Review security readiness by using the following points.

+- **Authentication:** Ensure Azure Active Directory (Azure AD) authentication is used whenever possible. If non-Azure AD authentication is used, ensure strong password mechanisms are in place and that passwords are rotated on a regular basis. For more information, see [Password Guidance](https://www.microsoft.com/research/publication/password-guidance/). Ensure monitoring is in place to detect suspicious actions related to user authentication. Consider using [Azure Identity Protection](/azure/active-directory/identity-protection/overview-identity-protection) to automate the detection and remediation of identity-based risks.

+- **Access control:** Ensure proper access controls are in place following the [principle of least privilege](/azure/active-directory/develop/secure-least-privileged-access). Use security features available with Azure services to strengthen the security of your solution. For example, Azure Synapse provides granular security features, including row-level security (RLS), column-level security, and dynamic data masking. For more information, see [Azure Synapse Analytics security white paper: Access control](security-white-paper-access-control.md).

+- **Threat protection:** Ensure proper threat detection mechanisms are place to prevent, detect, and respond to threats. Azure Synapse provides SQL Auditing, SQL Threat Detection, and Vulnerability Assessment to audit, protect, and monitor databases. For more information, see [Azure Synapse Analytics security white paper: Threat detection](security-white-paper-threat-protection.md).

+For more information, see the [Azure Synapse Analytics security white paper](security-white-paper-introduction.md).

+### Monitoring

+Set and document expectations for monitoring readiness with your business. These expectations should describe:

+- How to monitor the entire user experience, and whether it includes monitoring of a single-user experience.

+- The specific metrics of each service to monitor.

+- How and who to notify about poor user experience.

+- Details of proactive health checks.

+- Any mechanisms that are in place that automate actions in response to incidents, for example, raising tickets automatically.

+Consider using [Azure Monitor](/azure/azure-monitor/overview) to collect, analyze, and act on telemetry data from your Azure and on-premises environments. Azure Monitor helps you maximize performance and availability of your applications by proactively identify problems in seconds.

+List all the important metrics to monitor for each service in your solution along with their acceptable thresholds. For example, the following list includes important metrics to monitor for a dedicated SQL pool:

+- `DWULimit`

+- `DWUUsed`

+- `AdaptiveCacheHitPercent`

+- `AdaptiveCacheUsedPercent`

+- `LocalTempDBUsedPercent`

+- `ActiveQueries`

+- `QueuedQueries`

+Consider using [Azure Service Health](https://azure.microsoft.com/features/service-health/) to notify you about Azure service incidents and planned maintenance. That way, you can take action to mitigate downtime. You can set up customizable cloud alerts and use a personalized dashboard to analyze health issues, monitor the impact to your cloud resources, get guidance and support, and share details and updates.

+Lastly, ensure proper notifications are set up to notify appropriate people when incidents occur. Incidents could be proactive, such as when a certain metric exceeds a threshold, or reactive, such as a failure of a component or service. For more information, see [Overview of alerts in Microsoft Azure](/azure/azure-monitor/alerts/alerts-overview).

+### High availability

+Define and document *recovery time objective (RTO)* and *recovery point objective (RPO)* for your solution. RTO is how soon the service will be available to users, and RPO is how much data loss would occur in the event of a failover.

+Each of the Azure services publishes a set of guidelines and metrics on the expected high availability (HA) of the service. Ensure these HA metrics align with your business expectations. when they don't align, customizations may be necessary to meet your HA requirements. For example, Azure Synapse dedicated SQL pool supports an eight-hour RPO with automatic restore points. If that RPO isn't sufficient, you can set up user-defined restore points with an appropriate frequency to meet your RPO needs. For more information, see [Backup and restore in Azure Synapse dedicated SQL pool](../sql-data-warehouse/backup-and-restore.md).

+### Disaster recovery

+Define and document a detailed process for disaster recovery (DR) scenarios. DR scenarios can include a failover process, communication mechanisms, escalation process, war room setup, and others. Also document the process for identifying the causes of outages and the steps to take to recover from disasters.

+Use the built-in DR mechanisms available with Azure services for building your DR process. For example, Azure Synapse performs a standard geo-backup of SQL dedicated pools once every day to a paired data center. You can use a geo-backup to recover from a disaster at the primary location. You can also set up Azure Data Lake Storage (ADLS) to copy data to another Azure region that's hundreds of miles apart. If there's a disaster at the primary location, a failover can be initiated to transform the secondary storage location into the primary storage location. For more information, see [Disaster recovery and storage account failover](/azure/storage/common/storage-disaster-recovery-guidance).

+## Next steps

+In the [next article](implementation-success-perform-user-readiness-and-onboarding-plan-review.md) in the *Azure Synapse success by design* series, learn how to perform monitoring of your Azure Synapse solution.

+ Title: "Synapse implementation success methodology: Perform user readiness and onboarding plan review"

+description: "Learn how to perform user readiness and onboarding of new users to ensure successful adoption of your data warehouse."

+# Synapse implementation success methodology: Perform user readiness and onboarding plan review

+Training technical people, like administrators and developers, is important to deliver success. Don't overlook that you must also extend training to include end users. Review the use cases and roles identified during the [assessment stage](implementation-success-assess-environment.md), [project planning](implementation-success-evaluate-project-plan.md), and [solution development](implementation-success-evaluate-solution-development-environment-design.md) to ensure that *everyone* is readied for success.

+Evaluate your project plan and prepare an onboarding plan for each group of users, including:

+- Big data analytics users.

+- Structured data analytics users.

+- Users of each of your identified data consumption tools.

+- Operations support.

+- Help desk and user support.

+## Onboarding and readiness

+It's unrealistic to expect that users figure out how to use Azure Synapse, even when they have experience with similar technologies. So, plan to reach out to your users to ensure a smooth transition for them to the new environment. Specifically, ensure that:

+- Users understand what Azure Synapse does and how it does it.

+- Users understand how to use the Azure Synapse service or the platform that uses it.

+- Onboarding of users is a consistent and continuous process.

+- Users see and understand the value of the new environment.

+The onboarding of users starts with explanatory sessions or technical workshops. It also involves giving them access to the platform. The onboarding process can span several months depending on the complexity of the solution, and it should set the right tone for future interactions with the Azure Synapse platform and services.

+Take tracking steps and make sure users are capable of completing a set of core tasks that will form part of their daily operations. These tasks will be specific to different groups of user groups, roles, and use cases. Be sure to identify:

+- The core actions users need to be able to perform.

+- The steps users must take to perform each action.

+Focus your efforts on the tasks that users struggle the most with. Be sure to provide straightforward instructions and processes, yet be mindful of how long it takes for users to complete specific tasks. It's always a good idea to request qualitative feedback to better track user readiness and improve the onboarding experience.

+## Next steps

+In the [next article](implementation-success-perform-monitoring-review.md) in the *Azure Synapse success by design* series, learn how to monitor your Azure Synapse solution.

+ Title: "Synapse POC playbook: Data warehousing with dedicated SQL pool in Azure Synapse Analytics"

+description: "A high-level methodology for preparing and running an effective Azure Synapse Analytics proof of concept (POC) project for dedicated SQL pool."

+# Synapse POC playbook: Data warehousing with dedicated SQL pool in Azure Synapse Analytics

+This article presents a high-level methodology for preparing and running an effective Azure Synapse Analytics proof of concept (POC) project for dedicated SQL pool.

+> [!TIP]

+> If you're new to dedicated SQL pools, we recommend you work through the [Work with Data Warehouses using Azure Synapse Analytics](/learn/paths/work-with-data-warehouses-using-azure-synapse-analytics/) learning path.

+## Prepare for the POC

+Before deciding on your Azure Synapse POC goals, we recommend that you first read the [Azure Synapse SQL architecture](../sql/overview-architecture.md) article to familiarize yourself with how a dedicated SQL pool separates compute and storage to provide industry-leading performance.

+### Identify sponsors and potential blockers

+Once you're familiar with Azure Synapse, it's time to make sure that your POC has the necessary support and won't hit any roadblocks. You should:

+- Identify any restrictions or policies that your organization has about moving data to, and storing data in, the cloud.

+- Identify executive and business sponsorship for a cloud-based data warehouse project.

+- Verify that your workload is appropriate for Azure Synapse. For more information, see [Dedicated SQL pool architecture in Azure Synapse Analytics](../sql-data-warehouse/massively-parallel-processing-mpp-architecture.md).

+### Set the timeline

+A POC is a scoped, time-bounded exercise with specific, measurable goals and metrics that define success. Ideally, it should have some basis in business reality so that the results are meaningful.

+POCs have the best outcome when they're *timeboxed*. Timeboxing allocates a fixed and maximum unit of time to an activity. In our experience, two weeks provides enough time to complete the work without the burden of too many use cases or complex test matrices. Working within this fixed time period, we suggest that you follow this timeline:

+1. **Data loading:** Three days or less

+1. **Querying:** Five days or less

+1. **Value added tests:** Two days or less

+Here are some tips:

+> [!div class="checklist"]

+> - Make realistic estimates of the time that you will require to complete the tasks in your plan.

+> - Recognize that the time to complete your POC will be related to the size of your dataset, the number of database objects (tables, views, and stored procedures), the complexity of the database objects, and the number of interfaces you will test.

+> - If you estimate that your POC will run longer than four weeks, consider reducing the scope to focus only on the most important goals.

+> - Get support from all the lead resources and sponsors for the timeline before commencing the POC.

+Once you've determined that there aren't any immediate obstacles and you've set the timeline, you can scope a high-level architecture.

+### Create a high-level scoped architecture

+A high-level future architecture likely contains many data sources and data consumers, big data components, and possibly machine learning and AI data consumers. To keep your POC goals achievable (and within the bounds of your set timeline), decide which of these components will form part of the POC and which will be excluded.

+Additionally, if you're already using Azure, identify the following:

+- Any existing Azure resources that you can use during the POC. For example, resources can include Azure Active Directory (Azure AD), or Azure ExpressRoute.

+- What Azure region(s) your organization prefers.

+- A subscription you can use for non-production POC work.

+- The throughput of your network connection to Azure.

+ > [!IMPORTANT]

+ > Be sure to check that your POC can consume some of that throughput without having an adverse effect on production solutions.

+### Apply migration options

+If you're migrating from a legacy data warehouse system to Azure Synapse, here are some questions to consider:

+- Are you migrating and want to make as few changes to existing Extract, Transform, and Load (ETL) processes and data warehouse consumption as possible?

+- Are you migrating but want to do some extensive improvements along the way?

+- Are you building an entirely new data analytics environment (sometimes called a *greenfield project*)?

+Next, you need to consider your pain points.

+### Identify current pain points

+Your POC should contain use cases to prove potential solutions to address your current pain points. Here are some questions to consider:

+- What gaps in your current implementation do you expect Azure Synapse to fill?

+- What new business needs are you required to support?

+- What service level agreements (SLAs) are you required to meet?

+- What will be the workloads (for example, ETL, batch queries, analytics, reporting queries, or interactive queries)?

+Next, you need to set your POC success criteria.

+### Set POC success criteria

+Identify why you're doing a POC and be sure to define clear goals. It's also important to know what outputs you want from your POC and what you plan to do with them.

+Keep in mind that a POC should be a short and focused effort to quickly prove or test a limited set of concepts. If you have a long list of items to prove, you may want to dive them into multiple POCs. POCs can have gates between them so you can determine whether to proceed to the next POC.

+Here are some example POC goals:

+- We need to know that the query performance for our big complex reporting queries will meet our new SLAs.

+- We need to know the query performance for our interactive users.

+- We need to know whether our existing ETL processes are a good fit and where improvements need to be made.

+- We need to know whether we can shorten our ETL runtimes and by how much.

+- We need to know that Synapse Analytics has sufficient security capabilities to adequately secure our data.

+Next, you need to create a test plan.

+### Create a test plan

+Using your goals, identify specific tests to run in order to support those goals and provide your identified outputs. It's important to make sure that you have at least one test for each goal and the expected output. Identify specific queries, reports, ETL and other processes that you will run to provide quantifiable results.

+Refine your tests by adding multiple testing scenarios to clarify any table structure questions that arise.

+Good planning usually defines an effective POC execution. Make sure all stakeholders agree to a written test plan that ties each POC goal to a set of clearly stated test cases and measurements of success.

+Most test plans revolve around performance and the expected user experience. What follows is an example of a test plan. It's important to customize your test plan to meet your business requirements. Clearly defining what you are testing will pay dividends later in this process.

+|Goal|Test|Expected outcomes|

+||||

+|We need to know that the query performance for our big complex reporting queries will meet our new SLAs|- Sequential test of complex queries<br/>- Concurrency test of complex queries against stated SLAs|- Queries A, B, and C completed in 10, 13, and 21 seconds, respectively<br/>- With 10 concurrent users, queries A, B, and C completed in 11, 15, and 23 seconds, on average|

+|We need to know the query performance for our interactive users|- Concurrency test of selected queries at an expected concurrency level of 50 users.<br/>- Run the preceding query with result set caching|- At 50 concurrent users, average execution time is expected to be under 10 seconds, and without result set caching<br/>- At 50 concurrent users, average execution time is expected to be under five seconds with result set caching|

+|We need to know whether our existing ETL processes can run within the SLA|- Run one or two ETL processes to mimic production loads|- Loading incrementally into a core fact table must complete in less than 20 minutes (including staging and data cleansing)<br/>- Dimension processing needs to take less than five minutes|

+|We need to know that the data warehouse has sufficient security capabilities to secure our data|- Review and enable [network security](security-white-paper-network-security.md) (VNet and private endpoints), [access control](security-white-paper-access-control.md) (row-level security, dynamic data masking)|- Prove that data never leaves our tenant.<br/>- Ensure that customer content is easily secured|

+Next, you need to identify and validate the POC dataset.

+### Identify and validate the POC dataset

+Using the scoped tests, you can now identify the dataset required to execute those tests in Azure Synapse. Review your dataset by considering the following:

+- Verify that the dataset adequately represents your production dataset in terms of content, complexity, and scale.

+- Don't use a dataset that's too small (less than 1TB), as you might not achieve representative performance.

+- Don't use a dataset that's too large, as the POC isn't intended to complete a full data migration.

+- Identify the [distribution pattern](../sql-data-warehouse/sql-data-warehouse-tables-distribute.md), [indexing option](../sql-data-warehouse/sql-data-warehouse-tables-index.md), and [partitioning](../sql-data-warehouse/sql-data-warehouse-tables-partition.md) for each table. If there are any questions regarding distribution, indexing, or partitioning, add tests to your POC to answer them. Bear in mind that you may want to test more than one distribution option or indexing option for some tables.

+- Check with the business owners for any blockers for moving the POC dataset to the cloud.

+- Identify any security or privacy concerns.

+> [!IMPORTANT]

+> Make sure you check with business owners for any blockers before moving any data to the cloud. Identify any security or privacy concerns or any data obfuscation needs that should be done before moving data to the cloud.

+Next, you need to assemble the team of experts.

+### Assemble the team

+Identify the team members and their commitment to support your POC. Team members should include:

+- A project manager to run the POC project.

+- A business representative to oversee requirements and results.

+- An application data expert to source the data for the POC dataset.

+- An Azure Synapse specialist.

+- An expert advisor to optimize the POC tests.

+- Any person who will be required for specific POC project tasks but who aren't required for its entire duration. These supporting resources could include network administrators, Azure administrators, or Azure AD administrators.

+> [!TIP]

+> We recommend engaging an expert advisor to assist with your POC. [Microsoft's partner community](https://appsource.microsoft.com/marketplace/partner-dir) has global availability of expert consultants who can help you assess, evaluate, or implement Azure Synapse.

+Now that you are fully prepared, it's time to put your POC into practice.

+## Put the POC into practice

+It's important to keep the following in mind:

+- Implement your POC project with the discipline and rigor of any production project.

+- Run the POC according to plan.

+- Have a change request process in place to prevent your POC scope from growing or changing.

+Before tests can start, you need to set up the test environment. It involves four stages:

+1. Setup

+1. Data loading

+1. Querying

+1. Value added tests

+### Setup

+You can set up a POC on Azure Synapse by following these steps:

+1. Use [this quickstart](../sql-data-warehouse/create-data-warehouse-portal.md) to provision a Synapse workspace and set up storage and permissions according to the POC test plan.

+1. Use [this quickstart](../quickstart-create-sql-pool-portal.md) to add a dedicated SQL pool to the Synapse workspace.

+1. Set up [networking and security](security-white-paper-introduction.md) according to your requirements.

+1. Grant appropriate access to POC team members. See [this article](/azure-sql/database/logins-create-manage) about authentication and authorization for accessing dedicated SQL pools.

+> [!TIP]

+> We recommend that you *develop code and unit testing* by using the DW500c service level (or below). We recommend that you *run load and performance tests* by using the DW1000c service level (or above). You can [pause compute of the dedicated SQL pool](../sql-data-warehouse/pause-and-resume-compute-portal.md) at any time to cease compute billing, which will save on costs.

+### Data loading

+Once you've set up the dedicated SQL pool, you can follow these steps to load data:

+1. Load the data into [Azure Blob Storage](../../storage/blobs/storage-blobs-overview.md). For a POC, we recommend that you use a [general-purpose V2 storage account](../../storage/common/storage-account-overview.md) with [locally-redundant storage (LRS)](../../storage/common/storage-redundancy.md#locally-redundant-storage). While there are several tools for migrating data to Azure Blob Storage, the easiest way is to use [Azure Storage Explorer](https://azure.microsoft.com/features/storage-explorer/), which can copy files into a storage container.

+2. Load the data into the dedicated SQL pool. Azure Synapse supports two T-SQL loading methods: [PolyBase](../sql-data-warehouse/design-elt-data-loading.md) and the [COPY](/sql/t-sql/statements/copy-into-transact-sql?view=azure-sqldw-latest&preserve-view=true) statement. You can use SSMS to connect to the dedicated SQL pool to use either method.

+When you load data into the dedicated SQL pool for the first time, you need to consider which [distribution pattern](../sql-data-warehouse/sql-data-warehouse-tables-distribute.md) and [index option](../sql-data-warehouse/sql-data-warehouse-tables-index.md) to use. While a dedicated SQL pool supports a variety of both, it's a best practice to rely on default settings. Default settings use round-robin distribution and a clustered columnstore index. If necessary, you can adjust these settings later, which is described later in this article.

+The following example shows the COPY load method:

+```sql

+--Note when specifying the column list, input field numbers start from 1

+COPY INTO

+ test_1 (Col_1 default 'myStringDefault' 1, Col_2 default 1 3)

+FROM

+ 'https://myaccount.blob.core.windows.net/myblobcontainer/folder1/'

+WITH (

+ FILE_TYPE = 'CSV',

+ CREDENTIAL = (IDENTITY = 'Storage Account Key' SECRET = '<Your_Account_Key>'),

+ FIELDQUOTE = '"',

+ FIELDTERMINATOR = ',',

+ ROWTERMINATOR = '0x0A',

+ ENCODING = 'UTF8',

+ FIRSTROW = 2

+);

+```

+### Querying

+The primary purpose of an data warehouse is to perform analytics, which requires querying the data warehouse. Most POCs start by running a small number of representative queries against the data warehouse, at first sequentially and then concurrently. You should define both approaches in your test plan.

+#### Sequential query tests

+It's easy to run sequential query tests in SSMS. It's important to run these tests by using a user with a sufficiently large [resource class](../sql-data-warehouse/resource-classes-for-workload-management.md). A resource class is a pre-determined resource limit in dedicated SQL pool that governs compute resources and concurrency for query execution. For simple queries, we recommend using the pre-defined **staticrc20** resource class. For more complex queries, we recommend using the pre-defined **staticrc40** resource class.

+Notice that the following first query uses a [query label](../sql/develop-label.md) to provide a mechanism to keep track of the query. The second query uses the `sys.dm_pdw_exec_requests` dynamic management view to search by the label.

+```sql

+/* Use the OPTION(LABEL = '') Syntax to add a query label to track the query in DMVs */

+SELECT TOP (1000)

+ *

+FROM

+ [dbo].[Date]

+OPTION (LABEL = 'Test1');

+/* Use sys.dm_pdw_exec_requests to determine query execution duration (ms) */

+SELECT

+ Total_elapsed_time AS [Elapsed_Time_ms],

+ [label]

+FROM

+ sys.dm_pdw_exec_requests

+WHERE

+ [label] = 'Test1';

+```

+#### Concurrent query tests

+After recording sequential query performance, you can then run multiple queries concurrently. That way, you can simulate a business intelligence workload running against the dedicated SQL pool. The easiest way to run this test is to download a stress testing tool. The most popular tool is [Apache JMeter](https://jmeter.apache.org/download_jmeter.cgi), which is a third-party open source tool.

+The tool reports on minimum, maximum, and median query durations for a given concurrency level. For example, suppose that you want to simulate a business intelligence workload that generates 100 concurrent queries. You can setup JMeter to run those 100 concurrent queries in a loop and then review the steady state execution. It can be done with [result set caching](../sql-data-warehouse/performance-tuning-result-set-caching.md) on or off to evaluate the suitability of that feature.

+Be sure to document your results. Here's an example of some results:

+|Concurrency|# Queries run|DWU|Min duration(s)|Max duration(S)|Median duration(s)|

+|||||||

+|100|1,000|5,000|3|10|5|

+|50|5,000|5,000|3|6|4|

+#### Mixed workload tests

+Mixed workload testing is an extension of the [concurrent query tests](#concurrent-query-tests). By adding a data loading process into the workload mix, the workload will better simulate a real production workload.

+#### Optimize the data

+Depending on the query workload running on Azure Synapse, you may need to optimize your data warehouse's distributions and indexes and rerun the tests. For more information, see [Best practices for dedicated SQL pools in Azure Synapse Analytics](../sql-data-warehouse/sql-data-warehouse-best-practices.md).

+The most common mistakes seen during setup are:

+- Large queries run with a resource class that's too low.

+- The dedicated SQL pool service level DWUs are too low for the workload.

+- Large tables require hash distribution.

+To improve query performance, you can:

+- Create [materialized views](../sql-data-warehouse/performance-tuning-materialized-views.md), which can accelerate queries involving common aggregations.

+- [Replicate tables](../sql-data-warehouse/design-guidance-for-replicated-tables.md), especially for small dimension tables.

+- [Hash distribute](../sql-data-warehouse/sql-data-warehouse-tables-distribute.md) large fact tables that are joined or aggregated.

+### Value added tests

+Once query performance testing is complete, it's a good time to test specific features to verify that they satisfy your intended use cases. These features include:

+- [Row-level security](/sql/relational-databases/security/row-level-security?view=azure-sqldw-latest&preserve-view=true)

+- [Column-level security](../sql-data-warehouse/column-level-security.md)

+- [Dynamic data masking](/azure/azure-sql/database/dynamic-data-masking-overview)

+- Intra-cluster scaling via [workload isolation](../sql-data-warehouse/sql-data-warehouse-workload-isolation.md)

+Finally, you need to interpret your POC results.

+## Interpret the POC results

+Once you have test results for your data warehouse, it's important to interpret that data. A common approach you can take is to compare the runs in terms of *price/performance*. Simply put, price/performance removes the differences in price per DWU or service hardware and provides a single comparable number for each performance test.

+Here's an example:

+|Test|Test duration|DWU|$/hr for DWU|Cost of test|

+||||||

+|Test 1|10 min|1000|$12/hr|$2|

+|Test 2|30 min|500|$6/hr|$3|

+This example makes it easy to see that **Test 1** at DWU1000 is more cost effective at $2 per test run compared with $3 per test run.

+> [!NOTE]

+> You can also use this methodology to compare results *across vendors* in a POC.

+In summary, once you complete all the POC tests, you're ready to evaluate the results. Begin by evaluating whether the POC goals have been met and the desired outputs collected. Make note of where additional testing is warranted and additional questions that were raised.

+## Next steps

+> [!div class="nextstepaction"]

+> [Data lake exploration with serverless SQL pool in Azure Synapse Analytics](proof-of-concept-playbook-serverless-sql-pool.md)

+> [!div class="nextstepaction"]

+> [Big data analytics with Apache Spark pool in Azure Synapse Analytics](proof-of-concept-playbook-spark-pool.md)

+> [!div class="nextstepaction"]

+> [Azure Synapse Analytics frequently asked questions](../overview-faq.yml)

+ Title: Azure Synapse proof of concept playbook

+description: "Introduction to a series of articles that provide a high-level methodology for planning, preparing, and running an effective Azure Synapse Analytics proof of concept project."

+# Azure Synapse proof of concept playbook

+Whether it's an enterprise data warehouse migration, a big data re-platforming, or a greenfield implementation, each project traditionally starts with a proof of concept (POC).

+The *Synapse proof of concept playbook* is a series of related articles that provide a high-level methodology for planning, preparing, and running an effective Azure Synapse Analytics POC project. The overall objective of a POC is to validate potential solutions to technical problems, such as how to integrate systems or how to achieve certain results through a specific configuration. As emphasized by this series, an effective POC validates that certain concepts have the potential for real-world production application.

+> [!TIP]

+> If you're new to Azure Synapse, we recommend you work through the [Introduction to Azure Synapse Analytics](/learn/modules/introduction-azure-synapse-analytics/) module.

+## Playbook audiences

+The playbook helps you to evaluate the use of Azure Synapse when migrating from an existing workload. We designed it for the following audiences:

+- Technical experts who are planning their own in-house Azure Synapse POC project.

+- Business owners who will be part of the execution or evaluation of an Azure Synapse POC project.

+- Anyone looking to learn more about data warehousing POC projects.

+## Playbook content

+The playbook delivers the following content:

+- Guidance on what makes an effective POC.

+- Guidance on how to make valid comparisons between systems.

+- Guidance on the technical aspects of running an Azure Synapse POC.

+- A road map to relevant technical content from Azure Synapse.

+- Guidance on how to evaluate POC results to support business decisions.

+- Guidance on how and where to find additional help.

+The playbook includes three subjects:

+- [Data warehousing with dedicated SQL pool in Azure Synapse Analytics](proof-of-concept-playbook-dedicated-sql-pool.md)

+- [Data lake exploration with serverless SQL pool in Azure Synapse Analytics](proof-of-concept-playbook-serverless-sql-pool.md)

+- [Big data analytics with Apache Spark pool in Azure Synapse Analytics](proof-of-concept-playbook-spark-pool.md)

+## Next steps

+> [!div class="nextstepaction"]

+> [Data warehousing with dedicated SQL pool in Azure Synapse Analytics](proof-of-concept-playbook-dedicated-sql-pool.md)

+> [!div class="nextstepaction"]

+> [Data lake exploration with serverless SQL pool in Azure Synapse Analytics](proof-of-concept-playbook-serverless-sql-pool.md)

+> [!div class="nextstepaction"]

+> [Big data analytics with Apache Spark pool in Azure Synapse Analytics](proof-of-concept-playbook-spark-pool.md)

+> [!div class="nextstepaction"]

+> [Azure Synapse Analytics frequently asked questions](../overview-faq.yml)

+ Title: "Synapse POC playbook: Data lake exploration with serverless SQL pool in Azure Synapse Analytics"

+description: "A high-level methodology for preparing and running an effective Azure Synapse Analytics proof of concept (POC) project for serverless SQL pool."

+# Synapse POC playbook: Data lake exploration with serverless SQL pool in Azure Synapse Analytics

+This article presents a high-level methodology for preparing and running an effective Azure Synapse Analytics proof of concept (POC) project for serverless SQL pool.

+## Prepare for the POC

+A POC project can help you make an informed business decision about implementing a big data and advanced analytics environment on a cloud-based platform that leverages serverless SQL pool in Azure Synapse. If you need to explore or gain insights from data in the data lake, or optimize your existing data transformation pipeline, you can benefit from using the serverless SQL pool. It's suitable for the following scenarios:

+- **Basic discovery and exploration:** Quickly reason about data stored in various formats (Parquet, CSV, JSON) in your data lake, so you can plan how to unlock insights from it.

+- **Logical data warehouse:** Produce a relational abstraction on top of raw or disparate data without relocating or transforming it, providing an always up-to-date view of your data.

+- **Data transformation:** Run simple, scalable, and highly performant data lake queries by using T-SQL. You can feed query results to business intelligence (BI) tools, or load them into a relational database. Target systems can include Azure Synapse dedicated SQL pools or Azure SQL Database.

+Different professional roles can benefit from serverless SQL pool:

+- **Data engineers** can explore the data lake, transform and prepare data by using serverless SQL pool, and simplify their data transformation pipelines.

+- **Data scientists** can quickly reason about the contents and structure of the data stored in the data lake by using the [OPENROWSET](/sql/t-sql/functions/openrowset-transact-sql?view=sql-server-ver15&viewFallbackFrom=azure-sqldw-latest&preserve-view=true) T-SQL function and its automatic schema inference.

+- **Data analysts** can write T-SQL queries in their preferred query tools, which can connect to serverless SQL pool. They can explore data in Spark external tables that were created by data scientists or data engineers.

+- **BI professionals** can quickly create Power BI reports that connect to data lake or Spark tables.

+A serverless SQL pool POC project will identify your key goals and business drivers that serverless SQL pool is designed to support. It will also test key features and gather metrics to support your implementation decisions. A POC isn't designed to be deployed to a production environment. Rather, it's a short-term project that focuses on key questions, and its result can be discarded.

+Before you begin planning your serverless SQL Pool POC project:

+> [!div class="checklist"]

+> - Identify any restrictions or guidelines your organization has about moving data to the cloud.

+> - Identify executive or business sponsors for a big data and advanced analytics platform project. Secure their support for migration to the cloud.

+> - Identify availability of technical experts and business users to support you during the POC execution.

+Before you start preparing for the POC project, we recommend you first read the [serverless SQL pool documentation](../sql/on-demand-workspace-overview.md).

+> [!TIP]

+> If you're new to serverless SQL pools, we recommend you work through the [Build data analytics solutions using Azure Synapse serverless SQL pools](/learn/paths/build-data-analytics-solutions-using-azure-synapse-serverless-sql-pools/) learning path.

+### Set the goals

+A successful POC project requires planning. Start by identify why you're doing a POC to fully understand the real motivations. Motivations could include modernization, cost saving, performance improvement, or integrated experience. Be sure to document clear goals for your POC and the criteria that will define its success. Ask yourself:

+> [!div class="checklist"]

+> - What do you want as the outputs of your POC?

+> - What will you do with those outputs?

+> - Who will use the outputs?

+> - What will define a successful POC?

+Keep in mind that a POC should be a short and focused effort to quickly prove a limited set of concepts and capabilities. These concepts and capabilities should be representative of the overall workload. If you have a long list of items to prove, you may want to plan more than one POC. In that case, define gates between the POCs to determine whether you need to continue with the next one. Given the different professional roles that can use a serverless SQL pool (and the different scenarios that serverless SQL pool supports), you may choose to execute multiple POCs. For example, one POC could focus on requirements for the data scientist role, such as discovery and exploration of data in different formats. Another could focus on requirements for the data engineering role, such as data transformation and the creation of a logical data warehouse.

+As you consider your POC goals, ask yourself the following questions to help you shape the goals:

+> [!div class="checklist"]

+> - Are you migrating from an existing big data and advanced analytics platform (on-premises or cloud)?

+> - Are you migrating but want to make as few changes as possible to existing ingestion and data processing?

+> - Are you migrating but want to do some extensive improvements along the way?

+> - Are you building an entirely new big data and advanced analytics platform (greenfield project)?

+> - What are your current pain points? For example, scalability, performance, or flexibility.

+> - What new business requirements do you need to support?

+> - What are the SLAs that you're required to meet?

+> - What will be the workloads? For example, data exploration over different data formats, basic exploration, a logical data warehouse, data preparation and/or transformation, T-SQL interactive analysis, T-SQL querying of Spark tables, or reporting queries over the data lake.

+> - What are the skills of the users who will own the project (should the POC be implemented)?

+Here are some examples of POC goal setting:

+- Why are we doing a POC?

+ - We need to know if we can explore all of the raw file formats we store by using serverless SQL pool.

+ - We need to know if our data engineers can quickly evaluate new data feeds.

+ - We need to know if data lake query performance by using serverless SQL pool will meet our data exploration requirements.

+ - We need to know if serverless SQL pool is a good choice for some of our visualizations and reporting requirements.

+ - We need to know if serverless SQL pool is a good choice for some of our data ingestion and processing requirements.

+ - We need to know if our move to Azure Synapse will meet our budget.

+- At the conclusion of this PoC:

+ - We will have the data to identify the data transformations that are well suited to serverless SQL pool.

+ - We will have the data to identify when serverless SQL pool can be best used during data visualization.

+ - We will have the data to know the ease with which our data engineers and data scientists can adopt the new platform.

+ - We will have gained insight to better estimate the effort required to complete the implementation or migration project.

+ - We will have a list of items that may need more testing.

+ - Our POC will be successful if we have the data needed and have completed the testing identified to determine how serverless SQL pool will support our cloud-based big data and advance analytics platform.

+ - We will have determined whether we can move to the next phase or whether more POC testing is needed to finalize our decision.

+ - We will be able to make a sound business decision supported by specific data points.

+### Plan the project

+Use your goals to identify specific tests and to provide the outputs you identified. It's important to make sure that you have at least one test to support each goal and expected output. Also, identify specific data exploration and analysis tasks, specific transformations, and specific existing processing you want to test. Identify a specific dataset and codebase that you can use.

+Here's an example of the needed level of specificity in planning:

+- **Goal:** We need to know whether data engineers can achieve the equivalent processing of the existing ETL process named "Daily Batch Raw File Validation" within the required SLA.

+- **Output:** We will have the data to determine whether we can use T-SQL queries to execute the "Daily Batch Raw File Validation" ETL process within the required SLA.

+- **Test:** Validation queries A, B, and C are identified by data engineering, and they represent overall data processing needs. Compare the performance of these queries with the benchmark obtained from the existing system.

+### Evaluate the POC dataset

+Using the specific tests you identified, select a dataset to support the tests. Take time to review this dataset. You should verify that the dataset will adequately represent your future processing in terms of content, complexity, and scale. Don't use a dataset that's too small because it won't deliver representative performance. Conversely, don't use a dataset that's too large because the POC shouldn't become a full data migration. Be sure to obtain the appropriate benchmarks from existing systems so you can use them for performance comparisons.

+> [!IMPORTANT]

+> Make sure you check with business owners for any blockers before moving any data to the cloud. Identify any security or privacy concerns or any data obfuscation needs that should be done before moving data to the cloud.

+### Create a high-level architecture

+Based upon the high-level architecture of your proposed future state architecture, identify the components that will form part of your POC. Your high-level future state architecture likely contains many data sources, numerous data consumers, big data components, and possibly machine learning and artificial intelligence (AI) data consumers. Your POC architecture should specifically identify components that will be part of the POC. Importantly, it should identify any components that won't form part of the POC testing.

+If you're already using Azure, identify any resources you already have in place (Azure Active Directory, ExpressRoute, and others) that you can use during the POC. Also identify the Azure regions your organization uses. Now is a great time to identify the throughput of your ExpressRoute connection and to check with other business users that your POC can consume some of that throughput without adverse impact on production systems.

+### Identify POC resources

+Specifically identify the technical resources and time commitments required to support your POC. Your POC will need:

+- A business representative to oversee requirements and results.

+- An application data expert, to source the data for the POC and provide knowledge of the existing processes and logic.

+- A serverless SQL pool expert.

+- An expert advisor, to optimize the POC tests.

+- Resources that will be required for specific components of your POC project, but not necessarily required for the duration of the POC. These resources could include network admins, Azure admins, Active Directory admins, Azure portal admins, and others.

+- Ensure all the required Azure services resources are provisioned and the required level of access is granted, including access to storage accounts.

+- Ensure you have an account that has required data access permissions to retrieve data from all data sources in the POC scope.

+> [!TIP]

+> We recommend engaging an expert advisor to assist with your POC. [Microsoft's partner community](https://appsource.microsoft.com/marketplace/partner-dir) has global availability of expert consultants who can help you assess, evaluate, or implement Azure Synapse.

+### Set the timeline

+Review your POC planning details and business needs to identify a time frame for your POC. Make realistic estimates of the time that will be required to complete the POC goals. The time to complete your POC will be influenced by the size of your POC dataset, the number and complexity of tests, and the number of interfaces to test. If you estimate that your POC will run longer than four weeks, consider reducing the POC scope to focus on the highest priority goals. Be sure to obtain approval and commitment from all the lead resources and sponsors before continuing.

+## Put the POC into practice

+We recommend you execute your POC project with the discipline and rigor of any production project. Run the project according to plan and manage a change request process to prevent uncontrolled growth of the POC's scope.

+Here are some examples of high-level tasks:

+1. [Create a Synapse workspace](../quickstart-create-workspace.md), storage accounts, and the Azure resources identified in the POC plan.

+1. Set up [networking and security](security-white-paper-introduction.md) according to your requirements.

+1. Grant appropriate access to POC team members. See [this article](../sql/develop-storage-files-storage-access-control.md) about permissions for accessing files directly from Azure Storage.

+1. Load the POC dataset.

+1. Implement and configure the tests and/or migrate existing code to serverless SQL pool scripts and views.

+1. Execute the tests:

+ - Many tests can be executed in parallel.

+ - Record your results in a consumable and readily understandable format.

+1. Monitor for troubleshooting and performance.

+1. Evaluate your results and present findings.

+1. Work with technical stakeholders and the business to plan for the next stage of the project. The next stage could be a follow-up POC or a production implementation.

+## Interpret the POC results

+When you complete all the POC tests, you evaluate the results. Begin by evaluating whether the POC goals were met and the desired outputs were collected. Determine whether more testing is necessary or any questions need addressing.

+## Next steps

+> [!div class="nextstepaction"]

+> [Data lake exploration with dedicated SQL pool in Azure Synapse Analytics](proof-of-concept-playbook-dedicated-sql-pool.md)

+> [!div class="nextstepaction"]

+> [Big data analytics with Apache Spark pool in Azure Synapse Analytics](proof-of-concept-playbook-spark-pool.md)

+> [!div class="nextstepaction"]

+> [Build data analytics solutions using Azure Synapse serverless SQL pools](/learn/paths/build-data-analytics-solutions-using-azure-synapse-serverless-sql-pools/)

+> [!div class="nextstepaction"]

+> [Azure Synapse Analytics frequently asked questions](../overview-faq.yml)

+ Title: "Synapse POC playbook: Big data analytics with Apache Spark pool in Azure Synapse Analytics"

+description: "A high-level methodology for preparing and running an effective Azure Synapse Analytics proof of concept (POC) project for Apache Spark pool."

+# Synapse POC playbook: Big data analytics with Apache Spark pool in Azure Synapse Analytics

+This article presents a high-level methodology for preparing and running an effective Azure Synapse Analytics proof of concept (POC) project for Apache Spark pool.

+## Prepare for the POC

+A POC project can help you make an informed business decision about implementing a big data and advanced analytics environment on a cloud-based platform that leverages Apache Spark pool in Azure Synapse.

+A POC project will identify your key goals and business drivers that cloud-based big data and advance analytics platform must support. It will test key metrics and prove key behaviors that are critical to the success of your data engineering, machine learning model building, and training requirements. A POC isn't designed to be deployed to a production environment. Rather, it's a short-term project that focuses on key questions, and its result can be discarded.

+Before you begin planning your Spark POC project:

+> [!div class="checklist"]

+> - Identify any restrictions or guidelines your organization has about moving data to the cloud.

+> - Identify executive or business sponsors for a big data and advanced analytics platform project. Secure their support for migration to the cloud.

+> - Identify availability of technical experts and business users to support you during the POC execution.

+Before you start preparing for the POC project, we recommend you first read the [Apache Spark documentation](/hdinsight/spark/apache-spark-overview.md).

+> [!TIP]

+> If you're new to Spark pools, we recommend you work through the [Perform data engineering with Azure Synapse Apache Spark Pools](/learn/paths/perform-data-engineering-with-azure-synapse-apache-spark-pools/) learning path.

+By now you should have determined that there are no immediate blockers and then you can start preparing for your POC. If you are new to Apache Spark Pools in Azure Synapse Analytics you can refer to [this documentation](../spark/apache-spark-overview.md) where you can get an overview of the Spark architecture and learn how it works in Azure Synapse.

+Develop an understanding of these key concepts:

+- Apache Spark and its distributed architecture.

+- Spark concepts like Resilient Distributed Datasets (RDD) and partitions (in-memory and physical).

+- Azure Synapse workspace, the different compute engines, pipeline, and monitoring.

+- Separation of compute and storage in Spark pool.

+- Authentication and authorization in Azure Synapse.

+- Native connectors that integrate with Azure Synapse dedicated SQL pool, Azure Cosmos DB, and others.

+Azure Synapse decouples compute resources from storage so that you can better manage your data processing needs and control costs. The serverless architecture of Spark pool allows you to spin up and down as well as grow and shrink your Spark cluster, independent of your storage. You can pause (or setup auto-pause) a Spark cluster entirely. That way, you pay for compute only when it's in use. When it's not in use, you only pay for storage. You can scale up your Spark cluster for heavy data processing needs or large loads and then scale it back down during less intense processing times (or shut it down completely). You can effectively scale and pause a cluster to reduce costs. Your Spark POC tests should include data ingestion and data processing at different scales (small, medium, and large) to compare price and performance at different scale. For more information, see [Automatically scale Azure Synapse Analytics Apache Spark pools](../spark/apache-spark-autoscale.md).

+It's important to understand the difference between the different sets of Spark APIs so you can decide what works best for your scenario. You can choose the one that provides better performance or ease of use, taking advantage of your team's existing skill sets. For more information, see [A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets](https://databricks.com/session/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets).

+Data and file partitioning work slightly differently in Spark. Understanding the differences will help you to optimize for performance. For more information, see Apache Spark documentation: [Partition Discovery](https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#partition-discovery) and [Partition Configuration Options](https://spark.apache.org/docs/latest/sql-performance-tuning.html#other-configuration-options).

+### Set the goals

+A successful POC project requires planning. Start by identify why you're doing a POC to fully understand the real motivations. Motivations could include modernization, cost saving, performance improvement, or integrated experience. Be sure to document clear goals for your POC and the criteria that will define its success. Ask yourself:

+> [!div class="checklist"]

+> - What do you want as the outputs of your POC?

+> - What will you do with those outputs?

+> - Who will use the outputs?

+> - What will define a successful POC?

+Keep in mind that a POC should be a short and focused effort to quickly prove a limited set of concepts and capabilities. These concepts and capabilities should be representative of the overall workload. If you have a long list of items to prove, you may want to plan more than one POC. In that case, define gates between the POCs to determine whether you need to continue with the next one. Given the different professional roles that may use Spark pools and notebooks in Azure Synapse, you may choose to execute multiple POCs. For example, one POC could focus on requirements for the data engineering role, such as ingestion and processing. Another POC could focus on machine learning (ML) model development.

+As you consider your POC goals, ask yourself the following questions to help you shape the goals:

+> [!div class="checklist"]

+> - Are you migrating from an existing big data and advanced analytics platform (on-premises or cloud)?

+> - Are you migrating but want to make as few changes as possible to existing ingestion and data processing? For example, a Spark to Spark migration, or a Hadoop/Hive to Spark migration.

+> - Are you migrating but want to do some extensive improvements along the way? For example, re-writing MapReduce jobs as Spark jobs, or converting legacy RDD-based code to DataFrame/Dataset-based code.

+> - Are you building an entirely new big data and advanced analytics platform (greenfield project)?

+> - What are your current pain points? For example, scalability, performance, or flexibility.

+> - What new business requirements do you need to support?

+> - What are the SLAs that you're required to meet?

+> - What will be the workloads? For example, ETL, batch processing, stream processing, machine learning model training, analytics, reporting queries, or interactive queries?

+> - What are the skills of the users who will own the project (should the POC be implemented)? For example, PySpark vs Scala skills, notebook vs IDE experience.

+Here are some examples of POC goal setting:

+- Why are we doing a POC?

+ - We need to know that the data ingestion and processing performance for our big data workload will meet our new SLAs.

+ - We need to know whether near real-time stream processing is possible and how much throughput it can support. (Will it support our business requirements?)

+ - We need to know if our existing data ingestion and transformation processes are a good fit and where improvements will need to be made.

+ - We need to know if we can shorten our data integration run times and by how much.

+ - We need to know if our data scientists can build and train machine learning models and leverage AI/ML libraries as needed in a Spark pool.

+ - Will the move to cloud-based Synapse Analytics meet our cost goals?

+- At the conclusion of this POC:

+ - We will have the data to determine if our data processing performance requirements can be met for both batch and real-time streaming.

+ - We will have tested ingestion and processing of all our different data types (structured, semi and unstructured) that support our use cases.

+ - We will have tested some of our existing complex data processing and can identify the work that will need to be completed to migrate our portfolio of data integration to the new environment.

+ - We will have tested data ingestion and processing and will have the data points to estimate the effort required for the initial migration and load of historical data, as well as estimate the effort required to migrate our data ingestion (Azure Data Factory (ADF), Distcp, Databox, or others).

+ - We will have tested data ingestion and processing and can determine if our ETL/ELT processing requirements can be met.

+ - We will have gained insight to better estimate the effort required to complete the implementation project.

+ - We will have tested scale and scaling options and will have the data points to better configure our platform for better price-performance settings.

+ - We will have a list of items that may need more testing.

+### Plan the project

+Use your goals to identify specific tests and to provide the outputs you identified. It's important to make sure that you have at least one test to support each goal and expected output. Also, identify specific data ingestion, batch or stream processing, and all other processes that will be executed so you can identify a very specific dataset and codebase. This specific dataset and codebase will define the scope of the POC.

+Here's an example of the needed level of specificity in planning:

+- **Goal A:** We need to know whether our requirement for data ingestion and processing of batch data can be met under our defined SLA.

+- **Output A:** We will have the data to determine whether our batch data ingestion and processing can meet the data processing requirement and SLA.

+ - **Test A1:** Processing queries A, B, and C are identified as good performance tests as they are commonly executed by the data engineering team. Also, they represent overall data processing needs.

+ - **Test A2:** Processing queries X, Y, and Z are identified as good performance tests as they contain near real-time stream processing requirements. Also, they represent overall event-based stream processing needs.

+ - **Test A3:** Compare the performance of these queries at different scale of the Spark cluster (varying number of worker nodes, size of the worker nodes - like small, medium, and large - number and size of executors) with the benchmark obtained from the existing system. Keep the *law of diminishing returns* in mind; adding more resources (either by scaling up or scaling out) can help to achieve parallelism, however there's a certain limit that's unique to each scenario to achieve the parallelism. Discover the optimal configuration for each identified use case in your testing.

+- **Goal B:** We need to know if our data scientists can build and train machine learning models on this platform.

+- **Output B:** We will have tested some of our machine learning models by training them on data in a Spark pool or a SQL pool, leveraging different machine learning libraries. These tests will help to determine which machine learning models can be migrated to the new environment

+ - **Test B1:** Specific machine learning models will be tested.

+ - **Test B2:** Test base machine learning libraries that come with Spark (Spark MLLib) along with an additional library that can be installed on Spark (like scikit-learn) to meet the requirement.

+- **Goal C:** We will have tested data ingestion and will have the data points to:

+ - Estimate the effort for our initial historical data migration to data lake and/or the Spark pool.

+ - Plan an approach to migrate historical data.

+- **Output C:** We will have tested and determined the data ingestion rate achievable in our environment and can determine whether our data ingestion rate is sufficient to migrate historical data during the available time window.

+ - **Test C1:** Test different approaches of historical data migration. For more information, see [Transfer data to and from Azure](/architecture/data-guide/scenarios/data-transfer.md).

+ - **Test C2:** Identify allocated bandwidth of ExpressRoute and if there is any throttling setup by the infra team. For more information, see [What is Azure ExpressRoute? (Bandwidth options)](/expressroute/expressroute-introduction#bandwidth-options.md).

+ - **Test C3:** Test data transfer rate for both online and offline data migration. For more information, see [Copy activity performance and scalability guide](/data-factory/copy-activity-performance#copy-performance-and-scalability-achievable-using-azure-data-factory-and-synapse-pipelines).

+ - **Test C4:** Test data transfer from the data lake to the SQL pool by using either ADF, Polybase, or the COPY command. For more information, see [Data loading strategies for dedicated SQL pool in Azure Synapse Analytics](/sql-data-warehouse/design-elt-data-loading.md).

+- **Goal D:** We will have tested the data ingestion rate of incremental data loading and will have the data points to estimate the data ingestion and processing time window to the data lake and/or the dedicated SQL pool.

+- **Output D:** We will have tested the data ingestion rate and can determine whether our data ingestion and processing requirements can be met with the identified approach.

+ - **Test D1:** Test the daily update data ingestion and processing.

+ - **Test D1:** Test the processed data load to the dedicated SQL pool table from the Spark pool. For more information, see [Azure Synapse Dedicated SQL Pool Connector for Apache Spark](../spark/synapse-spark-sql-pool-import-export.md).

+ - **Test D1:** Execute the daily update load process concurrently while running end user queries.

+Be sure to refine your tests by adding multiple testing scenarios. Azure Synapse makes it easy to test different scale (varying number of worker nodes, size of the worker nodes like small, medium, and large) to compare performance and behavior.

+Here are some testing scenarios:

+- **Spark pool test A:** We will execute data processing across multiple node types (small, medium, and large) as well as different numbers of worker nodes.

+- **Spark pool test B:** We will load/retrieve processed data from the Spark pool to the dedicated SQL pool by using [the connector](../spark/synapse-spark-sql-pool-import-export.md).

+- **Spark pool test C:** We will load/retrieve processed data from the Spark pool to Cosmos DB by using Azure Synapse Link.

+### Evaluate the POC dataset

+Using the specific tests you identified, select a dataset to support the tests. Take time to review this dataset. You should verify that the dataset will adequately represent your future processing in terms of content, complexity, and scale. Don't use a dataset that's too small (less than 1TB) because it won't deliver representative performance. Conversely, don't use a dataset that's too large because the POC shouldn't become a full data migration. Be sure to obtain the appropriate benchmarks from existing systems so you can use them for performance comparisons.

+> [!IMPORTANT]

+> Make sure you check with business owners for any blockers before moving any data to the cloud. Identify any security or privacy concerns or any data obfuscation needs that should be done before moving data to the cloud.

+### Create a high-level architecture

+Based upon the high-level architecture of your proposed future state architecture, identify the components that will form part of your POC. Your high-level future state architecture likely contains many data sources, numerous data consumers, big data components, and possibly machine learning and artificial intelligence (AI) data consumers. Your POC architecture should specifically identify components that will be part of the POC. Importantly, it should identify any components that won't form part of the POC testing.

+If you're already using Azure, identify any resources you already have in place (Azure Active Directory, ExpressRoute, and others) that you can use during the POC. Also identify the Azure regions your organization uses. Now is a great time to identify the throughput of your ExpressRoute connection and to check with other business users that your POC can consume some of that throughput without adverse impact on production systems.

+For more information, see [Big data architectures](/architecture/data-guide/big-data.md).

+### Identify POC resources

+Specifically identify the technical resources and time commitments required to support your POC. Your POC will need:

+- A business representative to oversee requirements and results.

+- An application data expert, to source the data for the POC and provide knowledge of the existing processes and logic.

+- An Apache Spark and Spark pool expert.

+- An expert advisor, to optimize the POC tests.

+- Resources that will be required for specific components of your POC project, but not necessarily required for the duration of the POC. These resources could include network admins, Azure admins, Active Directory admins, Azure portal admins, and others.

+- Ensure all the required Azure services resources are provisioned and the required level of access is granted, including access to storage accounts.

+- Ensure you have an account that has required data access permissions to retrieve data from all data sources in the POC scope.

+> [!TIP]

+> We recommend engaging an expert advisor to assist with your POC. [Microsoft's partner community](https://appsource.microsoft.com/marketplace/partner-dir) has global availability of expert consultants who can help you assess, evaluate, or implement Azure Synapse.

+### Set the timeline

+Review your POC planning details and business needs to identify a time frame for your POC. Make realistic estimates of the time that will be required to complete the POC goals. The time to complete your POC will be influenced by the size of your POC dataset, the number and complexity of tests, and the number of interfaces to test. If you estimate that your POC will run longer than four weeks, consider reducing the POC scope to focus on the highest priority goals. Be sure to obtain approval and commitment from all the lead resources and sponsors before continuing.

+## Put the POC into practice

+We recommend you execute your POC project with the discipline and rigor of any production project. Run the project according to plan and manage a change request process to prevent uncontrolled growth of the POC's scope.

+Here are some examples of high-level tasks:

+1. [Create a Synapse workspace](../quickstart-create-workspace.md), Spark pools and dedicated SQL pools, storage accounts, and all Azure

+resources identified in the POC plan.

+1. Load POC dataset:

+ - Make data available in Azure by extracting from the source or by creating sample data in Azure. For more information, see:

+ - [Transferring data to and from Azure](/architecture/databox/data-box-overview.md#use-cases)

+ - [Azure Data Box](https://azure.microsoft.com/services/databox/)

+ - [Copy activity performance and scalability guide](/data-factory/copy-activity-performance#copy-performance-and-scalability-achievable-using-adf.md)

+ - [Data loading strategies for dedicated SQL pool in Azure Synapse Analytics](../sql-data-warehouse/design-elt-data-loading.md)

+ - [Bulk load data using the COPY statement](../sql-data-warehouse/quickstart-bulk-load-copy-tsql.md?view=azure-sqldw-latest&preserve-view=true)

+ - Test the dedicated connector for the Spark pool and the dedicated SQL pool.

+1. Migrate existing code to the Spark pool:

+ - If you're migrating from Spark, your migration effort is likely to be straightforward given that the Spark pool leverages the open-source Spark distribution. However, if you're using vendor specific features on top of core Spark features, you'll need to correctly map these features to the Spark pool features.

+ - If you're migrating from a non-Spark system, your migration effort will vary based on the complexity involved.

+1. Execute the tests:

+ - Many tests can be executed in parallel across multiple Spark pool clusters.

+ - Record your results in a consumable and readily understandable format.

+1. Monitor for troubleshooting and performance. For more information, see:

+ - [Monitor Apache Spark activities](../get-started-monitor.md#apache-spark-activities)

+ - [Monitor with web user interfaces - Spark's history server](https://spark.apache.org/docs/3.0.0-preview/web-ui.html)

+ - [Monitoring resource utilization and query activity in Azure Synapse Analytics](../../sql-data-warehouse/sql-data-warehouse-concept-resource-utilization-query-activity.md)

+1. Monitor data skewness, time skewness and executor usage percentage by opening the **Diagnostic** tab of Spark's history server.

+ :::image type="content" source="media/proof-of-concept-playbook-apache-spark-pool/apache-spark-history-server-diagnostic-tab.png" alt-text="Image shows the Diagnostic tab of Spark's history server.":::

+## Interpret the POC results

+When you complete all the POC tests, you evaluate the results. Begin by evaluating whether the POC goals were met and the desired outputs were collected. Determine whether more testing is necessary or any questions need addressing.

+## Next steps

+> [!div class="nextstepaction"]

+> [Data lake exploration with dedicated SQL pool in Azure Synapse Analytics](proof-of-concept-playbook-dedicated-sql-pool.md)

+> [!div class="nextstepaction"]

+> [Data lake exploration with serverless SQL pool in Azure Synapse Analytics](proof-of-concept-playbook-serverless-sql-pool.md)

+> [!div class="nextstepaction"]

+> [Azure Synapse Analytics frequently asked questions](../overview-faq.yml)

+ Title: Success by design

+description: "TODO: Success by design"

+# Success by design

+Welcome to the Azure Synapse Customer Success Engineering Success by Design repository.

+As part of the Synapse Engineering Group at Microsoft, the Azure Synapse Customer Success Engineering (CSE) team is relied upon to help with complex projects that involve Azure Synapse. As a team of experienced professionals, we provide guidance for all facets of Azure Synapse, including working with customers on architecture reviews, configuration guidance, and performance analysis and recommendations. It's from this experience that the CSE team created this repository of guidance articles to help you produce successful solutions.

+This guidance is intended to supplement the official Azure Synapse documentation with additional specialized content that:

+- Helps you evaluate Azure Synapse by using a Proof of Concept (POC) project.

+- Helps you successfully implement a solution that incorporates Azure Synapse by using our [implementation success method](implementation-success-overview.md). This method can guide you through the key phases of your implementation, including planning, solution development, deployment, and post go-live evaluation.

+- Provides detailed guidance on complex topics, including security, networking, troubleshooting, performance tuning, and migration.

+> [!NOTE]

+> This guidance undergoes continuous improvement by the Synapse CSE team. In time, guidance articles will cover additional topics, including enterprise development support using continuous integration and continuous deployment (CI/CD) methods, business continuity and disaster recovery (BCDR), high availability (HA), and Azure Synapse monitoring.

+Next steps:

+> [!div class="nextstepaction"]

+> [Synapse proof of concept playbook](proof-of-concept-playbook-overview.md)

+> [!div class="nextstepaction"]

+> [Synapse implementation success method](implementation-success-overview.md)

+> [!div class="nextstepaction"]

+> [Security white paper](security-white-paper-introduction.md)

+## May 2022

+Here's what changed in May 2022:

+### Background effects with Teams on Azure Virtual Desktop now generally available

+Users can now make meetings more personalized and avoid unexpected distractions by applying background effects. Meeting participants can select an available image in Teams to change their background or choose to blur their background. For more information, see [our blog post](https://techcommunity.microsoft.com/t5/azure-virtual-desktop-blog/microsoft-teams-background-effects-is-now-generally-available-on/ba-p/3401961).

+### Multi-window and "Call me with Teams" features now generally available

+The multi-window feature gives users the option to pop out chats, meetings, calls, or documents into separate windows to streamline their workflow. The "Call me" feature lets users transfer a Teams call to their phone. Both features are now generally available in Teams on Azure Virtual Desktop. For more information, see [our blog post](https://techcommunity.microsoft.com/t5/azure-virtual-desktop-blog/microsoft-teams-multi-window-support-and-call-me-are-now-in-ga/ba-p/3401830).

+### Japan metadata service in public preview

+The Azure Virtual Desktop metadata database located in Japan is now in public preview. This allows customers to store their Azure Virtual Desktop objects and metadata within a database located within our Japan geography, ensuring that the data will only reside within Japan. For more information, see [our blog post](https://techcommunity.microsoft.com/t5/azure-virtual-desktop/announcing-the-public-preview-of-the-azure-virtual-desktop/m-p/3417497).

+### FSLogix 2201 hotfix

+The latest update for FSLogix 2201 includes fixes to Cloud Cache and container redirection processes. No new features are included with this update. Learn more at [WhatΓÇÖs new in FSLogix](/fslogix/whats-new?context=%2Fazure%2Fvirtual-desktop%2Fcontext%2Fcontext) and [our blog post](https://techcommunity.microsoft.com/t5/azure-virtual-desktop/announcing-fslogix-2201-hotfix-1-2-9-8171-14983-has-been/m-p/3435445).

+> [!IMPORTANT]

+> You can't currently create a Flexible virtual machine scale set from an image shared by another tenant.

+> [!IMPORTANT]

+> You can't currently create a Flexible virtual machine scale set from an image shared by another tenant.

+1. Leave the defaults for the **Disks** page.

+1. On the **Networking** page, under **Load balancing**, select the **Use a load balancer** option to put the scale set instances behind a load balancer.

+> [!IMPORTANT]

+> You can't currently create a Flexible virtual machine scale set from an image shared by another tenant.

+1. In the left menu, select **Monitor**.

+1. In the Monitor menu, select **Service Health**.

+ :::image type="content" source="./media/virtual-machine-scale-sets-maintenance-notifications/monitor-service-health.png" alt-text="Select Service Health in the Monitor menu.":::

+1. In Service Health, select **+ Create service health alert**.

+ :::image type="content" source="./media/virtual-machine-scale-sets-maintenance-notifications/monitor-create-service-health-alert.png" alt-text="Select Create service health alert button.":::

+1. On the **Create an alert rule** page:

+ 1. Select the relevant **Subscription** and **Region** containing the resources to monitor for planned maintenance events.

+ 1. Specify the following:

+ - **Services**: *Virtual Machine Scale Sets* and *Virtual Machines*

+ - **Event type**: *Planned maintenance*

+1. Under **Actions**, add action groups to the alert rule in order to send notifications or invoke actions when a planned maintenance event is received.

+1. Fill out the details under **Alert rule details**.

+1. Select **Create alert rule**.

+| Cross tenant shared image gallery | No | Yes | Yes |

+| Managed Identity | [User Assigned Identity](../active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vmss.md#user-assigned-managed-identity) only¹ | System Assigned or User Assigned | N/A (can specify Managed Identity on individual instances) |

+¹ For Uniform scale sets, the `GET VMSS` response will have a reference to the *identity*, *clientID*, and *principalID*. For Flexible scale sets, the response will only get a reference the *identity*. You can make a call to `Identity` to get the *clientID* and *PrincipalID*.

+| Cross tenant shared image gallery | No |

+| Managed Identity | [User Assigned Identity](../active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vmss.md#user-assigned-managed-identity) only¹ |

+¹ For Uniform scale sets, the `GET VMSS` response will have a reference to the *identity*, *clientID*, and *principalID*. For Flexible scale sets, the response will only get a reference the *identity*. You can make a call to `Identity` to get the *clientID* and *PrincipalID*.

+### 2022-02-14

+**Improvements**:

+- [Validation Support](https://docs.microsoft.com/azure/virtual-machines/linux/image-builder-json#properties-validate)

+ - Shell (Linux) - Script or Inline

+ - PowerShell (Windows) - Script or Inline, run elevated, run as system

+ - Source-Validation-Only mode

+- [Customized staging resource group support](https://docs.microsoft.com/azure/virtual-machines/linux/image-builder-json#properties-stagingresourcegroup)

+ - Added support for [Gen2 images.](image-builder-overview.md#hyper-v-generation)

+[Flatcar Container Linux developer SDK guide](https://www.flatcar.org/docs/latest/reference/developer-guides/). When

+WAAGENT=/usr/sbin/waagent

+waagent -version 1> 2>&1

+if [ $? -eq 0 ]; then

+ WAAGENT=waagent

+fi

+$WAAGENT -force -deprovision+user && export HISTSIZE=0 && sync

+> [!IMPORTANT]

+> You can't currently create a Flexible virtual machine scale set from an image shared by another tenant.

+> [!IMPORTANT]

+> You can't currently create a Flexible virtual machine scale set from an image shared by another tenant.

+ Title: Compute benchmark scores for Azure Windows VMs

+description: Compare Coremark compute benchmark scores for Azure VMs running Windows Server.

+| Standard_F2s_v2 | Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70 GHz | 2 | 1 | 4.0 | 34,903 | 1,101 | 3.15% | 112 |

+| Standard_F2s_v2 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 2 | 1 | 4.0 | 34,738 | 1,331 | 3.83% | 224 |

+| Standard_F4s_v2 | Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70 GHz | 4 | 1 | 8.0 | 66,828 | 1,524 | 2.28% | 168 |

+| Standard_F4s_v2 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 4 | 1 | 8.0 | 66,903 | 1,047 | 1.57% | 182 |

+| Standard_F8s_v2 | Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70 GHz | 8 | 1 | 16.0 | 131,477 | 2,180 | 1.66% | 140 |

+| Standard_F8s_v2 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 8 | 1 | 16.0 | 132,533 | 1,732 | 1.31% | 210 |

+| Standard_F16s_v2 | Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70 GHz | 16 | 1 | 32.0 | 260,760 | 3,629 | 1.39% | 112 |

+| Standard_F16s_v2 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 16 | 1 | 32.0 | 265,158 | 2,185 | 0.82% | 182 |

+| Standard_F32s_v2 | Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70 GHz | 32 | 1 | 64.0 | 525,608 | 6,270 | 1.19% | 98 |

+| Standard_F32s_v2 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 32 | 1 | 64.0 | 530,137 | 6,085 | 1.15% | 140 |

+| Standard_F48s_v2 | Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70 GHz | 48 | 2 | 96.0 | 769,768 | 7,567 | 0.98% | 112 |

+| Standard_F48s_v2 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 48 | 1 | 96.0 | 742,828 | 17,316 | 2.33% | 112 |

+| Standard_F64s_v2 | Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70 GHz | 64 | 2 | 128.0 | 1,030,552 | 8,106 | 0.79% | 70 |

+| Standard_F64s_v2 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 64 | 2 | 128.0 | 1,028,052 | 9,373 | 0.91% | 168 |

+| Standard_F72s_v2 | Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70 GHz | 72 | 2 | 144.0 | N/A | - | - | - |

+| Standard_F72s_v2 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 72 | 2 | 144.0 | N/A | - | - | - |

+| Standard_F1s | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 1 | 1 | 2.0 | 16,445 | 825 | 5.02% | 42 |

+| Standard_F1s | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 1 | 1 | 2.0 | 17,614 | 2,873 | 16.31% | 210 |

+| Standard_F1s | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 1 | 1 | 2.0 | 16,053 | 1,802 | 11.22% | 70 |

+| Standard_F1s | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 1 | 1 | 2.0 | 20,007 | 1,684 | 8.42% | 28 |

+| Standard_F2s | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 2 | 1 | 4.0 | 33,451 | 3,424 | 10.24% | 70 |

+| Standard_F2s | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 2 | 1 | 4.0 | 33,626 | 2,990 | 8.89% | 154 |

+| Standard_F2s | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 2 | 1 | 4.0 | 34,386 | 3,851 | 11.20% | 98 |

+| Standard_F2s | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 2 | 1 | 4.0 | 36,826 | 344 | 0.94% | 28 |

+| Standard_F4s | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 4 | 1 | 8.0 | 67,351 | 4,407 | 6.54% | 42 |

+| Standard_F4s | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 4 | 1 | 8.0 | 67,009 | 4,637 | 6.92% | 196 |

+| Standard_F4s | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 4 | 1 | 8.0 | 63,668 | 3,375 | 5.30% | 84 |

+| Standard_F4s | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 4 | 1 | 8.0 | 79,153 | 15,034 | 18.99% | 28 |

+| Standard_F8s | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 8 | 1 | 16.0 | 128,232 | 1,272 | 0.99% | 42 |

+| Standard_F8s | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 8 | 1 | 16.0 | 127,871 | 5,109 | 4.00% | 154 |

+| Standard_F8s | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 8 | 1 | 16.0 | 122,811 | 5,481 | 4.46% | 126 |

+| Standard_F8s | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 8 | 1 | 16.0 | 154,842 | 10,354 | 6.69% | 28 |

+| Standard_F16s | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 16 | 2 | 32.0 | 260,883 | 15,853 | 6.08% | 42 |

+| Standard_F16s | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 16 | 1 | 32.0 | 255,762 | 4,966 | 1.94% | 182 |

+| Standard_F16s | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 16 | 1 | 32.0 | 248,884 | 11,035 | 4.43% | 70 |

+| Standard_F16s | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 16 | 1 | 32.0 | 310,303 | 21,942 | 7.07% | 28 |

+| Standard_F1 | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 1 | 1 | 2.0 | 17,356 | 1,151 | 6.63% | 112 |

+| Standard_F1 | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 1 | 1 | 2.0 | 16,508 | 1,740 | 10.54% | 154 |

+| Standard_F1 | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 1 | 1 | 2.0 | 16,076 | 2,065 | 12.84% | 70 |

+| Standard_F1 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 1 | 1 | 2.0 | 20,074 | 1,612 | 8.03% | 14 |

+| Standard_F2 | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 2 | 1 | 4.0 | 32,770 | 1,915 | 5.84% | 126 |

+| Standard_F2 | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 2 | 1 | 4.0 | 33,081 | 2,242 | 6.78% | 126 |

+| Standard_F2 | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 2 | 1 | 4.0 | 33,310 | 2,532 | 7.60% | 84 |

+| Standard_F2 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 2 | 1 | 4.0 | 40,746 | 2,027 | 4.98% | 14 |

+| Standard_F4 | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 4 | 1 | 8.0 | 65,694 | 3,512 | 5.35% | 126 |

+| Standard_F4 | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 4 | 1 | 8.0 | 65,054 | 3,457 | 5.31% | 154 |

+| Standard_F4 | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 4 | 1 | 8.0 | 61,607 | 3,662 | 5.94% | 56 |

+| Standard_F4 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 4 | 1 | 8.0 | 76,884 | 1,763 | 2.29% | 14 |

+| Standard_F8 | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 8 | 1 | 16.0 | 130,415 | 5,353 | 4.10% | 98 |

+| Standard_F8 | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 8 | 1 | 16.0 | 126,139 | 2,917 | 2.31% | 126 |

+| Standard_F8 | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 8 | 1 | 16.0 | 122,443 | 4,391 | 3.59% | 98 |

+| Standard_F8 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 8 | 1 | 16.0 | 144,696 | 2,172 | 1.50% | 14 |

+| Standard_F16 | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 16 | 2 | 32.0 | 253,473 | 8,597 | 3.39% | 140 |

+| Standard_F16 | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 16 | 1 | 32.0 | 257,457 | 7,596 | 2.95% | 126 |

+| Standard_F16 | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 16 | 1 | 32.0 | 244,559 | 8,036 | 3.29% | 70 |

+| Standard_F16 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 16 | 1 | 32.0 | 283,565 | 8,683 | 3.06% | 14 |

+| Standard_GS1 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 2 | 1 | 28.0 | 35,593 | 2,888 | 8.11% | 252 |

+| Standard_GS2 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 4 | 1 | 56.0 | 72,188 | 5,949 | 8.24% | 252 |

+| Standard_GS3 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 8 | 1 | 112.0 | 132,665 | 6,910 | 5.21% | 238 |

+| Standard_GS4 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 16 | 1 | 224.0 | 261,542 | 3,722 | 1.42% | 252 |

+| Standard_GS4-4 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 4 | 1 | 224.0 | 79,352 | 4,935 | 6.22% | 224 |

+| Standard_GS4-8 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 8 | 1 | 224.0 | 137,774 | 6,887 | 5.00% | 238 |

+| Standard_GS5 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.0 0GHz | 32 | 2 | 448.0 | 507,026 | 6,895 | 1.36% | 252 |

+| Standard_GS5-8 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 8 | 2 | 448.0 | 157,541 | 3,151 | 2.00% | 238 |

+| Standard_GS5-16 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 16 | 2 | 448.0 | 278,656 | 5,235 | 1.88% | 224 |

+| Standard_G1 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 2 | 1 | 28.0 | 36,386 | 4,100 | 11.27% | 252 |

+| Standard_G2 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 4 | 1 | 56.0 | 72,484 | 5,563 | 7.67% | 252 |

+| Standard_G3 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 8 | 1 | 112.0 | 136,618 | 5,714 | 4.18% | 252 |

+| Standard_G4 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 16 | 1 | 224.0 | 261,708 | 3,426 | 1.31% | 238 |

+| Standard_G5 | Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00 GHz | 32 | 2 | 448.0 | 507,423 | 7,261 | 1.43% | 252 |

+| Standard_B1ms | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 1 | 1 | 2.0 | 18,093 | 679 | 3.75% | 42 |

+| Standard_B1ms | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 1 | 1 | 2.0 | 18,197 | 1,341 | 7.37% | 168 |

+| Standard_B1ms | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 1 | 1 | 2.0 | 17,975 | 920 | 5.12% | 112 |

+| Standard_B1ms | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 1 | 1 | 2.0 | 20,176 | 1,568 | 7.77% | 28 |

+| Standard_B2s | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 2 | 1 | 4.0 | 35,546 | 660 | 1.86% | 42 |

+| Standard_B2s | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 2 | 1 | 4.0 | 36,569 | 2,172 | 5.94% | 154 |

+| Standard_B2s | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 2 | 1 | 4.0 | 36,136 | 924 | 2.56% | 140 |

+| Standard_B2s | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 2 | 1 | 4.0 | 42,546 | 834 | 1.96% | 14 |

+| Standard_B2hms | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 2 | 1 | 8.0 | 36,949 | 1,494 | 4.04% | 28 |

+| Standard_B2hms | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 2 | 1 | 8.0 | 36,512 | 2,537 | 6.95% | 70 |

+| Standard_B2hms | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 2 | 1 | 8.0 | 36,389 | 990 | 2.72% | 56 |

+| Standard_B2ms | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 2 | 1 | 8.0 | 35,758 | 1,028 | 2.88% | 42 |

+| Standard_B2ms | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 2 | 1 | 8.0 | 36,028 | 1,605 | 4.45% | 182 |

+| Standard_B2ms | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 2 | 1 | 8.0 | 36,122 | 2,128 | 5.89% | 112 |

+| Standard_B2ms | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 2 | 1 | 8.0 | 42,525 | 672 | 1.58% | 14 |

+| Standard_B4hms | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 4 | 1 | 16.0 | 71,028 | 879 | 1.24% | 28 |

+| Standard_B4hms | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 4 | 1 | 16.0 | 73,126 | 2,954 | 4.04% | 56 |

+| Standard_B4hms | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 4 | 1 | 16.0 | 68,451 | 1,571 | 2.29% | 56 |

+| Standard_B4hms | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 4 | 1 | 16.0 | 83,525 | 563 | 0.67% | 14 |

+| Standard_B4ms | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 4 | 1 | 16.0 | 70,831 | 1,135 | 1.60% | 28 |

+| Standard_B4ms | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 4 | 1 | 16.0 | 70,987 | 2,287 | 3.22% | 168 |

+| Standard_B4ms | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 4 | 1 | 16.0 | 68,796 | 1,897 | 2.76% | 84 |

+| Standard_B4ms | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 4 | 1 | 16.0 | 81,712 | 4,042 | 4.95% | 70 |

+| Standard_B8ms | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 8 | 1 | 32.0 | 141,620 | 2,256 | 1.59% | 42 |

+| Standard_B8ms | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 8 | 1 | 32.0 | 139,090 | 3,229 | 2.32% | 182 |

+| Standard_B8ms | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 8 | 1 | 32.0 | 135,510 | 2,653 | 1.96% | 112 |

+| Standard_B8ms | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 8 | 1 | 32.0 | 164,510 | 2,254 | 1.37% | 14 |

+| Standard_B12ms | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 12 | 1 | 48.0 | 206,957 | 5,240 | 2.53% | 56 |

+| Standard_B12ms | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 12 | 1 | 48.0 | 211,461 | 4,115 | 1.95% | 154 |

+| Standard_B12ms | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 12 | 1 | 48.0 | 200,729 | 3,475 | 1.73% | 140 |

+| Standard_B16ms | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 16 | 2 | 64.0 | 273,257 | 3,862 | 1.41% | 42 |

+| Standard_B16ms | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 16 | 1 | 64.0 | 282,187 | 5,030 | 1.78% | 154 |

+| Standard_B16ms | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 16 | 1 | 64.0 | 265,834 | 5,545 | 2.09% | 112 |

+| Standard_B16ms | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 16 | 1 | 64.0 | 331,694 | 3,537 | 1.07% | 28 |

+| Standard_B20ms | Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40 GHz | 20 | 2 | 80.0 | 334,369 | 8,555 | 2.56% | 42 |

+| Standard_B20ms | Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30 GHz | 20 | 1 | 80.0 | 345,686 | 6,702 | 1.94% | 154 |

+| Standard_B20ms | Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60 GHz | 20 | 1 | 80.0 | 328,900 | 7,625 | 2.32% | 126 |

+| Standard_B20ms | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60 GHz | 20 | 1 | 80.0 | 409,515 | 4,792 | 1.17% | 14 |

+| Standard_D96as_v4 | AMD EPYC 7452 32-Core Processor | 96 | 12 | 384.0 | N/A | - | - | - |

+| Standard_D96a_v4 | AMD EPYC 7452 32-Core Processor | 96 | 12 | 384.0 | N/A | - | - | - |

+| Standard_E96as_v4 | AMD EPYC 7452 32-Core Processor | 96 | 12 | 672.0 | N/A | - | - | - |

+| Standard_E96a_v4 | AMD EPYC 7452 32-Core Processor | 96 | 12 | 672.0 | N/A | - | - | - |

+### Edsv4 Isolated Extended

+(04/05/2021 PBIID:9198755)

+| VM Size | CPU | vCPUs | NUMA Nodes | Memory(GiB) | Avg Score | StdDev | StdDev% | #Runs |

+| | | : | : | : | : | : | : | : |

+| Standard_E80ids_v4 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz | 80 | 2 | 504.0 |N/A | - | - | - |

+### EIASv4

+(04/05/2021 PBIID:9198755)

+| VM Size | CPU | vCPUs | NUMA Nodes | Memory(GiB) | Avg Score | StdDev | StdDev% | #Runs |

+| | | : | : | : | : | : | : | : |

+| Standard_E96ias_v4 | AMD EPYC 7452 32-Core Processor | 96 | 12 | 672.0 | N/A | - | - | - |

+### Esv4 Isolated Extended

+| VM Size | CPU | vCPUs | NUMA Nodes | Memory(GiB) | Avg Score | StdDev | StdDev% | #Runs |

+| | | : | : | : | : | : | : | : |

+| Standard_E80is_v4 | Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz | 80 | 2 | 504.0 | N/A | - | - | - |

+[CoreMark](https://www.eembc.org/coremark/faq.php) is a benchmark that tests the functionality of a microctronoller (MCU) or central processing unit (CPU). CoreMark isn't system dependent, so it functions the same regardless of the platform (for example, big or little endian, high-end or low-end processor).

+Windows numbers were computed by running CoreMark on Windows Server 2019. CoreMark was configured with the number of threads set to the number of virtual CPUs, and concurrency set to `PThreads`. The target number of iterations was adjusted based on expected performance to provide a runtime of at least 20 seconds (typically much longer). The final score represents the number of iterations completed divided by the number of seconds it took to run the test. Each test was run at least seven times on each VM. Test run dates shown above. Tests run on multiple VMs across Azure public regions the VM was supported in on the date run. (Coremark doesn't properly support more than 64 vCPUs on Windows, therefore SKUs with > 64 vCPUs have been marked as N/A.)

+```run1.log``` contains CoreMark results with performance parameters.

+- Don't change source files other than ```core_portme*``` (use ```make check``` to validate)

+* For storage capacities, disk details, and other considerations for choosing among VM sizes, see [Sizes for virtual machines](../sizes.md).

+- June 02, 2022: Change in [HA for SAP NetWeaver on Azure VMs on Windows with Azure NetApp Files(SMB)](./high-availability-guide-windows-netapp-files-smb.md), [HA for SAP NW on Azure VMs on SLES with ANF](./high-availability-guide-suse-netapp-files.md) and [HA for SAP NW on Azure VMs on RHEL with ANF](./high-availability-guide-rhel-netapp-files.md) to add sizing considerations

+- The throughput and performance characteristics of an Azure NetApp Files volume is a function of the volume quota and service level, as documented in [Service level for Azure NetApp Files](../../../azure-netapp-files/azure-netapp-files-service-levels.md). While sizing the SAP Azure NetApp volumes, make sure that the resulting throughput meets the application requirements.

+- The minimum capacity pool is 4 TiB. The capacity pool size can be increased in 1 TiB increments.

+- The throughput and performance characteristics of an Azure NetApp Files volume is a function of the volume quota and service level, as documented in [Service level for Azure NetApp Files](../../../azure-netapp-files/azure-netapp-files-service-levels.md). While sizing the SAP Azure NetApp volumes, make sure that the resulting throughput meets the application requirements.

+### Important considerations

+When considering Azure NetApp Files for the SAP Netweaver architecture, be aware of the following important considerations:

+- The minimum capacity pool is 4 TiB. The capacity pool size can be increased in 1 TiB increments.

+- The minimum volume is 100 GiB

+- The selected virtual network must have a subnet, delegated to Azure NetApp Files.

+- The throughput and performance characteristics of an Azure NetApp Files volume is a function of the volume quota and service level, as documented in [Service level for Azure NetApp Files](../../../azure-netapp-files/azure-netapp-files-service-levels.md). While sizing the SAP Azure NetApp volumes, make sure that the resulting throughput meets the application requirements.

+

+## Availability Zone

+Public IP addresses with a Standard SKU can be created as non-zonal, zonal, or zone-redundant in [regions that support availability zones](../../availability-zones/az-region.md). A zone-redundant IP is created in all zones for a region and can survive any single zone failure. A zonal IP is tied to a specific availability zone, and shares fate with the health of the zone. A "non-zonal" public IP addresses is placed into a zone for you by Azure and does not give a guarantee of redundancy.

+In regions without availability zones, all public IP addresses are created as non-zonal. Public IP addresses created in a region that is later upgraded to have availability zones remain non-zonal.

+> [!NOTE]

+> All Basic SKU public IP addresses are created as non-zonal. Any IP that is upgraded from a Basic SKU to Standard SKU remains non-zonal.

+- [Public load balancer](/azure/virtual-network/ip-services/configure-public-ip-load-balancer)

+- [Application Gateway](/azure/virtual-network/ip-services/configure-public-ip-application-gateway)

+- [NAT gateway](/azure/virtual-network/ip-services/configure-public-ip-nat-gateway)

+- [Azure Bastion](/azure/virtual-network/ip-services/configure-public-ip-bastion)

+- [Azure Firewall](/azure/virtual-network/ip-services/configure-public-ip-firewall)

+> * Configure virtual hub Basic settings

+> * Connect a site to a virtual hub

+> * Connect a VPN site to a virtual hub

+> * Connect a VNet to a virtual hub

+## <a name="hub"></a>Configure virtual hub settings

+A virtual hub is a virtual network that can contain gateways for site-to-site, ExpressRoute, or point-to-site functionality. For this tutorial, you begin by filling out the **Basics** tab for the virtual hub and then continue on to fill out the site-to-site tab in the next section. It's also possible to create an empty virtual hub (a virtual hub that doesn't contain any gateways) and then add gateways (S2S, P2S, ExpressRoute, etc.) later. Once a virtual hub is created, you'll be charged for the virtual hub, even if you don't attach any sites or create any gateways within the virtual hub.

+Don't create the virtual hub yet. Continue on to the next section to configure additional settings.

+In this section, you configure site-to-site connectivity settings, and then proceed to create the virtual hub and site-to-site VPN gateway. A virtual hub and gateway can take about 30 minutes to create.

+In this section, you create site. Sites correspond to your physical locations. Create as many sites as you need. For example, if you have a branch office in NY, a branch office in London, and a branch office and LA, you'd create three separate sites. These sites contain your on-premises VPN device endpoints. You can create up to 1000 sites per virtual hub in a virtual WAN. If you had multiple virtual hubs, you can create 1000 per each of those virtual hubs. If you have Virtual WAN partner CPE device, check with them to learn about their automation to Azure. Typically, automation implies a simple click experience to export large-scale branch information into Azure, and setting up connectivity from the CPE to Azure Virtual WAN VPN gateway. For more information, see [Automation guidance from Azure to CPE partners](virtual-wan-configure-automation-providers.md).

+## <a name="connectsites"></a>Connect the VPN site to a virtual hub

+In this section, you connect your VPN site to the virtual hub.

+## <a name="vnet"></a>Connect a VNet to the virtual hub

+In this section, you create a connection between the virtual hub and your VNet.

+ * **Address space** of the VNets that are connected to the virtual hub.<br>Example:

+Use the [Get-AzVpnClientConfiguration](/powershell/module/az.network/get-azvpnclientconfiguration) cmdlet to generate the VPN client configuration for EapMSChapv2.

+# Configure WAF policies using Azure Firewall Manager (preview)

+> Configure Web Application Firewall policies using Azure Firewall Manager is currently in PREVIEW.

+Azure Firewall Manager is a platform to manage and protect your network security resources at scale. You can associate your WAF policies to an Application Gateway or Azure Front Door within Azure Firewall Manager, all in a single place.

+## View and manage WAF policies

+In Azure Firewall Manager, you can create and view all WAF policies in one central place across subscriptions and regions.

+To navigate to WAF policies, select the **Web Application Firewall Policies** tab on the left, under **Security**.

+## Associate of dissociate WAF policies

+In Azure Firewall Manager, you can create and view all WAF policies in your subscriptions. These policies can be associated or dissociated with an application delivery platform. Select the service and then select **Manage Security**.

+## Upgrade Application Gateway WAF configuration to WAF policy

+For Application Gateway with WAF configuration, you can upgrade the WAF configuration to a WAF policy associated with Application Gateway.

+The WAF policy can be shared to multiple application gateways. Also, a WAF policy allows you to take advantage of advanced and new features like bot protection, newer rule sets, and reduced false positives. New features are only released on WAF policies.

+To upgrade a WAF configuration to a WAF policy, select **Upgrade from WAF configuration** from the desired application gateway.