+1. In **Valid OAuth redirect URIs**, enter `https://your-tenant-name.b2clogin.com/your-tenant-id.onmicrosoft.com/oauth2/authresp`. If you use a [custom domain](custom-domain.md), enter `https://your-domain-name/your-tenant-id.onmicrosoft.com/oauth2/authresp`. Replace `your-tenant-id` with the id of your tenant, and `your-domain-name` with your custom domain.

+## Known Issues

+* Azure AD B2C does not support JWE (JSON Web Encryption) for exchanging encrypted tokens with OpenID connect identity providers.

+1. For **Key usage**, select `Signature`.

+> If you're facing `unauthorized` error while testing this identity provider, make sure you use the correct Twitter API Key and API Key Secret, or try to apply for [elevated](https://developer.twitter.com/en/portal/products/elevated) access. Also, we recommend you've a look at [Twitter's projects structure](https://developer.twitter.com/en/docs/projects/overview), if you registered your app before the feature was available.

+[cURL](https://curl.se/) is a popular, free, open-source, command-line tool used by API developers, and it's [available by default on Windows 10/11](https://curl.se/windows/microsoft.html). This tutorial describes how you can quickly test [API-driven inbound provisioning](inbound-provisioning-api-concepts.md) with cURL.

+1. Upon successful upload, you'll receive HTTP 202 Accepted response code.

+1. Click on any record in the provisioning logs to view more processing details.

+- [Quick start using PowerShell](inbound-provisioning-api-powershell.md)

+- [Quick start using Azure Logic Apps](inbound-provisioning-api-logic-apps.md)

+- [Quick start using cURL](inbound-provisioning-api-curl-tutorial.md)

+- [Quick start using Postman](inbound-provisioning-api-postman.md)

+- [Quick start using Postman](inbound-provisioning-api-graph-explorer.md)

+- [Quick start using PowerShell](inbound-provisioning-api-powershell.md)

+- [Quick start using Azure Logic Apps](inbound-provisioning-api-logic-apps.md)

+This tutorial describes how to use Azure Logic Apps workflow to implement Microsoft Entra ID [API-driven inbound provisioning](inbound-provisioning-api-concepts.md). Using the steps in this tutorial, you can convert a CSV file containing HR data into a bulk request payload and send it to the Microsoft Entra ID provisioning [/bulkUpload](/graph/api/synchronization-synchronizationjob-post-bulkupload) API endpoint. The article also provides guidance on how the same integration pattern can be used with any system of record.

+### Business requirement

+Your system of record periodically generates CSV file exports containing worker data. You want to implement an integration that reads data from the CSV file and automatically provisions user accounts in your target directory (on-premises Active Directory for hybrid users and Microsoft Entra ID for cloud-only users).

+### Implementation requirement

+From an implementation perspective:

+* You want to use an Azure Logic Apps workflow to read data from the CSV file exports available in an Azure File Share and send it to the inbound provisioning API endpoint.

+* In your Azure Logic Apps workflow, you don't want to implement the complex logic of comparing identity data between your system of record and target directory.

+* You want to use Microsoft Entra ID provisioning service to apply your IT managed provisioning rules to automatically create/update/enable/disable accounts in the target directory (on-premises Active Directory or Microsoft Entra ID).

+### Integration scenario variations

+While this tutorial uses a CSV file as a system of record, you can customize the sample Azure Logic Apps workflow to read data from any system of record. Azure Logic Apps provides a wide range of [built-in connectors](/azure/logic-apps/connectors/built-in/reference) and [managed connectors](/connectors/connector-reference/connector-reference-logicapps-connectors) with pre-built triggers and actions that you can use in your integration workflow.

+Here's a list of enterprise integration scenario variations, where API-driven inbound provisioning can be implemented with a Logic Apps workflow.

+|# |System of record |Integration guidance on using Logic Apps to read source data |

+||||

+| 1 | Files stored on SFTP server | Use either the [built-in SFTP connector](/azure/logic-apps/connectors/built-in/reference/sftp/) or [managed SFTP SSH connector](/azure/connectors/connectors-sftp-ssh) to read data from files stored on the SFTP server. |

+| 2 | Database table | If you're using an Azure SQL server or on-premises SQL Server, use the [SQL Server](/azure/connectors/connectors-create-api-sqlazure) connector to read your table data. <br> If you're using an Oracle database, use the [Oracle database](/azure/connectors/connectors-create-api-oracledatabase) connector to read your table data. |

+| 3 | On-premises and cloud-hosted SAP S/4 HANA or <br> Classic on-premises SAP systems, such as R/3 and ECC | Use the [SAP connector](/azure/logic-apps/logic-apps-using-sap-connector) to retrieve identity data from your SAP system. For examples on how to configure this connector, refer to [common SAP integration scenarios](/azure/logic-apps/sap-create-example-scenario-workflows) using Azure Logic Apps and the SAP connector. |

+| 4 | IBM MQ | Use the [IBM MQ connector](/azure/connectors/connectors-create-api-mq) to receive provisioning messages from the queue. |

+| 5 | Dynamics 365 Human Resources | Use the [Dataverse connector](/azure/connectors/connect-common-data-service) to read data from [Dataverse tables](/dynamics365/human-resources/hr-developer-entities) used by Microsoft Dynamics 365 Human Resources. |

+| 6 | Any system that exposes REST APIs | If you don't find a connector for your system of record in the Logic Apps connector library, You can create your own [custom connector](/azure/logic-apps/logic-apps-create-api-app) to read data from your system of record. |

+After reading the source data, apply your pre-processing rules and convert the output from your system of record into a bulk request that can be sent to the Microsoft Entra ID provisioning [bulkUpload](/graph/api/synchronization-synchronizationjob-post-bulkupload) API endpoint.

+> [!IMPORTANT]

+> If you'd like to share your API-driven inbound provisioning + Logic Apps integration workflow with the community, create a [Logic app template](/azure/logic-apps/logic-apps-create-azure-resource-manager-templates), document steps on how to use it and submit a pull request for inclusion in the GitHub repository [Entra-ID-Inbound-Provisioning](https://github.com/AzureAD/entra-id-inbound-provisioning).

+## How to use this tutorial

+The Logic Apps deployment template published in the [Microsoft Entra ID inbound provisioning GitHub repository](https://github.com/AzureAD/entra-id-inbound-provisioning/tree/main/LogicApps/CSV2SCIMBulkUpload) automates several tasks. It also has logic for handling large CSV files and chunking the bulk request to send 50 records in each request. Here's how you can test it and customize it per your integration requirements.

+|# | Automation task | Implementation guidance |

+||||

+|1 | Read worker data from the CSV file. | The Logic Apps workflow uses an Azure Function to read the CSV file stored in an Azure File Share. The Azure Function converts CSV data into JSON format. If your CSV file format is different, update the workflow step "Parse JSON" and "Construct SCIMUser". <br> If your system of record is different, check guidance provided in the section [Integration scenario variations](#integration-scenario-variations) on how you can customize the Logic Apps workflow by using an appropriate connector. |

+|2 | Pre-process and convert data to SCIM format. | By default, the Logic Apps workflow converts each record in the CSV file to a SCIM Core User + Enterprise User representation. If you plan to use custom SCIM schema extensions, you can update the step "Construct SCIMUser" to include your custom SCIM schema extensions. If you want to run C# code for advanced formatting and data validation, you can use [custom Azure Functions](../../logic-apps/logic-apps-azure-functions.md).|

+|3 | Use the right authentication method | You can either [use a service principal](inbound-provisioning-api-grant-access.md#configure-a-service-principal) or [use managed identity](inbound-provisioning-api-grant-access.md#configure-a-managed-identity) to access the inbound provisioning API. Update the step "Send SCIMBulkPayload to API endpoint" with the right authentication method. |

+|4 | Provision accounts in on-premises Active Directory or Microsoft Entra ID. | Configure [API-driven inbound provisioning app](inbound-provisioning-api-configure-app.md). This will generate a unique [/bulkUpload](/graph/api/synchronization-synchronizationjob-post-bulkupload) API endpoint. Update the step "Send SCIMBulkPayload to API endpoint" to use the right bulkUpload API endpoint. |

+|5 | Scan the provisioning logs and retry provisioning for failed records. | This automation is not yet implemented in the sample Logic Apps workflow. To implement it refer to the [provisioning logs Graph API](/graph/api/resources/provisioningobjectsummary) |

+|6 | Deploy your Logic Apps based automation to production. | Once you have verified your API-driven provisioning flow and customized the Logic Apps workflow to meet your requirements, you can deploy the automation in your environment. |

+- [Quick start using PowerShell](inbound-provisioning-api-powershell.md)

+- [Quick start using Azure Logic Apps](inbound-provisioning-api-logic-apps.md)

+This tutorial describes how to use a PowerShell script to implement Microsoft Entra ID [API-driven inbound provisioning](inbound-provisioning-api-concepts.md). Using the steps in this tutorial, you can convert a CSV file containing HR data into a bulk request payload and send it to the Microsoft Entra ID provisioning [/bulkUpload](/graph/api/synchronization-synchronizationjob-post-bulkupload) API endpoint. The article also provides guidance on how the same integration pattern can be used with any system of record.

+## Integration scenario

+### Business requirement

+Your system of record periodically generates CSV file exports containing worker data. You want to implement an integration that reads data from the CSV file and automatically provisions user accounts in your target directory (on-premises Active Directory for hybrid users and Microsoft Entra ID for cloud-only users).

+### Implementation requirement

+From an implementation perspective:

+* You want to use an unattended PowerShell script to read data from the CSV file exports and send it to the inbound provisioning API endpoint.

+* In your PowerShell script, you don't want to implement the complex logic of comparing identity data between your system of record and target directory.

+* You want to use Microsoft Entra ID provisioning service to apply your IT managed provisioning rules to automatically create/update/enable/disable accounts in the target directory (on-premises Active Directory or Microsoft Entra ID).

+### Integration scenario variations

+While this tutorial uses a CSV file as a system of record, you can customize the sample PowerShell script to read data from any system of record. Here's a list of enterprise integration scenario variations, where API-driven inbound provisioning can be implemented with a PowerShell script.

+|# |System of record |Integration guidance on using PowerShell to read source data |

+||||

+|1 | Database table | If you're using an Azure SQL database or an on-premises SQL Server, you can use the [Read-SqlTableData](/powershell/module/sqlserver/read-sqltabledata) cmdlet to read data stored in a table of a SQL database. You can use the [Invoke-SqlCmd](/powershell/module/sqlserver/invoke-sqlcmd) cmdlet to run Transact-SQL or XQuery scripts. <br> If you're using an Oracle / MySQL / Postgres database, you can find a PowerShell module either published by the vendor or available in the [PowerShell Gallery](https://www.powershellgallery.com/). Use the module to read data from your database table. |

+|2 | LDAP server | Use the `System.DirectoryServices.Protocols` .NET API or one of the LDAP modules available in the [PowerShell Gallery](https://www.powershellgallery.com/packages?q=ldap) to query your LDAP server. Understand the LDAP schema and hierarchy to retrieve user data from the LDAP server. |

+|3 | Any system that exposes REST APIs | To read data from a REST API endpoint using PowerShell, you can use the [Invoke-RestMethod](/powershell/module/microsoft.powershell.utility/invoke-restmethod) cmdlet from the `Microsoft.PowerShell.Utility` module. Check the documentation of your REST API and find out what parameters and headers it expects, what format it returns, and what authentication method it uses. You can then adjust your `Invoke-RestMethod` command accordingly. |

+|4 | Any system that exposes SOAP APIs | To read data from a SOAP API endpoint using PowerShell, you can use the [New-WebServiceProxy](/powershell/module/microsoft.powershell.management/new-webserviceproxy) cmdlet from the `Microsoft.PowerShell.Management` module. Check the documentation of your SOAP API and find out what parameters and headers it expects, what format it returns, and what authentication method it uses. You can then adjust your `New-WebServiceProxy` command accordingly. |

+After reading the source data, apply your pre-processing rules and convert the output from your system of record into a bulk request that can be sent to the Microsoft Entra ID provisioning [bulkUpload](/graph/api/synchronization-synchronizationjob-post-bulkupload) API endpoint.

+> [!IMPORTANT]

+> If you'd like to share your PowerShell integration script with the community, publish it on [PowerShell Gallery](https://www.powershellgallery.com/) and notify us on the GitHub repository [Entra-ID-Inbound-Provisioning](https://github.com/AzureAD/entra-id-inbound-provisioning), so we can add a reference it.

+## How to use this tutorial

+The PowerShell sample script published in the [Microsoft Entra ID inbound provisioning GitHub repository](https://github.com/AzureAD/entra-id-inbound-provisioning/tree/main/PowerShell/CSV2SCIM) automates several tasks. It has logic for handling large CSV files and chunking the bulk request to send 50 records in each request. Here's how you can test it and customize it per your integration requirements.

+|1 | Read worker data from the CSV file. | [Download the PowerShell script](#download-the-powershell-script). It has out-of-the-box logic to read data from any CSV file. Refer to [CSV2SCIM PowerShell usage details](#csv2scim-powershell-usage-details) to get familiar with the different execution modes of this script. <br> If your system of record is different, check guidance provided in the section [Integration scenario variations](#integration-scenario-variations) on how you can customize the PowerShell script. |

+|6 | Deploy your PowerShell based automation to production. | Once you have verified your API-driven provisioning flow and customized the PowerShell script to meet your requirements, you can deploy the automation as a [PowerShell Workflow runbook in Azure Automation](../../automation/learn/automation-tutorial-runbook-textual.md) or as a server process [scheduled to run on a Windows server](/troubleshoot/windows-server/system-management-components/schedule-server-process). |

+ ```powershell

+ @{

+ externalId = 'WorkerID'

+ name = @{

+ familyName = 'LastName'

+ givenName = 'FirstName'

+ }

+ active = { $_.'WorkerStatus' -eq 'Active' }

+ userName = 'UserID'

+ displayName = 'FullName'

+ nickName = 'UserID'

+ userType = 'WorkerType'

+ title = 'JobTitle'

+ addresses = @(

+ @{

+ type = { 'work' }

+ streetAddress = 'StreetAddress'

+ locality = 'City'

+ postalCode = 'ZipCode'

+ country = 'CountryCode'

+ }

+ )

+ phoneNumbers = @(

+ @{

+ type = { 'work' }

+ value = 'OfficePhone'

+ }

+ )

+ "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User" = @{

+ employeeNumber = 'WorkerID'

+ costCenter = 'CostCenter'

+ organization = 'Company'

+ division = 'Division'

+ department = 'Department'

+ manager = @{

+ value = 'ManagerID'

+ }

+ }

+ }

+ ```

+ 1. Sign in to the [Azure portal](https://portal.azure.com) as an administrator.

+ 1. Sign in to the [Azure portal](https://portal.azure.com).

+ 1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Sign in to the [Azure portal](https://portal.azure.com), go to **Enterprise applications** > **Application** > **Provisioning** and select **Authorize**.

+1. Sign in to the [Azure portal](https://portal.azure.com) with a hybrid administrator account.

+1. Sign in to the [Azure portal](https://portal.azure.com) and open your Workday to AD/Azure AD user provisioning app.

+1. Sign in to the [Azure portal](https://portal.azure.com) as an administrator.

+1. Sign in to the [Azure portal](https://portal.azure.com) as a global administrator.

+1. Sign in to the [Azure portal](https://portal.azure.com) as the test user by using CBA. The authentication policy is set where Issuer subject rule satisfies single-factor authentication.

+1. Sign in to the [Azure portal](https://portal.azure.com) using CBA. Since the policy was set to satisfy multifactor authentication, the user sign-in is successful without a second factor.

+1. Sign in to the [Azure portal](https://portal.azure.com) as a Global Administrator.

+1. Sign in to the [Azure portal](https://portal.azure.com) and search for the user account from which the FIDO key is to be removed.

+1. Sign in to the [Azure portal](https://portal.azure.com) by using one of the preceding roles.

+| **NPS Extension for Azure AD MFA (AccessReject):** <br> NPS Extension for Azure AD MFA only performs Secondary Auth for Radius requests in AccessAccept State. Request received for User username with response state AccessReject, ignoring request. | This error usually reflects an authentication failure in AD or that the NPS server is unable to receive responses from Azure AD. Verify that your firewalls are open bidirectionally for traffic to and from `https://adnotifications.windowsazure.com` and `https://login.microsoftonline.com` using ports 80 and 443. It is also important to check that on the DIAL-IN tab of Network Access Permissions, the setting is set to "control access through NPS Network Policy". This error can also trigger if the user is not assigned a license. |

+| **NPS Extension for Azure AD MFA (AccessChallenge):** <br> NPS Extension for Azure AD MFA only performs Secondary Auth for Radius requests in AccessAccept State. Request received for User username with response state AccessChallenge, ignoring request. | This response is used when additional information is required from the user to complete the authentication or authorization process. The NPS server sends a challenge to the user, requesting further credentials or information. It usually preceeds an Access-Accept or Access-Reject response. |

+1. Open a new browser window in InPrivate or incognito mode and sign in to the [Azure portal](https://portal.azure.com).

+1. Close the browser window, and sign in to the [Azure portal](https://portal.azure.com) again to test the authentication method that you configured. For example, if you configured a mobile app for authentication, you should see a prompt like the following.

+1. Sign in to the [Azure portal](https://portal.azure.com) and create a [new support request](https://portal.azure.com/#create/Microsoft.Support).

+- Web APIs must validate access tokens sent to them by a client. They must only accept tokens containing one of their AppId URIs as the `aud` claim.

+- Web apps must validate ID tokens sent to them by using the user's browser in the hybrid flow, before allowing access to a user's data or establishing a session.

+If none of the above scenarios apply, there's no need to validate the token, and this may present a security and reliability risk when basing decisions on the validity of the token. Public clients like native, desktop or single-page applications don't benefit from validating ID tokens because the application communicates directly with the IDP where SSL protection ensures the ID tokens are valid. They shouldn't validate the access tokens, as these are for the web API to validate, not the client.

+The Azure AD middleware has built-in capabilities for validating access tokens, see [samples](sample-v2-code.md) to find one in the appropriate language. There are also several third-party open-source libraries available for JWT validation. For more information about Azure AD authentication libraries and code samples, see the [authentication libraries](reference-v2-libraries.md). If you web app or web API is on ASP.NET or ASP.NET Core, use Microsoft.Identity.Web which handles the validation for you.

+### v1.0 and v2.0 tokens

+- When your web app/API is validating a v1.0 token (`ver` claim ="1.0"), it needs to read the OpenID Connect metadata document from the v1.0 endpoint (`https://login.microsoftonline.com/{example-tenant-id}/.well-known/openid-configuration`), even if the authority configured for your web API is a v2.0 authority.

+- When your web app/API is validating a v2.0 token (`ver` claim ="2.0"), it needs to read the OpenID Connect metadata document from the v2.0 endpoint (`https://login.microsoftonline.com/{example-tenant-id}/v2.0/.well-known/openid-configuration`), even if the authority configured for your web API is a v1.0 authority.

+The examples below suppose that your application is validating a v2.0 access token (and therefore reference the v2.0 versions of the OIDC metadata documents and keys). Just remove the "/v2.0" in the URL if you validate v1.0 tokens.

+### Validate the issuer

+Web apps validating ID tokens, and web APIs validating access tokens need to validate the issuer of the token (`iss` claim) against:

+1. the issuer available in the OpenID connect metadata document associated with the application configuration (authority). The metadata document to verify against depends on:

+ - the version of the token

+ - the accounts supported by your application.

+1. the tenant ID (`tid` claim) of the token,

+1. the issuer of the signing key.

+#### Single tenant applications

+[OpenID Connect Core](https://openid.net/specs/openid-connect-core-1_0.html#IDTokenValidation) says "The Issuer Identifier \[...\] MUST exactly match the value of the `iss` (issuer) Claim." For applications that use a tenant-specific metadata endpoint (like [https://login.microsoftonline.com/{example-tenant-id}/v2.0/.well-known/openid-configuration](https://login.microsoftonline.com/{example-tenant-id}/v2.0/.well-known/openid-configuration) or [https://login.microsoftonline.com/contoso.onmicrosoft.com/v2.0/.well-known/openid-configuration](https://login.microsoftonline.com/contoso.onmicrosoft.com/v2.0/.well-known/openid-configuration)), this is all that is needed.

+This applies to applications which support:

+- Accounts in one organizational directory (**example-tenant-id** only): `https://login.microsoftonline.com/{example-tenant-id}`

+- Personal Microsoft accounts only: `https://login.microsoftonline.com/consumers` (**consumers** being a nickname for the tenant 9188040d-6c67-4c5b-b112-36a304b66dad)

+#### Multi-tenant applications

+Azure AD also supports multi-tenant applications. These applications support:

+- Accounts in any organizational directory (any Azure AD directory): `https://login.microsoftonline.com/organizations`

+- Accounts in any organizational directory (any Azure AD directory) and personal Microsoft accounts (e.g. Skype, XBox): `https://login.microsoftonline.com/common`

+For these applications Azure AD exposes tenant-independent versions of the OIDC document at [https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration](https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration) and [https://login.microsoftonline.com/organizations/v2.0/.well-known/openid-configuration](https://login.microsoftonline.com/organizations/v2.0/.well-known/openid-configuration) respectively. These endpoints return an issuer value, which is a template parametrized by the `tenantid`: `https://login.microsoftonline.com/{tenantid}/v2.0`. Applications may use these tenant-independent endpoints to validate tokens from every tenant with the following modifications: instead of expecting the issuer claim in the token to exactly match the issuer value from metadata, the application should replace the `{tenantid}` value in the issuer metadata with the tenant ID that is the target of the current request, and then check the exact match (`tid` claim of the token).

+### Validate the signing key issuer

+In addition to the issuer of the token, applications using the v2.0 tenant-independant metadata need to validate the signing key issuer.

+#### Keys document and signing key issuer

+As discussed, from the OpenID Connect document, your application accesses the keys used to sign the tokens. It gets the corresponding keys document by accessing the URL exposed in the **jwks_uri** property of the OpenIdConnect document.

+```json

+ "jwks_uri": "https://login.microsoftonline.com/{example-tenant-id}/discovery/v2.0/keys",

+```

+As above, `{example-tenant-id}` can be replaced by a GUID, a domain name, or **common**, **organizations** and **consumers**

+The "keys" documents exposed by Azure AD v2.0 contains, for each key, the issuer that uses this signing key. See, for instance, the

+tenant-independent "common" key endpoint [https://login.microsoftonline.com/common/discovery/v2.0/keys](https://login.microsoftonline.com/common/discovery/v2.0/keys) returns a document like:

+

+ ```json

+ {

+ "keys":[

+ {"kty":"RSA","use":"sig","kid":"jS1Xo1OWDj_52vbwGNgvQO2VzMc","x5t":"jS1Xo1OWDj_52vbwGNgvQO2VzMc","n":"spv...","e":"AQAB","x5c":["MIID..."],"issuer":"https://login.microsoftonline.com/{tenantid}/v2.0"},

+ {"kty":"RSA","use":"sig","kid":"2ZQpJ3UpbjAYXYGaXEJl8lV0TOI","x5t":"2ZQpJ3UpbjAYXYGaXEJl8lV0TOI","n":"wEM...","e":"AQAB","x5c":["MIID..."],"issuer":"https://login.microsoftonline.com/{tenantid}/v2.0"},

+ {"kty":"RSA","use":"sig","kid":"yreX2PsLi-qkbR8QDOmB_ySxp8Q","x5t":"yreX2PsLi-qkbR8QDOmB_ySxp8Q","n":"rv0...","e":"AQAB","x5c":["MIID..."],"issuer":"https://login.microsoftonline.com/9188040d-6c67-4c5b-b112-36a304b66dad/v2.0"}

+ ]

+ }

+ ```

+#### Validation of the signing key issuer

+The application should use the `issuer` property of the keys document, associated with the key used to sign the token, in order to restrict the scope of keys:

+- Keys that have an issuer value with a GUID like `https://login.microsoftonline.com/9188040d-6c67-4c5b-b112-36a304b66dad/v2.0` should only be used when the `iss` claim in the token matches the value exactly.

+- Keys that have a templated issuer value like `https://login.microsoftonline.com/{tenantid}/v2.0` need to ensure that:

+ - the `tid` claim is a GUID and the `iss` claim is of the form `https://login.microsoftonline.com/{tid}/v2.0` where `{tid}` is the exact `tid` claim. This ties the tenant back to the issuer. back to the scope of the signing key creating a chain of trust.

+ - Multi-tenant applications must use `tid` claim when they locate data associated with the subject of the claim. In other words, the `tid` claim must be part of the key used to access the user's data.

+Using tenant-independent metadata is more efficient for applications that accept tokens from many tenants.

+> [!NOTE]

+> With Azure AD tenant-independent metadata, claims should be interpreted within the tenant, just as under standard OpenID Connect, claims are interpreted within the issuer. That is, `{"sub":"ABC123","iss":"https://login.microsoftonline.com/{example-tenant-id}/v2.0","tid":"{example-tenant-id}"}` and `{"sub":"ABC123","iss":"https://login.microsoftonline.com/{another-tenand-id}/v2.0","tid":"{another-tenant-id}"}` describe different users, even though the `sub` is the same, because claims like `sub` are interpreted within the context of the issuer/tenant.

+#### Recap

+Here is some pseudo code that recapitulates how to validate issuer and signing key issuer:

+1. Fetch keys from configured metadata URL

+1. Check token if signed with one of the published keys, fail if not

+1. Identify key in the metadata based on the kid header. Check the "issuer" property attached to the key in the metadata document:

+ ```c

+ var issuer = metadata["kid"].issuer;

+ if (issuer.contains("{tenantId}", CaseInvariant)) issuer = issuer.Replace("{tenantid}", token["tid"], CaseInvariant);

+ if (issuer != token["iss"]) throw validationException;

+ if (configuration.allowedIssuer != "*" && configuration.allowedIssuer != issuer) throw validationException;

+ var issUri = new Uri(token["iss"]);

+ if (issUri.Segments.Count < 1) throw validationException;

+ if (issUri.Segments[1] != token["tid"]) throw validationException;

+ ```

+The **Microsoft Enterprise SSO plug-in for Apple devices** provides single sign-on (SSO) for Azure Active Directory (Azure AD) accounts on macOS, iOS, and iPadOS across all applications that support Apple's [enterprise single sign-on](https://developer.apple.com/documentation/authenticationservices) feature. The plug-in provides SSO for even old applications that your business might depend on but that don't yet support the latest identity libraries or protocols. Microsoft worked closely with Apple to develop this plug-in to increase your application's usability while providing the best protection available.

+| `AppPrefixAllowList` | String<br/>*(comma-delimited list)* | Bundle ID prefixes of applications allowed to participate in SSO. For iOS, the default value would be set to `com.apple.` and that would enable SSO for all Apple apps. For macOS, the default value would be set to `com.apple.` and `com.microsoft.` and that would enable SSO for all Apple and Microsoft apps. Developers, Customers, or Admins could override the default value or add apps to `AppBlockList` to prevent them from participating in SSO. |

+> [!TIP]

+> Learn more about how the SSO plug-in works and how to troubleshoot the Microsoft Enterprise SSO Extension with the [SSO troubleshooting guide for Apple devices](../devices/troubleshoot-mac-sso-extension-plugin.md).

+Learn about [troubleshooting the Microsoft Enterprise SSO Extension](../devices/troubleshoot-mac-sso-extension-plugin.md).

+With a multi-tenant application, because the application can't immediately tell which tenant the user is from, requests can't be sent to a tenantΓÇÖs endpoint. Instead, requests are sent to an endpoint that multiplexes across all Azure AD tenants: `https://login.microsoftonline.com/common`.

+> [!NOTE]

+> There are, in reality 2 authorities for multi-tenant applications:

+> - `https://login.microsoftonline.com/common` for applications processing accounts in any organizational directory (any Azure AD directory) and personal Microsoft accounts (e.g. Skype, XBox).

+> - `https://login.microsoftonline.com/organizations` for applications processing accounts in any organizational directory (any Azure AD directory):

+>

+> The explanations in this document use `common`. But you can replace it by `organizations` if your application doesn't support Microsoft personal accounts.

+## Update your code to handle multiple issuer values

+Web applications and web APIs receive and validate tokens from the Microsoft identity platform. Native client applications don't validate access tokens and must treat them as opaque. They instead request and receive tokens from the Microsoft identity platform, and do so to send them to APIs, where they're then validated.

+Multi-tenant applications must perform additional checks when validating a token. A multi-tenant application is configured to consume keys metadata from `/organizations` or `/common` keys URLs. The application must validate that the `issuer` property in the published metadata matches the `iss` claim in the token, in addition to the usual check that the `iss` claim in the token contains the tenant ID (`tid`) claim. For more information see [Validate tokens](access-tokens.md#validate-tokens).

+* [Access tokens](access-tokens.md)

+- [Invalidate refresh token](https://learn.microsoft.com/powershell/module/microsoft.graph.beta.users.actions/invoke-mgbetainvalidateuserrefreshtoken?view=graph-powershell-beta.md)

+1. Sign in to the [Azure portal](https://portal.azure.com) to access your tenant. Select **Azure Active Directory**. Select **Security** in the left navigation pane and then select **Conditional Access**.

+1. Sign in to the [Azure portal](https://portal.azure.com) by using an account that has access to create VMs, and then select **+ Create a resource**.

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Sign in to the [Azure portal](https://portal.azure.com).

+| Credential | Challenge and response authentication types like **Kerberos** (on-premises Active Directory Domain Services)| Request is sent from the application to the authentication server (AD domain controller). Credential extensions are configured with HOSTS in the MDM configuration profile. If the authentication server returns a challenge that matches a host listed in the profile, the operating system routes the challenge to the extension. The extension has the choice of handling or rejecting the challenge. If handled, the extension returns the authorization headers to complete the request, and the authentication server returns a response to the caller. | Request data then get challenged for authentication. Use HOSTs in MDM configuration profile. |

+The following flowchart outlines a logical flow for approaching troubleshooting the SSO Extension. The rest of this article goes into detail on the steps depicted in this flowchart. The troubleshooting can be broken down into two separate focus areas: [Deployment](#deployment-troubleshooting) and [Application Auth Flow](#application-auth-flow-troubleshooting).

+# [iOS](#tab/flowchart-ios)

+# [macOS](#tab/flowchart-macos)

+Most issues that customers encounter stem from either improper Mobile Device Management (MDM) configuration(s) of the SSO extension profile, or an inability for the Apple device to receive the configuration profile from the MDM. This section covers the steps you can take to ensure that the MDM profile has been deployed to a Mac and that it has the correct configuration.

+- Device managed by any MDM vendor that supports [Apple macOS and/or iOS](https://support.apple.com/guide/deployment/dep1d7afa557/web) (MDM Enrollment).

+Use the following steps to check the operating system (OS) version on the macOS device. Apple SSO Extension profiles are only deployed to devices running **macOS 10.15 (Catalina)** or greater. You can check the macOS version from either the [User Interface](#user-interface) or from the [Terminal](#terminal).

+1. From the macOS device, select on the Apple icon in the top left corner and select **About This Mac**.

+1. The Operating system version is listed beside **macOS**.

+1. From the macOS device, double-click on the **Applications** folder, then double-click on the **Utilities** folder.

+1. Double-click on the **Terminal** application.

+#### Check iOS operating system version

+Use the following steps to check the operating system (OS) version on the iOS device. Apple SSO Extension profiles are only deployed to devices running **iOS 13** or greater. You can check the iOS version from the **Settings app**. Open the **Settings app**:

+Navigate to **General** and then **About**. This screen lists information about the device, including the iOS version number:

+1. From the macOS device, select on the **System Settings**.

+1. When the **System Settings** appears type **Profiles** and hit **return**.

+1. This action should bring up the **Profiles** panel.

+ |**2**| There may be multiple profiles to choose from. In this example, the Microsoft Enterprise SSO Extension Profile is called **Extensible Single Sign On Profile-32f37be3-302e-4549-a3e3-854d300e117a**. |

+ :::image type="content" source="media/troubleshoot-mac-sso-extension-plugin/sso-extension-config-profile.png" alt-text="Screenshot showing SSO extension configuration profile.":::

+ |**4**|**Extension**| Identifier that maps to the **bundle ID** of the application that is running the **Microsoft Enterprise Extension Plugin**. The identifier must **always** be set to **`com.microsoft.CompanyPortalMac.ssoextension`** and the Team Identifier must appear as **(UBF8T346G9)** if the profile is installed on a macOS device. If any values differ, then the MDM doesn't invoke the extension correctly.|

+1. From the macOS device, double-click on the **Applications** folder, then double-click on the **Utilities** folder.

+1. Double-click on the **Console** application.

+| [**Native MSAL App**](../develop/apple-sso-plugin.md#applications-that-use-msal) |X|X| MSAL (Microsoft Authentication Library) is an application developer framework tailored for building applications with the Microsoft Identity platform (Azure AD).<br>Apps built on **MSAL version 1.1 or greater** are able to integrate with the Microsoft Enterprise SSO Extension.<br>*If the application is SSO extension (broker) aware it utilizes the extension without any further configuration* for more information, see our [MSAL developer sample documentation](https://github.com/AzureAD/microsoft-authentication-library-for-objc). | Microsoft To Do |

+1. From the macOS device, double-click on the **Applications** folder, then double-click on the **Utilities** folder.

+1. Double-click on the **Terminal** application.

+By default, only MSAL apps invoke the SSO Extension, and then in turn the Extension acquires a shared credential (PRT) from Azure AD. However, the **Safari** browser application or other **Non-MSAL** applications can be configured to acquire the PRT. See [Allow users to sign in from applications that don't use MSAL and the Safari browser](../develop/apple-sso-plugin.md#allow-users-to-sign-in-from-applications-that-dont-use-msal-and-the-safari-browser). After the SSO extension acquires a PRT, it will store the credential in the user login Keychain. Next, check to ensure that the PRT is present in the user's keychain:

+1. From the macOS device, double-click on the **Applications** folder, then double-click on the **Utilities** folder.

+1. Double-click on the **Keychain Access** application.

+ - In the search bar, on the right-hand side, type `primaryrefresh` (To filter).

+ |**6** |**Modified**|Shows when the credential was last updated. For the Azure AD PRT credential, anytime the credential is bootstrapped or updated by an interactive sign-on event it updates the date/timestamp|

+ |**7** |**Keychain** |Indicates which Keychain the selected credential resides. The Azure AD PRT credential resides in the **Local Items** or **iCloud** Keychain. When iCloud is enabled on the macOS device, the **Local Items** Keychain will become the **iCloud** keychain|

+1. From the macOS device, double-click on the **Applications** folder.

+1. Double-click on the **Company Portal** application.

+1. When the **Company Portal** loads, navigate to the top menu bar: **Help**->**Save diagnostic report**. There's no need to Sign into the app.

+#### Tailing SSO extension logs on macOS with terminal

+1. From the macOS device, double-click on the **Applications** folder, then double-click on the **Utilities** folder.

+1. Double-click on the **Terminal** application.

+#### Exporting SSO extension logs on iOS

+It isn't possible to view iOS SSO Extension logs in real time, as it is on macOS. The iOS SSO extension logs can be exported from the Microsoft Authenticator app, and then reviewed from another device:

+1. Open the Microsoft Authenticator app:

+ :::image type="content" source="media/troubleshoot-mac-sso-extension-plugin/auth-app.jpg" alt-text="Screenshot showing the icon of the Microsoft Authenticator app on iOS.":::

+1. Press the menu button in the upper left:

+ :::image type="content" source="media/troubleshoot-mac-sso-extension-plugin/auth-app-menu-button.png" alt-text="Screenshot showing the location of the menu button in the Microsoft Authenticator app.":::

+1. Choose the "Send feedback" option:

+ :::image type="content" source="media/troubleshoot-mac-sso-extension-plugin/auth-app-send-feedback.png" alt-text="Screenshot showing the location of the send feedback option in the Microsoft Authenticator app.":::

+1. Choose the "Having trouble" option:

+ :::image type="content" source="media/troubleshoot-mac-sso-extension-plugin/auth-app-having-trouble.png" alt-text="Screenshot showing the location of having trouble option in the Microsoft Authenticator app.":::

+1. Press the View diagnostic data option:

+ :::image type="content" source="media/troubleshoot-mac-sso-extension-plugin/auth-app-view-diagnostic-data.png" alt-text="Screenshot showing the view diagnostic data button in the Microsoft Authenticator app.":::

+ > [!TIP]

+ > If you are working with Microsoft Support, at this stage you can press the **Send** button to send the logs to support. This will provide you with an Incident ID, which you can provide to your Microsoft Support contact.

+1. Press the "Copy all" button to copy the logs to your iOS device's clipboard. You can then save the log files elsewhere for review or send them via email or other file sharing methods:

+ :::image type="content" source="media/troubleshoot-mac-sso-extension-plugin/auth-app-copy-all-logs.png" alt-text="Screenshot showing the Copy all logs option in the Microsoft Authenticator app.":::

+During the MDM configuration of the Microsoft Enterprise SSO Extension, an optional extension specific data can be sent as instructions to change how the SSO extension behaves. These configuration specific instructions are known as **Feature Flags**. The Feature Flag configuration is especially important for Non-MSAL/Browser SSO authorization requests types, as the Bundle ID can determine if the Extension is invoked or not. See [Feature Flag documentation](../develop/apple-sso-plugin.md#more-configuration-options). Every authorization request begins with a Feature Flag configuration report. The following screenshot walks through an example feature flag configuration:

+|**2**|**browser_sso_disable_mfa**|(Now deprecated) During bootstrapping of the PRT credential, by default MFA is required. Notice this configuration is set to **null** which means that the default configuration is enforced|

+The following section walks through how to examine the SSO extension logs for the Native MSAL Application auth flow. For this example, we're using the [MSAL macOS/iOS sample application](https://github.com/AzureAD/microsoft-authentication-library-for-objc) as the client application, and the application is making a call to the Microsoft Graph API to display the sign-in user's information.

+1. The user signs in to the MSAL macOS sample app.

+1. The Microsoft SSO Extension Broker is invoked and handles the request.

+1. Microsoft SSO Extension Broker undergoes the bootstrapping process to acquire a PRT for the signed in user.

+The user clicks on the **Call Microsoft Graph API** button to invoke the sign-in process.

+Next, examine server-side [Azure AD Sign-in logs](../reports-monitoring/reference-basic-info-sign-in-logs.md#correlation-id) based on the correlation ID collected from the client-side SSO extension logs. For more information, see [Sign-in logs in Azure Active Directory](../reports-monitoring/concept-sign-ins.md).

+There are also non-interactive sign-in events, due to the fact the PRT is used to acquire the access token for the client application's request. Follow the [View Azure AD Sign-in logs by Correlation ID Filter](#view-azure-ad-sign-in-logs-by-correlation-id-filter) but in step 2, select **User sign-ins (non-interactive)**.

+After a period of time, the access token will no longer be valid. So, if the user reclicks on the **Call Microsoft Graph API** button. The SSO extension attempts to refresh the access token with the already acquired PRT.

+|**PRT Refresh and Acquire Access Token** | This operation revalidates the PRT and refreshes it if necessary, before returning the access token back to the calling client application. |

+The following section walks through how to examine the SSO extension logs for the Non-MSAL/Browser Application auth flow. For this example, we're using the Apple Safari browser as the client application, and the application is making a call to the Office.com (OfficeHome) web application.

+1. On a device, with the **Microsoft SSO Extension Broker** deployed, the configured **feature flags** are checked to ensure that the application can be handled by the SSO Extension.

+1. Since the Safari browser adheres to the **Apple Networking Stack**, the SSO extension tries to intercept the Azure AD auth request.

+1. The PRT is used to acquire a token for the resource being requested.

+1. If the device is Azure AD Registered, it passes the Device ID along with the request.

+1. The SSO extension populates the header of the Browser request to sign-in to the resource.

+> [!TIP]

+> If you use Jamf Connect, it is recommended that you follow the [latest Jamf guidance on integrating Jamf Connect with Azure AD](https://learn.jamf.com/bundle/jamf-connect-documentation-current/page/Jamf_Connect_and_Microsoft_Conditional_Access.html). The recommended integration pattern ensures that Jamf Connect works properly with your Conditional Access policies and Azure AD Identity Protection.

+1. Sign in to the [Azure portal](https://portal.azure.com) and open the [Identity Governance](https://portal.azure.com/#blade/Microsoft_AAD_ERM/DashboardBlade/) page.

+needed.

+9. Sign in to the [Azure portal](https://portal.azure.com) again, and remove any new admin account that you created in step 3.

+1. Sign in to the [Azure portal](https://portal.azure.com) with an account that has Global Administrator, Intune Administrator, or User Administrator role permissions.

+1. Sign in to the [Azure portal](https://portal.azure.com) with an account that is the global administrator for your organization.

+1. Sign in to the [Azure portal](https://portal.azure.com) with an account that is a User administrator in the organization.

+## Sponsor field for B2B users (preview)

+You can also manage and track your guest users in the organization using the sponsor feature (preview). The **Sponsor** field on the user account displays who is responsible for the guest user. A sponsor can be a user or a group. To learn more about the sponsor feature (preview), see [Add sponsors to a guest user](b2b-sponsors.md).

+| Manage the B2B account lifecycle with the Sponsor (preview) feature | A sponsor is a user or group responsible for their guest users. For more details about this new feature see [Sponsor field for B2B users (preview)](b2b-sponsors.md).|

+ Title: Add sponsors to a guest user in the Azure portal - Azure AD (preview)

+description: Shows how an admin can add sponsors to guest users in Azure Active Directory (Azure AD) B2B collaboration.

+# Customer intent: As a tenant administrator, I want to know how to add sponsors to guest users in Azure AD.

+# Sponsors field for B2B users (preview)

+To ensure proper governance of B2B users in their directory, organizations need to have a system in place for tracking who oversees each guest user. Currently, Entitlement Management provides this capability for guests within specified domains, but it doesn't extend to guests outside of these domains.

+By implementing the sponsor feature, you can identify a responsible individual or group for each guest user. This allows you to track who invited the guest user and to help with accountability.

+This article provides an overview of the sponsor feature and explains how to use it in B2B scenarios.

+## Sponsors field on the user object

+The **Sponsors** field on the user object refers to the person or a group who invited the guest user to the organization. You can use this field to track who invited the guest user and to help with accountability.

+Being a sponsor doesn't grant administrative powers for the sponsor user or the group, but it can be used for approval processes in Entitlement Management. You can also use it for custom solutions, but it doesn't provide any other built-in directory powers.

+## Who can be a sponsor?

+If you send an invitation to a guest user, you'll automatically become the sponsor of that guest user, unless you specify another user in the invite process as a sponsor. Your name will be added to the **Sponsors** field on the user object automatically. If you want to add a different sponsor, you can also specify the sponsor user or group when sending an invitation to a guest user.

+You can also assign multiple people or groups when inviting the guest user. You can assign a maximum of five sponsors to a single guest user.

+When a sponsor leaves the organization, as part of the offboarding process the tenant administrator can change the **Sponsors** field on the user object to a different person or group. With this transition, they can ensure that the guest user's account remains properly tracked and accounted for.

+## Other scenarios using the B2B sponsors feature

+The Azure Active Directory B2B collaboration sponsor feature serves as a foundation for other scenarios that aim to provide a full governance lifecycle for external partners. These scenarios aren't part of the sponsor feature but rely on it for managing guest users:

+- Administrators can transfer sponsorship to another user or group, if the guest user starts working on a different project.

+- When requesting new access packages, sponsors can be added as approvers to provide additional support in Entitlement Management, which can help reduce the workload on existing reviewers.

+## Add sponsors when inviting a new guest user

+You can add up to five sponsors when inviting a new guest user. If you donΓÇÖt specify a sponsor, the inviter will be added as a sponsor. To invite a guest user, you need to have the Global Administrator role or a limited administrator directory role such as Guest Inviter or User Administrator.

+1. Sign in to the [Azure portal](https://portal.azure.com/) .

+1. Navigate to **Azure Active Directory** > **Users**.

+1. Select **Invite external user** from the menu.

+1. Entered the details on the Basics tab and select **Next: Properties**.

+1. You can add sponsors under ΓÇ»**Job information** on the **Properties** tab,

+ :::image type="content" source="media/b2b-sponsors/add-sponsors.png" alt-text="Screenshot showing the Add sponsor option.":::

+1. Select the **Review and invite** button to finalize the process.

+You can also add sponsors with the Microsoft Graph API, using invitation manager for any new guest users, by passing through the payload. If there are no sponsors in the payload, the inviter will be stamped as the sponsor. To learn more about adding guest users with the Microsoft Graph API, see [Assign sponsors](/graph/api/user-post-sponsors).

+

+## Edit the Sponsors field

+When you invite a guest user, you became their sponsor by default. If you need to manually change the guest user's sponsor, follow these steps:

+1. Sign in to the [Azure portal](https://portal.azure.com) as an Azure AD administrator.

+2. Search for and select **Azure Active Directory** from any page.

+3. Under **Manage**, select **Users**.

+4. In the list, select the user's name to open their user profile

+5. Under **Properties** > **Job information** check the **Sponsors** field. If the guest user already has a sponsor, you can select **View** to see the sponsor's name.

+ :::image type="content" source="media/b2b-sponsors/sponsors-under-properties.png" alt-text="Screenshot of the sponsors field under the job information.":::

+6. Close the window with the sponsor name list, if you want to edit the **Sponsors** field.

+7. There are two ways to edit the **Sponsors** field. Either select the pencil icon next to the **Job Information**, or select **Edit properties** from the top of the page and go to the **Job Information** tab.

+8. If the user has only one sponsor, you can see the sponsor's name:

+ :::image type="content" source="media/b2b-sponsors/single-sponsor.png" alt-text="Screenshot of the sponsors 'name.":::

+ If the user has multiple sponsors, you can't see the individual names:

+ :::image type="content" source="media/b2b-sponsors/multiple-sponsors.png" alt-text="Screenshot of multiple sponsors option.":::

+ To add or remove sponsors, select **Edit**, select or remove the users or groups and select **Save** on the **Job Information** tab.

+9. If the guest user doesn't have a sponsor, select **Add sponsors**.

+ :::image type="content" source="media/b2b-sponsors/add-sponsors-existing-user.png" alt-text="Screenshot of adding a sponsor to an existing user.":::

+10. Once you selected sponsor users or groups, save the changes on the **Job Information** tab.

+## Next steps

+- [Add and invite guest users](add-users-administrator.md)

+- [Crete a new access package](/azure/active-directory/governance/entitlement-management-access-package-create#approval)

+- [Manage user profile info](/azure/active-directory/fundamentals/how-to-manage-user-profile-info)

+1. Sign in to the [Azure portal](https://portal.azure.com) with an account that is a global administrator in the organization.

+1. Sign in to the [Azure portal](https://portal.azure.com) with an account that is a User administrator in the organization.

+B2B collaboration user accounts are the result of inviting guest users to collaborate by using the guest users' own credentials. When the invitation is initially sent to the guest user, an account is created in your tenant. This account doesnΓÇÖt have any credentials associated with it because authentication is performed by the guest user's identity provider. The **Identities** property for the guest user account in your directory is set to the host's organization domain until the guest redeems their invitation. The user sending the invitation is added as a default value for the **Sponsor** (preview) attribute on the guest user account. In the portal, the invited userΓÇÖs profile will show an **External user state** of **PendingAcceptance**. Querying for `externalUserState` using the Microsoft Graph API will return `Pending Acceptance`.

+1. Sign in to the [Azure portal](https://portal.azure.com) as a Conditional Access Administrator, Security Administrator, or Global Administrator.

+1. Sign in to the [Azure portal](https://portal.azure.com) as a Conditional Access Administrator, Security Administrator, or Global Administrator.

+After you get your domain name, you can create your first Azure AD directory. Sign in to the [Azure portal](https://portal.azure.com) for your directory, using an account with the **Owner** role for the subscription.

+1. Sign in to the [Azure portal](https://portal.azure.com) as a tenant administrator.

+- [Add or change profile information for a user in Azure Active Directory](active-directory-users-profile-azure-portal.md)

+1. Sign in to the [Azure portal](https://portal.azure.com) and open the [Identity Governance page](https://portal.azure.com/#blade/Microsoft_AAD_ERM/DashboardBlade/).

+1. Sign in to the [Azure portal](https://portal.azure.com).

+- [What is Conditional Access in Azure Active Directory?](../conditional-access/overview.md)

+1. Sign in to the [Azure portal](https://portal.azure.com) and open the [Identity Governance](https://portal.azure.com/#blade/Microsoft_AAD_ERM/DashboardBlade/) page.

+1. Sign in to the [Azure portal](https://portal.azure.com) and open the [Identity Governance](https://portal.azure.com/#blade/Microsoft_AAD_ERM/DashboardBlade/) page.

+1. Sign in to the [Azure portal](https://portal.azure.com) as a Global Administrator, User Admin or Identity Governance Admin.

+1. Sign in to the [Azure portal](https://portal.azure.com) and open the [Identity Governance page](https://portal.azure.com/#blade/Microsoft_AAD_ERM/DashboardBlade/).

+1. Sign in to the [Azure portal](https://portal.azure.com) as a user who is a Global Administrator. Make sure you have access to the resource group containing the Azure Monitor workspace.

+- [Create interactive reports with Azure Monitor workbooks](../../azure-monitor/visualize/workbooks-overview.md)

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Sign in to the [Azure portal](https://portal.azure.com). Make sure you have access to the subscription or resource group where the Azure Automation account will be located.

+ 1. Sign in to the [Azure portal](https://portal.azure.com).

+ 1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Sign in to the [Azure portal](https://portal.azure.com) as a Global Administrator and select **Azure Active Directory** > **Security** > **Authentication methods** > **Temporary Access Pass**

+ 1. Sign in to the [Azure portal](https://portal.azure.com).

+ 2. Sign in to the [Azure portal](https://portal.azure.com), and then go to **Azure Active Directory**.

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Sign in to the [Azure portal](https://portal.azure.com) as an admin.

+1. Sign in to the [Azure portal](https://portal.azure.com) as an admin.

+1. Sign in to the [Azure portal](https://portal.azure.com) with an administrator account.

+1. Sign in to the [Azure portal](https://portal.azure.com) with an administrator account.

+1. Sign in to the [Azure portal](https://portal.azure.com) with an administrator account.

+1. Sign in to the [Azure portal](https://portal.azure.com) as a Global Administrator.

+1. Sign in to the [Azure portal](https://portal.azure.com) as a Global Administrator.

+1. Sign in to the [Azure portal](https://portal.azure.com) as a Global Administrator.

+- Sign in to the [Azure portal](https://portal.azure.com) as a global administrator, application administrator or cloud application administrator for your directory.

+1. Sign in to the [Azure portal](https://portal.azure.com) with application admin rights.

+Sign in to the [Azure portal](https://portal.azure.com) and go to **Azure AD** and select **Audit log** from the **Monitoring** section.

+ Title: Azure Active Directory SSO integration with WhosOff

+description: Learn how to configure single sign-on between Azure Active Directory and WhosOff.

+# Azure Active Directory SSO integration with WhosOff

+In this article, you'll learn how to integrate WhosOff with Azure Active Directory (Azure AD). WhosOff is an online leave management platform. Azure's WhosOff integration allows customers to sign in to their WhosOff account using Azure as a single sign-on provider. When you integrate WhosOff with Azure AD, you can:

+* Control in Azure AD who has access to WhosOff.

+* Enable your users to be automatically signed-in to WhosOff with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+You'll configure and test Azure AD single sign-on for WhosOff in a test environment. WhosOff supports both **SP** and **IDP** initiated single sign-on.

+## Prerequisites

+To integrate Azure Active Directory with WhosOff, you need:

+* An Azure AD user account. If you don't already have one, you can [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* One of the following roles: Global Administrator, Cloud Application Administrator, Application Administrator, or owner of the service principal.

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* WhosOff single sign-on (SSO) enabled subscription.

+## Add application and assign a test user

+Before you begin the process of configuring single sign-on, you need to add the WhosOff application from the Azure AD gallery. You need a test user account to assign to the application and test the single sign-on configuration.

+### Add WhosOff from the Azure AD gallery

+Add WhosOff from the Azure AD application gallery to configure single sign-on with WhosOff. For more information on how to add application from the gallery, see the [Quickstart: Add application from the gallery](../manage-apps/add-application-portal.md).

+### Create and assign Azure AD test user

+Follow the guidelines in the [create and assign a user account](../manage-apps/add-application-portal-assign-users.md) article to create a test user account in the Azure portal called B.Simon.

+Alternatively, you can also use the [Enterprise App Configuration Wizard](https://portal.office.com/AdminPortal/home?Q=Docs#/azureadappintegration). In this wizard, you can add an application to your tenant, add users/groups to the app, and assign roles. The wizard also provides a link to the single sign-on configuration pane in the Azure portal. [Learn more about Microsoft 365 wizards.](/microsoft-365/admin/misc/azure-ad-setup-guides).

+## Configure Azure AD SSO

+Complete the following steps to enable Azure AD single sign-on in the Azure portal.

+1. In the Azure portal, on the **WhosOff** application integration page, find the **Manage** section and select **single sign-on**.

+1. On the **Select a single sign-on method** page, select **SAML**.

+1. On the **Set up single sign-on with SAML** page, select the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+1. On the **Basic SAML Configuration** section, the user does not have to perform any step as the app is already pre-integrated with Azure.

+1. Perform the following step, if you wish to configure the application in **SP** initiated mode:

+ In the **Sign on URL** textbox, type a URL using the following pattern:

+ `https://app.whosoff.com/int/<Integration_ID>/sso/azure/`

+ > [!NOTE]

+ > This value is not real. Update this value with the actual Sign on URL. Contact [WhosOff support team](mailto:support@whosoff.com) to get the value. You can also refer to the patterns shown in the **Basic SAML Configuration** section in the Azure portal.

+1. On the **Set-up single sign-on with SAML** page, in the **SAML Signing Certificate** section, find **Federation Metadata XML** and select **Download** to download the certificate and save it on your computer.

+ 

+1. On the **Set up WhosOff** section, copy the appropriate URL(s) based on your requirement.

+ 

+## Configure WhosOff SSO

+To configure single sign-on on **WhosOff** side, you need to send the downloaded **Federation Metadata XML** and appropriate copied URLs from Azure portal to [WhosOff support team](mailto:support@whosoff.com). They set this setting to have the SAML SSO connection set properly on both sides.

+### Create WhosOff test user

+In this section, you create a user called Britta Simon at WhosOff SSO. Work with [WhosOff support team](mailto:support@whosoff.com) to add the users in the WhosOff SSO platform. Users must be created and activated before you use single sign-on.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+#### SP initiated:

+* Click on **Test this application** in Azure portal. This will redirect to WhosOff Sign-on URL where you can initiate the login flow.

+* Go to WhosOff Sign-on URL directly and initiate the login flow from there.

+#### IDP initiated:

+* Click on **Test this application** in Azure portal and you should be automatically signed in to the WhosOff for which you set up the SSO.

+You can also use Microsoft My Apps to test the application in any mode. When you click the WhosOff tile in the My Apps, if configured in SP mode you would be redirected to the application sign-on page for initiating the login flow and if configured in IDP mode, you should be automatically signed in to the WhosOff for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Additional resources

+* [What is single sign-on with Azure Active Directory?](../manage-apps/what-is-single-sign-on.md)

+* [Plan a single sign-on deployment](../manage-apps/plan-sso-deployment.md).

+## Next steps

+Once you configure WhosOff you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+```azurecli

+- Access granted to the Azure OpenAI Service in the desired Azure subscription

+The `speak` element contains information such as version, language, and the markup vocabulary definition. The `speak` element is the root element that's required for all SSML documents. You must specify the default language within the `speak` element, whether or not the language is adjusted elsewhere such as within the [`lang`](speech-synthesis-markup-voice.md#adjust-speaking-languages) element.

+The `<lang xml:lang>` element is primarily intended for multilingual neural voices. You can adjust the speaking language for the multilingual neural voice at the sentence level and word level. The supported languages for multilingual voices are [provided in a table](#multilingual-voices-with-the-lang-element) following the `<lang>` syntax and attribute definitions.

+### Multilingual voices with the lang element

+| Voice | Supported locales |

+| - | - |

+| `en-US-JennyMultilingualNeural`¹ | `de-DE`, `en-AU`, `en-CA`, `en-GB`, `es-ES`, `es-MX`, `fr-CA`, `fr-FR`, `it-IT`, `ja-JP`, `ko-KR`, `pt-BR`, `zh-CN` |

+| `en-US-JennyMultilingualV2Neural`² | `ar-EG`, `ar-SA`, `ca-ES`, `cs-CZ`, `da-DK`, `de-AT`, `de-CH`, `de-DE`, `en-AU`, `en-CA`, `en-GB`, `en-HK`, `en-IE`, `en-IN`, `en-US`, `es-ES`, `es-MX`, `fi-FI`, `fr-BE`, `fr-CA`, `fr-CH`, `fr-FR`, `hi-IN`, `hu-HU`, `id-ID`, `it-IT`, `ja-JP`, `ko-KR`, `nb-NO`, `nl-BE`, `nl-NL`, `pl-PL`, `pt-BR`, `pt-PT`, `ru-RU`, `sv-SE`, `th-TH`, `tr-TR`, `zh-CN`, `zh-HK`, `zh-TW`. |

+| `en-US-RyanMultilingualNeural` | `ar-EG`, `ar-SA`, `ca-ES`, `cs-CZ`, `da-DK`, `de-AT`, `de-CH`, `de-DE`, `en-AU`, `en-CA`, `en-GB`, `en-HK`, `en-IE`, `en-IN`, `en-US`, `es-ES`, `es-MX`, `fi-FI`, `fr-BE`, `fr-CA`, `fr-CH`, `fr-FR`, `hi-IN`, `hu-HU`, `id-ID`, `it-IT`, `ja-JP`, `ko-KR`, `nb-NO`, `nl-BE`, `nl-NL`, `pl-PL`, `pt-BR`, `pt-PT`, `ru-RU`, `sv-SE`, `th-TH`, `tr-TR`, `zh-CN`, `zh-HK`, `zh-TW`. |

+¹ In order to speak in a language other than English, the current implementation of the `en-US-JennyMultilingualNeural` voice requires that you set the `<lang xml:lang>` element. We anticipate that during Q4 calendar year 2023, the `en-US-JennyMultilingualNeural` voice will be updated to speak in the language of the input text without the `<lang xml:lang>` element. This will be in parity with the `en-US-JennyMultilingualV2Neural` voice.

+² The `en-US-JennyMultilingualV2Neural` voice is provided temporarily in public preview soley for evaluation purposes. It will be removed in the future.

+> [!NOTE]

+> Multilingual voices don't fully support certain SSML elements, such as break, emphasis, silence, and sub.

+You must specify `en-US` as the default language within the `speak` element, whether or not the language is adjusted elsewhere. In this example, the primary language for `en-US-JennyMultilingualNeural` is `en-US`.

+This SSML snippet shows how to use `<lang xml:lang>` to speak `de-DE` with the `en-US-JennyMultilingualNeural` neural voice.

+2. [Cordon the existing Ubuntu nodes][cordon-and-drain].

+[use-tags]: use-tags.md

+* Synthetic GraphQL APIs currently aren't supported in API Management [workspaces](workspaces-overview.md)

+- [Import a GraphQL schema and set up field resolvers](graphql-schema-resolve-api.md)

+> Importing an OData service as an API from its metadata description is in preview. Currently, testing OData APIs isn't supported in the test console of the Azure portal or in the API Management developer portal.

+description: Learn how to import OData metadata from SAP as an API to Azure API Management, either directly or by converting the metadata to an OpenAPI specification.

+This article shows how to import an OData service using its metadata description. In this article, [SAP Gateway Foundation](https://help.sap.com/viewer/product/SAP_GATEWAY) serves as an example.

+> * Retrieve OData metadata from your SAP service

+> * Import OData metadata to API Management, either directly or after conversion to an OpenAPI specification

+> Importing an OData API to API Management from its metadata description is in preview. [Learn more](import-api-from-odata.md).

+## Retrieve OData metadata from your SAP service

+Retrieve metadata XML from your SAP service, using one of the following methods. If you plan to convert the metadata XML to an OpenAPI specification, save the file locally.

+* Use the SAP Gateway Client (transaction `/IWFND/GW_CLIENT`), or

+* Make a direct HTTP call to retrieve the XML:

+`http://<OData server URL>:<port>/<path>/$metadata`

+## Import API to API Management

+Choose one of the following methods to import your API to API Management: import the metadata XML as an OData API directly, or convert the metadata XML to an OpenAPI specification.

+#### [OData metadata](#tab/odata)

+#### [OpenAPI specification](#tab/openapi)

+## Convert OData metadata to OpenAPI JSON

+* Control access to an SAP backend using API Management policies. For example, if the API is imported as an OData API, use the [validate OData request](validate-odata-request-policy.md) policy. See also policy snippets for [SAP principal propagation](https://github.com/Azure/api-management-policy-snippets/blob/master/examples/Request%20OAuth2%20access%20token%20from%20SAP%20using%20AAD%20JWT%20token.xml) and [fetching an X-CSRF token](https://github.com/Azure/api-management-policy-snippets/blob/master/examples/Get%20X-CSRF%20token%20from%20SAP%20gateway%20using%20send%20request.policy.xml).

+> OpenSSL v3 changed default cipher from 3DES to AES256, but this can be overridden on the command line -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -macalg SHA1.

+Using the new IPs, update any of your resources or networking components to ensure your new environment functions as intended once migration is complete. It's your responsibility to make any necessary updates. This step is also a good time to review the [inbound and outbound network](networking.md#ports-and-network-restrictions) dependency changes when moving to App Service Environment v3 including the port change for the Azure Load Balancer, which now uses port 80. Don't migrate until you've completed this step.

+When the previous step finishes, you'll be shown the IP addresses for your new App Service Environment v3. Using the new IPs, update any resources and networking components to ensure your new environment functions as intended once migration is complete. It's your responsibility to make any necessary updates. This step is also a good time to review the [inbound and outbound network](networking.md#ports-and-network-restrictions) dependency changes when moving to App Service Environment v3 including the port change for the Azure Load Balancer, which now uses port 80. Don't move on to the next step until you confirm that you have made these updates.

+Once the new IPs are created, you have the new default outbound to the internet public addresses. In preparation for the migration, you can adjust any external firewalls, DNS routing, network security groups, and any other resources that rely on these IPs. For ELB App Service Environment, you also have the new inbound IP address that you can use to set up new endpoints with services like [Traffic Manager](../../traffic-manager/traffic-manager-overview.md) or [Azure Front Door](../../frontdoor/front-door-overview.md). **It's your responsibility to update any and all resources that will be impacted by the IP address change associated with the new App Service Environment v3. Don't move on to the next step until you've made all required updates.** This step is also a good time to review the [inbound and outbound network](networking.md#ports-and-network-restrictions) dependency changes when moving to App Service Environment v3 including the port change for the Azure Load Balancer, which now uses port 80.

+For any migration method that doesn't use the [migration feature](migrate.md), you need to [create the App Service Environment v3](creation.md) and a new subnet using the method of your choice. There are [feature differences](overview.md#feature-differences) between App Service Environment v1/v2 and App Service Environment v3 as well as [networking changes](networking.md) that involve new (and for internet-facing environments, additional) IP addresses. You need to update any infrastructure that relies on these IPs as well as account for inbound dependency changes such as the Azure Load Balancer port.

+|Networking dependencies |Must [manage all inbound and outbound traffic](app-service-app-service-environment-network-architecture-overview.md). Network security groups must allow management traffic. |Must [manage all inbound and outbound traffic](network-info.md). Network security groups must allow management traffic. Ensure that [Azure Load Balancer is able to connect to the subnet on port 16001](network-info.md#inbound-dependencies). |No [networking dependencies](networking.md) on the customer's virtual network. Ensure that [Azure Load Balancer is able to connect to the subnet on port 80](networking.md#ports-and-network-restrictions). |

+```bash

+EOF

+```

+## <a name="cpuutil"></a>Scenario 3: Stop automatically based on CPU utilization

+* See [Monitoring Azure resources with Azure Monitor](../azure-monitor/essentials/monitor-azure-resource.md) for details on monitoring Azure resources.

+* See [Monitoring Azure resources with Azure Monitor](../azure-monitor/essentials/monitor-azure-resource.md) for details on monitoring Azure resources.

+> [!NOTE]

+> In order to purchase ESUs, you must have Software Assurance through Volume Licensing Programs such as an Enterprise Agreement (EA), Enterprise Agreement Subscription (EAS), Enrollment for Education Solutions (EES), or Server and Cloud Enrollment (SCE).

+>

+* Find out more about [planning for Windows Server and SQL Server end of support](https://www.microsoft.com/en-us/windows-server/extended-security-updates) and [getting Extended Security Updates](/windows-server/get-started/extended-security-updates-deploy).

+| AZCM0041 | The credentials supplied are invalid | For device logins, verify that the user account specified has access to the tenant and subscription where the server resource will be created^{[1](#footnote3)}.<br> For service principal logins, check the client ID and secret for correctness, the expiration date of the secret^{[2](#footnote4)}, and that the service principal is from the same tenant where the server resource will be created^{[1](#footnote3)}.<br> <a name="footnote3"></a>¹See [How to find your Azure Active Directory tenant ID](/azure/active-directory-b2c/tenant-management-read-tenant-name).<br> <a name="footnote4"></a>²In Azure portal, open Azure Active Directory and select the App registration blade. Select the application to be used and the Certificates and secrets within it. Check whether the expiration data has passed. If it has, create new credentials with sufficient roles and try again. See [Connected Machine agent prerequisites-required permissions](prerequisites.md#required-permissions). |

+* File an Azure support incident. Go to the [Azure support site](https://azure.microsoft.com/support/options/), and select **Get Support**.

+ The `StylesObject` is a `StyleObject` array representing stateset styles. Use the Azure Maps Creator [Feature State service] to apply your stateset styles to indoor map data features. Once you've created your stateset styles and associated them with indoor map features, you can use them to create dynamic indoor maps. For more information on creating dynamic indoor maps, see [Implement dynamic styling for Creator indoor maps].

+* [`BooleanTypeStyleRule`]

+* [`NumericTypeStyleRule`]

+* [`StringTypeStyleRule`]

+The following JSON shows example usage of each of the three style types. The `BooleanTypeStyleRule` is used to determine the dynamic style for features whose `occupied` property is true and false. The `NumericTypeStyleRule` is used to determine the style for features whose `temperature` property falls within a certain range. Finally, the `StringTypeStyleRule` is used to match specific styles to `meetingType`.

+ A `NumericTypeStyleRule` is a [`StyleObject`] and consists of the following properties:

+| `type` | string | Value is `numeric`. | Yes |

+| `rules` | [`NumberRuleObject`][]| An array of numeric style ranges with associated colors. Each range defines a color that's to be used when the *state* value satisfies the range.| Yes |

+A `NumberRuleObject` consists of a [`RangeObject`](#rangeobject) and a `color` property. If the *state* value falls into the range, its color for display is the color specified in the `color` property.

+In the following JSON sample, both ranges hold true when the *state* value is between 50-60. However, the color that is used is `#343deb` because it's the first range in the list that has been satisfied.

+| `range` | [RangeObject] | The [RangeObject] defines a set of logical range conditions, which, if `true`, change the display color of the *state* to the color specified in the `color` property. If `range` is unspecified, then the color defined in the `color` property is always used. | No |

+The `RangeObject` defines a numeric range value of a [`NumberRuleObject`]. For the *state* value to fall into the range, all defined conditions must hold true.

+The following JSON illustrates a `NumericTypeStyleRule` *state* named `temperature`. In this example, the [`NumberRuleObject`] contains two defined temperature ranges and their associated color styles. If the temperature range is 50-69, the display should use the color `#343deb`. If the temperature range is 31-70, the display should use the color `#eba834`.

+A `StringTypeStyleRule` is a [`StyleObject`] and consists of the following properties:

+| `type` | string |Value is `string`. | Yes |

+| `rules` | [`StringRuleObject`][]| An array of N number of *state* values.| Yes |

+| Property | Type | Description | Required |

+||--|--|-|

+| `stateValue1` | string | The color when value string is stateValue1.| No |

+| `stateValueN` | string | The color when value string is stateValueN.| No |

+A `BooleanTypeStyleRule` is a [`StyleObject`] and consists of the following properties:

+| `type` | string |Value is `boolean`. | Yes |

+| `rules` | [`BooleanRuleObject`]| A boolean pair with colors for `true` and `false` *state* values.| Yes |

+The following JSON illustrates a `BooleanTypeStyleRule` *state* named `occupied`. The [`BooleanRuleObject`] defines colors for `true` and `false` values.

+> [Creator for indoor maps]

+[`BooleanRuleObject`]: #booleanruleobject

+[`BooleanTypeStyleRule`]: #booleantypestylerule

+[`NumberRuleObject`]: #numberruleobject

+[`NumericTypeStyleRule`]: #numerictypestylerule

+[`StringRuleObject`]: #stringruleobject

+[`StringTypeStyleRule`]: #stringtypestylerule

+[`StyleObject`]: #styleobject

+[Creator for indoor maps]: creator-indoor-maps.md

+[Feature State service]: /rest/api/maps/v2/feature-state

+[Implement dynamic styling for Creator  indoor maps]: indoor-map-dynamic-styling.md

+[RangeObject]: #rangeobject

+ Title: Set a map style in Android maps

+This article shows you two ways to set map styles using the Azure Maps Android SDK. Azure Maps has six different maps styles to choose from. For more information about supported map styles, see [supported map styles in Azure Maps].

+Be sure to complete the steps in the Quickstart: [Create an Android app].

+>The procedure in this section requires an Azure Maps account in Gen 1 or Gen 2 pricing tier. For more information on pricing tiers, see [Choose the right pricing tier in Azure Maps].

+The aspect ratio of a bounding box may not be the same as the aspect ratio of the map, as such the map often show the full bounding box area, but are often only tight vertically or horizontally.

+| `animationType(AnimationType animationType)` | Specifies the type of animation transition to perform.<br><br> - `JUMP` - an immediate change.<br> - `EASE` - gradual change of the camera's settings.<br> - `FLY` - gradual change of the camera's settings that create an arc resembling flight. |

+This code shows how to animate the map view using a `FLY` animation over a duration of three seconds:

+The above code demonstrates animating the map view from New York to Seattle:

+> [Add a symbol layer]

+> [Add a bubble layer]

+[Add a bubble layer]: map-add-bubble-layer-android.md

+[Add a symbol layer]: how-to-add-symbol-to-android-map.md

+[Choose the right pricing tier in Azure Maps]: choose-pricing-tier.md

+[Create an Android app]: quick-android-map.md

+[supported map styles in Azure Maps]: supported-map-styles.md

+description: This article describes how to set drawing options data using the Microsoft Azure Maps Web SDK

+The Azure Maps Web SDK provides a [drawing tools module]. This module makes it easy to draw and edit shapes on the map using an input device such as a mouse or touch screen. The core class of this module is the [drawing manager]. The drawing manager provides all the capabilities needed to draw and edit shapes on the map. It can be used directly, and it's integrated with a custom toolbar UI. You can also use the built-in [drawing toolbar] class.

+1. Create a new HTML file and [implement the map as usual].

+ - Use the globally hosted, Azure Content Delivery Network version of the Azure Maps services module. Add reference to the JavaScript and CSS in the `<head>` element of the file:

+ - Or, you can load the drawing tools module for the Azure Maps Web SDK source code locally by using the [azure-maps-drawing-tools] npm package, and then host it with your app. This package also includes TypeScript definitions. Use this command:

+ You would also need to embed the CSS for various controls to display correctly. If you're using a JavaScript bundler to bundle the dependencies and package your code, refer to your bundler's documentation on how it's done. For [Webpack], it's commonly done via a combination of `style-loader` and `css-loader` with documentation available at [style-loader].

+Once the drawing tools module is loaded in your application, you can enable drawing and editing capabilities using the [drawing manager]. You can specify options for the drawing manager while instantiating it or alternatively use the `drawingManager.setOptions()` function.

+- `freehand` - Coordinates are added when the mouse or touch is dragged on the map.

+Learn how to use more features of the drawing tools module:

+> [Add a drawing toolbar]

+> [Get shape data]

+> [React to drawing events]

+> [Interaction types and keyboard shortcuts]

+> [Map]

+> [Drawing manager]

+> [drawing toolbar]

+[Add a drawing toolbar]: map-add-drawing-toolbar.md

+[azure-maps-drawing-tools]: https://www.npmjs.com/package/azure-maps-drawing-tools

+[Drawing manager options]: https://samples.azuremaps.com/drawing-tools-module/drawing-manager-options

+[drawing manager]: /javascript/api/azure-maps-drawing-tools/atlas.drawing.drawingmanager

+[drawing toolbar]: /javascript/api/azure-maps-drawing-tools/atlas.control.drawingtoolbar

+[Get shape data]: map-get-shape-data.md

+[implement the map as usual]: how-to-use-map-control.md

+[Interaction types and keyboard shortcuts]: drawing-tools-interactions-keyboard-shortcuts.md

+[Map]: /javascript/api/azure-maps-control/atlas.map

+[React to drawing events]: drawing-tools-events.md

+[style-loader]: https://webpack.js.org/loaders/style-loader/

+[Webpack]: https://webpack.js.org/

+description: This article describes how to display traffic data on a map using the Microsoft Azure Maps iOS SDK.

+Complete the [Create an iOS app] quickstart. Code blocks from this quickstart can be inserted into the `viewDidLoad` function of `ViewController`.

+ |Flow enum | Description |

+ | :-- | :-- |

+Details about a traffic incident are available within the properties of the feature used to display the incident on the map. Traffic incidents are added to the map using the Azure Maps traffic incident vector tile service. The format of the data in these vector tiles can be found in the [Vector Incident Tiles] article on the TomTom site. The following code adds a delegate to the map. This delegate handles a click event, retrieves the traffic incident feature that was selected and displays an alert with some of the details.

+| Category enum | Description |

+|--|-|

+| `IncidentCategory.unknown` | An incident that either doesn't fit any of the defined categories or hasn't yet been classified. |

+| `IncidentCategory.fog` | Fog that reduces visibility, likely reducing traffic flow, and possibly increasing the risk of an accident. |

+| `IncidentCategory.ice` | Icy road conditions that may make driving difficult or dangerous. |

+| `IncidentCategory.jam` | Traffic jam resulting in slower moving traffic. |

+| Magnitude enum | Description |

+The following articles describe different ways to add data to your map:

+- [Add a tile layer]

+- [Add a symbol layer]

+- [Add a bubble layer]

+- [Add a line layer]

+- [Add a polygon layer]

+[Add a bubble layer]: add-bubble-layer-map-ios.md

+[Add a line layer]: add-line-layer-map-ios.md

+[Add a polygon layer]: add-polygon-layer-map-ios.md

+[Add a symbol layer]: add-symbol-layer-ios.md

+[Add a tile layer]: add-tile-layer-map-ios.md

+[Create an iOS app]: quick-ios-app.md

+[Vector Incident Tiles]: https://developer.tomtom.com/traffic-api/traffic-api-documentation-traffic-incidents/vector-incident-tiles

+In addition to styling features, the `SimpleDataLayer` provides a built-in popup feature with a popup template. The popup displays when a feature is clicked. The default popup feature can be disabled, if desired. This layer also supports clustered data. When a cluster is clicked, the map zooms into the cluster and expands it into individual points and subclusters.

+The `SimpleDataLayer` class is used like the other rendering layers are used. The following code shows how to use a simple data layer in a map:

+For example when parsing XML data feeds, you may not know the exact styles and geometry types of the features. The [Simple data layer options] sample shows the power of the simple data layer by rendering the features of a KML file. It also demonstrates various options that the simple data layer class provides. For the source code for this sample, see [Simple data layer options.html] in the Azure Maps code samples in GitHub.

+> This simple data layer uses the [popup template] class to display KML balloons or feature properties as a table. By default, all content rendered in the popup will be sandboxed inside of an iframe as a security feature. However, there are limitations:

+Azure Maps and GitHub style properties are the two main sets of supported property names. Most property names of the different Azure maps layer options are supported as style properties of features in the simple data layer. Expressions have been added to some layer options to support style property names that are commonly used by GitHub. [GitHub's GeoJSON map support] defines these property names, and they're used to style GeoJSON files that are stored and rendered within the platform. All GitHub's styling properties are supported in the simple data layer, except the `marker-symbol` styling properties.

+If the reader comes across a less common style property, it converts it to the closest Azure Maps style property. Additionally, the default style expressions can be overridden by using the `getLayers` function of the simple data layer and updating the options on any of the layers.

+The following sections provide details on the default style properties supported by the simple data layer. The order of the supported property name is also the priority of the property. If two style properties are defined for the same layer option, then the first one in the list has higher precedence. Colors can be any CSS3 color value; HEX, RGB, RGBA, HSL, HSLA, or named color value.

+If a feature is a `Point` or a `MultiPoint`, and the feature doesn't have an `image` property that would be used as a custom icon to render the point as a symbol, then the feature is rendered with a `BubbleLayer`.

+\[1\] The `size` and `scale` values are considered scalar values, and are multiplied by `8`

+\[2\] If the GitHub `marker-size` option is specified, then the following values are used for the radius.

+Clusters are also rendered using the bubble layer. By default the radius of a cluster is set to `16`. The color of the cluster varies depending on the number of points in the cluster, as defined in the following table:

+If a feature is a `Point` or a `MultiPoint`, and the feature and has an `image` property that would be used as a custom icon to render the point as a symbol, then the feature is rendered with a `SymbolLayer`.

+| `image` | `image` | ``none`` |

+| `size` | `size`, `marker-size`¹ | `1` |

+| `rotation` | `rotation` | `0` |

+| `offset` | `offset` | `[0, 0]` |

+| `anchor` | `anchor` | `'bottom'` |

+\[1\] If the GitHub `marker-size` option is specified, then the following values are used for the icon size option.

+If the point feature is a cluster, the `point_count_abbreviated` property is rendered as a text label. No image is rendered.

+If the feature is a `LineString`, `MultiLineString`, `Polygon`, or `MultiPolygon`, then the feature is rendered with a `LineLayer`.

+| Layer option | Supported property name(s) | Default value |

+|--|-||

+| `strokeColor` | `strokeColor`, `stroke` | `'#1E90FF'` |

+| `strokeWidth` | `strokeWidth`, `stroke-width`, `stroke-thickness` | `3` |

+| `strokeOpacity` | `strokeOpacity`, `stroke-opacity` | `1` |

+If the feature is a `Polygon` or a `MultiPolygon`, and the feature either doesn't have a `height` property or the `height` property is zero, then the feature is rendered with a `PolygonLayer`.

+| `fillColor` | `fillColor`, `fill` | `'#1E90FF'` |

+| `fillOpacity`|`fillOpacity`, '`fill-opacity`| `0.5` |

+If the feature is a `Polygon` or a `MultiPolygon`, and has a `height` property with a value greater than zero, the feature is rendered with an `PolygonExtrusionLayer`.

+| `base` | `base` | `0` |

+| `fillColor` | `fillColor`, `fill` | `'#1E90FF'` |

+| `height` | `height` | `0` |

+> [SimpleDataLayer]

+> [SimpleDataLayerOptions]

+> [Read and write spatial data]

+> [Add an OGC map layer]

+> [Connect to a WFS service]

+> [Leverage core operations]

+> [Supported data format details]

+[Add an OGC map layer]: spatial-io-add-ogc-map-layer.md

+[Connect to a WFS service]: spatial-io-connect-wfs-service.md

+[GitHub's GeoJSON map support]: https://docs.github.com/en/repositories/working-with-files/using-files/working-with-non-code-files#mapping-geojsontopojson-files-on-github

+[Leverage core operations]: spatial-io-core-operations.md

+[popup template]: map-add-popup.md#add-popup-templates-to-the-map

+[Read and write spatial data]: spatial-io-read-write-spatial-data.md

+[Simple data layer options.html]: https://github.com/Azure-Samples/AzureMapsCodeSamples/blob/main/Samples/Spatial%20IO%20Module/Simple%20data%20layer%20options/Simple%20data%20layer%20options.html

+[SimpleDataLayer]: /javascript/api/azure-maps-spatial-io/atlas.layer.simpledatalayer

+[SimpleDataLayerOptions]: /javascript/api/azure-maps-spatial-io/atlas.simpledatalayeroptions

+[Supported data format details]: spatial-io-supported-data-format-details.md

+A Web Feature Service (WFS) is a web service for querying spatial data that has a standardized API defined by the Open Geospatial Consortium (OGC). The `WfsClient` class in the spatial IO module lets developers connect to a WFS service and query data from the service.

+The `WfsClient` class supports the following features:

+The specification for the WFS standard makes use of OGC filters. The WFS client supports the following filters, assuming that the service being called also supports these filters. Custom filter strings can be passed into the `CustomFilter` class.

+To access WFS services hosted on non-CORS enabled endpoints, a CORS enabled proxy service can be passed into the `proxyService` option of the WFS client as shown in the following example.

+> [WfsClient]

+> [WfsServiceOptions]

+> [Leverage core operations]

+> [Supported data format details]

+[Leverage core operations]: spatial-io-core-operations.md

+[Simple WFS example]: https://samples.azuremaps.com/spatial-io-module/simple-wfs-example

+[Supported data format details]: spatial-io-supported-data-format-details.md

+[WFS filter example]: https://samples.azuremaps.com/spatial-io-module/wfs-filter-examples

+[WFS service explorer]: https://samples.azuremaps.com/spatial-io-module/wfs-service-explorer

+[WfsClient]: /JavaScript/api/azure-maps-spatial-io/atlas.io.ogc.wfsclient

+[WfsServiceOptions]: /JavaScript/api/azure-maps-spatial-io/atlas.wfsserviceoptions

+ Title: Core IO operations

+The `atlas.io.core` namespace contains two low-level classes that can quickly read and write CSV and XML data. These base classes power the spatial data readers and writers in the Spatial IO module. Feel free to use them to add more reading and writing support for CSV or XML files.

+- The `read` function reads the full data set and return a two-dimensional array of strings representing all cells of the delimited data set.

+By default, the reader uses the comma character as the delimiter. However, the delimiter can be changed to any single character or set to `'auto'`. When set to `'auto'`, the reader analyzes the first line of text in the string. Then, it selects the most common character from the following table to use as the delimiter.

+Follow the steps to use this class:

+- Call the `toString` function to retrieve the delimited string.

+The `atlas.io.core.SimpleXmlReader` class is faster at parsing XML files than `DOMParser`. However, the `atlas.io.core.SimpleXmlReader` class requires XML files to be well formatted. XML files that aren't well formatted, for example missing closing tags, may result in an error.

+The following table lists the spatial file formats that are supported for reading and writing operations with the Spatial IO module.

+When reading a compressed file, either as a zip or a KMZ, it's unzipped and scanned for the first valid file. For example, doc.kml, or a file with other valid extension, such as: .kml, .xml, geojson, .json, .csv, .tsv, or .txt. Then, images referenced in KML and GeoRSS files are preloaded to ensure they're accessible. Inaccessible image data may load an alternative fallback image or removed from the styles. Images extracted from KMZ files are converted to data URIs.

+The [Load spatial data] sample shows how to read a spatial data set, and renders it on the map using the `SimpleDataLayer` class. The code uses a GPX file pointed to by a URL. For the source code of this sample, see [Load spatial data source code].

+The next code demo shows how to read and load KML, or KMZ, to the map. KML can contain ground overlays, which is in the form of an `ImageLyaer` or `OgcMapLayer`. These overlays must be added on the map separately from the features. Additionally, if the data set has custom icons, those icons need to be loaded to the maps resources before the features are loaded.

+You may optionally provide a proxy service for accessing cross domain assets that may not have CORS enabled. The read function tries to access files on another domain using CORS first. After the first time it fails to access any resource on another domain using CORS it only requests more files if a proxy service has been provided. The read function appends the file URL to the end of the proxy URL provided. This snippet of code shows how to pass a proxy service into the read function:

+The following code snippet shows how to read a delimited file and render it on the map. In this case, the code uses a CSV file that has spatial data columns. You must add a reference to the Azure Maps Spatial IO module.

+There are two main write functions in the spatial IO module. The `atlas.io.write` function generates a string, while the `atlas.io.writeCompressed` function generates a compressed zip file. The compressed zip file would contain a text-based file with the spatial data in it. Both of these functions return a promise to add the data to the file. And, they both can write any of the following data: `SpatialDataSet`, `DataSource`, `ImageLayer`, `OgcMapLayer`, feature collection, feature, geometry, or an array of any combination of these data types. When writing using either functions, you can specify the wanted file format. If the file format isn't specified, then the data is written as KML.

+The [Spatial data write options] sample is a tool that demonstrates most the write options that can be used with the `atlas.io.write` function. For the source code of this sample, see [Spatial data write options source code].

+[Well-Known Text] (WKT) is an Open Geospatial Consortium (OGC) standard for representing spatial geometries as text. Many geospatial systems support WKT, such as Azure SQL and Azure PostgreSQL using the PostGIS plugin. Like most OGC standards, coordinates are formatted as "longitude latitude" to align with the "x y" convention. As an example, a point at longitude -110 and latitude 45 can be written as `POINT(-110 45)` using the WKT format.

+- The `isAxisOrderLonLat` option - The axis order of coordinates "latitude, longitude" or "longitude, latitude" can vary between data sets, and it isn't always well defined. By default the GML reader reads the coordinate data as "latitude, longitude", but setting this option to `true` reads it as "longitude, latitude".

+- The `propertyTypes` option - This option is a key value lookup table where the key is the name of a property in the data set. The value is the object type to cast the value to when parsing. The supported type values are: `string`, `number`, `boolean`, and `date`. If a property isn't in the lookup table or the type isn't defined, the property is parsed as a string.

+The `atlas.io.read` function defaults to the `atlas.io.core.GmlReader.read` function when it detects that the input data is XML, but the data isn't one of the other support spatial XML formats.

+The `GmlReader` parses coordinates that have one of the following SRIDs:

+[Drag and drop spatial files onto map]: https://samples.azuremaps.com/spatial-io-module/drag-and-drop-spatial-files-onto-map

+[Load KML onto map source code]: https://github.com/Azure-Samples/AzureMapsCodeSamples/blob/main/Samples/Spatial%20IO%20Module/Load%20KML%20onto%20map/Load%20KML%20onto%20map.html

+[Load KML onto map]: https://samples.azuremaps.com/spatial-io-module/load-kml-onto-map

+[Load spatial data source code]: https://github.com/Azure-Samples/AzureMapsCodeSamples/blob/main/Samples/Spatial%20IO%20Module/Load%20spatial%20data%20(simple)/Load%20spatial%20data%20(simple).html

+[Load spatial data]: https://samples.azuremaps.com/spatial-io-module/load-spatial-data-(simple)

+[Read and write Well Known Text]: https://samples.azuremaps.com/spatial-io-module/read-and-write-well-known-text

+[Read Well Known Text source code]: https://github.com/Azure-Samples/AzureMapsCodeSamples/blob/main/Samples/Spatial%20IO%20Module/Read%20Well%20Known%20Text/Read%20Well%20Known%20Text.html

+[Read Well Known Text]: https://samples.azuremaps.com/spatial-io-module/read-well-known-text

+[Spatial data write options source code]: https://github.com/Azure-Samples/AzureMapsCodeSamples/blob/main/Samples/Spatial%20IO%20Module/Spatial%20data%20write%20options/Spatial%20data%20write%20options.html

+[Spatial data write options]: https://samples.azuremaps.com/spatial-io-module/spatial-data-write-options

+[Well-Known Text]: https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry

+The spatial IO module supports the following XML elements. Any XML tags that aren't supported are converted into a JSON object. Then, each tag is added as a property in the `properties` field of the parent shape or layer.

+| `extrude` | partial | partial | Only supported for polygons. MultiGeometry that have polygons of different heights are broken out into individual features. Line styles aren't supported. Polygons with an altitude of 0 is rendered as a flat polygon. When reading, the altitude of the first coordinate in the exterior ring is added as a height property of the polygon. Then, the altitude of the first coordinate is used to render the polygon on the map. |

+| `IconStyle` | partial | partial | `icon`, `heading`, `colorMode`, and `hotspots` values are parsed, but not rendered by `SimpleDataLayer` |

+| `viewRefreshMode` | partial | no | If pointing to a WMS service, then only `onStop` is supported for ground overlays. Appends `BBOX={bboxWest},{bboxSouth},{bboxEast},{bboxNorth}` to the URL and update as the map moves. |

+| `position` | yes | no | Some XML feeds wrap GML with a position tag instead of wrapping it with a `georss:where` tag. Read this tag, but writes using a `georss:where` tag. |

+#### More notes

+- Member elements are searched for a geometry that may be buried within child elements. This search operation is necessary as many XML formats that extend from GML may not place a geometry as a direct child of a member element.

+- `srsName` is partially supported for WGS84 coordinates and the following codes:[EPSG:4326]), and web Mercator ([EPSG:3857] or one of its alternative codes. Any other coordinate system is parsed as WGS84 as-is.

+- Unless a custom GML namespace is specified for the properties when writing to a GML file, other property information isn't added.

+| `gpx:extensions` | partial | partial | When read, style information is extracted. All other extensions are flattened into a simple JSON object. Only shape style information is written. |

+| `gpxx:DisplayColor` | yes | no | Used to specify the color of a shape. If writing, `gpx_style:line` color is used instead.|

+#### More notes

+- MultiPoints is broken up into individual waypoints.

+- Polygons and MultiPolygons are written as tracks.

+When reading a delimited file that contains spatial data, the header is analyzed to determine which columns contain location fields. If the header contains type information, it's used to cast the cell values to the appropriate type. If no header is specified, the first row is analyzed to generate a header. When analyzing the first row, a check is executed to match column names with the following names in a case-insensitive way. The order of the names is the priority, in case two or more names exist in a file.

+The first row of data is scanned for strings that are in Well-Known Text format.

+When scanning the header row, any type information that is in the column name is extracted and used to cast the cells in that column. Here's an example of a column name that has a type value: "ColumnName (typeName)". The following case-insensitive type names are supported:

+If no type information can be extracted from the header, and the dynamic typing option is enabled when reading, then each cell is individually analyzed to determine what data type it's best suited to be cast as.

+[Read and write spatial data]

+[EPSG:4326]: https://epsg.io/4326

+[EPSG:3857]: https://epsg.io/3857

+[Read and write spatial data]: spatial-io-read-write-spatial-data.md

+| June 2023| **Windows** <ul><li>Add new file path column to custom logs table</li><li>Config setting to disable custom IMDS endpoint in Tenant.json file</li><li>FluentBit binaries signed with Microsoft customer Code Sign cert</li><li>Minimize number of retries on calls to refresh tokens</li><li>Don't overwrite resource ID with empty string</li><li>AzSecPack updated to version 4.27</li><li>AzureProfiler and AzurePerfCollector updated to version 1.0.0.990</li><li>MetricsExtension updated to version 2.2023.513.10</li><li>Troubleshooter updated to version 1.5.0</li></ul>**Linux** <ul><li>Add new column CollectorHostName to syslog table to identify forwarder/collector machine</li><li>Link OpenSSL dynamically</li><li>**Fixes**<ul><li>Allow uploads soon after AMA start up</li><li>Run LocalSink GC on a dedicated thread to avoid thread pool scheduling issues</li><li>Fix upgrade restart of disabled services</li><li>Handle Linux Hardening where sudo on root is blocked</li><li>CEF processing fixes for noncomliant RFC 5424 logs</li><li>ASA tenant can fail to start up due to config-cache directory permissions</li><li>Fix auth proxy in AMA</li><li>Fix to remove null characters in agentlauncher.log after log rotation</li></ul></li></ul>|1.17.0 |1.27.2|

+> [!NOTE]

+> As of September 2022, we are starting the 3-year process of deprecating support for using ITSM actions to send alerts and events to ServiceNow.

+ > As of September 2022, we are starting the 3-year process of deprecating support for using ITSM actions to send alerts and events to ServiceNow.

+ > [!NOTE]

+ > As of September 2022, we are starting the 3-year process of deprecating support for using ITSM actions to send alerts and events to ServiceNow. For information about legal terms and the privacy policy, see the [Microsoft privacy statement](https://go.microsoft.com/fwLink/?LinkID=522330&clcid=0x9).

+|ApplicationInsightsAgent_EXTENSION_VERSION | Main extension, which controls runtime monitoring. | `~3` |

+kubectl port-forward <ama-metrics pod name> -n kube-system 9090

+Go to `127.0.0.1:9090/metrics` in a browser to see if the metrics were scraped by the OpenTelemetry Collector. This user interface can be accessed for every `ama-metrics-*` pod. If metrics aren't there, there could be an issue with the metric or label name lengths or the number of labels. Also check for exceeding the ingestion quota for Prometheus metrics as specified in this article.

+## Microsoft.ServiceNetworking

+> [!div class="mx-tableFixed"]

+> | Resource type | Resource group | Subscription | Region move |

+> | - | -- | - | -- |

+> | trafficcontrollers | No | No | No |

+> | associations | No | No | No |

+> | frontends | No | No | No |

+User subscription mode requires [Azure Key Vault](/azure/key-vault/general/overview). The key vault must be in the same subscription and region as the Batch account and use a [Vault Access Policy](/azure/key-vault/general/assign-access-policy).

+1. On the **Access configuration** tab, select **Vault access policy** under **Permission model**.

+ Map.of("rule1", new ExceptionRule(

+ new QueueLengthExceptionTrigger().setThreshold(1)

+ Map.of("cancelAction", new CancelExceptionAction())))

+ "rule1", new ExceptionRule(

+ new WaitTimeExceptionTrigger(60),

+ Map.of("increasePriority", new ManualReclassifyExceptionAction().setPriority(10))),

+ "rule2", new ExceptionRule(

+ new WaitTimeExceptionTrigger(300),

+ Map.of("changeQueue", new ManualReclassifyExceptionAction().setQueueId("queue2"))))

+ "queue2": RouterQueueAssignment()

+ "queue1", new RouterQueueAssignment(),

+ "queue2", new RouterQueueAssignment()))

+ "voice", new ChannelConfiguration(2),

+ "chat", new ChannelConfiguration(1)))

+ new RouterWorkerSelector("English", LabelOperator.EQUAL, new LabelValue(true)),

+ new RouterWorkerSelector("Skill", LabelOperator.GREATER_THAN, new LabelValue(10))))

+ "voice", new ChannelConfiguration(100),

+ "chat", new ChannelConfiguration(20))));

+ "voice", new ChannelConfiguration(60),

+ "chat", new ChannelConfiguration(10).setMaxNumberOfJobs(2),

+ "email", new ChannelConfiguration(10).setMaxNumberOfJobs(2))));

+ queueSelectors: new List<RouterQueueSelector>

+ queueSelectors: [{

+ queue_selectors = [

+ .setQueueSelectors(List.of(new ConditionalQueueSelectorAttachment(

+ new ExpressionRouterRule("job.Escalated = true"),

+ List.of(new RouterQueueSelector("Id", LabelOperator.EQUAL, new LabelValue("XBOX_Escalation_Queue"))))))

+ .setPrioritizationRule(new ExpressionRouterRule("If(job.Escalated = true, 10, 1)")));

+ Map.of("Escalated_Rule", new ExceptionRule(new WaitTimeExceptionTrigger(5 * 60),

+ Map.of("EscalateReclassifyExceptionAction", new ReclassifyExceptionAction()

+ RequestedWorkerSelectors =

+ new RouterWorkerSelector(key: "XBOX_Hardware", labelOperator: LabelOperator.GreaterThanEqual, value: new LabelValue(7))

+ .setRequestedWorkerSelectors(List.of(

+ new RouterWorkerSelector("XBOX_Hardware", LabelOperator.GREATER_THAN_EQUAL, new LabelValue(7)))));

+ .setQueueSelectors(List.of(new ConditionalQueueSelectorAttachment(

+ new ExpressionRouterRule("job.Region = \"NA\""),

+ List.of(new RouterQueueSelector("Id", LabelOperator.EQUAL, new LabelValue("XBOX_NA_QUEUE"))))))

+ .setWorkerSelectors(List.of(

+ new StaticWorkerSelectorAttachment(new RouterWorkerSelector("Foo", LabelOperator.EQUAL, new LabelValue("Bar"))))));

+ .setWorkerSelectors(List.of(new ConditionalRouterWorkerSelectorAttachment(

+ new ExpressionRouterRule("job.Urgent = true"),

+ List.of(new RouterWorkerSelector("Foo", LabelOperator.EQUAL, new LabelValue("Bar")))))));

+ .setWorkerSelectors(List.of(new PassThroughWorkerSelectorAttachment("Foo", LabelOperator.EQUAL))));

+ .setWorkerSelectors(List.of(new WeightedAllocationWorkerSelectorAttachment(

+ List.of(new WorkerWeightedAllocation(0.3, List.of(

+ new RouterWorkerSelector("Vendor", LabelOperator.EQUAL, new LabelValue("A")),

+ new RouterWorkerSelector("Vendor", LabelOperator.EQUAL, new LabelValue("B"))

+ .setRequestedWorkerSelectors(List.of(

+ new RouterWorkerSelector("Id", LabelOperator.EQUAL, new LabelValue("<preferred_worker_id>"))

+ .setExpireAfterSeconds(45.0)

+ .setExpedite(true))));

+ ```

+ MatchingMode = new JobMatchingMode(

+ new ScheduleAndSuspendMode(scheduleAt: DateTimeOffset.UtcNow.Add(TimeSpan.FromMinutes(3))))

+ .setMatchingMode(new JobMatchingMode(new ScheduleAndSuspendMode(OffsetDateTime.now().plusMinutes(3)))));

+ MatchingMode = new JobMatchingMode(new QueueAndMatchMode()),

+ matchingMode: { queueAndMatchMode: {} },

+ matching_mode = JobMatchingMode(queue_and_match_mode = {}),

+ .setMatchingMode(new JobMatchingMode(new QueueAndMatchMode()))

+* In Standard workflows, to use the SQL built-in triggers, you must enable change tracking in the table where you want to use the trigger. For more information, see [Enable and disable change tracking](/sql/relational-databases/track-changes/enable-and-disable-change-tracking-sql-server).

+| [Deploy Drupal on Azure Container Apps](https://github.com/Azure-Samples/drupal-on-azure-container-apps) | Demonstrates how to deploy a Drupal site to Azure Container Apps, with Azure Database for MariaDB, and Azure Files to store static assets. |

+16-MB document support raises the size limit for your documents from 2 MB to 16 MB. This limit only applies to collections created after this feature has been enabled. Once this feature is enabled for your database account, it can't be disabled.

+16-MB document support raises the size limit for your documents from 2 MB to 16 MB. This limit applies only to collections that are created after this feature is enabled. When this feature is enabled for your database account, it can't be disabled.

+| `EnableUniqueIndexReIndex` | Enables support for unique index re-indexing for Cosmos DB for MongoDB RU. | No |

+> [!NOTE]

+> Support for unique index on existing collections with data is available in preview. This feature can be enabled for your database account by enabling the ['EnableUniqueIndexReIndex' capability](./how-to-configure-capabilities.md#available-capabilities).

+- If the input value isn't a valid object, the result is `undefined`.

+The following example returns the constant value of Pi.

+- This function doesn't use the index.

+- [`SQRT`](sqrt.md)

+ Title: RAND

+description: An Azure Cosmos DB for NoSQL system function that returns a randomly generated numeric value from zero to one.

+# RAND (NoSQL query)

+Returns a randomly generated numeric value from zero to one.

+RAND()

+```

+Returns a numeric expression.

+## Examples

+The following example returns randomly generated numeric values.

+## Remarks

+- This function doesn't use the index.

+- This function is nondeterministic. Repetitive calls of this function don't return the same results.

+- [`IS_NUMBER`](is-number.md)

+ Title: RegexMatch

+description: An Azure Cosmos DB for NoSQL system function that provides regular expression capabilities.

+# RegexMatch (NoSQL query)

+This function provides regular expression capabilities. Regular expressions are a concise and flexible notation for finding patterns of text.

+> [!NOTE]

+> Azure Cosmos DB for NoSQL uses [PERL compatible regular expressions (PCRE)](https://pcre2project.github.io/pcre2/).

+RegexMatch(<string_expr_1>, <string_expr_2>, [, <string_expr_3>])

+| | Description |

+| | |

+| **`string_expr_1`** | A string expression to be searched. |

+| **`string_expr_2`** | A string expression with a regular expression defined to use when searching `string_expr_1`. |

+| **`string_expr_3`** *(Optional)* | An optional string expression with the selected modifiers to use with the regular expression (`string_expr_2`). If not provided, the default is to run the regular expression match with no modifiers. |

+> [!NOTE]

+> Providing an empty string for `string_expr_3` is functionally equivalent to omitting the argument.

+## Return types

+Returns a boolean expression.

+The following example illustrates regular expression matches using a few different modifiers.

+The next example assumes that you have a container with items including a `name` field.

+This example uses a regular expression match as a filter to return a subset of items.

+- This function benefits from a [range index](../../index-policy.md#includeexclude-strategy) only if the regular expression can be broken down into either `StartsWith`, `EndsWith`, `Contains`, or `StringEquals` equivalent system functions.

+- Returns `undefined` if the string expression to be searched (`string_expr_1`), the regular expression (`string_expr_2`), or the selected modifiers (`string_expr_3`) are invalid.

+- This function supports the following four modifiers:

+ | | Format | Description |

+ | | | |

+ | **Multiple lines** | `m` | Treat the string expression to be searched as multiple lines. Without this option, the characters `^` and `$` match at the beginning or end of the string and not each individual line. |

+ | **Match any string** | `s` | Allow "." to match any character, including a newline character. |

+ | **Ignore case** | `i` | Ignore case when pattern matching. |

+ | **Ignore whitespace** | `x` | Ignore all whitespace characters. |

+- If you'd like to use a meta-character in a regular expression and don't want it to have special meaning, you should escape the metacharacter using `\`.

+- [`IS_STRING`](is-string.md)

+ Title: REPLACE

+description: An Azure Cosmos DB for NoSQL system function that returns a string with all occurrences of a specified string replaced.

+# REPLACE (NoSQL query)

+Replaces all occurrences of a specified string value with another string value.

+REPLACE(<string_expr_1>, <string_expr_2>, <string_expr_3>)

+```

+| | Description |

+| | |

+| **`string_expr_1`** | A string expression to be searched. |

+| **`string_expr_2`** | A string expression to be found within `string_expr_1`. |

+| **`string_expr_3`** | A string expression with the text to replace all occurrences of `string_expr_2` within `string_expr_1`. |

+Returns a string expression.

+The following example shows how to use this function to replace static values.

+- This function doesn't use the index.

+- [`SUBSTRING`](substring.md)

+ Title: REPLICATE

+description: An Azure Cosmos DB for NoSQL system function that returns a string value repeated a specific number of times.

+# REPLICATE (NoSQL query)

+Repeats a string value a specified number of times.

+REPLICATE(<string_expr>, <numeric_expr>)

+| | Description |

+| | |

+| **`string_expr`** | A string expression. |

+| **`numeric_expr`** | A numeric expression. |

+Returns a string expression.

+The following example shows how to use this function to build a repeating string.

+## Remarks

+- This function doesn't use the index.

+- The maximum length of the result is **10,000** characters.

+ - `(length(string_expr) * numeric_expr) <= 10,000`

+- If `numeric_expr` is *negative* or *nonfinite*, the result is `undefined`.

+- [`REPLACE`](replace.md)

+ Title: REVERSE

+description: An Azure Cosmos DB for NoSQL system function that returns a reversed string.

+# REVERSE (NoSQL query)

+Returns the reverse order of a string value.

+REVERSE(<string_expr>)

+```

+| | Description |

+| | |

+| **`string_expr`** | A string expression. |

+Returns a string expression.

+The following example shows how to use this function to reverse multiple strings.

+- This function doesn't use the index.

+- [`LENGTH`](length.md)

+ Title: ROUND

+description: An Azure Cosmos DB for NoSQL system function that returns the number rounded to the closest integer.

+# ROUND (NoSQL query)

+Returns a numeric value, rounded to the closest integer value.

+```

+| | Description |

+| | |

+| **`numeric_expr`** | A numeric expression. |

+Returns a numeric expression.

+The following example rounds positive and negative numbers to the nearest integer.

+## Remarks

+- This function benefits from a [range index](../../index-policy.md#includeexclude-strategy).

+- The rounding operation performed follows midpoint rounding away from zero. If the input is a numeric expression, which falls exactly between two integers then the result is the closest integer value away from `0`. Examples are provided here:

+ | | Rounded |

+ | | |

+ | **`-6.5000`** | `-7` |

+ | **`-0.5`** | `-1` |

+ | **`0.5`** | `1` |

+ | **`6.5000`** | `7` |

+- [`POWER`](power.md)

+```

+- This function doesn't use the index.

+This last example uses a single item that share values within two array properties.

+The query selects the appropriate field from the item\[s\] in the container.

+- This function doesn't use the index.

+This last example uses an item that share values within multiple array properties.

+The query returns the union of the two arrays as a new property.

+- This function doesn't use the index.

+ Title: SIGN

+description: An Azure Cosmos DB for NoSQL system function that returns the sign of the specified number.

+# SIGN (NoSQL query)

+Returns the positive (+1), zero (0), or negative (-1) sign of the specified numeric expression.

+```

+| | Description |

+| | |

+| **`numeric_expr`** | A numeric expression. |

+Returns a numeric expression.

+The following example returns the sign of various numbers from -2 to 2.

+- This function doesn't use the index.

+- [`ABS`](abs.md)

+```

+- This function doesn't use the index.

+- This function doesn't use the index.

+- [`POWER`](power.md)

+- This function doesn't use the index.

+- [`SQRT`](sqrt.md)

+- [`POWER`](power.md)

+ Title: ST_AREA

+description: An Azure Cosmos DB for NoSQL system function that returns the total area of a GeoJSON polygon or multi-polygon.

+# ST_AREA (NoSQL query)

+Returns the total area of a GeoJSON **Polygon** or **MultiPolygon** expression.

+> [!NOTE]

+> For more information, see [Geospatial and GeoJSON location data](geospatial-intro.md).

+ST_AREA(<spatial_expr>)

+| | Description |

+| | |

+| **`spatial_expr`** | Any valid GeoJSON **Polygon** or **MultiPolygon** expression. |

+Returns a numeric expression that enumerates the total area of a set of points.

+The following example shows how to return the area of a polygon.

+## Remarks

+- The result is expressed in square meters for the default reference system.

+- Using this function to calculate the area of zero or one-dimensional figures like GeoJSON **Points** and **LineStrings** results in an area of `0`.

+- The GeoJSON specification requires that points within a Polygon be specified in counter-clockwise order. A Polygon specified in clockwise order represents the inverse of the region within it.

+- [`ST_WITHIN`](st-within.md)

+ Title: ST_DISTANCE

+description: An Azure Cosmos DB for NoSQL system function that returns the distance between two GeoJSON Point, Polygon, MultiPolygon or LineStrings.

+# ST_DISTANCE (NoSQL query)

+Returns the distance between two GeoJSON Point, Polygon, MultiPolygon or LineString expressions.

+> [!NOTE]

+> For more information, see [Geospatial and GeoJSON location data](geospatial-intro.md).

+ST_DISTANCE(<spatial_expr_1>, <spatial_expr_2>)

+```

+| | Description |

+| | |

+| **`spatial_expr_1`** | Any valid GeoJSON **Point**, **Polygon**, **MultiPolygon** or **LineString** expression. |

+| **`spatial_expr_2`** | Any valid GeoJSON **Point**, **Polygon**, **MultiPolygon** or **LineString** expression. |

+Returns a numeric expression that enumerates the distance between two expressions.

+The following example assumes a container exists with two items.

+The example shows how to use the function as a filter to return items within a specified distance.

+## Remarks

+- The result is expressed in meters for the default reference system.

+- This function benefits from a [geospatial index](../../index-policy.md#spatial-indexes) except in queries with aggregates.

+- The GeoJSON specification requires that points within a Polygon be specified in counter-clockwise order. A Polygon specified in clockwise order represents the inverse of the region within it.

+- [`ST_INTERSECTS`](st-intersects.md)

+ Title: ST_INTERSECTS

+description: An Azure Cosmos DB for NoSQL system function that returns whether two GeoJSON objects intersect.

+# ST_INTERSECTS (NoSQL query)

+Returns a boolean indicating whether the GeoJSON object (**Point**, **Polygon**, **MultiPolygon**, or **LineString**) specified in the first argument intersects the GeoJSON object in the second argument.

+ST_INTERSECTS(<spatial_expr_1>, <spatial_expr_2>)

+| | Description |

+| | |

+| **`spatial_expr_1`** | Any valid GeoJSON **Point**, **Polygon**, **MultiPolygon** or **LineString** expression. |

+| **`spatial_expr_2`** | Any valid GeoJSON **Point**, **Polygon**, **MultiPolygon** or **LineString** expression. |

+Returns a boolean value.

+The following example shows how to find if two polygons intersect.

+## Remarks

+- This function benefits from a [geospatial index](../../index-policy.md#spatial-indexes) except in queries with aggregates.

+- The GeoJSON specification requires that points within a Polygon be specified in counter-clockwise order. A Polygon specified in clockwise order represents the inverse of the region within it.

+- [`ST_WITHIN`](st-within.md)

+ Title: ST_ISVALID

+description: An Azure Cosmos DB for NoSQL system function that returns if a GeoJSON object is valid.

+# ST_ISVALID (NoSQL query)

+Returns a boolean value indicating whether the specified GeoJSON **Point**, **Polygon**, **MultiPolygon**, or **LineString** expression is valid.

+```

+| | Description |

+| | |

+| **`spatial_expr`** | Any valid GeoJSON **Point**, **Polygon**, **MultiPolygon**, or **LineString** expression. |

+Returns a boolean value.

+The following example how to check validity of multiple objects.

+## Remarks

+- The GeoJSON specification requires that points within a Polygon be specified in counter-clockwise order. A Polygon specified in clockwise order represents the inverse of the region within it.

+- [`ST_ISVALIDDETAILED`](st-isvaliddetailed.md)

+ Title: ST_ISVALIDDETAILED

+description: An Azure Cosmos DB for NoSQL system function that returns if a GeoJSON object is valid along with the reason.

+# ST_ISVALIDDETAILED (NoSQL query)

+Returns a JSON value containing a Boolean value if the specified GeoJSON **Point**, **Polygon**, or **LineString** expression is valid, and if invalid, the reason.

+```

+| | Description |

+| | |

+| **`spatial_expr`** | Any valid GeoJSON **Point**, **Polygon**, or **LineString** expression. |

+Returns a JSON object containing a boolean value indicating if the specified GeoJSON point or polygon expression is valid. If invalid, the object additionally contains the reason as a string value.

+The following example how to check validity of multiple objects.

+## Remarks

+- The GeoJSON specification requires that points within a Polygon be specified in counter-clockwise order. A Polygon specified in clockwise order represents the inverse of the region within it.

+- [`ST_ISVALID`](st-isvalid.md)

+ Title: ST_WITHIN

+description: An Azure Cosmos DB for NoSQL system function that returns if one GeoJSON object is within another.

+# ST_WITHIN (NoSQL query)

+Returns a boolean expression indicating whether the GeoJSON object (GeoJSON **Point**, **Polygon**, or **LineString** expression) specified in the first argument is within the GeoJSON object in the second argument.

+ST_WITHIN(<spatial_expr_1>, <spatial_expr_2>)

+```

+| | Description |

+| | |

+| **`spatial_expr_1`** | Any valid GeoJSON **Point**, **Polygon**, **MultiPolygon** or **LineString** expression. |

+| **`spatial_expr_2`** | Any valid GeoJSON **Point**, **Polygon**, **MultiPolygon** or **LineString** expression. |

+Returns a boolean value.

+The following example shows how to find if a **Point** is within a **Polygon**.

+## Remarks

+- This function benefits from a [geospatial index](../../index-policy.md#spatial-indexes) except in queries with aggregates.

+- The GeoJSON specification requires that points within a Polygon be specified in counter-clockwise order. A Polygon specified in clockwise order represents the inverse of the region within it.

+- [`ST_INTERSECT`](st-intersects.md)

+STARTSWITH(<string_expr_1>, <string_expr_2> [, <bool_expr>])

+| **`string_expr_1`** | A string expression. |

+| **`string_expr_2`** | A string expression to be compared to the beginning of `string_expr_1`. |

+| **`bool_expr`** *(Optional)* | Optional value for ignoring case. When set to `true`, `STARTSWITH` does a case-insensitive search. When unspecified, this default value is `false`. |

+ Title: StringEquals

+description: An Azure Cosmos DB for NoSQL system function that returns a boolean indicating whether two strings are equivalent.

+# StringEquals (NoSQL query)

+Returns a boolean indicating whether the first string expression matches the second.

+STRINGEQUALS(<string_expr_1>, <string_expr_2> [, <boolean_expr>])

+```

+| | Description |

+| | |

+| **`string_expr_1`** | The first string expression to compare. |

+| **`string_expr_2`** | The second string expression to compare. |

+| **`boolean_expr` *(Optional)*** | An optional boolean expression for ignoring case. When set to `true`, this function performs a case-insensitive search. If not specified, the default value is `false`. |

+Returns a boolean expression.

+The following example checks if "abc" matches "abc" and if "abc" matches "ABC."

+- [`SUBSTRING`](substring.md)

+ Title: StringToArray

+description: An Azure Cosmos DB for NoSQL system function that returns a string expression converted to an array.

+# StringToArray (NoSQL query)

+Converts a string expression to an array.

+StringToArray(<string_expr>)

+## Arguments

+| | Description |

+| | |

+| **`string_expr`** | A string expression. |

+## Return types

+Returns an array.

+## Examples

+

+The following example illustrates how this function works with various inputs.

+## Remarks

+- This function doesn't use the index.

+- If the expression can't be converted, the function returns `undefined`.

+- Nested string values must be written with double quotes to be valid.

+- Single quotes within the array aren't valid JSON. Even though single quotes are valid within a query, they don't parse to valid arrays. Strings within the array string must either be escaped `\"` or the surrounding quote must be a single quote.

+> [!NOTE]

+> For more information on the JSON format, see [https://json.org](https://json.org/).

+- [`StringToObject`](stringtoobject.md)

+ Title: StringToBoolean

+description: An Azure Cosmos DB for NoSQL system function that returns a string expression converted to a boolean.

+# StringToBoolean (NoSQL query)

+Converts a string expression to a boolean.

+StringToBoolean(<string_expr>)

+## Arguments

+| | Description |

+| | |

+| **`string_expr`** | A string expression. |

+## Return types

+Returns a boolean value.

+## Examples

+

+The following example illustrates how this function works with various data types.

+- This function doesn't use the index.

+- If the expression can't be converted, the function returns `undefined`.

+> [!NOTE]

+> For more information on the JSON format, see [https://json.org](https://json.org/).

+- [`StringToNumber`](stringtonumber.md)

+ Title: StringToNull

+description: An Azure Cosmos DB for NoSQL system function that returns a string expression converted to null.

+# StringToNull (NoSQL query)

+Converts a string expression to `null`.

+```sql

+StringToNull(<string_expr>)

+```

+## Arguments

+| | Description |

+| | |

+| **`string_expr`** | A string expression. |

+## Return types

+Returns a `null`.

+## Examples

+The following example illustrates how this function works with various data types.

+- This function doesn't use the index.

+- If the expression can't be converted, the function returns `undefined`.

+> [!NOTE]

+> For more information on the JSON format, see [https://json.org](https://json.org/).

+- [`StringToBoolean`](stringtoboolean.md)

+ Title: StringToNumber

+description: An Azure Cosmos DB for NoSQL system function that returns a string expression converted to a number.

+# StringToNumber (NoSQL query)

+Converts a string expression to a number.

+StringToNumber(<string_expr>)

+| | Description |

+| | |

+| **`string_expr`** | A string expression. |

+## Return types

+Returns a number value.

+## Examples

+The following example illustrates how this function works with various data types.

+- This function doesn't use the index.

+- String expressions are parsed as a JSON number expression.

+- Numbers in JSON must be an integer or a floating point.

+- If the expression can't be converted, the function returns `undefined`.

+> [!NOTE]

+> For more information on the JSON format, see [https://json.org](https://json.org/).

+- [`StringToBoolean`](stringtoboolean.md)

+ Title: StringToObject

+description: An Azure Cosmos DB for NoSQL system function that returns a string expression converted to an object.

+# StringToObject (NoSQL query)

+Converts a string expression to an object.

+## Syntax

+StringToObject(<string_expr>)

+## Arguments

+| | Description |

+| | |

+| **`string_expr`** | A string expression. |

+## Return types

+Returns an object.

+## Examples

+

+The following example illustrates how this function works with various inputs.

+- This function doesn't use the index.

+- If the expression can't be converted, the function returns `undefined`.

+- Nested string values must be written with double quotes to be valid.

+> [!NOTE]

+> For more information on the JSON format, see [https://json.org](https://json.org/).

+- [`StringToArray`](stringtoarray.md)

+ Title: SUBSTRING

+description: An Azure Cosmos DB for NoSQL system function that returns a portion of a string using a starting position and length.

+# SUBSTRING (NoSQL query)

+Returns part of a string expression starting at the specified position and of the specified length, or to the end of the string.

+SUBSTRING(<string_expr>, <numeric_expr_1>, <numeric_expr_2>)

+```

+| | Description |

+| | |

+| **`string_expr`** | A string expression. |

+| **`numeric_expr_1`** | A numeric expression to denote the start character. |

+| **`numeric_expr_2`** | A numeric expression to denote the maximum number of characters of `string_expr` to be returned. |

+Returns a string expression.

+The following example returns substrings with various lengths and starting positions.

+- This function benefits from a [range index](../../index-policy.md#includeexclude-strategy) if the starting position is `0`.

+- `numeric_expr_1` positions are zero-based, therefore a value of `0` starts from the first character of `string_expr`.

+- A value of `0` or less for `numeric_expr_2` results in empty string.

+- [`StringEquals`](stringequals.md)

+```

+- This function doesn't use the index.

+ Title: TicksToDateTime

+description: An Azure Cosmos DB for NoSQL system function that returns the number of ticks as a date and time value.

+# TicksToDateTime (NoSQL query)

+Converts the specified number of ticks to a date and time value.

+TicksToDateTime(<numeric_expr>)

+| | Description |

+| | |

+| **`numeric_expr`** | A numeric expression. |

+> [!NOTE]

+> For more information on the ISO 8601 format, see [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601).

+Returns a UTC date and time string in the ISO 8601 format `YYYY-MM-DDThh:mm:ss.fffffffZ`.

+## Examples

+The following example converts the ticks to a date and time value.

+## Remarks

+- This function returns `undefined` if the ticks value specified is invalid.

+- [`TimestampToDateTime`](timestamptodatetime.md)

+ Title: TimestampToDateTime

+description: An Azure Cosmos DB for NoSQL system function that returns the timestamp as a date and time value.

+# TimestampToDateTime (NoSQL query)

+Converts the specified timestamp to a date and time value.

+TimestampToDateTime(<numeric_expr>)

+| | Description |

+| | |

+| **`numeric_expr`** | A numeric expression. |

+> [!NOTE]

+> For more information on the ISO 8601 format, see [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601).

+Returns a UTC date and time string in the ISO 8601 format `YYYY-MM-DDThh:mm:ss.fffffffZ`.

+The following example converts the ticks to a date and time value.

+## Remarks

+- This function returns `undefined` if the timestamp value specified is invalid.

+- [`TicksToDateTime`](tickstodatetime.md)

+ Title: ToString

+description: An Azure Cosmos DB for NoSQL system function that returns a value converted to a string.

+# ToString (NoSQL query)

+Returns a string representation of a value.

+```

+| | Description |

+| | |

+| **`expr`** | Any expression. |

+Returns a string expression.

+This example converts multiple scalar and object values to a string.

+- This function doesn't use the index.

+- [`IS_OBJECT`](is-object.md)

+ Title: TRIM

+description: An Azure Cosmos DB for NoSQL system function that returns a string with leading or trailing whitespace trimmed.

+# TRIM (NoSQL query)

+Returns a string expression after it removes leading and trailing whitespace or custom characters.

+TRIM(<string_expr_1> [, <string_expr_2>])

+```

+| | Description |

+| | |

+| **`string_expr_1`** | A string expression. |

+| **`string_expr_2` *(Optional)*** | An optional string expression with a string to trim from `string_expr_1`. If not specified, the default is to trim whitespace. |

+Returns a string expression.

+This example illustrates various ways to trim a string expression.

+- This function doesn't use the index.

+- [`TRUNC`](trunc.md)

+ Title: TRUNC

+description: An Azure Cosmos DB for NoSQL system function that returns a truncated numeric value.

+# TRUNC (NoSQL query)

+Returns a numeric value truncated to the closest integer value.

+TRUNC(<numeric_expr>)

+```

+| | Description |

+| | |

+| **`numeric_expr`** | A numeric expression. |

+Returns a numeric expression.

+This example illustrates various ways to truncate a number to the closest integer.

+- This function benefits from a [range index](../../index-policy.md#includeexclude-strategy).

+- [`TRIM`](trim.md)

+UPPER(<string_expr>)

+```

+- This function doesn't use the index.

+ - If getting started with cost reporting in Power BI, consider using these [Power BI sample reports](https://github.com/flanakin/cost-management-powerbi).

+The **Checkpoint Key** is used by the SAP CDC runtime to store status information about the change data capture process. This, for example, allows SAP CDC mapping data flows to automatically recover from error situations, or know whether a change data capture process for a given data flow has already been established. It is therefore important to use a unique **Checkpoint Key** for each source. Otherwise status information of one source will be overwritten by another source.

+>[!NOTE]

+ > - To avoid conflicts, a unique id is generated as **Checkpoint Key** by default.

+ > - When using parameters to leverage the same data flow for multiple sources, make sure to parametrize the **Checkpoint Key** with unique values per source.

+ > - The **Checkpoint Key** property is not shown if the **Run mode** within the SAP CDC source is set to **Full on every run** (see next section), because in this case no change data capture process is established.

+ Title: SAP CDC advanced topics

+description: Learn about advanced features and best practices for SAP change data capture in Azure Data Factory.

+# SAP CDC advanced topics

+Learn about advanced topics for the SAP CDC connector like metadata driven data integration, debugging, and more.

+## Parametrizing an SAP CDC mapping data flow

+One of the key strengths of pipelines and mapping data flows in Azure Data Factory and Azure Synapse Analytics is the support for metadata driven data integration. With this feature, it's possible to design a single (or few) parametrized pipeline that can be used to handle integration of potentially hundreds or even thousands of sources.

+The SAP CDC connector has been designed with this principle in mind: all relevant properties, whether it's the source object, run mode, key columns, etc., can be provided via parameters to maximize flexibility and reuse potential of SAP CDC mapping data flows.

+To understand the basic concepts of parametrizing mapping data flows, read [Parameterizing mapping data flows](parameters-data-flow.md).

+In the template gallery of Azure Data Factory and Azure Synapse Analytics, you find a [template pipeline and data flow](solution-template-replicate-multiple-objects-sap-cdc.md) which shows how to parametrize SAP CDC data ingestion.

+### Parametrizing source and run mode

+Mapping data flows don't necessarily require a Dataset artifact: both source and sink transformations offer a **Source type** (or **Sink type**) **Inline**. In this case, all source properties otherwise defined in an ADF dataset can be configured in the **Source options** of the source transformation (or **Settings** tab of the sink transformation). Using an inline dataset provides better overview and simplifies parametrizing a mapping data flow since the complete source (or sink) configuration is maintained in a one place.

+For SAP CDC, the properties that are most commonly set via parameters are found in the tabs **Source options** and **Optimize**.

+When **Source type** is **Inline**, the following properties can be parametrized in **Source options**.

+- **ODP context**: valid parameter values are

+ - **ABAP_CDS** for ABAP Core Data Services Views

+ - **BW** for SAP BW or SAP BW/4HANA InfoProviders

+ - **HANA** for SAP HANA Information Views

+ - **SAPI** for SAP DataSources/Extractors

+ - when SAP Landscape Transformation Replication Server (SLT) is used as a source, the ODP context name is SLT~\<Queue Alias\>. The **Queue Alias** value can be found under **Administration Data** in the SLT configuration in the SLT cockpit (SAP transaction LTRC).

+ - **ODP_SELF** and **RANDOM** are ODP contexts used for technical validation and testing, and are typically not relevant.

+- **ODP name**: provide the ODP name you want to extract data from.

+- **Run mode**: valid parameter values are

+ - **fullAndIncrementalLoad** for **Full on the first run, then incremental**, which initiates a change data capture process and extracts a current full data snapshot.

+ - **fullLoad** for **Full on every run**, which extracts a current full data snapshot without initiating a change data capture process.

+ - **incrementalLoad** for **Incremental changes only**, which initiates a change data capture process without extracting a current full snapshot.

+- **Key columns**: key columns are provided as an array of (double-quoted) strings. For example, when working with SAP table **VBAP** (sales order items), the key definition would have to be \["VBELN", "POSNR"\] (or \["MANDT","VBELN","POSNR"\] in case the client field is taken into account as well).

+### Parametrizing the filter conditions for source partitioning

+In the **Optimize** tab, a source partitioning scheme (see [optimizing performance for full or initial loads](connector-sap-change-data-capture.md#optimizing-performance-of-full-or-initial-loads-with-source-partitioning)) can be defined via parameters. Typically, two steps are required:

+1. Define the source partitioning scheme.

+2. Ingest the partitioning parameter into the mapping data flow.

+#### Define a source partitioning scheme

+The format in step 1 follows the JSON standard, consisting of an array of partition definitions, each of which itself is an array of individual filter conditions. These conditions themselves are JSON objects with a structure aligned with so-called **selection options** in SAP. In fact, the format required by the SAP ODP framework is basically the same as dynamic DTP filters in SAP BW:

+```json

+{ "fieldName": <>, "sign": <>, "option": <>, "low": <>, "high": <> }

+```

+For example

+```json

+{ "fieldName": "VBELN", "sign": "I", "option": "EQ", "low": "0000001000" }

+```

+corresponds to a SQL WHERE clause ... **WHERE "VBELN" = '0000001000'**, or

+```json

+{ "fieldName": "VBELN", "sign": "I", "option": "BT", "low": "0000000000", "high": "0000001000" }

+```

+corresponds to a SQL WHERE clause ... **WHERE "VBELN" BETWEEN '0000000000' AND '0000001000'**

+A JSON definition of a partitioning scheme containing two partitions thus looks as follows

+```json

+[

+ [

+ { "fieldName": "GJAHR", "sign": "I", "option": "BT", "low": "2011", "high": "2015" }

+ ],

+ [

+ { "fieldName": "GJAHR", "sign": "I", "option": "BT", "low": "2016", "high": "2020" }

+ ]

+]

+```

+where the first partition contains fiscal years (GJAHR) 2011 through 2015, and the second partition contains fiscal years 2016 through 2020.

+>[!NOTE]

+ > Azure Data Factory doesn't perform any checks on these conditions. For example, it is in the user's responsibility to ensure that partition conditions don't overlap.

+Partition conditions can be more complex, consisting of multiple elementary filter conditions themselves. There are no logical conjunctions that explicitly define how to combine multiple elementary conditions within one partition. The implicit definition in SAP is as follows:

+1. **including** conditions ("sign": "I") for the same field name are combined with **OR** (mentally, put brackets around the resulting condition)

+2. **excluding** conditions ("sign": "E") for the same field name are combined with **OR** (again, mentally, put brackets around the resulting condition)

+3. the resulting conditions of steps 1 and 2 are

+ - combined with **AND** for **including** conditions,

+ - combined with **AND NOT** for **excluding** conditions.

+As an example, the partition condition

+```json

+ [

+ { "fieldName": "BUKRS", "sign": "I", "option": "EQ", "low": "1000" },

+ { "fieldName": "BUKRS", "sign": "I", "option": "EQ", "low": "1010" },

+ { "fieldName": "GJAHR", "sign": "I", "option": "BT", "low": "2010", "high": "2025" },

+ { "fieldName": "GJAHR", "sign": "E", "option": "EQ", "low": "2023" },

+ { "fieldName": "GJAHR", "sign": "E", "option": "EQ", "low": "2021" }

+ ]

+```

+corresponds to a SQL WHERE clause ... **WHERE ("BUKRS" = '1000' OR "BUKRS" = '1010') AND ("GJAHR" BETWEEN '2010' AND '2025') AND NOT ("GJAHR" = '2021' or "GJARH" = '2023')**

+>[!NOTE]

+ > Make sure to use the SAP internal format for the low and high values, include leading zeroes, and express calendar dates as an eight character string with the format \"YYYYMMDD\".

+#### Ingesting the partitioning parameter into mapping data flow

+To ingest the partitioning scheme into a mapping data flow, create a data flow parameter (for example, "sapPartitions"). To pass the JSON format to this parameter, it has to be converted to a string using the **@string()** function:

+Finally, in the **optimize** tab of the source transformation in your mapping data flow, select **Partition type** "Source", and enter the data flow parameter in the **Partition conditions** property.

+### Parametrizing the Checkpoint Key

+When using a parametrized data flow to extract data from multiple SAP CDC sources, it's important to parametrize the **Checkpoint Key** in the data flow activity of your pipeline. The checkpoint key is used by Azure Data Factory to manage the status of a change data capture process. To avoid that the status of one CDC process overwrites the status of another one, make sure that the checkpoint key values are unique for each parameter set used in a dataflow.

+>[!NOTE]

+ > A best practice to ensure uniqueness of the **Checkpoint Key** is to add the checkpoint key value to the set of parameters for your dataflow.

+For more information on the checkpoint key, see [Transform data with the SAP CDC connector](connector-sap-change-data-capture.md#transform-data-with-the-sap-cdc-connector).

+## Debugging

+Azure Data Factory pipelines can be executed via **triggered** or **debug runs**. A fundamental difference between these two options is, that debug runs execute the dataflow and pipeline based on the current version modeled in the user interface, while triggered runs execute the last published version of a dataflow and pipeline.

+For SAP CDC, there's one more aspect that needs to be understood: to avoid an impact of debug runs on an existing change data capture process, debug runs use a different "subscriber process" value (see [Monitor SAP CDC data flows](sap-change-data-capture-management.md#monitor-sap-cdc-data-flows)) than triggered runs. Thus, they create separate subscriptions (that is, change data capture processes) within the SAP system. In addition, the "subscriber process" value for debug runs has a life time limited to the browser UI session.

+>[!NOTE]

+ > To test stability of a change data capture process with SAP CDC over a longer period of time (say, multiple days), data flow and pipeline need to be published, and **triggered** runs need to be executed.

+The SAP CDC connector is the core of the SAP CDC capabilities. It can connect to all SAP systems that support ODP, which includes SAP ECC, SAP S/4HANA, SAP BW, and SAP BW/4HANA. The solution works either directly at the application layer or indirectly via an SAP Landscape Transformation Replication Server (SLT) as a proxy. It doesn't rely on watermarking to extract SAP data either fully or incrementally. The data the SAP CDC connector extracts includes not only physical tables but also logical objects that are created by using the tables. An example of a table-based object is an SAP Advanced Business Application Programming (ABAP) Core Data Services (CDS) view.

+The Azure side includes the mapping data flow that can transform and load the SAP data into any data sink supported by mapping data flows. Some of these options are storage destinations like Azure Data Lake Storage Gen2 or databases like Azure SQL Database or Azure Synapse Analytics. The mapping data flow activity also can load the results in Data Lake Storage Gen2 in delta format. You can use the Delta Lake Time Travel feature to produce snapshots of SAP data for a specific period. You can run your pipeline and mapping data flows frequently by using a Data Factory tumbling window trigger to replicate SAP data in Azure with low latency and without using watermarking.

+To get started, create an SAP CDC linked service, an SAP CDC source dataset, and a pipeline with a mapping data flow activity in which you use the SAP CDC source dataset. To extract the data from SAP, a self-hosted integration runtime is required that you install on an on-premises computer or on a virtual machine (VM) that has a line of sight to your SAP source systems or your SLT server. The mapping data flow activity runs on a serverless Azure Databricks or Apache Spark cluster, or on an Azure integration runtime. A staging storage is required to be configured in mapping data flow activity to make your self-hosted integration runtime work seamlessly with mapping data flow integration runtime.

+In this process, the SAP data sources are *providers*. The providers run on SAP systems to produce either full or incremental data in an operational delta queue (ODQ). The mapping data flow source is a *subscriber* of the ODQ.

+The ODP framework is part of many SAP systems, including SAP ECC and SAP S/4HANA. It's also contained in SAP BW and SAP BW/4HANA. To ensure that your SAP releases have ODP, see the following SAP documentation or support notes. Even though the guidance primarily refers to SAP BW and SAP Data Services, the information also applies to Data Factory.

+- Ensure that DataSources are activated on your SAP source systems. This requirement applies only to DataSources SAP or its partners deliver out-of-the-box. Customer-created DataSources are automatically active. If you already use a certain DataSource with SAP BW or BW/4HANA, it's already activate. For more information about DataSources and their activation, see [Installing BW Content DataSources](https://help.sap.com/saphelp_nw73/helpdata/en/4a/1be8b7aece044fe10000000a421937/frameset.htm).

+- Make sure that DataSources are released for extraction via ODP. This requirement applies to DataSources that customers create and DataSources created by SAP in older releases of SAP ECC. For more information, see the following SAP support note [2232584 - To release SAP extractors for ODP API](https://launchpad.support.sap.com/#/notes/2232584).

+### Set up the SAP Landscape Transformation Replication Server (optional)

+SAP Landscape Transformation Replication Server (SLT) is a database trigger-enabled CDC solution that can replicate SAP application tables and simple views in near real time. SLT replicates from SAP source systems to various targets, including the operational delta queue (ODQ).

+>[!NOTE]

+ > SAP Landscape Transformation Replication Server (SLT) is only required if you want to replicate data from SAP tables with the SAP CDC connector. All other sources work out-of-the-box without SLT.

+You can use SLT as a proxy in data extraction ODP. You can install SLT on an SAP source system as an SAP Data Migration Server (DMIS) add-on or use it on a standalone replication server. To use SLT as a proxy, complete the following steps:

+- [2321589 - To resolve missing Business add-in (BAdI) implementation for RSODP_ODATA subscriber type](https://launchpad.support.sap.com/#/notes/2321589)

+ Optionally, if you select **Organization**, a management project and an organization custom role are created on your GCP project for the onboarding process. Autoprovisioning is enabled for the onboarding of new projects.

+ | CSPM service account reader role <br><br> Microsoft Defender for Cloud identity federation <br><br> CSPM identity pool <br><br>Microsoft Defender for Servers service account (when the servers plan is enabled) <br><br>*Azure Arc for servers onboarding* service account (when Azure Arc for servers autoprovisioning is enabled) | Microsoft Defender for Containers service account role <br><br> Microsoft Defender Data Collector service account role <br><br> Microsoft Defender for Cloud identity pool |

+After you create the connector, a scan starts on your GCP environment. New recommendations appear in Defender for Cloud after up to 6 hours. If you enabled autoprovisioning, Azure Arc and any enabled extensions are installed automatically for each newly detected resource.

+We recommend that you use the autoprovisioning process to install Azure Arc on your VM instances. Autoprovisioning is enabled by default in the onboarding process and requires **Owner** permissions on the subscription. The Azure Arc autoprovisioning process uses the OS Config agent on the GCP end. [Learn more about the availability of the OS Config agent on GCP machines](https://cloud.google.com/compute/docs/images/os-details#vm-manager).

+The Azure Arc autoprovisioning process uses the VM manager on GCP to enforce policies on your VMs through the OS Config agent. A VM that has an [active OS Config agent](https://cloud.google.com/compute/docs/manage-os#agent-state) incurs a cost according to GCP. To see how this cost might affect your account, refer to the [GCP technical documentation](https://cloud.google.com/compute/docs/vm-manager#pricing).

+Microsoft Defender for Servers doesn't install the OS Config agent to a VM that doesn't have it installed. However, Microsoft Defender for Servers enables communication between the OS Config agent and the OS Config service if the agent is already installed but not communicating with the service. This communication can change the OS Config agent from `inactive` to `active` and lead to more costs.

+The respective Azure Arc servers for EC2 instances or GCP virtual machines that no longer exist (and the respective Azure Arc servers with a status of [Disconnected or Expired](/azure/azure-arc/servers/overview)) are removed after seven days. This process removes irrelevant Azure Arc entities to ensure that only Azure Arc servers related to existing instances are displayed.

+Defender for Servers assigns tags to your GCP resources to manage the autoprovisioning process. You must have these tags properly assigned to your resources so that Defender for Servers can manage your resources: `Cloud`, `InstanceName`, `MDFCSecurityConnector`, `MachineId`, `ProjectId`, and `ProjectNumber`.

+>

+> - If you choose to disable the available configuration options, no agents or components will be deployed to your clusters. [Learn more about feature availability](supported-machines-endpoint-solutions-clouds-containers.md).

+> - Defender for Containers when deployed on GCP, may incur external costs such as [logging costs](https://cloud.google.com/stackdriver/pricing), [pub/sub costs](https://cloud.google.com/pubsub/pricing) and [egress costs](https://cloud.google.com/vpc/network-pricing#:~:text=Platform%20SKUs%20apply.-%2cInternet%20egress%20rates%2c-Premium%20Tier%20pricing).

+ - Enable Defender for Containers autoprovisioning at the project level, as explained in the instructions in this section. We recommend this method.

+| [Preview alerts for DNS servers to be deprecated](#preview-alerts-for-dns-servers-to-be-deprecated) | August 2023 |

+### Preview alerts for DNS servers to be deprecated

+**Estimated date for change: August 2023**

+Following quality improvement process, security alerts for DNS servers are set to be deprecated in August. For cloud resources, use [Azure DNS](defender-for-dns-introduction.md) to receive the same security value.

+The following table lists the alerts to be deprecated:

+| AlertDisplayName | AlertType |

+|--|--|

+| Communication with suspicious random domain name (Preview) | DNS_RandomizedDomain

+| Communication with suspicious domain identified by threat intelligence (Preview) | DNS_ThreatIntelSuspectDomain |

+| Digital currency mining activity (Preview) | DNS_CurrencyMining |

+| Network intrusion detection signature activation (Preview) | DNS_SuspiciousDomain |

+| Attempted communication with suspicious sinkholed domain (Preview) | DNS_SinkholedDomain |

+| Communication with possible phishing domain (Preview) | DNS_PhishingDomain|

+| Possible data transfer via DNS tunnel (Preview) | DNS_DataObfuscation |

+| Possible data exfiltration via DNS tunnel (Preview) | DNS_DataExfiltration |

+| Communication with suspicious algorithmically generated domain (Preview) | DNS_DomainGenerationAlgorithm |

+| Possible data download via DNS tunnel (Preview) | DNS_DataInfiltration |

+| Anonymity network activity (Preview) | DNS_DarkWeb |

+| Anonymity network activity using web proxy (Preview) | DNS_DarkWebProxy |

+# Azure Event Grid's pull delivery (Preview) - Concepts

+# Create, view, and manage event subscriptions in namespace topics (Preview)

+# Create, view, and manage namespace topics (Preview)

+ Title: Create, view, and manage Azure Event Grid namespaces (Preview)

+# Create, view, and manage namespaces (Preview)

+> [!IMPORTANT]

+> The Namespace resource is currently in PREVIEW.

+# Monitor data reference for Azure Event Grid's MQTT delivery (Preview)

+# Monitor data reference for Azure Event Grid's pull event delivery (Preview)

+# Quickstart: Publish and subscribe to MQTT messages on Event Grid Namespace with Azure CLI (Preview)

+# Quickstart: Publish and subscribe to MQTT messages on Event Grid Namespace with Azure portal (Preview)

+ step ca init --deployment-type standalone --name MqttAppSamplesCA --dns localhost --address 127.0.0.1:443 --provisioner MqttAppSamplesCAProvisioner

+2. Using the CA files generated in step 1 to create certificate for the client.

+ step certificate create client1-authnID client1-authnID.pem client1-authnID.key --ca .step/certs/intermediate_ca.crt --ca-key .step/secrets/intermediate_ca_key --no-password --insecure --not-after 2400h

+ step certificate fingerprint client1-authnID.pem

+# Tutorial: Route MQTT messages to Azure Event Hubs from Azure Event Grid with Azure CLI (Preview)

+# Tutorial: Route MQTT messages to Azure Event Hubs from Azure Event Grid with Azure portal (Preview)

+ Title: Publish and consume events or messages using namespace topics (Preview)

+# Publish to namespace topics and consume events (Preview)

+- [Azure Databricks: Azure Blob storage][Azure Databricks: Azure Blob Storage]. See the following sample: [Streaming at Scale with Event Hubs Capture](https://github.com/Azure-Samples/streaming-at-scale/tree/main/eventhubs-capture-databricks-delta).

+> [!NOTE]

+> ExpressRoute supports redundant pair of cross connection. If you exceed the configured bandwidth of your ExpressRoute circuit in a cross connection, your traffic would be subjected to rate limiting within that cross connection.

+>

+#### Enrichments

+There are three types of enrichment that you can add to an export: custom strings, system properties, and custom properties:

+The following example shows how to use the `enrichments` node to add a custom string to the outgoing message:

+```json

+"enrichments": {

+ "My custom string": {

+ "value": "My value"

+ },

+ //...

+}

+```

+The following example shows how to use the `enrichments` node to add a system property to the outgoing message:

+```json

+"enrichments": {

+ "Device template": {

+ "path": "$templateDisplayName"

+ },

+ //...

+}

+```

+You can add the following system properties:

+| Property | Description |

+| -- | -- |

+| `$enabled` | Is the device enabled? |

+| `$displayName` | The device name. |

+| `$templateDisplayName` | The device template name. |

+| `$organizations` | The organizations the device belongs to. |

+| `$provisioned` | Is the device provisioned? |

+| `$simulated` | Is the device simulated? |

+The following example shows how to use the `enrichments` node to add a custom property to the outgoing message. Custom properties are properties defined in the device template the device is associated with:

+```json

+"enrichments": {

+ "Device model": {

+ "target": "dtmi:azure:DeviceManagement:DeviceInformation;1",

+ "path": "model"

+ },

+ //...

+}

+```

+#### Filters

+You can filter the exported messages based on telemetry or property values.

+The following example shows how to use the `filter` field to export only messages where the accelerometer-X telemetry value is greater than 0:

+```json

+{

+ "id": "export-001",

+ "displayName": "Enriched Export",

+ "enabled": true,

+ "source": "telemetry",

+ "filter": "SELECT * FROM dtmi:azurertos:devkit:gsgmxchip;1 WHERE accelerometerX > 0",

+ "destinations": [

+ {

+ "id": "dest-001"

+ }

+ ],

+ "status": "healthy"

+}

+```

+The following example shows how to use the `filter` field to export only messages where the `temperature` telemetry value is greater than the `targetTemperature` property:

+```json

+{

+ "id": "export-001",

+ "displayName": "Enriched Export",

+ "enabled": true,

+ "source": "telemetry",

+ "filter": "SELECT * FROM dtmi:azurertos:devkit:gsgmxchip;1 AS A, dtmi:contoso:Thermostat;1 WHERE A.temperature > targetTemperature",

+ "destinations": [

+ {

+ "id": "dest-001"

+ }

+ ],

+ "status": "healthy"

+}

+```

+To learn how to manage jobs by using the IoT Central REST API, see [How to use the IoT Central REST API to manage devices](../core/howto-manage-jobs-with-rest-api.md).

+> [!TIP]

+> When you create a recurring job, sign in to your application using a Microsoft account or Azure Active Directory account. If you sign in using an Azure Active Directory group, it's possible that the Azure Active Directory token associated with the group will expire at some point in the future and cause the job to fail.

+### Resource health alerts

+Azure Resource Health alerts can notify you in near real-time when the health state of your Load balancer resource changes. It's recommended that you set resource health alerts to notify you when your Load balancer resource is in a **Degraded** or **Unavailable** state.

+When you create Azure resource health alerts for Load balancer, Azure sends resource health notifications to your Azure subscription. You can create and customize alerts based on:

+* The subscription affected

+* The resource group affected

+* The resource type affected (Load balancer)

+* The specific resource (any Load balancer resource you choose to set up an alert for)

+* The event status of the Load balancer resource affected

+* The current status of the Load balancer resource affected

+* The previous status of the Load balancer resource affected

+* The reason type of the Load balancer resource affected

+You can also configure who the alert should be sent to:

+* A new action group (that can be used for future alerts)

+* An existing action group

+For more information on how to set up these resource health alerts, see:

+* [Resource health alerts using Azure portal](/azure/service-health/resource-health-alert-monitor-guide#create-a-resource-health-alert-rule-in-the-azure-portal)

+* [Resource health alerts using Resource Manager templates](/azure/service-health/resource-health-alert-arm-template-guide)

+> [!IMPORTANT]

+> This feature is currently in public preview.

+> This preview version is provided without a service-level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+| Tool | Windows 2019 Server DSVM | Windows 2022 Server DSVM (Preview) | Linux DSVM | Usage notes |

+|--|:-:|:-:|:-:|:-|

+| [Microsoft 365](https://www.microsoft.com/microsoft-365) (Word, Excel, PowerPoint) | <span class='green-check'>&#9989; </span> | <span class='green-check'>&#9989; </span> | <span class='red-x'>&#10060; </span> | |

+| [Microsoft Teams](https://www.microsoft.com/microsoft-teams) | <span class='green-check'>&#9989; </span> | <span class='green-check'>&#9989; </span> | <span class='red-x'>&#10060; </span> | |

+| [Power BI Desktop](https://powerbi.microsoft.com/) | <span class='green-check'>&#9989; </span> | <span class='green-check'>&#9989; </span>| <span class='red-x'>&#10060; </span> | |

+| [Microsoft Edge Browser](https://www.microsoft.com/edge) | <span class='green-check'>&#9989; </span> | <span class='green-check'>&#9989; </span> | <span class='red-x'>&#10060; </span> | |

+> [!IMPORTANT]

+> This feature is currently in public preview.

+> This preview version is provided without a service-level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+|Feature |Data Science<br>VM |Azure Machine Learning<br>Compute Instance |

+|Preinstalled Tools | Jupyter(lab), VS Code,<br> Visual Studio, PyCharm, Juno,<br>Power BI Desktop, SSMS, <br>Microsoft Office 365, Apache Drill | Jupyter(lab) |

+The Data Science Virtual Machine comes with the most useful data-science tools pre-installed.

+> [!IMPORTANT]

+> This feature is currently in public preview.

+> This preview version is provided without a service-level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+| Tool | Windows Server 2019 DSVM | Windows Server 2022 DSVM (Preview) | Ubuntu 20.04 DSVM | Usage notes |

+|--|:-:|:-:|:-:|:-:|

+| [CUDA, cuDNN, NVIDIA Driver](https://developer.nvidia.com/cuda-toolkit) | <span class='green-check'>&#9989;</span> |<span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span></br> | [CUDA, cuDNN, NVIDIA Driver on the DSVM](./dsvm-tools-deep-learning-frameworks.md#cuda-cudnn-nvidia-driver) |

+| [Horovod](https://github.com/horovod/horovod) | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | [Horovod on the DSVM](./dsvm-tools-deep-learning-frameworks.md#horovod) |

+| [NVidia System Management Interface (nvidia-smi)](https://developer.nvidia.com/nvidia-system-management-interface) | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | [nvidia-smi on the DSVM](./dsvm-tools-deep-learning-frameworks.md#nvidia-system-management-interface-nvidia-smi) |

+| [PyTorch](https://pytorch.org) | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | [PyTorch on the DSVM](./dsvm-tools-deep-learning-frameworks.md#pytorch) |

+| [TensorFlow](https://www.tensorflow.org) | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span> | [TensorFlow on the DSVM](./dsvm-tools-deep-learning-frameworks.md#tensorflow) |

+| Integration with [Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/) (Python) | <span class='green-check'>&#9989;</span></br> (Python SDK, samples) | <span class='green-check'>&#9989;</span></br> (Python SDK, samples) | <span class='green-check'>&#9989;</span></br> (Python SDK,CLI, samples) | [Azure Machine Learning SDK](./dsvm-tools-data-science.md#azure-machine-learning-sdk-for-python) |

+| [XGBoost](https://github.com/dmlc/xgboost) | <span class='green-check'>&#9989;</span></br> (CUDA support) | <span class='green-check'>&#9989;</span></br> (CUDA support) | <span class='green-check'>&#9989;</span></br> (CUDA support) | [XGBoost on the DSVM](./dsvm-tools-data-science.md#xgboost) |

+| [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit) | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span></br> | [Vowpal Wabbit on the DSVM](./dsvm-tools-data-science.md#vowpal-wabbit) |

+| [Weka](https://www.cs.waikato.ac.nz/ml/weka/) | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | |

+| LightGBM | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span></br> (GPU, MPI support) | |

+| H2O | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| CatBoost | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| Intel MKL | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| OpenCV | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| Dlib | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| Docker | <span class='green-check'>&#9989;</span><br/> (Windows containers only) | <span class='green-check'>&#9989;</span> <br/> (Windows containers only) | <span class='green-check'>&#9989;</span> | |

+| Nccl | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| Rattle | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | |

+| PostgreSQL | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| ONNX Runtime | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| Tool | Windows Server 2019 DSVM | Windows Server 2022 DSVM (Preview)| Ubuntu 20.04 DSVM | Usage notes |

+|--|:-:|:-:|:-:|:-:|

+| Relational databases | [SQL Server 2019](https://www.microsoft.com/sql-server/sql-server-2019) <br/> Developer Edition | [SQL Server 2019](https://www.microsoft.com/sql-server/sql-server-2019) <br/> Developer Edition | [SQL Server 2019](https://www.microsoft.com/sql-server/sql-server-2019) <br/> Developer Edition | [SQL Server on the DSVM](./dsvm-tools-data-platforms.md#sql-server-developer-edition) |

+| Database tools | SQL Server Management Studio<br/> SQL Server Integration Services<br/> [bcp, sqlcmd](/sql/tools/command-prompt-utility-reference-database-engine) | SQL Server Management Studio<br/> SQL Server Integration Services<br/> [bcp, sqlcmd](/sql/tools/command-prompt-utility-reference-database-engine) | [SQuirreL SQL](http://squirrel-sql.sourceforge.net/) (querying tool), <br /> bcp, sqlcmd <br /> ODBC/JDBC drivers | |

+| [Azure Storage Explorer](https://azure.microsoft.com/features/storage-explorer/) | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | |

+| [Azure CLI](/cli/azure) | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | |

+| [AzCopy](../../storage/common/storage-use-azcopy-v10.md) | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | <span class='red-x'>&#10060;</span> | [AzCopy on the DSVM](./dsvm-tools-ingestion.md#azcopy) |

+| [Blob FUSE driver](https://github.com/Azure/azure-storage-fuse) | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span></br> | [blobfuse on the DSVM](./dsvm-tools-ingestion.md#blobfuse) |

+| [Azure Cosmos DB Data Migration Tool](../../cosmos-db/import-data.md) | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='red-x'>&#10060;</span> | [Azure Cosmos DB on the DSVM](./dsvm-tools-ingestion.md#azure-cosmos-db-data-migration-tool) |

+| Unix/Linux command-line tools | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| Apache Spark 3.1 (standalone) | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span></br> | |

+| Tool | Windows Server 2019 DSVM | Windows Server 2022 DSVM (Preview) | Ubuntu 20.04 DSVM | Usage notes |

+|--|:-:|:-:|:-:|:-:|

+| [CRAN-R](https://cran.r-project.org/) with popular packages pre-installed | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | |

+| [Anaconda Python](https://www.continuum.io/) with popular packages pre-installed | <span class='green-check'>&#9989;</span> |<span class='green-check'>&#9989;</span><br/> (Miniconda) | <span class='green-check'>&#9989;</span></br> (Miniconda) | |

+| [Julia (Julialang)](https://julialang.org/) | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | |

+| JupyterHub (multiuser notebook server) | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| JupyterLab (multiuser notebook server) | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | |

+| Node.js | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | |

+| [Jupyter Notebook Server](https://jupyter.org/) with the following kernels: | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span> | [Jupyter Notebook samples](./dsvm-samples-and-walkthroughs.md) |

+| &nbsp;&nbsp;&nbsp;&nbsp; R | | | | [R Jupyter Samples](./dsvm-samples-and-walkthroughs.md#r-language) |

+| &nbsp;&nbsp;&nbsp;&nbsp; Python | | | | [Python Jupyter Samples](./dsvm-samples-and-walkthroughs.md#python-language) |

+| &nbsp;&nbsp;&nbsp;&nbsp; Julia | | | | [Julia Jupyter Samples](./dsvm-samples-and-walkthroughs.md#julia-language) |

+| &nbsp;&nbsp;&nbsp;&nbsp; PySpark | | | | [pySpark Jupyter Samples](./dsvm-samples-and-walkthroughs.md#sparkml) |

+**Ubuntu 20.04 DSVM, Windows Server 2019 DSVM, and Windows Server 2022 DSVM (Preview)** have the following Jupyter Kernels:-</br>

+**Ubuntu 20.04 DSVM, Windows Server 2019 DSVM, and Windows Server 2022 DSVM (Preview)** have the following conda environments:-</br>

+| Tool | Windows Server 2019 DSVM | Windows Server 2022 DSVM (Preview)| Ubuntu 20.04 DSVM | Usage notes |

+|--|:-:|:-:|:-:|:-:|

+| [Notepad++](https://notepad-plus-plus.org/) | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | <span class='red-x'>&#10060;</span></br> | |

+| [Nano](https://www.nano-editor.org/) | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | <span class='red-x'>&#10060;</span></br> | |

+| [Visual Studio 2019 Community Edition](https://www.visualstudio.com/community/) | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span> | <span class='red-x'>&#10060;</span> | [Visual Studio on the DSVM](dsvm-tools-development.md#visual-studio-community-edition) |

+| [Visual Studio Code](https://code.visualstudio.com/) | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | [Visual Studio Code on the DSVM](./dsvm-tools-development.md#visual-studio-code) |

+| [PyCharm Community Edition](https://www.jetbrains.com/pycharm/) | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | [PyCharm on the DSVM](./dsvm-tools-development.md#pycharm) |

+| [IntelliJ IDEA](https://www.jetbrains.com/idea/) | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span> | |

+| [Vim](https://www.vim.org) | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span></br> | |

+| [Emacs](https://www.gnu.org/software/emacs) | <span class='red-x'>&#10060;</span> | <span class='red-x'>&#10060;</span> | <span class='green-check'>&#9989;</span></br> | |

+| [Git](https://git-scm.com/) and Git Bash | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | |

+| [OpenJDK](https://openjdk.java.net) 11 | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | |

+| .NET Framework | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span></br> | <span class='red-x'>&#10060;</span> | |

+| Azure SDK | <span class='green-check'>&#9989;</span></br> | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | |

+| Tool | Windows Server 2019 DSVM | Windows Server 2022 DSVM (Preview) | Ubuntu 20.04 DSVM | Usage notes |

+|--|:-:|:-:|:-:|:-:|

+| [Microsoft 365](https://www.microsoft.com/microsoft-365) (Word, Excel, PowerPoint) | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='red-x'>&#10060;</span> | |

+| [Microsoft Teams](https://www.microsoft.com/microsoft-teams) | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='red-x'>&#10060;</span> | |

+| [Power BI Desktop](https://powerbi.microsoft.com/) | <span class='green-check'>&#9989;</span> |<span class='green-check'>&#9989;</span></br> | <span class='red-x'>&#10060;</span> | |

+| Microsoft Edge Browser | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | <span class='green-check'>&#9989;</span> | |

+ Title: Create a vector index in an Azure Machine Learning prompt flow (preview)

+description: Learn how to create a vector index in Azure Machine Learning and use it in a prompt flow.

+# Create a vector index in an Azure Machine Learning prompt flow (preview)

+You can use Azure Machine Learning to create a vector index from files or folders on your machine, a location in cloud storage, an Azure Machine Learning data asset, a Git repository, or a SQL database. Azure Machine Learning can currently process .txt, .md, .pdf, .xls, and .docx files. You can also reuse an existing Azure Cognitive Search index instead of creating a new index.

+When you create a vector index, Azure Machine Learning chunks the data, creates embeddings, and stores the embeddings in a Faiss index or Azure Cognitive Search index. In addition, Azure Machine Learning creates:

+* A sample prompt flow, which uses the vector index that you created. Features of the sample prompt flow include:

+ * Automatically generated prompt variants.

+ * Evaluation of each prompt variant by using the [generated test data](https://aka.ms/prompt_flow_blog).

+ * Metrics against each prompt variant to help you choose the best variant to run.

+ You can use this sample to continue developing your prompt.

+* Access to Azure OpenAI Service.

+* Prompt flows enabled in your Azure Machine Learning workspace. You can enable prompt flows by turning on **Build AI solutions with Prompt flow** on the **Manage preview features** panel.

+## Create a vector index by using Machine Learning studio

+1. Select **Prompt flow** on the left menu.

+ :::image type="content" source="media/how-to-create-vector-index/prompt.png" alt-text="Screenshot that shows the location of prompt flow on the left menu.":::

+1. Select the **Vector Index** tab.

+ :::image type="content" source="./media/how-to-create-vector-index/vector-index.png" alt-text="Screenshot that shows the tab for vector index.":::

+1. Select **Create**.

+1. When the form for creating a vector index opens, provide a name for your vector index.

+ :::image type="content" source="media/how-to-create-vector-index/new-vector-creation.png" alt-text="Screenshot that shows basic settings for creating a vector index.":::

+1. Select your data source type.

+1. Based on the chosen type, provide the location details of your source. Then, select **Next**.

+1. Review the details of your vector index, and then select the **Create** button.

+1. On the overview page that appears, you can track and view the status of creating your vector index. The process might take a while, depending on the size of your data.

+## Add a vector index to a prompt flow

+After you create a vector index, you can add it to a prompt flow from the prompt flow canvas.

+1. Open an existing prompt flow.

+1. On the top menu of the prompt flow designer, select **More tools**, and then select **Vector Index Lookup**.

+ :::image type="content" source="media/how-to-create-vector-index/vector-lookup.png" alt-text="Screenshot that shows the list of available tools.":::

+ The Vector Index Lookup tool is added to the canvas. If you don't see the tool immediately, scroll to the bottom of the canvas.

+ :::image type="content" source="media/how-to-create-vector-index/vector-index-lookup-tool.png" alt-text="Screenshot that shows the Vector Index Lookup tool.":::

+1. Enter the path to your vector index, along with the query that you want to perform against the index.

+[Get started with RAG by using a prompt flow sample (preview)](how-to-use-pipelines-prompt-flow.md)

+[Use vector stores with Azure Machine Learning (preview)](concept-vector-stores.md)

+* Training course on [MLOps with Machine Learning](https://learn.microsoft.com/training/paths/introduction-machine-learn-operations/)

+* Training course on [MLOps with Machine Learning](https://learn.microsoft.com/training/paths/introduction-machine-learn-operations/)

+1. Have basic understanding on managed identities. [Learn more about managed identities.](../../active-directory/managed-identities-azure-resources/overview.md)

+If you didn't complete the tutorial, you need to build a flow. Testing the flow properly by bulk tests and evaluation before deployment is a recommended best practice.

+In the flow authoring page or run detail page, select **Deploy**.

+**Flow authoring page**:

+**Run detail page**:

+A wizard for you to configure the endpoint occurs and include following steps.

+System-assigned identity will be autocreated after your endpoint is created, while user-assigned identity is created by user. The advantage of user-assigned identity is that you can assign multiple endpoints with the same user-assigned identity, and you just need to grant needed permissions to the user-assigned identity once. [Learn more about managed identities.](../../active-directory/managed-identities-azure-resources/overview.md)

+Select the identity you want to use, and you'll notice a warning message to remind you to grant correct permissions to the identity.

+> [!IMPORTANT]

+> When creating the deployment, Azure tries to pull the user container image from the workspace Azure Container Registry (ACR) and mount the user model and code artifacts into the user container from the workspace storage account.

+>

+> To do these, Azure uses managed identities to access the storage account and the container registry.

+>

+> - If you created the associated endpoint with **System Assigned Identity**, Azure role-based access control (RBAC) permission is automatically granted, and no further permissions are needed.

+>

+> - If you created the associated endpoint with **User Assigned Identity**, the user's managed identity must have Storage blob data reader permission on the storage account for the workspace, and AcrPull permission on the Azure Container Registry (ACR) for the workspace. Make sure your User Assigned Identity has the right permission **before the deployment creation**; otherwise, the deployment creation will fail. If you need to create multiple endpoints, it is recommended to use the same user-assigned identity for all endpoints in the same workspace, so that you only need to grant the permissions to the identity once.

+|Property| System Assigned Identity | User Assigned Identity|

+||||

+|| if you select system assigned identity, it will be auto-created by system for this endpoint <br> | created by user. [Learn more about how to create user assigned identities](../../active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities.md#create-a-user-assigned-managed-identity). <br> one user assigned identity can be assigned to multiple endpoints|

+|Pros| Permissions needed to pull image and mount model and code artifacts from workspace storage are auto-granted.| Can be shared by multiple endpoints.|

+|Required permissions|**Workspace**: **AzureML Data Scientist** role **OR** a customized role with ΓÇ£Microsoft.MachineLearningServices/workspaces/connections/listsecrets/actionΓÇ¥ <br> |**Workspace**: **AzureML Data Scientist** role **OR** a customized role with ΓÇ£Microsoft.MachineLearningServices/workspaces/connections/listsecrets/actionΓÇ¥ <br> **Workspace container registry**: **Acr pull** <br> **Workspace default storage**: **Storage Blob Data Reader**|

+See detailed guidance about how to grant permissions to the endpoint identity in [Grant permissions to the endpoint](#grant-permissions-to-the-endpoint).

+ > If you select **System Assigned Identity**, make sure you have granted correct permissions by adding role assignment to the managed identity of the endpoint **before you test or consume the endpoint**. Otherwise, the endpoint will fail to perform inference due to lacking of permissions.

+ >

+ > If you select **User Assigned Identity**, the user's managed identity must have Storage blob data reader permission on the storage account for the workspace, and AcrPull permission on the Azure Container Registry (ACR) for the workspace. Make sure your User Assigned Identity has the right permission **before the deployment creation** - better do it before you finisht the deploy wizard; otherwise, the deployment creation will fail. If you need to create multiple endpoints, it is recommended to use the same user-assigned identity for all endpoints in the same workspace, so that you only need to grant the permissions to the identity once.

+- You can use Azure Resource Manager template to grant all permissions. You can find related Azure Resource Manager templates in [Prompt flow GitHub repo](https://github.com/cloga/azure-quickstart-templates/tree/lochen/promptflow/quickstarts/microsoft.machinelearningservices/machine-learning-prompt-flow).

+- It might be because you ran your flow in an old version runtime and then deployed the flow, the deployment used the environment of the runtime which was in old version as well. Update the runtime following [this guidance](./how-to-create-manage-runtime.md#update-runtime-from-ui) and rerun the flow in the latest runtime and then deploy the flow again.

+This tutorial series shows how features seamlessly integrate all phases of the ML lifecycle: prototyping, training and operationalization.

+Part 1 of this tutorial showed how to create a feature set spec with custom transformations, and use that feature set to generate training data. This tutorial describes materialization, which computes the feature values for a given feature window, and then stores those values in a materialization store. All feature queries can then use the values from the materialization store. A feature set query applies the transformations to the source on the fly, to compute the features before it returns the values. This works well for the prototyping phase. However, for training and inference operations in a production environment, it's recommended that you materialize the features, for greater reliability and availability.

+This tutorial is part two of a four part series. In this tutorial, you'll learn how to:

+> [!div class="checklist"]

+> * Enable offline store on the feature store by creating and attaching an Azure Data Lake Storage Gen2 container and a user assigned managed identity

+> * Enable offline materialization on the feature sets, and backfill the feature data

+> [!IMPORTANT]

+> This feature is currently in public preview. This preview version is provided without a service-level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+* Complete the part 1 tutorial, to create the required feature store, account entity and transaction feature set

+* An Azure Resource group, where you (or the service principal you use) have `User Access Administrator`and `Contributor` roles.

+To proceed with this article, your user account needs the owner role or contributor role for the resource group that holds the created feature store.

+## Set up

+This list summarizes the required setup steps:

+1. In your project workspace, create an Azure Machine Learning compute resource, to run the training pipeline

+1. In your feature store workspace, create an offline materialization store: create an Azure gen2 storage account and a container inside it, and attach it to the feature store. Optional: you can use an existing storage container

+1. Create and assign a user-assigned managed identity to the feature store. Optionally, you can use an existing managed identity. The system managed materialization jobs - in other words, the recurrent jobs - use the managed identity. Part 3 of the tutorial relies on this

+1. Grant required role-based authentication control (RBAC) permissions to the user-assigned managed identity

+1. Grant required role-based authentication control (RBAC) to your Azure AD identity. Users, including yourself, need read access to the sources and the materialization store

+### Configure the Azure Machine Learning spark notebook

+1. Running the tutorial:

+ You can create a new notebook, and execute the instructions in this document, step by step. You can also open the existing notebook named `2. Enable materialization and backfill feature data.ipynb`, and then run it. You can find the notebooks in the `featurestore_sample/notebooks directory`. You can select from `sdk_only`, or `sdk_and_cli`. You can keep this document open, and refer to it for documentation links and more explanation.

+1. Select Azure Machine Learning Spark compute in the "Compute" dropdown, located in the top nav.

+1. Configure the session:

+ * Select "configure session" in the bottom nav

+ * Select **upload conda file**

+ * Upload the **conda.yml** file you [uploaded in Tutorial #1](./tutorial-get-started-with-feature-store.md#prepare-the-notebook-environment-for-development)

+ * Increase the session time-out (idle time) to avoid frequent prerequisite reruns

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=start-spark-session)]

+### Set up the root directory for the samples

+[!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=root-dir)]

+ 1. Set up the CLI

+ # [Python SDK](#tab/python)

+

+ Not applicable

+

+ # [Azure CLI](#tab/cli)

+

+ 1. Install the Azure Machine Learning extension

+

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=install-ml-ext-cli)]

+

+ 1. Authentication

+

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=auth-cli)]

+

+ 1. Set the default subscription

+

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=set-default-subs-cli)]

+

+

+1. Initialize the project workspace properties

+ This is the current workspace. You'll run the tutorial notebook from this workspace.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=init-ws-crud-client)]

+1. Initialize the feature store properties

+ Make sure that you update the `featurestore_name` and `featurestore_location` values shown, to reflect what you created in part 1 of this tutorial.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=init-fs-crud-client)]

+1. Initialize the feature store core SDK client

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=init-fs-core-sdk)]

+1. Set up the offline materialization store

+ You can create a new gen2 storage account and a container. You can also reuse an existing gen2 storage account and container as the offline materialization store for the feature store.

+ # [Python SDK](#tab/python)

+ You can optionally override the default settings.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=setup-utility-fns)]

+ # [Azure CLI](#tab/cli)

+ Not applicable

+

+## Set values for the Azure Data Lake Storage Gen2 storage

+ The materialization store uses these values. You can optionally override the default settings.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=set-offline-store-params)]

+1. Storage containers

+ Option 1: create new storage and container resources

+ # [Python SDK](#tab/python)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=create-new-storage)]

+ # [Azure CLI](#tab/cli)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=create-new-storage)]

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=create-new-storage-container)]

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=set-container-arm-id-cli)]

+

+ Option 2: reuse an existing storage container

+ # [Python SDK](#tab/python)

+

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=use-existing-storage)]

+

+ # [Azure CLI](#tab/cli)

+

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=use-existing-storage)]

+

+

+1. Set up user assigned managed identity (UAI)

+ The system-managed materialization jobs will use the UAI. For example, the recurrent job in part 3 of this tutorial uses this UAI.

+### Set the UAI values

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=set-uai-params)]

+### User assigned managed identity (option 1)

+ Create a new one

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=create-new-uai)]

+### User assigned managed identity (option 2)

+ Reuse an existing managed identity

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=use-existing-uai)]

+### Retrieve UAI properties

+ Run this code sample in the SDK to retrieve the UAI properties:

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=retrieve-uai-properties)]

+

+## Grant RBAC permission to the user assigned managed identity (UAI)

+ This UAI is assigned to the feature store shortly. It requires these permissions:

+ | **Scope** | **Action/Role** |

+ ||--|

+ | Feature Store | Azure Machine Learning Data Scientist role |

+ | Storage account of feature store offline store | Blob storage data contributor role |

+ | Storage accounts of source data | Blob storage data reader role |

+ The next CLI commands will assign the first two roles to the UAI. In this example, "Storage accounts of source data" doesn't apply because we read the sample data from a public access blob storage. To use your own data sources, you must assign the required roles to the UAI. To learn more about access control, see the [access control document]() in the documentation resources.

+ # [Python SDK](#tab/python)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=grant-rbac-to-uai)]

+ # [Azure CLI](#tab/cli)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=grant-rbac-to-uai-fs)]

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=grant-rbac-to-uai-offline-store)]

+

+### Grant the blob data reader role access to your user account in the offline store

+ If the feature data is materialized, you need this role to read feature data from the offline materialization store.

+ Obtain your Azure AD object ID value from the Azure portal as described [here](/partner-center/find-ids-and-domain-names#find-the-user-object-id).

+ To learn more about access control, see the [access control document](./how-to-setup-access-control-feature-store.md).

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=grant-rbac-to-user-identity)]

+ The following steps grant the blob data reader role access to your user account.

+ 1. Attach the offline materialization store and UAI, to enable the offline store on the feature store

+ # [Python SDK](#tab/python)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=enable-offline-store)]

+ # [Azure CLI](#tab/cli)

+ Action: inspect file `xxxx`. This command attaches the offline store and the UAI, to update the feature store.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=dump_featurestore_yaml)]

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=enable-offline-store)]

+

+ 2. Enable offline materialization on the transactions feature set

+ Once materialization is enabled on a feature set, you can perform a backfill, as explained in this tutorial. You can also schedule recurrent materialization jobs. See [part 3](./tutorial-experiment-train-models-using-features.md) of this tutorial series for more information.

+ # [Python SDK](#tab/python)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=enable-offline-mat-txns-fset)]

+ # [Azure CLI](#tab/cli)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/2. Enable materialization and backfill feature data.ipynb?name=enable-offline-mat-txns-fset)]

+

+ Optional: you can save the feature set asset as a YAML resource

+ # [Python SDK](#tab/python)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=dump-txn-fset-yaml)]

+ # [Azure CLI](#tab/cli)

+ Not applicable

+

+ 3. Backfill data for the transactions feature set

+ As explained earlier in this tutorial, materialization computes the feature values for a given feature window, and stores these computed values in a materialization store. Feature materialization increases the reliability and availability of the computed values. All feature queries now use the values from the materialization store. This step performs a one-time backfill, for a feature window of three months.

+ > [!NOTE]

+ > You might need to determine a backfill data window. The window must match the window of your training data. For example, to use two years of data for training, you need to retrieve features for the same window. This means you should backfill for a two year window.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=backfill-txns-fset)]

+ We'll print sample data from the feature set. The output information shows that the data was retrieved from the materialization store. The `get_offline_features()` method retrieved the training and inference data, and it also uses the materialization store by default.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/2. Enable materialization and backfill feature data.ipynb?name=sample-txns-fset-data)]

+The Tutorial #4 [clean up step](./tutorial-enable-recurrent-materialization-run-batch-inference.md#cleanup) describes how to delete the resources

+* Reference: [YAML reference](./reference-yaml-overview.md)

+This tutorial series shows how features seamlessly integrate all phases of the ML lifecycle: prototyping, training and operationalization.

+Part 1 of this tutorial showed how to create a feature set spec with custom transformations, and use that feature set to generate training data. Part 2 of the tutorial showed how to enable materialization and perform a backfill. Part 3 of this tutorial showed how to experiment with features, as a way to improve model performance. Part 3 also showed how a feature store increases agility in the experimentation and training flows. Tutorial 4 explains how to

+> [!div class="checklist"]

+> * Run batch inference for the registered model

+> * Enable recurrent materialization for the `transactions` feature set

+> * Run a batch inference pipeline on the registered model

+> [!IMPORTANT]

+> This feature is currently in public preview. This preview version is provided without a service-level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+Before you proceed with this article, make sure you complete parts 1, 2, and 3 of this tutorial series.

+## Set up

+ 1. In the "Compute" dropdown in the top nav, select "Configure session"

+ To run this tutorial, you can create a new notebook, and execute the instructions in this document, step by step. You can also open and run the existing notebook named `4. Enable recurrent materialization and run batch inference`. You can find that notebook, and all the notebooks in this series, at the `featurestore_sample/notebooks directory`. You can select from `sdk_only`, or `sdk_and_cli`. You can keep this document open, and refer to it for documentation links and more explanation.

+ 1. Select Azure Machine Learning Spark compute in the "Compute" dropdown, located in the top nav.

+ 1. Configure session:

+ * Upload the **conda.yml** file you [uploaded in Tutorial #1](./tutorial-get-started-with-feature-store.md#prepare-the-notebook-environment-for-development)

+### Start the spark session

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable recurrent materialization and run batch inference.ipynb?name=start-spark-session)]

+### Set up the root directory for the samples

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable recurrent materialization and run batch inference.ipynb?name=root-dir)]

+ ### [Python SDK](#tab/python)

+ Not applicable

+ ### [Azure CLI](#tab/cli)

+ **Set up the CLI**

+ 1. Install the Azure Machine Learning extension

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/4. Enable recurrent materialization and run batch inference.ipynb?name=install-ml-ext-cli)]

+ 1. Authentication

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/4. Enable recurrent materialization and run batch inference.ipynb?name=auth-cli)]

+ 1. Set the default subscription

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/4. Enable recurrent materialization and run batch inference.ipynb?name=set-default-subs-cli)]

+

+1. Initialize the project workspace CRUD client

+ The tutorial notebook runs from this current workspace

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable recurrent materialization and run batch inference.ipynb?name=init-ws-crud-client)]

+1. Initialize the feature store variables

+ Make sure that you update the `featurestore_name` value, to reflect what you created in part 1 of this tutorial.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable recurrent materialization and run batch inference.ipynb?name=init-fs-crud-client)]

+1. Initialize the feature store SDK client

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable recurrent materialization and run batch inference.ipynb?name=init-fs-core-sdk)]

+## Enable recurrent materialization on the `transactions` feature set

+We enabled materialization in tutorial part 2, and we also performed backfill on the transactions feature set. Backfill is an on-demand, one-time operation that computes and places feature values in the materialization store. However, to handle inference of the model in production, you might want to set up recurrent materialization jobs to keep the materialization store up-to-date. These jobs run on user-defined schedules. The recurrent job schedule works this way:

+ * interval = 3

+ * frequency = Hour

+ define a three-hour window.

+* The first recurrent job is submitted at the start of the next window after the update time.

+[!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable recurrent materialization and run batch inference.ipynb?name=enable-recurrent-mat-txns-fset)]

+## (Optional) Save the feature set asset yaml file

+ We use the updated settings to save the yaml file

+ ### [Python SDK](#tab/python)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable recurrent materialization and run batch inference.ipynb?name=dump-txn-fset-with-mat-yaml)]

+ ### [Azure CLI](#tab/cli)

+ Not applicable

+

+## Run the batch-inference pipeline

+ The batch-inference has these steps:

+ 1. Feature retrieval: this uses the same built-in feature retrieval component used in the training pipeline, covered in tutorial part 3. For pipeline training, we provided a feature retrieval spec as a component input. However, for batch inference, we pass the registered model as the input, and the component looks for the feature retrieval spec in the model artifact.

+

+ Additionally, for training, the observation data had the target variable. However, the batch inference observation data doesn't have the target variable. The feature retrieval step joins the observation data with the features, and outputs the data for batch inference.

+ 1. Batch inference: This step uses the batch inference input data from previous step, runs inference on the model, and appends the predicted value as output.

+ > [!NOTE]

+ > We use a job for batch inference in this example. You can also use Azure ML's batch endpoints.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable recurrent materialization and run batch inference.ipynb?name=run-batch-inf-pipeline)]

+ ### Inspect the batch inference output data

+ In the pipeline view

+ 1. Select `inference_step` in the `outputs` card

+ 1. Copy the Data field value. It looks something like `azureml_995abbc2-3171-461e-8214-c3c5d17ede83_output_data_data_with_prediction:1`

+ 1. Paste the Data field value in the following cell, with separate name and version values (note that the last character is the version, preceded by a `:`).

+ 1. Note the `predict_is_fraud` column that the batch inference pipeline generated

+ Explanation: In the batch inference pipeline (`/project/fraud_mode/pipelines/batch_inference_pipeline.yaml`) outputs, since we didn't provide `name` or `version` values in the `outputs` of the `inference_step`, the system created an untracked data asset with a guid as the name value, and 1 as the version value. In this cell, we derive and then display the data path from the asset:

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/4. Enable recurrent materialization and run batch inference.ipynb?name=inspect-batch-inf-output-data)]

+If you created a resource group for the tutorial, you can delete the resource group, to delete all the resources associated with this tutorial. Otherwise, you can delete the resources individually:

+1. To delete the feature store, go to the resource group in the Azure portal, select the feature store, and delete it

+1. Follow [these instructions](../active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities.md) to delete the user-assigned managed identity

+1. To delete the offline store (storage account), go to the resource group in the Azure portal, select the storage you created, and delete it

+* Reference: [YAML reference](./reference-yaml-overview.md)

+Part 1 of this tutorial showed how to create a feature set spec with custom transformations, and use that feature set to generate training data. Part 2 of the tutorial showed how to enable materialization and perform a backfill. Tutorial 3 shows how to experiment with features, as a way to improve model performance. This tutorial also shows how a feature store increases agility in the experimentation and training flows. It shows how to:

+> [!div class="checklist"]

+> * Prototype a new `accounts` feature set spec, using existing precomputed values as features. Then, register the local feature set spec as a feature set in the feature store. This differs from tutorial part 1, where we created a feature set that had custom transformations

+> * Select features for the model from the `transactions` and `accounts` feature sets, and save them as a feature-retrieval spec

+> * Run a training pipeline that uses the feature retrieval spec to train a new model. This pipeline uses the built-in feature-retrieval component, to generate the training data

+> [!IMPORTANT]

+> This feature is currently in public preview. This preview version is provided without a service-level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+Before you proceed with this article, make sure you complete parts 1 and 2 of this tutorial series.

+## Set up

+1. Configure the Azure Machine Learning spark notebook

+ 1. Running the tutorial: You can create a new notebook, and execute the instructions in this document step by step. You can also open and run existing notebook `3. Experiment and train models using features.ipynb`. You can find the notebooks in the `featurestore_sample/notebooks directory`. You can select from `sdk_only`, or `sdk_and_cli`. You can keep this document open, and refer to it for documentation links and more explanation.

+ 1. Select Azure Machine Learning Spark compute in the "Compute" dropdown, located in the top nav. Wait for a status bar in the top to display "configure session".

+ 1. Configure the session:

+ * Upload the **conda.yml** file you [uploaded in Tutorial #1](./tutorial-get-started-with-feature-store.md#prepare-the-notebook-environment-for-development)

+ 1. Start the spark session

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=start-spark-session)]

+ 1. Set up the root directory for the samples

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=root-dir)]

+ ### [Python SDK](#tab/python)

+

+ Not applicable

+

+ ### [Azure CLI](#tab/cli)

+

+ Set up the CLI

+

+ 1. Install the Azure Machine Learning extension

+

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/3. Experiment and train models using features.ipynb?name=install-ml-ext-cli)]

+

+ 1. Authentication

+

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/3. Experiment and train models using features.ipynb?name=auth-cli)]

+

+ 1. Set the default subscription

+

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/3. Experiment and train models using features.ipynb?name=set-default-subs-cli)]

+

+

+1. Initialize the project workspace variables

+ This is the current workspace, and the tutorial notebook runs in this resource.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=init-ws-crud-client)]

+1. Initialize the feature store variables

+ Make sure that you update the `featurestore_name` and `featurestore_location` values shown, to reflect what you created in part 1 of this tutorial.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=init-fs-crud-client)]

+1. Initialize the feature store consumption client

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=init-fs-core-sdk)]

+1. Create a compute cluster

+ We'll create a compute cluster named `cpu-cluster` in the project workspace. We need this compute cluster when we run the training / batch inference jobs.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=create-compute-cluster)]

+## Create the accounts feature set locally

+To onboard precomputed features, you can create a feature set spec without writing any transformation code. A feature set spec is a specification that we use to develop and test a feature set, in a fully local development environment. We don't need to connect to a feature store. In this step, you create the feature set spec locally, and then sample the values from it. For managed feature store capabilities, you must use a feature asset definition to register the feature set spec with a feature store. Later steps in this tutorial provide more details.

+1. Explore the source data for the accounts

+ > [!NOTE]

+ > This notebook uses sample data hosted in a publicly-accessible blob container. Only a `wasbs` driver can read it in Spark. When you create feature sets using your own source data, please host those feature sets in an adls gen2 account, and use an `abfss` driver in the data path.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=explore-accts-fset-src-data)]

+1. Create the `accounts` feature set spec in local, from these precomputed features

+ We don't need any transformation code here, because we reference precomputed features.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=create-accts-fset-spec)]

+1. Export as a feature set spec

+ To register the feature set spec with the feature store, you must save the feature set spec in a specific format.

+ Action: After you run the next cell, inspect the generated `accounts` feature set spec. To see the spec, open the `featurestore/featuresets/accounts/spec/FeatureSetSpec.yaml` file from the file tree to see the spec.

+ The spec has these important elements:

+ 1. `source`: a reference to a storage resource, in this case, a parquet file in a blog storage resource

+

+ 1. `features`: a list of features and their datatypes. With provided transformation code (see the Day 2 section), the code must return a dataframe that maps to the features and datatypes. Without the provided transformation code (in this case, the generated `accounts` feature set spec, because it's precomputed), the system builds the query to map the features and datatypes to the source

+

+ 1. `index_columns`: the join keys required to access values from the feature set

+ See the [top level feature store entities document](./concept-top-level-entities-in-managed-feature-store.md) and the [feature set spec yaml reference](./reference-yaml-featureset-spec.md) to learn more.

+ As an extra benefit, persisting supports source control.

+ We don't need any transformation code here, because we reference precomputed features.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=dump-accts-fset-spec)]

+## Locally experiment with unregistered features

+As you develop features, you might want to locally test and validate them, before you register them with the feature store or run training pipelines in the cloud. A combination of a local unregistered feature set (`accounts`), and a feature set registered in the feature store (`transactions`), generates training data for the ML model.

+1. Select features for the model

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=select-unreg-features-for-model)]

+1. Locally generate training data

+ This step generates training data for illustrative purposes. As an option, you can locally train models here. Later steps in this tutorial explain how to train a model in the cloud.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=gen-training-data-locally)]

+1. Register the `accounts` feature set with the feature store

+ After you locally experiment with different feature definitions, and they seem reasonable, you can register a feature set asset definition with the feature store.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=reg-accts-fset)]

+1. Get the registered feature set, and sanity test it

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=sample-accts-fset-data)]

+## Run a training experiment

+In this step, you select a list of features, run a training pipeline, and register the model. You can repeat this step until the model performs as you'd like.

+1. (Optional) Discover features from the feature store UI

+ Part 1 of this tutorial covered this, when you registered the transactions feature set. Since you also have an accounts feature set, you can browse the available features:

+ * Go to the [Azure Machine Learning global landing page](https://ml.azure.com/home?flight=FeatureStores).

+ * In the left nav, select `feature stores`

+ * The list of feature stores that you can access appears. Select the feature store that you created earlier.

+ You can see the feature sets and entity that you created. Select the feature sets to browse the feature definitions. You can use the global search box to search for feature sets across feature stores.

+1. (Optional) Discover features from the SDK

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=discover-features-from-sdk)]

+1. Select features for the model, and export the model as a feature-retrieval spec

+ In the previous steps, you selected features from a combination of registered and unregistered feature sets, for local experimentation and testing. You can now experiment in the cloud. Your model shipping agility increases if you save the selected features as a feature-retrieval spec, and use the spec in the mlops/cicd flow for training and inference.

+1. Select features for the model

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=select-reg-features)]

+1. Export selected features as a feature-retrieval spec

+ > [!NOTE]

+ > A **feature retrieval spec** is a portable definition of the feature list associated with a model. It can help streamline ML model development and operationalization. It will become an input to the training pipeline which generates the training data. Then, it will be packaged with the model. The inference phase uses it to look up the features. It becomes a glue that integrates all phases of the machine learning lifecycle. Changes to the training/inference pipeline can stay at a minimum as you experiment and deploy.

+ Use of the feature retrieval spec and the built-in feature retrieval component is optional. You can directly use the `get_offline_features()` API, as shown earlier. The name of the spec should be **feature_retrieval_spec.yaml** when it's packaged with the model. This way, the system can recognize it.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=export-as-frspec)]

+## Train in the cloud with pipelines, and register the model

+In this step, you manually trigger the training pipeline. In a production scenario, a ci/cd pipeline could trigger it, based on changes to the feature-retrieval spec in the source repository. You can register the model if it's satisfactory.

+1. Run the training pipeline

+ The training pipeline has these steps:

+ 1. Feature retrieval: For its input, this built-in component takes the feature retrieval spec, the observation data, and the timestamp column name. It then generates the training data as output. It runs these steps as a managed spark job.

+

+ 1. Training: Based on the training data, this step trains the model, and then generates a model (not yet registered)

+

+ 1. Evaluation: This step validates whether or not the model performance and quality fall within a threshold (in our case, it's a placeholder/dummy step for illustration purposes)

+

+ 1. Register the model: This step registers the model

+ > [!NOTE]

+ > In part 2 of this tutorial, you ran a backfill job to materialize data for the `transactions` feature set. The feature retrieval step reads feature values from the offline store for this feature set. The behavior will be the same, even if you use the `get_offline_features()` API.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/3. Experiment and train models using features.ipynb?name=run-training-pipeline)]

+ 1. Inspect the training pipeline and the model

+ 1. Open the above pipeline, and run "web view" in a new window to see the pipeline steps.

+1. Use the feature retrieval spec in the model artifacts

+ 1. In the left nav of the current workspace, select `Models`

+ 1. Select open in a new tab or window

+ 1. Select **fraud_model**

+ 1. In the top nav, select Artifacts

+ The feature retrieval spec is packaged along with the model. The model registration step in the training pipeline handled this step. You created the feature retrieval spec during experimentation. Now it became part of the model definition. In the next tutorial, you'll see how inferencing uses it.

+## View the feature set and model dependencies

+1. View the list of feature sets associated with the model

+ In the same models page, select the `feature sets` tab. This tab shows both the `transactions` and the `accounts` feature sets on which this model depends.

+1. View the list of models that use the feature sets

+ 1. Open the feature store UI (explained earlier in this tutorial)

+ 1. Select `Feature sets` on the left nav

+ 1. Select a feature set

+ 1. Select the `Models` tab

+ You can see the list of models that use the feature sets. The feature retrieval spec determined this list when the model was registered.

+The Tutorial #4 [clean up step](./tutorial-enable-recurrent-materialization-run-batch-inference.md#cleanup) describes how to delete the resources

+* Reference: [YAML reference](./reference-yaml-overview.md)

+This tutorial series shows how features seamlessly integrate all phases of the ML lifecycle: prototyping, training and operationalization.

+Azure Machine Learning managed feature store lets you discover, create and operationalize features. The machine learning lifecycle includes a prototyping phase, where you experiment with various features. It also involves an operationalization phase, where models are deployed and inference steps look up feature data. Features serve as the connective tissue in the machine learning lifecycle. To learn more about basic feature store concepts, see [what is managed feature store](./concept-what-is-managed-feature-store.md) and [top level entities in managed feature store](./concept-top-level-entities-in-managed-feature-store.md).

+This tutorial is the first part of a four part series. Here, you'll learn how to:

+> [!div class="checklist"]

+> * Create a new minimal feature store resource

+> * Develop and locally test a feature set with feature transformation capability

+> * Register a feature store entity with the feature store

+> * Register the feature set that you developed with the feature store

+> * Generate a sample training dataframe using the features you created

+> [!IMPORTANT]

+> This feature is currently in public preview. This preview version is provided without a service-level agreement, and it's not recommended for production workloads. Certain features might not be supported, or might have constrained capabilities. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+> [!NOTE]

+> This tutorial series has two tracks:

+> * SDK only track: Uses only Python SDKs. Choose this track for pure, Python-based development and deployment.

+> * SDK & CLI track: This track uses the CLI for CRUD operations (create, update, and delete), and the Python SDK for feature set development and testing only. This is useful in CI / CD, or GitOps, scenarios, where CLI/yaml is preferred.

+* An Azure Machine Learning workspace. See [Quickstart: Create workspace resources](./quickstart-create-resources.md) article for more information about workspace creation.

+* To proceed with this article, your user account must be assigned the owner or contributor role to the resource group where the feature store is created

+ (Optional): If you use a new resource group for this tutorial, you can easily delete all the resources by deleting the resource group

+## Set up

+> [!NOTE]

+> This tutorial uses an Azure Machine Learning Spark notebook for development.

+1. In the Azure Machine Learning studio environment, first select **Notebooks** in the left nav, and then select the **Samples** tab. Navigate to the **featurestore_sample** directory

+ **Samples -> SDK v2 -> sdk -> python -> featurestore_sample**

+ and then select **Clone**, as shown in this screenshot:

+ :::image type="content" source="media/tutorial-get-started-with-feature-store/clone-featurestore-example-notebooks.png" lightbox="media/tutorial-get-started-with-feature-store/clone-featurestore-example-notebooks.png" alt-text="Screenshot showing selection of the featurestore_sample directory in Azure Machine Learning studio UI.":::

+1. The **Select target directory** panel opens next. Select the User directory, in this case **testUser**, and then select **Clone**, as shown in this screenshot:

+ :::image type="content" source="media/tutorial-get-started-with-feature-store/select-target-directory.png" lightbox="media/tutorial-get-started-with-feature-store/select-target-directory.png" alt-text="Screenshot showing selection of the target directory location in Azure Machine Learning studio UI for the featurestore_sample resource.":::

+1. To configure the notebook environment, you must upload the **conda.yml** file. Select **Notebooks** in the left nav, and then select the **Files** tab. Navigate to the **env** directory

+ **Users -> testUser -> featurestore_sample -> project -> env**

+ and select the **conda.yml** file. In this navigation, **testUser** is the user directory. Select **Download**, as shown in this screenshot:

+ :::image type="content" source="media/tutorial-get-started-with-feature-store/download-conda-file.png" lightbox="media/tutorial-get-started-with-feature-store/download-conda-file.png" alt-text="Screenshot showing selection of the conda.yml file in Azure Machine Learning studio UI.":::

+1. At the Azure Machine Learning environment, open the notebook, and select **Configure Session**, as shown in this screenshot:

+ :::image type="content" source="media/tutorial-get-started-with-feature-store/open-configure-session.png" lightbox="media/tutorial-get-started-with-feature-store/open-configure-session.png" alt-text="Screenshot showing Open Configure Session for this notebook.":::

+1. At the **Configure Session** panel, select **Python packages**. To upload the Conda file, select **Upload Conda file**, and **Browse** to the directory that hosts the Conda file. Select **conda.yml**, and then select **Open**, as shown in this screenshot:

+ :::image type="content" source="media/tutorial-get-started-with-feature-store/open-conda-file.png" lightbox="media/tutorial-get-started-with-feature-store/open-conda-file.png" alt-text="Screenshot showing the directory hosting the Conda file.":::

+1. Select **Apply**, as shown in this screenshot:

+ :::image type="content" source="media/tutorial-get-started-with-feature-store/upload-conda-file.png" lightbox="media/tutorial-get-started-with-feature-store/upload-conda-file.png" alt-text="Screenshot showing the Conda file upload.":::

+## Start the Spark session

+[!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=start-spark-session)]

+## Set up the root directory for the samples

+[!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=root-dir)]

+### [SDK Track](#tab/SDK-track)

+Not applicable

+### [SDK and CLI Track](#tab/SDK-and-CLI-track)

+### Set up the CLI

+1. Install the Azure Machine Learning extension

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/1. Develop a feature set and register with managed feature store.ipynb?name=install-ml-ext-cli)]

+1. Authentication

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/1. Develop a feature set and register with managed feature store.ipynb?name=auth-cli)]

+1. Set the default subscription

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/1. Develop a feature set and register with managed feature store.ipynb?name=set-default-subs-cli)]

+> [!NOTE]

+> Feature store Vs Project workspace: You'll use a feature store to reuse features across projects. You'll use a project workspace (an Azure Machine Learning workspace) to train and inference models, by leveraging features from feature stores. Many project workspaces can share and reuse the same feature store.

+### [SDK Track](#tab/SDK-track)

+This tutorial uses two SDKs:

+* The Feature Store CRUD SDK

+* You use the same MLClient (package name azure-ai-ml) SDK that you use with the Azure Machine Learning workspace. A feature store is implemented as a type of workspace. As a result, this SDK is used for feature store CRUD operations for feature store, feature set, and feature store entity.

+* The feature store core SDK

+

+ This SDK (azureml-featurestore) is intended for feature set development and consumption. Later steps in this tutorial describe these operations:

+

+ * Feature set specification development

+ * Feature data retrieval

+ * List and Get registered feature sets

+ * Generate and resolve feature retrieval specs

+ * Generate training and inference data using point-in-time joins

+This tutorial doesn't require explicit installation of those SDKs, because the earlier **conda YAML** instructions cover this step.

+### [SDK and CLI Track](#tab/SDK-and-CLI-track)

+This tutorial uses both the Feature store core SDK, and the CLI, for CRUD operations. It only uses the Python SDK for Feature set development and testing. This approach is useful for GitOps or CI / CD scenarios, where CLI / yaml is preferred.

+* Use the CLI for CRUD operations on feature store, feature set, and feature store entities

+* Feature store core SDK: This SDK (`azureml-featurestore`) is meant for feature set development and consumption. This tutorial covers these operations:

+ * List / Get a registered feature set

+ * Generate / resolve a feature retrieval spec

+ * Execute a feature set definition, to generate a Spark dataframe

+ * Generate training with a point-in-time join

+This tutorial doesn't need explicit installation of these resources, because the instructions cover these steps. The **conda.yaml** file includes them in an earlier step.

+## Create a minimal feature store

+1. Set feature store parameters

+ Set the name, location, and other values for the feature store

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=fs-params)]

+1. Create the feature store

+ ### [SDK Track](#tab/SDK-track)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=create-fs)]

+ ### [SDK and CLI Track](#tab/SDK-and-CLI-track)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/1. Develop a feature set and register with managed feature store.ipynb?name=create-fs-cli)]

+1. Initialize an Azure Machine Learning feature store core SDK client

+ As explained earlier in this tutorial, the feature store core SDK client is used to develop and consume features

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=init-fs-core-sdk)]

+## Prototype and develop a feature set

+We'll build a feature set named `transactions` that has rolling, window aggregate-based features

+1. Explore the transactions source data

+ > [!NOTE]

+ > This notebook uses sample data hosted in a publicly-accessible blob container. It can only be read into Spark with a `wasbs` driver. When you create feature sets using your own source data, host them in an adls gen2 account, and use an `abfss` driver in the data path.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=explore-txn-src-data)]

+1. Locally develop the feature set

+ A feature set specification is a self-contained feature set definition that you can locally develop and test. Here, we create these rolling window aggregate features:

+ * transactions three-day count

+ * transactions amount three-day sum

+ * transactions amount three-day avg

+ * transactions seven-day count

+ * transactions amount seven-day sum

+ * transactions amount seven-day avg

+ **Action:**

+ - Review the feature transformation code file: `featurestore/featuresets/transactions/transformation_code/transaction_transform.py`. Note the rolling aggregation defined for the features. This is a spark transformer.

+ See [feature store concepts](./concept-what-is-managed-feature-store.md) and **transformation concepts** to learn more about the feature set and transformations.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=develop-txn-fset-locally)]

+1. Export as a feature set spec

+ To register the feature set spec with the feature store, you must save that spec in a specific format.

+ **Action:** Review the generated `transactions` feature set spec: Open this file from the file tree to see the spec: `featurestore/featuresets/accounts/spec/FeaturesetSpec.yaml`

+ The spec contains these elements:

+

+ 1. `source`: a reference to a storage resource. In this case, it's a parquet file in a blob storage resource.

+ 1. `features`: a list of features and their datatypes. If you provide transformation code (see the Day 2 section), the code must return a dataframe that maps to the features and datatypes.

+ 1. `index_columns`: the join keys required to access values from the feature set

+ To learn more about the spec, see [top level feature store entities document](./concept-top-level-entities-in-managed-feature-store.md) and the [feature set spec yaml reference](./reference-yaml-feature-set.md).

+ Persisting the feature set spec offers another benefit: the feature set spec can be source controlled.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=dump-transactions-fs-spec)]

+## Register a feature-store entity

+As a best practice, entities help enforce use of the same join key definition across feature sets that use the same logical entities. Examples of entities can include accounts, customers, etc. Entities are typically created once, and then reused across feature sets. To learn more, see [feature store concepts](./concept-top-level-entities-in-managed-feature-store.md).

+ ### [SDK Track](#tab/SDK-track)

+ 1. Initialize the Feature Store CRUD client

+ As explained earlier in this tutorial, the MLClient is used for feature store asset CRUD (create, update, and delete). The notebook code cell sample shown here searches for the feature store we created in an earlier step. Here, we can't reuse the same ml_client we used earlier in this tutorial, because the earlier ml_client is scoped at the resource group level. Proper scoping is a prerequisite for feature store creation. In this code sample, the client is scoped at feature store level.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=init-fset-crud-client)]

+ 1. Register the `account` entity with the feature store

+ Create an account entity that has the join key `accountID`, of type string.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=register-acct-entity)]

+ ### [SDK and CLI Track](#tab/SDK-and-CLI-track)

+ 1. Initialize the Feature Store CRUD client

+ As explained earlier in this tutorial, MLClient is used for feature store asset CRUD (create, update, and delete). The notebook code cell sample shown here searches for the feature store we created in an earlier step. Here, we can't reuse the same ml_client we used earlier in this tutorial, because the earlier ml_client is scoped at the resource group level. Proper scoping is a prerequisite for feature store creation. In this code sample, the client is scoped at the feature store level, and it registers the `account` entity with the feature store. Additionally, it creates an account entity that has the join key `accountID`, of type string.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/1. Develop a feature set and register with managed feature store.ipynb?name=register-acct-entity-cli)]

+

+## Register the transaction feature set with the feature store

+First, register a feature set asset with the feature store. You can then reuse that asset, and easily share it. Feature set asset registration offers managed capabilities, including versioning and materialization. Later steps in this tutorial series cover managed capabilities.

+ ### [SDK Track](#tab/SDK-track)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=register-txn-fset)]

+ ### [SDK and CLI Track](#tab/SDK-and-CLI-track)

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_and_cli/1. Develop a feature set and register with managed feature store.ipynb?name=register-txn-fset-cli)]

+

+## Explore the feature store UI

+* From this list of accessible feature stores, select the feature store you created earlier in this tutorial.

+> [!NOTE]

+> Feature store asset creation and updates can happen only through the SDK and CLI. You can use the UI to search or browse the feature store.

+## Generate a training data dataframe using the registered feature set

+1. Load observation data

+ Observation data typically involves the core data used for training and inferencing. This data joins with the feature data to create the full training data resource. Observation data is data captured during the event itself. Here, it has core transaction data, including transaction ID, account ID, and transaction amount values. Since we use it for training, it also has an appended target variable (**is_fraud**).

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=load-obs-data)]

+1. Get the registered feature set, and list its features

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=get-txn-fset)]

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=print-txn-fset-sample-values)]

+1. Select features, and generate training data

+ Here, we select the features that become part of the training data. Then, we use the feature store SDK to generate the training data itself.

+ [!notebook-python[] (~/azureml-examples-main/sdk/python/featurestore_sample/notebooks/sdk_only/1. Develop a feature set and register with managed feature store.ipynb?name=select-features-and-gen-training-data)]

+ A point-in-time join appends the features to the training data.

+This tutorial built the training data with features from feature store. Optional: you can save the training data to storage for later use, or you can run model training on it directly.

+The Tutorial #4 [clean up step](./tutorial-enable-recurrent-materialization-run-batch-inference.md#cleanup) describes how to delete the resources

+* Reference: [YAML reference](./reference-yaml-overview.md)

+Learn how to [import your own data](how-to-designer-import-data.md) in Azure Machine Learning designer.

+- Discover Azure Migrate from Operations Manager console: Operations Manager 2022 allows you to discover Azure Migrate from console. You can now generate a complete inventory of your on-premises environment without appliance. This can be used in Azure Migrate to assess machines at scale. [Learn more](https://support.microsoft.com/topic/discover-azure-migrate-for-operations-manager-04b33766-f824-4e99-9065-3109411ede63).

+- Support for export of errors and notifications from the portal for software inventory and agentless dependency. [Learn more](troubleshoot-dependencies.md)

+ Title: Azure Database for MySQL - Flexible Server storage cops

+description: This article describes the storage IOPS in Azure Database for MySQL - Flexible Server.

+# Storage IOPS in Azure Database for MySQL - Flexible Server

+Storage IOPS (I/O Operations Per Second) refer to the number of read and write operations that can be performed by the storage system per second. Higher IOPS values indicate better storage performance, allowing your database to handle more simultaneous read and write operations, resulting in faster data retrieval and improved overall efficiency. When the IOPS setting is set too low, the database server may experience delays in processing requests, resulting in slow performance and reduced throughput. On the other hand, if the IOPS setting is set too high, it may lead to unnecessary resource allocation and potentially increased costs without significant performance improvements.

+Azure database for MySQL Flexible Server currently offers two settings for IOPS management, Pre-provisioned IOPS and Autoscale IOPS.

+## Pre-provisioned IOPS

+Azure Database for MySQL Flexible Server offers pre-provisioned IOPS, allowing you to allocate a specific number of IOPS to your MySQL database server. This setting ensures consistent and predictable performance for your workloads. With pre-provisioned IOPS, you can define a specific IOPS limit for your storage volume, guaranteeing the ability to handle a certain number of requests per second. This results in a reliable and assured level of performance.

+Moreover, Additional IOPS with pre-provisioned refers to the flexibility of increasing the provisioned IOPS for the storage volume associated with the server. You have the option to add extra IOPS beyond the default provisioned level, allowing you to customize the performance aligning with your workload requirements at any time.

+## Autoscale IOPS

+Autoscale IOPS offer the flexibility to scale IOPS on demand, eliminating the need to pre-provision a specific amount of IO per second. By enabling Autoscale IOPS, your server will automatically adjust IOPS based on workload requirements. With the Autoscale IOPS featured enable, you can now enjoy worry free IO management in Azure Database for MySQL - Flexible Server because the server scales IOPs up or down automatically depending on workload needs.ΓÇ»

+With this feature, you'll only be charged for the IO your server actually utilizes, avoiding unnecessary provisioning and expenses for underutilized resources. This ensures both cost savings and optimal performance, making it a smart choice for managing your database workload efficiently.

+## Monitor Storage performance

+Monitoring Storage IOPS utilization is easy with [Metrics available under Monitoring](./concepts-monitoring.md#list-of-metrics) .

+#### Overview

+To obtain a comprehensive view of the IO utilization for the selected time period.

+Navigate to the Monitoring in the Azure portal for Azure Database for MySQL Flexible Server under the Overview blade.

+[:::image type="content" source="./media/concepts-storage-iops/1-overview.png" alt-text="Screenshot of overview metrics.":::](./media/concepts-storage-iops/1-overview.png#lightbox)

+#### Enhanced Metrics Workbook

+- Navigate to Workbooks under Monitoring section on your Azure portal.

+- Select "Enhanced Metrics" workbook.

+- Check for Storage IO Percentage metrics under Overview section of the workbook.

+[:::image type="content" source="./media/concepts-storage-iops/2-workbook.png" alt-text="Screenshot of enhanced metrics.":::](./media/concepts-storage-iops/2-workbook.png#lightbox)

+#### Metrics under Monitoring

+- Navigate to Metrics, under Monitoring section on your Azure portal.

+- Select "Add metric" option.

+- Choose ΓÇ£Storage IO PercentΓÇ¥ from the drop-down of available metrics.

+- Choose "Storage IO count" from the drop-down of available metrics.

+[:::image type="content" source="./media/concepts-storage-iops/3-metrics.png" alt-text="Screenshot of monitoring metrics.":::](./media/concepts-storage-iops/3-metrics.png#lightbox)

+## Selecting the Optimal IOPS Setting

+Having learned how to monitor your IOPS usage effectively, you're now equipped to explore the best settings for your server. When choosing the IOPS setting for your Azure Database for MySQL Flexible Server, several important factors should be considered. Understanding these factors can help you make an informed decision to ensure the best performance and cost-efficiency for your workload.

+### Performance Optimization

+With Autoscale IOPS, consistent requirements can be met for workload, which is predictable without facing the drawback of storage throttling and manual interaction to add more IOPS.

+If your workload has consistent throughput or requires consistent IOPS, Pre-provisioned IOPS may be preferable. It provides a predictable performance level, and the fixed allocation of IOPS correlates with workload within the specified limits.

+Although for any requirement of higher throughput from usual requirement, Additional IOPS can be allotted with Pre-provisioned IOPS, which requires manual interaction and understanding of throughput increase time.

+### Throttling impact

+Consider the impact of throttling on your workload. If the potential performance degradation due to throttling is a concern, Autoscale IOPS can dynamically handle workload spikes, minimizing the risk of throttling and maintaining performance to optimal level.

+

+Ultimately, the decision between Autoscale and Pre-provisioned IOPS depends on your specific workload requirements and performance expectations. Analyze your workload patterns, evaluate the cost implications, and consider the potential impact of throttling to make an informed choice that aligns with your priorities.

+By considering the specific characteristics of your database workload, such as traffic fluctuations, query patterns, and performance requirements, you can make an informed decision regarding the choice between Autoscale and Pre-provisioned IOPS.

+| **Workload Considerations** | **Pre-Provisioned IOPS** | **Autoscale IOPS** |

+||||

+| Workloads with consistent and predictable I/O patterns | Recommended as it utilizes only provisioned IOPS | Compatible, no manual provisioning of IOPS required |

+| Workloads with varying usage patterns | Not Recommended as it may not provide efficient performance during high usage periods. | Recommended as it automatically adjusts to handle varying workloads |

+| Workloads with dynamic growth or changing performance need | Not recommended as it requires constant adjustments as per changing IOPS requirement | Recommended as no extra settings is required for specific throughput requirement |

+### Cost considerations

+If you have a fluctuating workload with unpredictable peaks, opting for Autoscale IOPS may be more cost-effective. It ensures that you only pay for the higher IOPS used during peak periods, offering flexibility and cost savings. Pre-provisioned IOPS, while providing consistent and max IOPS, may come at a higher cost depending on the workload. Consider the trade-off between cost and performance required from your server.

+### Test and Evaluate

+If unsure about the optimal IOPS setting, consider running performance tests using both Autoscale IOPS and Pre-provisioned IOPS. Assess the results and determine which setting meets your workload requirements and performance expectations.

+**Example workloads: E-commerce websites**

+If you own an e-commerce website that experiences fluctuations in traffic throughout the year. During normal periods, the workload is moderate, but during holiday seasons or special promotions, the traffic surges exponentially.

+Autoscale IOPS: With Autoscale IOPS, your database can dynamically adjust its IOPS to handle the increased workload during peak periods. When traffic spikes, such as during Black Friday sales, the auto scale feature allows your database to seamlessly scale up the IOPS to meet the demand. This ensures smooth and uninterrupted performance, preventing slowdowns or service disruptions. After the peak period, when the traffic subsides, the IOPS scale back down, allowing for cost savings as you only pay for the resources utilized during the surge.

+

+Pre-provisioned IOPS: If you opt for pre-provisioned IOPS, you need to estimate the maximum workload capacity and allocate a fixed number of IOPS accordingly. However, during peak periods, the workload might exceed the predetermined IOPS limit. As a result, the storage I/O could throttle, impacting performance and potentially causing delays or timeouts for your users.

+**Example workloads: Reporting /Data Analytics Platforms**

+Suppose you have Azure Database for MySQL Flexible Server used for data analytics where users submit complex queries and large-scale data processing tasks.

+The workload pattern is relatively consistent, with a steady flow of queries throughout the day.

+Pre-provisioned IOPS: With pre-provisioned IOPS, you can select a suitable number of IOPS based on the expected workload. As long as the chosen IOPS adequately handle the daily query volume, there's no risk of throttling or performance degradation. This approach provides cost predictability and allows you to optimize resources efficiently without the need for dynamic scaling.

+Autoscale IOPS: The Autoscale feature might not provide significant advantages in this case. Since the workload is consistent, the database can be provisioned with a fixed number of IOPS that comfortably meets the demand. Autoscaling might not be necessary as there are no sudden bursts of activity that require additional IOPS. By using Pre-provisioned IOPS, you have predictable performance without the need for scaling, and the cost is directly tied to the allocated storage.

+## Frequent Asked Questions

+#### How to move from pre-provisioned IOPS to Autoscale IOPS?

+- Access your Azure portal and locate the relevant Azure database for MySQL Flexible Server.

+- Go to the Settings blade and choose the Compute + Storage section.

+- Within the IOPS section, opt for Auto Scale IOPS and save the settings to apply the modifications.

+#### How soon does Autoscale IOPS take effect after making the change?

+Once you enable Autoscale IOPS for your Azure database for MySQL Flexible Server and save the settings, the changes take effect immediately after the deployment to the resource has completed successfully. This means that the Autoscale IOPS feature will be applied to your database without any delay.

+#### How to know when IOPS have scaled up and scaled down when the server is using Autoscale IOPS feature? Or Can I monitor IOPS usage for my server?

+Refer to [ΓÇ£Monitor Storage performanceΓÇ¥](#monitor-storage-performance) section, which will help to identify if your server has scaled up or scaled down during specific time window.

+#### Can I switch between Autoscale IOPS and pre-provisioned IOPS later?

+Yes, you can move back to pre-provisioned IOPS by opting for pre-provisioned IOPS under Compute + Storage section under Settings blade.

+#### How do I know how much IOPS have been utilized for Azure database for MySQL Flexible Server?

+My navigating to Monitoring under Overview section. Or navigate to [IO count metrics](./concepts-monitoring.md#list-of-metrics) under Monitoring blade. IO count metric gives sum of IOPS used by server in the selected timeframe.

+## Next steps

+- Learn more about [service limitations](./concepts-limitations.md).

+- Learn more about [Pricing](./concepts-service-tiers-storage.md#pricing) information.

+## July 2023

+- **Autoscale IOPS in Azure Database for MySQL - Flexible Server (General Availability)**

+You can now scale IOPS on demand without having to pre-provision a certain amount of IOPS. With this feature, you can now enjoy worry free IO management in Azure Database for MySQL - Flexible Server because the server scales IOPs up or down automatically depending on workload needs. With this feature, you pay only for the IO you use and no longer need to provision and pay for resources they aren't fully using, saving time and money. Autoscale IOPS eliminates the administration required to provide the best performance for Azure Database for MySQL customers at the least cost. [Learn more](./concepts-service-tiers-storage.md#autoscale-iops)

+| Degraded | Your NAT gateway resource has platform or user initiated events impacting the health of your NAT gateway. The metric for the data-path availability has reported less than 80% but greater than 25% health for the last 15 minutes. You'll experience moderate to severe performance impact. |

+## Resource health alerts

+Azure Resource Health alerts can notify you in near real-time when the health state of your NAT gateway resource changes. It's recommended that you set resource health alerts to notify you when your NAT gateway resource is in a **Degraded** or **Unavailable** state.

+When you create Azure resource health alerts for NAT gateway, Azure sends resource health notifications to your Azure subscription. You can create and customize alerts based on:

+* The subscription affected

+* The resource group affected

+* The resource type affected (Microsoft.Network/NATGateways)

+* The specific resource (any NAT gateway resource you choose to set up an alert for)

+* The event status of the NAT gateway resource affected

+* The current status of the NAT gateway resource affected

+* The previous status of the NAT gateway resource affected

+* The reason type of the NAT gateway resource affected

+You can also configure who the alert should be sent to:

+* A new action group (that can be used for future alerts)

+* An existing action group

+For more information on how to set up these resource health alerts, see:

+* [Resource health alerts using Azure portal](/azure/service-health/resource-health-alert-monitor-guide#create-a-resource-health-alert-rule-in-the-azure-portal)

+* [Resource health alerts using Resource Manager templates](/azure/service-health/resource-health-alert-arm-template-guide)

+> * Create a virtual network

+> * Enable flow logging for a network security group using Network Watcher flow logs

+## Create a virtual network

+In this section, you create **myVNet** virtual network with one subnet for the virtual machine.

+1. In the search box at the top of the portal, enter *virtual networks*. Select **Virtual networks** from the search results.

+ | Region | Select **(US) East US**. |

+1. In the search box at the top of the portal, enter *virtual machines*. Select **Virtual machines** from the search results.

+1. Select **+ Create** and then select **Azure virtual machine**.

+1. In **Create a virtual machine**, enter or select the following values in the **Basics** tab:

+1. Select the **Networking** tab, or select **Next: Disks**, then **Next: Networking**.

+1. In the Networking tab, select the following values:

+ | Public IP | Select **(new) myVM-ip**. |

+ | Public inbound ports | Select **Allow selected ports**. |

+ | Select inbound ports | Select **RDP (3389)**. |

+ > [!CAUTION]

+ > Leaving the RDP port open to the internet is only recommended for testing. For production environments, it's recommended to restrict access to the RDP port to a specific IP address or range of IP addresses. You can also block internet access to the RDP port and use [Azure Bastion](../bastion/bastion-overview.md) to securely connect to your virtual machine from the Azure portal.

+1. Select **Review + create**.

+1. Review the settings, and then select **Create**.

+1. Once the deployment is complete, select **Go to resource** to go to the **Overview** page of **myVM**.

+1. Select **Connect** then select **RDP**.

+1. Select **Download RDP File** and open the downloaded file.

+1. Select **Connect** and then enter the username and password that you created in the previous steps. Accept the certificate if prompted.

+ | Redundancy | Select **Locally-redundant storage (LRS)** or different replication strategy that matches your durability requirements. |

+1. Go back to your RDP session with **myVM** virtual machine.

+1. Open Microsoft Edge and go to `www.bing.com`.

+ ```

+ https://{storageAccountName}.blob.core.windows.net/insights-logs-networksecuritygroupflowevent/resourceId=/SUBSCRIPTIONS/{subscriptionID}/RESOURCEGROUPS/{resourceGroupName}/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/{networSecurityGroupName}/y={year}/m={month}/d={day}/h={hour}/m=00/macAddress={acAddress}/PT1H.json

+ ```

+**Delete the resource group**:

+1. In the search box at the top of the portal, enter ***myResourceGroup***. Select **myResourceGroup** from the search results.

+1. Select **Delete resource group**.

+1. In **Delete a resource group**, enter ***myResourceGroup***, and then select **Delete**.

+1. Select **Delete** to confirm the deletion of the resource group and all its resources.

+**Delete the flow log**:

+1. In the search box at the top of the portal, enter ***network watcher***. Select **Network Watcher** from the search results.

+1. Under **Logs**, select **Flow logs**.

+1. In **Network Watcher | Flow logs**, select the checkbox of the flow log.

+1. Select **Delete**.

+ Title: Delete Azure Private 5G Core resources

+description: In this how-to guide, you'll learn how to delete all Azure Private 5G Core resources.

+# Delete Azure Private 5G Core resources

+In this how-to guide, you'll learn how to delete all resources associated with Azure Private 5G Core (AP5GC). This includes Azure Stack Edge (ASE) resources that are required to deploy AP5GC. You should do this only when advised by your Microsoft support representative; for example, if your deployment has encountered an unrecoverable error.

+If you want to delete your entire AP5GC deployment, you must complete all sections of this guide in order or you may be left with resources that cannot be deleted without intervention from Microsoft. You can also follow one or more sections to delete a subset of the resources in your deployment.

+If you want to move resources instead, see [Move your private mobile network resources to a different region](region-move-private-mobile-network-resources.md).

+> [!CAUTION]

+> This procedure will destroy your AP5GC deployment. You will lose all data that isn't backed up. Do not delete resources that are in use.

+## Prerequisites

+- Ensure you can sign in to the Azure portal using an account with access to the active subscription you used to create your private mobile network. This account must have the built-in Contributor or Owner role at the subscription scope.

+- Make a note of the resource group that contains your private mobile network, which was collected in [Collect the required information to deploy a private mobile network](collect-required-information-for-private-mobile-network.md).

+- Make a note of the resource group that contains your Azure Stack Edge and custom location resources.

+## Back up deployment information

+All data will be lost when deleting your deployment. Back up any information you'd like to preserve. You can use this information to help set up a new deployment.

+1. Refer to [Collect the required information for your SIMs](provision-sims-azure-portal.md#collect-the-required-information-for-your-sims) to take a backup of all the information you'll need to recreate your SIMs.

+1. Depending on your authentication method when signing in to the [distributed tracing](distributed-tracing.md) and [packet core dashboards](packet-core-dashboards.md):

+ - If you use Azure AD, save a copy of the Kubernetes Secret Object YAML file you created in [Create Kubernetes Secret Objects](enable-azure-active-directory.md#create-kubernetes-secret-objects).

+ - If you use local usernames and passwords and want to keep using the same credentials, save a copy of the current passwords to a secure location.

+1. If you want to retain any traces, [export and save](distributed-tracing-share-traces.md#export-trace-from-the-distributed-tracing-web-gui) them securely before continuing.

+1. Refer to [Exporting a dashboard](https://grafana.com/docs/grafana/v6.1/reference/export_import/#exporting-a-dashboard) in the Grafana documentation to save a backed-up copy of your dashboards.

+## Delete private mobile network resources

+The private mobile network resources represent your private 5G core network. If you followed the recommendations in this documentation when creating your resources, you should have a single resource group containing all private mobile network resources. You must ensure that you do not delete any unrelated resources.

+> [!IMPORTANT]

+> Deleting this resource group will delete the resources for all sites in your deployment. If you only want to delete a single site, see [Delete sites using the Azure portal](delete-a-site.md). You can can then return to this procedure to delete the custom location, delete the AKS cluster and reset ASE if required.

+1. Sign in to the [Azure portal](https://portal.azure.com/).

+1. Search for and select the resource group containing the private mobile network resources.

+1. Select **Delete resource group**. You will be prompted to enter the resource group name to confirm deletion.

+1. Select **Yes** when prompted to delete the resource group.

+## Delete the custom location

+The custom location resource represents the physical location of the hardware that runs the packet core software.

+1. Navigate to the resource group containing the **Custom location** resource.

+1. Select the tick box for the **Custom location** resource and select **Delete**.

+1. Confirm the deletion.

+If you are deleting multiple sites, repeat this step for each site.

+## Delete the AKS cluster

+The Azure Kubernetes Service (AKS) cluster is an orchestration layer used to manage the packet core software components. To delete the Azure Kubernetes Service (AKS) connected cluster, follow [Remove the Azure Kubernetes service](https://learn.microsoft.com/azure/databox-online/azure-stack-edge-deploy-aks-on-azure-stack-edge#remove-the-azure-kubernetes-service).

+If you are deleting multiple sites, repeat this step for each site.

+## Reset ASE

+ Azure Stack Edge (ASE) hardware runs the packet core software at the network edge. To reset your ASE device, follow [Reset and reactivate your Azure Stack Edge device](https://learn.microsoft.com/azure/databox-online/azure-stack-edge-reset-reactivate-device).

+If you are deleting multiple sites, repeat this step for each site.

+## Next steps

+To create a new AP5GC deployment, refer to [Commission the AKS cluster](commission-cluster.md) and [Deploy a private mobile network through Azure Private 5G Core - Azure portal](how-to-guide-deploy-a-private-mobile-network-azure-portal.md).

+Once you have created a new deployment, complete the following steps to restore the data you backed up in [Back up deployment information](#back-up-deployment-information).

+1. Retrieve your backed-up SIM information and recreate your SIMs by following one of:

+ - [Provision new SIMs for Azure Private 5G Core - Azure portal](provision-sims-azure-portal.md)

+ - [Provision new SIMs for Azure Private 5G Core - ARM template](provision-sims-arm-template.md)

+1. Depending on your authentication method when signing in to the [distributed tracing](distributed-tracing.md) and [packet core dashboards](packet-core-dashboards.md):

+ - If you use Azure AD, [reapply the Secret Object for distributed tracing and the packet core dashboards](enable-azure-active-directory.md#apply-kubernetes-secret-objects).

+ - If you use local usernames and passwords, follow [Access the distributed tracing web GUI](distributed-tracing.md#access-the-distributed-tracing-web-gui) and [Access the packet core dashboards](packet-core-dashboards.md#access-the-packet-core-dashboards) to restore access to your local monitoring tools.

+1. If you backed up any packet core dashboards, follow [Importing a dashboard](https://grafana.com/docs/grafana/v6.1/reference/export_import/#importing-a-dashboard) in the Grafana documentation to restore them.

+| Azure Web Apps - Azure Function Apps (Microsoft.Web/sites) / sites | privatelink.azurewebsites.net </br> scm.privatelink.azurewebsites.net | azurewebsites.net </br> scm.azurewebsites.net |

+[Azure Virtual Machines](reliability-virtual-machines.md)|

+ Title: Reliability in Azure Virtual Machines

+description: Find out about reliability in Azure Virtual Machines

+# Reliability in Virtual Machines

+This article contains [specific reliability recommendations for Virtual Machines](#reliability-recommendations), as well as detailed information on VM regional resiliency with [availability zones](#availability-zone-support) and [cross-region resiliency with disaster recovery](#disaster-recovery-cross-region-failover).

+For an architectural overview of reliability in Azure, see [Azure reliability](/azure/architecture/framework/resiliency/overview).

+## Reliability recommendations

+

+### Reliability recommendations summary

+| Category | Priority |Recommendation |

+||--||

+| [**High Availability**](#high-availability) |:::image type="icon" source="media/icon-recommendation-high.svg":::| [VM-1: Run production workloads on two or more VMs using Azure Virtual Machine Scale Sets(VMSS) Flex](#-vm-1-run-production-workloads-on-two-or-more-vms-using-vmss-flex) |

+||:::image type="icon" source="media/icon-recommendation-high.svg"::: |[VM-2: Deploy VMs across availability zones or use VMSS Flex with zones](#-vm-2-deploy-vms-across-availability-zones-or-use-vmss-flex-with-zones) |

+||:::image type="icon" source="media/icon-recommendation-high.svg":::|[VM-3: Migrate VMs using availability sets to VMSS Flex](#-vm-3-migrate-vms-using-availability-sets-to-vmss-flex) |

+||:::image type="icon" source="media/icon-recommendation-high.svg"::: |[VM-5: Use managed disks for VM disks](#-vm-5-use-managed-disks-for-vm-disks)|

+|[**Disaster Recovery**](#disaster-recovery)| :::image type="icon" source="media/icon-recommendation-medium.svg"::: |[ VM-4: Replicate VMs using Azure Site Recovery](#-vm-4-replicate-vms-using-azure-site-recovery) |

+||:::image type="icon" source="media/icon-recommendation-medium.svg"::: |[VM-7: Backup data on your VMs with Azure Backup service](#-vm-7-backup-data-on-your-vms-with-azure-backup-service) |

+|[**Performance**](#performance) |:::image type="icon" source="media/icon-recommendation-low.svg"::: | [VM-6: Host application and database data on a data disk](#-vm-6-host-application-and-database-data-on-a-data-disk)|

+||:::image type="icon" source="media/icon-recommendation-high.svg"::: | [VM-8: Production VMs should be using SSD disks](#-vm-8-production-vms-should-be-using-ssd-disks)|

+||:::image type="icon" source="media/icon-recommendation-medium.svg"::: |[VM-10: Enable Accelerated Networking (AccelNet)](#-vm-10-enable-accelerated-networking-accelnet) |

+||:::image type="icon" source="media/icon-recommendation-low.svg"::: |[VM-11: Accelerated Networking is enabled, make sure you update the GuestOS NIC driver every 6 months](#-vm-11-when-accelnet-is-enabled-you-must-manually-update-the-guestos-nic-driver) |

+|[**Management**](#management)|:::image type="icon" source="media/icon-recommendation-low.svg"::: |[VM-9: Watch for VMs in Stopped state](#-vm-9-review-vms-in-stopped-state) |

+||:::image type="icon" source="media/icon-recommendation-high.svg"::: |[VM-22: Use maintenance configurations for the VM](#-vm-22-use-maintenance-configurations-for-the-vm) |

+|[**Security**](#security)|:::image type="icon" source="media/icon-recommendation-medium.svg"::: |[VM-12: VMs should not have a Public IP directly associated](#-vm-12-vms-should-not-have-a-public-ip-directly-associated) |

+||:::image type="icon" source="media/icon-recommendation-low.svg"::: |[VM-13: Virtual Network Interfaces have an NSG associated](#-vm-13-vm-network-interfaces-have-a-network-security-group-nsg-associated) |

+||:::image type="icon" source="media/icon-recommendation-medium.svg"::: |[VM-14: IP Forwarding should only be enabled for Network Virtual Appliances](#-vm-14-ip-forwarding-should-only-be-enabled-for-network-virtual-appliances) |

+||:::image type="icon" source="media/icon-recommendation-low.svg"::: |[VM-17: Network access to the VM disk should be set to "Disable public access and enable private access"](#-vm-17-network-access-to-the-vm-disk-should-be-set-to-disable-public-access-and-enable-private-access) |

+||:::image type="icon" source="media/icon-recommendation-medium.svg"::: |[VM-19: Enable disk encryption and data at rest encryption by default](#-vm-19-enable-disk-encryption-and-data-at-rest-encryption-by-default) |

+|[**Networking**](#networking) | :::image type="icon" source="media/icon-recommendation-low.svg"::: |[VM-15: Customer DNS Servers should be configured in the Virtual Network level](#-vm-15-dns-servers-should-be-configured-in-the-virtual-network-level) |

+|[**Storage**](#storage) |:::image type="icon" source="media/icon-recommendation-medium.svg"::: |[VM-16: Shared disks should only be enabled in clustered servers](#-vm-16-shared-disks-should-only-be-enabled-in-clustered-servers) |

+|[**Compliance**](#compliance)| :::image type="icon" source="media/icon-recommendation-low.svg"::: |[VM-18: Ensure that your VMs are compliant with Azure Policies](#-vm-18-ensure-that-your-vms-are-compliant-with-azure-policies) |

+|[**Monitoring**](#monitoring)| :::image type="icon" source="media/icon-recommendation-low.svg"::: |[VM-20: Enable VM Insights](#-vm-20-enable-vm-insights) |

+||:::image type="icon" source="media/icon-recommendation-low.svg"::: |[VM-21: Configure diagnostic settings for all Azure resources](#-vm-21-configure-diagnostic-settings-for-all-azure-resources) |

+### High availability

+

+#### :::image type="icon" source="media/icon-recommendation-high.svg"::: **VM-1: Run production workloads on two or more VMs using VMSS Flex**

+To safeguard application workloads from downtime due to the temporary unavailability of a disk or VM, it's recommended that you run production workloads on two or more VMs using VMSS Flex.

+To achieve this you can use:

+- [Azure Virtual Machine Scale Sets](/azure/virtual-machine-scale-sets/overview) to create and manage a group of load balanced VMs. The number of VM instances can automatically increase or decrease in response to demand or a defined schedule.

+- **Availability zones**. For more information on availability zones and VMs, see [Availability zone support](#availability-zone-support).

+# [Azure Resource Graph](#tab/graph)

+-

+#### :::image type="icon" source="media/icon-recommendation-high.svg"::: **VM-2: Deploy VMs across availability zones or use VMSS Flex with zones**

+

+When you create your VMs, use availability zones to protect your applications and data against unlikely datacenter failure. For more information about availability zones for VMs, see [Availability zone support](#availability-zone-support) in this document.

+For information on how to enable availability zones support when you create your VM, see [create availability zone support](#create-a-resource-with-availability-zone-enabled).

+For information on how to migrate your existing VMs to availability zone support, see [Availability zone support redeployment and migration](#availability-zone-redeployment-and-migration).

+# [Azure Resource Graph](#tab/graph)

+-

+#### :::image type="icon" source="media/icon-recommendation-high.svg"::: **VM-3: Migrate VMs using availability sets to VMSS Flex**

+Availability sets will be retired in the near future. Modernize your workloads by migrating them from VMs to VMSS Flex.

+With VMSS Flex, you can deploy your VMs in one of two ways:

+- Across zones

+- In the same zone, but across fault domains (FDs) and update domains (UD) automatically.

+In an N-tier application, it's recommended that you place each application tier into its own VMSS Flex.

+# [Azure Resource Graph](#tab/graph)

+-

+#### :::image type="icon" source="media/icon-recommendation-high.svg"::: **VM-5: Use managed disks for VM disks**

+To provide better reliability for VMs in an availability set, use managed disks. Managed disks are sufficiently isolated from each other to avoid single points of failure. Also, managed disks arenΓÇÖt subject to the IOPS limits of VHDs created in a storage account.

+# [Azure Resource Graph](#tab/graph)

+### Disaster recovery

+#### :::image type="icon" source="media/icon-recommendation-low.svg"::: **VM-4: Replicate VMs using Azure Site Recovery**

+When you replicate Azure VMs using Site Recovery, all the VM disks are continuously replicated to the target region asynchronously. The recovery points are created every few minutes. This gives you a Recovery Point Objective (RPO) in the order of minutes. You can conduct disaster recovery drills as many times as you want, without affecting the production application or the ongoing replication.

+To learn how to run a disaster recovery drill, see [Run a test failover](/azure/site-recovery/site-recovery-test-failover-to-azure).

+# [Azure Resource Graph](#tab/graph)

+#### :::image type="icon" source="media/icon-recommendation-medium.svg"::: **VM-7: Backup data on your VMs with Azure Backup service**

+The Azure Backup service provides simple, secure, and cost-effective solutions to back up your data and recover it from the Microsoft Azure cloud. For more information, see [What is the Azure Backup Service](/azure/backup/backup-overview).

+# [Azure Resource Graph](#tab/graph)

+### Performance

+#### :::image type="icon" source="media/icon-recommendation-low.svg"::: **VM-6: Host application and database data on a data disk**

+A data disk is a managed disk thatΓÇÖs attached to a VM. Use the data disk to store application data, or other data you need to keep. Data disks are registered as SCSI drives and are labeled with a letter that you choose. Hosting your data on a data disk makes it easy to backup or restore your data. You can also migrate the disk without having to move the entire VM and Operating System. Also, you'll be able to select a different disk SKU, with different type, size, and performance that meet your requirements. For more information on data disks, see [Data Disks](/azure/virtual-machines/managed-disks-overview#data-disk).

+# [Azure Resource Graph](#tab/graph)

+#### :::image type="icon" source="media/icon-recommendation-high.svg"::: **VM-8: Production VMs should be using SSD disks**

+Premium SSD disks offer high-performance, low-latency disk support for I/O-intensive applications and production workloads. Standard SSD Disks are a cost-effective storage option optimized for workloads that need consistent performance at lower IOPS levels.

+It is recommended that you:

+- Use Standard HDD disks for Dev/Test scenarios and less critical workloads at lowest cost.

+- Use Premium SSD disks instead of Standard HDD disks with your premium-capable VMs. For any Single Instance VM using premium storage for all Operating System Disks and Data Disks, Azure guarantees VM connectivity of at least 99.9%.

+If you want to upgrade from Standard HDD to Premium SSD disks, consider the following issues:

+- Upgrading requires a VM reboot and this process takes 3-5 minutes to complete.

+- If VMs are mission-critical production VMs, evaluate the improved availability against the cost of premium disks.

+

+For more information on Azure managed disks and disks types, see [Azure managed disk types](/azure/virtual-machines/disks-types#premium-ssd).

+# [Azure Resource Graph](#tab/graph)

+### :::image type="icon" source="media/icon-recommendation-medium.svg"::: **VM-10: Enable Accelerated Networking (AccelNet)**

+AccelNet enables single root I/O virtualization (SR-IOV) to a VM, greatly improving its networking performance. This high-performance path bypasses the host from the data path, which reduces latency, jitter, and CPU utilization for the most demanding network workloads on supported VM types.

+For more information on Accelerated Networking, see [Accelerated Networking](/azure/virtual-network/accelerated-networking-overview?tabs=redhat.)

+# [Azure Resource Graph](#tab/graph)

+#### :::image type="icon" source="media/icon-recommendation-low.svg"::: **VM-11: When AccelNet is enabled, you must manually update the GuestOS NIC driver**

+When AccelNet is enabled, the default Azure Virtual Network interface in the GuestOS is replaced for a Mellanox interface. As a result, the GuestOS NIC driver is provided from Mellanox, a 3rd party vendor. Although Marketplace images maintained by Microsoft are offered with the latest version of Mellanox drivers, once the VM is deployed, you'll need to manually update GuestOS NIC driver every six months.

+# [Azure Resource Graph](#tab/graph)

+### Management

+#### :::image type="icon" source="media/icon-recommendation-low.svg"::: **VM-9: Review VMs in stopped state**

+VM instances go through different states, including provisioning and power states. If a VM is in a stopped state, the VM may be facing an issue or is no longer necessary and could be removed to help reduce costs.

+# [Azure Resource Graph](#tab/graph)

+#### :::image type="icon" source="media/icon-recommendation-high.svg"::: **VM-22: Use maintenance configurations for the VM**

+To ensure that VM updates/interruptions are done in a planned time frame, use maintenance configuration settings to schedule and manage updates. For more information on managing VM updates with maintenance configurations, see [Managing VM updates with Maintenance Configurations](../virtual-machines/maintenance-configurations.md).

+# [Azure Resource Graph](#tab/graph)

+### Security

+#### :::image type="icon" source="media/icon-recommendation-medium.svg"::: **VM-12: VMs should not have a Public IP directly associated**

+If a VM requires outbound internet connectivity, it's recommended that you use NAT Gateway or Azure Firewall. NAT Gateway or Azure Firewall help to increase security and resiliency of the service, since both services have much higher availability and [Source Network Address Translation (SNAT)](/azure/load-balancer/load-balancer-outbound-connections) ports. For inbound internet connectivity, it's recommended that you use a load balancing solution such as Azure Load Balancer and Application Gateway.

+# [Azure Resource Graph](#tab/graph)

+#### :::image type="icon" source="media/icon-recommendation-low.svg"::: **VM-13: VM network interfaces have a Network Security Group (NSG) associated**

+It's recommended that you associate a NSG to a subnet, or a network interface, but not both. Since rules in a NSG associated to a subnet can conflict with rules in a NSG associated to a network interface, you can have unexpected communication problems that require troubleshooting. For more information, see [Intra-Subnet traffic](/azure/virtual-network/network-security-group-how-it-works#intra-subnet-traffic).

+# [Azure Resource Graph](#tab/graph)

+#### :::image type="icon" source="media/icon-recommendation-medium.svg"::: **VM-14: IP forwarding should only be enabled for network virtual appliances**

+IP forwarding enables the virtual machine network interface to:

+- Receive network traffic not destined for one of the IP addresses assigned to any of the IP configurations assigned to the network interface.

+- Send network traffic with a different source IP address than the one assigned to one of a network interfaceΓÇÖs IP configurations.

+The IP forwarding setting must be enabled for every network interface that's attached to the VM receiving traffic to be forwarded. A VM can forward traffic whether it has multiple network interfaces, or a single network interface attached to it. While IP forwarding is an Azure setting, the VM must also run an application that's able to forward the traffic, such as firewall, WAN optimization, and load balancing applications.

+To learn how to enable or disable IP forwarding, see [Enable or disable IP forwarding](/azure/virtual-network/virtual-network-network-interface?tabs=azure-portal#enable-or-disable-ip-forwarding).

+# [Azure Resource Graph](#tab/graph)

+#### :::image type="icon" source="media/icon-recommendation-low.svg"::: **VM-17: Network access to the VM disk should be set to "Disable public access and enable private access"**

+It's recommended that you set VM disk network access to ΓÇ£Disable public access and enable private accessΓÇ¥ and create a private endpoint. To learn how to create a private endpoint, see [Create a private endpoint](/azure/virtual-machines/disks-enable-private-links-for-import-export-portal#create-a-private-endpoint).

+# [Azure Resource Graph](#tab/graph)

+#### :::image type="icon" source="media/icon-recommendation-medium.svg"::: **VM-19: Enable disk encryption and data at rest encryption by default**

+There are several types of encryption available for your managed disks, including Azure Disk Encryption (ADE), Server-Side Encryption (SSE) and encryption at host.

+- Azure Disk Encryption helps protect and safeguard your data to meet your organizational security and compliance commitments.

+- Azure Disk Storage Server-Side Encryption (also referred to as encryption-at-rest or Azure Storage encryption) automatically encrypts data stored on Azure managed disks (OS and data disks) when persisting on the Storage Clusters.

+- Encryption at host ensures that data stored on the VM host hosting your VM is encrypted at rest and flows encrypted to the Storage clusters.

+- Confidential disk encryption binds disk encryption keys to the VMΓÇÖs TPM and makes the protected disk content accessible only to the VM.

+For more information about managed disk encryption options, see [Overview of managed disk encryption options](../virtual-machines/disk-encryption-overview.md).

+# [Azure Resource Graph](#tab/graph)

+### Networking

+#### :::image type="icon" source="media/icon-recommendation-low.svg"::: **VM-15: DNS Servers should be configured in the Virtual Network level**

+Configure the DNS Server in the Virtual Network to avoid name resolution inconsistency across the environment. For more information on name resolution for resources in Azure virtual networks, see [Name resolution for VMs and cloud services](/azure/virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances?tabs=redhat).

+# [Azure Resource Graph](#tab/graph)

+### Storage

+#### :::image type="icon" source="media/icon-recommendation-medium.svg"::: **VM-16: Shared disks should only be enabled in clustered servers**

+Azure shared disks is a feature for Azure managed disks that enables you to attach a managed disk to multiple VMs simultaneously. Attaching a managed disk to multiple VMs allows you to either deploy new or migrate existing clustered applications to Azure and should only be used in those situations where the disk will be assigned to more than one VM member of a cluster.

+To learn more about how to enable shared disks for managed disks, see [Enable shared disk](/azure/virtual-machines/disks-shared-enable?tabs=azure-portal).

+# [Azure Resource Graph](#tab/graph)

+### Compliance

+#### :::image type="icon" source="media/icon-recommendation-low.svg"::: **VM-18: Ensure that your VMs are compliant with Azure Policies**

+ItΓÇÖs important to keep your virtual machine (VM) secure for the applications that you run. Securing your VMs can include one or more Azure services and features that cover secure access to your VMs and secure storage of your data. For more information on how to keep your VM and applications secure, see [Azure Policy Regulatory Compliance controls for Azure Virtual Machines](/azure/virtual-machines/security-controls-policy).

+# [Azure Resource Graph](#tab/graph)

+### Monitoring

+#### :::image type="icon" source="media/icon-recommendation-low.svg"::: **VM-20: Enable VM Insights**

+Enable [VM Insights](/azure/azure-monitor/vm/vminsights-overview) to get more visibility into the health and performance of your virtual machine. VM Insights gives you information on the performance and health of your VMs and virtual machine scale sets, by monitoring their running processes and dependencies on other resources. VM Insights can help deliver predictable performance and availability of vital applications by identifying performance bottlenecks and network issues. Insights can also help you understand whether an issue is related to other dependencies.

+# [Azure Resource Graph](#tab/graph)

+#### :::image type="icon" source="media/icon-recommendation-low.svg"::: **VM-21: Configure diagnostic settings for all Azure resources**

+Platform metrics are sent automatically to Azure Monitor Metrics by default and without configuration. Platform logs provide detailed diagnostic and auditing information for Azure resources and the Azure platform they depend on and are one of the following types:

+- **Resource logs** that arenΓÇÖt collected until theyΓÇÖre routed to a destination.

+- **Activity logs** that exist on their own but can be routed to other locations.

+Each Azure resource requires its own diagnostic setting, which defines the following criteria:

+- **Sources** The type of metric and log data to send to the destinations defined in the setting. The available types vary by resource type.

+- **Destinations**: One or more destinations to send to.

+A single diagnostic setting can define no more than one of each of the destinations. If you want to send data to more than one of a particular destination type (for example, two different Log Analytics workspaces), create multiple settings. Each resource can have up to five diagnostic settings.

+Fore information, see [Diagnostic settings in Azure Monitor](/azure/azure-monitor/essentials/diagnostic-settings?tabs=portal).

+# [Azure Resource Graph](#tab/graph)

+## Availability zone support

+Virtual machines support availability zones with three availability zones per supported Azure region and are also zone-redundant and zonal. For more information, see [availability zones support](availability-zones-service-support.md). The customer will be responsible for configuring and migrating their virtual machines for availability. Refer to the following readiness options below for availability zone enablement:

+- See [availability options for VMs](../virtual-machines/availability.md)

+- Review [availability zone service and region support](availability-zones-service-support.md)

+- [Migrate existing VMs](migrate-vm.md) to availability zones

+

+### Prerequisites

+- Your virtual machine SKUs must be available across the zones in for your region. To review which regions support availability zones, see the [list of supported regions](availability-zones-service-support.md#azure-regions-with-availability-zone-support).

+- Your VM SKUs must be available across the zones in your region. To check for VM SKU availability, use one of the following methods:

+ - Use PowerShell to [Check VM SKU availability](../virtual-machines/windows/create-PowerShell-availability-zone.md#check-vm-sku-availability).

+ - Use the Azure CLI to [Check VM SKU availability](../virtual-machines/linux/create-cli-availability-zone.md#check-vm-sku-availability).

+ - Go to [Foundational Services](availability-zones-service-support.md#an-icon-that-signifies-this-service-is-foundational-foundational-services).

+

+### SLA improvements

+Because availability zones are physically separate and provide distinct power source, network, and cooling, SLAs (Service-level agreements) increase. For more information, see the [SLA for Virtual Machines](https://azure.microsoft.com/support/legal/sla/virtual-machines/v1_9/).

+#### Create a resource with availability zone enabled

+Get started by creating a virtual machine (VM) with availability zone enabled from the following deployment options below:

+- [Azure CLI](../virtual-machines/linux/create-cli-availability-zone.md)

+- [PowerShell](../virtual-machines/windows/create-powershell-availability-zone.md)

+- [Azure portal](../virtual-machines/create-portal-availability-zone.md)

+### Zonal failover support

+Customers can set up virtual machines to failover to another zone using the Site Recovery service. For more information, see [Site Recovery](../site-recovery/site-recovery-overview.md).

+### Fault tolerance

+Virtual machines can failover to another server in a cluster, with the VM's operating system restarting on the new server. Customers should refer to the failover process for disaster recovery, gathering virtual machines in recovery planning, and running disaster recovery drills to ensure their fault tolerance solution is successful.

+For more information, see the [site recovery processes](../site-recovery/site-recovery-failover.md#before-you-start).

+### Zone down experience

+During a zone-wide outage, you should expect a brief degradation of performance until the virtual machine service self-healing re-balances underlying capacity to adjust to healthy zones. This isn't dependent on zone restoration; it's expected that the Microsoft-managed service self-healing state will compensate for a lost zone, leveraging capacity from other zones.

+Customers should also prepare for the possibility that there's an outage of an entire region. If there's a service disruption for an entire region, the locally redundant copies of your data would temporarily be unavailable. If geo-replication is enabled, three additional copies of your Azure Storage blobs and tables are stored in a different region. In the event of a complete regional outage or a disaster in which the primary region isn't recoverable, Azure remaps all of the DNS entries to the geo-replicated region.

+#### Zone outage preparation and recovery

+The following guidance is provided for Azure virtual machines in the case of a service disruption of the entire region where your Azure virtual machine application is deployed:

+- Configure [Azure Site Recovery](/azure/virtual-machines/virtual-machines-disaster-recovery-guidance#option-1-initiate-a-failover-by-using-azure-site-recovery) for your VMs

+- Check the [Azure Service Health Dashboard](/azure/virtual-machines/virtual-machines-disaster-recovery-guidance#option-2-wait-for-recovery) status if Azure Site Recovery hasn't been configured

+- Review how the [Azure Backup service](../backup/backup-azure-vms-introduction.md) works for VMs

+ - See the [support matrix](../backup/backup-support-matrix-iaas.md) for Azure VM backups

+- Determine which [VM restore option and scenario](../backup/about-azure-vm-restore.md) will work best for your environment

+### Low-latency design

+Cross Region (secondary region), Cross Subscription (preview), and Cross Zonal (preview) are available options to consider when designing a low-latency virtual machine solution. For more information on these options, see the [supported restore methods](../backup/backup-support-matrix-iaas.md#supported-restore-methods).

+>[!IMPORTANT]

+>By opting out of zone-aware deployment, you forego protection from isolation of underlying faults. Use of SKUs that don't support availability zones or opting out from availability zone configuration forces reliance on resources that don't obey zone placement and separation (including underlying dependencies of these resources). These resources shouldn't be expected to survive zone-down scenarios. Solutions that leverage such resources should define a disaster recovery strategy and configure a recovery of the solution in another region.

+### Safe deployment techniques

+When you opt for availability zones isolation, you should utilize safe deployment techniques for application code, as well as application upgrades. In addition to configuring Azure Site Recovery, below are recommended safe deployment techniques for VMs:

+- [Virtual Machine Scale Sets](/azure/virtual-machines/flexible-virtual-machine-scale-sets)

+- [Azure Load Balancer](../load-balancer/load-balancer-overview.md)

+- [Azure Storage Redundancy](../storage/common/storage-redundancy.md)

+ As Microsoft periodically performs planned maintenance updates, there may be rare instances when these updates require a reboot of your virtual machine to apply the required updates to the underlying infrastructure. To learn more, see [availability considerations](../virtual-machines/maintenance-and-updates.md#availability-considerations-during-scheduled-maintenance) during scheduled maintenance.

+Follow the health signals below for monitoring before upgrading your next set of nodes in another zone:

+- Check the [Azure Service Health Dashboard](https://azure.microsoft.com/status/) for the virtual machines service status for your expected regions

+- Ensure that [replication](../site-recovery/azure-to-azure-quickstart.md) is enabled on your VMs

+### Availability zone redeployment and migration

+For migrating existing virtual machine resources to a zone redundant configuration, refer to the below resources:

+- Move a VM to another subscription or resource group

+ - [CLI](/azure/azure-resource-manager/management/move-resource-group-and-subscription#use-azure-cli)

+ - [PowerShell](/azure/azure-resource-manager/management/move-resource-group-and-subscription#use-azure-powershell)

+- [Azure Resource Mover](/azure/resource-mover/tutorial-move-region-virtual-machines)

+- [Move Azure VMs to availability zones](../site-recovery/move-azure-vms-avset-azone.md)

+- [Move region maintenance configuration resources](../virtual-machines/move-region-maintenance-configuration-resources.md)

+## Disaster recovery: cross-region failover

+In the case of a region-wide disaster, Azure can provide protection from regional or large geography disasters with disaster recovery by making use of another region. For more information on Azure disaster recovery architecture, see [Azure to Azure disaster recovery architecture](../site-recovery/azure-to-azure-architecture.md).

+Customers can use Cross Region to restore Azure VMs via paired regions. You can restore all the Azure VMs for the selected recovery point if the backup is done in the secondary region. For more details on Cross Region restore, refer to the Cross Region table row entry in our [restore options](../backup/backup-azure-arm-restore-vms.md#restore-options).

+### Cross-region disaster recovery in multi-region geography

+While Microsoft is working diligently to restore the virtual machine service for region-wide service disruptions, customers will have to rely on other application-specific backup strategies to achieve the highest level of availability. For more information, see the section on [Data strategies for disaster recovery](/azure/architecture/reliability/disaster-recovery#disaster-recovery-plan).

+#### Outage detection, notification, and management

+When the hardware or the physical infrastructure for the virtual machine fails unexpectedly. This can include local network failures, local disk failures, or other rack level failures. When detected, the Azure platform automatically migrates (heals) your virtual machine to a healthy physical machine in the same data center. During the healing procedure, virtual machines experience downtime (reboot) and in some cases loss of the temporary drive. The attached OS and data disks are always preserved.

+For more detailed information on virtual machine service disruptions, see [disaster recovery guidance](/azure/virtual-machines/virtual-machines-disaster-recovery-guidance).

+#### Set up disaster recovery and outage detection

+When setting up disaster recovery for virtual machines, understand what [Azure Site Recovery provides](../site-recovery/site-recovery-overview.md#what-does-site-recovery-provide). Enable disaster recovery for virtual machines with the below methods:

+- Set up disaster recovery to a [secondary Azure region for an Azure VM](../site-recovery/azure-to-azure-quickstart.md)

+- Create a Recovery Services vault

+ - [Bicep](../site-recovery/quickstart-create-vault-bicep.md)

+ - [ARM template](../site-recovery/quickstart-create-vault-template.md)

+- Enable disaster recovery for [Linux virtual machines](../virtual-machines/linux/tutorial-disaster-recovery.md)

+- Enable disaster recovery for [Windows virtual machines](../virtual-machines/windows/tutorial-disaster-recovery.md)

+- Failover virtual machines to [another region](../site-recovery/azure-to-azure-tutorial-failover-failback.md)

+- Failover virtual machines to the [primary region](../site-recovery/azure-to-azure-tutorial-failback.md#fail-back-to-the-primary-region)

+### Single-region geography disaster recovery

+With disaster recovery set up, Azure VMs will continuously replicate to a different target region. If an outage occurs, you can fail over VMs to the secondary region, and access them from there.

+When you replicate Azure VMs using [Site Recovery](../site-recovery/site-recovery-overview.md), all the VM disks are continuously replicated to the target region asynchronously. The recovery points are created every few minutes. This gives you a Recovery Point Objective (RPO) in the order of minutes. You can conduct disaster recovery drills as many times as you want, without affecting the production application or the ongoing replication. For more information, see [Run a disaster recovery drill to Azure](../site-recovery/tutorial-dr-drill-azure.md).

+For more information, see [Azure VMs architectural components](../site-recovery/azure-to-azure-architecture.md#architectural-components) and [region pairing](../virtual-machines/regions.md#region-pairs).

+### Capacity and proactive disaster recovery resiliency

+Microsoft and its customers operate under the Shared Responsibility Model. This means that for customer-enabled DR (customer-responsible services), the customer must address DR for any service they deploy and control. To ensure that recovery is proactive, customers should always pre-deploy secondaries because there's no guarantee of capacity at time of impact for those who haven't pre-allocated.

+For deploying virtual machines, customers can use [flexible orchestration](../virtual-machine-scale-sets/virtual-machine-scale-sets-orchestration-modes.md#scale-sets-with-flexible-orchestration) mode on Virtual Machine Scale Sets. All VM sizes can be used with flexible orchestration mode. Flexible orchestration mode also offers high availability guarantees (up to 1000 VMs) by spreading VMs across fault domains in a region or within an Availability Zone.

+## Additional guidance

+- [Well-Architected Framework for virtual machines](/azure/architecture/framework/services/compute/virtual-machines/virtual-machines-review)

+- [Azure to Azure disaster recovery architecture](/azure/site-recovery/azure-to-azure-architecture)

+- [Accelerated networking with Azure VM disaster recovery](/azure/site-recovery/azure-vm-disaster-recovery-with-accelerated-networking)

+- [Express Route with Azure VM disaster recovery](../site-recovery/azure-vm-disaster-recovery-with-expressroute.md)

+- [Virtual Machine Scale Sets](../virtual-machine-scale-sets/index.yml)

+## Next steps

+> [!div class="nextstepaction"]

+> [Resiliency in Azure](/azure/reliability/availability-zones-overview)

+### Azure China Commercial Marketplace

+

+To learn which commercial marketplace features are available for Azure China Marketplace operated by 21Vianet, as compared to the Azure global commercial marketplace, see [Feature availability for Azure China Commercial Marketplace operated by 21Vianet](/partner-center/marketplace/azure-in-china-feature-availability).

+The [Azure Retail Prices API for China](/rest/api/cost-management/retail-prices/azure-retail-prices-china) article is applicable only to Azure in China and isn't available in Azure Global.

+The [Markup - China](../cost-management-billing/manage/markup-china.md) article is applicable only to Azure China and isn't available in Azure Global.

+> Looking for a Cognitive Search solution with ChatGPT interaction? See [this demo](https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/README.md) for details.

+This article explains how to configure private, outbound calls from Azure Cognitive Search to an Azure PaaS resource that runs within a virtual network.

+Setting up a private connection allows a search service to connect to a virtual network IP address instead of a port that's open to the internet. The object created for the connection is called a *shared private link*. On the connection, Search uses the shared private link internally to reach an Azure PaaS resource inside the network boundary.

+Shared private link is a premium feature that's billed by usage. When you set up a shared private link, charges for the private endpoint are added to your Azure invoice. As you use the shared private link, data transfer rates for inbound and outbound access are also invoiced. For details, see [Azure Private Link pricing](https://azure.microsoft.com/pricing/details/private-link/).

+**Information** (properties.incidentType == Informational)

+## Authorize access and connect to data resources

+To work with the code examples in this article, you need to create an authorized [DataLakeServiceClient](/dotnet/api/azure.storage.files.datalake.datalakeserviceclient) instance that represents the storage account. You can authorize a `DataLakeServiceClient` object using Azure Active Directory (Azure AD), an account access key, or a shared access signature (SAS).

+### [Azure AD](#tab/azure-ad)

+To learn more about using `DefaultAzureCredential` to authorize access to data, see [How to authenticate .NET applications with Azure services](/dotnet/azure/sdk/authentication#defaultazurecredential).

+### [SAS token](#tab/sas-token)

+To use a shared access signature (SAS) token, provide the token as a string and initialize a [DataLakeServiceClient](/dotnet/api/azure.storage.files.datalake.datalakeserviceclient) object. If your account URL includes the SAS token, omit the credential parameter.

+To learn more about generating and managing SAS tokens, see the following article:

+- [Grant limited access to Azure Storage resources using shared access signatures (SAS)](../common/storage-sas-overview.md?toc=/azure/storage/blobs/toc.json)

+### [Account key](#tab/account-key)

+A container acts as a file system for your files. You can create a container by using the following method:

+- [DataLakeServiceClient.CreateFileSystem](/dotnet/api/azure.storage.files.datalake.datalakeserviceclient.createfilesystemasync)

+This example creates a container and returns a [DataLakeFileSystemClient](/dotnet/api/azure.storage.files.datalake.datalakefilesystemclient) object for later use:

+You can create a directory reference in the container by using the following method:

+- [DataLakeFileSystemClient.CreateDirectoryAsync](/dotnet/api/azure.storage.files.datalake.datalakefilesystemclient.createdirectoryasync)

+The following code example adds a directory to a container, then adds a subdirectory and returns a [DataLakeDirectoryClient](/dotnet/api/azure.storage.files.datalake.datalakedirectoryclient) object for later use:

+You can rename or move a directory by using the following method:

+- [DataLakeDirectoryClient.RenameAsync](/dotnet/api/azure.storage.files.datalake.datalakedirectoryclient.renameasync)

+Pass the path of the desire directory as a parameter. The following code example shows how to rename a subdirectory:

+The following code example shows how to move a subdirectory from one directory to a different directory:

+## Upload a file to a directory

+You can upload content to a new or existing file by using the following method:

+- [DataLakeFileClient.UploadAsync](/dotnet/api/azure.storage.files.datalake.datalakefileclient.uploadasync)

+The following code example shows how to upload a local file to a directory using the `UploadAsync` method:

+You can use this method to create and upload content to a new file, or you can set the `overwrite` parameter to `true` to overwrite an existing file.

+## Append data to a file

+You can upload data to be appended to a file by using the following method:

+- [DataLakeFileClient.AppendAsync](/dotnet/api/azure.storage.files.datalake.datalakefileclient.appendasync)

+The following code example shows how to append data to the end of a file using these steps:

+- Create a [DataLakeFileClient](/dotnet/api/azure.storage.files.datalake.datalakefileclient) object to represent the file resource you're working with.

+- Upload data to the file using the [DataLakeFileClient.AppendAsync](/dotnet/api/azure.storage.files.datalake.datalakefileclient.appendasync) method.

+- Complete the upload by calling the [DataLakeFileClient.FlushAsync](/dotnet/api/azure.storage.files.datalake.datalakefileclient.flushasync) method to write the previously uploaded data to the file.

+## Download from a directory

+The following code example shows how to download a file from a directory to a local file using these steps:

+- Create a [DataLakeFileClient](/dotnet/api/azure.storage.files.datalake.datalakefileclient) instance to represent the file that you want to download.

+- Use the [DataLakeFileClient.ReadAsync](/dotnet/api/azure.storage.files.datalake.datalakefileclient.readasync) method, then parse the return value to obtain a [Stream](/dotnet/api/system.io.stream) object. Use any .NET file processing API to save bytes from the stream to a file.

+This example uses a [BinaryReader](/dotnet/api/system.io.binaryreader) and a [FileStream](/dotnet/api/system.io.filestream) to save bytes to a file.

+## List directory contents

+You can list directory contents by using the following method and enumerating the result:

+- [FileSystemClient.GetPathsAsync](/dotnet/api/azure.storage.files.datalake.datalakefilesystemclient.getpathsasync)

+Enumerating the paths in the result may make multiple requests to the service while fetching the values.

+The following code example prints the names of each file that is located in a directory:

+## Delete a directory

+You can delete a directory by using the following method:

+- [DataLakeDirectoryClient.Delete](/dotnet/api/azure.storage.files.datalake.datalakedirectoryclient.delete)

+The following code example shows how to delete a directory:

+## Restore a soft-deleted directory

+You can use the Azure Storage client libraries to restore a soft-deleted directory. Use the following method to list deleted paths for a [DataLakeFileSystemClient](/dotnet/api/azure.storage.files.datalake.datalakefilesystemclient) instance:

+- [GetDeletedPathsAsync](/dotnet/api/azure.storage.files.datalake.datalakefilesystemclient.getdeletedpathsasync)

+Use the following method to restore a soft-deleted directory:

+- [UndeletePathAsync](/dotnet/api/azure.storage.files.datalake.datalakefilesystemclient.undeletepathasync)

+The following code example shows how to list deleted paths and restore a soft-deleted directory:

+If you rename the directory that contains the soft-deleted items, those items become disconnected from the directory. If you want to restore those items, you have to revert the name of the directory back to its original name or create a separate directory that uses the original directory name. Otherwise, you receive an error when you attempt to restore those soft-deleted items.

+az k8s-extension create --cluster-type managedClusters --cluster-name <cluster name> --resource-group <resource group name> --name <name of extension> --extension-type microsoft.azurecontainerstorage --scope cluster --release-train stable --release-namespace acstor

+For code samples using the latest version 12.x client library version, see [Quickstart: Azure Queue Storage client library for .NET](storage-quickstart-queues-dotnet.md).

+## Create a Queue Storage client

+For code samples using the latest version 12.x client library version, see [Quickstart: Azure Queue Storage client library for Python](storage-quickstart-queues-python.md).

+## Create a queue

+For code samples using the latest version 12.x client library version, see [Quickstart: Azure Queue Storage client library for Java](storage-quickstart-queues-java.md).

+## Create a queue

+Synapse Studio creates a new session for each SQL script execution. Once a SQL script execution completes, the session is automatically closed.

+Temporary tables are only visible to the session in which they were created and are automatically dropped when the session closes.

+For complete syntax, refer to [CREATE EXTERNAL TABLE AS SELECT (Transact-SQL)](/sql/t-sql/statements/create-external-table-as-select-transact-sql).

+ Title: An overview of dynamic scoping (preview)

+description: This article provides information about dynamic scoping (preview), its purpose and advantages.

+# About Dynamic Scoping (preview)

+**Applies to:** :heavy_check_mark: Windows VMs :heavy_check_mark: Linux VMs :heavy_check_mark: On-premises environment :heavy_check_mark: Azure VMs :heavy_check_mark: Azure Arc-enabled servers.

+Dynamic scoping (preview) is an advanced capability of schedule patching that allows users to:

+- Group machines based on criteria such as subscription, resource group, location, resource type, OS Type, and Tags. This becomes the definition of the scope.

+- Associate the scope to a schedule/maintenance configuration to apply updates at scale as per a pre-defined scope.

+The criteria will be evaluated at the scheduled run time, which will be the final list of machines that will be patched by the schedule. The machines evaluated during create or edit phase may differ from the group at schedule run time.

+## Key benefits

+

+**At Scale and simplified patching** - You don't have to manually change associations between machines and schedules. For example, if you want to remove a machine from a schedule and your scope was defined based on tag(s) criteria, removing the tag on the machine will automatically drop the association. These associations can be dropped and added for multiple machines at scale.

+ > [!NOTE]

+ > Subscription is mandatory for the creation of dynamic scope and you can't edit it after the dynamic scope is created.

+**Reusability of the same schedule** - You can associate a schedule to multiple machines dynamically, statically, or both.

+ > [!NOTE]

+ > You can associate one dynamic scope to one schedule.

+## Permissions

+For dynamic scoping (preview) and configuration assignment, ensure that you have the following permissions:

+- Write permissions to create or modify a schedule.

+- Read permissions to assign or read a schedule.

+## Prerequisites for Azure VMs

+- Patch Orchestration must be set to Customer Managed Schedules (Preview). This sets patch mode to AutomaticByPlatform and the **BypassPlatformSafetyChecksOnUserSchedule** = *True*.

+- Associate a Schedule with the VM.

+> [!NOTE]

+> For Arc VMs, there are no patch orchestration pre-requisites. However, you must associate a schedule with the VM for Schedule patching. For more information, see [Configure schedule patching on Azure VMs to ensure business continuity](prerequsite-for-schedule-patching.md).

+## Next steps

+ Learn about deploying updates to your machines to maintain security compliance by reading [deploy updates](deploy-updates.md)

+ Title: Manage various operations of dynamic scoping (preview).

+description: This article describes how to manage dynamic scoping (preview) operations

+# Manage a Dynamic scope

+**Applies to:** :heavy_check_mark: Windows VMs :heavy_check_mark: Linux VMs :heavy_check_mark: On-premises environment :heavy_check_mark: Azure Arc-enabled servers.

+This article describes how to view, add, edit and delete a dynamic scope (preview).

+## Add a Dynamic scope (preview)

+To add a Dynamic scope to an existing configuration, follow these steps:

+1. Sign in to the [Azure portal](https://portal.azure.com) and navigate to Update management center (preview).

+1. Select **Machines** > **Browse maintenance configurations** > **Maintenance configurations**.

+1. In the **Maintenance configurations** page, select the name of the maintenance configuration for which you want to add a Dynamic scope.

+1. In the given maintenance configuration page > select **Dynamic scopes** > **Add a dynamic scope**.

+1. In the **Add a dynamic scope** page, select **subscriptions**(mandatory).

+1. In **Filter by**, choose **Select** and in the **Select Filter by**, specify the Resource group, Resource type, Location, Tags and OS type and then select **Ok**. These filters are optional fields.

+1. In the **Preview of machines based on above scope**, you can view the list of machines for the selected criteria and then select **Add**.

+ > [!NOTE]

+ > The list of machines may be different at run time.

+1. In the **Configure Azure VMs for schedule updates** page, select any one of the following options to provide your consent:

+ 1. **Change the required options to ensure schedule supportability** - this option confirms that you want to update the patch orchestration from existing option to *Customer Managed Schedules (Preview)*: This updates the following two properties on your behalf:

+

+ - *Patch mode = AutomaticByPlatform*

+ - *Set the BypassPlatformSafetyChecksOnUserSchedule = True*.

+ 1. **Continue with supported machines only** - this option confirms that you want to proceed with only the machines that already have patch orchestration set to *Customer Managed Schedules (Preview)*.

+

+ > [!NOTE]

+ > In the **Preview of machines based on above scope** page, you can view only the machines that don't have patch orchestration set to *Customer Managed Schedules (Preview)*.

+1. Select **Save** to go back to the Dynamic scopes tab. In this tab, you can view and edit the Dynamic scope that you have created.

+## View Dynamic scope (preview)

+To view the list of Dynamic scopes (preview) associated to a given maintenance configuration, follow these steps:

+1. Sign in to the [Azure portal](https://portal.azure.com) and navigate to **Update management center (preview)**.

+1. Select **Machines** > **Browse maintenance configurations** > **Maintenance configurations**.

+1. In the **Maintenance configurations** page, select the name of the maintenance configuration for which you want to view the Dynamic scope.

+1. In the given maintenance configuration page, select **Dynamic scopes** to view all the Dynamic scopes that are associated with the maintenance configuration.

+## Edit a Dynamic scope (preview)

+1. Sign in to the [Azure portal](https://portal.azure.com) and navigate to Update management center (preview).

+1. Select **Machines** > **Browse maintenance configurations** > **Maintenance configurations**.

+1. In the **Maintenance configurations** page, select the name of the maintenance configuration for which you want to edit an existing Dynamic scope.

+1. In the given maintenance configuration page > select **Dynamic scopes** and select the scope you want to edit. Under **Actions** column, select the edit icon.

+1. In the **Edit Dynamic scope**, select the edit icon in the **Filter By** to edit the filters as needed and select **Ok**.

+ > [!NOTE]

+ > Subscription is mandatory for the creation of dynamic scope and you can't edit it after the dynamic scope is created.

+1. Select **Save**.

+## Delete a Dynamic scope (preview)

+1. Sign in to the [Azure portal](https://portal.azure.com) and navigate to Update management center (preview).

+1. Select **Machines** > **Browse maintenance configurations** > **Maintenance configurations**.

+1. In the **Maintenance configurations** page, select the name of the maintenance configuration for which you want to edit an existing Dynamic scope.

+1. In the given maintenance configuration page > select **Dynamic scopes** and select the scope you want to delete. Select **Remove dynamic scope** and then select **Ok**.

+## View patch history of a Dynamic scope (preview)

+1. Sign in to the [Azure portal](https://portal.azure.com) and navigate to Update management center (preview).

+1. Select **History** > **Browse maintenance configurations** > **Maintenance configurations** to view the patch history of a dynamic scope.

+## Provide consent to apply updates

+Obtaining consent to apply updates is an important step in the workflow of dynamic scoping and listed are the various ways to provide consent.

+#### [From Virtual Machine](#tab/vm)

+1. In [Azure portal](https://portal.azure.com), go to **+Create a resource** > **Virtual machine** > **Create**.

+1. In **Create a virtual machine**, select **Management** tab and under the **Guest OS Updates**, in **Patch orchestration options**, you can do the following:

+ 1. Select **Azure-orchestrated with user managed schedules (Preview)** to confirm that:

+ - Patch Orchestration is set to *Azure orchestration*

+ - Set the Bypass platform safety checks on user schedule = *True*.

+ The selection allows you to provide consent to apply the update settings, ensures that auto patching isn't applied and that patching on the VM(s) runs as per the schedule you've defined.

+1. Complete the details under **Monitoring**, **Advanced** and **Tags** tabs.

+1. Select **Review + Create** and under the **Management** you can view the values as **Periodic assessment** - *Off* and **Patch orchestration options** - *Azure-orchestrated with user managed schedules (Preview)*.

+1. Select **Create**.

+

+#### [From Schedule updates tab](#tab/sc)

+1. Follow the steps from 1 to 5 listed in [Add a Dynamic scope (preview)](#add-a-dynamic-scope-preview).

+1. In **Machines** tab, select **Add machine**, In **Select resources** page, select the machines and select **Add**

+1. In **Configure Azure VMs for schedule updates**, select **Continue to schedule updates** option to confirm that:

+ - Patch Orchestration is set to *Azure orchestration*

+ - Set the Bypass platform safety checks on user schedule = *True*.

+1. Select **Continue to schedule updates** to update the patch mode as **Azure-orchestrated** and enable the scheduled patching for the VMs after obtaining the consent.

+#### [From Update Settings](#tab/us)

+1. In **Update management center**, go to **Overview** > **Update settings**.

+1. In **Change Update settings**, select **+Add machine** to add the machines.

+1. In the list of machines sorted as per the operating system, go to the **Patch orchestration** option and select **Azure-orchestrated with user managed schedules (Preview)** to confirm that:

+ - Patch Orchestration is set to *Azure orchestration*

+ - Set the Bypass platform safety checks on user schedule = *True*

+1. Select **Save**.

+ The selection made in this workflow automatically applies the update settings and no consent is explicitly obtained.

+## Next steps

+* [View updates for single machine](view-updates.md)

+* [Deploy updates now (on-demand) for single machine](deploy-updates.md)

+* [Schedule recurring updates](scheduled-patching.md)

+* [Manage update settings via Portal](manage-update-settings.md)

+* [Manage multiple machines using update management center](manage-multiple-machines.md)

+ Title: Schedule updates on Dynamic scoping (preview).

+description: In this tutorial, you learn how to group machines, dynamically apply the updates at scale.

+#Customer intent: As an IT admin, I want dynamically apply patches on the machines as per a schedule.

+# Tutorial: Schedule updates on Dynamic scopes (preview)

+**Applies to:** :heavy_check_mark: Windows VMs :heavy_check_mark: Linux VMs :heavy_check_mark: On-premises environment :heavy_check_mark: Azure VMs :heavy_check_mark: Azure Arc-enabled servers.

+

+This tutorial explains how you can create a dynamic scope, and apply patches based on the criteria.

+In this tutorial, you learn how to:

+> [!div class="checklist"]

+> - Create and edit groups

+> - Associate a schedule

+If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.

+## Prerequisites

+- Patch Orchestration must be set to Customer Managed Schedules (Preview). This sets patch mode to AutomaticByPlatform and the **BypassPlatformSafetyChecksOnUserSchedule** = *True*.

+- Associate a Schedule with the VM.

+## Create a Dynamic scope

+To create a dynamic scope, follow the steps:

+1. Sign in to the [Azure portal](https://portal.azure.com) and navigate to Update management center (preview).

+1. Select **Overview** > **Schedule updates** > **Create a maintenance configuration**.

+1. In the **Create a maintenance configuration** page, enter the details in the **Basics** tab and select **Maintenance scope** as *Guest* (Azure VM, Arc-enabled VMs/servers).

+1. Select **Dynamic Scopes** and follow the steps to [Add Dynamic scope](manage-dynamic-scoping.md#add-a-dynamic-scope-preview).

+1. In **Machines** tab, select **Add machines** to add any individual machines to the maintenance configuration and select **Updates**.

+1. In the **Updates** tab, select the patch classification that you want to include/exclude and select **Tags**.

+1. Provide the tags in **Tags** tab.

+1. Select **Review** and then **Review + Create**.

+>[!NOTE]

+> A dynamic scope exists within the context of a schedule only. You can use one schedule to link to a machine, dynamic scope, or both. One dynamic scope cannot have more than one schedule.

+## Provide the consent

+Obtaining consent to apply updates is an important step in the workflow of scheduled patching and follow the steps on various ways to [provide the consent](manage-dynamic-scoping.md#provide-consent-to-apply-updates).

+## Next steps

+Learn about [managing multiple machines](manage-multiple-machines.md).

+

+## July 2023

+### Dynamic scope (preview)

+Dynamic scope (preview) is an advanced capability of schedule patching. You can now create a group of [machines based on a schedule and apply patches](dynamic-scope-overview.md) on those machines at scale. [Learn more](tutorial-dynamic-grouping-for-scheduled-patching.md).

+

+The limit on the number of subscriptions that you can manage to use the Update management center (preview) portal has now been removed. You can now manage all your subscriptions using the update management center (preview) portal.

+ > [!NOTE]

+ > The host pool name must not contain spaces.

+ Call redirection only affects the connection between the local client device and the telephony app server, as shown in the following diagram.

+ :::image type="content" source="media/multimedia-redirection-intro/call-redirection.png" alt-text="A diagram depicting the relationship between the telephony web app server, the Azure Virtual Desktop user, the web app, and other callers." lightbox="media/multimedia-redirection-intro/call-redirection.png":::

+ Call redirection offloads WebRTC calls from session hosts to local client devices to reduce latency and improve call quality. However, after the connection is established, call quality becomes dependent on the website or app providers just as it would with a non-redirected call.

+- Fixed an issue where the client doesn't auto-reconnect when the Gateway WebSocket connection shuts down normally.

+ Title: We're retiring Azure VMs (classic) on September 6, 2023

+# Migrate your IaaS resources to Azure Resource Manager by September 6, 2023

+In 2014, we launched infrastructure as a service (IaaS) on [Azure Resource Manager](https://azure.microsoft.com/features/resource-manager/). We've been enhancing capabilities ever since. Because Azure Resource Manager now has full IaaS capabilities and other advancements, we deprecated the management of IaaS virtual machines (VMs) through [Azure Service Manager](./migration-classic-resource-manager-faq.yml) (ASM) on February 28, 2020. This functionality will be fully retired on September 6, 2023.

+Today, about 90 percent of the IaaS VMs are using Azure Resource Manager. If you use IaaS resources through ASM, start planning your migration now. Complete it by September 6, 2023, to take advantage of [Azure Resource Manager](../azure-resource-manager/management/index.yml).

+- On September 6, 2023, customers will no longer be able to start IaaS VMs by using ASM. Any that are still running or allocated will be stopped and deallocated.

+- On September 6, 2023, subscriptions that are not migrated to Azure Resource Manager will be informed regarding timelines for deleting any remaining VMs (classic).

+ Title: 'Quickstart: Create a Linux VM cluster in Azure using Terraform'

+description: In this article, you learn how to create a Linux VM cluster in Azure using Terraform

+content_well_notification:

+ - AI-contribution

+# Quickstart: Create a Linux VM cluster in Azure using Terraform

+**Applies to:** :heavy_check_mark: Linux VMs

+This article shows you how to create a Linux VM cluster (containing two Linux VM instances) in Azure using Terraform.

+In this article, you learn how to:

+> [!div class="checklist"]

+> * Create a random value for the Azure resource group name using [random_pet](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/pet).

+> * Create an Azure resource group using [azurerm_resource_group](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group).

+> * Create a virtual network using [azurerm_virtual_network](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/virtual_network)

+> * Create a subnet using [azurerm_subnet](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet)

+> * Create a public IP using [azurerm_public_ip](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/public_ip)

+> * Create a load balancer using [azurerm_lb](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/lb)

+> * Create a load balancer address pool using [azurerm_lb_backend_address_pool](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/lb_backend_address_pool)

+> * Create a network interface using [azurerm_network_interface](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/network_interface)

+> * Create a managed disk using [azurerm_managed_disk](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/managed_disk)

+> * Create a availability set using [azurerm_availability_set](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/availability_set)

+> * Create a Linux virtual machine using [azurerm_linux_virtual_machine](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/linux_virtual_machine)

+> * Create an AzAPI resource [azapi_resource](https://registry.terraform.io/providers/Azure/azapi/latest/docs/resources/azapi_resource).

+> * Create an AzAPI resource to generate an SSH key pair using [azapi_resource_action](https://registry.terraform.io/providers/Azure/azapi/latest/docs/resources/azapi_resource_action).

+## Prerequisites

+- [Install and configure Terraform](/azure/developer/terraform/quickstart-configure)

+## Implement the Terraform code

+> [!NOTE]

+> The sample code for this article is located in the [Azure Terraform GitHub repo](https://github.com/Azure/terraform/tree/master/quickstart/101-vm-cluster-linux). You can view the log file containing the [test results from current and previous versions of Terraform](https://github.com/Azure/terraform/tree/master/quickstart/101-vm-cluster-linux/TestRecord.md).

+>

+> See more [articles and sample code showing how to use Terraform to manage Azure resources](/azure/terraform)

+1. Create a directory in which to test the sample Terraform code and make it the current directory.

+1. Create a file named `providers.tf` and insert the following code:

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-cluster-linux/providers.tf":::

+1. Create a file named `ssh.tf` and insert the following code:

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-cluster-linux/ssh.tf":::

+1. Create a file named `main.tf` and insert the following code:

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-cluster-linux/main.tf":::

+1. Create a file named `variables.tf` and insert the following code:

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-cluster-linux/variables.tf":::

+1. Create a file named `outputs.tf` and insert the following code:

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-cluster-linux/outputs.tf":::

+## Initialize Terraform

+## Create a Terraform execution plan

+## Apply a Terraform execution plan

+## Verify the results

+#### [Azure CLI](#tab/azure-cli)

+1. Get the Azure resource group name.

+ ```console

+ resource_group_name=$(terraform output -raw resource_group_name)

+ ```

+1. Run [az vm list](/cli/azure/vm#az-vm-list) with a [JMESPath](/cli/azure/query-azure-cli) query to display the names of the virtual machines created in the resource group.

+ ```azurecli

+ az vm list \

+ --resource-group $resource_group_name \

+ --query "[].{\"VM Name\":name}" -o table

+ ```

+#### [Azure PowerShell](#tab/azure-powershell)

+1. Get the Azure resource group name.

+ ```console

+ $resource_group_name=$(terraform output -raw resource_group_name)

+ ```

+1. Run [Get-AzVm](/powershell/module/az.compute/get-azvm) to display the names of all the virtual machines in the resource group.

+ ```azurepowershell

+ Get-AzVm -ResourceGroupName $resource_group_name

+ ```

+## Clean up resources

+## Troubleshoot Terraform on Azure

+[Troubleshoot common problems when using Terraform on Azure](/azure/developer/terraform/troubleshoot)

+## Next steps

+> [!div class="nextstepaction"]

+> [Azure Linux virtual machine tutorials](./tutorial-manage-vm.md)

+content_well_notification:

+ - AI-contribution

+> * Create a random value for the Azure resource group name using [random_pet](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/pet).

+> * Create an Azure resource group using [azurerm_resource_group](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group).

+> * Create a virtual network (VNET) using [azurerm_virtual_network](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/virtual_network).

+> * Create a subnet using [azurerm_subnet](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet).

+> * Create a public IP using [azurerm_public_ip](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/public_ip).

+> * Create a network security group using [azurerm_network_security_group](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/network_security_group).

+> * Create a network interface using [azurerm_network_interface](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/network_interface).

+> * Create an association between the network security group and the network interface using [azurerm_network_interface_security_group_association](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/network_interface_security_group_association).

+> * Generate a random value for a unique storage account name using [random_id](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id).

+> * Create a storage account for boot diagnostics using [azurerm_storage_account](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account).

+> * Create a Linux VM using [azurerm_linux_virtual_machine](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/linux_virtual_machine)

+> * Create an AzAPI resource [azapi_resource](https://registry.terraform.io/providers/Azure/azapi/latest/docs/resources/azapi_resource).

+> * Create an AzAPI resource to generate an SSH key pair using [azapi_resource_action](https://registry.terraform.io/providers/Azure/azapi/latest/docs/resources/azapi_resource_action).

+> [!NOTE]

+> The sample code for this article is located in the [Azure Terraform GitHub repo](https://github.com/Azure/terraform/tree/master/quickstart/101-vm-with-infrastructure). You can view the log file containing the [test results from current and previous versions of Terraform](https://github.com/Azure/terraform/tree/master/quickstart/101-vm-with-infrastructure/TestRecord.md).

+>

+> See more [articles and sample code showing how to use Terraform to manage Azure resources](/azure/terraform)

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-with-infrastructure/providers.tf":::

+1. Create a file named `ssh.tf` and insert the following code:

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-with-infrastructure/ssh.tf":::

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-with-infrastructure/main.tf":::

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-with-infrastructure/variables.tf":::

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-with-infrastructure/outputs.tf":::

+#### [Azure CLI](#tab/azure-cli)

+1. Get the Azure resource group name.

+ resource_group_name=$(terraform output -raw resource_group_name)

+1. Run [az vm list](/cli/azure/vm#az-vm-list) with a [JMESPath](/cli/azure/query-azure-cli) query to display the names of the virtual machines created in the resource group.

+ ```azurecli

+ az vm list \

+ --resource-group $resource_group_name \

+ --query "[].{\"VM Name\":name}" -o table

+#### [Azure PowerShell](#tab/azure-powershell)

+1. Get the Azure resource group name.

+ $resource_group_name=$(terraform output -raw resource_group_name)

+1. Run [Get-AzVm](/powershell/module/az.compute/get-azvm) to display the names of all the virtual machines in the resource group.

+ ```azurepowershell

+ Get-AzVm -ResourceGroupName $resource_group_name

+ ```

+ :::image type="content" source="./media/maintenance-configurations/add-schedule-maintenance-window.png" alt-text="Screenshot of the upper maintenance window time.":::

+> Today, about 90% of IaaS VMs are using [Azure Resource Manager](https://azure.microsoft.com/features/resource-manager/). As of February 28, 2020, classic VMs have been deprecated and will be fully retired on September 6, 2023. [Learn more]( https://aka.ms/classicvmretirement) about this deprecation and [how it affects you](classic-vm-deprecation.md#how-does-this-affect-me).

+> Today, about 90% of IaaS VMs are using [Azure Resource Manager](https://azure.microsoft.com/features/resource-manager/). As of February 28, 2020, classic VMs have been deprecated and will be fully retired on September 6, 2023. [Learn more]( https://aka.ms/classicvmretirement) about this deprecation and [how it affects you](classic-vm-deprecation.md#how-does-this-affect-me).

+> Today, about 90% of IaaS VMs are using [Azure Resource Manager](https://azure.microsoft.com/features/resource-manager/). As of February 28, 2020, classic VMs have been deprecated and will be fully retired on September 6, 2023. [Learn more]( https://aka.ms/classicvmretirement) about this deprecation and [how it affects you](./classic-vm-deprecation.md#how-does-this-affect-me).

+> Today, about 90% of IaaS VMs are using [Azure Resource Manager](https://azure.microsoft.com/features/resource-manager/). As of February 28, 2020, classic VMs have been deprecated and will be fully retired on September 6, 2023. [Learn more]( https://aka.ms/classicvmretirement) about this deprecation and [how it affects you](classic-vm-deprecation.md#how-does-this-affect-me).

+> Today, about 90% of IaaS VMs are using [Azure Resource Manager](https://azure.microsoft.com/features/resource-manager/). As of February 28, 2020, classic VMs have been deprecated and will be fully retired on September 6, 2023. [Learn more]( https://aka.ms/classicvmretirement) about this deprecation and [how it affects you](classic-vm-deprecation.md#how-does-this-affect-me).

+> Today, about 90% of IaaS VMs are using [Azure Resource Manager](https://azure.microsoft.com/features/resource-manager/). As of February 28, 2020, classic VMs have been deprecated and will be fully retired on September 6, 2023. [Learn more]( https://aka.ms/classicvmretirement) about this deprecation and [how it affects you](classic-vm-deprecation.md#how-does-this-affect-me).

+2. What are the technical reasons for Azure Resource Manager? What (if any) other Azure services would you like to use?

+- **Do a Validate/Prepare/Abort Dry Run** - This is perhaps the most important step to ensure Classic to Azure Resource Manager migration success. The migration API has three main steps: Validate, Prepare and Commit. Validate will read the state of your classic environment and return a result of all issues. However, because some issues might exist in the Azure Resource Manager stack, Validate won't catch everything. The next step in migration process, Prepare will help expose those issues. Prepare will move the metadata from Classic to Azure Resource Manager, but won't commit the move, and won't remove or change anything on the Classic side. The dry run involves preparing the migration, then aborting (**not committing**) the migrations prepare. The goal of validate/prepare/abort dry run is to see all of the metadata in the Azure Resource Manager stack, examine it (*programmatically or in Portal*), and verify that everything migrates correctly, and work through technical issues. It will also give you a sense of migration duration so you can plan for downtime accordingly. A validate/prepare/abort doesn't cause any user downtime; therefore, it's nondisruptive to application usage.

+ - The items below will need to be solved before the dry run, but a dry run test will also safely flush out these preparation steps if they're missed. During enterprise migration, we've found the dry run to be a safe and invaluable way to ensure migration readiness.

+- **Express Route Circuits and VPN**. Currently Express Route Gateways with authorization links can't be migrated without downtime. For the workaround, see [Migrate ExpressRoute circuits and associated virtual networks from the classic to the Resource Manager deployment model](../expressroute/expressroute-migration-classic-resource-manager.md).

+- **VM Extensions** - Virtual Machine extensions are potentially one of the biggest roadblocks to migrating running VMs. Remediation of VM Extensions could take upwards of 1-2 days, so plan accordingly. A working Azure agent is needed to report back VM Extension status of running VMs. If the status comes back as bad for a running VM, this will halt migration. The agent itself doesn't need to be in working order to enable migration, but if extensions exist on the VM, then both a working agent AND outbound internet connectivity (with DNS) will be needed for migration to move forward.

+ - If connectivity to a DNS server is lost during migration, all VM Extensions except BGInfo v1.\* need to first be removed from every VM before migration prepare, and subsequently re-added back to the VM after Azure Resource Manager migration. **This is only for VMs that are running.** If the VMs are stopped deallocated, VM Extensions don't need to be removed. **Note:** Many extensions like Azure diagnostics and Defender for Cloud monitoring will reinstall themselves after migration, so removing them isn't a problem.

+ - In addition, make sure Network Security Groups aren't restricting outbound internet access. This can happen with some Network Security Groups configurations. Outbound internet access (and DNS) is needed for VM Extensions to be migrated to Azure Resource Manager.

+ - There are two versions of the BGInfo extension: v1 and v2. If the VM was created using the Azure portal or PowerShell, the VM will likely have the v1 extension on it. This extension doesn't need to be removed and will be skipped (not migrated) by the migration API. However, if the Classic VM was created with the new Azure portal, it will likely have the JSON-based v2 version of BGInfo, which can be migrated to Azure Resource Manager provided the agent is working and has outbound internet access (and DNS).

+- **Availability Sets** - For a virtual network (vNet) to be migrated to Azure Resource Manager, the Classic deployment (i.e. cloud service) contained VMs must all be in one availability set, or the VMs must all not be in any availability set. Having more than one availability set in the cloud service isn't compatible with Azure Resource Manager and will halt migration. Additionally, there can't be some VMs in an availability set, and some VMs not in an availability set. To resolve this, you'll need to remediate or reshuffle your cloud service. Plan accordingly as this might be time consuming.

+- **Web/Worker Role Deployments** - Cloud Services containing web and worker roles can't migrate to Azure Resource Manager. The web/worker roles must first be removed from the virtual network before migration can start. A typical solution is to just move web/worker role instances to a separate Classic virtual network that is also linked to an ExpressRoute circuit, or to migrate the code to newer PaaS App Services (this discussion is beyond the scope of this document). In the former redeploy case, create a new Classic virtual network, move/redeploy the web/worker roles to that new virtual network, then delete the deployments from the virtual network being moved. No code changes required. The new [Virtual Network Peering](../virtual-network/virtual-network-peering-overview.md) capability can be used to peer together the classic virtual network containing the web/worker roles and other virtual networks in the same Azure region such as the virtual network being migrated (**after virtual network migration is completed as peered virtual networks cannot be migrated**), hence providing the same capabilities with no performance loss and no latency/bandwidth penalties. Given the addition of [Virtual Network Peering](../virtual-network/virtual-network-peering-overview.md), web/worker role deployments can now easily be mitigated and not block the migration to Azure Resource Manager.

+- **RoleStateUnknown VM Status** - If migration halts due to a **role state unknown** error message, inspect the VM using the portal and ensure it's running. This error will typically go away on its own (no remediation required) after a few minutes and is often a transient type often seen during a Virtual Machine **start**, **stop**, **restart** operations. **Recommended practice:** re-try migration again after a few minutes.

+- **Fabric Cluster does not exist** - In some cases, certain VMs can't be migrated for various odd reasons. One of these known cases is if the VM was recently created (within the last week or so) and happened to land an Azure cluster that isn't yet equipped for Azure Resource Manager workloads. You'll get an error that says **fabric cluster does not exist** and the VM can't be migrated. Waiting a couple of days will usually resolve this particular problem as the cluster will soon get Azure Resource Manager enabled. However, one immediate workaround is to `stop-deallocate` the VM, then continue forward with migration, and start the VM back up in Azure Resource Manager after migrating.

+- Don't take shortcuts and omit the validate/prepare/abort dry run migrations.

+Now you're ready because you have worked through the known issues with your environment.

+> Today, about 90% of IaaS VMs are using [Azure Resource Manager](https://azure.microsoft.com/features/resource-manager/). As of February 28, 2020, classic VMs have been deprecated and will be fully retired on September 6, 2023. [Learn more]( https://aka.ms/classicvmretirement) about this deprecation and [how it affects you](classic-vm-deprecation.md#how-does-this-affect-me).

+description: NC-series retirement by September 6, 2023

+# Migrate your NC and NC_Promo series virtual machines by September 6, 2023

+Based on feedback weΓÇÖve received from customers weΓÇÖre happy to announce that we're extending the retirement date by 1 year to 6 September 2023, for the Azure NC-Series virtual machine to give you more time to plan your migration.

+With this in mind, we're retiring our NC (v1) GPU VM sizes, powered by NVIDIA Tesla K80 GPUs on 6 September 2023.

+After 6 September 2023, any remaining NC size virtual machines remaining in your subscription will be set to a deallocated state. These virtual machines will be stopped and removed from the host. These virtual machines will no longer be billed in the deallocated state.

+> NC and NC_Promo series Azure virtual machines (VMs) will be retired on September 6, 2023. For more information, see the [NC and NC_Promo retirement information](nc-series-retirement.md). For how to migrate your workloads to other VM sizes, see the [GPU compute migration guide](n-series-migration.md).

+description: NCv2-series retirement by September 6, 2023

+# Migrate your NCv2 series virtual machines by September 6, 2023

+WeΓÇÖre happy to announce that we're extending the retirement date by one year to September 6, 2023, for the Azure NCv2-Series virtual machine to give you more time to plan your migration.

+We are retiring our NC (v2) GPU VM sizes, powered by NVIDIA Tesla P100 GPUs on 6 September 2023.

+After 6 September 2023, any remaining NCv2 size virtual machines remaining in your subscription will be set to a deallocated state. These virtual machines will be stopped and removed from the host. These virtual machines will no longer be billed in the deallocated state.

+This VM size retirement only impacts the VM sizes in the [NCv2-series](ncv2-series.md). This doesn't impact the newer [NCv3](ncv3-series.md), [NCasT4 v3](nct4-v3-series.md), and [NC A100 v4](nc-a100-v4-series.md) series virtual machines.

+> NCv2 series Azure virtual machines (VMs) will be retired on September 6, 2023. For more information, see the [NCv2 retirement information](ncv2-series-retirement.md). For how to migrate your workloads to other VM sizes, see the [GPU compute migration guide](n-series-migration.md).

+description: ND-series retirement by September 6, 2023

+# Migrate your ND series virtual machines by September 6, 2023

+Based on feedback weΓÇÖve received from customers weΓÇÖre happy to announce that we're extending the retirement date by one year to 6 September 2023, for the Azure ND-Series virtual machine to give you more time to plan your migration.

+With this in mind, we're retiring our ND GPU VM sizes, powered by NVIDIA Tesla P40 GPUs on 6 September 2023.

+After 6 September 2023, any remaining ND size virtual machines remaining in your subscription will be set to a deallocated state. These virtual machines will be stopped and removed from the host. These virtual machines will no longer be billed in the deallocated state.

+description: NV series retirement starting September 6, 2023

+# Migrate your NV and NV_Promo series virtual machines by September 6, 2023

+WeΓÇÖre happy to announce that we're extending the retirement date by one year to September 6, 2023, for the Azure NV-Series and NV_Promo Series virtual machine to give you more time to plan your migration.

+We continue to bring modern and optimized virtual machine (VM) instances to Azure by using the latest innovations in datacenter technologies. As we innovate, we also thoughtfully plan how we retire aging hardware. With this context in mind, we're retiring our NV-series Azure VM sizes on September 6, 2023.

+After September 6, 2023, any remaining NV and NV_Promo-size VMs remaining in your subscription will be set to a deallocated state. These VMs will be stopped and removed from the host. These VMs will no longer be billed in the deallocated state.

+> NV and NV_Promo series Azure virtual machines (VMs) will be retired on September 6, 2023. For more information, see the [NV and NV_Promo retirement information](nv-series-retirement.md). For how to migrate your workloads to other VM sizes, see the [NV and NV_Promo series migration guide](nv-series-migration-guide.md).

+ Title: 'Quickstart: Create a Windows VM cluster in Azure using Terraform'

+description: In this article, you learn how to create a Windows VM cluster in Azure using Terraform

+content_well_notification:

+ - AI-contribution

+# Quickstart: Create a Windows VM cluster in Azure using Terraform

+**Applies to:** :heavy_check_mark: Windows VMs

+This article shows you how to create a Windows VM cluster (containing three Windows VM instances) in Azure using Terraform.

+> [!div class="checklist"]

+> * Create a random value for the Azure resource group name using [random_pet](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/pet).

+> * Create an Azure resource group using [azurerm_resource_group](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group).

+> * Create a random value for the Windows VM host name [random_string](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string).

+> * Create a random password for the Windows VMs using [random_password](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/password).

+> * Create a Windows VM using the [compute module](https://registry.terraform.io/modules/Azure/compute/azurerm).

+> * Create a virtual network along with subnet using the [network module](https://registry.terraform.io/modules/Azure/network/azurerm).

+## Prerequisites

+- [Install and configure Terraform](/azure/developer/terraform/quickstart-configure)

+## Implement the Terraform code

+> [!NOTE]

+> The sample code for this article is located in the [Azure Terraform GitHub repo](https://github.com/Azure/terraform/tree/UserStory89540/quickstart/101-vm-cluster-windows). You can view the log file containing the [test results from current and previous versions of Terraform](https://github.com/Azure/terraform/tree/UserStory89540/quickstart/101-vm-cluster-windows/TestRecord.md).

+>

+> See more [articles and sample code showing how to use Terraform to manage Azure resources](/azure/terraform)

+1. Create a directory in which to test the sample Terraform code and make it the current directory.

+1. Create a file named `providers.tf` and insert the following code:

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-cluster-windows/providers.tf":::

+1. Create a file named `main.tf` and insert the following code:

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-cluster-windows/main.tf":::

+1. Create a file named `variables.tf` and insert the following code:

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-cluster-windows/variables.tf":::

+1. Create a file named `outputs.tf` and insert the following code:

+ :::code language="Terraform" source="~/terraform_samples/quickstart/101-vm-cluster-windows/outputs.tf":::

+## Initialize Terraform

+## Create a Terraform execution plan

+## Apply a Terraform execution plan

+## Verify the results

+#### [Azure CLI](#tab/azure-cli)

+1. Get the Azure resource group name.

+ ```console

+ resource_group_name=$(terraform output -raw resource_group_name)

+ ```

+1. Run [az vm list](/cli/azure/vm#az-vm-list) with a [JMESPath](/cli/azure/query-azure-cli) query to display the names of the virtual machines created in the resource group.

+ ```azurecli

+ az vm list \

+ --resource-group $resource_group_name \

+ --query "[].{\"VM Name\":name}" -o table

+ ```

+#### [Azure PowerShell](#tab/azure-powershell)

+1. Get the Azure resource group name.

+ ```console

+ $resource_group_name=$(terraform output -raw resource_group_name)

+ ```

+1. Run [Get-AzVm](/powershell/module/az.compute/get-azvm) to display the names of all the virtual machines in the resource group.

+ ```azurepowershell

+ Get-AzVm -ResourceGroupName $resource_group_name

+ ```

+## Clean up resources

+## Troubleshoot Terraform on Azure

+[Troubleshoot common problems when using Terraform on Azure](/azure/developer/terraform/troubleshoot)

+## Next steps

+> [!div class="nextstepaction"]

+> [Azure Linux virtual machine tutorials](./tutorial-manage-vm.md)

+* Palo Alto Networks Cloud NGFW is only available in the following Azure regions: Central US, East US, East US 2, West US, West Europe, Australia East, Australia Southeast, UK South, UK West, Canada Central and East Asia. Other Azure regions are on the roadmap.

+-VpnType "RouteBased" -GatewaySku VpnGw2 -VpnGatewayGeneration "Generation2"