+- [Known issues](known-issues.md)

+**Discover schema** - If your end point supports /Schema, this option lets the tool discover the supported attributes. We recommend this option as it reduces the overhead of updating your app as you build it out.

+In addition to using the SCIM Validator tool, you can also use Postman to validate an endpoint. This example provides a set of tests in Postman. The example validates create, read, update, and delete (CRUD) operations. The operations are validated on users and groups, filtering, updates to group membership, and disabling users.

+- The time zone format is randomly generated and fails for systems that try to validate it.

+- The preferred language format is randomly generated and fails for systems that try to validate it.

+- [Learn how to add an app that's not in the Azure AD app gallery](../manage-apps/overview-application-gallery.md)

+Requests to a SCIM endpoint require authorization. The SCIM standard has multiple options available. Requests can use cookies, basic authentication, TLS client authentication, or any of the methods listed in [RFC 7644](https://tools.ietf.org/html/rfc7644#section-2).

+You cannot set token lifetime policies for refresh tokens and session tokens. For lifetime, timeout, and revocation information on refresh tokens, see [Refresh tokens](refresh-tokens.md).

+> As of January 30, 2021 you cannot configure refresh and session token lifetimes. Azure Active Directory no longer honors refresh and session token configuration in existing policies. New tokens issued after existing tokens have expired are now set to the [default configuration](#configurable-token-lifetime-properties). You can still configure access, SAML, and ID token lifetimes after the refresh and session token configuration retirement.

+For an example, see [Create a policy for web sign-in](registration-config-change-token-lifetime-how-to.md).

+Non-persistent session tokens have a Max Inactive Time of 24 hours whereas persistent session tokens have a Max Inactive Time of 90 days. Anytime the SSO session token is used within its validity period, the validity period is extended another 24 hours or 90 days. If the SSO session token isn't used within its Max Inactive Time period, it's considered expired and will no longer be accepted. Any changes to this default period should be changed using [Conditional Access](../conditional-access/howto-conditional-access-session-lifetime.md).

+This article discusses what you need to know about the public keys that are used by the Microsoft identity platform to sign security tokens. It's important to note that these keys roll over on a periodic basis and, in an emergency, could be rolled over immediately. All applications that use the Microsoft identity platform should be able to programmatically handle the key rollover process. Continue reading to understand how the keys work, how to assess the impact of the rollover to your application and how to update your application or establish a periodic manual rollover process to handle key rollover if necessary.

+The Microsoft identity platform uses public-key cryptography built on industry standards to establish trust between itself and the applications that use it. In practical terms, this works in the following way: The Microsoft identity platform uses a signing key that consists of a public and private key pair. When a user signs in to an application that uses the Microsoft identity platform for authentication, the Microsoft identity platform creates a security token that contains information about the user. This token is signed by the Microsoft identity platform using its private key before it's sent back to the application. To verify that the token is valid and originated from Microsoft identity platform, the application must validate the tokenΓÇÖs signature using the public keys exposed by the Microsoft identity platform that is contained in the tenantΓÇÖs [OpenID Connect discovery document](https://openid.net/specs/openid-connect-discovery-1_0.html) or SAML/WS-Fed [federation metadata document](../azuread-dev/azure-ad-federation-metadata.md).

+For security purposes, the Microsoft identity platformΓÇÖs signing key rolls on a periodic basis and, in the case of an emergency, could be rolled over immediately. There's no set or guaranteed time between these key rolls - any application that integrates with the Microsoft identity platform should be prepared to handle a key rollover event no matter how frequently it may occur. If your application doesn't handle sudden refreshes, and attempts to use an expired key to verify the signature on a token, your application will incorrectly reject the token. Checking every 24 hours for updates is a best practice, with throttled (once every five minutes at most) immediate refreshes of the key document if a token is encountered that doesn't validate with the keys in your application's cache.

+There's always more than one valid key available in the OpenID Connect discovery document and the federation metadata document. Your application should be prepared to use any and all of the keys specified in the document, since one key may be rolled soon, another may be its replacement, and so forth. The number of keys present can change over time based on the internal architecture of the Microsoft identity platform as we support new platforms, new clouds, or new authentication protocols. Neither the order of the keys in the JSON response nor the order in which they were exposed should be considered meaningful to your app.

+Applications that support only a single signing key, or those that require manual updates to the signing keys, are inherently less secure and less reliable. They should be updated to use [standard libraries](reference-v2-libraries.md) to ensure that they're always using up-to-date signing keys, among other best practices.

+Applications that are only accessing resources (for example, Microsoft Graph, KeyVault, Outlook API, and other Microsoft APIs) only obtain a token and pass it along to the resource owner. Given that they aren't protecting any resources, they don't inspect the token and therefore don't need to ensure it's properly signed.

+Applications that are only accessing resources (such as Microsoft Graph, KeyVault, Outlook API, and other Microsoft APIs) only obtain a token and pass it along to the resource owner. Given that they aren't protecting any resources, they don't inspect the token and therefore don't need to ensure it's properly signed.

+If you added authentication to your solution manually, your application might not have the necessary key rollover logic. You'll need to write it yourself, or follow the steps in [Web applications / APIs using any other libraries or manually implementing any of the supported protocols](#other).

+If you added authentication to your solution manually, your application might not have the necessary key rollover logic. You'll need to write it yourself, or follow the steps in [Web applications / APIs using any other libraries or manually implementing any of the supported protocols.](#other).

+The following steps help you verify that the logic is working properly in your application.

+1. In Visual Studio 2013, open the solution, and then select on the **Server Explorer** tab on the right window.

+2. Expand **Data Connections**, **DefaultConnection**, and then **Tables**. Locate the **IssuingAuthorityKeys** table, right-click it, and then select **Show Table Data**.

+In desktop apps, if you want the token cache to persist, you can customize the [token cache serialization](msal-net-token-cache-serialization.md). By implementing dual token cache serialization, you can use backward-compatible and forward-compatible token caches. These tokens support previous generations of authentication libraries. Specific libraries include Azure AD Authentication Library for .NET (ADAL.NET) version 3 and version 4.

+When developing applications with the Microsoft identity platform, you need to direct your customers when they want to use their work or school account (managed in Azure AD), or their personal account for sign-up and sign-in to your application.

+In an earlier version of these guidelines, we recommended using a ΓÇ£blue badgeΓÇ¥ pictogram. Based on user and developer feedback, we now recommend the use of the Microsoft logo instead. The Microsoft logo helps users understand that they can reuse the account they use with Microsoft 365 or other Microsoft business services to sign into your app.

+**If your app supports end-user sign-up (for example, free to trial or freemium model)**: You can show a **sign-in** button that allows users to access your app with their work account or their personal account. Azure AD shows a consent prompt the first time they access your app.

+After admins consent to your app, they can choose to add it to their usersΓÇÖ Microsoft 365 app launcher experience (accessible from the waffle and from [https://www.office.com/](https://www.office.com/)). If you want to advertise this capability, you can use terms like ΓÇ£Add this app to your organizationΓÇ¥ and show a button like the following example:

+1. Choose an app and click **Branding & properties**.

+Make sure you meet the [pre-requisites](publisher-verification-overview.md#requirements), then follow these steps to mark your app(s) as Publisher Verified.

+1. Sign in using [multi-factor authentication](../fundamentals/concept-fundamentals-mfa-get-started.md) to an organizational (Azure AD) account authorized to make changes to the app you want to mark as Publisher Verified and on the MPN Account in Partner Center.

+ - The Azure AD user must have one of the following [roles](../roles/permissions-reference.md): Application Admin, Cloud Application Admin, or Global Administrator.

+ - The user in Partner Center must have the following [roles](/partner-center/permissions-overview): MPN Admin, Accounts Admin, or a Global Administrator (a shared role mastered in Azure AD).

+1. Click on an app you would like to mark as Publisher Verified and open the **Branding & properties** blade.

+1. Enter the **MPN ID** for:

+1. If the verification was successful, the publisher verification window closes, returning you to the **Branding & properties** blade. You see a blue verified badge next to your verified **Publisher display name**.

+1. Users who get prompted to consent to your app start seeing the badge soon after you've gone through the process successfully, although it may take some time for updates to replicate throughout the system.

+1. Test this functionality by signing into your application and ensuring the verified badge shows up on the consent screen. If you're signed in as a user who has already granted consent to the app, you can use the *prompt=consent* query parameter to force a consent prompt. This parameter should be used for testing only, and never hard-coded into your app's requests.

+1. Repeat these steps as needed for any more apps you would like the badge to be displayed for. You can use Microsoft Graph to do this more quickly in bulk, and PowerShell cmdlets will be available soon. See [Making Microsoft API Graph calls](troubleshoot-publisher-verification.md#making-microsoft-graph-api-calls) for more info.

+- When writing a desktop application, use the cross-platform token cache as explained in [desktop apps](msal-net-token-cache-serialization.md?tabs=desktop).

+- No action required for [mobile and Universal Windows Platform (UWP) apps](msal-net-token-cache-serialization.md?tabs=mobile). MSAL.NET provides secure storage for the cache.

+- In ASP.NET Core [web apps](scenario-web-app-call-api-overview.md) and [web APIs](scenario-web-api-call-api-overview.md), use [Microsoft.Identity.Web](microsoft-identity-web.md) as a higher-level API. `Microsoft.Identity.Web` provides token caches as explained in [ASP.NET Core web apps and web APIs](msal-net-token-cache-serialization.md?tabs=aspnetcore).

+ - If you want to use an in-memory cache and you're only using [`AcquireTokenForClient`](/dotnet/api/microsoft.identity.client.acquiretokenforclientparameterbuilder), either reuse the confidential client application instance and don't add a serializer, or create a new confidential client application and enable the [shared cache option](msal-net-token-cache-serialization.md?tabs=aspnet#no-token-cache-serialization).

+ A shared cache is faster because it's not serialized. However, the memory grows as tokens are cached. The number of tokens is equal to the number of tenants times the number of downstream APIs. An app token is about 2 KB in size, whereas tokens for a user are about 7 KB in size. It's great for development, or if you have few users.

+ - If you want to use an in-memory token cache and control its size and eviction policies, use the [Microsoft.Identity.Web in-memory cache option](msal-net-token-cache-serialization.md?tabs=aspnet#in-memory-token-cache-1).

+- If you build an SDK and want to write your own token cache serializer for confidential client applications, inherit from [Microsoft.Identity.Web.MsalAbstractTokenCacheProvider](https://github.com/AzureAD/microsoft-identity-web/blob/master/src/Microsoft.Identity.Web.TokenCache/MsalAbstractTokenCacheProvider.cs) and override the [WriteCacheBytesAsync](/dotnet/api/microsoft.identity.web.tokencacheproviders.msalabstracttokencacheprovider.writecachebytesasync) and [ReadCacheBytesAsync](/dotnet/api/microsoft.identity.web.tokencacheproviders.msalabstracttokencacheprovider.readcachebytesasync) methods.

+If you're using the [MSAL library](/dotnet/api/microsoft.identity.client) directly in an ASP.NET Core app, consider using [Microsoft.Identity.Web](https://github.com/AzureAD/microsoft-identity-web), which provides a simpler, higher-level API. Otherwise, see [Non-ASP.NET Core web apps and web APIs](?tabs=aspnet#configuring-the-token-cache), which covers direct MSAL usage.

+| [AddInMemoryTokenCaches](/dotnet/api/microsoft.identity.web.microsoftidentityappcallswebapiauthenticationbuilder.addinmemorytokencaches) | Creates a temporary cache in memory for token storage and retrieval. In-memory token caches are faster than other cache types, but their tokens aren't persisted between application restarts, and you can't control the cache size. In-memory caches are good for applications that don't require tokens to persist between app restarts. Use an in-memory token cache in apps that participate in machine-to-machine auth scenarios like services, daemons, and others that use [AcquireTokenForClient](/dotnet/api/microsoft.identity.client.acquiretokenforclientparameterbuilder) (the client credentials grant). In-memory token caches are also good for sample applications and during local app development. Microsoft.Identity.Web versions 1.19.0+ share an in-memory token cache across all application instances.

+| [AddSessionTokenCaches](/dotnet/api/microsoft.identity.web.microsoftidentityappcallswebapiauthenticationbuilder.addsessiontokencaches) | The token cache is bound to the user session. This option isn't ideal if the ID token contains many claims, because the cookie becomes too large.

+If you're using MSAL.NET in your web app or web API, you can benefit from token cache serializers in [Microsoft.Identity.Web.TokenCache](https://www.nuget.org/packages/Microsoft.Identity.Web.TokenCache)

+You can specify that you don't want to have any token cache serialization and instead rely on the MSAL.NET internal cache. Use `.WithCacheOptions(CacheOptions.EnableSharedCacheOptions)` when building the application and don't add any serializer.

+r.

+`WithCacheOptions(CacheOptions.EnableSharedCacheOptions)` makes the internal MSAL token cache shared between MSAL client application instances. Sharing a token cache is faster than using any token cache serialization, but the internal in-memory token cache doesn't have eviction policies. Existing tokens are refreshed in place, but fetching tokens for different users, tenants, and resources makes the cache grow accordingly.

+For details, see the [Cross platform Token Cache](https://github.com/AzureAD/microsoft-authentication-extensions-for-dotnet/wiki/Cross-platform-Token-Cache). Here's an example using the cross-platform token cache:

+- `ITokenCache` defines events to subscribe to token cache serialization requests and methods to serialize or deserialize the cache at various formats (MSAL 2.x and MSAL 3.x).

+- `TokenCacheCallback` is a callback passed to the events so that you can handle the serialization. They are called with arguments of type `TokenCacheNotificationArgs`.

+MSAL.NET v2.x and later versions provide several options for serializing the token cache of a public client. You can serialize the cache only to the MSAL.NET format. (The unified format cache is common across MSAL and the platforms.) You can also support the [legacy](https://github.com/AzureAD/azure-activedirectory-library-for-dotnet/wiki/Token-cache-serialization) token cache serialization of ADAL v3.

+#### Dual token cache serialization (MSAL unified cache)

+If you want to implement token cache serialization with the unified cache format (common to MSAL.NET 2.x and other MSALs of the same generation or older, on the same platform), take a look at the following sample: https://github.com/Azure-Samples/active-directory-dotnet-v1-to-v2/tree/master/TokenCacheMigration/ADAL2MSAL.

+| `DurationTotalInMs` | Total time spent in MSAL, including network calls and cache. | Alarm on overall high latency (> 1 second). Value depends on token source. From the cache: one cache access. From Azure Active Directory (Azure AD): two cache accesses plus one HTTP call. First ever call (per-process) takes longer because of one extra HTTP call. |

+ 3. If an MPN account already exists, this is recognized and you are added to the account.

+2. Go to the [MPN tenant management page](https://partner.microsoft.com/dashboard/account/v3/tenantmanagement) and confirm that the tenant the app is registered in and that you're signing with a user account from is on the list of associated tenants. To add another tenant, follow the [multi-tenant-account instructions](/partner-center/multi-tenant-account). All Global Admins of any tenant you add will be granted Global Administrator privileges on your Partner Center account.

+* **Session key**: The session key is an encrypted symmetric key, generated by the Azure AD authentication service, issued as part of the PRT. The session key acts as the proof of possession when a PRT is used to obtain tokens for other applications. Session key is rolled on Windows 10 or newer Azure AD joined or Hybrid Azure AD joined devices if it's older than 30 days.

+> [!NOTE]

+> When using password to sign into Windows 10 or newer Azure AD joined or Hybrid Azure AD joined device, MFA during WAM interactive sign in may be required after session key associated with PRT is rolled.

+ Title: 'Check execution user scope of a workflow - Azure Active Directory'

+description: Describes how to check the users who fall into the execution scope of a Lifecycle Workflow.

+# Check execution user scope of a workflow (Preview)

+Workflow scheduling will automatically process the workflow for users meeting the workflows execution conditions. This article walks you through the steps to check the users who fall into the execution scope of a workflow. For more information about execution conditions, see: [workflow basics](../governance/understanding-lifecycle-workflows.md#workflow-basics).

+## Check execution user scope of a workflow using the Azure portal

+To check the users who fall under the execution scope of a workflow, you'd follow these steps:

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Type in **Identity Governance** on the search bar near the top of the page and select it.

+1. In the left menu, select **Lifecycle workflows (Preview)**.

+1. From the list of workflows, select the workflow you want to check the execution scope of.

+1. On the workflow overview page, select **Execution conditions (Preview)**.

+1. On the Execution conditions page, select the **Execution User Scope** tab.

+1. On this page you're presented with a list of users who currently meet the scope for execution for the workflow.

+ :::image type="content" source="media/check-workflow-execution-scope/execution-user-scope-list.png" alt-text="Screenshot of users under scope of workflow execution." lightbox="media/check-workflow-execution-scope/execution-user-scope-list.png":::

+> [!NOTE]

+> The workflow engine routinely evaluates the users that meet the execution conditions. The results will not be up to date if the execution conditions have been changed recently, relevant attributes on the user have been changed recently, or the time based trigger has recently passed.

+## Check execution user scope of a workflow using Microsoft Graph

+To check execution user scope of a workflow using API via Microsoft Graph, see: [List executionScope](/graph/api/workflow-list-executionscope).

+## Next steps

+- [Manage workflow properties](manage-workflow-properties.md)

+- [Delete Lifecycle Workflows](delete-lifecycle-workflow.md)

+- [Manage Workflow Versions](manage-workflow-tasks.md)

+The Lifecycle Workflows preview requires Azure AD Premium P2. For more information, see: [License requirements](what-are-lifecycle-workflows.md#license-requirements).

+ Title: Customize emails sent out by workflow tasks

+description: A step by step guide for customizing emails sent out using tasks within Lifecycle Workflows

+# Customize emails sent out by workflow tasks (Preview)

+Lifecycle Workflows provide several tasks that send out email notifications. Email notifications can be customized to suit the needs of a specific workflow. For a list of these tasks, see: [Lifecycle Workflows tasks and definitions (Preview)](lifecycle-workflow-tasks.md).

+Emails tasks allow for the customization of the following aspects:

+- Additional CC recipients

+- Sender domain

+- Organizational branding of the email

+- Subject

+- Message body

+- Email language

+> [!NOTE]

+> To avoid additional security disclaimers, you should opt in to using customized domain and organizational branding.

+For more information on these customizable parameters, see: [Common email task parameters](lifecycle-workflow-tasks.md#common-email-task-parameters).

+## Prerequisites

+- Azure AD Premium P2

+For more information, see: [License requirements](what-are-lifecycle-workflows.md#license-requirements)

+## Customize email using the Azure portal

+When customizing an email sent via Lifecycle workflows, you can choose to customize either a new or existing task. These customizations are done the same way no matter if the task is new or existing, but the following steps walk you through updating an existing task. To customize emails sent out from tasks within workflows using the Azure portal, you'd follow these steps:

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Type in **Identity Governance** in the search bar near the top of the page, and select it.

+1. In the left menu, select **Lifecycle workflows (Preview)**.

+1. In the left menu, select **workflows (Preview)**.

+

+1. On the left side of the screen, select **Tasks (Preview)**.

+1. On the tasks screen, select the task for which you want to customize the email.

+1. On the specific task screen, you're able to set CC to include others in the email outside of the default audience.

+1. Select the **Email Customization** tab.

+1. On the email customization screen, enter a custom subject, message body, and the email language translation option that will be used to translate the message body of the email.

+ :::image type="content" source="media/customize-workflow-email/customize-workflow-email-example.png" alt-text="Screenshot of an example of a customized email from a workflow.":::

+1. After making changes, select **save** to capture changes to the customized email.

+## Format attributes within customized emails

+To further personalize customized emails, you can take advantage of dynamic attributes. With dynamic attributes by placing in specific attributes, you're able to specifically call out values such as a user's name, their generated Temporary Access Pass, or even their manager's email.

+To use dynamic attributes within your customized emails, you must follow the formatting rules within the customized email. The proper format is:

+{{**dynamic attribute**}}

+The following screenshot is an example of the proper format for dynamic attributes within a customized email:

+When typing this it's written the following way:

+```html

+Welcome to the team, {{userGivenName}}

+We're excited to have you join our growing team and look forward to a successful and memorable journey together.

+We've already set up a few things to help you get started quickly and make your onboarding process as smooth as possible.

+For more information and next steps, please contact your manager, {{managerDisplayName}}

+```

+For a full list of dynamic attributes that can be used with customized emails, see:[Dynamic attributes within email](lifecycle-workflow-tasks.md#dynamic-attributes-within-email).

+## Use custom branding and domain in emails sent out using workflows

+Emails sent out using Lifecycle workflows can be customized to have your own company branding, and be sent out using your company domain. When you opt in to using custom branding and domain, every email sent out using Lifecycle Workflows reflect these settings. To enable these features the following prerequisites are required:

+- A verified domain. To add a custom domain, see: [Managing custom domain names in your Azure Active Directory](../enterprise-users/domains-manage.md)

+- Custom Branding set within Azure AD if you want to have your custom branding used in emails. To set organizational branding within your Azure tenant, see: [Configure your company branding (preview)](../fundamentals/how-to-customize-branding.md).

+After these prerequisites are satisfied, you'd follow these steps:

+1. On the Lifecycle workflows page, select **Workflow settings (Preview)**.

+1. On the settings page, with **email domain** you're able to select your domain from a drop-down list of your verified domains.

+ :::image type="content" source="media/customize-workflow-email/workflow-email-settings.png" alt-text="Screenshot of workflow domain settings.":::

+1. With the Use company branding banner logo setting, you're able to turn on whether or not company branding is used in emails.

+ :::image type="content" source="media/customize-workflow-email/customize-email-logo-setting.png" alt-text="Screenshot of email logo setting.":::

+## Customize email using Microsoft Graph

+To customize email using Microsoft Graph API see: [workflow: createNewVersion](/graph/api/identitygovernance-workflow-createnewversion).

+## Set custom branding and domain workflow settings in Lifecycle Workflows using Microsoft Graph

+To turn on custom branding and domain feature settings in Lifecycle Workflows using Microsoft Graph API, see: [lifecycleManagementSettings resource type](/graph/api/resources/identitygovernance-lifecyclemanagementsettings)

+## Next steps

+- [Lifecycle Workflow tasks](lifecycle-workflow-tasks.md)

+- [Manage workflow versions](manage-workflow-tasks.md)

+# Customize the schedule of workflows (Preview)

+The Lifecycle Workflows preview requires Azure AD Premium P2. For more information, see: [License requirements](what-are-lifecycle-workflows.md#license-requirements).

+1. Type in **Identity Governance** on the search bar near the top of the page and select it.

+Lifecycle Workflows allow you to create workflows that can be triggered based on joiner, mover, or leaver scenarios. While Lifecycle Workflows provide several built-in tasks to automate common scenarios throughout the lifecycle of users, eventually you may reach the limits of these built-in tasks. With the extensibility feature, you're able to utilize the concept of custom task extensions to call-out to external systems as part of a workflow. By calling out to the external systems, you're able to accomplish things, which can extend the purpose of your workflows. When a user joins your organization you can have a workflow with a custom task extension that assigns a Teams number, or have a separate workflow that grants access to an email account for a manager when a user leaves. With the extensibility feature, Lifecycle Workflows currently support creating custom tasks extensions to call-out to [Azure Logic Apps](../../logic-apps/logic-apps-overview.md).

+When creating custom task extensions, the scenarios for how it interacts with Lifecycle Workflows can be one of two ways:

+- **Launch and wait** - The Azure Logic App is started, and the following task's execution waits on the response from the Logic App. You enter a time duration for how long the custom task extension should wait for a response from the Azure Logic App. If no response is received within a customer defined duration window, the task is considered failed.

+ :::image type="content" source="media/lifecycle-workflow-extensibility/custom-task-launch-wait.png" alt-text="Screenshot of custom task launch and wait task choice." lightbox="media/lifecycle-workflow-extensibility/custom-task-launch-wait.png":::

+## Response authorization

+When creating a custom task extension that waits for a response from the Logic App, you're able to define which applications can send a response

+Response authorization can be utilized in one of the following ways:

+- **System-assigned managed identity (Default)** - Enables and utilizes the Logic Apps system-assigned managed identity. For more information on this, see: [Authenticate access to Azure resources with managed identities in Azure Logic Apps](/azure/logic-apps/create-managed-service-identity)

+- **No authorization** - Grants no authorization to the Logic App. You're responsible for assigning an application permission, or role assignment.

+- **Existing application** - You can choose an existing application to respond.

+Separating processing of the workflow from the tasks is important because, in a workflow, processing a user certain tasks could be successful, while others could fail. Whether or not a task runs after a failed task in a workflow depends on parameters such as enabling continue On Error, and their placement within the workflow. For more information, see [Common task parameters (preview)](lifecycle-workflow-tasks.md#common-task-parameters).

+# Lifecycle Workflow built-in tasks (Preview)

+## Supported tasks

+## Common task parameters

+## Common email task parameters

+Emails, sent from tasks, are able to be customized. If you choose to customize the email, you're able to set the following arguments:

+- **Subject:** Customizes the subject of emails.

+- **Message body:** Customizes the body of the emails being sent out.

+- **Email language translation:** Overrides the email recipient's language settings. Custom text is not customized, and it is recommended to set this language to the same language as the custom text.

+For a step by step guide on this, see: [Customize emails sent out by workflow tasks](customize-workflow-email.md).

+### Dynamic attributes within email

+With customized emails, you're able to include dynamic attributes within the subject and body to personalize these emails. The list of dynamic attributes that can be included are as follows:

+|Attribute |Definition |

+|||

+|userDisplayName | The userΓÇÖs display name. |

+|userEmployeeHireDate | The userΓÇÖs employee hire date. |

+|userEmployeeLeaveDateTime | The userΓÇÖs employee leave date time. |

+|managerDisplayName | The display name of the userΓÇÖs manager. |

+|temporaryAccessPass | The generated Temporary Access Pass. Only available with the **Generate TAP And Send Email** task. |

+|userPrincipalName | The userΓÇÖs userPrincipalName. |

+|managerEmail | The managerΓÇÖs email. |

+|userSurname | UserΓÇÖs last name. |

+|userGivenName | UserΓÇÖs first name. |

+> [!NOTE]

+> When adding these attributes to a customized email, or subject, they must be properly embedded. For a step by step guide on doing this, see: [Format attributes within customized emails](customize-workflow-email.md).

+## Task details

+In this section is each specific task, and detailed information such as parameters and prerequisites, required for them to run successfully. The parameters are noted as they appear both in the Azure portal, and within Microsoft Graph. For information about editing Lifecycle Workflow tasks in general, see: [Manage workflow Versions](manage-workflow-tasks.md).

+ "arguments": [

+ {

+ "name": "cc",

+ "value": "b47471b9-af8f-4a5a-bfa2-b78e82398f6e, a7a23ce0-909b-40b9-82cf-95d31f0aaca2"

+ },

+ {

+ "name": "customSubject",

+ "value": "Welcome to the organization {{userDisplayName}}!"

+ },

+ {

+ "name": "customBody",

+ "value": "Welcome to our organization {{userGivenName}} {{userSurname}}. \nFor more information, reach out to your manager {{managerDisplayName}} at {{managerEmail}}."

+ },

+ {

+ "name": "locale",

+ "value": "en-us"

+ }

+ ]

+ "arguments": [

+ {

+ "name": "cc",

+ "value": "b47471b9-af8f-4a5a-bfa2-b78e82398f6e, a7a23ce0-909b-40b9-82cf-95d31f0aaca2"

+ },

+ {

+ "name": "customSubject",

+ "value": "Reminder to onboard {{userDisplayName}}!"

+ },

+ {

+ "name": "customBody",

+ "value": "Hello {{managerDisplayName}}. \n This is a reminder to onboard {{userDisplayName}}."

+ },

+ {

+ "name": "locale",

+ "value": "en-us"

+ }

+ ]

+When a compatible user joins your organization, Lifecycle Workflows allow you to automatically generate a Temporary Access Pass (TAP), and have it sent to the new user's manager. You're also able to customize the email that is sent to the user's manager.

+ "name": "cc",

+ "value": "b47471b9-af8f-4a5a-bfa2-b78e82398f6e, a7a23ce0-909b-40b9-82cf-95d31f0aaca2"

+ },

+ {

+ "name": "customSubject",

+ "value": "Your new employees Temporary Access Pass {{managerDisplayName}}"

+ },

+ {

+ "name": "customBody",

+ "value": "Hello {{managerDisplayName}}. \nThe temporary Access Pass {{temporaryAccessPass}} has been generated for {{userDisplayName}}."

+ },

+ {

+ "name": "locale",

+ "value": "en-us"

+ },

+ {

+ },

+ {

+ }

+## Send email to manager before user's last day

+The Azure AD prerequisite to run the **Send email before user's last day** task are:

+For Microsoft Graph the parameters for the **Send email before user's last day** task are as follows:

+ "arguments": [

+ {

+ "name": "cc",

+ "value": "b47471b9-af8f-4a5a-bfa2-b78e82398f6e, a7a23ce0-909b-40b9-82cf-95d31f0aaca2"

+ },

+ {

+ "name": "customSubject",

+ "value": "Reminder that {{userDisplayName}}'s last day is coming up."

+ },

+ {

+ "name": "customBody",

+ "value": "Hello {{managerDisplayName}}. \nThis is a reminder that {{userDisplayName}}'s last date is coming up."

+ },

+ {

+ "name": "locale",

+ "value": "en-us"

+ },

+ ]

+## Send email on user's last day

+ "arguments": [

+ {

+ "name": "cc",

+ "value": "b47471b9-af8f-4a5a-bfa2-b78e82398f6e, a7a23ce0-909b-40b9-82cf-95d31f0aaca2"

+ },

+ {

+ "name": "customSubject",

+ "value": "{{userDisplayName}}'s last day"

+ },

+ {

+ "name": "customBody",

+ "value": "Hello {{managerDisplayName}}. \nThis is a reminder that {{userDisplayName}}'s last day is today, {{userEmployeeLeaveDateTime}}."

+ },

+ {

+ "name": "locale",

+ "value": "en-us"

+ },

+ ]

+## Send email to user's manager after their last day

+Allows an email containing off-boarding information to be sent to the user's manager after their last day. You're able to customize the task name and description for this task in the Azure portal.

+The Azure AD prerequisite to run the **Send email to users manager after their last day** task are:

+For Microsoft Graph the parameters for the **Send email to users manager after their last day** task are as follows:

+|displayName | Send email to users manager after their last day |

+|description | Send offboarding email to userΓÇÖs manager after the last day of work (Customizable by user) |

+ "arguments": [

+ {

+ "name": "cc",

+ "value": "b47471b9-af8f-4a5a-bfa2-b78e82398f6e, a7a23ce0-909b-40b9-82cf-95d31f0aaca2"

+ },

+ {

+ "name": "customSubject",

+ "value": "{{userDisplayName}} left on {{userEmployeeLeaveDateTime}}"

+ },

+ {

+ "name": "customBody",

+ "value": "Hello {{managerDisplayName}}. This is a reminder that {{userDisplayName}} left on{{UserEmployeeLeaveDateTime}}."

+ },

+ {

+ "name": "locale",

+ "value": "en-us"

+ },

+]

+- **Security Owner** ensures that the plan meets the security requirements of your organization. This team:

+ * Helps determine attributes that are used to populate employeeHireDate and employeeLeaveDateTime.

+|Attribute synchronization|The accounts in Azure AD have the employeeHireDate and employeeLeaveDateTime attributes populated. The values may be populated when the accounts are created from an HR system or synchronized from AD using Azure AD Connect or cloud sync. You have extra attributes that are used to determine the scope such as department, populated or the ability to populate, with data.|[How to synchronize attributes for Lifecycle Workflows](how-to-lifecycle-workflow-sync-attributes.md)

+|Value range for offsetInDays|Between -180 and 180 days|

+|[Determine the execution conditions](#determine-the-execution-conditions)|Determine who and when the workflow runs|

+Before building a Lifecycle Workflow in the portal, you should determine which scenario or scenarios you wish to deploy. You can use the following table to see a current list of the available scenarios. These are based on the templates that are available in the portal and list the task associated with each one.

+|Scenario|Predefined Tasks|

+|Onboard prehire employee| Generate TAP and Send Email|

+Now that you've determined your scenarios, you need to look at what users in your organization the scenarios apply to.

+The [scope](understanding-lifecycle-workflows.md#configure-scope) determines who the workflow runs against. This is defined by a rule that will filter users based on a condition. For example, the rule, `"rule": "(department eq 'sales')"` runs the task only on users who are members of the sales department.

+The [trigger](understanding-lifecycle-workflows.md#trigger-details) determines when the workflow runs. This can either be, on-demand, which is immediate, or time based. Most of the predefined templates in the portal are time based.

+Now that we've determined the scenario and the who and when, you should consider whether the predefined tasks are sufficient or are you going to need extra tasks. The following table has a list of the predefined tasks that are currently in the portal. Use this table to determine if you want to add more tasks.

+If you're using a group or team task, the workflow needs you to specify the group or groups. In the following screenshot, you see the yellow triangle on the task indicating that it's missing information.

+By clicking on the task, you are presented with a navigation bar to add or remove groups. Select the "x groups selected" link to add groups.

+Lifecycle Workflows allow you to create workflows that can be triggered based on joiner, mover, or leaver scenarios. While Lifecycle Workflows provide several built-in tasks to automate common scenarios throughout the lifecycle of users, eventually you may reach the limits of these built-in tasks. With the extensibility feature, you are able to utilize the concept of custom task extensions to call-out to external systems as part of a Lifecycle Workflow.

+When creating custom task extensions, the scenarios for how it interacts with Lifecycle Workflows can be one of three ways:

+Using the on-demand feature allows you to test and evaluate whether the Lifecycle Workflow is working as intended.

+|Determine the scenario| A prehire workflow that sends email to new manager. |

+|Determine the execution conditions|The workflow runs on new employees in the sales department, two (2) days before the employeeHireDate.|

+|Review the tasks.|We use the predefined tasks in the workflow. No extra tasks are added.|

+|Create the workflow in the portal|Use the predefined template for new hire in the portal.|

+1. Type in **Identity Governance** on the search bar near the top of the page and select it.

+1. Type in **Identity Governance** on the search bar near the top of the page and select it.

+1. Type in **Identity Governance** on the search bar near the top of the page and select it.

+## Create a custom task extension using the Azure portal

+To use a custom task extension in your workflow, first a custom task extension must be created to be linked with an Azure Logic App. You're able to create a Logic App at the same time you're creating a custom task extension. To do this, you complete these steps:

+1. On the Lifecycle workflows screen, select **Custom task extension**.

+1. On the **Task behavior** page, you specify how the custom task extension will behave after executing the Azure Logic App. If you choose **Launch and continue** you can immediately select **Next: Details**.

+1. If you select **Launch and wait**, you're given an option of how long to wait for a response from the logic app before the task is considered a failure, and also options to set **Response authorization**. After choosing these options, you would be able to select **Next: Details**.

+ :::image type="content" source="media/trigger-custom-task/custom-task-extension-launch-wait.png" alt-text="Screenshot of launch and wait option for custom task extension." lightbox="media/trigger-custom-task/custom-task-extension-launch-wait.png":::

+ > [!NOTE]

+ > For more information about custom task extension behavior, see: [Lifecycle Workflow extensibility](lifecycle-workflow-extensibility.md)

+ > [!IMPORTANT]

+ > A Logic App must be configured to be compatible with the custom task extension. For more information, see [Configure a Logic App for Lifecycle Workflow use](configure-logic-app-lifecycle-workflows.md)

+1. If deployed successfully, you get confirmation on the **Logic App details** page immediately, and then you can select **Next**.

+1. On the **Review** page, you can review the details of the custom task extension and the Azure Logic App you've created. Select **Create** if the details match what you desire for the custom task extension.

+1. In the **Select tasks** side menu, select **Run a Custom Task Extension**, and select **Add**.

+1. On the custom task extension page, you can give the task a name and description. You also choose from a list of configured custom task extensions to use.

+The Lifecycle Workflows preview requires Azure AD Premium P2. For more information, see: [License requirements](what-are-lifecycle-workflows.md#license-requirements).

+This tutorial provides a step-by-step guide on how to automate prehire tasks with Lifecycle workflows using the Azure portal.

+This prehire scenario generates a temporary access pass for our new employee and sends it via email to the user's new manager.

+The Lifecycle Workflows preview requires Azure AD Premium P2. For more information, see [License requirements](what-are-lifecycle-workflows.md#license-requirements).

+- Department must be set to sales

+- Manager attribute must be set, and the manager account should have a mailbox to receive an email

+The prehire scenario can be broken down into the following:

+Use the following steps to create a prehire workflow that will generate a TAP and send it via email to the user's manager using the Azure portal.

+ 1. Sign in to Azure portal.

+ 7. Next, you'll configure the basic information about the workflow. This information includes when the workflow triggers, known as **Days from event**. So in this case, the workflow triggers two days before the employee's hire date. On the onboard prehire employee screen, add the following settings and then select **Next: Configure Scope**.

+ 8. Next, you'll configure the scope. The scope determines which users this workflow runs against. In this case, it is on all users in the Sales department. On the configure scope screen, under **Rule** add the following settings and then select **Next: Review tasks**. For a full list of supported user properties, see [Supported user properties and query parameters](/graph/api/resources/identitygovernance-rulebasedsubjectset?view=graph-rest-beta&preserve-view=true#supported-user-properties-and-query-parameters).

+ 9. On the following page, you may inspect the task if desired but no additional configuration is needed. Select **Next: Review + Create** when you're finished.

+Now that the workflow is created, it will automatically run the workflow every 3 hours. Lifecycle workflows check every 3 hours for users in the associated execution condition and execute the configured tasks for those users. However, for the tutorial, we would like to run it immediately. To run a workflow immediately, we can use the on-demand feature.

+At any time, you may monitor the status of the workflows and the tasks. As a reminder, there are three different data pivots, users runs, and tasks that are currently available in public preview. You may learn more in the how-to guide [Check the status of a workflow (preview)](check-status-workflow.md). In the course of this tutorial, we'll look at the status using the user focused reports.

+1. Once the **Workflow history (Preview)** tab has been selected, you'll land on the workflow history page as shown.

+- [Automate employee onboarding tasks before their first day of work using Lifecycle Workflows APIs](/graph/tutorial-lifecycle-workflows-onboard-custom-workflow)

+This post off-boarding scenario runs a scheduled workflow and accomplishes the following tasks:

+The Lifecycle Workflows preview requires Azure AD Premium P2. For more information, see [License requirements](what-are-lifecycle-workflows.md#license-requirements).

+As part of the prerequisites for completing this tutorial, you'll need an account that has licenses and Teams memberships that can be deleted during the tutorial. For more comprehensive instructions on how to complete these prerequisite steps, you may refer to the [Preparing user accounts for Lifecycle workflows tutorial](tutorial-prepare-azure-ad-user-accounts.md).

+ 1. Sign in to Azure portal.

+ 7. Next, you'll configure the basic information about the workflow. This information includes when the workflow triggers, known as **Days from event**. So in this case, the workflow will trigger seven days after the employee's leave date. On the post-offboarding of an employee screen, add the following settings and then select **Next: Configure Scope**.

+ 8. Next, you'll configure the scope. The scope determines which users this workflow runs against. In this case, it is on all users in the Marketing department. On the configure scope screen, under **Rule** add the following and then select **Next: Review tasks**. For a full list of supported user properties, see [Supported user properties and query parameters](/graph/api/resources/identitygovernance-rulebasedsubjectset?view=graph-rest-beta&preserve-view=true#supported-user-properties-and-query-parameters)

+ 9. On the following page, you may inspect the tasks if desired but no additional configuration is needed. Select **Next: Select users** when you're finished.

+Now that the workflow is created, it will automatically run the workflow every 3 hours. Lifecycle workflows check every 3 hours for users in the associated execution condition and execute the configured tasks for those users. However, for the tutorial, we would like to run it immediately. To run a workflow immediately, we can use the on-demand feature.

+At any time, you may monitor the status of the workflows and the tasks. As a reminder, there are three different data pivots, users runs, and tasks that are currently available in public preview. You may learn more in the how-to guide [Check the status of a workflow (preview)](check-status-workflow.md). In the course of this tutorial, we'll look at the status using the user focused reports.

+1. Once the **Workflow history (Preview)** tab has been selected, you'll land on the workflow history page as shown.

+|offsetInDays range of triggerAndScopeBasedConditions executionConditions | 180 days |

+> If creating, or updating, a workflow via API the offsetInDays range will be between -180-180 days. The negative value will signal happening before the timeBasedAttribute, while the positive value will signal happening afterwards.

+The trigger of a workflow defines when a scheduled workflow will run for users in scope for the workflow. The trigger is a combination of a time-based attribute, and an offset value. For example, if the attribute is employeeHireDate and offsetInDays is -1, then the workflow should trigger one day before the employee hire date. The value can range between -180 and 180 days.

+To preview the Lifecycle Workflows feature, you must have an Azure AD Premium P2 license in your tenant. During this preview, you're able to:

+

+- Create, manage, and delete workflows up to the total limit of 50 workflows.

+

+Azure AD also supports provisioning users into applications hosted on-premises or in a virtual machine, without having to open up any firewalls. If your application supports [SCIM](https://aka.ms/scimoverview), or you've built a SCIM gateway to connect to your legacy application, you can use the Azure AD Provisioning agent to [directly connect](https://learn.microsoft.com/azure/active-directory/app-provisioning/on-premises-scim-provisioning) with your application and automate provisioning and deprovisioning. If you have legacy applications that don't support SCIM and rely on an [LDAP](https://learn.microsoft.com/azure/active-directory/app-provisioning/on-premises-ldap-connector-configure) user store or a [SQL](https://learn.microsoft.com/azure/active-directory/app-provisioning/on-premises-sql-connector-configure) database, Azure AD can support those as well.

+- [What is inter-directory provisioning?](../hybrid/what-is-inter-directory-provisioning.md)

+>[!IMPORTANT]

+>When a group membership or ownership is activated, Azure AD PIM temporarily adds an active assignment. Azure AD PIM creates an active assignment (adds user as member or owner of the group) within seconds. When deactivation (manual or through activation time expiration) happens, Azure AD PIM removes userΓÇÖs group membership or ownership within seconds as well.

+>

+>Application may provide access to users based on their group membership. In some situations, application access may not immediately reflect the fact that user was added to the group or removed from it. If application previously cached the fact that user is not member of the group ΓÇô when user tries to access application again, access may not be provided. Similarly, if application previously cached the fact that user is member of the group ΓÇô when group membership is deactivated, user may still get access. Specific situation depends on the applicationΓÇÖs architecture. For some applications, signing out and signing back in may help to get access added or removed.

+>[!IMPORTANT]

+>When a role is activated, Azure AD PIM temporarily adds active assignment for the role. Azure AD PIM creates active assignment (assigns user to a role) within seconds. When deactivation (manual or through activation time expiration) happens, Azure AD PIM removes the active assignment within seconds as well.

+>

+>Application may provide access based on the role the user has. In some situations, application access may not immediately reflect the fact that user got role assigned or removed. If application previously cached the fact that user does not have a role ΓÇô when user tries to access application again, access may not be provided. Similarly, if application previously cached the fact that user has a role ΓÇô when role is deactivated, user may still get access. Specific situation depends on the applicationΓÇÖs architecture. For some applications, signing out and signing back in may help get access added or removed.

+When a role assignment is activated, you'll see a **Deactivate** option in the PIM portal for the role assignment. Also, you can't deactivate a role assignment within five minutes after activation.

+>[!IMPORTANT]

+>When a role is activated, Azure AD PIM temporarily adds active assignment for the role. Azure AD PIM creates active assignment (assigns user to a role) within seconds. When deactivation (manual or through activation time expiration) happens, Azure AD PIM removes the active assignment within seconds as well.

+>

+>Application may provide access based on the role the user has. In some situations, application access may not immediately reflect the fact that user got role assigned or removed. If application previously cached the fact that user does not have a role ΓÇô when user tries to access application again, access may not be provided. Similarly, if application previously cached the fact that user has a role ΓÇô when role is deactivated, user may still get access. Specific situation depends on the applicationΓÇÖs architecture. For some applications, signing out and signing back in may help get access added or removed.

+When a role assignment is activated, you'll see a **Deactivate** option in the PIM portal for the role assignment. Also, you can't deactivate a role assignment within five minutes after activation.

+While the [Cloud Native Computing Foundation][cloud-native-computing-foundation] (CNCF) owns and maintains most of the code running in AKS, Microsoft takes responsibility for building the open-source packages that we deploy on AKS. With that responsibility, it includes having complete ownership of the build, scan, sign, validate, and hotfix process and control over the binaries in container images. By us having responsibility for building the open-source packages deployed on AKS, it enables us to both establish a software supply chain over the binary, and patch the software as needed.  

+Microsoft is active in the broader Kubernetes ecosystem to help build the future of cloud-native compute in the wider CNCF community. This work not only ensures the quality of every Kubernetes release for the world, but also enables AKS quickly get new Kubernetes releases out into production for several years. In some cases, ahead of other cloud providers by several months. Microsoft collaborates with other industry partners in the Kubernetes security organization. For example, the Security Response Committee (SRC) receives, prioritizes, and patches embargoed security vulnerabilities before they're  announced to the public. This commitment ensures Kubernetes is secure for everyone, and enables AKS to patch and respond to vulnerabilities faster to keep our customers safe. In addition to Kubernetes, Microsoft has signed up to receive pre-release notifications for software vulnerabilities for products such as Envoy, container runtimes, and many other open-source projects.

+[azure-bounty-program-overview]: https://www.microsoft.com/msrc/bounty-microsoft-azure

+# Obtain the name of the resource group containing the Virtual Machine Scale set of your AKS cluster, commonly called the node resource group

+# Obtain the id of the node resource group

+# Create a role assignment granting your managed identity permissions on the node resource group

+* Permissions to collaborate in the workspace. If needed, ask an administrator of your API Management instance to assign you appropriate [roles](api-management-role-based-access-control.md#built-in-workspace-roles) in the service and the workspace.

+| API Management Service Workspace API Developer | service | Has read access to tags and products and write access to allow: <br/><br/> ▪️ Assigning APIs to products<br/> ▪️ Assigning tags to products and APIs<br/><br/> This role should be assigned on the service scope. |

+After creating a workspace, assign permissions to users to manage the workspace's resources. Each workspace user must be assigned both a service-scoped workspace RBAC role and a workspace-scoped RBAC role, or granted equivalent permissions using custom roles.

+> [!NOTE]

+> For easier management, set up Azure AD groups to assign workspace permissions to multiple users.

+>

+### Assign a service-scoped role

+1. Assign one of the following service-scoped roles to each member of the workspace:

+ * **API Management Service Workspace API Developer**

+### Assign a workspace-scoped role

+1. Assign one of the following workspace-scoped roles to the workspace members to manage workspace APIs and other resources.

+1. A central API platform team that manages the API Management instance creates a workspace and assigns permissions to workspace collaborators using RBAC roles - for example, permissions to create or read resources in the workspace.

+Workspace members must be assigned both a service-scoped role and a workspace-scoped role, or granted equivalent permissions using custom roles. The service-scoped role enables referencing service-level resources from workspace-level resources. For example, publish an API from a workspace with a service-level product, assign a service-level tag to an API, or organize a user into a workspace-level group to control API and product visibility.

+> [!NOTE]

+> For easier management, set up Azure AD groups to assign workspace permissions to multiple users.

+>

+[1(I1v2) = **$281.78**](https://azure.com/e/c2cfb6f810374f31b563e2f8a2c877e7)

+[14(I1v2) = **$3,944.92**](https://azure.com/e/a7b6240644824273bebd358c5919ae4f)

+Learn more about logging in Python apps in the series on [setting up Azure Monitor for your Python application](/azure/azure-monitor/app/opencensus-python).

+> [!IMPORTANT]

+>

+> * Managed identities eliminate the need for you to manage credentials, including Shared Access Signature (SAS) tokens.

+>

+> * Managed identities are a safer way to grant access to data without having credentials in your code.

+ Private Azure storage account access and authentication support [managed identities for Azure resources](../../active-directory/managed-identities-azure-resources/overview.md). If you have an Azure storage account, protected by a Virtual Network (VNet) or firewall, Form Recognizer can't directly access your storage account data. However, once a managed identity is enabled, Form Recognizer can access your storage account using an assigned managed identity credential.

+1. Next, you're going to assign a **Storage Blob Data Reader** role to your Form Recognizer service resource. In the **Add role assignment** pop-up window, complete the fields as follows and select **Save**:

+- **Supported distributions**: All Cloud Native Computing Foundation (CNCF) certified Kubernetes clusters.

+### 1.7.0 (March 2023)

+Flux version: [Release v0.39.0](https://github.com/fluxcd/flux2/releases/tag/v0.39.0)

+- source-controller: v0.34.0

+- kustomize-controller: v0.33.0

+- helm-controller: v0.29.0

+- notification-controller: v0.31.0

+- image-automation-controller: v0.29.0

+- image-reflector-controller: v0.24.0

+Changes made for this version:

+- Upgrades Flux to [v0.39.0](https://github.com/fluxcd/flux2/releases/tag/v0.39.0)

+- Flux extension is now supported on ARM64-based clusters

+There is no limit to how many Arc-enabled servers and VM extensions you can deploy in a resource group or subscription. The standard 800 resource limit per resource group applies to the Azure Arc Private Link Scope resource type.

+Azure Arc-enabled servers stores customer data. By default, customer data stays within the region the customer deploys the service instance in. For region with data residency requirements, customer data is always kept within the same region.

+To write to an output binding, you must apply an output binding attribute to the function method, which defined how to write to the bound service. The value returned by the method is written to the output binding. For example, the following example writes a string value to a message queue named `output-queue` by using an output binding:

+> The new programming model for authoring Functions in Python (V2) is currently in preview. Compared to the current model, the new experience is designed to be more idiomatic and intuitive for Python programmers. To learn more, see Azure Functions Python [developer guide](../functions-reference-python.md?pivots=python-mode-decorators).

+|AzureWebJobsSecretStorageType | `kubernetes` | Supported only when running the Functions runtime in Kubernetes. When `AzureWebJobsKubernetesSecretName` isn't set, the repository is considered read-only. In this case, the values must be generated before deployment. The [Azure Functions Core Tools](functions-run-local.md) generates the values automatically when deploying to Kubernetes.|

+ [EventHub("dest", Connection = "EventHubConnectionAppSetting")]IAsyncCollector<EventData> outputEvents,

+ // Do some processing:

+ string newEventBody = DoSomething(eventData);

+

+ // Queue the message to be sent in the background by adding it to the collector.

+ // If only the event is passed, an Event Hub partition to be be assigned via

+ // round-robin for each batch.

+ await outputEvents.AddAsync(new EventData(newEventBody));

+

+ // If your scenario requires that certain events are grouped together in an

+ // Event Hub partition, you can specify a partition key. Events added with

+ // the same key will always be assigned to the same partition.

+ await outputEvents.AddAsync(new EventData(newEventBody), "sample-key");

+Send messages by using a method parameter such as `out string paramName`. To write multiple messages, you can use `ICollector<EventData>` or `IAsyncCollector<EventData>` in place of `out string`. Partition keys may only be used with `IAsyncCollector<EventData>`.

+|**sessionIdleTimeout**|n/a|The maximum amount of time to wait for a message to be received for the currently active session. After this time has elapsed, the session will be closed and the function will attempt to process another session.

+* [Quarkus guide to deploying on Azure](https://quarkus.io/guides/deploying-to-azure-cloud)

+* [Zip Deploy with Run-From-Package enabled](functions-deployment-technologies.md#zip-deploy): Recommended for Azure Functions deployments.

+To request a specific Python version when you create your function app in Azure, use the `--runtime-version` option of the [`az functionapp create`](/cli/azure/functionapp#az-functionapp-create) command. The Functions runtime version is set by the `--functions-version` option. The Python version is set when the function app is created, and it can't be changed for apps running in a Consumption plan.

+| **[Dedicated plan]**³ | Manual/autoscale |10-30|

+When your function starts any tasks, callbacks, threads, processes, they must complete before your function code returns. Because Functions doesn't track these background threads, site shutdown can occur regardless of background thread status, which can cause unintended behavior in your functions.

+[wayfinding path]: /rest/api/maps/v20220901preview/wayfinding/get-path

+>

+|operation_ParentId|string|ParentId|string|

+|Infrastructure|**- Container.** Data about containers, such as [Azure Kubernetes Service](../aks/intro-kubernetes.md), [Prometheus](./essentials/prometheus-metrics-overview.md), and about the applications running inside containers.<br>**- Operating system.** Data about the guest operating system on which your application is running.|

+|[Log Analytics](logs/log-analytics-overview.md)|The Log Analytics user interface in the Azure portal helps you query the log data collected by Azure Monitor so that you can quickly retrieve, consolidate, and analyze collected data. After creating test queries, you can then directly analyze the data with Azure Monitor tools, or you can save the queries for use with visualizations or alert rules. Log Analytics workspaces are based on Azure Data Explorer, using a powerful analysis engine and the rich Kusto query language (KQL). Azure Monitor Logs uses a version of the Kusto Query Language suitable for simple log queries, and advanced functionality such as aggregations, joins, and smart analytics. You can [get started with KQL](logs/get-started-queries.md) quickly and easily.|

+Azure NetApp FilesΓÇÖ integration with Azure native services like Azure Kubernetes Service, Azure Batch, and Azure Machine Learning provides users with a seamless experience and enables them to leverage the full power of Azure's cloud-native services. This integration allows businesses to run their workloads in a scalable, secure, and highly performant environment, providing them with the confidence they need to run mission-critical workloads in the cloud.

+ "userDefinedTypes": true,

+> Visual Studio Code enables you to paste JSON as Bicep. It automatically runs the decompile command. For more information, see [Paste JSON as Bicep](./visual-studio-code.md#paste-as-bicep).

+> Visual Studio Code enables you to paste JSON as Bicep. For more information, see [Paste JSON as Bicep](./visual-studio-code.md#paste-as-bicep).

+If you define allowed values for an array parameter, the actual value can be any subset of the allowed values.

+## Paste as Bicep

+A control that's used to select a new or existing storage account.

+Storage account names must be globally unique across Azure with a length of 3-24 characters, and include only lowercase letters or numbers.

+The `StorageAccountSelector` control shows the default name for a storage account. The default is set in your code.

+The `StorageAccountSelector` control allows you to create a new storage account or select an existing storage account.

+ "label": "Storage account selector",

+ "resourceGroup": "demoRG",

+ "type": "Standard_LRS",

+ "newOrExisting": "new",

+ "kind": "StorageV2"

+- The `defaultValue.name` is required and the value is automatically validated for uniqueness. If the storage account name isn't unique, the user must specify a different name or choose an existing storage account.

+- The default value for `defaultValue.type` is **Premium_LRS**. You can set any storage account type as the default value. For example, _Standard_LRS_ or _Standard_GRS_.

+- If `options.hideExisting` is **true**, the user can't choose an existing storage account. The default value is **false**. The control only shows storage accounts as _existing_ if they are in same resource group and region as the selections made on the **Basics** tab.

+- The `kind` property displays the value if a new storage account was created, or an existing storage account's value.

+## Example

+The default values for the storage account name and type are examples. You can set your own default values for your environment.

+In the `outputs` section, the `storageSelector` output includes all the values for a storage account. The `storageKind` and `storageName` are examples of how to output specific values.

+```json

+{

+ "$schema": "https://schema.management.azure.com/schemas/0.1.2-preview/CreateUIDefinition.MultiVm.json#",

+ "handler": "Microsoft.Azure.CreateUIDef",

+ "version": "0.1.2-preview",

+ "parameters": {

+ "basics": [

+ {}

+ ],

+ "steps": [

+ {

+ "name": "StorageAccountSelector",

+ "label": "Storage account selector",

+ "elements": [

+ {

+ "name": "storageSelectorElement",

+ "type": "Microsoft.Storage.StorageAccountSelector",

+ "label": "Storage account name",

+ "toolTip": "",

+ "defaultValue": {

+ "name": "storageaccount01",

+ "type": "Premium_LRS"

+ },

+ "options": {

+ "hideExisting": false

+ },

+ "visible": true

+ }

+ ]

+ }

+ ],

+ "outputs": {

+ "location": "[location()]",

+ "storageSelector": "[steps('StorageAccountSelector').storageSelectorElement]",

+ "storageKind": "[steps('StorageAccountSelector').storageSelectorElement.kind]",

+ "storageName": "[steps('StorageAccountSelector').storageSelectorElement.name]"

+ }

+ }

+}

+```

+## Example output

+The output for a _new_ storage account.

+```json

+{

+ "location": {

+ "value": "westus3"

+ },

+ "storageSelector": {

+ "value": {

+ "name": "demostorageaccount01",

+ "resourceGroup": "demoRG",

+ "type": "Standard_GRS",

+ "newOrExisting": "new",

+ "kind": "StorageV2"

+ }

+ },

+ "storageKind": {

+ "value": "StorageV2"

+ },

+ "storageName": {

+ "value": "demostorageaccount01"

+ }

+}

+```

+The output for an _existing_ storage account.

+```json

+{

+ "location": {

+ "value": "westus3"

+ },

+ "storageSelector": {

+ "value": {

+ "name": "demostorage99",

+ "resourceGroup": "demoRG",

+ "type": "Standard_LRS",

+ "newOrExisting": "existing",

+ "kind": "StorageV2"

+ }

+ },

+ "storageKind": {

+ "value": "StorageV2"

+ },

+ "storageName": {

+ "value": "demostorage99"

+ }

+}

+```

+- For an introduction to creating UI definitions, go to [CreateUiDefinition.json for Azure managed application's create experience](create-uidefinition-overview.md).

+- For a description of common properties in UI elements, go to [CreateUiDefinition elements](create-uidefinition-elements.md).

+ Title: Azure VMware Solution known issues

+description: This article provides details about the known issues of Azure VMware Solution.

+# Known issues: Azure VMware Solution

+This article describes the currently known issues with Azure VMware Solution.

+Refer to the table below to find details about resolution dates or possible workarounds. For more information about the different feature enhancements and bug fixes in Azure VMware Solution, see [What's New](azure-vmware-solution-platform-updates.md).

+|Issue | Date discovered | Workaround | Date resolved |

+| :- | : | :- | :- |

+| [VMSA-2021-002 ESXiArgs](https://www.vmware.com/security/advisories/VMSA-2021-0002.html) OpenSLP vulnerability publicized in February 2023 | 2021 | [Disable OpenSLP service](https://kb.vmware.com/s/article/76372) | February 2021 - Resolved in [ESXi 7.0 U3c](concepts-private-clouds-clusters.md#vmware-software-versions) |

+In this article, you learned about the currently known issues with the Azure VMware Solution. For more information about the Azure VMware Solution, see:

+>[!div class="nextstepaction"]

+>[About Azure VMware Solution](introduction.md)

+All new Azure VMware Solution private clouds are being deployed with VMware NSX-T Data Center version 3.2.2. NSX-T Data Center versions in existing private clouds will be upgraded to NSX-T Data Center version 3.2.2 through April 2023.

+VMware HCX Enterprise is now available and supported on Azure VMware Solution at no extra cost. VMware HCX Enterprise brings valuable [services](https://docs.vmware.com/en/VMware-HCX/4.5/hcx-user-guide/GUID-32AF32BD-DE0B-4441-95B3-DF6A27733EED.html) like, Replicated Assisted vMotion (RAV), and Mobility Optimized Networking (MON). VMware HCX Enterprise is now automatically installed for all new VMware HCX add-on requests, and existing VMware HCX Advanced customers can upgrade to VMware HCX Enterprise using the Azure portal. Learn more on how to [Install and activate VMware HCX in Azure VMware Solution](install-vmware-hcx.md).

+ >This is non-disruptive and should not impact the Azure VMware Solution service or workloads. During maintenance, various VMware vSphere alerts, such as _Lost network connectivity on DVPorts_ and _Lost uplink redundancy on DVPorts_, appear in vCenter Server and clear automatically as the maintenance progresses.

+ > [!NOTE]

+ > The Azure key vault Managed HSM option is only supported with the Key URI option.

+ Title: Migrate Microsoft SQL Server Always-On cluster to Azure VMware Solution

+description: Learn how to migrate Microsoft SQL Server Always-On cluster to Azure VMware Solution.

+# Migrate Microsoft SQL Server Always-On cluster to Azure VMware Solution

+In this article, you learn how to migrate Microsoft SQL Server Always-On Cluster to Azure VMware Solution. For VMware HCX, you can follow the VMware vMotion migration procedure.

+## Prerequisites

+These are the prerequisites to migrating your SQL server instance to Azure VMware Solution.

+- Review and record the storage and network configuration of every node in the cluster.

+- Backup the full database.

+- Backup the virtual machine running the Microsoft SQL Server instance.

+- Remove the virtual machine from any VMware vSphere Distributed Resource Scheduler (DRS) groups and rules.

+- VMware HCX must be configured between your on-premises datacenter and the Azure VMware Solution private cloud that runs the migrated workloads. For more information on how to configure HCX, see [Azure VMware Solution documentation](install-vmware-hcx.md).

+- Ensure that all the network segments in use by the Microsoft SQL Server are extended into your Azure VMware Solution private cloud. To verify this step, see [Configure VMware HCX network extension](configure-hcx-network-extension.md).

+VMware HCX over VPN is supported in Azure VMware Solution for workload migration. However, due to the size of database workloads, VMware HCX over VPN is not recommended for Microsoft SQL Server Always-On migrations for production workloads. ExpressRoute connectivity is recommended as more performant and reliable. For Microsoft SQL Server Standalone and non-production workloads this may be suitable, depending upon the size of the database, to migrate.

+Microsoft SQL Server (2019 and 2022) was tested with Windows Server (2019 and 2022) Data Center edition with the virtual machines deployed in the on-premises environment. Windows Server and SQL Server have been configured following best practices and recommendations from Microsoft and VMware. The on-premises source infrastructure was VMware vSphere 7.0 Update 3 and VMware vSAN running on Dell PowerEdge servers and Intel Optane P4800X SSD NVMe devices.

+## Downtime considerations

+Predicting downtime during a migration depends upon the size of the database to be migrated and the speed of the private network connection to Azure cloud. Always-On migrations are intended to be executed with low database downtime. However, plan to conduct the migration during off-peak hours within a pre-approved change window.

+The table below indicates the estimated downtime for each Microsoft SQL Server topology.

+| **Scenario** | **Downtime expected** | **Notes** |

+|:|:--|:--|

+| **Standalone instance** | LOW | Migrate with VMware vMotion, the DB is available during migration, but it is not recommended to commit any critical data during it. |

+| **Always-On Availability Group** | LOW | The primary replica will always be available during the migration of the first secondary replica and the secondary replica will become the primary after the initial failover to Azure. |

+| **Failover Cluster Instance** | HIGH | All nodes of the cluster are shutdown and migrated using VMware HCX Cold Migration. Downtime duration depends upon database size and private network speed to Azure cloud. |

+## Windows Server Failover Cluster quorum considerations

+Microsoft SQL Server Always-On Availability Groups rely on Windows Server Failover Cluster, which requires a quorum voting mechanism to maintain the coherence of the cluster.

+An odd number of voting elements is required, which is achieved by an odd number of nodes in the cluster or by using a witness. Witness can be configured in three different ways:

+- Disk witness

+- File share witness

+- Cloud witness

+If the cluster uses **Disk witness**, then the disk must be migrated with the rest of cluster shared storage using the procedure described in this document.

+If the cluster uses a **File share witness** running on-premises, then the type of witness for your migrated cluster depends upon the Azure VMware Solution scenario, there are several options to consider.

+- **Datacenter Extension**: Maintain the file share witness on-premises. Your workloads are distributed across your datacenter and Azure. Therefore the connectivity between your datacenter and Azure should always be available. In any case, take into consideration bandwidth constraints and plan accordingly.

+- **Datacenter Exit**: For this scenario, there are two options. In both options, you can maintain the file share witness on-premises during the migration in case you need to do rollback during the process.

+ - Deploy a new **File share witness** in your Azure VMware Solution private cloud.

+ - Deploy a **Cloud witness** running in Azure Blob Storage in the same region as the Azure VMware Solution private cloud.

+- **Disaster Recovery and Business Continuity**: For a disaster recovery scenario, the best and most reliable option is to create a **Cloud Witness** running in Azure Storage.

+- **Application Modernization**: For this use case, the best option is to deploy a **Cloud Witness**.

+For details about configuring and managing the quorum, see [Failover Clustering documentation](https://learn.microsoft.com/windows-server/failover-clustering/manage-cluster-quorum). For information about deployment of Cloud witness in Azure Blob Storage, see [Manage a cluster quorum for a Failover Cluster](https://learn.microsoft.com/windows-server/failover-clustering/deploy-cloud-witness).

+## Migrate Microsoft SQL Server Always-On cluster

+1. Access your Always-On cluster with SQL Server Management Studio using administration credentials.

+ 1. Select your primary replica and open **Availability Group** **Properties**.

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-always-on-1.png" alt-text="Diagram showing Always On Availability Group properties." border="false" lightbox="media/sql-server-hybrid-benefit/sql-always-on-1.png":::

+ 1. Change **Availability Mode** to **Asynchronous commit** only for the replica to be migrated.

+ 1. Change **Failover Mode** to **Manual** for every member of the availability group.

+1. Access the on-premises vCenter Server and proceed to HCX area.

+1. Under **Services** select **Migration** > **Migrate**.

+ 1. Select one virtual machine running the secondary replica of the database the is going to be migrated.

+ 1. Set the vSphere cluster in the remote private cloud to run the migrated SQL cluster as the **Compute Container**.

+ 1. Select the **vSAN Datastore** as remote storage.

+ 1. Select a folder. This not mandatory, but is recommended to separate the different workloads in your Azure VMware Solution private cloud.

+ 1. Keep **Same format as source**.

+ 1. Select **vMotion** as **Migration profile**.

+ 1. In **Extended Options** select **Migrate Custom Attributes**.

+ 1. Verify that on-premises network segments have the correct remote stretched segment in Azure.

+ 1. Select **Validate** and ensure that all checks are completed with pass status. The most common error is related to the storage configuration. Verify again that there are no virtual SCSI controllers have the physical sharing setting.

+ 1. Click **Go** to start the migration.

+1. Once the migration has been completed, access the migrated replica and verify connectivity with the rest of the members in the availability group.

+1. In SQL Server Management Studio, open the **Availability Group Dashboard** and verify that the replica appears as **Online**.

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-always-on-2.png" alt-text="Diagram showing Always On Availability Group Dashboard." border="false" lightbox="media/sql-server-hybrid-benefit/sql-always-on-2.png":::

+

+ 1. **Data Loss** status in the **Failover Readiness** column is expected since the replica has been out-of-sync with the primary during the migration.

+1. Edit the **Availability Group** **Properties** again and set **Availability Mode** back to **Synchronous commit**.

+ 1. The secondary replica starts to synchronize back all the changes made to the primary replica during the migration. Wait until it appears in Synchronized state.

+1. From the **Availability Group Dashboard** in SSMS click on **Start Failover Wizard**.

+1. Select the migrated replica and click **Next**.

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-always-on-3.png" alt-text="Diagram showing new primary replica selection for always on." border="false" lightbox="media/sql-server-hybrid-benefit/sql-always-on-3.png":::

+1. Connect to the replica in the next screen with your DB admin credentials.

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-always-on-4.png" alt-text="Diagram showing new primary replica admin credentials connection." border="false" lightbox="media/sql-server-hybrid-benefit/sql-always-on-4.png":::

+

+1. Review the changes and click **Finish** to start the failover operation.

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-always-on-5.png" alt-text="Diagram showing Availability Group always on operation review." border="false" lightbox="media/sql-server-hybrid-benefit/sql-always-on-5.png":::

+

+1. Monitor the progress of the failover in the next screen, and click **Close** when the operation is finished.

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-always-on-6.png" alt-text="Diagram showing that always on SQL server cluster successfully finished." border="false" lightbox="media/sql-server-hybrid-benefit/sql-always-on-6.png":::

+1. Refresh the **Object Explorer** view in SQL Server Management Studio (SSMS), and verify that the migrated instance is now the primary replica.

+1. Repeat steps 1 to 6 for the rest of the replicas of the availability group.

+

+ >[!Note]

+ > Migrate one replica at a time and verify that all changes are synchronized back to the replica after each migration. Do not migrate all the replicas at the same time using **HCX Bulk Migration**.

+1. After the migration of all the replicas is completed, access your Always-On availability group with **SQL Server Management Studio**.

+ 1. Open the Dashboard and verify there is no data loss in any of the replicas and that all are in a **Synchronized** state.

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-always-on-7.png" alt-text="Diagram showing availability Group Dashboard with new primary replica and all migrated secondary replicas in synchronized state." border="false" lightbox="media/sql-server-hybrid-benefit/sql-always-on-7.png":::

+ 1. Edit the **Properties** of the availability group and set **Failover Mode** to **Automatic** in all replicas.

+

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-always-on-8.png" alt-text="Diagram showing a setting for failover back to Automatic for all replicas." border="false" lightbox="media/sql-server-hybrid-benefit/sql-always-on-8.png":::

+## Next steps

+[Enable SQL Azure hybrid benefit for Azure VMware Solution](enable-sql-azure-hybrid-benefit.md).

+[Create a placement policy in Azure VMware Solution](create-placement-policy.md)

+[Windows Server Failover Clustering Documentation](https://learn.microsoft.com/windows-server/failover-clustering/failover-clustering-overview)

+[Microsoft SQL Server 2019 Documentation](https://learn.microsoft.com/sql/sql-server/)

+[Microsoft SQL Server 2022 Documentation](https://learn.microsoft.com/sql/sql-server/)

+[Windows Server Technical Documentation](https://learn.microsoft.com/windows-server/)

+[Planning Highly Available, Mission Critical SQL Server Deployments with VMware vSphere](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/vmware-vsphere-highly-available-mission-critical-sql-server-deployments.pdf)

+[Microsoft SQL Server on VMware vSphere Availability and Recovery Options](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/sql-server-on-vmware-availability-and-recovery-options.pdf)

+[VMware KB 100 2951 ΓÇô Tips for configuring Microsoft SQL Server in a virtual machine](https://kb.vmware.com/s/article/1002951)

+[Microsoft SQL Server 2019 in VMware vSphere 7.0 Performance Study](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere7-sql-server-perf.pdf)

+[Architecting Microsoft SQL Server on VMware vSphere ΓÇô Best Practices Guide](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/sql-server-on-vmware-best-practices-guide.pdf)

+[Setup for Windows Server Failover Cluster in VMware vSphere 7.0](https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-703-setup-wsfc.pdf)

+ Title: Migrate SQL Server failover cluster to Azure VMware Solution

+description: Learn how to migrate SQL Server failover cluster to Azure VMware Solution

+# Migrate SQL Server failover cluster to Azure VMware Solution

+In this article, you'll learn how to migrate a Microsoft SQL Server Failover cluster instance to Azure VMware Solution. Currently Azure VMware Solution service doesn't support VMware Hybrid Linked Mode to connect an on-premises vCenter Server with one running in Azure VMware Solution. Due to this constraint, this process requires the use of VMware HCX for the migration. For more details about configuring HCX, see [Install and activate VMware HCX in Azure VMware Solution](install-vmware-hcx.md).

+VMware HCX doesn't support migrating virtual machines with SCSI controllers in physical sharing mode attached to a virtual machine. However, you can overcome this limitation by performing the steps shown in this procedure and by using VMware HCX Cold Migration to move the different virtual machines that make up the cluster.

+> [!NOTE]

+> This procedure requires a full shutdown of the cluster. Since the Microsoft SQL Server service will be unavailable during the migration, plan accordingly for the downtime period .

+## Prerequisites

+- Review and record the storage and network configuration of every node in the cluster.

+- Review and record the WSFC configuration.

+- Back up the database(s) being executed in the cluster.

+- Back up the cluster virtual machines.

+- Remove all cluster node VMs from any Distributed Resource Scheduler (DRS) groups and rules they're part of.

+- VMware HCX must be configured between your on-premises datacenter and the Azure VMware Solution private cloud that runs the migrated workloads. For more details about installing VMware HCX, see [Azure VMware Solution documentation](install-vmware-hcx.md).

+- Ensure that all the network segments in use by the Microsoft SQL Server are extended into your Azure VMware Solution private cloud. To verify this step, see [Configure VMware HCX network extension](configure-hcx-network-extension.md).

+VMware HCX over VPN is supported in Azure VMware Solution for workload migration. However, due to the size of database workloads it isn't recommended for Microsoft SQL Server Failover Cluster Instance and Microsoft SQL Server Always-On migrations, especially for production workloads. ExpressRoute connectivity is recommended as more performant and reliable. For Microsoft SQL Server Standalone and non-production workloads this can be suitable, depending upon the size of the database, to migrate.

+Microsoft SQL Server 2019 and 2022 were tested with Windows Server 2019 and 2022 Data Center edition with the virtual machines deployed in the on-premises environment. Windows Server and SQL Server have been configured following best practices and recommendations from Microsoft and VMware. The on-premises source infrastructure was VMware vSphere 7.0 Update 3 and VMware vSAN running on Dell PowerEdge servers and Intel Optane P4800X SSD NVMe devices.

+## Downtime considerations

+Predicting downtime during a migration will depend upon the size of the database to be migrated and the speed of the private network connection to Azure cloud. Migration of SQL Server Failover Cluster Instances Always On to Azure VMware Solution requires a full downtime of the database and all cluster nodes, however you should plan for the migration to be executed during off-peak hours with an approved change window.

+The table below indicates the downtime for each Microsoft SQL Server topology.

+| **Scenario** | **Downtime expected** | **Notes** |

+|:|:--|:--|

+| **Standalone instance** | LOW | Migration will be done using vMotion, the DB will be available during migration time, but it isn't recommended to commit any critical data during it. |

+| **Always-On Availability Group** | LOW | The primary replica will always be available during the migration of the first secondary replica and the secondary replica will become the primary after the initial failover to Azure. |

+| **Failover Cluster Instance** | HIGH | All nodes of the cluster will be shut down and migrated using VMware HCX Cold Migration. Downtime duration will depend upon database size and private network speed to Azure cloud. |

+## Windows Server Failover Cluster quorum considerations

+Windows Server Failover Cluster requires a quorum mechanism to maintain the cluster.

+Use an odd number of voting elements to achieve by an odd number of nodes in the cluster or by using a witness. Witnesses can be configured in three different forms:

+- Disk witness

+- File share witness

+- Cloud witness

+If the cluster uses **Disk witness**, then the disk must be migrated with the cluster shared storage using the [Migrate fail over cluster](#migrate-failover-cluster).

+If the cluster uses a **File** **share witness** running on-premises, then the type of witness for your migrated cluster depends on the Azure VMware Solution scenario:

+- **Datacenter Extension**: Maintain the file share witness on-premises. Your workloads are distributed across your datacenter and Azure VMware Solution, therefore connectivity between both should always be available. In any case take into consideration bandwidth constraints and plan accordingly.

+- **Datacenter Exit**: For this scenario, there are two options. In both cases, you can maintain the file share witness on-premises during the migration in case you need to do roll back.

+ - Deploy a new **File share witness** in your Azure VMware Solution private cloud.

+ - Deploy a **Cloud witness** running in Azure Blob Storage in the same region as the Azure VMware Solution private cloud.

+- Disaster Recovery and Business Continuity: For a disaster recovery scenario, the best and most reliable option is to create a **Cloud Witness** running in Azure Storage.

+- Application Modernization: For this use case, the best option is to deploy a **Cloud Witness**.

+For more information about quorum configuration and management, see [Failover Clustering documentation](https://learn.microsoft.com/windows-server/failover-clustering/manage-cluster-quorum). For more information about deploying a Cloud witness in Azure Blob Storage, see [Deploy a Cloud Witness for a Failover Cluster](https://learn.microsoft.com/windows-server/failover-clustering/deploy-cloud-witness) documentation for the details.

+## Migrate failover cluster

+For illustration purposes, in this document we're using a two-node cluster with Windows Server 2019 Datacenter and SQL Server 2019 Enterprise. Windows Server 2022 and SQL Server 2022 are also supported with this procedure.

+1. From vSphere Client shutdown the second node of the cluster.

+1. Access the first node of the cluster and open **Failover Cluster Manager**.

+ 1. Verify that the second node is in **Offline** state and that all clustered services and storage are under the control of the first node.

+

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-failover-1.png" alt-text="Diagram showing Windows Server Failover Cluster Manager cluster storage verification." border="false" lightbox="media/sql-server-hybrid-benefit/sql-failover-1.png":::

+

+ 1. Shut down the cluster.

+

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-failover-2.png" alt-text="Diagram showing a shut down cluster using Windows Server Failover Cluster Manager." border="false" lightbox="media/sql-server-hybrid-benefit/sql-failover-2.png":::

+

+ 1. Check that all cluster services are successfully stopped without errors.

+1. Shut down first node of the cluster.

+1. From the **vSphere Client**, edit the settings of the second node of the cluster.

+ 1. Remove all shared disks from the virtual machine configuration.

+ 1. Ensure that the **Delete files from datastore** checkbox isn't selected as this will permanently delete the disk from the datastore, and you'll need to recover the cluster from a previous backup.

+ 1. Set **SCSI Bus Sharing** from **Physical** to **None** in the virtual SCSI controllers used for the shared storage. Usually, these controllers are of VMware Paravirtual type.

+1. Edit the first node virtual machine settings. Set **SCSI Bus Sharing** from **Physical** to **None** in the SCSI controllers.

+

+1. From the vSphere Client,** go to the HCX plugin area. Under **Services**, select **Migration** > **Migrate**.

+ 1. Select the second node virtual machine.

+ 1. Set the vSphere cluster in the remote private cloud that will run the migrated SQL cluster as the **Compute Container**.

+ 1. Select the **vSAN Datastore** as remote storage.

+ 1. Select a folder if you want to place the virtual machines in specific folder, this not mandatory but is recommended to separate the different workloads in your Azure VMware Solution private cloud.

+ 1. Keep **Same format as source**.

+ 1. Select **Cold migration** as **Migration profile**.

+ 1. In **Extended** **Options** select **Migrate Custom Attributes**.

+ 1. Verify that on-premises network segments have the correct remote stretched segment in Azure.

+ 1. Select **Validate** and ensure that all checks are completed with pass status. The most common error here will be one related to the storage configuration. Verify again that there are no SCSI controllers with physical sharing setting.

+ 1. Select **Go** and the migration will initiate.

+1. Repeat the same process for the first node.

+1. Access **Azure VMware Solution vSphere Client** and edit the first node settings and set back to physical SCSI Bus sharing the SCSI controller(s) managing the shared disks.

+1. Edit node 2 settings in **vSphere Client**.

+ 1. Set SCSI Bus sharing back to physical in the SCSI controller managing shared storage.

+ 1. Add the cluster shared disks to the node as additional storage. Assign them to the second SCSI controller.

+ 1. Ensure that all the storage configuration is the same as the one recorded before the migration.

+1. Power on the first node virtual machine.

+1. Access the first node VM with **VMware Remote Console**.

+ 1. Verify virtual machine network configuration and ensure it can reach on-premises and Azure resources.

+ 1. Open **Failover Cluster Manager** and verify cluster services.

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-failover-3.png" alt-text="Diagram showing a cluster summary in Failover Cluster Manager." border="false" lightbox="media/sql-server-hybrid-benefit/sql-failover-3.png":::

+1. Power on the second node virtual machine.

+1. Access the second node VM from the **VMware Remote Console**.

+ 1. Verify that Windows Server can reach the storage.

+ 1. In the **Failover Cluster Manager** review that the second node appears as **Online** status.

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-failover-4.png" alt-text="Diagram showing a cluster node status in Failover Cluster Manager." border="false" lightbox="media/sql-server-hybrid-benefit/sql-failover-4.png":::

+1. Using the **SQL Server Management Studio** connect to the SQL Server cluster resource network name. Check the database is online and accessible.

+

+Finally, check the connectivity to SQL from other systems and applications in your infrastructure and verify that all applications using the database(s) can still access them.

+## Next steps

+- [Enable SQL Azure hybrid benefit for Azure VMware Solution](enable-sql-azure-hybrid-benefit.md).

+- [Create a placement policy in Azure VMware Solution](create-placement-policy.md)

+- [Windows Server Failover Clustering Documentation](https://learn.microsoft.com/windows-server/failover-clustering/failover-clustering-overview)

+- [Microsoft SQL Server 2019 Documentation](https://learn.microsoft.com/sql/sql-server/?view=sql-server-ver15)

+- [Microsoft SQL Server 2022 Documentation](https://learn.microsoft.com/sql/sql-server/?view=sql-server-ver16)

+- [Windows Server Technical Documentation](https://learn.microsoft.com/windows-server/)

+- [Planning Highly Available, Mission Critical SQL Server Deployments with VMware vSphere](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/vmware-vsphere-highly-available-mission-critical-sql-server-deployments.pdf)

+- [Microsoft SQL Server on VMware vSphere Availability and Recovery Options](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/sql-server-on-vmware-availability-and-recovery-options.pdf)

+- [VMware KB 100 2951 ΓÇô Tips for configuring Microsoft SQL Server in a virtual machine](https://kb.vmware.com/s/article/1002951)

+- [Microsoft SQL Server 2019 in VMware vSphere 7.0 Performance Study](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere7-sql-server-perf.pdf)

+- [Architecting Microsoft SQL Server on VMware vSphere ΓÇô Best Practices Guide](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/sql-server-on-vmware-best-practices-guide.pdf)

+- [Setup for Windows Server Failover Cluster in VMware vSphere 7.0](https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-703-setup-wsfc.pdf)

+ Title: Migrate Microsoft SQL Server Standalone to Azure VMware Solution

+description: Learn how to migrate Microsoft SQL Server Standalone to Azure VMware Solution.

+# Migrate Microsoft SQL Server Standalone to Azure VMware Solution

+In this article, you learn how to migrate Microsoft SQL Server standalone to Azure VMware Solution.

+When migrating Microsoft SQL Server Standalone to Azure VMware Solution, VMware HCX offers two migration profiles that can be used:

+- HCX vMotion

+- HCX Cold Migration

+In both cases, consider the size and criticality of the database being migrated. For this how-to procedure, we have validated VMware HCX vMotion. VMware HCX Cold Migration is also valid, but it requires a longer downtime period.

+## Prerequisites

+- Review and record the storage and network configuration of every node in the cluster.

+- Back up the full database.

+- Back up the virtual machine running the Microsoft SQL Server instance.

+- Remove all cluster node VMs from any Distributed Resource Scheduler (DRS) groups and rules.

+- Configure VMware HCX between your on-premises datacenter and the Azure VMware Solution private cloud that runs the migrated workloads. For more information about configuring VMware HCX, see [Azure VMware Solution documentation](install-vmware-hcx.md) .

+- Ensure that all the network segments in use by the Microsoft SQL Server are extended into your Azure VMware Solution private cloud.For verify this step in the procedure, see [Configure VMware HCX network extension](configure-hcx-network-extension.md).

+VMware HCX over VPN is supported in Azure VMware Solution for workload migration. However, due to the size of database workloads, VMware HCX over VPN isn't recommended for Microsoft SQL Server Failover Cluster Instance and Microsoft SQL Server Always-On migrations, especially for production workloads. ExpressRoute connectivity is recommended as more performant and reliable. For Microsoft SQL Server Standalone and non-production workloads HCX over VPN can be suitable, depending on the size of the database, to migrate.

+Microsoft SQL Server (2019 and 2022) were tested with Windows Server (2019 and 2022) Data Center edition with the virtual machines deployed in the on-premises environment. Windows Server and SQL Server have been configured following best practices and recommendations from Microsoft and VMware. The on-premises source infrastructure was VMware vSphere 7.0 Update 3 and VMware vSAN running on Dell PowerEdge servers and Intel Optane P4800X SSD NVMe devices.

+## Downtime considerations

+Predicting downtime during a migration depends upon the size of the database to be migrated and the speed of the private network connection to Azure cloud. Migration of SQL Server standalone instance doesn't require database downtime since it will be done using the VMware HCX vMotion mechanism. We recommend the migration during off-peak hours with an pre-approved change window.

+This table indicates the estimated downtime for each Microsoft SQL Server topology.

+| **Scenario** | **Downtime expected** | **Notes** |

+|:|:--|:--|

+| **Standalone instance** | LOW | Migration is done using VMware vMotion, the DB is available during migration time, but it isn't recommended to commit any critical data during it. |

+| **Always-On Availability Group** | LOW | The primary replica will always be available during the migration of the first secondary replica and the secondary replica will become the primary after the initial failover to Azure. |

+| **Failover Cluster Instance** | HIGH | All nodes of the cluster are shutdown and migrated using VMware HCX Cold Migration. Downtime duration depends upon database size and private network speed to Azure cloud. |

+## Migrate Microsoft SQL Server standalone

+1. Log into your on-premises **vCenter Server** and access the VMware HCX plugin.

+1. Under **Services** select **Migration** > **Migrate**.

+ a. Select the Microsoft SQL Server virtual machine.

+ a. Set the vSphere cluster in the remote private cloud of the migrated SQL cluster as the **Compute Container**.

+ a. Select the vSAN Datastore as remote storage.

+ a. Select a folder. This isn't mandatory, but we recommended separating the different workloads in your Azure VMware Solution private cloud.

+ a. Keep **Same format as source**.

+ a. Select **vMotion** as Migration profile.

+ a. In **Extended Options** select **Migrate Custom Attributes**.

+ a. Verify that on-premises network segments have the correct remote stretched segment in Azure VMware Solution.

+ a. Select **Validate** and ensure that all checks are completed with pass status.

+ a. Select **Go** to start the migration.

+1. After the migration has completed, access the virtual machine using VMware Remote Console in the vSphere Client.

+ a. Verify the network configuration and check connectivity both with on-premises and Azure VMware Solution resources.

+ a. Using SQL Server Management Studio verify you can access the database.

+ :::image type="content" source="media/sql-server-hybrid-benefit/sql-standalone-1.png" alt-text="Diagram showing a SQL Server Management Studio connection to the migrated database." border="false" lightbox="media/sql-server-hybrid-benefit/sql-standalone-1.png":::

+## Next steps

+- [Enable SQL Azure hybrid benefit for Azure VMware Solution](enable-sql-azure-hybrid-benefit.md).

+- [Create a placement policy in Azure VMware Solution](create-placement-policy.md)

+- [Windows Server Failover Clustering Documentation](https://learn.microsoft.com/windows-server/failover-clustering/failover-clustering-overview)

+- [Microsoft SQL Server 2019 Documentation](https://learn.microsoft.com/sql/sql-server/?view=sql-server-ver15)

+- [Microsoft SQL Server 2022 Documentation](https://learn.microsoft.com/sql/sql-server/?view=sql-server-ver16)

+- [Windows Server Technical Documentation](https://learn.microsoft.com/windows-server/)

+- [Planning Highly Available, Mission Critical SQL Server Deployments with VMware vSphere](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/vmware-vsphere-highly-available-mission-critical-sql-server-deployments.pdf)

+- [Microsoft SQL Server on VMware vSphere Availability and Recovery Options](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/sql-server-on-vmware-availability-and-recovery-options.pdf)

+- [VMware KB 100 2951 ΓÇô Tips for configuring Microsoft SQL Server in a virtual machine](https://kb.vmware.com/s/article/1002951)

+- [Microsoft SQL Server 2019 in VMware vSphere 7.0 Performance Study](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere7-sql-server-perf.pdf)

+- [Architecting Microsoft SQL Server on VMware vSphere ΓÇô Best Practices Guide](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/sql-server-on-vmware-best-practices-guide.pdf)

+- [Setup for Windows Server Failover Cluster in VMware vSphere 7.0](https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-703-setup-wsfc.pdf)

+ Title: Azure Hybrid benefit for Windows server, SQL server, or Linux subscriptions

+description: Learn about Azure Hybrid benefit for Windows server, SQL server, or Linux subscriptions.

+# Azure Hybrid benefit for Windows server, SQL server, or Linux subscriptions

+Azure Hybrid benefit is a cost saving offering from Microsoft you can use to save on cost while optimizing your hybrid environment by applying your existing Windows Server, SQL Server licenses or Linux subscriptions.

+- Save up to 85% over standard pay-as-you-go rate leveraging Windows Server and SQL Server licenses with Azure Hybrid benefit.

+- Use Azure Hybrid Benefit in Azure SQL platform as a service (PaaS) environment.

+- Apply to SQL Server one to four vCPUs exchange: For every one core of SQL Server Enterprise Edition, you get four vCPUs of SQL Managed Instance or Azure SQL Database general purpose and Hyperscale tiers, or 4 vCPUs of SQL Server Standard edition on Azure VMs.

+- Use existing SQL licensing to adopt Azure ArcΓÇôenabled SQL Managed Instance.

+- Help meet compliance requirements with unlimited virtualization on Azure Dedicated Host and the Azure VMware Solution.

+- Get 180 days of dual-use rights between on-premises and Azure.

+## Microsoft SQL server

+Microsoft SQL Server is a core component of many business-critical applications currently running on VMware vSphere and is one of the most widely used database platforms in the market with customers running hundreds of SQL Server instances with VMware vSphere on-premises.

+Azure VMware Solution is an ideal solution for customers looking to migrate and modernize their vSphere-based applications to the cloud, including their Microsoft SQL databases.

+## Next steps

+Now that you've covered Azure Hybrid benefit, you may want to learn about:

+- [Migrate Microsoft SQL Server Standalone to Azure VMware Solution](migrate-sql-server-standalone-cluster.md)

+- [Migrate SQL Server failover cluster to Azure VMware Solution](migrate-sql-server-failover-cluster.md)

+- [Migrate Microsoft SQL Server Always-On cluster to Azure VMware Solution](migrate-sql-server-always-on-cluster.md)

+| Workloads | Generally available |

+### My privatelink.azure.com cannot resolve to management.privatelink.azure.com

+### Limitations

+### Limitations

+Currently, the Windows agent doesn't reduce memory pressure when other applications increase their memory usage. If the overall memory usage exceeds 100%, the Windows agent may crash.

+* Some Speech Studio [scenarios](speech-studio-overview.md#speech-studio-scenarios) are available to try without an Azure subscription.

+For a demonstration of these scenarios in Speech Studio, view this [introductory video](https://youtu.be/mILVerU6DAw).

+> [!VIDEO https://www.youtube.com/embed/mILVerU6DAw]

+ <voice name="string" effect="string">

+Some examples of contents that are allowed in each element are described in the following list:

+| `effect` |The audio effect processor that's used to optimize the quality of the synthesized speech output for specific scenarios on devices. <br/><br/>For some scenarios in production environments, the auditory experience may be degraded due to the playback distortion on certain devices. For example, the synthesized speech from a car speaker may sound dull and muffled due to environmental factors such as speaker response, room reverberation, and background noise. The passenger might have to turn up the volume to hear more clearly. To avoid manual operations in such a scenario, the audio effect processor can make the sound clearer by compensating the distortion of playback.<br/><br/>The following values are supported:<br/><ul><li>`eq_car` ΓÇô Optimize the auditory experience when providing high-fidelity speech in cars, buses, and other enclosed automobiles.</li><li>`eq_telecomhp8k` ΓÇô Optimize the auditory experience for narrowband speech in telecom or telephone scenarios. We recommend a sampling rate of 8 kHz. If the sample rate isn't 8 kHz, the auditory quality of the output speech won't be optimized.</li></ul><br/>If the value is missing or invalid, this attribute will be ignored and no effect will be applied.| Optional |

+#### Audio effect example

+You use the `effect` attribute to optimize the auditory experience for scenarios such as cars and telecommunications. The following SSML example uses the `effect` attribute with the configuration in car scenarios.

+

+```xml

+<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US">

+ <voice name="en-US-JennyNeural" effect="eq_car">

+ This is the text that is spoken.

+ </voice>

+</speak>

+```

+ :::code language="json" source="../../../../../cognitive-services-rest-samples/curl/Translator/translate-with-glossary.json" range="1-23" highlight="13-14":::

+

+ > [!NOTE]

+ > The example used an enabled [**system-assigned managed identity**](create-use-managed-identities.md#enable-a-system-assigned-managed-identity) with a [**Storage Blob Data Contributor**](create-use-managed-identities.md#grant-access-to-your-storage-account) role assignment for authorization. For more information, *see* [**Managed identities for Document Translation**](./create-use-managed-identities.md).

+ :::image type="content" source="../media/managed-identity-rbac-flow.png" alt-text="Screenshot of managed identity flow (RBAC).":::

+> [!IMPORTANT]

+> * Currently, Document Translation doesn't support managed identity in the global region. If you intend to use managed identities for Document Translation operations, [create your Translator resource](https://portal.azure.com/#create/Microsoft.CognitiveServicesTextTranslation) in a non-global Azure region.

+>

+> * Document Translation is **only** available in the S1 Standard Service Plan (Pay-as-you-go) or in the D3 Volume Discount Plan. _See_ [Cognitive Services pricingΓÇöTranslator](https://azure.microsoft.com/pricing/details/cognitive-services/translator/).

+>

+* If successful, the POST method returns a `202 Accepted` response code and the service creates a batch request.

+ Title: How to use autolabeling in custom named entity recognition

+description: Learn how to use autolabeling in custom named entity recognition.

+# How to use autolabeling for Custom Named Entity Recognition

+[Labeling process](tag-data.md) is an important part of preparing your dataset. Since this process requires both time and effort, you can use the autolabeling feature to automatically label your entities. You can start autolabeling jobs based on a model you've previously trained or using GPT models. With autolabeling based on a model you've previously trained, you can start labeling a few of your documents, train a model, then create an autolabeling job to produce entity labels for other documents based on that model. With autolabeling with GPT, you may immediately trigger an autolabeling job without any prior model training. This feature can save you the time and effort of manually labeling your entities.

+### [Autolabel based on a model you've trained](#tab/autolabel-model)

+Before you can use autolabeling based on a model you've trained, you need:

+* A successfully [created project](create-project.md) with a configured Azure blob storage account.

+* Text data that [has been uploaded](design-schema.md#data-preparation) to your storage account.

+* [Labeled data](tag-data.md)

+* A [successfully trained model](train-model.md)

+### [Autolabel with GPT](#tab/autolabel-gpt)

+Before you can use autolabeling with GPT, you need:

+* A successfully [created project](create-project.md) with a configured Azure blob storage account.

+* Text data that [has been uploaded](design-schema.md#data-preparation) to your storage account.

+* Entity names that are meaningful. The GPT models label entities in your documents based on the name of the entity you've provided.

+* [Labeled data](tag-data.md) isn't required.

+* An Azure OpenAI [resource and deployment](../../../openai/how-to/create-resource.md).

+## Trigger an autolabeling job

+### [Autolabel based on a model you've trained](#tab/autolabel-model)

+When you trigger an autolabeling job based on a model you've trained, there's a monthly limit of 5,000 text records per month, per resource. This means the same limit applies on all projects within the same resource.

+1. From the left navigation menu, select **Data labeling**.

+2. Select the **Autolabel** button under the Activity pane to the right of the page.

+

+3. Choose Autolabel based on a model you've trained and click on Next.

+ :::image type="content" source="../media/choose-models.png" alt-text="A screenshot showing model choice for auto labeling." lightbox="../media/choose-models.png":::

+

+4. Choose a trained model. It's recommended to check the model performance before using it for autolabeling.

+ :::image type="content" source="../media/choose-model-trained.png" alt-text="A screenshot showing how to choose trained model for autotagging." lightbox="../media/choose-model-trained.png":::

+5. Choose the entities you want to be included in the autolabeling job. By default, all entities are selected. You can see the total labels, precision and recall of each entity. It's recommended to include entities that perform well to ensure the quality of the automatically labeled entities.

+6. Choose the documents you want to be automatically labeled. The number of text records of each document is displayed. When you select one or more documents, you should see the number of texts records selected. It's recommended to choose the unlabeled documents from the filter.

+ > * If an entity was automatically labeled, but has a user defined label, only the user defined label is used and visible.

+7. Select **Autolabel** to trigger the autolabeling job.

+You should see the model used, number of documents included in the autolabeling job, number of text records and entities to be automatically labeled. Autolabeling jobs can take anywhere from a few seconds to a few minutes, depending on the number of documents you included.

+ :::image type="content" source="../media/review-autotag.png" alt-text="A screenshot showing the review screen for an autotag job." lightbox="../media/review-autotag.png":::

+### [Autolabel with GPT](#tab/autolabel-gpt)

+When you trigger an autolabeling job with GPT, you're charged to your Azure OpenAI resource as per your consumption. You're charged an estimate of the number of tokens in each document being autolabeled. Refer to the [Azure OpenAI pricing page](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) for a detailed breakdown of pricing per token of different models.

+1. From the left navigation menu, select **Data labeling**.

+2. Select the **Autolabel** button under the Activity pane to the right of the page.

+ :::image type="content" source="../media/trigger-autotag.png" alt-text="A screenshot showing how to trigger an autotag job from the activity pane." lightbox="../media/trigger-autotag.png":::

+4. Choose Autolabel with GPT and click on Next.

+ :::image type="content" source="../media/choose-models.png" alt-text="A screenshot showing model choice for auto labeling." lightbox="../media/choose-models.png":::

+5. Choose your Azure OpenAI resource and deployment. You must [create an Azure OpenAI resource and deploy a model](../../../openai/how-to/create-resource.md) in order to proceed.

+ :::image type="content" source="../media/autotag-choose-open-ai.png" alt-text="A screenshot showing how to choose OpenAI resource and deployments" lightbox="../media/autotag-choose-open-ai.png":::

+

+6. Choose the entities you want to be included in the autolabeling job. By default, all entities are selected. Having descriptive names for labels, and including examples for each label is recommended to achieve good quality labeling with GPT.

+ :::image type="content" source="../media/choose-entities.png" alt-text="A screenshot showing which entities to be included in autotag job." lightbox="../media/choose-entities.png":::

+

+7. Choose the documents you want to be automatically labeled. It's recommended to choose the unlabeled documents from the filter.

+ > [!NOTE]

+ > * If an entity was automatically labeled, but has a user defined label, only the user defined label is used and visible.

+ > * You can view the documents by clicking on the document name.

+

+ :::image type="content" source="../media/choose-files.png" alt-text="A screenshot showing which documents to be included in the autotag job." lightbox="../media/choose-files.png":::

+8. Select **Start job** to trigger the autolabeling job.

+You should be directed to the autolabeling page displaying the autolabeling jobs initiated. Autolabeling jobs can take anywhere from a few seconds to a few minutes, depending on the number of documents you included.

+When the autolabeling job is complete, you can see the output documents in the **Data labeling** page of Language Studio. Select **Review documents with autolabels** to view the documents with the **Auto labeled** filter applied.

+Entities that have been automatically labeled appear with a dotted line. These entities have two selectors (a checkmark and an "X") that allow you to accept or reject the automatic label.

+Once an entity is accepted, the dotted line changes to a solid one, and the label is included in any further model training becoming a user defined label.

+> * All labels that were not accepted are be deleted when you train your model.

+ Title: How to use autolabeling in custom text classification

+description: Learn how to use autolabeling in custom text classification.

+# How to use autolabeling for Custom Text Classification

+[Labeling process](tag-data.md) is an important part of preparing your dataset. Since this process requires much time and effort, you can use the autolabeling feature to automatically label your documents with the classes you want to categorize them into. You can currently start autolabeling jobs based on a model using GPT models where you may immediately trigger an autolabeling job without any prior model training. This feature can save you the time and effort of manually labeling your documents.

+## Prerequisites

+Before you can use autolabeling with GPT, you need:

+* A successfully [created project](create-project.md) with a configured Azure blob storage account.

+* Text data that [has been uploaded](design-schema.md#data-preparation) to your storage account.

+* Class names that are meaningful. The GPT models label documents based on the names of the classes you've provided.

+* [Labeled data](tag-data.md) isn't required.

+* An Azure OpenAI [resource and deployment](../../../openai/how-to/create-resource.md).

+## Trigger an autolabeling job

+When you trigger an autolabeling job with GPT, you're charged to your Azure OpenAI resource as per your consumption. You're charged an estimate of the number of tokens in each document being autolabeled. Refer to the [Azure OpenAI pricing page](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) for a detailed breakdown of pricing per token of different models.

+1. From the left navigation menu, select **Data labeling**.

+2. Select the **Autolabel** button under the Activity pane to the right of the page.

+ :::image type="content" source="../media/trigger-autotag.png" alt-text="A screenshot showing how to trigger an autotag job from the activity pane." lightbox="../media/trigger-autotag.png":::

+4. Choose Autolabel with GPT and click on Next.

+ :::image type="content" source="../media/choose-models.png" alt-text="A screenshot showing model choice for auto labeling." lightbox="../media/choose-models.png":::

+5. Choose your Azure OpenAI resource and deployment. You must [create an Azure OpenAI resource and deploy a model](../../../openai/how-to/create-resource.md) in order to proceed.

+ :::image type="content" source="../media/autotag-choose-open-ai.png" alt-text="A screenshot showing how to choose OpenAI resource and deployments" lightbox="../media/autotag-choose-open-ai.png":::

+

+6. Select the classes you want to be included in the autolabeling job. By default, all classes are selected. Having descriptive names for classes, and including examples for each class is recommended to achieve good quality labeling with GPT.

+ :::image type="content" source="../media/choose-classes.png" alt-text="A screenshot showing which labels to be included in autotag job." lightbox="../media/choose-classes.png":::

+

+7. Choose the documents you want to be automatically labeled. It's recommended to choose the unlabeled documents from the filter.

+ > [!NOTE]

+ > * If a document was automatically labeled, but this label was already user defined, only the user defined label is used.

+ > * You can view the documents by clicking on the document name.

+

+ :::image type="content" source="../media/choose-files.png" alt-text="A screenshot showing which documents to be included in the autotag job." lightbox="../media/choose-files.png":::

+8. Select **Start job** to trigger the autolabeling job.

+You should be directed to the autolabeling page displaying the autolabeling jobs initiated. Autolabeling jobs can take anywhere from a few seconds to a few minutes, depending on the number of documents you included.

+ :::image type="content" source="../media/review-autotag.png" alt-text="A screenshot showing the review screen for an autotag job." lightbox="../media/review-autotag.png":::

+## Review the auto labeled documents

+When the autolabeling job is complete, you can see the output documents in the **Data labeling** page of Language Studio. Select **Review documents with autolabels** to view the documents with the **Auto labeled** filter applied.

+Documents that have been automatically classified have suggested labels in the activity pane highlighted in purple. Each suggested label has two selectors (a checkmark and a cancel icon) that allow you to accept or reject the automatic label.

+Once a label is accepted, the purple color changes to the default blue one, and the label is included in any further model training becoming a user defined label.

+After you accept or reject the labels for the autolabeled documents, select **Save labels** to apply the changes.

+> [!NOTE]

+> * We recommend validating automatically labeled documents before accepting them.

+> * All labels that were not accepted are deleted when you train your model.

+## Next steps

+* Learn more about [labeling your data](tag-data.md).

+ Title: Country availability of telephone numbers and subscription eligibility

+description: Learn about Country Availability, Subscription Eligibility and Number Capabilities for PSTN and SMS Numbers in Communication Services.

+# Country availability of telephone numbers and subscription eligibility

+More details on eligible subscription types are as follows:

+\* In some countries, number purchases are only allowed for own use. Reselling or suballcoating to another parties is not allowed. Due to this, purchases for CSP and LSP customers is not allowed.

+\** Applications from all other subscription types will be reviewed and approved on a case-by-case basis. Reach out to acstns@microsoft.com for assistance with your application.

+## Number capabilities and availability

+The capabilities and numbers that are available to you depend on the country that you're operating within, your use case, and the phone number type that you've selected. These capabilities vary by country due to regulatory requirements.

+The following tables summarize current availability:

+\** Phone numbers in Italy can only be purchased for own use. Reselling or suballocating to another party is not allowed.

+## Customers with France Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| France | Local** | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+\** Phone numbers in France can only be purchased for own use. Reselling or suballocating to another party is not allowed.

+## Customers with Spain Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| Spain | Toll-Free | - | - | Public Preview | Public Preview\* |

+| Spain | Local | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+## Customers with Switzerland Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| Switzerland | Local | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+## Customers with Belgium Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| Belgium | Toll-Free | - | - | Public Preview | Public Preview\* |

+| Belgium | Local | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+## Customers with Luxembourg Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| Luxembourg | Local | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+## Customers with Austria Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| Austria | Toll-Free** | - | - | Public Preview | Public Preview\* |

+| Austria | Local** | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+\** Phone numbers in Austria can only be purchased for own use. Reselling or suballocating to another party is not allowed.

+## Customers with Portugal Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| Portugal | Toll-Free** | - | - | Public Preview | Public Preview\* |

+| Portugal | Local** | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+\** Phone numbers in Portugal can only be purchased for own use. Reselling or suballocating to another party is not allowed.

+## Customers with Slovakia Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| Slovakia | Local | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+## Customers with Norway Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| Norway | Local** | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+\** Phone numbers in Norway can only be purchased for own use. Reselling or suballocating to another party is not allowed.

+## Customers with Netherlands Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| Netherlands | Local | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+## Customers with Germany Azure billing addresses

+| Number | Type | Send SMS | Receive SMS | Make Calls | Receive Calls |

+| :- | :-- | :- | :- | :- | : |

+| Germany | Local | - | - | Public Preview | Public Preview\* |

+\* Please refer to [Inbound calling capabilities page](../telephony/inbound-calling-capabilities.md) for details.

+For more information about Azure Communication Services' telephony options please see the following pages:

+## France telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 1.00/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0.0160/min |USD 0.0100/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+## Spain telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 5.00/mo |

+|Toll-Free |USD 20.00/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0165/min |USD 0.0072/min |

+|Toll-free |Starting at USD 0165/min | USD 0.2200/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+## Switzerland telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 1.00/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0.0234/min |USD 0.0100/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+## Belgium telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 0.70/mo |

+|Toll-Free |USD 25.00/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0.1300/min |USD 0.0100/min |

+|Toll-free |Starting at USD 0.1300/min |Starting at USD 0.0505/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+## Luxembourg telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 3.00/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0.2300/min |USD 0.0100/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+## Austria telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 1.00/mo |

+|Toll-Free |USD 25.00/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0.1550/min |USD 0.0100/min |

+|Toll-free |Starting at USD 0.1550/min |Starting at USD 0.0897/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+## Portugal telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 1.00/mo |

+|Toll-Free |USD 18.00/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0.0130/min |USD 0.0100/min |

+|Toll-free |Starting at USD 0.0130/min | USD 0.0601/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+## Slovakia telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 1.00/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0.0270/min |USD 0.0100/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+## Norway telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 5.00/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0.0200/min |USD 0.0300/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+## Netherlands telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 1.50/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0.3500/min |USD 0.0100/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+## Germany telephony offers

+### Phone number leasing charges

+|Number type |Monthly fee |

+|--|--|

+|Geographic |USD 0.80/mo |

+### Usage charges

+|Number type |To make calls* |To receive calls|

+|--|--||

+|Geographic |Starting at USD 0.0150/min |USD 0.0100/min |

+\* For destination-specific pricing for making outbound calls, please refer to details [here](https://github.com/Azure/Communication/blob/master/pricing/communication-services-pstn-rates.csv)

+ Title: Outbound calling with Toll-Free numbers - Azure Communication Services

+description: Information about outbound calling limitations with Toll-Free numbers

+# Toll-Free telephone numbers and outbound calling

+Outbound calling capability with Toll-Free telephone numbers is available in many countries where Azure Communication Services is available. However, there can be some limitations when trying to place outbound calls with toll-free telephone numbers.

+**Why outbound calls from Toll-Free numbers may not work?**

+Microsoft provides Toll-Free telephone numbers that have outbound calling capabilities, but it's important to note that this feature is only provided on a "best-effort" basis. In some countries and regions, toll-free numbers are considered as an "inbound only" service from regulatory perspective. This means, that in some scenarios, the receiving carrier may not allow incoming calls from toll-free telephone numbers. Since Microsoft and our carrier-partners don't have control over other carrier networks, we can't guarantee that outbound calls will reach all possible destinations.

+Call Recording enables you to record multiple calling scenarios available in Azure Communication Services by providing you with a set of APIs to start, stop, pause and resume recording. Whether it's a PSTN, WebRTC, or SIP call, these APIs can be accessed from your server-side business logic. Also, recordings can be triggered by a user action that tells the server application to start recording.

+Regardless of how you establish the call, Call Recording allows you to produce mixed or unmixed media files that are stored for 48 hours on a built-in temporary storage. You can retrieve the files and take them to the long-term storage solution of your choice. Call Recording supports all Azure Communication Services data regions.

+Call Recording supports multiple media outputs and content types to address your business needs and use cases. You might use mixed formats for scenarios such as keeping records, meeting notes, coaching and training, or even compliance and adherence. Or, you can use unmixed audio format to address quality assurance use cases or even more complex scenarios like advanced analytics or AI-based (Artificial Intelligence) sophisticated post-call processes.

+| mixed | mp4 | 1920x1080, 16 FPS (frames per second) | 16 kHz | single file, single channel | mixed video in a default 3x3 (most active speakers) tile arrangement with display name support |

+Call Recording uses [Azure Event Grid](https://learn.microsoft.com/azure/event-grid/event-schema-communication-services) to provide you with notifications related to media and metadata.

+Many countries and states have laws and regulations that apply to call recording. PSTN, voice, and video calls often require that users consent to the recording of their communications. It is your responsibility to use the call recording capabilities in compliance with the law. You must obtain consent from the parties of recorded communications in a manner that complies with the laws applicable to each participant.

+It's possible that when a call is created using Call Automation, you don't get a value in the `serverCallId`. If that's the case, get the `serverCallId` from the `CallConnected` event method described in [Get serverCallId](https://learn.microsoft.com/azure/communication-services/quickstarts/voice-video-calling/callflows-for-customer-interactions?pivots=programming-language-csharp#configure-programcs-to-answer-the-call).

+> This API is provided as a Public Preview ('beta') for developers and may change based on feedback that we receive. Do not use this API in a production environment.

+1. Use the information you collected in [Collect Azure Communications Gateway resource values](prepare-to-deploy.md#4-collect-basic-information-for-deploying-an-azure-communications-gateway) to fill out the fields in the **Basics** configuration section and then select **Next: Service Regions**.

+1. Use the information you collected in [Collect Service Regions configuration values](prepare-to-deploy.md#5-collect-service-regions-configuration-values) to fill out the fields in the **Service Regions** section and then select **Next: Tags**.

+## 3. Provide additional information to your onboarding team

+>This step and the next step ([2. Assign an Admin user to the Project Synergy application](#2-assign-an-admin-user-to-the-project-synergy-application)) set you up as an Operator in the Teams Phone Mobile (TPM) and Operator Connect (OC) environments. If you've already gone through onboarding, go to [3. Create a network design](#3-create-a-network-design).

+1. If you don't have the Azure Active Directory module installed, install it:

+ Install-Module AzureAD

+1. In your Azure portal, navigate to **Enterprise applications** using the left-hand side menu. Alternatively, you can search for it in the search bar; it's under the **Services** subheading.

+## 3. Create a network design

+## 4. Collect basic information for deploying an Azure Communications Gateway

+ |The management Azure region: the region in which your monitoring and billing data is processed. We recommend that you select a region near or colocated with the two regions for handling call traffic. |**Instance details: Region**

+ |The scope at which Azure Communications Gateway's autogenerated domain name label is unique. Communications Gateway resources get assigned an autogenerated domain name label that depends on the name of the resource. You'll need to register the domain name later when you deploy Azure Communications Gateway. Selecting **Tenant** gives a resource with the same name in the same tenant but a different subscription the same label. Selecting **Subscription** gives a resource with the same name in the same subscription but a different resource group the same label. Selecting **Resource Group** gives a resource with the same name in the same resource group the same label. Selecting **No Re-use** means the label doesn't depend on the name, resource group, subscription or tenant. |**Instance details: Auto-generated Domain Name Scope**|

+## 5. Collect Service Regions configuration values

+ |The Azure regions to use for call traffic. |**Service Region One/Two: Region**|

+## 6. Collect Test Lines configuration values

+## 7. Decide if you want tags

+## 8. Get access to Azure Communications Gateway for your Azure subscription

+Access to Azure Communications Gateway is restricted. When you've completed the previous steps in this article, contact your onboarding team and ask them to enable your subscription. If you don't already have an onboarding team, contact azcog-enablement@microsoft.com with your Azure subscription ID and contact details.

+Wait for confirmation that Azure Communications Gateway is enabled before moving on to the next step.

+## 9. Register the Microsoft Voice Services resource provider

+If the **Microsoft.VoiceServices** resource provider isn't already registered in your subscription, register this provider.

+1. Sign in to the [Azure portal](https://portal.azure.com/) and go to your Azure subscription.

+1. Select **Resource providers** under the **Settings** tab.

+1. Search for the **Microsoft.VoiceServices** resource provider.

+1. Check if the resource provider is already marked as registered. If it isn't, choose the resource provider and select **Register**.

+## 10. Set up application roles for Azure Communications Gateway in your Project Synergy application

+Azure Communications Gateway contains services that need to access the Operator Connect API on your behalf. To enable this access, you must grant specific application roles to an AzureCommunicationsGateway service principal under the Project Synergy Enterprise Application. You created the Project Synergy application in [1. Add the Project Synergy application to your Azure tenancy](#1-add-the-project-synergy-application-to-your-azure-tenancy). Microsoft created the Azure Communications Gateway service principal for you when you followed [9. Register the Microsoft Voice Services resource provider](#9-register-the-microsoft-voice-services-resource-provider).

+You need to do the following steps in the tenant that contains your Project Synergy application.

+1. Sign in to the [Azure portal](https://ms.portal.azure.com/) as an Azure Active Directory Global Admin.

+1. Select **Azure Active Directory**.

+1. Select **Properties**.

+1. Scroll down to the Tenant ID field. Your tenant ID is in the box. Make a note of your tenant ID.

+1. Open PowerShell.

+1. If you didn't install the Azure Active Directory module as part of [1. Add the Project Synergy application to your Azure tenancy](#1-add-the-project-synergy-application-to-your-azure-tenancy), install it:

+ ```azurepowershell

+ Install-Module AzureAD

+ ```

+1. Run the following cmdlet, replacing *`<AADTenantID>`* with the tenant ID you noted down in step 4.

+ ```azurepowershell

+ Connect-AzureAD -TenantId "<AADTenantID>"

+ ```

+1. Run the following PowerShell commands. These commands add the following roles for Azure Communications Gateway: `TrunkManagement.Read`, `NumberManagement.Read`, `NumberManagement.Write`, `Data.Read`, `Data.Write`, `TrunkManagement.Write`, `PartnerSettings.Read`.

+ ```azurepowershell

+ # Get the Service Principal ID for Azure Communications Gateway

+ $commGwayApplicationId = "8502a0ec-c76d-412f-836c-398018e2312b"

+ $commGwayEnterpriseApplication = Get-AzureADServicePrincipal -Filter "AppId eq '$commGwayApplicationId'"

+ $commGwayObjectId = $commGwayEnterpriseApplication.ObjectId

+

+ # Get the Service Principal ID for Project Synergy (Operator Connect)

+ $projectSynergyApplicationId = "eb63d611-525e-4a31-abd7-0cb33f679599"

+ $projectSynergyEnterpriseApplication = Get-AzureADServicePrincipal -Filter "AppId eq '$projectSynergyApplicationId'"

+ $projectSynergyObjectId = $projectSynergyEnterpriseApplication.ObjectId

+

+ # Required Operator Connect - Project Synergy Roles

+ $trunkManagementRead = "72129ccd-8886-42db-a63c-2647b61635c1"

+ $trunkManagementWrite = "e907ba07-8ad0-40be-8d72-c18a0b3c156b"

+

+ $partnerSettingsRead = "d6b0de4a-aab5-4261-be1b-0e1800746fb2"

+

+ $numberManagementRead = "130ecbe2-d1e6-4bbd-9a8d-9a7a909b876e"

+ $numberManagementWrite = "752b4e79-4b85-4e33-a6ef-5949f0d7d553"

+

+ $dataRead = "eb63d611-525e-4a31-abd7-0cb33f679599"

+ $dataWrite = "98d32f93-eaa7-4657-b443-090c23e69f27"

+

+ $requiredRoles = $trunkManagementRead, $numberManagementRead, $numberManagementWrite, $dataRead, $dataWrite, $trunkManagementWrite, $partnerSettingsRead

+

+ foreach ($role in $requiredRoles) {

+ # Assign the relevant Role to the Azure Communications Gateway Service Principal

+ New-AzureADServiceAppRoleAssignment -ObjectId $commGwayObjectId -PrincipalId $commGwayObjectId -ResourceId $projectSynergyObjectId -Id $role

+ }

+ ```

+## 11. Add the application ID for Azure Communications Gateway to Operator Connect

+Before you can use the roles that you set up in [10. Set up application roles for Azure Communications Gateway in your Project Synergy application](#10-set-up-application-roles-for-azure-communications-gateway-in-your-project-synergy-application), you must enable the Azure Communications Gateway application within the Operator Connect or Teams Phone Mobile environment.

+To enable the Azure Communications Gateway application and the roles, add the application ID of the AzureCommunicationsGateway service principal to your Operator Connect or Teams Phone Mobile environment:

+1. Optionally, check the application ID of the service principal to confirm that you're adding the right application.

+ 1. Search for `AzureCommunicationsGateway` with the search bar: it's under the **Azure Active Directory** subheading.

+ 1. On the overview page, check that the value of **Object ID** is `8502a0ec-c76d-412f-836c-398018e2312b`.

+1. Log into the [Operator Connect Number Management Portal](https://operatorconnect.microsoft.com/operator/configuration).

+1. Add a new **Application Id**, pasting in the following value. This value is the application ID for Azure Communications Gateway.

+ ```

+ 8502a0ec-c76d-412f-836c-398018e2312b

+ ```

+ Title: What's new in Azure Communications Gateway?

+description: Discover what's new in Azure Communications Gateway

+# What's new in Azure Communications Gateway?

+This article covers new features and improvements for Azure Communications Gateway.

+## March 2023: Simpler authentication for Operator Connect APIs

+Azure Communications Gateway contains services that need to access the Operator Connect API on your behalf. Azure Communications Gateway therefore needs to authenticate with your Operator Connect or Teams Phone Mobile environment.

+From March 2023, Azure Communications Gateway automatically provides a service principal for this authentication. You must set up specific permissions for this service principal and then add the service principal to your Operator Connect or Teams Phone Mobile environment. For more information, see [Prepare to deploy Azure Communications Gateway](prepare-to-deploy.md).

+This new authentication model replaces an earlier model that required you to create an App registration and manage secrets for it. With the new model, you no longer need to create, manage or rotate secrets.

+## Next steps

+- [Learn more about Azure Communications Gateway](overview.md).

+- [Prepare to deploy Azure Communications Gateway](prepare-to-deploy.md).

+ Title: Secure Key Release with Azure Key Vault and Azure Confidential Computing

+description: Concept guide on what SKR is and its usage with Azure Confidential Computing Offerings

+# Secure Key Release feature with AKV and Azure Confidential Computing (ACC)

+Secure Key Release (SKR) is a functionality of Azure Key Vault (AKV) Managed HSM and Premium offering. Secure key release enables the release of an HSM protected key from AKV to an attested Trusted Execution Environment (TEE), such as a secure enclave, VM based TEEs etc. SKR adds another layer of access protection to your data decryption/encryption keys where you can target an application + TEE runtime environment with known configuration get access to the key material. The SKR policies defined at the time of exportable key creation govern the access to these keys.

+## SKR support with AKV offerings

+- [Azure Key Vault Premium](../security/fundamentals/key-management.md)

+- [Azure Key Vault Managed HSM](../key-vault/managed-hsm/overview.md)

+## Overall Secure Key Release Flow with TEE

+SKR can only release keys based on the Microsoft Azure Attestation (MAA) generated claims. There's a tight integration on the SKR policy definition to MAA claims.

+

+The below steps are for AKV Premium.

+### Step 1: Create a Key Vault Premium HSM Backed

+[Follow the details here for Az CLI based AKV creation](../key-vault/general/quick-create-cli.md)

+Make sure to set the value of [--sku] to "premium".

+### Step 2: Create a Secure Key Release Policy

+A Secure Key Release Policy is a json format release policy as defined [here](/rest/api/keyvault/keys/create-key/create-key?tabs=HTTP#keyreleasepolicy) that specifies a set of claims required in addition to authorization to release the key. The claims here are MAA based claims as referenced [here for SGX](/azure/attestation/attestation-token-examples#sample-jwt-generated-for-sgx-attestation) and here for [AMD SEV-SNP CVM](/azure/attestation/attestation-token-examples#sample-jwt-generated-for-sev-snp-attestation).

+Visit the TEE specific [examples page for more details](skr-policy-examples.md)

+Before you set an SKR policy make sure to run your TEE application through the remote attestation flow. Remote attestation isn't covered as part of this tutorial.

+Example

+```json

+{

+ "version": "1.0.0",

+ "anyOf": [ // Always starts with "anyOf", meaning you can multiple, even varying rules, per authority.

+ {

+ "authority": "https://sharedweu.weu.attest.azure.net",

+ "allOf": [ // can be replaced by "anyOf", though you cannot nest or combine "anyOf" and "allOf" yet.

+ {

+ "claim": "x-ms-isolation-tee.x-ms-attestation-type", // These are the MAA claims.

+ "equals": "sevsnpvm"

+ },

+ {

+ "claim": "x-ms-isolation-tee.x-ms-compliance-status",

+ "equals": "azure-compliant-cvm"

+ }

+ ]

+ }

+ ]

+}

+```

+### Step 3: Create an exportable key in AKV with attached SKR policy

+Exact details of the type of key and other attributes associated can be found [here](../key-vault/general/quick-create-cli.md).

+```azurecli

+az keyvault key create --exportable true --vault-name "vault name from step 1" --kty RSA-HSM --name "keyname" --policy "jsonpolicyfromstep3 -can be a path to JSON" --protection hsm --vault-name "name of vault created from step1"

+```

+### Step 4: Application running within a TEE doing a remote attestation

+This step can be specific to the type of TEE you're running your application Intel SGX Enclaves or AMD SEV-SNP based Confidential Virtual Machines (CVM) or Confidential Containers running in CVM Enclaves with AMD SEV-SNP etc.

+Follow these references examples for various TEE types offering with Azure:

+- [Application within AMD EV-SNP based CVM's performing Secure Key Release](skr-flow-confidential-vm-sev-snp.md)

+- [Confidential containers with Azure Container Instances (ACI) with SKR side-car containers](skr-flow-confidential-containers-azure-container-instance.md)

+- [Intel SGX based applications performing Secure Key Release - Open Source Solution Mystikos Implementation](https://github.com/deislabs/mystikos/tree/main/samples/confidential_ml#environment)

+## Frequently Asked Questions (FAQ)

+### Can I perform SKR with non confidential computing offerings?

+No. The policy attached to SKR only understands MAA claims that are associated to hardware based TEEs.

+### Can I bring my own attestation provider or service and use those claims for AKV to validate and release?

+No. AKV only understands and integrates with MAA today.

+### Can I use AKV SDKs to perform key RELEASE?

+Yes. Latest SDK integrated with 7.3 AKV API's support key RELEASE.

+### Can you share some examples of the key release policies?

+Yes, detailed examples by TEE type are listed [here.](./skr-policy-examples.md)

+## Can I attach SKR type of policy with certificates and secrets?

+No. Not at this time.

+## References

+[SKR Policy Examples](skr-policy-examples.md)

+[Azure Container Instance with confidential containers Secure Key Release with container side-cars](skr-flow-confidential-containers-azure-container-instance.md)

+[CVM on AMD SEV-SNP Applications with Secure Key Release Example](skr-flow-confidential-vm-sev-snp.md)

+[AKV REST API With SKR Details](/rest/api/keyvault/keys/create-key/create-key?tabs=HTTP)

+[AKV SDKs](../key-vault/general/client-libraries.md)

+ Title: Confidential containers with Intel SGX enclaves on Azure

+ Title: Secure Key Release with Azure Key Vault and Confidential Containers on Azure Container Instance

+description: Learn how to build an application that securely gets the key from AKV to an attested Azure Container Instances confidential container environment

+# Secure Key Release with Confidential containers on Azure Container Instance (ACI)

+Secure Key Release (SKR) flow with Azure Key Vault (AKV) with confidential container offerings can implement in couple of ways. Confidential containers run a guest enlightened exposting AMD SEV-SNP device through a Linux Kernel that uses an in guest firmware with necessary Hyper-V related patches that we refer as Direct Linux Boot (DLB). This platform doesn't use vTPM and HCL based that Confidential VMs with AMD SEV-SNP support. This concept document assumes you plan to run the containers in [Azure Container Support choosing a confidential computing SKU](../container-instances/container-instances-tutorial-deploy-confidential-containers-cce-arm.md)

+- Side-Car Helper Container provided by Azure

+- Custom implementation with your container application

+## Side-Car helper container provided by Azure

+An [open sourced GitHub project "confidential side-cars"](https://github.com/microsoft/confidential-sidecar-containers) details how to build this container and what parameters/environment variables are required for you to prepare and run this side-car container. The current side car implementation provides various HTTP REST APIs that your primary application container can use to fetch the key from AKV. The integration through Microsoft Azure Attestation(MAA) is already built in. The preparation steps to run the side-car SKR container can be found in details [here](https://github.com/microsoft/confidential-sidecar-containers/tree/main/examples/skr).

+Your main application container application can call the side-car WEB API end points as defined in the example blow. Side-cars runs within the same container group and is a local endpoint to your application container. Full details of the API can be found [here](https://github.com/microsoft/confidential-sidecar-containers/blob/main/cmd/skr/README.md)

+

+The `key/release` POST method expects a JSON of the following format:

+```json

+{

+ "maa_endpoint": "<maa endpoint>", //https://learn.microsoft.com/en-us/azure/attestation/quickstart-portal#attestation-provider

+ "akv_endpoint": "<akv endpoint>", //AKV URI

+ "kid": "<key identifier>" //key name,

+ "access_token": "optional aad token if the command will run in a resource without proper managed identity assigned"

+}

+```

+Upon success, the `key/release` POST method response carries a `StatusOK` header and a payload of the following format:

+```json

+{

+ "key": "<key in JSON Web Key format>"

+}

+```

+Upon error, the `key/release` POST method response carries a `StatusForbidden` header and a payload of the following format:

+```json

+{

+ "error": "<error message>"

+}

+```

+## Custom implementation with your container application

+To perform a custom container application that extends the capability of Azure Key Vault (AKV) - Secure Key Release and Microsoft Azure Attestation (MAA), use the below as a high level reference flow. An easy approach is to review the current side-car implementation code in this [side-car Github project](https://github.com/microsoft/confidential-sidecar-containers/tree/d933d0f4e3d5498f7ed9137189ab6a23ade15466/pkg/common).

+

+1. **Step 1:** Set up AKV with Exportable Key and attach the release policy. More [here](concept-skr-attestation.md)

+1. **Step 2:** Set up a managed identity with Azure Active Directory and attach that to AKV. More [here](../container-instances/container-instances-managed-identity.md)

+1. **Step 3:** Deploy your container application with required parameters within ACI by setting up a confidential computing enforcement policy. More [here](../container-instances/container-instances-tutorial-deploy-confidential-containers-cce-arm.md)

+1. **Step 4:** In this step, your application shall fetch a RAW AMD SEV-SNP hardware report by doing a IOCTL Linux Socket call. You don't need any guest attestation library to perform this action. More on existing side-car [implementation](https://github.com/microsoft/confidential-sidecar-containers/blob/d933d0f4e3d5498f7ed9137189ab6a23ade15466/pkg/attest/snp.go)

+1. **Step 5:** Fetch the AMD SEV-SNP cert chain for the container group. These certs are delivered from Azure host IMDS endpoint. More [here](https://github.com/microsoft/confidential-sidecar-containers/blob/d933d0f4e3d5498f7ed9137189ab6a23ade15466/pkg/common/info.go)

+1. **Step 6:** Send the SNP RAW hardware report and cert details to MAA for verification and return claims. More [here](../attestation/basic-concepts.md)

+1. **Step 7:** Send the MAA token and the managed identity token generated by ACI to AKV for key release. More [here](../container-instances/container-instances-managed-identity.md)

+On success of the key fetch from AKV, you can consume the key for decrypting the data sets or encrypt the data going out of the confidential container environment.

+## References

+[ACI with Confidential container deployments](../container-instances/container-instances-tutorial-deploy-confidential-containers-cce-arm.md)

+[Side-Car Implementation with encrypted blob fetch and decrypt with SKR AKV key](https://github.com/microsoft/confidential-sidecar-containers/#encrypted-filesystem-sidecar)

+[AKV SKR with Confidential VM's AMD SEV-SNP](skr-flow-confidential-vm-sev-snp.md)

+[Microsoft Azure Attestation (MAA)](../attestation/overview.md)

+[SKR Policy Examples](skr-policy-examples.md)

+ Title: Secure Key Release with Azure Key Vault and application on Confidential VMs with AMD SEV-SNP

+description: Learn how to build an application that securely gets the key from AKV to a Confidential VM attested environment and in an Azure Kubernetes Service cluster

+# Secure Key Release with Confidential VMs How To Guide

+The below article describes how to perform a Secure Key Release from Azure Key Value when your applications are running with an AMD SEV-SNP confidential. To learn more about Secure Key Release and Azure Confidential Computing, [go here.](./concept-skr-attestation.md).

+SKR requires that an application performing SKR shall go through a remote guest attestation flow using Microsoft Azure Attestation (MAA) as described [here](guest-attestation-confidential-vms.md).

+## Overall flow and architecture

+To allow Azure Key Vault to release a key to an attested confidential virtual machine, there are certain steps that need to be followed:

+1. Assign a managed identity to the confidential virtual machine. System-assigned managed identity or a user-assigned managed identity are allowed.

+1. Set a Key Vault access policy to grant the managed identity the "release" key permission. A policy allows the confidential virtual machine to access the Key Vault and perform the release operation. If using Key Vault Managed HSM, assign "Managed HSM Crypto Service Release User" role membership.

+1. Create a Key Vault key that is marked as exportable and has an associated release policy. Key release policy associates the key to an attested confidential virtual machine and that the key can only be used for the desired purpose.

+1. To perform the release, send an HTTP request to the Key Vault from the confidential virtual machine. HTTP request must include the Confidential VMs attested platform report in the request body. The attested platform report is used to verify the trustworthiness of the state of the Trusted Execution Environment-enabled platform, such as the Confidential VM. The Microsoft Azure Attestation service can be used to create the attested platform report and include it in the request.

+

+## Deploying an Azure Key Vault

+Set up AKV Premium or AKV mHSM with an exportable key. Follow the detailed instructions from here [setting up SKR exportable keys](concept-skr-attestation.md)

+### Bicep

+```bicep

+@description('Required. Specifies the Azure location where the key vault should be created.')

+param location string = resourceGroup().location

+@description('Specifies the Azure Active Directory tenant ID that should be used for authenticating requests to the key vault. Get it by using Get-AzSubscription cmdlet.')

+param tenantId string = subscription().tenantId

+resource keyVault 'Microsoft.KeyVault/vaults@2021-11-01-preview' = {

+ name: 'mykeyvault'

+ location: location

+ properties: {

+ tenantId: tenantId

+ sku: {

+ name: 'premium'

+ family: 'A'

+ }

+ }

+}

+```

+### ARM template

+```json

+ {

+ "type": "Microsoft.KeyVault/vaults",

+ "apiVersion": "2021-11-01-preview",

+ "name": "mykeyvault",

+ "location": "[parameters('location')]",

+ "properties": {

+ "tenantId": "[parameters('tenantId')]",

+ "sku": {

+ "name": "premium",

+ "family": "A"

+ }

+ }

+ }

+```

+## Deploy a confidential virtual machine

+Follow the quickstart instructions on how to "[Deploy confidential VM with ARM template](quick-create-confidential-vm-arm-amd.md)"

+## Enable system-assigned managed identity

+[Managed identities](../active-directory/managed-identities-azure-resources/overview.md) for Azure resources provide Azure services with an automatically managed identity in Azure Active Directory. You can use this identity to authenticate to any service that supports Azure AD authentication, without having credentials in your code.

+To enable system-assigned managed identity on a CVM, your account needs the [Virtual Machine Contributor](../role-based-access-control/built-in-roles.md#virtual-machine-contributor) role assignment. No other Azure AD directory role assignments are required.

+### [Bicep 1](#tab/bicep)

+1. Whether you sign in to Azure locally or via the Azure portal, use an account that is associated with the Azure subscription that contains the VM.

+2. To enable system-assigned managed identity, load the template into an editor, locate the `Microsoft.Compute/virtualMachines` resource of interest and add the `"identity"` property at the same level as the `name: vmName` property. Use the following syntax:

+ ```bicep

+ identity:{

+ type: 'SystemAssigned'

+ }

+ ```

+3. Add the `resource` details to the template.

+ ```bicep

+ resource confidentialVm 'Microsoft.Compute/virtualMachines@2021-11-01' = {

+ name: vmName

+ location: location

+ identity:{

+ type: 'SystemAssigned'

+ }

+ // other resource provider properties

+ }

+ ```

+### [ARM template 1](#tab/arm-template)

+1. Use an Azure account that is associated to the Azure subscription that contains the VM.

+2. To enable system-assigned managed identity, load the template into an editor, locate the `Microsoft.Compute/virtualMachines` resource of interest within the `resources` section and add the `"identity"` property at the same level as the `"type": "Microsoft.Compute/virtualMachines"` property. Use the following syntax:

+ ```json

+ "identity": {

+ "type": "SystemAssigned"

+ },

+ ```

+3. The final template looks the below example

+ ```json

+ "resources": [

+ {

+ "apiVersion": "2021-11-01",

+ "type": "Microsoft.Compute/virtualMachines",

+ "name": "[parameters('vmName')]",

+ "location": "[parameters('location')]",

+ "identity": {

+ "type": "SystemAssigned",

+ },

+ //other resource provider properties...

+ }

+ ]

+ ```

+## Add the access policy to Azure Key Vault

+Once you turn on a system-assigned managed identity for your CVM, you have to provide it with access to the Azure Key Vault data plane where key objects are stored. To ensure that only our confidential virtual machine can execute the release operation, we'll only grant specific permission required for that.

+> [!NOTE]

+> You can find the managed identity object ID in the virtual machine identity options, in the Azure portal. Alternatively you can retrieve it with [PowerShell](../active-directory/managed-identities-azure-resources/how-to-assign-app-role-managed-identity-powershell.md), [Azure CLI](../active-directory/managed-identities-azure-resources/how-to-assign-app-role-managed-identity-cli.md), Bicep or ARM templates.

+### [Bicep 1]

+```bicep

+@description('Required. Specifies the object ID of a user, service principal or security group in the Azure Active Directory tenant for the vault. The object ID must be unique for the list of access policies. Get it by using Get-AzADUser or Get-AzADServicePrincipal cmdlets.')

+param objectId string

+resource keyVaultCvmAccessPolicy 'Microsoft.KeyVault/vaults/accessPolicies@2022-07-01' = {

+ parent: keyVault

+ name: 'add'

+ properties: {

+ accessPolicies: [

+ {

+ objectId: objectId

+ tenantId: tenantId

+ permissions: {

+ keys: [

+ 'release'

+ ]

+ }

+ }

+ ]

+ }

+}

+```

+### [ARM template 2]

+```json

+ {

+ "type": "Microsoft.KeyVault/vaults/accessPolicies",

+ "apiVersion": "2022-07-01",

+ "name": "[format('{0}/{1}', 'mykeyvault', 'add')]",

+ "properties": {

+ "accessPolicies": [

+ {

+ "objectId": "[parameters('objectId')]",

+ "tenantId": "[parameters('tenantId')]",

+ "permissions": {

+ "keys": [

+ "release"

+ ]

+ }

+ }

+ ]

+ },

+ "dependsOn": [

+ "[resourceId('Microsoft.KeyVault/vaults', 'mykeyvault')]"

+ ]

+ }

+```

+## Prepare the release policy

+Key Vault Secure Key Release Policies are modeled after __Azure Policy__, with a [slightly different grammar](../key-vault/keys/policy-grammar.md).

+The idea is when we __pass the attested platform report__, in the form of a JSON Web Token (JWT), to Key Vault. It will, in turn, look at the JWT and check whether or not the attested platform report claims match the claims in the policy.

+For example, let's say we want to release a key only when our attested platform report has properties like:

+- Attested by [Microsoft Azure Attestation (MAA)](../attestation/overview.md) service endpoint "https://sharedweu.weu.attest.azure.net".

+ - This `authority` value from the policy is compared to the `iss` (issuer) property, in the token.

+- And that it also contains an object called `x-ms-isolation-tee` with a property called `x-ms-attestation-type`, which holds value `sevsnpvm`.

+ - MAA as an Azure service has attested that the CVM is running in an AMD SEV-SNP genuine processor.

+- And that it also contains an object called `x-ms-isolation-tee` with a property called `x-ms-compliance-status`, which holds the value `azure-compliant-cvm`.

+ - MAA as an Azure service has the ability to attest that the CVM is a compliant Azure confidential virtual machine.

+Create a new folder called `assets` and add the following JSON content to a file named `cvm-release-policy.json`:

+```json

+{

+ "version": "1.0.0",

+ "anyOf": [

+ {

+ "authority": "https://sharedweu.weu.attest.azure.net",

+ "allOf": [

+ {

+ "claim": "x-ms-isolation-tee.x-ms-attestation-type",

+ "equals": "sevsnpvm"

+ },

+ {

+ "claim": "x-ms-isolation-tee.x-ms-compliance-status",

+ "equals": "azure-compliant-cvm"

+ }

+ ]

+ }

+ ]

+}

+```

+Release policy is an `anyOf` condition containing an array of key authorities. A `claim` condition is a JSON object that identifies a claim name, a condition for matching, and a value. The `AnyOf` and `AllOf` condition objects allow for the modeling of a logical `OR` and `AND`. Currently, we can only perform an `equals` comparison on a `claim`. Condition properties are placed together with `authority` properties.

+> [!IMPORTANT]

+> An environment assertion contains at least __a key encryption key and one or more claims about the target environment__ (for example, TEE type, publisher, version) that are matched against the Key Release Policy. The key-encryption key is a public RSA key owned and protected by the target execution environment that is used for key export. It must appear in the TEE keys claim (x-ms-runtime/keys). This claim is a JSON object representing a JSON Web Key Set. Within the JWKS, one of the keys must meet the requirements for use as an encryption key (key_use is "enc", or key_ops contains "encrypt"). The first suitable key is chosen.

+Key Vault picks the first suitable key from "`keys`" array property in the "`x-ms-runtime`" object, it looks for a public RSA key with `"key_use": ["enc"]` or `"key_ops": ["encrypt"]`. An example of an attested platform report would look like:

+```json

+{

+ //...

+ "x-ms-runtime": {

+ "client-payload": {

+ "nonce": "MTIzNA=="

+ },

+ "keys": [

+ {

+ "e": "AQAB",

+ "key_ops": [

+ "encrypt"

+ ],

+ "kid": "TpmEphemeralEncryptionKey",

+ "kty": "RSA",

+ "n": "9v2XQgAA6y18CxV8dSGnh..."

+ }

+ ]

+ },

+ //...

+}

+```

+In this example, we have only one key under the `$.x-ms-runtime.keys` path. Key Vault uses the `TpmEphemeralEncryptionKey` key as the key-encryption key.

+> [!NOTE]

+> Notice that there may be a key under `$.x-ms-isolation-tee.x-ms-runtime.keys`, this is __not__ the key that Key Vault will be using.

+## Create an exportable key with release policy

+We create a Key Vault access policy that lets an Azure Confidential Virtual Machine perform the `release` key operation. Finally, we must include our release policy as a base64 encoded string during the key creation. The key must be an __exportable__ key, backed by an HSM.

+> [!NOTE]

+> HSM-backed keys are available with Azure Key Vault Premium and Azure Key Vault Managed HSM.

+### [Bicep 2]

+```bicep

+@description('The type of the key. For valid values, see JsonWebKeyType. Must be backed by HSM, for secure key release.')

+@allowed([

+ 'EC-HSM'

+ 'RSA-HSM'

+])

+param keyType string = 'RSA-HSM'

+@description('Not before date in seconds since 1970-01-01T00:00:00Z.')

+param keyNotBefore int = -1

+@description('Expiry date in seconds since 1970-01-01T00:00:00Z.')

+param keyExpiration int = -1

+@description('The elliptic curve name. For valid values, see JsonWebKeyCurveName.')

+@allowed([

+ 'P-256'

+ 'P-256K'

+ 'P-384'

+ 'P-521'

+])

+param curveName string

+@description('The key size in bits. For example: 2048, 3072, or 4096 for RSA.')

+param keySize int = -1

+resource exportableKey 'Microsoft.KeyVault/vaults/keys@2022-07-01' = {

+ parent: keyVault

+ name: 'mykey'

+ properties: {

+ kty: keyType

+ attributes: {

+ exportable: true

+ enabled: true

+ nbf: keyNotBefore == -1 ? null : keyNotBefore

+ exp: keyExpiration == -1 ? null : keyExpiration

+ }

+ curveName: curveName // applicable when using key type (kty) 'EC'

+ keySize: keySize == -1 ? null : keySize

+ keyOps: ['encrypt','decrypt'] // encrypt and decrypt only work with RSA keys, not EC

+ release_policy: {

+ contentType: 'application/json; charset=utf-8'

+ data: loadFileAsBase64('assets/cvm-release-policy.json')

+ }

+ }

+}

+```

+### [ARM template 2]

+```json

+ {

+ "type": "Microsoft.KeyVault/vaults/keys",

+ "apiVersion": "2022-07-01",

+ "name": "[format('{0}/{1}', 'mykeyvault', 'mykey')]",

+ "properties": {

+ "kty": "RSA-HSM",

+ "attributes": {

+ "exportable": true,

+ "enabled": true,

+ "nbf": "[if(equals(parameters('keyNotBefore'), -1), null(), parameters('keyNotBefore'))]",

+ "exp": "[if(equals(parameters('keyExpiration'), -1), null(), parameters('keyExpiration'))]"

+ },

+ "curveName": "[parameters('curveName')]",

+ "keySize": "[if(equals(parameters('keySize'), -1), null(), parameters('keySize'))]",

+ "keyOps": [

+ "encrypt",

+ "decrypt"

+ ],

+ "release_policy": {

+ "contentType": "application/json; charset=utf-8",

+ "data": "[variables('cvmReleasePolicyBase64EncodedString')]"

+ }

+ },

+ "dependsOn": [

+ "[resourceId('Microsoft.KeyVault/vaults', 'mykeyvault')]"

+ ]

+ }

+```

+We can verify that Key Vault has created a new, __HSM-backed__, key and that it contains our secure key __release policy__ by navigating to the Azure portal and selecting our key. The intended key will be marked as "__exportable__".

+

+## Guest attestation client

+Attestation helps us to _cryptographically assess_ that something is running in the intended operating state. It is the process by which one party, the verifier, assesses the trustworthiness of a potentially untrusted peer, the attester. With remote guest attestation, the trusted execution environment offers a platform that allows you to run an entire operating system inside of it.

+> [!IMPORTANT]

+> Microsoft offers a C/C++ library, for both [Windows](https://www.nuget.org/packages/Microsoft.Azure.Security.GuestAttestation) and [Linux](https://packages.microsoft.com/repos/azurecore/pool/main/a/azguestattestation1/) that can help your development efforts. The library makes it easy to acquire a __a SEV-SNP platform report__ from the hardware and to also have it attested by an instance of Azure Attestation service. The Azure Attestation service can either be one hosted by Microsoft (shared) or your own private instance.

+A [open sourced](https://github.com/Azure/confidential-computing-cvm-guest-attestation) Windows and Linux client binary that utilizes the guest attestation library can be chosen to make the guest attestation process easy with CVMs. The client binary returns the attested platform report as a JSON Web Token, which is what is needed for Key Vault's `release` key operation.

+> [!NOTE]

+> A token from the Azure Attestation service is valid for [8 hours](../attestation/faq.yml).

+### [Linux]

+1. Sign in to your VM.

+1. Clone the [sample Linux application](https://github.com/Azure/confidential-computing-cvm-guest-attestation/tree/main/cvm-platform-checker-exe/Linux).

+1. Install the `build-essential` package. This package installs everything required for compiling the sample application.

+ ```bash

+ sudo apt-get install build-essential

+ ```

+1. Install the `libcurl4-openssl-dev` and `libjsoncpp-dev` packages.

+ ```bash

+ sudo apt-get install libcurl4-openssl-dev

+ ```

+ ```bash

+ sudo apt-get install libjsoncpp-dev

+ ```

+1. [Download](https://packages.microsoft.com/repos/azurecore/pool/main/a/azguestattestation1/) the attestation package.

+1. Install the attestation package. Make sure to replace `<version>` with the version that you downloaded.

+ ```bash

+ sudo dpkg -i azguestattestation1_<latest-version>_amd64.deb

+ ```

+1. To run the sample client, navigate inside the unzipped folder and run the below command:

+ ```sh

+ sudo ./AttestationClient -a <attestation-url> -n <nonce-value> -o token

+ ```

+> [!NOTE]

+> If `-o` is not specified to as `token`, the exe prints a binary result true or false depending on the attestation result and the platform being `sevsnp`.

+### [Windows](#tab/windows)

+1. Sign in to your VM.

+1. Clone the [sample Windows application](https://github.com/Azure/confidential-computing-cvm-guest-attestation/tree/main/cvm-platform-checker-exe/Windows).

+1. Navigate inside the unzipped folder and run `VC_redist.x64.exe`. VC_redist will install Microsoft C and C++ (MSVC) runtime libraries on the machine.

+1. To run the sample client, navigate inside the unzipped folder and run the below command:

+ ```sh

+ sudo ./AttestationClient -a <attestation-url> -n <nonce-value> -o token

+ ```

+> [!NOTE]

+> If `-o` is not specified to as `token`, the exe prints a binary result true or false depending on the attestation result and the platform being `sevsnp`.

+### Guest Attestation result

+The result from the Guest Attestation client simply is a base64 encoded string! This encoded string value is a signed JSON Web Token (__JWT__), with a header, body and signature. You can split the string by the `.` (dot) value and base64 decode the results.

+```text

+eyJhbGciO...

+```

+The header contains a `jku`, also known as [JWK Set URI](https://www.rfc-editor.org/rfc/rfc7515#section-4.1.2) which links to a set of JSON-encoded public keys. One of which corresponds to the key used to digitally sign the JWS. The `kid` indicates which key was used to sign the JWS.

+```json

+{

+ "alg": "RS256",

+ "jku": "https://sharedweu.weu.attest.azure.net/certs",

+ "kid": "dRKh+hBcWUfQimSl3Iv6ZhStW3TSOt0ThwiTgUUqZAo=",

+ "typ": "JWT"

+}

+```

+The body of the guest attestation response will get validated by Azure Key Vault as input to test against the key release policy. As previously noted, Azure Key Vault uses the "`TpmEphemeralEncryptionKey`" as the key-encryption key.

+```json

+{

+ "exp": 1671865218,

+ "iat": 1671836418,

+ "iss": "https://sharedweu.weu.attest.azure.net",

+ "jti": "ce395e5de9c638d384cd3bd06041e674edee820305596bba3029175af2018da0",

+ "nbf": 1671836418,

+ "secureboot": true,

+ "x-ms-attestation-type": "azurevm",

+ "x-ms-azurevm-attestation-protocol-ver": "2.0",

+ "x-ms-azurevm-attested-pcrs": [

+ 0,

+ 1,

+ 2,

+ 3,

+ 4,

+ 5,

+ 6,

+ 7

+ ],

+ "x-ms-azurevm-bootdebug-enabled": false,

+ "x-ms-azurevm-dbvalidated": true,

+ "x-ms-azurevm-dbxvalidated": true,

+ "x-ms-azurevm-debuggersdisabled": true,

+ "x-ms-azurevm-default-securebootkeysvalidated": true,

+ "x-ms-azurevm-elam-enabled": false,

+ "x-ms-azurevm-flightsigning-enabled": false,

+ "x-ms-azurevm-hvci-policy": 0,

+ "x-ms-azurevm-hypervisordebug-enabled": false,

+ "x-ms-azurevm-is-windows": false,

+ "x-ms-azurevm-kerneldebug-enabled": false,

+ "x-ms-azurevm-osbuild": "NotApplication",

+ "x-ms-azurevm-osdistro": "Ubuntu",

+ "x-ms-azurevm-ostype": "Linux",

+ "x-ms-azurevm-osversion-major": 20,

+ "x-ms-azurevm-osversion-minor": 4,

+ "x-ms-azurevm-signingdisabled": true,

+ "x-ms-azurevm-testsigning-enabled": false,

+ "x-ms-azurevm-vmid": "6506B531-1634-431E-99D2-42B7D3414AD0",

+ "x-ms-isolation-tee": {

+ "x-ms-attestation-type": "sevsnpvm",

+ "x-ms-compliance-status": "azure-compliant-cvm",

+ "x-ms-runtime": {

+ "keys": [

+ {

+ "e": "AQAB",

+ "key_ops": [

+ "encrypt"

+ ],

+ "kid": "HCLAkPub",

+ "kty": "RSA",

+ "n": "tXkRLAABQ7vgX96..1OQ"

+ }

+ ],

+ "vm-configuration": {

+ "console-enabled": true,

+ "current-time": 1671835548,

+ "secure-boot": true,

+ "tpm-enabled": true,

+ "vmUniqueId": "6506B531-1634-431E-99D2-42B7D3414AD0"

+ }

+ },

+ "x-ms-sevsnpvm-authorkeydigest": "0000000000000..00",

+ "x-ms-sevsnpvm-bootloader-svn": 3,

+ "x-ms-sevsnpvm-familyId": "01000000000000000000000000000000",

+ "x-ms-sevsnpvm-guestsvn": 2,

+ "x-ms-sevsnpvm-hostdata": "0000000000000000000000000000000000000000000000000000000000000000",

+ "x-ms-sevsnpvm-idkeydigest": "57486a44..96",

+ "x-ms-sevsnpvm-imageId": "02000000000000000000000000000000",

+ "x-ms-sevsnpvm-is-debuggable": false,

+ "x-ms-sevsnpvm-launchmeasurement": "ad6de16..23",

+ "x-ms-sevsnpvm-microcode-svn": 115,

+ "x-ms-sevsnpvm-migration-allowed": false,

+ "x-ms-sevsnpvm-reportdata": "c6500..0000000",

+ "x-ms-sevsnpvm-reportid": "cf5ea742f08cb45240e8ad4..7eb7c6c86da6493",

+ "x-ms-sevsnpvm-smt-allowed": true,

+ "x-ms-sevsnpvm-snpfw-svn": 8,

+ "x-ms-sevsnpvm-tee-svn": 0,

+ "x-ms-sevsnpvm-vmpl": 0

+ },

+ "x-ms-policy-hash": "wm9mHlvTU82e8UqoOy1..RSNkfe99-69IYDq9eWs",

+ "x-ms-runtime": {

+ "client-payload": {

+ "nonce": ""

+ },

+ "keys": [

+ {

+ "e": "AQAB",

+ "key_ops": [

+ "encrypt"

+ ],

+ "kid": "TpmEphemeralEncryptionKey", // key-encryption key candidate!

+ "kty": "RSA",

+ "n": "kVTLSwAAQpg..Q"

+ }

+ ]

+ },

+ "x-ms-ver": "1.0"

+}

+```

+The documentation for Microsoft Azure Attestation service has an extensive list containing descriptions of all of these [SEV-SNP-related claims](../attestation/claim-sets.md#sev-snp-attestation).

+## Performing the key release operation

+We can use any scripting or programming language to receive an attested platform report using the AttestationClient binary. Since the virtual machine we deployed in a previous step has managed identity enabled, we should get an __Azure AD token for Key Vault__ from the instance metadata service (__IMDS__).

+By configuring the attested platform report as the body payload and the Azure AD token in our __authorization header__, you have everything needed to perform the key `release` operation.

+```powershell

+#Requires -Version 7

+#Requires -RunAsAdministrator

+#Requires -PSEdition Core

+<#

+.SYNOPSIS

+ Perform Secure Key Release operation in Azure Key Vault, provided this script is running inside an Azure Confidential Virtual Machine.

+.DESCRIPTION

+ Perform Secure Key Release operation in Azure Key Vault, provided this script is running inside an Azure Confidential Virtual Machine.

+ The release key operation is applicable to all key types. The target key must be marked exportable. This operation requires the keys/release permission.

+.PARAMETER -AttestationTenant

+ Provide the attestation instance base URI, for example https://mytenant.attest.azure.net.

+.PARAMETER -VaultBaseUrl

+ Provide the vault name, for example https://myvault.vault.azure.net.

+.PARAMETER -KeyName

+ Provide the name of the key to get.

+.PARAMETER -KeyName

+ Provide the version parameter to retrieve a specific version of a key.

+.INPUTS

+ None.

+.OUTPUTS

+ System.Management.Automation.PSObject

+.EXAMPLE

+ PS C:\> .\Invoke-SecureKeyRelease.ps1 -AttestationTenant "https://sharedweu.weu.attest.azure.net" -VaultBaseUrl "https://mykeyvault.vault.azure.net/" -KeyName "mykey" -KeyVersion "e473cd4c66224d16870bbe2eb4c58078"

+#>

+param (

+ [Parameter(Mandatory = $true)]

+ [string]

+ $AttestationTenant,

+ [Parameter(Mandatory = $true)]

+ [string]

+ $VaultBaseUrl,

+ [Parameter(Mandatory = $true)]

+ [string]

+ $KeyName,

+ [Parameter(Mandatory = $false)]

+ [string]

+ $KeyVersion

+)

+# Check if AttestationClient* exists.

+$fileExists = Test-Path -Path "AttestationClient*"

+if (!$fileExists) {

+ throw "AttestationClient binary not found. Please download it from 'https://github.com/Azure/confidential-computing-cvm-guest-attestation'."

+}

+$cmd = $null

+if ($isLinux) {

+ $cmd = "sudo ./AttestationClient -a $attestationTenant -o token"

+}

+elseif ($isWindows) {

+ $cmd = "./AttestationClientApp.exe -a $attestationTenant -o token"

+}

+$attestedPlatformReportJwt = Invoke-Expression -Command $cmd

+if (!$attestedPlatformReportJwt.StartsWith("eyJ")) {

+ throw "AttestationClient failed to get an attested platform report."

+}

+## Get access token from IMDS for Key Vault

+$imdsUrl = 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net'

+$kvTokenResponse = Invoke-WebRequest -Uri $imdsUrl -Headers @{Metadata = "true" }

+if ($kvTokenResponse.StatusCode -ne 200) {

+ throw "Unable to get access token. Ensure Azure Managed Identity is enabled."

+}

+$kvAccessToken = ($kvTokenResponse.Content | ConvertFrom-Json).access_token

+# Perform release key operation

+if ([string]::IsNullOrEmpty($keyVersion)) {

+ $kvReleaseKeyUrl = "{0}/keys/{1}/release?api-version=7.3" -f $vaultBaseUrl, $keyName

+}

+else {

+ $kvReleaseKeyUrl = "{0}/keys/{1}/{2}/release?api-version=7.3" -f $vaultBaseUrl, $keyName, $keyVersion

+}

+$kvReleaseKeyHeaders = @{

+ Authorization = "Bearer $kvAccessToken"

+ 'Content-Type' = 'application/json'

+}

+$kvReleaseKeyBody = @{

+ target = $attestedPlatformReportJwt

+}

+$kvReleaseKeyResponse = Invoke-WebRequest -Method POST -Uri $kvReleaseKeyUrl -Headers $kvReleaseKeyHeaders -Body ($kvReleaseKeyBody | ConvertTo-Json)

+if ($kvReleaseKeyResponse.StatusCode -ne 200) {

+ Write-Error -Message "Unable to perform release key operation."

+ Write-Error -Message $kvReleaseKeyResponse.Content

+}

+else {

+ $kvReleaseKeyResponse.Content | ConvertFrom-Json

+}

+```

+### Key Release Response

+The secure key release operation only returns a single property inside of its JSON payload. The contents, however, have been base64 encoded as well.

+```json

+{

+ "value": "eyJhbGciOiJSUzI1NiIsImtpZCI6Ijg4RUFDM.."

+}

+```

+Here we have another header, though this one has a [X.509 certificate chain](https://www.rfc-editor.org/rfc/rfc7515#section-4.1.6) as a property.

+```json

+{

+ "alg": "RS256",

+ "kid": "88EAC2DB6BE4E051B0E05AEAF6CB79E675296121",

+ "x5t": "iOrC22vk4FGw4Frq9st55nUpYSE",

+ "typ": "JWT",

+ "x5t#S256": "BO7jbeU3BG0FEjetF8rSisRbkMfcdy0olhcnmYEwApA",

+ "x5c": [

+ "MIIIfDCCBmSgA..XQ==",

+ "MII..8ZZ8m",

+ "MII..lMrY="

+ ]

+}

+```

+You can read from the "`x5c`" array in PowerShell if you wanted to, this can help you verify that this is a valid certificate. Below is an example:

+```powershell

+$certBase64 = "MIIIfDCCBmSgA..XQ=="

+$cert = [System.Security.Cryptography.X509Certificates.X509Certificate2]([System.Convert]::FromBase64String($certBase64))

+$cert | Format-List *

+# NotAfter : 9/18/2023 6:14:06 PM

+# NotBefore : 9/23/2022 6:14:06 PM

+# ...

+# Issuer : CN=Microsoft Azure TLS Issuing CA 06, O=Microsoft Corporation, C=US

+# Subject : CN=vault.azure.net, O=Microsoft Corporation, L=Redmond, S=WA, C=US

+```

+The response's JWT token body looks incredibly similar to the response that you get when invoking the `get` key operation. However, the `release` operation includes the `key_hsm` property, amongst other things.

+```json

+{

+ "request": {

+ "api-version": "7.3",

+ "enc": "CKM_RSA_AES_KEY_WRAP",

+ "kid": "https://mykeyvault.vault.azure.net/keys/mykey"

+ },

+ "response": {

+ "key": {

+ "key": {

+ "kid": "https://mykeyvault.vault.azure.net/keys/mykey/e473cd4c66224d16870bbe2eb4c58078",

+ "kty": "RSA-HSM",

+ "key_ops": [

+ "encrypt",

+ "decrypt"

+ ],

+ "n": "nwFQ8p..20M",

+ "e": "AQAB",

+ "key_hsm": "eyJzY2hlbW..GIifQ"

+ },

+ "attributes": {

+ "enabled": true,

+ "nbf": 1671577355,

+ "exp": 1703113355,

+ "created": 1671577377,

+ "updated": 1671827011,

+ "recoveryLevel": "Recoverable+Purgeable",

+ "recoverableDays": 90,

+ "exportable": true

+ },

+ "tags": {},

+ "release_policy": {

+ "data": "eyJ2ZXJzaW9uIjoiMS4wLjAiLCJhbnlPZiI6W3siYXV0aG9yaXR5IjoiaHR0cHM6Ly9zaGFyZWR3ZXUud2V1LmF0dGVzdC5henVyZS5uZXQiLCJhbGxPZiI6W3siY2xhaW0iOiJ4LW1zLWlzb2xhdGlvbi10ZWUueC1tcy1hdHRlc3RhdGlvbi10eXBlIiwiZXF1YWxzIjoic2V2c25wdm0ifSx7ImNsYWltIjoieC1tcy1pc29sYXRpb24tdGVlLngtbXMtY29tcGxpYW5jZS1zdGF0dXMiLCJlcXVhbHMiOiJhenVyZS1jb21wbGlhbnQtY3ZtIn1dfV19",

+ "immutable": false

+ }

+ }

+ }

+}

+```

+Should your base64 decode the value under `$.response.key.release_policy.data`, you get the JSON representation of the Key Vault key release policy that we defined in an earlier step.

+The `key_hsm` property base64 decoded value looks like this:

+```json

+{

+ "schema_version": "1.0",

+ "header": {

+ "kid": "TpmEphemeralEncryptionKey", // (key identifier of KEK)

+ "alg": "dir", // Direct mode, i.e. the referenced 'kid' is used to directly protect the ciphertext

+ "enc": "CKM_RSA_AES_KEY_WRAP"

+ },

+ "ciphertext": "Rftxvr..lb"

+}

+```

+## Next steps

+[SKR Policy Examples](skr-policy-examples.md)

+[Learn how to use Microsoft Defender for Cloud integration with confidential VMs with guest attestation installed](guest-attestation-defender-for-cloud.md)

+[Learn more about the guest attestation feature](guest-attestation-confidential-vms.md)

+[Learn about Azure confidential VMs](confidential-vm-overview.md)

+ Title: Secure Key Release Policy with Azure Key Vault and Azure Confidential Computing

+description: Examples of AKV SKR policies across offered Azure Confidential Computing Trusted Execution Environments

+# Secure Key Release Policy (SKR) Examples for Confidential Computing (ACC)

+SKR can only release exportable marked keys based on the Microsoft Azure Attestation (MAA) generated claims. There's a tight integration on the SKR policy definition to MAA claims. MAA claims by trusted execution environment (TEE) can be found [here.](../attestation/attestation-token-examples.md)

+Follow the policy [grammar](../key-vault/keys/policy-grammar.md) for more examples on how you can customize the SKR policies.

+## Intel SGX Application Enclaves SKR policy examples

+**Example 1:** Intel SGX based SKR policy validating the MR Signer (SGX enclave signer) details as part of the MAA claims

+```json

+{

+ "anyOf": [

+ {

+ "authority": "https://sharedeus2.eus2.attest.azure.net",

+ "allOf": [

+ {

+ "claim": "x-ms-sgx-mrsigner",

+ "equals": "9fa48b1629bd246a1de3d38fb7df97f6554cd65d6b3b72e85b86848ae6b578ba"

+ }

+ ]

+ }

+ ],

+ "version": "1.0.0"

+}

+```

+**Example 2:** Intel SGX based SKR policy validating the MR Signer (SGX enclave signer) or MR Enclave details as part of the MAA claims

+```json

+{

+ "anyOf": [

+ {

+ "authority": "https://sharedeus2.eus2.attest.azure.net",

+ "allOf": [

+ {

+ "claim": "x-ms-sgx-mrsigner",

+ "equals": "9fa48b1629bd246a1de3d38fb7df97f6554cd65d6b3b72e85b86848ae6b578ba"

+ },

+ {

+ "claim": "x-ms-sgx-mrenclave",

+ "equals": "9fa48b1629bg677jfsaawed7772e85b86848ae6b578ba"

+ }

+ ]

+ }

+ ],

+ "version": "1.0.0"

+}

+```

+**Example 3:** Intel SGX based SKR policy validating the MR Signer (SGX enclave signer) and MR Enclave with a min SVN number details as part of the MAA claims

+```json

+{

+ "anyOf": [

+ {

+ "authority": "https://sharedeus2.eus2.attest.azure.net",

+ "allOf": [

+ {

+ "claim": "x-ms-sgx-mrsigner",

+ "equals": "9fa48b1629bd246a1de3d38fb7df97f6554cd65d6b3b72e85b86848ae6b578ba"

+ },

+ {

+ "claim": "x-ms-sgx-mrenclave",

+ "equals": "9fa48b1629bg677jfsaawed7772e85b86848ae6b578ba"

+ },

+ {

+ "claim": "x-ms-sgx-svn",

+ "greater": 1

+ }

+ ]

+ }

+ ],

+ "version": "1.0.0"

+}

+```

+## Confidential VM AMD SEV-SNP based VM TEE SKR policy examples

+**Example 1:** A SKR policy that validates if this is Azure compliant CVM and is running on a genuine AMD SEV-SNP hardware and the MAA URL authority is spread across many regions.

+```json

+{

+ "version": "1.0.0",

+ "anyOf": [

+ {

+ "authority": "https://sharedweu.weu.attest.azure.net",

+ "allOf": [

+ {

+ "claim": "x-ms-attestation-type",

+ "equals": "sevsnpvm"

+ },

+ {

+ "claim": "x-ms-compliance-status",

+ "equals": "azure-compliant-cvm"

+ }

+ ]

+ },

+ {

+ "authority": "https://sharedeus2.weu2.attest.azure.net",

+ "allOf": [

+ {

+ "claim": "x-ms-attestation-type",

+ "equals": "sevsnpvm"

+ },

+ {

+ "claim": "x-ms-compliance-status",

+ "equals": "azure-compliant-cvm"

+ }

+ ]

+ }

+ ]

+}

+```

+**Example 2:** A SKR policy that validates if the CVM is an Azure compliant CVM and is running on a genuine AMD SEV-SNP hardware and is of a known Virtual Machine ID. (VMIDs are unique across Azure)

+```json

+{

+ "version": "1.0.0",

+ "allOf": [

+ {

+ "authority": "https://sharedweu.weu.attest.azure.net",

+ "allOf": [

+ {

+ "claim": "x-ms-isolation-tee.x-ms-attestation-type",

+ "equals": "sevsnpvm"

+ },

+ {

+ "claim": "x-ms-isolation-tee.x-ms-compliance-status",

+ "equals": "azure-compliant-cvm"

+ },

+ {

+ "claim": "x-ms-azurevm-vmid",

+ "equals": "B958DC88-E41D-47F1-8D20-E57B6B7E9825"

+ }

+ ]

+ }

+ ]

+}

+```

+## Confidential containers on Azure Container Instances (ACI) SKR policy examples

+**Example 1:** Confidential containers on ACI validating the containers initiated and container configuration metadata as part of container group launch with added validations that this is an AMD SEV-SNP hardware.

+> [!NOTE]

+> The containers metadata is a rego based policy hash reflected as in this [example.](https://github.com/microsoft/confidential-sidecar-containers/tree/main).

+```json

+{

+ "version": "1.0.0",

+ "anyOf": [

+ {

+ "authority": "https://fabrikam1.wus.attest.azure.net",

+ "allOf": [

+ {

+ "claim": "x-ms-attestation-type",

+ "equals": "sevsnpvm"

+ },

+ {

+ "claim": "x-ms-compliance-status",

+ "equals": "azure-compliant-uvm"

+ },

+ {

+ "claim": "x-ms-sevsnpvm-hostdata",

+ "equals": "532eaabd9574880dbf76b9b8cc00832c20a6ec113d682299550d7a6e0f345e25"

+ }

+ ]

+ }

+ ]

+}

+```

+## References

+[Microsoft Azure Attestation (MAA)](../attestation/overview.md)

+[Secure Key Release Concept and Basic Steps](concept-skr-attestation.md)

+ Title: Burst capacity (preview)

+description: Use your database or container's idle throughput capacity to handle spikes of traffic with burst capacity in Azure Cosmos DB.

+Burst capacity applies only to Azure Cosmos DB accounts using provisioned throughput (manual and autoscale) and doesn't apply to serverless containers. The feature is configured at the Azure Cosmos DB account level and automatically applies to all databases and containers in the account that have physical partitions with less than 3000 RU/s of provisioned throughput. Resources that have greater than or equal to 3000 RU/s per physical partition can't benefit from or use burst capacity.

+Before enabling the feature, verify that your Azure Cosmos DB account(s) meet all the [preview eligibility criteria](#limitations-preview-eligibility-criteria). Once you've enabled the feature, it takes 15-20 minutes to take effect.

+ Title: Configure periodic backup

+description: Configure Azure Cosmos DB accounts with periodic backup and retention at a specified interval through the portal or a support ticket.

+- Azure Cosmos DB automatically takes a full backup of your database every 4 hours and at any point of time, only the latest two backups are stored by default. If the default intervals aren't sufficient for your workloads, you can change the backup interval and the retention period from the Azure portal. You can change the backup configuration during or after the Azure Cosmos DB account is created. If the container or database is deleted, Azure Cosmos DB retains the existing snapshots of a given provisioned throughput container or shared throughput database for 30 days. If throughput is provisioned at the database level, the backup and restore process happens across the entire database scope.

+- Azure Cosmos DB stores these backups in Azure Blob storage whereas the actual data resides locally within Azure Cosmos DB.

+- To guarantee low latency, the snapshot of your backup is stored in Azure Blob storage in the same region as the current write region (or **one** of the write regions, in case you have a multi-region write configuration). For resiliency against regional disaster, each snapshot of the backup data in Azure Blob storage is again replicated to another region through geo-redundant storage (GRS). The region to which the backup is replicated is based on your source region and the regional pair associated with the source region. To learn more, see the [list of geo-redundant pairs of Azure regions](../availability-zones/cross-region-replication-azure.md) article. You can't access this backup directly. Azure Cosmos DB team restores your backup when you request through a support request.

+ The following image shows how an Azure Cosmos DB container with all the three primary physical partitions in West US. The container is backed up in a remote Azure Blob Storage account in West US and then replicated to East US:

+ :::image type="content" source="./media/configure-periodic-backup-restore/automatic-backup.png" alt-text="Diagram of periodic full backups taken of multiple Azure Cosmos DB entities in geo-redundant Azure Storage." lightbox="./media/configure-periodic-backup-restore/automatic-backup.png" border="false":::

+- The backups are taken without affecting the performance or availability of your application. Azure Cosmos DB performs data backup in the background without consuming any extra provisioned throughput (RUs) or affecting the performance and availability of your database.

+> [!NOTE]

+Change the default geo-redundant backup storage to ensure that your backup data stays within the same region where your Azure Cosmos DB account is provisioned. You can configure the geo-redundant backup to use either locally redundant or zone-redundant storage. Storage redundancy mechanisms store multiple copies of your backups so that it's protected from planned and unplanned events. These events can include transient hardware failure, network or power outages, or massive natural disasters.

+- **Geo-redundant backup storage:** This option copies your data asynchronously across the paired region.

+- **Zone-redundant backup storage:** This option copies your data synchronously across three Azure availability zones in the primary region. For more information, see [Zone-redundant storage.](../storage/common/storage-redundancy.md#redundancy-in-the-primary-region)

+- **Locally-redundant backup storage:** This option copies your data synchronously three times within a single physical location in the primary region. For more information, see [locally redundant storage.](../storage/common/storage-redundancy.md#redundancy-in-the-primary-region)

+Azure Cosmos DB automatically takes a full backup of your data for every 4 hours and at any point of time, the latest two backups are stored. This configuration is the default option and itΓÇÖs offered without any extra cost. You can change the default backup interval and retention period during the Azure Cosmos DB account creation or after the account is created. The backup configuration is set at the Azure Cosmos DB account level and you need to configure it on each account. After you configure the backup options for an account, itΓÇÖs applied to all the containers within that account. You can modify these settings using the Azure portal as described later in this article, or via [PowerShell](configure-periodic-backup-restore.md#modify-backup-options-using-azure-powershell) or the [Azure CLI](configure-periodic-backup-restore.md#modify-backup-options-using-azure-cli).

+If you've accidentally deleted or corrupted your data, **before you create a support request to restore the data, make sure to increase the backup retention for your account to at least seven days. ItΓÇÖs best to increase your retention within 8 hours of this event.** This way, the Azure Cosmos DB team has enough time to restore your account.

+ - **Backup Interval** - ItΓÇÖs the interval at which Azure Cosmos DB attempts to take a backup of your data. Backup takes a nonzero amount of time and in some case it could potentially fail due to downstream dependencies. Azure Cosmos DB tries its best to take a backup at the configured interval, however, it doesnΓÇÖt guarantee that the backup completes within that time interval. You can configure this value in hours or minutes. Backup Interval can't be less than 1 hour and greater than 24 hours. When you change this interval, the new interval takes into effect starting from the time when the last backup was taken.

+ - **Backup Retention** - It represents the period where each backup is retained. You can configure it in hours or days. The minimum retention period canΓÇÖt be less than two times the backup interval (in hours) and it canΓÇÖt be greater than 720 hours.

+ - **Copies of data retained** - By default, two backup copies of your data are offered at free of charge. There's an extra charge if you need more than two copies. See the Consumed Storage section in the [Pricing page](https://azure.microsoft.com/pricing/details/cosmos-db/) to know the exact price for extra copies.

+ - **Backup storage redundancy** - Choose the required storage redundancy option, see the [Backup storage redundancy](#backup-storage-redundancy) section for available options. By default, your existing periodic backup mode accounts have geo-redundant storage if the region where the account is being provisioned supports it. Otherwise, the account fallback to the highest redundancy option available. You can choose other storage such as locally redundant to ensure the backup isn't replicated to another region. The changes made to an existing account are applied to only future backups. After the backup storage redundancy of an existing account is updated, it may take up to twice the backup interval time for the changes to take effect, and **you will lose access to restore the older backups immediately.**

+ > [!NOTE]

+ > You must have the Azure [Azure Cosmos DB Operator role](../role-based-access-control/built-in-roles.md#cosmos-db-operator) role assigned at the subscription level to configure backup storage redundancy.

+ :::image type="content" source="./media/configure-periodic-backup-restore/configure-backup-options-existing-accounts.png" alt-text="Screenshot of configuration options including backup interval, retention, and storage redundancy for an existing Azure Cosmos DB account." border="true":::

+If you accidentally delete your database or a container, you can [file a support ticket](https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade) or [call the Azure support](https://azure.microsoft.com/support/options/) to restore the data from automatic online backups. Azure support is available for selected plans only such as **Standard**, **Developer**, and plans higher than those tiers. Azure support isn't available with **Basic** plan. To learn about different support plans, see the [Azure support plans](https://azure.microsoft.com/support/plans/) page.

+- Have your subscription ID ready.

+- Based on how your data was accidentally deleted or modified, you should prepare to have additional information. It's advised that you have the information available ahead to minimize the back-and-forth that can be detrimental in some time sensitive cases.

+- If the entire Azure Cosmos DB account is deleted, you need to provide the name of the deleted account. If you create another account with the same name as the deleted account, share that with the support team because it helps to determine the right account to choose. It's recommended to file different support tickets for each deleted account because it minimizes the confusion for the state of restore.

+- If one or more databases are deleted, you should provide the Azure Cosmos DB account, and the Azure Cosmos DB database names and specify if a new database with the same name exists.

+- If one or more containers are deleted, you should provide the Azure Cosmos DB account name, database names, and the container names. And specify if a container with the same name exists.

+- If you've accidentally deleted or corrupted your data, you should contact [Azure support](https://azure.microsoft.com/support/options/) within 8 hours so that the Azure Cosmos DB team can help you restore the data from the backups. **Before you create a support request to restore the data, make sure to [increase the backup retention](#modify-the-backup-interval-and-retention-period) for your account to at least seven days. ItΓÇÖs best to increase your retention within 8 hours of this event.** This way the Azure Cosmos DB support team has enough time to restore your account.

+In addition to Azure Cosmos DB account name, database names, container names, you should specify the point in time to use for data restoration. It's important to be as precise as possible to help us determine the best available backups at that time. **It is also important to specify the time in UTC.**

+- Delete the entire Azure Cosmos DB account.

+- Delete one or more Azure Cosmos DB databases.

+- Delete one or more Azure Cosmos DB containers.

+- Delete or modify the Azure Cosmos DB items (for example, documents) within a container. This specific case is typically referred to as data corruption.

+- A shared offer database or containers within a shared offer database are deleted or corrupted.

+Azure Cosmos DB can restore data in all the above scenarios. A new Azure Cosmos DB account is created to hold the restored data when restoring from a backup. The name of the new account, if it's not specified, has the format `<Azure_Cosmos_account_original_name>-restored1`. The last digit is incremented when multiple restores are attempted. You can't restore data to a precreated Azure Cosmos DB account.

+When you accidentally delete an Azure Cosmos DB account, we can restore the data into a new account with the same name, if the account name isn't in use. So, we recommend that you don't re-create the account after deleting it. Because it not only prevents the restored data to use the same name, but also makes discovering the right account to restore from difficult.

+When you accidentally delete an Azure Cosmos DB database, we can restore the whole database or a subset of the containers within that database. It's also possible to select specific containers across databases and restore them to a new Azure Cosmos DB account.

+When you accidentally delete or modify one or more items within a container (the data corruption case), you need to specify the time to restore to. Time is important if there's data corruption. Because the container is live, the backup is still running, so if you wait beyond the retention period (the default is eight hours) the backups would be overwritten. In order to prevent the backup from being overwritten, increase the backup retention for your account to at least seven days. ItΓÇÖs best to increase your retention within 8 hours from the data corruption.

+If you've accidentally deleted or corrupted your data, you should contact [Azure support](https://azure.microsoft.com/support/options/) within 8 hours so that the Azure Cosmos DB team can help you restore the data from the backups. This way the Azure Cosmos DB support team has enough time to restore your account.

+>

+> - VNET access control lists

+> - Stored procedures, triggers and user-defined functions

+> - Multi-region settings

+> - Managed identity settings

+>

+If you assign throughput at the database level, the backup and restore process in this case happen at the entire database level, and not at the individual containers level. In such cases, you can't select a subset of containers to restore.

+Two backups are provided free and extra backups are charged according to the region-based pricing for backup storage described in [backup storage pricing](https://azure.microsoft.com/pricing/details/cosmos-db/). For example, consider a scenario where Backup Retention is configured to **240 hrs** (or 10 days) and Backup Interval is configured to **24** hrs. This configuration implies that there are 10 copies of the backup data. If you have **1 TB** of data in the West US 2 region, the cost would be `0.12 * 1000 * 8` for backup storage in given month.

+1. Open the **Tags** page. This page should have the tags **restoredAtTimestamp** and **restoredSourceDatabaseAccountName**. These tags describe the timestamp and the source account name that were used for the periodic restore.

+Run the following command to get the restore details. The `restoreSourceAccountName` and `restoreTimestamp` fields are within the `tags` field:

+Import the Az.CosmosDB module and run the following command to get the restore details. The `restoreSourceAccountName` and `restoreTimestamp` are within the `tags` field:

+- Use [Azure Data Factory](../data-factory/connector-azure-cosmos-db.md) to move data periodically to a storage solution of your choice.

+- Use Azure Cosmos DB [change feed](change-feed.md) to read data periodically for full backups or for incremental changes, and store it in your own storage.

+The primary goal of the data restore is to recover the data that you've accidentally deleted or modified. So, we recommend that you first inspect the content of the recovered data to ensure it contains what you are expecting. If everything looks good, you can migrate the data back to the primary account. Although it's possible to use the restored account as your new active account, it's not a recommended option if you have production workloads.

+After you restore the data, you get a notification about the name of the new account (itΓÇÖs typically in the format `<original-name>-restored1`) and the time when the account was restored to. The restored account has the same provisioned throughput, indexing policies and it is in same region as the original account. A user who is the subscription admin or a coadmin can see the restored account.

+- Use the [Azure Data Factory](../data-factory/connector-azure-cosmos-db.md).

+- Use the [change feed](change-feed.md) in Azure Cosmos DB.

+- You can write your own custom code.

+It's advised that you delete the container or database immediately after migrating the data. If you don't delete the restored databases or containers, they incur cost for request units, storage, and egress.

+- To make a restore request, contact Azure Support by [filing a ticket in the Azure portal](https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade).

+- [Create account with continuous backup](provision-account-continuous-backup.md).

+- [Restore continuous backup account](restore-account-continuous-backup.md).

+> [!TIP]

+- [Provision a dedicated gateway using the Azure portal](how-to-configure-integrated-cache.md#provision-the-dedicated-gateway)

+> [!NOTE]

+> You can provision a dedicated gateway in Azure Cosmos DB accounts with [availability zones](../availability-zones/az-region.md) by request. Reach out to cosmoscachefeedback@microsoft.com for more information.

+The Azure Cosmos DB integrated cache is an in-memory cache that helps you ensure manageable costs and low latency as your request volume grows. The integrated cache is easy to set up and you donΓÇÖt need to spend time writing custom code for cache invalidation or managing backend infrastructure. The integrated cache uses the [dedicated gateway](dedicated-gateway.md) within your Azure Cosmos DB account. When provisioning your dedicated gateway, you can choose the number of nodes and the node size based on the number of cores and memory needed for your workload. Each dedicated gateway node has a separate integrated cache from the others.

+The integrated cache is a read-through, write-through cache with a Least Recently Used (LRU) eviction policy. The item cache and query cache share the same capacity within the integrated cache and the LRU eviction policy applies to both. Data is evicted from the cache strictly based on when it was least recently used, regardless of whether it's a point read or query. The cached data within each node depends on the data that was recently [written or read](integrated-cache.md#item-cache) through that specific node. If an item or query is cached on one node, it isn't necessarily cached on the others.

+Point reads and queries that hit the integrated cache have an RU charge of 0. Cache hits have a much lower per-operation cost than reads from the backend database.

+Workloads that fit the following characteristics should evaluate if the integrated cache helps lower costs:

+The biggest factor in expected savings is the degree to which reads repeat themselves. If your workload consistently executes the same point reads or queries within a short period of time, it's a great candidate for the integrated cache. When using the integrated cache for repeated reads, you only use RUs for the first read. Subsequent reads routed through the same dedicated gateway node (within the `MaxIntegratedCacheStaleness` window and if the data hasn't been evicted) don't use throughput.

+Item cache is used for point reads (key/value look ups based on the Item ID and partition key).

+- New writes, updates, and deletes are automatically populated in the item cache of the node that the request is routed through

+- Items from point read requests where the item isnΓÇÖt already in the cache (cache miss) of the node the request is routed through are added to the item cache

+- Requests that are part of a [transactional batch](./nosql/transactional-batch.md) or written in [bulk mode](./nosql/how-to-migrate-from-bulk-executor-library.md#enable-bulk-support) don't populate the item cache

+Because each node has an independent cache, it's possible items are invalidated or evicted in the cache of one node and not the others. Items in the cache of a given node are invalidated and evicted based on the below criteria:

+The query cache is used to cache queries. The query cache transforms a query into a key/value lookup where the key is the query text and the value is the query results. The integrated cache doesn't have a query engine, it only stores the key/value lookup for each query. Query results are stored as a set, and the cache doesn't keep track of individual items. A given item can be stored in the query cache multiple times if it appears in the result set of multiple queries. Updates to the underlying items won't be reflected in query results unless the [max integrated cache staleness](#maxintegratedcachestaleness) for the query is reached and the query is served from the backend database.

+- If the cache doesn't have a result for that query (cache miss) on the node it was routed through, the query is sent to the backend. After the query is run, the cache will store the results for that query

+- Queries with the same shape but different parameters or request options that affect the results (ex. max item count) will be stored as their own key/value pair

+Query cache eviction is based on the node the request was routed through. It's possible queries could be evicted or refreshed on one node and not the others.

+The query cache automatically caches query continuation tokens where applicable. If you have a query with multiple pages of results, any pages that are stored in the integrated cache have an RU charge of 0. If subsequent pages of query results require backend execution, they'll have a continuation token from the previous page so they can avoid duplicating previous work.

+> [!IMPORTANT]

+The integrated cache supports read requests with session and eventual [consistency](consistency-levels.md) only. If a read has consistent prefix, bounded staleness, or strong consistency, it bypasses the integrated cache and is served from the backend.

+> Write requests with other consistencies still populate the cache, but in order to read from the cache the request must have either session or eventual consistency.

+[Session consistency](consistency-levels.md#session-consistency) is the most widely used consistency level for both single region and globally distributed Azure Cosmos DB accounts. With session consistency, single client sessions can read their own writes. Clients outside of the session performing writes will see eventual consistency when they are using the integrated cache.

+It's important to understand that the `MaxIntegratedCacheStaleness`, when configured on a request that ends up populating the cache, doesn't affect how long that request is cached. `MaxIntegratedCacheStaleness` enforces consistency when you try to use cached data. There's no global TTL or cache retention setting, so data is only evicted from the cache if either the integrated cache is full or a new read is run with a lower `MaxIntegratedCacheStaleness` than the age of the current cached entry.

+This is an improvement from how most caches work and allows for the following other customizations:

+- If you wanted to modify read consistency for cached data, changing `MaxIntegratedCacheStaleness` has an immediate effect on read consistency

+All existing metrics are available, by default, from **Metrics** in the Azure portal (not Metrics classic):

+ Title: Use stored procedures, triggers, and UDFs in SDKs

+| Java | [Quickstart: Build a Java app to manage Azure Cosmos DB for NoSQL data](quickstart-java.md) |

+> [!IMPORTANT]

+> The following code samples assume that you have already have `client` and `container` variables. If you need to create those variables, refer to the appropriate quickstart for your platform.

+## <a id="how-to-run-pre-triggers"></a>How to run pretriggers

+The following examples show how to register and call a pretrigger by using the Azure Cosmos DB SDKs. For the source of this pretrigger example, saved as *trgPreValidateToDoItemTimestamp.js*, see [Pretriggers](how-to-write-stored-procedures-triggers-udfs.md#pre-triggers).

+When you run an operation by specifying `PreTriggerInclude` and then passing the name of the trigger in a `List` object, pretriggers are passed in the `RequestOptions` object.

+The following code shows how to register a pretrigger using the .NET SDK v2:

+The following code shows how to call a pretrigger using the .NET SDK v2:

+The following code shows how to register a pretrigger using the .NET SDK v3:

+The following code shows how to call a pretrigger using the .NET SDK v3:

+The following code shows how to register a pretrigger using the Java SDK:

+The following code shows how to call a pretrigger using the Java SDK:

+The following code shows how to register a pretrigger using the JavaScript SDK:

+The following code shows how to call a pretrigger using the JavaScript SDK:

+The following code shows how to register a pretrigger using the Python SDK:

+The following code shows how to call a pretrigger using the Python SDK:

+ Title: ODBC driver for BI and analytics

+description: Use the ODBC driver for Azure Cosmos DB to create normalized data tables and views for SQL queries, analytics, BI, and visualizations.

+The ODBC driver normalizes Azure Cosmos DB data into tables and views that fit your data analytics and reporting needs. The normalized schemas let you use ODBC-compliant tools to access the data. The schemas have no effect on the underlying data, and don't require developers to adhere to them. The ODBC driver helps make Azure Cosmos DB databases useful for data analysts and development teams.

+> [!IMPORTANT]

+> Consider using [Azure Synapse Link for Azure Cosmos DB](../synapse-link.md) to create tables and views for your data. Synapse Link has distinct performance benefits for large datasets over the ODBC driver. You can also connect the normalized Azure Cosmos DB data to other software solutions, such as SQL Server Integration Services (SSIS), QlikSense, Tableau and other analytics software, BI, and data integration tools. You can use those solutions to analyze, move, transform, and create visualizations with your Azure Cosmos DB data.

+>

+>

+ | Installer | Supported operating systems |

+ | | |

+ | [Microsoft Azure Cosmos DB ODBC 64-bit.msi](https://aka.ms/cosmos-odbc-64x64) for 64-bit Windows | 64-bit versions of Windows 8.1 or later, Windows 8, Windows 7. 64-bit versions of Windows Server 2012 R2, Windows Server 2012, and Windows Server 2008 R2. |

+ | [Microsoft Azure Cosmos DB ODBC 32x64-bit.msi](https://aka.ms/cosmos-odbc-32x64) for 32-bit on 64-bit Windows | 64-bit versions of Windows 8.1 or later, Windows 8, Windows 7, Windows XP, Windows Vista. 64-bit versions of Windows Server 2012 R2, Windows Server 2012, Windows Server 2008 R2, and Windows Server 2003. |

+ | [Microsoft Azure Cosmos DB ODBC 32-bit.msi](https://aka.ms/cosmos-odbc-32x32) for 32-bit Windows | 32-bit versions of Windows 8.1 or later, Windows 8, Windows 7, Windows XP, and Windows Vista. |

+ :::image type="content" source="./media/odbc-driver/odbc-driver.png" alt-text="Screenshot of the ODBC Data Source Administrator window.":::

+1. In the **DocumentDB ODBC Driver DSN Setup** window, fill in the following information:

+ :::image type="content" source="./media/odbc-driver/odbc-driver-dsn-setup.png" alt-text="Screenshot of the domain name server (DNS) setup window.":::

+ - **Data Source Name**: A friendly name for the ODBC DSN. This name is unique to this Azure Cosmos DB account.

+ - **Description**: A brief description of the data source.

+ - **Host**: The URI for your Azure Cosmos DB account. You can get this information from the **Keys** page in your Azure Cosmos DB account in the Azure portal.

+ - **Access Key**: The primary or secondary, read-write or read-only key from the Azure Cosmos DB **Keys** page in the Azure portal. It's best to use the read-only keys, if you use the DSN for read-only data processing and reporting.

+ To avoid an authentication error, use the copy buttons to copy the URI and key from the Azure portal.

+ :::image type="content" source="./media/odbc-driver/odbc-cosmos-account-keys.png" alt-text="Screenshot of the Azure Cosmos DB Keys page.":::

+ - **Encrypt Access Key for**: Select the best choice, based on who uses the machine.

+ - **REST API Version**: Select the [REST API version](/rest/api/cosmos-db) for your operations. The default is **2015-12-16**.

+ If you have containers with [large partition keys](../large-partition-keys.md) that need REST API version `2018-12-31`, type `2018-12-31`, and then [follow the steps at the end of this procedure](#edit-the-windows-registry-to-support-rest-api-version-2018-12-31).

+ - **Query Consistency**: Select the [consistency level](../consistency-levels.md) for your operations. The default is **Session**.

+ - **Number of Retries**: Enter the number of times to retry an operation if the initial request doesn't complete due to service rate limiting.

+ - **Schema File**: If you don't select a schema file, the driver scans the first page of data for each container to determine its schema, called container mapping, for each session. This process can cause long startup time for applications that use the DSN. It's best to associate a schema file to the DSN.

+ - If you already have a schema file, select **Browse**, navigate to the file, select **Save**, and then select **OK**.

+ - If you don't have a schema file yet, select **OK**, and then follow the steps in the next section to [create a schema definition](#create-a-schema-definition). After you create the schema, come back to this **Advanced Options** window to add the schema file.

+ - Value Name: **IgnoreSessionToken**

+ - Value data: **1**

+ :::image type="content" source="./media/odbc-driver/cosmos-odbc-edit-registry.png" alt-text="Screenshot that shows the Windows Registry Editor settings.":::

+ For example, if a document contains a **Type** property, you can scope the sampling to the values of this property. The end result of the sampling is a set of tables for each of the **Type** values you specified. **Type = Car** produces a **Car** table, while **Type = Plane** produces a **Plane** table.

+ :::image type="content" source="./media/odbc-driver/odbc-driver-schema-editor.png" alt-text="Screenshot that shows the Schema Editor button in the D S N Setup window.":::

+1. In the **Schema Editor** window, select **Create New**.

+ Or, to use *table-delimiter* mapping, take the following steps to define attributes and values for scoping the sample.

+ 1. Select **Edit** in the **Mapping Definition** column for your DSN.

+ 1. In the **Mapping Definition** window, under **Mapping Method**, select **Table Delimiters**.

+ 1. In the **Attributes** box, type the name of a delimiter property in your document that you want to scope the sampling to, for instance, *City*. Press Enter.

+ 1. If you want to scope the sampling to certain values for the attribute you entered, select the attribute, and then enter a value in the **Value** box, such as *Seattle*, and press Enter. You can add multiple values for attributes. Just make sure that the correct attribute is selected when you enter values.

+ 1. When you're done entering attributes and values, select **OK**.

+ 1. In the **Generate Schema** window, select **Sample**.

+ For each column, you can modify the **SQL name**, the **SQL type**, **SQL length**, **Scale**, **Precision**, and **Nullable** as applicable.

+ You can set **Hide Column** to **true** if you want to exclude that column from query results. Columns marked **Hide Column = true** aren't returned for selection and projection, although they're still part of the schema. For example, you can hide all of the Azure Cosmos DB system required properties that start with **_**. The **id** column is the only field you can't hide, because it's the primary key in the normalized schema.

+ :::image type="content" source="./media/odbc-driver/odbc-driver-create-view.png" alt-text="Screenshot of creating a view within the driver.":::

+ :::image type="content" source="./media/odbc-driver/odbc-driver-power-bi-get-data.png" alt-text="Screenshot showing Get Data in Power B I Desktop.":::

+ :::image type="content" source="./media/odbc-driver/odbc-driver-power-bi-get-data-2.png" alt-text="Screenshot that shows choosing ODBC data source in Power B I Get Data.":::

+ :::image type="content" source="./media/odbc-driver/odbc-driver-power-bi-get-data-3.png" alt-text="Screenshot that shows choosing the D S N in Power B I Get Data.":::

+ :::image type="content" source="./media/odbc-driver/odbc-driver-power-bi-get-data-4.png" alt-text="Screenshot of selecting the table in Power B I Get Data.":::

+ ```output

+ [HY000]: [Microsoft][Azure Cosmos DB] (401) HTTP 401 Authentication Error: {"code":"Unauthorized","message":"The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'get\ndbs\n\nfri, 20 jan 2017 03:43:55 gmt\n\n'\r\nActivityId: 9acb3c0d-cb31-4b78-ac0a-413c8d33e373"}

+ ```

+ **Solution:** Make sure the **Host** and **Access Key** values you copied from the Azure portal are correct, and retry.

+ ```output

+ Msg 7312, Level 16, State 1, Line 44

+

+ Invalid use of schema or catalog for OLE DB provider "MSDASQL" for linked server "DEMOCOSMOS". A four-part name was supplied, but the provider does not expose the necessary interfaces to use a catalog or schema.

+ ```

+ **Solution**: A linked Azure Cosmos DB server doesn't support four-part naming.

+ Title: |

+ Tutorial: Query data

+description: In this tutorial, learn how to query data in Azure Cosmos DB for NoSQL with the built-in query syntax using the Data Explorer.

+# Tutorial: Query data in Azure Cosmos DB for NoSQL

+[Azure Cosmos DB for NoSQL](../introduction.md) supports querying documents using the built-in query syntax. This article provides a sample document and two sample queries and results.

+This article covers the following tasks:

+>

+> - Query NoSQL data with the built-in query syntax

+>

+## Prerequisites

+This tutorial assumes you have an Azure Cosmos DB account, database, and container.

+Don't have any of those resources? Complete this quickstart: [Create an Azure Cosmos DB account, database, container, and items from the Azure portal](quickstart-portal.md).

+You can run the queries using the [Azure Cosmos DB Explorer](../data-explorer.md) in the Azure portal. You can also run queries by using the [REST API](/rest/api/cosmos-db/) or [various SDKs](sdk-dotnet-v3.md).

+For more information about queries, see [setting started with queries](query/getting-started.md).

+The queries in this article use the following sample document.

+ { "familyName": "Wakefield", "givenName": "Robin" },

+ { "familyName": "Miller", "givenName": "Ben" }

+ {

+ "familyName": "Merriam",

+ "givenName": "Jesse",

+ "gender": "female", "grade": 1,

+ "pets": [

+ { "givenName": "Goofy" },

+ { "givenName": "Shadow" }

+ ]

+ },

+ {

+ "familyName": "Miller",

+ "givenName": "Lisa",

+ "gender": "female",

+ "grade": 8

+ }

+## Select all fields and apply a filter

+Given the sample family document, the following query returns the documents where the ID field matches `WakefieldFamily`. Since it's a `SELECT *` statement, the output of the query is the complete JSON document:

+Query:

+SELECT *

+FROM Families f

+WHERE f.id = "WakefieldFamily"

+Results:

+ { "familyName": "Wakefield", "givenName": "Robin" },

+ { "familyName": "Miller", "givenName": "Ben" }

+ {

+ "familyName": "Merriam",

+ "givenName": "Jesse",

+ "gender": "female", "grade": 1,

+ "pets": [

+ { "givenName": "Goofy" },

+ { "givenName": "Shadow" }

+ ]

+ },

+ {

+ "familyName": "Miller",

+ "givenName": "Lisa",

+ "gender": "female",

+ "grade": 8

+ }

+## Select a cross-product of a child collection field

+Query:

+SELECT c.givenName

+FROM Families f

+JOIN c IN f.children

+WHERE f.id = 'WakefieldFamily'

+Results:

+```json

+ {

+ "givenName": "Jesse"

+ },

+ {

+ "givenName": "Lisa"

+ }

+>

+> - Learned how to query using the built-in query syntax

+>

+ Title: Get started with Azure Cosmos DB Partial Document Update

+description: Learn how to use Partial Document Update with .NET, Java, and Node SDKs for Azure Cosmos DB with these examples.

+# Get started with Azure Cosmos DB Partial Document Update

+This article provides examples that illustrate how to use Partial Document Update with .NET, Java, and Node SDKs. It also describes common errors that you might encounter.

+This article links to code samples for the following scenarios:

+- Run a single patch operation

+- Combine multiple patch operations

+- Use conditional patch syntax based on filter predicate

+- Run patch operation as part of a transaction

+Support for Partial Document Update (Patch API) in the [Azure Cosmos DB .NET v3 SDK](nosql/sdk-dotnet-v3.md) is available starting with version *3.23.0*. You can download it from the [NuGet Gallery](https://www.nuget.org/packages/Microsoft.Azure.Cosmos/3.23.0).

+> Find a complete Partial Document Update sample in the [.NET v3 samples repository](https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos.Samples/Usage/ItemManagement/Program.cs) on GitHub.

+- Run a single patch operation:

+- Combine multiple patch operations:

+- Use conditional patch syntax based on filter predicate:

+- Run patch operation as a part of a transaction:

+Support for Partial Document Update (Patch API) in the [Azure Cosmos DB Java v4 SDK](nosql/sdk-java-v4.md) is available starting with version *4.21.0*. You can either add it to the list of dependencies in your `pom.xml` or download it directly from [Maven](https://mvnrepository.com/artifact/com.azure/azure-cosmos).

+> Find the full sample in the [Java SDK v4 samples repository](https://github.com/Azure-Samples/azure-cosmos-java-sql-api-samples/tree/main/src/main/java/com/azure/cosmos/examples/patch/sync) on GitHub.

+- Run a single patch operation:

+- Combine multiple patch operations:

+- Use conditional patch syntax based on filter predicate:

+- Run patch operation as a part of a transaction:

+Support for Partial Document Update (Patch API) in the [Azure Cosmos DB JavaScript SDK](nosql/sdk-nodejs.md) is available starting with version *3.15.0*. You can download it from the [npm Registry](https://www.npmjs.com/package/@azure/cosmos).

+> Find a complete Partial Document Update sample in the [.js v3 samples repository](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/cosmosdb/cosmos/samples/v3/typescript/src/ItemManagement.ts#L167) on GitHub. In the sample, as the container is created without a partition key specified, the JavaScript SDK resolves the partition key values from the items through the container's partition key definition.

+- Run a single patch operation:

+- Combine multiple patch operations:

+- Use conditional patch syntax based on filter predicate:

+## Support for server-side programming

+Partial Document Update operations can also be [executed on the server-side](stored-procedures-triggers-udfs.md) using stored procedures, triggers, and user-defined functions.

+> Find the definition of `validateOptionsAndCallback` in the [.js DocDbWrapperScript](https://github.com/Azure/azure-cosmosdb-js-server/blob/1dbe69893d09a5da29328c14ec087ef168038009/utils/DocDbWrapperScript.js#L289) on GitHub.

+Sample parameter for patch operation:

+```javascript

+function () {

+ var doc = {

+ "id": "exampleDoc",

+ "field1": {

+ "field2": 10,

+ "field3": 20

+ }

+ };

+ var isAccepted = __.createDocument(__.getSelfLink(), doc, (err, doc) => {

+ if (err) throw err;

+ var patchSpec = [

+ {"op": "add", "path": "/field1/field2", "value": 20},

+ {"op": "remove", "path": "/field1/field3"}

+ ];

+ isAccepted = __.patchDocument(doc._self, patchSpec, (err, doc) => {

+ if (err) throw err;

+ else {

+ getContext().getResponse().setBody(docPatched);

+ }

+ }

+ }

+ if(!isAccepted) throw new Error("patch was't accepted")

+ }

+ }

+ if(!isAccepted) throw new Error("create wasn't accepted")

+}

+```

+Here's some common errors that you might encounter while using this feature:

+| Invalid patch request: check syntax of patch specification. | The patch operation syntax is invalid. For more information, see [the Partial Document Update specification](partial-document-update.md#rest-api-reference-for-partial-document-update). |

+| Invalid patch request: Can't patch system property `SYSTEM_PROPERTY`. | System-generated properties likeΓÇ»`_id`,ΓÇ»`_ts`,ΓÇ»`_etag`,ΓÇ»`_rid` aren't modifiable using a patch operation. For more information, see [Partial Document Update FAQs](partial-document-update-faq.yml#is-partial-document-update-supported-for-system-generated-properties-). |

+| The number of patch operations can't exceed 10. | There's a limit of 10 patch operations that can be added in a single patch specification. For more information, see [Partial Document Update FAQs](partial-document-update-faq.yml#is-there-a-limit-to-the-number-of-partial-document-update-operations-). |

+| For Operation(`PATCH_OPERATION_INDEX`): Index(`ARRAY_INDEX`) to operate on is out of array bounds. | The index of array element to be patched is out of bounds. |

+| For Operation(`PATCH_OPERATION_INDEX`)): Node(`PATH`) to be replaced has been removed earlier in the transaction. | The path you're trying to patch doesn't exist. |

+| For Operation(`PATCH_OPERATION_INDEX`): Node(`PATH`) to be removed is absent. Note: it might also have been removed earlier in the transaction.ΓÇ»| The path you're trying to patch doesn't exist. |

+| For Operation(`PATCH_OPERATION_INDEX`): Node(`PATH`) to be replaced is absent. | The path you're trying to patch doesn't exist. |

+| For Operation(`PATCH_OPERATION_INDEX`): Node(`PATH`) isn't a number. | Increment operation can only work on integer and float. For more information, see: [Supported Operations](partial-document-update.md#supported-operations). |

+| For Operation(`PATCH_OPERATION_INDEX`): Add Operation can only create a child object of an existing node (array or object) and can't create path recursively, no path found beyond: `PATH`. | Child paths can be added to an object or array node type. Also, to create `n`th child, `n-1`th child should be present. |

+| For Operation(`PATCH_OPERATION_INDEX`): Given Operation can only create a child object of an existing node(array or object) and can't create path recursively, no path found beyond: `PATH`. | Child paths can be added to an object or array node type. Also, to create `n`th child, `n-1`th child should be present. |

+- [Partial Document Update in Azure Cosmos DB](partial-document-update.md)

+- [Frequently asked questions about Partial Document Update in Azure Cosmos DB](partial-document-update-faq.yml)

+This article describes how to add or change the administrator role for a user using Azure RBAC at the subscription scope.

+* [Administrator role permissions in Azure Active Directory](../../active-directory/roles/permissions-reference.md)

+ :::image type="content" source="media/concept-managed-airflow/architecture.png" lightbox="media/concept-managed-airflow/architecture.png" alt-text="Screenshot shows architecture in Managed Airflow.":::

+| Worker Nodes | Driver Nodes | Total Nodes | Notes |

+

+- **Altering repository**. ADF manages GIT repository content automatically. Altering or adding manually unrelated files or folder into anywhere in ADF Git repository data folder could cause resource loading errors. For example, presence of *.bak* files can cause ADF CI/CD error, so they should be removed for ADF to load.

+ "checkPointKey":"CheckPointFor_ZPERFCDPOS$F",

+ "checkPointKey":"CheckPointFor_Z0131",

+ from airflow.models import DAG, BaseOperator

+ try:

+ from airflow.operators.empty import EmptyOperator

+ except ModuleNotFoundError:

+ from airflow.operators.dummy import DummyOperator as EmptyOperator # type: ignore

+ from airflow.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperator

+ from airflow.providers.microsoft.azure.sensors.data_factory import AzureDataFactoryPipelineRunStatusSensor

+ from airflow.utils.edgemodifier import Label

+ with DAG(

+ dag_id="example_adf_run_pipeline",

+ start_date=datetime(2022, 5, 14),

+ schedule_interval="@daily",

+ catchup=False,

+ default_args={

+ "retries": 1,

+ "retry_delay": timedelta(minutes=3),

+ "azure_data_factory_conn_id": "<connection_id>", #This is a connection created on Airflow UI

+ "factory_name": "<FactoryName>", # This can also be specified in the ADF connection.

+ "resource_group_name": "<ResourceGroupName>", # This can also be specified in the ADF connection.

+ },

+ default_view="graph",

+ ) as dag:

+ begin = EmptyOperator(task_id="begin")

+ end = EmptyOperator(task_id="end")

+ # [START howto_operator_adf_run_pipeline]

+ run_pipeline1: BaseOperator = AzureDataFactoryRunPipelineOperator(

+ task_id="run_pipeline1",

+ pipeline_name="<PipelineName>",

+ parameters={"myParam": "value"},

+ )

+ # [END howto_operator_adf_run_pipeline]

+ # [START howto_operator_adf_run_pipeline_async]

+ run_pipeline2: BaseOperator = AzureDataFactoryRunPipelineOperator(

+ task_id="run_pipeline2",

+ pipeline_name="<PipelineName>",

+ wait_for_termination=False,

+ pipeline_run_sensor: BaseOperator = AzureDataFactoryPipelineRunStatusSensor(

+ task_id="pipeline_run_sensor",

+ run_id=run_pipeline2.output["run_id"],

+ )

+ # [END howto_operator_adf_run_pipeline_async]

+ begin >> Label("No async wait") >> run_pipeline1

+ begin >> Label("Do async wait with sensor") >> run_pipeline2

+ [run_pipeline1, pipeline_run_sensor] >> end

+ # Task dependency created via `XComArgs`:

+ # run_pipeline2 >> pipeline_run_sensor

+ You will have to create the connection using the Airflow UI (Admin -> Connections -> '+' -> Choose 'Connection type' as 'Azure Data Factory', then fill in your **client_id**, **client_secret**, **tenant_id**, **subscription_id**, **resource_group_name**, **data_factory_name**, and **pipeline_name**.

+`.\Add-AdlaJobUser.ps1 -Account myadlsaccount -EntityIdToAdd 546e153e-0ecf-417b-ab7f-aa01ce4a7bff -EntityType User -FullReplication`

+Microsoft Dev Box Preview gives you self-service access to high-performance, preconfigured, and ready-to-code cloud-based workstations called dev boxes. You can set up dev boxes with tools, source code, and prebuilt binaries that are specific to a project, so developers can immediately start work. If you're a developer, you can use dev boxes in your day-to-day workflows.

+The Dev Box service was designed with three organizational roles in mind: dev infrastructure (infra) admins, developer team leads, and developers.

+Dev infra admins and IT admins work together to provide developer infrastructure and tools to the developer teams. Dev infra admins set and manage security settings, network configurations, and organizational policies to ensure that dev boxes can access resources securely.

+Developer team leads are experienced developers who have in-depth knowledge of their projects. They can be assigned the DevCenter Project Admin role and assist with creating and managing the developer experience. Project admins create and manage pools of dev boxes.

+Members of a development team are assigned the DevCenter Dev Box User role. They can then self-serve one or more dev boxes on demand from the dev box pools that have been enabled for a project. Dev box users can work on multiple projects or tasks by creating multiple dev boxes.

+- Manage dev boxes like any other device on your network:

+### Developer team lead scenarios

+After a developer team lead is assigned the DevCenter Project Admin role, they can help manage the project. Project Admins can:

+- Create dev box pools and add appropriate dev box definitions.

+- Control costs by using auto-stop schedules.

+### Developer scenarios

+An organization that has globally distributed development teams can configure Dev Box to enable developers to create their own dev boxes in their closest region. Developers can create dev boxes as needed, without waiting for the IT admin team. Users can access dev boxes from any device and from any operating system.

+Dev Box supports developers who are working on multiple projects. Developers can create and use separate dev boxes for separate workloads, projects, or tasks. Developers can create multiple dev boxes from a predefined pool whenever they need them, and then delete those dev boxes when they're done.

+Organizations can even define dev boxes for various roles on a team. You might configure standard dev boxes with admin rights to give full-time developers greater control, while applying more restricted permissions for contractors.

+Azure network connections enable dev boxes to communicate with your organization's network. The network connection provides a link between the dev center and your organization's virtual networks. In the network connection, you define how a dev box joins Azure AD. Use an Azure AD join to connect exclusively to cloud-based resources, or use a hybrid Azure AD join to connect to on-premises resources and cloud-based resources.

+ Title: 'Quickstart: Create an Azure DNS zone and record using Terraform'

+description: 'In this article, you create an Azure DNS zone and record using Terraform'

+# Quickstart: Create an Azure DNS zone and record using Terraform

+This article shows how to use [Terraform](/azure/terraform) to create an [Azure DNS zone](/azure/dns/dns-zones-records) and an [A record](/azure/dns/dns-alias) in that zone.

+In this article, you learn how to:

+> [!div class="checklist"]

+> * Create a random pet name for the Azure resource group name using [random_pet](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/pet)

+> * Create an Azure resource group using [azurerm_resource_group](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group)

+> * Create a random string using [random_string](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string)

+> * Create an Azure DNS zone using [azurerm_dns_zone](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/dns_zone)

+> * Create an Azure DNS A record using [azurerm_dns_a_record](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/dns_a_record)

+## Prerequisites

+- [Install and configure Terraform](/azure/developer/terraform/quickstart-configure)

+## Implement the Terraform code

+> [!NOTE]

+> The example code for this article is located in the [Azure Terraform GitHub repo](https://github.com/Azure/terraform/tree/master/quickstart/101-dns_zone). See more [articles and sample code showing how to use Terraform to manage Azure resources](/azure/terraform)

+1. Create a directory in which to test and run the sample Terraform code and make it the current directory.

+1. Create a file named `providers.tf` and insert the following code:

+ [!code-terraform[master](~/terraform_samples/quickstart/101-dns_zone/providers.tf)]

+1. Create a file named `main.tf` and insert the following code:

+ [!code-terraform[master](~/terraform_samples/quickstart/101-dns_zone/main.tf)]

+1. Create a file named `variables.tf` and insert the following code:

+ [!code-terraform[master](~/terraform_samples/quickstart/101-dns_zone/variables.tf)]

+1. Create a file named `outputs.tf` and insert the following code:

+ [!code-terraform[master](~/terraform_samples/quickstart/101-dns_zone/outputs.tf)]

+## Initialize Terraform

+## Create a Terraform execution plan

+## Apply a Terraform execution plan

+## Verify the results

+#### [Azure CLI](#tab/azure-cli)

+1. Get the Azure resource group name.

+ ```console

+ resource_group_name=$(terraform output -raw resource_group_name)

+ ```

+1. Get the DNS zone name.

+ ```console

+ dns_zone_name=$(terraform output -raw dns_zone_name)

+ ```

+1. Run [az network dns zone show](/cli/azure/network/dns/zone#az-network-dns-zone-show) to display information about the new DNS zone.

+ ```azurecli

+ az network dns zone show \

+ --resource-group $resource_group_name \

+ --name $dns_zone_name

+ ```

+#### [Azure PowerShell](#tab/azure-powershell)

+1. Get the Azure resource group name.

+ ```console

+ $resource_group_name=$(terraform output -raw resource_group_name)

+ ```

+1. Get the DNS zone name.

+ ```console

+ $dns_zone_name=$(terraform output -raw dns_zone_name)

+ ```

+1. Run [Get-AzDnsZone](/powershell/module/az.dns/get-azdnszone) to display information about the new service.

+ ```azurepowershell

+ Get-AzDnsZone -ResourceGroupName $resource_group_name `

+ -Name $dns_zone_name

+ ```

+## Clean up resources

+## Troubleshoot Terraform on Azure

+[Troubleshoot common problems when using Terraform on Azure](/azure/developer/terraform/troubleshoot)

+## Next steps

+> [!div class="nextstepaction"]

+> [Learn more about Azure DNS](/azure/dns)

+ "subjectBeginsWith": "/blobServices/default/containers/mycontainer/blobs/log",

+ using Microsoft.Azure.ResourceManager.EventHubs;

+ using Microsoft.Azure.ResourceManager.EventHubs.Models;

+>Enabling BFD on the ExpressRoute circuits will help with faster link failure detection between Microsoft Enterprise Edge (MSEE) devices and the Customer/Partner Edge routers. However, the overall failover and convergence to redundant site may take up to 180 seconds under some failure conditions and you may experience increased latency or performance degradation during this time.

+> Deploying NAT gateway with a [zone redundant firewall](deploy-availability-zone-powershell.md) is not recommended deployment option, as the NAT gateway does not support zonal deployment at this time. In order to use NAT gateway with Azure Firewall, a zonal Firewall deployment is required.

+ Title: Open-source components and versions - Azure HDInsight 5.x

+description: Learn about the open-source components and versions in Azure HDInsight 5.x

+# HDInsight 5.x component versions

+In this article, you learn about the open-source components and their versions in Azure HDInsight 5.x.

+## Public preview

+From February 27, 2023 we have started rolling out a new version of HDInsight 5.1, this version is backward compatible with HDInsight 4.0. and 5.0. All new open-source releases added as incremental releases on HDInsight 5.1.

+**Only Kafka and HBase clusters are supported now.**

+## Open-source components available with HDInsight version 5.x

+The Open-source component versions associated with HDInsight 5.1 listed in the following table.

+| Component | HDInsight 5.1 | HDInsight 5.0 |

+||||

+| Apache Spark | 3.3 * | 3.1.3 |

+| Apache Hive | 3.1.2 * | 3.1.2 |

+| Apache Kafka | 3.2.0 ** | 2.4.1 |

+| Apache Hadoop with YARN | 3.3.4 * | 3.1.1 |

+| Apache Tez | 0.9.1 * | 0.9.1 |

+| Apache Pig | 0.17.0 * | 0.16.1 |

+| Apache Ranger | 2.1.0 * | 1.1.0 |

+| Apache HBase | 2.4.11 ** | - |

+| Apache Sqoop | 1.5.0 * | 1.5.0 |

+| Apache Oozie | 5.2.1 * | 4.3.1 |

+| Apache Zookeeper | 3.6.3 * | 3.4.6 |

+| Apache Livy | 0.7.1 * | 0.5 |

+| Apache Ambari | 2.7.0 ** | 2.7.0 |

+| Apache Zeppelin | 0.10.0 * | 0.8.0 |

+| Apache Phoenix | 5.1.2 ** | - |

+\* Under development/Planned

+** Public Preview

+> [!NOTE]

+> ESP isn't supported for Kafka and HBase in this release.

+### Spark versions supported in Azure HDInsight

+Apache Spark versions supported in Azure HDIinsight

+|Apache Spark version on HDInsight|Release date|Release stage|End of life announcement date|End of standard support|End of basic support|

+|--|--|--|--|--|--|

+|2.4|July 8, 2019|End of Life Announced (EOLA)| Feb10,2023| Aug 10,2023|Feb 10,2024|

+|3.1|March 11,2022|GA |-|-|-|

+|3.3|To be announced for Public Preview|-|-|-|-|

+### Apache Spark 2.4 to Spark 3.x Migration Guides

+Spark 2.4 to Spark 3.x Migration Guides see [here](https://spark.apache.org/docs/latest/migration-guide.html).

+## HDInsight version 5.0

+Starting from June 1, 2022, we have started rolling out a new version of HDInsight 5.0, this version is backward compatible with HDInsight 4.0. All new open-source releases will be added as incremental releases on HDInsight 5.0.

+### Spark

+> [!NOTE]

+> * If you are using Azure User Interface to create a Spark Cluster for HDInsight, you will see from the dropdown list an additional version Spark 3.1.(HDI 5.0) along with the older versions. This version is a renamed version of Spark 3.1.(HDI 4.0) and it is backward compatible.

+> * This is only a UI level change, which doesnΓÇÖt impact anything for the existing users and users who are already using the ARM template to build their clusters.

+> * For backward compatibility, ARM supports creating Spark 3.1 with HDI 4.0 and 5.0 versions which maps to same versions Spark 3.1 (HDI 5.0)

+> * Spark 3.1 (HDI 5.0) cluster comes with HWC 2.0 which works well together with Interactive Query (HDI 5.0) cluster.

+### Interactive Query

+> [!NOTE]

+> * If you are creating an Interactive Query Cluster, you will see from the dropdown list another version as Interactive Query 3.1 (HDI 5.0).

+> * If you are going to use Spark 3.1 version along with Hive which require ACID support via Hive Warehouse Connector (HWC). You need to select this version Interactive Query 3.1 (HDI 5.0).

+### Kafka

+Current ARM template supports HDI 5.0 for Kafka 2.4.1

+`HDI Version '5.0' is supported for clusterType "Kafka" and component Version '2.4'.`

+We have fixed the arm templated issue.

+### Upcoming version upgrades.

+HDInsight team is working on upgrading other open-source components.

+1. Spark 3.2.0

+1. Kafka 3.2.1

+1. HBase 2.4.11

+## Next steps

+- [Cluster setup for Apache Hadoop, Spark, and more on HDInsight](hdinsight-hadoop-provision-linux-clusters.md)

+- [Enterprise Security Package](./enterprise-security-package.md)

+- [Work in Apache Hadoop on HDInsight from a Windows PC](hdinsight-hadoop-windows-tools.md)

+| HDInsight 5.1 |Ubuntu 18.0.4 LTS |Feb 27, 2022 | [Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions) | Not announced |Not announced| Yes |

+| HDInsight 5.0 |Ubuntu 18.0.4 LTS |July 01, 2022 | [Standard](hdinsight-component-versioning.md#support-options-for-hdinsight-versions) | Not announced |Not announced| Yes |

+* [HDInsight 5.x component versions](./hdinsight-5x-component-versioning.md)

+* [HDInsight 4.x component versions](./hdinsight-40-component-versioning.md)

+## Create new search parameter

+To create a new search parameter, you need to `POST` a `SearchParameter` resource to the FHIR service database.

+```

+The examples below demonstrate creating new custom search parameter

+### Create new search parameter per definition in Implementation Guide

+The code example below shows how to add the [US Core Race search parameter](http://hl7.org/fhir/us/core/STU3.1.1/SearchParameter-us-core-race.html) to the `Patient` resource type in your FHIR service database.

+```rest

+### Create new search parameter for resource attributes with reference type

+The code example shows how to create a custom search parameter to search MedicationDispense resources based on the location where they were dispensed. This is an example of adding custom search parameter for a Reference type.

+```rest

+{

+ "resourceType": "SearchParameter",

+ "id": "a3c28d46-fd06-49ca-aea7-5f9314ef0497",

+ "url": "{{An absolute URI that is used to identify this search parameter}}",

+ "version": "1.0",

+ "name": "MedicationDispenseLocationSearchParameter",

+ "status": "active",

+ "description": "Search parameter for MedicationDispense by location",

+ "code": "location",

+ "base": ["MedicationDispense"],

+ "target": ["Location"],

+ "type": "reference",

+ "expression": "MedicationDispense.location"

+}

+```

+* `code`: The value stored in the **code** element is the name used for the search parameter when it's included in an API call. For the example above with the "US Core Race" extension, you would search with `GET {{FHIR_URL}}/Patient?race=<code>` where `<code>` is in the value set from the specified coding system. This call would retrieve all patients of a certain race.

+* `target`: Describes which resource type(s) the search parameter matches to.

+First, you can test a new search parameter to see what values will be returned. By running the command below against a specific resource instance (by supplying the resource ID), you get back a list of value pairs with the search parameter name and the value stored in the corresponding element. This list includes all of the search parameters for the resource. You can scroll through to find the search parameter you created. Running this command won't change any behavior in your FHIR service.

+GET https://{{FHIR_URL}}/{{RESOURCE}}/{{RESOURCE_ID}}/$reindex

+The result looks like this:

+Continuing with our example, you could index one patient to enable `SearchParameter`:

+And then do a test search

+1. For the patient by race:

+1. For Location (reference type):

+```rest

+{{fhirurl}}/MedicationDispense?location=<locationid referenced in MedicationDispense Resource>

+x-ms-use-partial-indices: true

+```

+After you've tested your new search parameter and confirmed that it's working as expected, run or schedule your reindex job so the new search parameter(s) can be used in live production.

+See [Running a reindex job](../fhir/how-to-run-a-reindex.md) for information on how to reindex your FHIR service database.

+In this article, youΓÇÖve learned how to create a custom search parameter. Next you can learn how to reindex your FHIR service database.

+For more information, see

+ Title: Access control and security for DPS with Azure AD

+description: Control access to Azure IoT Hub Device Provisioning Service (DPS) (DPS) for back-end apps. Includes information about Azure Active Directory and RBAC.

+ Title: Access control and security for Azure DPS

+description: Overview on controlling access to Azure IoT Hub Device Provisioning Service, links to articles on Azure Active Directory integration and SAS options.

+ Title: Using custom allocation policies with Azure DPS

+description: Understand how custom allocation policies enable provisioning to multiple IoT hubs with the Azure IoT Hub Device Provisioning Service (DPS)

+ Title: Best practices for large-scale IoT deployments

+description: Best practices, patterns, and sample code you can use to help with large-scale deployments of Azure IoT Hub and Device Provisioning Service.

+ Title: Device lifecycle and reprovisioning concepts

+description: Describes device reprovisioning concepts and policies for the Azure IoT Hub Device Provisioning Service (DPS)

+ Title: Roles and operations for Azure DPS

+description: Conceptual overview of the roles and operations involved when developing and IoT solution using the IoT Device Provisioning Service (DPS).

+ Title: Terminology and glossary for Azure DPS

+description: This article describes common terminology used with the Device Provisioning Service (DPS) and IoT Hub

+ Title: Symmetric key attestation with Azure DPS

+ Title: TPM Attestation with Azure DPS

+ Title: X.509 certificate attestation with Azure DPS

+ Title: Access control and security for DPS with security tokens

+description: Control access to Azure IoT Hub Device Provisioning Service (DPS) for backend apps by using shared access signatures and security tokens.

+ Title: Tutorial - Provision devices using a symmetric key enrollment group in DPS

+ Title: Tutorial - Provision devices for geo latency in DPS

+ Title: Reprovision devices with DPS

+ Title: Disenroll or revoke device from DPS

+ Title: Roll X.509 certificates in DPS

+description: How to update or replace X.509 certificates with your Azure IoT Hub Device Provisioning Service (DPS) instance

+ Title: How to transfer a payload between devices and DPS

+ Title: Diagnose and troubleshoot provisioning errors with DPS

+ Title: Deprovision devices that were provisioned with DPS

+ Title: How to use allocation policies with DPS

+ Title: Verify X.509 CA certificates with DPS

+ Title: High availability and disaster recovery with DPS

+ Title: Understand DPS MQTT support

+ Title: Understanding the IP address of your DPS instance

+description: Query your DPS IP address and its properties. The IP address of your DPS instance can change during scenarios like disaster recovery or regional failover.

+ Title: Monitoring DPS data reference

+description: Important reference material needed when you monitor Azure IoT Hub Device Provisioning Service using Azure Monitor

+ Title: Monitor DPS using Azure Monitor

+description: Start here to learn how to monitor metrics and logs from the Azure IoT Hub Device Provisioning Service by using Azure Monitor

+ Title: Manage public network access for DPS

+ Title: TLS support with DPS

+description: Best practices in using secure TLS connections for devices and services communicating with the IoT Device Provisioning Service (DPS)

+This article describes how to configure the Azure IoT Edge for Linux (EFLOW) virtual machine (VM) to support multiple network interface cards (NICs) and connect to multiple networks. By enabling multiple NIC support, applications running on the EFLOW VM can communicate with devices connected to the offline network, while using IoT Edge to send data to the cloud.

+Industrial IoT is overtaking the era of information technology (IT) and operational technology (OT) convergence. However, making traditional OT assets more intelligent with IT technologies also means a larger exposure to cyber attacks. This scenario is one of the main reasons why multiple environments are designed using demilitarized zones, also known as, DMZs.

+In this scenario, you have an environment with some devices like programmable logic controllers (PLCs) or open platform communications unified architecture (OPC UA)-compatible devices connected to the offline network, and you want to upload all the devices' information to Azure using the OPC Publisher module running on the EFLOW VM.

+- No internet connectivity - access restricted.

+- PLCs or UPC UA-compatible devices connected.

+> The steps in this article assume that the EFLOW VM was deployed with an *external virtual switch* connected to the *secure network (offline)*. You can change the following steps to the specific network configuration you want to achieve. For more information about EFLOW multiple NICs support, see [Azure IoT Edge for Linux on Windows virtual multiple NIC configurations](./how-to-configure-multiple-nics.md).

+For this scenario, you assign an *external virtual switch* connected to the DMZ network. For more information, review [Create a virtual switch for Hyper-V virtual machines](/windows-server/virtualization/hyper-v/get-started/create-a-virtual-switch-for-hyper-v-virtual-machines).

+Once the external virtual switch is created, you need to attach it to the EFLOW VM using the following steps. If you need to attach multiple NICs, see [EFLOW Multiple NICs](https://github.com/Azure/iotedge-eflow/wiki/Multiple-NICs).

+For the custom new *external virtual switch* you created, use the following PowerShell commands to:

+1. Attach the switch to your EFLOW VM.

+ ```powershell

+ Add-EflowNetwork -vswitchName "OnlineOPCUA" -vswitchType "External"

+ ```

+ :::image type="content" source="./media/how-to-configure-iot-edge-for-linux-on-windows-iiot-dmz/add-eflow-network.png" alt-text="Screenshot of a successful creation of the external network named OnlineOPCUA." lightbox="./media/how-to-configure-iot-edge-for-linux-on-windows-iiot-dmz/add-eflow-network.png":::

+1. Set a static IP.

+ ```powershell

+ Add-EflowVmEndpoint -vswitchName "OnlineOPCUA" -vEndpointName "OnlineEndpoint" -ip4Address 192.168.0.103 -ip4PrefixLength 24 -ip4GatewayAddress 192.168.0.1

+ ```

+ :::image type="content" source="./media/how-to-configure-iot-edge-for-linux-on-windows-iiot-dmz/add-eflow-vm-endpoint.png" alt-text="Screenshot of a successful configuration of the OnlineOPCUA switch.." lightbox="./media/how-to-configure-iot-edge-for-linux-on-windows-iiot-dmz/add-eflow-vm-endpoint.png":::

+Once complete, you have the *OnlineOPCUA* switch assigned to the EFLOW VM. To check the multiple NIC attachment, use the following steps:

+1. Once you're in your VM, list all the network interfaces assigned to the EFLOW virtual machine.

+1. Once you're in your VM, list all the network routes configured in the EFLOW virtual machine.

+The final step is to create a Linux service that runs on startup, and executes the bash script to set the routes. You have to create a *systemd* unit file to load the service. The following is an example of that file.

+ Title: Control access to IoT Hub using SAS tokens

+ Title: Azure IoT Hub cloud-to-device options

+ Title: Azure IoT Hub device-to-cloud options

+ Title: Understand Azure IoT Hub device twins

+ Title: Understand Azure IoT Hub direct methods

+ Title: Understand Azure IoT Hub endpoints

+ Title: Understand Azure IoT Hub file upload

+ Title: Understand Azure IoT Hub jobs

+ Title: Understand Azure IoT Hub cloud-to-device messaging

+ Title: Understand Azure IoT Hub message format

+ Title: Understand the Azure IoT Hub built-in endpoint

+ Title: Understand Azure IoT Hub messaging

+ Title: Understand Azure IoT Hub module twins

+ Title: Develop without an Azure IoT SDK

+ Title: Azure IoT Hub communication protocols and ports

+ Title: Azure IoT Hub device and service SDKs

+ Title: Access control and security for IoT Hub

+ Title: Concepts overview for Azure IoT Hub

+description: The Azure IoT Hub conceptual documentation includes discussions of endpoints, security, the identity registry, device management, direct methods, device twins, file uploads, jobs, the IoT Hub query language, messaging and many other features.

+You can migrate your application from the Baltimore CyberTrust Root to the DigiCert Global G2 Root on your own schedule. We recommend the following process: 

+1. **Keep the Baltimore CyberTrust Root on your device until the transition period is completed on 15 February 2024** (necessary to prevent connection interruption).

+2. **In addition** to the Baltimore Root, ensure the DigiCert Global G2 Root is added to your trusted root store.

+3. Make sure you arenΓÇÖt pinning any intermediate or leaf certificates and are using the public roots to perform TLS server validation.

+4. In your IoT Central application you can find the Root Certification settings underΓÇ»**Settings**ΓÇ»>ΓÇ»**Application**ΓÇ»>ΓÇ»**Baltimore Cybertrust Migration**.ΓÇ»

+ 1. Select **DigiCert Global G2 Root** to migrate to the new certificate root.

+ 2. Click **Save** to initiate the migration.

+ 3. If needed, you can migrate back to the Baltimore root by selecting **Baltimore CyberTrust Root** and saving the changes. This option is available until 15 May 2023 and will then be disabled as Microsoft will start initiating the migration.

+ Title: Azure IoT Hub support for virtual networks

+description: How to use virtual networks connectivity pattern with IoT Hub

+ Title: Create example Standard logic app workflow in the Azure portal

+description: Create your first example Standard logic app workflow that runs in single-tenant Azure Logic Apps using the Azure portal.

+# Customer intent: As a developer, I want to create my first example Standard logic app workflow that runs in single-tenant Azure Logic Apps using the Azure portal.

+# Create an example Standard workflow in single-tenant Azure Logic Apps with the Azure portal

+This guide shows how to create an example automated workflow that waits for an inbound web request and then sends a message to an email account. More specifically, you'll create a [Standard logic app resource](logic-apps-overview.md#resource-environment-differences), which can include multiple [stateful and stateless workflows](single-tenant-overview-compare.md#stateful-stateless) that run in single-tenant Azure Logic Apps.

+> [!NOTE]

+> [Create Standard workflows in single-tenant Azure Logic Apps with Visual Studio Code](create-single-tenant-workflows-visual-studio-code.md).

+While this example workflow is cloud-based and has only two steps, you can create workflows from hundreds of operations that can connect a wide range of apps, data, services, and systems across cloud, on premises, and hybrid environments. The example workflow starts with the Request built-in trigger, which is followed by an Office 365 Outlook action. The trigger creates a callable endpoint for the workflow and waits for an inbound HTTPS request from any caller. When the trigger receives a request and fires, the next action runs by sending email to the specified email address along with selected outputs from the trigger.

+

+* Create a Standard logic app resource and add a blank [*stateful* workflow](single-tenant-overview-compare.md#stateful-stateless).

+In single-tenant Azure Logic Apps, workflows in the same logic app resource and tenant run in the same process as the runtime, so they share the same resources and provide better performance. For more information about single-tenant Azure Logic Apps, see [Single-tenant versus multi-tenant and integration service environment](single-tenant-overview-compare.md).

+ >

+ > The Standard logic app resource type is powered by Azure Functions and has [storage requirements similar to function apps](../azure-functions/storage-considerations.md).

+* To create the same example workflow in this guide, you need an Office 365 Outlook email account that uses a Microsoft work or school account to sign in.

+ If you don't have an Office 365 account, you can use [any other available email connector](/connectors/connector-reference/connector-reference-logicapps-connectors) that can send messages from your email account, for example, Outlook.com. If you use a different email connector, you can still follow the example, and the general overall steps are the same. However, your options might differ in some ways. For example, if you use the Outlook.com connector, use your personal Microsoft account instead to sign in.

+* To test the example workflow in this guide, you need a tool that can send calls to the endpoint created by the Request trigger. If you don't have such a tool, you can download, install, and use [Postman](https://www.postman.com/downloads/).

+* If you create your logic app resource and enable [Application Insights](../azure-monitor/app/app-insights-overview.md), you can optionally enable diagnostics logging and tracing for your logic app. You can do so either when you create your logic app or after deployment. You need to have an Application Insights instance, but you can create this resource either [in advance](../azure-monitor/app/create-workspace-resource.md), when you create your logic app, or after deployment.

+* To deploy your Standard logic app resource to an [App Service Environment v3 (ASEv3) - Windows plan only](../app-service/environment/overview.md), you have to create this environment resource first. You can then select this environment as the deployment location when you create your logic app resource. For more information, review [Resources types and environments](single-tenant-overview-compare.md#resource-environment-differences) and [Create an App Service Environment](../app-service/environment/creation.md).

+1. In the [Azure portal](https://portal.azure.com), sign in with your Azure account.

+1. In the Azure portal search box, enter **logic apps**, and select **Logic apps**.

+ | **Logic App name** | Yes | <*logic-app-name*> | Your logic app resource name, which must be unique across regions and can contain only letters, numbers, hyphens (**-**), underscores (**_**), parentheses (**()**), and periods (**.**). <br><br>**Note**: Your logic app's name automatically gets the suffix, **.azurewebsites.net**, because the Standard logic app resource is powered by the single-tenant Azure Logic Apps runtime, which uses the Azure Functions extensibility model and is hosted as an extension on the Azure Functions runtime. Azure Functions uses the same app naming convention. <br><br>This example creates a logic app named **Fabrikam-Workflows**. |

+ | **Windows Plan** | Yes | <*plan-name*> | The plan name to use. Either select an existing plan name or provide a name for a new plan. <br><br>This example uses the name **Fabrikam-Service-Plan**. <br><br>**Note**: Only the Windows-based App Service plan is supported. Don't use a Linux-based App Service plan. |

+ | **SKU and size** | Yes | <*pricing-tier*> | The [pricing tier](../app-service/overview-hosting-plans.md) to use for your logic app and workflows. Your selection affects the pricing, compute, memory, and storage that your logic app and workflows use. <br><br>To change the default pricing tier, select **Change size**. You can then select other pricing tiers, based on the workload that you need. <br><br>For more information, review [Hosting plans and pricing tiers](logic-apps-pricing.md#standard-pricing). |

+ | **Publish** | Yes | **Workflow** | This option appears and applies only when **Plan type** is set to the **Standard** logic app type. By default, this option is set to **Workflow** and creates an empty logic app resource where you add your first workflow. <br><br>**Note**: Currently, the **Docker Container** option requires a [*custom location*](../azure-arc/kubernetes/conceptual-custom-locations.md) on an Azure Arc enabled Kubernetes cluster, which you can use with [Azure Arc enabled Logic Apps (Standard)](azure-arc-enabled-logic-apps-overview.md). The resource locations for your logic app, custom location, and cluster must all be the same. |

+ | **Storage type** | Yes | - **Azure Storage** <br>- **SQL and Azure Storage** | The storage type that you want to use for workflow-related artifacts and data. <br><br>- To deploy only to Azure, select **Azure Storage**. <br><br>- To use SQL as primary storage and Azure Storage as secondary storage, select **SQL and Azure Storage**, and review [Set up SQL database storage for Standard logic apps in single-tenant Azure Logic Apps](set-up-sql-db-storage-single-tenant-standard-workflows.md). <br><br>**Note**: If you're deploying to an Azure region, you still need an Azure storage account, which is used to complete the one-time hosting of the logic app's configuration on the Azure Logic Apps platform. The ongoing workflow state, run history, and other runtime artifacts are stored in your SQL database. <br><br>For deployments to a custom location that's hosted on an Azure Arc cluster, you only need SQL as your storage provider. |

+ | **Storage account** | Yes | <*Azure-storage-account-name*> | The [Azure Storage account](../storage/common/storage-account-overview.md) to use for storage transactions. <br><br>This resource name must be unique across regions and have 3-24 characters with only numbers and lowercase letters. Either select an existing account or create a new account. <br><br>This example creates a storage account named **fabrikamstorageacct**. |

+1. On the **Networking** tab, you can leave the default options for this example.

+ For your specific, real-world scenarios, make sure to review and select the appropriate options. You can also change this configuration after you deploy your logic app. For more information, see [Secure traffic between Standard logic apps and Azure virtual networks using private endpoints](secure-single-tenant-workflow-virtual-network-private-endpoint.md).

+ | Enable public access | Behavior |

+ |-|-|

+ | **On** | Your logic app has a public endpoint with an inbound address that's open to the internet and can't access an Azure virtual network. |

+ | **Off** | Your logic app has no public endpoint, but has a private endpoint instead for communication within an Azure virtual network, and is isolated to that virtual network. The private endpoint can communicate with endpoints in the virtual network, but only from clients within that network. This configuration also means that logic app traffic can be governed by network security groups or affected by virtual network routes. |

+ To enable your logic app to access endpoints in a virtual network, make sure to select the appropriate option:

+ | Enable network injection | Behavior |

+ |--|-|

+ | **On** | Your logic app workflows can privately and securely communicate with endpoints in the virtual network. |

+ | **Off** | Your logic app workflows can't communicate with endpoints in the virtual network. |

+1. If your creation and deployment settings support using [Application Insights](../azure-monitor/app/app-insights-overview.md), you can optionally enable diagnostics logging and tracing for your logic app workflows.

+ This example adds a blank stateful workflow named **Fabrikam-Stateful-Workflow**. By default, the workflow is enabled but doesn't do anything until you add a trigger and actions.

+1. In the **Choose an operation** search box, enter **when a http request**, and select the built-in Request trigger that's named **When an HTTP request is received**.

+ When the trigger appears on the designer, the trigger's information pane opens to show the trigger's properties, settings, and other actions.

+ 

+ > If the information pane doesn't appear, makes sure that the trigger is selected on the designer.

+ >

+ > If the **Add an action** pane shows the error message, **"Cannot read property 'filter' of undefined"**,

+ > If the **Add an action** pane shows the error message,

+ > **"The access token expiry UTC time {*token-expiration-date-time*} is earlier than current UTC time {*current-date-time*}"**,

+ 

+1. In the action's information pane, on the **Create Connection** tab, select **Sign in** so that you can create a connection to your email account.

+ 

+ > If you get the error message, **"Failed with error: 'The browser is closed.'. Please sign in again"**,

+ > try adding **https://portal.azure.com** to the list of sites that can use cookies.

+ After Azure creates the connection, the **Send an email** action appears on the designer and is selected by default. If the action isn't selected, select the action so that its information pane is also open.

+1. In the action information pane, on the **Parameters** tab, provide the required information for the action, for example:

+ 

+ | **To** | Yes | <*your-email-address*> | The email recipient, which can be your email address for test purposes. This example uses the fictitious email, **sophiaowen@fabrikam.com**. |

+ | **Subject** | Yes | **An email from your example workflow** | The email subject |

+ | **Body** | Yes | **Hello from your example workflow!** | The email body content |

+ > When making any changes in the information pane on the **Settings**, **Static Result**, or **Run After** tabs,

+1. Copy and save the **connectionRuntimeUrl** property value somewhere safe so that you can set up your firewall with this information.

+ 

+1. After the information pane opens, on the **Parameters** tab, find the **HTTP POST URL** property. To copy the generated URL, select the **Copy Url** (copy file icon), and save the URL somewhere else for now. The URL follows this format:

+ **`https://<*logic-app-name*>.azurewebsites.net:443/api/<*workflow-name*>/triggers/manual/invoke?api-version=2020-05-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=<*shared-access-signature*>`**

+ **`https://fabrikam-workflows.azurewebsites.net:443/api/Fabrikam-Stateful-Workflow/triggers/manual/invoke?api-version=2020-05-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=xxxxxXXXXxxxxxXXXXxxxXXXXxxxxXXXX`**

+ 1. In the **Save Request** window, under **Request name**, provide a name for the request, for example, **Test workflow trigger**.

+ 1. Under **All Collections**, provide a name for the collection to create for organizing your requests, press Enter, and select **Save to <*collection-name*>**. This example uses **Logic Apps requests** as the collection name.

+ | **Running** | The run was triggered and is in progress, but this status can also appear for a run that is throttled due to [action limits](logic-apps-limits-and-config.md) or the [current pricing plan](https://azure.microsoft.com/pricing/details/logic-apps/). <br><br>**Tip**: If you set up [diagnostics logging](monitor-workflows-collect-diagnostic-data.md), you can get information about any throttle events that happen. |

+ | **Timed out** | The run timed out because the current duration exceeded the run duration limit, which is controlled by the [**Run history retention in days** setting](logic-apps-limits-and-config.md#run-duration-retention-limits). A run's duration is calculated by using the run's start time and run duration limit at that start time. <br><br>**Note**: If the run's duration also exceeds the current *run history retention limit*, which is also controlled by the [**Run history retention in days** setting](logic-apps-limits-and-config.md#run-duration-retention-limits), the run is cleared from the runs history by a daily cleanup job. Whether the run times out or completes, the retention period is always calculated by using the run's start time and *current* retention limit. So, if you reduce the duration limit for an in-flight run, the run times out. However, the run either stays or is cleared from the runs history based on whether the run's duration exceeded the retention limit. |

+ | **Skipped** | The action was skipped because its **runAfter** conditions weren't met, for example, a preceding action failed. Each action has a `runAfter` object where you can set up conditions that must be met before the current action can run. |

+## Best practices and recommendations

+For optimal designer responsiveness and performance, review and follow these guidelines:

+- Use no more than 50 actions per workflow. Exceeding this number of actions raises the possibility for slower designer performance.

+- Consider splitting business logic into multiple workflows where necessary.

+- Have no more than 10-15 workflows per logic app resource.

+1. In the [Azure portal](https://portal.azure.com), open your Standard logic app resource.

+ **Workflows.{*yourWorkflowName*}.OperationOptions**

+1. In the **Value** box, enter the following value: **WithStatelessRunHistory**

+ 

+1. To disable the run history when you're done, either set the property named **Workflows.{*your-workflow-name*}.OperationOptions** to **None**, or delete the property and its value.

+* Select the item so that information pane opens for that item. In the pane's upper right corner, open the ellipses (**...**) menu, and select **Delete**. To confirm, select **OK**.

+ 

+ > the information pane shows the ellipses (**...**) button in the upper right corner.

+1. In the Azure portal's main search box, enter **logic apps**, and select **Logic apps**.

+1. In the Azure portal's main search box, enter **logic apps**, and select **Logic apps**.

+1. When the confirmation box appears, enter **yes**, and select **Delete**.

+* If you delete a workflow and then recreate the same workflow, the recreated workflow won't have the same metadata as the deleted workflow. To refresh the metadata, you have to resave any workflow that called the deleted workflow. That way, the caller gets the correct information for the recreated workflow. Otherwise, calls to the recreated workflow fail with an **Unauthorized** error. This behavior also applies to workflows that use artifacts in integration accounts and workflows that call Azure functions.

+If you use source control, you can seamlessly redeploy a deleted Standard logic app resource to single-tenant Azure Logic Apps. However, if you're not using source control, try the following steps to recover your deleted logic app.

+>

+> * You can recover only deleted Standard logic app resources that use the **Workflow Standard**

+> hosting plan. You can't recover deleted Consumption logic app resources.

+ **DefaultEndpointsProtocol=https;AccountName=<*storage-account-name*>;AccountKey=<*access-key*>;EndpointSuffix=core.windows.net**

+1. Create a new Standard logic app resource using the same hosting plan and pricing tier. You can either use a new name or reuse the name from the deleted logic app.

+ | **AzureWebJobsStorage** | Replace the existing value with the previously copied connection string from your storage account. |

+ | **WEBSITE_CONTENTAZUREFILECONNECTIONSTRING** | Replace the existing value with the previously copied string from your storage account. |

+ | **WEBSITE_CONTENTSHARE** | Replace the existing value with the previously copied file share name. |

+Single-tenant Azure Logic Apps supports built-in actions for Azure Function Operations, Liquid Operations, and XML Operations, such as **XML Validation** and **Transform XML**. However, for previously created logic apps, these actions might not appear in the designer for you to select if your logic app uses an outdated version of the extension bundle, **Microsoft.Azure.Functions.ExtensionBundle.Workflows**.

+>

+> This specific solution applies only to Standard logic app resources that you create using

+ **...\home\data\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle.Workflows**

+1. Delete the version folder for the existing bundle. In the console window, you can run this command where you replace **{*bundle-version*}** with the existing version:

+ >

+ > If you get an error such as **"permission denied"** or **"file in use"**, refresh the

+ > page in your browser, and try the previous steps again until the folder is deleted.

+* For questions, requests, comments, and other feedback, [use this feedback form](https://aka.ms/lafeedback).

+* You can also **[use a setup script](how-to-customize-compute-instance.md)** for an automated way to customize and configure the compute instance as per your needs.

+You can also [use a setup script](how-to-customize-compute-instance.md) to create the compute instance with your own custom environment.

+ Title: Customize compute instance with a script

+* A registered model.

+### Unexpected Dockerfile Format

+<!--issueDescription-->

+This issue can happen when your Dockerfile is formatted incorrectly.

+**Potential causes:**

+* Your Dockerfile contains invalid syntax

+* Your Dockerfile contains characters that aren't compatible with UTF-8

+**Affected areas (symptoms):**

+* Failure in building environments from UI, SDK, and CLI.

+* Failure in running jobs because it will implicitly build the environment in the first step.

+<!--/issueDescription-->

+**Troubleshooting steps**

+* Ensure Dockerfile is formatted correctly and is encoded in UTF-8

+**Resources**

+* [Dockerfile format](https://docs.docker.com/engine/reference/builder/#format)

+<!--/issueDescription-->

+## *Copy issues*

+### File not found

+<!--issueDescription-->

+This issue can happen when Docker fails to find and copy a file.

+**Potential causes:**

+* Source file not found in Docker build context

+* Source file excluded by `.dockerignore`

+**Affected areas (symptoms):**

+* Failure in building environments from UI, SDK, and CLI.

+* Failure in running jobs because it will implicitly build the environment in the first step.

+**Troubleshooting steps**

+* Ensure that the source file exists in the Docker build context

+* Ensure that the source and destination paths exist and are spelled correctly

+* Ensure that the source file isn't listed in the `.dockerignore` of the current and parent directories

+* Remove any trailing comments from the same line as the `COPY` command

+**Resources**

+* [Docker COPY](https://docs.docker.com/engine/reference/builder/#copy)

+* [Docker Build Context](https://docs.docker.com/engine/context/working-with-contexts/)

+## *Unknown Docker command*

+### Unknown Docker instruction

+<!--issueDescription-->

+This issue can happen when Docker doesn't recognize an instruction in the Dockerfile.

+**Potential causes:**

+* Unknown Docker instruction being used in Dockerfile

+* Your Dockerfile contains invalid syntax

+**Affected areas (symptoms):**

+* Failure in building environments from UI, SDK, and CLI.

+* Failure in running jobs because it will implicitly build the environment in the first step.

+<!--/issueDescription-->

+**Troubleshooting steps**

+* Ensure that the Docker command is valid and spelled correctly

+* Ensure there's a space between the Docker command and arguments

+* Ensure there's no unnecessary whitespace in the Dockerfile

+* Ensure Dockerfile is formatted correctly and is encoded in UTF-8

+**Resources**

+* [Dockerfile reference](https://docs.docker.com/engine/reference/builder/)

+ To link two servers and start replication, login to the target replica server in the Azure Database for MySQL service and set the external instance as the source server. This is done by using the `mysql.az_replication_change_master` stored procedure on the Azure Database for MySQL server.

+Network Watcher packet capture allows you to create packet capture sessions to track traffic to and from a virtual machine. Packet capture helps to diagnose network anomalies both reactively and proactively. Other uses include gathering network statistics, gaining information on network intrusions, to debug client-server communications and much more.

+Packet capture is an extension that is remotely started through Network Watcher. This capability eases the burden of running a packet capture manually on the desired virtual machine or virtual machine scale set instance(s), which saves valuable time. Packet capture can be triggered through the portal, PowerShell, CLI, or REST API. One example of how packet capture can be triggered is with Virtual Machine alerts. Filters are provided for the capture session to ensure you capture traffic you want to monitor. Filters are based on 5-tuple (protocol, local IP address, remote IP address, local port, and remote port) information. The captured data is stored in the local disk or a storage blob.

+|**Maximum bytes per packet (bytes)** | The number of bytes from each packet that are captured, all bytes are captured if left blank. If you need only the IPv4 header ΓÇô indicate 34 here |

+|**Maximum bytes per session (bytes)** | Total number of bytes that are captured, once the value is reached the session ends.|

+After you've registered the feature flag, create the cluster [using the command above](#create-the-cluster).

+Backups on flexible servers are snapshot based. The first snapshot backup is scheduled immediately after a server is created. Snapshot backups are currently taken once daily. **The first snapshot is a full backup and consecutive snapshots are differential backups.**

+- [Single Server](../overview-single-server.md)

+

+| China East 3 | :heavy_check_mark: | :x: | :heavy_check_mark: | :heavy_check_mark: |

+| China North 3 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |

+After creating the policy, any of the Azure AD users in the Subject should now be able to connect to the data sources in the scope of the policy. To test, use SSMS or any SQL client and try to query. Attempt access to a SQL table you have provided read access to.

+If you require additional troubleshooting, see the [Next steps](#next-steps) section in this guide.

+## Role definition detail

+This section contains a reference of how relevant Microsoft Purview data policy roles map to specific actions in SQL data sources.

+| **Microsoft Purview policy role definition** | **Data source specific actions** |

+* [Enable Microsoft Purview data owner policies on an Azure SQL Database](./how-to-policies-data-owner-azure-sql-db.md)

+* Doc: [Troubleshoot Microsoft Purview policies for SQL data sources](./troubleshoot-policy-sql.md)

+After creating the policy, any of the Azure AD users in the Subject should now be able to connect to the data sources in the scope of the policy. To test, use SSMS or any SQL client and try to query. Attempt access to a SQL table you have provided read access to.

+If you require additional troubleshooting, see the [Next steps](#next-steps) section in this guide.

+## Role definition detail

+This section contains a reference of how relevant Microsoft Purview data policy roles map to specific actions in SQL data sources.

+| **Microsoft Purview policy role definition** | **Data source specific actions** |

+* Doc: [Troubleshoot Microsoft Purview policies for SQL data sources](./troubleshoot-policy-sql.md)

+If you require additional troubleshooting, see the [Next steps](#next-steps) section in this guide.

+This section contains a reference of how relevant Microsoft Purview data policy roles map to specific actions in SQL data sources.

+| **Microsoft Purview policy role definition** | **Data source specific actions** |

+* [Provision read/modify access on a single storage account](./how-to-policies-data-owner-storage.md#create-and-publish-a-data-owner-policy)

+* [Provision access to system health, performance and audit information in SQL Server 2022](./how-to-policies-devops-arc-sql-server.md#create-a-new-devops-policy)

+* [Provision read/modify access on a single SQL Server 2022](./how-to-policies-data-owner-arc-sql-server.md#create-and-publish-a-data-owner-policy)

+To create an access policy for Azure Blob Storage, follow this guide: [Provision read/modify access on a single storage account](./how-to-policies-data-owner-storage.md#create-and-publish-a-data-owner-policy).

+* [Provision read/modify access to all sources in a subscription or resource group](./how-to-policies-data-owner-resource-group.md#create-and-publish-a-data-owner-policy)

+* [Provision access to system health, performance and audit information in Azure SQL Database](./how-to-policies-devops-azure-sql-db.md#create-a-new-devops-policy). Use this guide to apply a DevOps policy on a single SQL database.

+* [Provision read/modify access on a single Azure SQL Database](./how-to-policies-data-owner-azure-sql-db.md#create-and-publish-a-data-owner-policy). Use this guide to provision access on a single SQL database account in your subscription.

+* [Self-service access policies for Azure SQL Database](./how-to-policies-self-service-azure-sql-db.md). Use this guide to allow data consumers to request access to data assets by using a self-service workflow.

+To create policies that cover all data sources inside a resource group or Azure subscription, see [Discover and govern multiple Azure sources in Microsoft Purview](register-scan-azure-multiple-sources.md#access-policy).

+Microsoft Purview supports basic authentication (username and password) for scanning Oracle. The Oracle user must have read access to system tables in order to access advanced metadata.

+For classification, user also needs to be the owner of the table.

+>[!IMPORTANT]

+>If the user is not the owner of the table, the scan will run successfully and ingest metadata, but will not identify any classifications.

+ var modelOptions = LoadModelOptions.CreateForBlobStorage(

+All spatial queries are evaluated on the server. Accordingly, the queries are asynchronous operations and results arrive with a delay that depends on your network latency.

+A *ray cast* is a spatial query where the runtime checks which objects intersect a ray, starting at a given position and pointing into a certain direction. As an optimization, a maximum ray distance is also given, to not search for objects that are too far away.

+* **`Closest`:** In this mode, only the closest hit is reported.

+* **`HitType`:** What is hit by the ray: `TriangleFrontFace`, `TriangleBackFace` or `Point`. By default, [ARR renders double sided](single-sided-rendering.md#prerequisites) so the triangles the user sees aren't necessarily front facing. If you want to differentiate between `TriangleFrontFace` and `TriangleBackFace` in your code, make sure your models are authored with correct face directions first.

+* [C# RenderingConnection.RayCastQueryAsync()](/dotnet/api/microsoft.azure.remoterendering.renderingconnection.raycastqueryasync)

+* [C# RenderingConnection.SpatialQueryAabbAsync()](/dotnet/api/microsoft.azure.remoterendering.renderingconnection.spatialqueryaabbasync)

+* [C# RenderingConnection.SpatialQuerySphereAsync()](/dotnet/api/microsoft.azure.remoterendering.renderingconnection.spatialquerysphereasync)

+* [C# RenderingConnection.SpatialQueryObbAsync()](/dotnet/api/microsoft.azure.remoterendering.renderingconnection.spatialqueryobbasync)

+* [C++ RenderingConnection::RayCastQueryAsync()](/cpp/api/remote-rendering/renderingconnection#raycastqueryasync)

+* [C++ RenderingConnection::SpatialQueryAabbAsync()](/cpp/api/remote-rendering/renderingconnection#spatialqueryaabbasync)

+* [C++ RenderingConnection::SpatialQuerySphereAsync()](/cpp/api/remote-rendering/renderingconnection#spatialquerysphereasync)

+* [C++ RenderingConnection::SpatialQueryObbAsync()](/cpp/api/remote-rendering/renderingconnection#spatialqueryobbasync)

+When using a linked blob storage, you use slightly different methods for loading models:

+var loadModelParams = LoadModelOptions.CreateForBlobStorage(storageAccountPath, blobName, modelPath, modelEntity);

+ var loadModelParams = LoadModelOptions.CreateForBlobStorage($"{storageAccountName}.blob.core.windows.net", blobName, modelPath, modelEntity);

+ The additional inputs `storageAccountName` and `blobName` have also been added to the arguments. We call this new **LoadModel** method from another method similar to the first **LoadTestModel** method we created in the first tutorial.

+ This code adds three extra string variables to your **RemoteRenderingCoordinator** component.

+ * **Storage Account Name**: Your storage account name, the globally unique name you choose for your storage account. In the quickstart this was *arrtutorialstorage*, your value is different.

+ * **Model Path**: The combination of the "outputFolderPath" and the "outputAssetFileName" defined in the *arrconfig.json* file. In the quickstart, this was "outputFolderPath":"converted/robot", "outputAssetFileName": "robot.arrAsset". Which would result in a Model Path value of "converted/robot/robot.arrAsset", your value is different.

+AAD authentication allows you to determine which individuals or groups are using ARR in a more controlled way. ARR has built in support for accepting [Access Tokens](../../../../active-directory/develop/access-tokens.md) instead of using an Account Key. You can think of Access Tokens as a time-limited, user-specific key, that only unlocks certain parts of the specific resource it was requested for.

+With the Azure side of things in place, we now need to modify how your code connects to the AAR service. We do that by implementing an instance of **BaseARRAuthentication**, which returns a new **SessionConfiguration** object. In this case, the account info is configured with the Azure Access Token.

+The code first tries to get the token silently using **AquireTokenSilent**. This is successful if the user has previously authenticated this application. If it's not successful, move on to a more user-involved strategy.

+Since the User Credentials aren't stored on the device (or in this case even entered on the device), their exposure risk is low. Now the device is using a user-specific, time-limited Access Token to access ARR, which uses access control (IAM) to access the Blob Storage. These two steps have removed the "passwords" from the source code and increased security considerably. However, this isn't the most security available, moving the model and session management to a web service will improve security further. Additional security considerations are discussed in the [Commercial Readiness](../commercial-ready/commercial-ready.md) chapter.

+In the Unity Editor, when AAD Auth is active, you'll need to authenticate every time you launch the application. On device, the authentication step happens the first time and only be required again when the token expires or is invalidated.

+ * **Azure Tenant ID** is the *Directory (tenant) ID* found in your AAD app registration (see image below).

+ Since the **AAD Authentication** component has a view controller, it's automatically hooked up to display a prompt after the session authorization modal panel.

+If you're building an application using MSAL to device, you need to include a file in your project's **Assets** folder. This helps the compiler build the application correctly using the *Microsoft.Identity.Client.dll* included in the **Tutorial Assets**.

+The remainder of this tutorial set contains conceptual articles for creating a production-ready application that uses Azure Remote Rendering.

+# Configure SAP Installation parameters

+The Ansible playbooks use a combination of default parameters and parameters defined by the Terraform deployment for the SAP installation.

+## Default Parameters

+This table contains the default parameters defined by the framework.

+### User IDs

+> [!div class="mx-tdCol2BreakAll "]

+> | Parameter | Description | Default Value | Type |

+> | - | -- | - | - |

+> | `sapadm_uid` | The UID for the sapadm account. | 2100 | Required |

+> | `sidadm_uid` | The UID for the sidadm account. | 2003 | Required |

+> | `hdbadm_uid` | The UID for the hdbadm account. | 2200 | Required |

+> | `sapinst_gid` | The GID for the sapinst group. | 2001 | Required |

+> | `sapsys_gid` | The GID for the sapsys group. | 2000 | Required |

+> | `hdbshm_gid` | The GID for the hdbshm group. | 2002 | Required |

+> | | | | |

+> | `db2sidadm_uid` | The UID for the db2sidadm account. | 3004 | Required |

+> | `db2sapsid_uid` | The UID for the db2sapsid account. | 3005 | Required |

+> | `db2sysadm_gid` | The UID for the db2sysadm group. | 3000 | Required |

+> | `db2sysctrl_gid` | The UID for the db2sysctrl group. | 3001 | Required |

+> | `db2sysmaint_gid` | The UID for the db2sysmaint group. | 3002 | Required |

+> | `db2sysmon_gid` | The UID for the db2sysmon group. | 2003 | Required |

+> | | | | |

+> | `orasid_uid` | The UID for the orasid account. | 3100 | Required |

+> | `oracle_uid` | The UID for the oracle account. | 3101 | Required |

+> | `observer_uid` | The UID for the observer account. | 4000 | Required |

+> | `dba_gid` | The GID for the dba group. | 3100 | Required |

+> | `oper_gid` | The GID for the oper group. | 3101 | Required |

+> | `asmoper_gid` | The GID for the asmoper group. | 3102 | Required |

+> | `asmadmin_gid` | The GID for the asmadmin group. | 3103 | Required |

+> | `asmdba_gid` | The GID for the asmdba group. | 3104 | Required |

+> | `oinstall_gid` | The GID for the oinstall group. | 3105 | Required |

+> | `backupdba_gid` | The GID for the backupdba group. | 3106 | Required |

+> | `dgdba_gid` | The GID for the dgdba group. | 3107 | Required |

+> | `kmdba_gid` | The GID for the kmdba group. | 3108 | Required |

+> | `racdba_gid` | The GID for the racdba group. | 3108 | Required |

+This table contains the parameters stored in the sap-parameters.yaml file, most of the values are prepopulated via the Terraform deployment.

+Disks define a dictionary with information about the disks of all the virtual machines in the SAP Application virtual machines.

+Example of the disks dictionary:

+From the v3.4 release, it's possible to deploy SAP on Azure systems in a Shared Home configuration using an Oracle database backend. For more information on running SAP on Oracle in Azure, see [Azure Virtual Machines Oracle DBMS deployment for SAP workload](../workloads/dbms-guide-oracle.md).

+The [SAP on Azure Deployment Automation Framework](https://github.com/Azure/sap-automation) is an open-source orchestration tool for deploying, installing and maintaining SAP environments. You can create infrastructure for SAP landscapes based on SAP HANA and NetWeaver with AnyDB using [Terraform](https://www.terraform.io/), and [Ansible](https://www.ansible.com/) for the operating system and application configuration. The systems can be deployed on any of the SAP-supported operating system versions and deployed into any Azure region.

+Hashicorp [Terraform](https://www.terraform.io/) is an open-source tool for provisioning and managing cloud infrastructure.

+- Deployment infrastructure (control plane, hub component)

+- SAP Infrastructure (SAP Workload, spoke component)

+You'll use the control plane of the SAP on Azure Deployment Automation Framework to deploy the SAP Infrastructure and the SAP application. The deployment uses Terraform templates to create the [infrastructure as a service (IaaS)](https://azure.microsoft.com/overview/what-is-iaas) defined infrastructure to host the SAP Applications.

+>

+The automation framework can be used to deploy the following SAP architectures:

+The control plane houses the deployment infrastructure from which other environments will be deployed. Once the control plane is deployed, it rarely needs to be redeployed, if ever.

+- Deployment agents for running:

+ - Terraform Deployment

+ - Ansible configuration

+- Persistent storage for the Terraform state files

+- Persistent storage for the Downloaded SAP Software

+- Azure Key Vault for secure storage for deployment credentials

+- Private DNS zone (optional)

+The control plane is typically a regional resource deployed in to the hub subscription in a [hub and spoke architecture](/azure/architecture/reference-architectures/hybrid-networking/hub-spoke).

+The application configuration will be performed from the deployment agents in the Control plane using a set of pre-defined playbooks. These playbooks will:

+- Install the SAP system components

+The framework also provides an Ansible playbook that can be used to download the software from SAP and persist it in the storage accounts in the Control Plane's SAP Library resource group.

+### Deployer Virtual Machines

+These virtual machines are used to run the orchestration scripts that will deploy the Azure resources using Terraform. They are also Ansible Controllers and are used to execute the Ansible playbooks on all the managed nodes, i.e the virtual machines of an SAP deployment.

+The SAP Workload contains all the Azure infrastructure resources for the SAP Deployments. These resources are deployed from the control plane.

+- SAP System(s)

+The workload zone allows for partitioning of the deployments into different environments (Development, Test, Production). The Workload zone will provide the shared services (networking, credentials management) to the SAP systems.

+> | Term | Description |

+> | - | -- |

+> [Planning for the automation framwework](plan-deployment.md)

+| China North 3 | September 7, 2022 or later |

+| South Africa North | September 7, 2022 or later |

+| Switzerland North | September 7, 2022 or later |

+| [Azure Firewall](../../firewall/overview.md) | A cloud-native and intelligent network firewall security service that provides threat protection for your cloud workloads running in Azure. It's a fully stateful firewall as a service with built-in high availability and unrestricted cloud scalability. Azure Firewall is offered in three SKUs: [Standard](../../firewall/features.md), [Premium](../../firewall/premium-features.md), and [Basic](../../firewall/overview.md#azure-firewall-basic). |

+Perimeter networks are useful because you can focus your network access control management, monitoring, logging, and reporting on the devices at the edge of your Azure virtual network. A perimeter network is where you typically enable [distributed denial of service (DDoS) protection](../../ddos-protection/ddos-protection-overview.md), intrusion detection/intrusion prevention systems (IDS/IPS), firewall rules and policies, web filtering, network antimalware, and more. The network security devices sit between the internet and your Azure virtual network and have an interface on both networks.

+- Azure native controls. [Azure Firewall](../../firewall/overview.md) and [Azure Web Application Firewall](../../web-application-firewall/overview.md) offer basic security advantages. Advantages are a fully stateful firewall as a service, built-in high availability, unrestricted cloud scalability, FQDN filtering, support for OWASP core rule sets, and simple setup and configuration.

+While the Microsoft Sentinel solution for **Zero Trust (TIC 3.0)** provides best practice guidance, Microsoft does not guarantee nor imply compliance. All Trusted Internet Connection (TIC) requirements, validations, and controls are governed by the [Cybersecurity & Infrastructure Security Agency](https://www.cisa.gov/resources-tools/programs/trusted-internet-connections-tic).

+1. Save the script for Azure Spring Apps [Standard tier](https://raw.githubusercontent.com/Azure/azure-spring-apps-landing-zone-accelerator/reference-architecture/CLI/brownfield-deployment/azuredeploySpringStandard.sh) or [Enterprise tier](https://raw.githubusercontent.com/Azure/azure-spring-apps-landing-zone-accelerator/reference-architecture/CLI/brownfield-deployment/azuredeploySpringEnterprise.sh) locally, then run it from the Bash prompt.