+>[!NOTE]

+>User accounts that were recently deleted, also known as [soft-deleted users](../fundamentals/active-directory-users-restore.md), are not listed in user registration details.

+- Methods registered (Alternate Mobile Phone, Certificate-based authentication, Email, FIDO2 security key, Hardware OATH token, Microsoft Authenticator app, Microsoft Passwordless phone sign-in, Mobile phone, Office phone, Security questions, Software OATH token, Temporary Access Pass, Windows Hello for Business)

+# View privileged role assignments in your organization

+- If you're using Azure AD for customers (preview), the support request feature is currently unavailable in customer tenants. However, you can use the **Give Feedback** link on the **New support request** page to provide feedback. Or, you can switch to your Azure AD workforce tenant and [open a support request](https://portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/newsupportrequest).

+The `optionalClaims` object declares the optional claims requested by an application. An application can configure optional claims that are returned in ID tokens, access tokens, and SAML 2 tokens. The application can configure a different set of optional claims to be returned in each token type.

+>[!NOTE]

+> The following Cloud Kerberos diagnostics fields were added in the original release of Windows 11 (version 21H2).

+- **OnPremTgt**: Set the state to *YES* if a Cloud Kerberos ticket to access on-premises resources is present on the device for the logged-in user.

+- **CloudTgt**: Set the state to *YES* if a Cloud Kerberos ticket to access cloud resources is present on the device for the logged-in user.

+- **KerbTopLevelNames**: List of top level Kerberos realm names for Cloud Kerberos.

+ OnPremTgt : YES

+ CloudTgt : YES

+ KerbTopLevelNames : .windows.net,.windows.net:1433,.windows.net:3342,.azure.net,.azure.net:1433,.azure.net:3342

+>[!NOTE]

+> The following Cloud Kerberos diagnostics fields were added in the Windows 10 May 2021 update (version 21H1).

+>[!NOTE]

+> Prior to Windows 11 version 23H2, the setting **OnPremTGT** was named **CloudTGT**.

+- **OnPremTGT**: This setting is specific to Cloud Kerberos trust deployment and present only if the CertEnrollment state is *none*. Set the state to *YES* if the device has a Cloud Kerberos ticket to access on-premises resources. Prior to Windows 11 version 23H2, this setting was named **CloudTGT**.

+## Configure group settings using Microsoft Graph

+To configure the _Users can create security groups in Azure portals, API or PowerShell_ setting using Microsoft Graph, configure the **EnableGroupCreation** object in the groupSettings object. For more information, see [Overview of group settings](/graph/group-directory-settings).

+To configure the _Users can create security groups in Azure portals, API or PowerShell_ setting using Microsoft Graph, update the **allowedToCreateSecurityGroups** property of **defaultUserRolePermissions** in the [authorizationPolicy](/graph/api/resources/authorizationpolicy) object.

+>This information last updated on June 1st, 2023.<br/>You can also download a CSV version of this table [here](https://download.microsoft.com/download/e/3/e/e3e9faf2-f28b-490a-9ada-c6089a1fc5b0/Product%20names%20and%20service%20plan%20identifiers%20for%20licensing.csv).

+| Windows 10/11 Enterprise A3 for faculty | WIN10_ENT_A3_FAC | 8efbe2f6-106e-442f-97d4-a59aa6037e06 | EXCHANGE_S_FOUNDATION (113feb6c-3fe4-4440-bddc-54d774bf0318)<br/>UNIVERSAL_PRINT_01 (795f6fe0-cc4d-4773-b050-5dde4dc704c9)<br/>Virtualization Rights for Windows 10 (E3/E5+VDA) (e7c91390-7625-45be-94e0-e16907e03118)<br/>WINDOWSUPDATEFORBUSINESS_DEPLOYMENTSERVICE (7bf960f6-2cd9-443a-8046-5dbff9558365) | Exchange Foundation (113feb6c-3fe4-4440-bddc-54d774bf0318)<br/>Universal Print (795f6fe0-cc4d-4773-b050-5dde4dc704c9)<br/>Windows 10 Enterprise (New) (e7c91390-7625-45be-94e0-e16907e03118)<br/>Windows Update for Business Deployment Service (7bf960f6-2cd9-443a-8046-5dbff9558365) |

+| Windows 10/11 Enterprise A3 for students | WIN10_ENT_A3_STU | d4ef921e-840b-4b48-9a90-ab6698bc7b31 | EXCHANGE_S_FOUNDATION (113feb6c-3fe4-4440-bddc-54d774bf0318)<br/>UNIVERSAL_PRINT_01 (795f6fe0-cc4d-4773-b050-5dde4dc704c9)<br/>Virtualization Rights for Windows 10 (E3/E5+VDA) (e7c91390-7625-45be-94e0-e16907e03118)<br/>WINDOWSUPDATEFORBUSINESS_DEPLOYMENTSERVICE (7bf960f6-2cd9-443a-8046-5dbff9558365) | Exchange Foundation (113feb6c-3fe4-4440-bddc-54d774bf0318)<br/>Universal Print (795f6fe0-cc4d-4773-b050-5dde4dc704c9)<br/>Windows 10 Enterprise (New) (e7c91390-7625-45be-94e0-e16907e03118)<br/>Windows Update for Business Deployment Service (7bf960f6-2cd9-443a-8046-5dbff9558365) |

+| WINDOWS 10/11 ENTERPRISE E3 | WIN10_PRO_ENT_SUB | cb10e6cd-9da4-4992-867b-67546b1db821 | WIN10_PRO_ENT_SUB (21b439ba-a0ca-424f-a6cc-52f954a5b111) | WINDOWS 10 ENTERPRISE (21b439ba-a0ca-424f-a6cc-52f954a5b111) |

+| WINDOWS 10/11 ENTERPRISE E3 | WIN10_VDA_E3 | 6a0f6da5-0b87-4190-a6ae-9bb5a2b9546a | EXCHANGE_S_FOUNDATION (113feb6c-3fe4-4440-bddc-54d774bf0318)<br/>UNIVERSAL_PRINT_01 (795f6fe0-cc4d-4773-b050-5dde4dc704c9)<br/>Virtualization Rights for Windows 10 (E3/E5+VDA) (e7c91390-7625-45be-94e0-e16907e03118)<br/>WINDOWSUPDATEFORBUSINESS_DEPLOYMENTSERVICE (7bf960f6-2cd9-443a-8046-5dbff9558365)<br/>Windows_Autopatch (9a6eeb79-0b4b-4bf0-9808-39d99a2cd5a3) | EXCHANGE FOUNDATION (113feb6c-3fe4-4440-bddc-54d774bf0318)<br/>UNIVERSAL PRINT (795f6fe0-cc4d-4773-b050-5dde4dc704c9)<br/>WINDOWS 10 ENTERPRISE (NEW) (e7c91390-7625-45be-94e0-e16907e03118)<br/>WINDOWS UPDATE FOR BUSINESS DEPLOYMENT SERVICE (7bf960f6-2cd9-443a-8046-5dbff9558365)<br/>Windows Autopatch (9a6eeb79-0b4b-4bf0-9808-39d99a2cd5a3) |

+| Windows 10/11 Enterprise E5 | WIN10_VDA_E5 | 488ba24a-39a9-4473-8ee5-19291e71b002 | EXCHANGE_S_FOUNDATION (113feb6c-3fe4-4440-bddc-54d774bf0318)<br/>WINDEFATP (871d91ec-ec1a-452b-a83f-bd76c7d770ef)<br/>UNIVERSAL_PRINT_01 (795f6fe0-cc4d-4773-b050-5dde4dc704c9)<br/>Virtualization Rights for Windows 10 (E3/E5+VDA) (e7c91390-7625-45be-94e0-e16907e03118)<br/>WINDOWSUPDATEFORBUSINESS_DEPLOYMENTSERVICE (7bf960f6-2cd9-443a-8046-5dbff9558365)<br/>Windows_Autopatch (9a6eeb79-0b4b-4bf0-9808-39d99a2cd5a3) | Exchange Foundation (113feb6c-3fe4-4440-bddc-54d774bf0318)<br/>Microsoft Defender For Endpoint (871d91ec-ec1a-452b-a83f-bd76c7d770ef)<br/>Universal Print (795f6fe0-cc4d-4773-b050-5dde4dc704c9)<br/>Windows 10 Enterprise (New) (e7c91390-7625-45be-94e0-e16907e03118)<br/>Windows Update for Business Deployment Service (7bf960f6-2cd9-443a-8046-5dbff9558365)<br/>Windows Autopatch (9a6eeb79-0b4b-4bf0-9808-39d99a2cd5a3) |

+| Windows 10/11 Enterprise E5 Commercial (GCC Compatible) | WINE5_GCC_COMPAT | 938fd547-d794-42a4-996c-1cc206619580 | EXCHANGE_S_FOUNDATION_GOV (922ba911-5694-4e99-a794-73aed9bfeec8)<br/>WINDEFATP (871d91ec-ec1a-452b-a83f-bd76c7d770ef)<br/>Virtualization Rights for Windows 10 (E3/E5+VDA) (e7c91390-7625-45be-94e0-e16907e03118) | Exchange Foundation for Government (922ba911-5694-4e99-a794-73aed9bfeec8)<br/>Microsoft Defender For Endpoint (871d91ec-ec1a-452b-a83f-bd76c7d770ef)<br/>Windows 10 Enterprise (New) (e7c91390-7625-45be-94e0-e16907e03118) |

+### Updating sign-in methods

+At any time, you can update the sign-in options you've selected for an app. For example, you can add social identity providers or change the local account sign-in method.

+Be aware that when you change sign-in methods, the change affects only new users. Existing users will continue to sign in using their original method. For example, suppose you start out with the email and password sign-in method, and then change to email with one-time passcode. New users will sign in using a one-time passcode, but any users who have already signed up with an email and password will continue to be prompted for their email and password.

+# Customize the neutral default authentication experience for the customer tenant (preview)

+# Planning for customer identity and access management (preview)

+Microsoft Entra External ID for customers, also known as Azure Active Directory (Azure AD) for customers, is a customizable, extensible solution for adding customer identity and access management (CIAM) to your app. Because it's built on the Azure AD platform, you benefit from consistency in app integration, tenant management, and operations across your workforce and customer scenarios. When designing your configuration, it's important to understand the components of a customer tenant and the Azure AD features that are available for your customer scenarios.

+- **Number of user flows**. Each application can have just one sign-up and sign-in user flow. If you have several applications, you can use a single user flow for all of them. Or, if you want a different experience for each application, you can create multiple user flows. The maximum is 10 user flows per customer tenant.

+- **Company branding and language customizations**. Although we describe configuring company branding and language customizations later in Step 4, you can configure them anytime, either before or after you integrate an app with a user flow. If you configure company branding before you create the user flow, the sign-in pages reflect that branding. Otherwise, the sign-in pages reflect the default, neutral branding.

+# Supported features in Azure Active Directory for customers (preview)

+# Create a customer identity and access management (CIAM) tenant (preview)

+# Customize the neutral branding in your customer tenant (preview)

+Your Azure AD tenant supports Microsoft look and feel as a default state for authentication experience. You can [customize the default Microsoft sign-in experience](/azure/active-directory/fundamentals/how-to-customize-branding) with a custom background image or color, favicon, layout, header, and footer. You can also upload a [custom CSS](/azure/active-directory/fundamentals/reference-company-branding-css-template). If the custom company branding fails to load for any reason, the sign-in page will revert to the default Microsoft branding.

+Before you customize any settings, the neutral default branding will appear in your sign-in and sign-up pages. You can customize this default experience with a custom background image or color, favicon, layout, header, and footer. You can also upload a [custom CSS](/azure/active-directory/fundamentals/reference-company-branding-css-template).

+# Add and manage customer accounts (preview)

+# Quickstart: Get started with Azure AD for customers (preview)

+> [!NOTE]

+> The next time you return to your tenant, you might be prompted to set up additional authentication factors for added security of your tenant admin account.

+ Title: Known issues in customer tenants

+description: Learn about known issues in customer tenants.

+# Known issues with Azure Active Directory (Azure AD) for customers

+This article describes known issues that you may experience when you use Azure Active Directory (Azure AD) for customers, and provides help to resolve these issues.

+## Tenant creation and management

+### Tenant creation fails when you choose an unsupported region

+During customer tenant creation, the **Country/Region** dropdown menu lists countries and regions where Azure Azure AD for customers isn't yet available. If you choose Japan or Australia, tenant creation fails.

+**Cause**: Public preview is currently available in the Americas and Europe, with more regions to follow shortly.

+**Workaround**: Select a different region and try again

+### Customer trial tenants can't be extended or linked with an existing Azure subscription

+Customer trial tenants can't be supported beyond 30 days.

+**Workaround**: Take one of the following actions.

+- To continue beyond 30 days, if you're an existing Azure AD customer, [create a new customer tenant](how-to-create-customer-tenant-portal.md) with your subscription.

+- If you donΓÇÖt have an Azure AD account, delete the trial tenant and [set up an Azure free account](https://azure.microsoft.com/free/).

+### The get started guide UI lacks client-side validation for the Domain name field

+When you manually update the autopopulated value for the **Domain name** field, it may appear as though the value is accepted, but then an error occurs.

+**Cause**: Currently there's no client-side validation in the get started guide for setting up a trial tenant.

+**Workaround**: Enter a value that meets the domain name requirements. The **Domain name** field accepts an alphanumeric value with a length of up to 27 characters.

+### Using your admin email to create a local customer account prevents you from administering the tenant

+If you're the admin who created the customer tenant, and you use the same email address as your admin account to create a local customer account in that same tenant, you can't sign in directly to the tenant with admin privileges.

+**Cause**: Using your tenant admin email to create a customer account via self-service sign-up creates a second user with the same email address, but with customer-level privileges. When you sign in to the tenant via `https://entra.microsoft.com/<tenantID>` or `<tenantName>.onmicrosoft.com`, the least-privileged account takes precedence, and you're signed in as the customer instead of the admin. You have insufficient privileges to manage the tenant.

+**Workaround**: Take one of the following actions.

+- When creating a local customer account, use a different email address than the one used by the admin who created the tenant.

+- If you've already created a customer account with the same email address as the admin, sign out of the admin center, and then use `https://entra.microsoft.com` instead of `https://entra.microsoft.com/<tenantID>` or `<tenantName>.onmicrosoft.com` to sign in with the correct admin account.

+### Unable to delete your customer tenant

+You get the following error when you try to delete a customer tenant:

+ `Unable to delete tenant`

+**Cause**: This error occurs when you try to delete a customer tenant but you haven't deleted the b2c-extensions-app.

+**Workaround**: When deleting a customer tenant, delete the **b2c-extensions-app**, found in **App registrations** under **All applications**.

+## Branding

+### Device code flows display Microsoft branding instead of custom branding

+The device code flows display Microsoft branding even when you've configured custom branding.

+**Cause**: Device code flows don't yet support custom branding

+**Workaround**: None currently.

+### The sign-up page displays Microsoft branding and "Can't access your account?"

+After you set up a tenant and create a sign-up user flow, you see Microsoft branding instead of neutral branding, along with **Can't access your account?** under the sign-in email box instead of **No account? Create one**.

+**Cause**: The sign-in page for a workforce tenant is displaying instead of sign-in for a customer tenant. This issue can occur when you refresh the sign-in page too many times in quick succession.

+**Workaround**: Wait a few minutes and then refresh. The customer sign-in page should appear.

+## Samples

+### Error when signing in to a sample

+When you follow the get started guide to run a sample and try to sign in as a customer, you see an error message that starts with the following text:

+ `AADSTS50011: The redirect URI specified in the request does not match the redirect URIs configured for the application...`

+**Cause**: This error can occur when there is a replication delay in updating the redirect URI in the app registration.

+**Workaround**: Take one of the following actions.

+- Try running the sample sign in again after a few minutes.

+- Check the app registration to confirm that the redirect URI in the error is configured.

+### Error "Invalid client secret providedΓÇ¥ (ASP.NET Core) or ΓÇ£Cannot read properties of undefined (reading 'verifier')ΓÇ¥ (Node.js)

+When you run the ASP.NET Core sample from the get started guide and try to sign in as a customer, you see an error that starts with the following text:

+ `AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app...`

+Or, when you run the Node.js sample, you see an error containing the following line:

+ `TypeError: Cannot read properties of undefined (reading 'verifier')`

+**Cause**: These errors can be caused by a replication delay in updating the secret in the app registration.

+**Workaround**: Take one of the following actions.

+- Try running the sample sign in again after a few minutes.

+- Check the app registration to confirm there's a client secret configured and it matches the value in the application configuration.

+## Token version in Web API

+### Error when running a web API

+When you create your own web API in a customer tenant (without using the app creation scripts in the web API samples), and then run it and send an access token, you enable logging and see the following error:

+ `IDX20804: Unable to retrieve document from: https://<tenant>.ciamlogin.com/common/discovery/keys`

+**Cause**: This error occurs if you haven't set the accepted access token version to 2.

+**Workaround**: Do the following.

+1. Go to the app registration for your application.

+1. Choose to edit the manifest.

+1. Change the **accessTokenAcceptedVersion** property from null to **2**.

+## Next steps

+See also [Supported features in Azure Active Directory for customers](concept-supported-features-customers.md)

+## May 2023

+### New article

+- [Set up tenant restrictions V2 (Preview)](tenant-restrictions-v2.md)

+### Updated articles

+- [Overview: Cross-tenant access with Azure AD External Identities](cross-tenant-access-overview.md) Graph API links were updated.

+- [Reset redemption status for a guest user](reset-redemption-status.md) Screenshots were updated.

+description: Learn about a baseline design for a multilateral federation solution for universities.

+Microsoft often speaks with research universities that operate in hybrid environments in which applications are either cloud based or hosted on-premises. In both cases, applications can use various authentication protocols. In some cases, these protocols are reaching end of life or aren't providing the required level of security.

+[](media/multilateral-federation-baseline/typical-baseline-environment.png#lightbox)

+Applications drive much of the need for different authentication protocols and different identity management (IdM) mechanisms.

+In research university environments, research apps often drive IdM requirements. A university might use a federation provider, such as Shibboleth, as a primary identity provider (IdP). If so, Azure Active Directory (Azure AD) is often configured to federate with Shibboleth. If Microsoft 365 apps are also in use, Azure AD enables you to configure integration.

+Applications used in research universities operate in various parts of the overall IT footprint:

+* Research and multilateral federation applications are available through InCommon and eduGAIN.

+* Some applications use legacy authentication protocols such as Central Authentication Service to enable single sign-on.

+* Student and faculty applications often use multiple authentication mechanisms. For example, some are integrated with Shibboleth or other federation providers, whereas others are integrated with Azure AD.

+* Windows Server Active Directory might be in use and synchronized with Azure AD.

+* Lightweight Directory Access Protocol (LDAP) is in use at many universities that might have an external LDAP directory or identity registry. These registries are often used to house confidential attributes, role hierarchy information, and even certain types of users, such as applicants.

+* On-premises Active Directory, or an external LDAP directory, is often used to enable single-credential sign-in for non-web applications and various non-Microsoft operating system sign-ins.

+Baseline architectures often evolve over time, introducing complexity and rigidness to the design and the ability to update. Some of the challenges with using the baseline architecture include:

+* **Hard to react to new requirements**: Having a complex environment makes it hard to quickly adapt and keep up with the most recent regulations and requirements. For example, if you have apps in lots of locations, and these apps are connected in different ways with different IdMs, you have to decide where to locate multifactor authentication (MFA) services and how to enforce MFA.

+ Higher education also experiences fragmented service ownership. The people responsible for key services such as enterprise resource planning, learning management systems, division, and department solutions might resist efforts to change or modify the systems that they operate.

+* **Can't take advantage of all Microsoft 365 capabilities for all apps** (for example, Intune, Conditional Access, passwordless): Many universities want to move toward the cloud and use their existing investments in Azure AD. However, with a different federation provider as their primary IdP, universities can't take advantage of all the Microsoft 365 capabilities for the rest of their apps.

+* **Complexity of a solution**: There are many components to manage. Some components are in the cloud, and some are on-premises or in infrastructure as a service (IaaS) instances. Apps are operated in many places. From a user perspective, this experience can be disjointed. For example, users sometime see a Shibboleth sign-in page and other times see an Azure AD sign-in page.

+We present three solutions to solve these challenges, while also addressing the following requirements:

+* Ability to support all types of apps (even apps that require legacy protocols)

+We present the three solutions in order, from most preferred to least preferred. Each satisfies requirements but introduces tradeoff decisions that are expected in a complex architecture. Based on your requirements and starting point, select the one that best suits your environment. We also provide a decision tree to aid in this decision.

+See these related articles about multilateral federation:

+[Multilateral federation Solution 1: Azure AD with Cirrus Bridge](multilateral-federation-solution-one.md)

+[Multilateral federation Solution 2: Azure AD with Shibboleth as a SAML proxy](multilateral-federation-solution-two.md)

+[Multilateral federation Solution 3: Azure AD with AD FS and Shibboleth](multilateral-federation-solution-three.md)

+Use this decision tree to determine the multilateral federation solution that's best suited for your environment.

+[](media/multilateral-federation-decision-tree/tradeoff-decision-matrix.png#lightbox)

+The following resources can help with your migration to the solutions covered in this content.

+| Migration resource | Description | Relevant for migrating to... |

+| [Resources for migrating applications to Azure Active Directory](../manage-apps/migration-resources.md) | List of resources to help you migrate application access and authentication to Azure Active Directory (Azure AD) | Solution 1, Solution 2, and Solution 3 |

+| [Azure AD custom claims provider](../develop/custom-claims-provider-overview.md)| Overview of the Azure AD custom claims provider | Solution 1 |

+| [Custom security attributes](../fundamentals/custom-security-attributes-manage.md) | Steps for managing access to custom security attributes | Solution 1 |

+| [Azure AD SSO integration with Cirrus Bridge](../saas-apps/cirrus-identity-bridge-for-azure-ad-tutorial.md) | Tutorial to integrate Cirrus Bridge with Azure AD | Solution 1 |

+| [Cirrus Bridge overview](https://blog.cirrusidentity.com/documentation/azure-bridge-setup-rev-6.0) | Cirrus Identity documentation for configuring Cirrus Bridge with Azure AD | Solution 1 |

+| [Configuring Shibboleth as a SAML proxy](https://shibboleth.atlassian.net/wiki/spaces/KB/pages/1467056889/Using+SAML+Proxying+in+the+Shibboleth+IdP+to+connect+with+Azure+AD) | Shibboleth article that describes how to use the SAML proxying feature to connect the Shibboleth identity provider (IdP) to Azure AD | Solution 2 |

+| [Azure AD Multi-Factor Authentication deployment considerations](../authentication/howto-mfa-getstarted.md) | Guidance for configuring Azure AD Multi-Factor Authentication | Solution 1 and Solution 2 |

+See these related articles about multilateral federation:

+[Multilateral federation baseline design](multilateral-federation-baseline.md)

+[Multilateral federation Solution 1: Azure AD with Cirrus Bridge](multilateral-federation-solution-one.md)

+[Multilateral federation Solution 2: Azure AD with Shibboleth as a SAML proxy](multilateral-federation-solution-two.md)

+[Multilateral federation Solution 3: Azure AD with AD FS and Shibboleth](multilateral-federation-solution-three.md)

+Research universities need to collaborate with one another. To accomplish collaboration, they require multilateral federation to enable authentication and access between universities globally.

+Universities face many challenges. For example, a university might use one identity management system and a set of protocols. Other universities might use a different set of technologies, depending on their requirements. In general, universities can:

+* Use different identity management systems.

+* Use different protocols.

+* Use customized solutions.

+* Need support for a long history of legacy functionality.

+* Need support for solutions that are built in different IT generations.

+* Single sign-on across multiple applications.

+* Modern security controls, including passwordless authentication, multifactor authentication, adaptive Conditional Access, and identity protection.

+* Enhanced reporting and monitoring.

+Because Azure AD doesn't natively support multilateral federation, this content describes three solutions for federating authentication and access between universities with a typical research university architecture. These scenarios mention non-Microsoft products for illustrative purposes only and to represent the broader class of products. For example, this content uses Shibboleth as an example of a federation provider.

+See these related articles about multilateral federation:

+[Multilateral federation baseline design](multilateral-federation-baseline.md)

+[Multilateral federation Solution 1: Azure AD with Cirrus Bridge](multilateral-federation-solution-one.md)

+[Multilateral federation Solution 2: Azure AD with Shibboleth as a SAML proxy](multilateral-federation-solution-two.md)

+[Multilateral federation Solution 3: Azure AD with AD FS and Shibboleth](multilateral-federation-solution-three.md)

+ Title: 'Solution 1: Azure AD with Cirrus Bridge'

+description: This article describes design considerations for using Azure AD with Cirrus Bridge as a multilateral federation solution for universities.

+Solution 1 uses Azure Active Directory (Azure AD) as the primary identity provider (IdP) for all applications. A managed service provides multilateral federation. In this example, Cirrus Bridge is the managed service for integration of Central Authentication Service (CAS) and multilateral federation apps.

+[](media/multilateral-federation-solution-one/cirrus-bridge.png#lightbox)

+If you're also using an on-premises Active Directory instance, you can [configure Active Directory](../hybrid/whatis-hybrid-identity.md) with hybrid identities. Implementing a solution of using Azure AD with Cirrus Bridge provides:

+* **Security Assertion Markup Language (SAML) bridge**: Configure multilateral federation and participation in InCommon and eduGAIN. You can also use the SAML bridge to configure Azure AD Conditional Access policies, app assignment, governance, and other features for each multilateral federation app.

+* **CAS bridge**: Provide protocol translation to support on-premises CAS apps to authenticate with Azure AD. You can use the CAS bridge to configure Azure AD Conditional Access policies, app assignment, and governance for all CAS apps as a whole.

+When you implement Azure AD with Cirrus Bridge, you can take advantage of more capabilities in Azure AD:

+* **Custom claims provider support**: With the [Azure AD custom claims provider](../develop/custom-claims-provider-overview.md), you can use an external attribute store (like an external LDAP directory) to add claims into tokens for individual apps. The custom claims provider uses a custom extension that calls an external REST API to fetch claims from external systems.

+* **Custom security attributes**: You can add custom attributes to objects in the directory and control who can read them. [Custom security attributes](../fundamentals/custom-security-attributes-overview.md) enable you to store more of your attributes directly in Azure AD.

+Here are some of the advantages of implementing Azure AD with Cirrus Bridge:

+ * All apps authenticate through Azure AD.

+ * Elimination of all on-premises identity components in a managed service can potentially lower your operational and administrative costs, reduce security risks, and free up resources for other efforts.

+ * Cirrus Identity provides continuous support.

+* **Conditional Access support for multilateral federation apps**

+ * Implementation of Conditional Access controls helps you comply with [NIH](https://auth.nih.gov/CertAuthV3/forms/help/compliancecheckhelp.html) and [REFEDS](https://refeds.org/category/research-and-scholarship) requirements.

+ * This solution is the only architecture that enables you to configure granular Azure AD Conditional Access for both multilateral federation apps and CAS apps.

+* **Use of other Azure AD-related solutions for all apps**

+ * You can use Intune and Azure AD join for device management.

+ * Azure AD join enables you to use Windows Autopilot, Azure AD Multi-Factor Authentication, and passwordless features. Azure AD join supports achieving a Zero Trust posture.

+ > [!NOTE]

+ > Switching to Azure AD Multi-Factor Authentication might help you save significant costs over other solutions that you have in place.

+Here are some of the trade-offs of using this solution:

+* **Limited ability to customize the authentication experience**: This scenario provides a managed solution. It might not offer you the flexibility or granularity to build a custom solution by using federation provider products.

+* **Limited third-party MFA integration**: The number of integrations available to third-party MFA solutions might be limited.

+* **One-time integration effort required**: To streamline integration, you need to perform a one-time migration of all student and faculty apps to Azure AD. You also need to set up Cirrus Bridge.

+* **Subscription required for Cirrus Bridge**: The subscription fee for Cirrus Bridge is based on anticipated annual authentication usage of the bridge.

+The following resources help with your migration to this solution architecture.

+| Migration resource | Description |

+| [Resources for migrating applications to Azure Active Directory](../manage-apps/migration-resources.md) | List of resources to help you migrate application access and authentication to Azure AD |

+| [Azure AD custom claims provider](../develop/custom-claims-provider-overview.md)| Overview of the Azure AD custom claims provider |

+| [Custom security attributes](../fundamentals/custom-security-attributes-manage.md) | Steps for managing access to custom security attributes |

+| [Azure AD single sign-on integration with Cirrus Bridge](../saas-apps/cirrus-identity-bridge-for-azure-ad-tutorial.md) | Tutorial to integrate Cirrus Bridge with Azure AD |

+| [Cirrus Bridge overview](https://blog.cirrusidentity.com/documentation/azure-bridge-setup-rev-6.0) | Cirrus Identity documentation for configuring Cirrus Bridge with Azure AD |

+| [Azure AD Multi-Factor Authentication deployment considerations](../authentication/howto-mfa-getstarted.md) | Guidance for configuring Azure AD Multi-Factor Authentication |

+See these related articles about multilateral federation:

+[Multilateral federation baseline design](multilateral-federation-baseline.md)

+[Multilateral federation Solution 2: Azure AD with Shibboleth as a SAML proxy](multilateral-federation-solution-two.md)

+[Multilateral federation Solution 3: Azure AD with AD FS and Shibboleth](multilateral-federation-solution-three.md)

+ Title: 'Solution 3: Azure AD with AD FS and Shibboleth'

+description: This article describes design considerations for using Azure AD with AD FS and Shibboleth as a multilateral federation solution for universities.

+# Solution 3: Azure AD with AD FS and Shibboleth

+In Solution 3, the federation provider is the primary identity provider (IdP). In this example, Shibboleth is the federation provider for the integration of multilateral federation apps, on-premises Central Authentication Service (CAS) apps, and any Lightweight Directory Access Protocol (LDAP) directories.

+[](media/multilateral-federation-solution-three/shibboleth-adfs-azure-ad.png#lightbox)

+Student apps, faculty apps, and Microsoft 365 apps are integrated with Azure Active Directory (Azure AD). Any on-premises instance of Active Directory is synced with Azure AD. Active Directory Federation Services (AD FS) provides integration with third-party multifactor authentication (MFA). AD FS performs protocol translation and enables certain Azure AD features, such as Azure AD join for device management, Windows Autopilot, and passwordless features.

+Here are some of the advantages of using this solution:

+* **Customized authentication**: You can customize the experience for multilateral federation apps through Shibboleth.

+* **Ease of execution**: The solution is simple to implement in the short term for institutions that already use Shibboleth as their primary IdP. You need to migrate student and faculty apps to Azure AD and add an AD FS instance.

+* **Minimal disruption**: The solution allows third-party MFA. You can keep existing MFA solutions, such as Duo, in place until you're ready for an update.

+Here are some of the trade-offs of using this solution:

+* **Higher complexity and security risk**: An on-premises footprint might mean higher complexity for the environment and extra security risks, compared to a managed service. Increased overhead and fees might also be associated with managing on-premises components.

+* **Suboptimal authentication experience**: For multilateral federation and CAS apps, there's no cloud-based authentication mechanism and there might be multiple redirects.

+* **No Azure AD Multi-Factor Authentication support**: This solution doesn't enable Azure AD Multi-Factor Authentication support for multilateral federation or CAS apps. You might miss potential cost savings.

+* **No granular Conditional Access support**: The lack of granular Conditional Access support limits your ability to make granular decisions.

+* **Significant ongoing staff allocation**: IT staff must maintain infrastructure and software for the authentication solution. Any staff attrition might introduce risk.

+The following resources can help with your migration to this solution architecture.

+| Migration resource | Description |

+| [Resources for migrating applications to Azure Active Directory](../manage-apps/migration-resources.md) | List of resources to help you migrate application access and authentication to Azure AD |

+See these related articles about multilateral federation:

+[Multilateral federation baseline design](multilateral-federation-baseline.md)

+[Multilateral federation Solution 1: Azure AD with Cirrus Bridge](multilateral-federation-solution-one.md)

+[Multilateral federation Solution 2: Azure AD with Shibboleth as a SAML proxy](multilateral-federation-solution-two.md)

+ Title: 'Solution 2: Azure AD with Shibboleth as a SAML proxy'

+description: This article describes design considerations for using Azure AD with Shibboleth as a SAML proxy as a multilateral federation solution for universities.

+# Solution 2: Azure AD with Shibboleth as a SAML proxy

+In Solution 2, Azure Active Directory (Azure AD) acts as the primary identity provider (IdP). The federation provider acts as a Security Assertion Markup Language (SAML) proxy to the Central Authentication Service (CAS) apps and the multilateral federation apps. In this example, [Shibboleth acts as the SAML proxy](https://shibboleth.atlassian.net/wiki/spaces/KB/pages/1467056889/Using+SAML+Proxying+in+the+Shibboleth+IdP+to+connect+with+Azure+AD) to provide a reference link.

+[](media/multilateral-federation-solution-two/azure-ad-shibboleth-as-sp-proxy.png#lightbox)

+Because Azure AD is the primary IdP, all student and faculty apps are integrated with Azure AD. All Microsoft 365 apps are also integrated with Azure AD. If Azure Active Directory Domain Services is in use, it also is synchronized with Azure AD.

+Advantages of using this solution include:

+* **Cloud authentication for all apps**: All apps authenticate through Azure AD.

+* **Ease of execution**: This solution provides short-term ease of execution for universities that are already using Shibboleth.

+Here are some of the trade-offs of using this solution:

+* **Higher complexity and security risk**: An on-premises footprint might mean higher complexity for the environment and extra security risks, compared to a managed service. Increased overhead and fees might also be associated with managing on-premises components.

+* **Suboptimal authentication experience**: For multilateral federation and CAS apps, the authentication experience for users might not be seamless because of redirects through Shibboleth. The options for customizing the authentication experience for users are limited.

+* **Limited third-party multifactor authentication (MFA) integration**: The number of integrations available to third-party MFA solutions might be limited.

+* **No granular Conditional Access support**: Without granular Conditional Access support, you have to choose between the least common denominator (optimize for less friction but have limited security controls) or the highest common denominator (optimize for security controls at the expense of user friction). Your ability to make granular decisions is limited.

+The following resources can help with your migration to this solution architecture.

+| Migration resource | Description |

+| [Resources for migrating applications to Azure Active Directory](../manage-apps/migration-resources.md) | List of resources to help you migrate application access and authentication to Azure AD |

+| [Configuring Shibboleth as a SAML proxy](https://shibboleth.atlassian.net/wiki/spaces/KB/pages/1467056889/Using+SAML+Proxying+in+the+Shibboleth+IdP+to+connect+with+Azure+AD) | Shibboleth article that describes how to use the SAML proxying feature to connect the Shibboleth IdP to Azure AD |

+| [Azure AD Multi-Factor Authentication deployment considerations](../authentication/howto-mfa-getstarted.md) | Guidance for configuring Azure AD Multi-Factor Authentication |

+See these related articles about multilateral federation:

+[Multilateral federation baseline design](multilateral-federation-baseline.md)

+[Multilateral federation Solution 1: Azure AD with Cirrus Bridge](multilateral-federation-solution-one.md)

+[Multilateral federation Solution 3: Azure AD with AD FS and Shibboleth](multilateral-federation-solution-three.md)

+1. In the Azure portal, select **Azure Active Directory** and then select **Identity Governance**.

+1. In the left menu, select **Catalog** and then open the catalog.

+1. In the left menu, select **Resources** to see the list of resources in this catalog.

+ * Groups can be cloud-created Microsoft 365 Groups or cloud-created Azure AD security groups. Groups that originate in an on-premises Active Directory can't be assigned as resources because their owner or member attributes can't be changed in Azure AD. To give users access to an application that uses AD security group memberships, create a new group in Azure AD, configure [group writeback to AD](../hybrid/how-to-connect-group-writeback-v2.md), and [enable that group to be written to AD](../enterprise-users/groups-write-back-portal.md). Groups that originate in Exchange Online as Distribution groups can't be modified in Azure AD either.

+ * Applications can be Azure AD enterprise applications, which include both software as a service (SaaS) applications and your own applications integrated with Azure AD. If your application hasn't yet been integrated with Azure AD, see [govern access for applications in your environment](identity-governance-applications-prepare.md) and [integrate an application with Azure AD](identity-governance-applications-integrate.md).

+ * Sites can be SharePoint Online sites or SharePoint Online site collections.

+1. If you're an access package manager and you need to add resources to the catalog, you can ask the catalog owner to add them.

+A resource role is a collection of permissions associated with a resource. Resources can be made available for users to request if you add resource roles from each of the catalog's resources to your access package. You can add resource roles that are provided by groups, teams, applications, and SharePoint sites. When a user receives an assignment to an access package, they are added to all the resource roles in the access package.

+If you want some users to receive different roles than others, then you need to create multiple access packages in the catalog, with separate access packages for each of the resource roles. You can also mark the access packages as [incompatible](entitlement-management-access-package-incompatible.md) with each other so users can't request access to access packages that would give them excessive access.

+1. In the Azure portal, select **Azure Active Directory** and then select **Identity Governance**.

+1. In the left menu, select **Access packages** and then open the access package.

+1. In the left menu, select **Resource roles**.

+1. Select **Add resource roles** to open the Add resource roles to access package page.

+You can have entitlement management automatically add users to a group or a team in Microsoft Teams when they're assigned an access package.

+- When a user's access package assignment expires, they're removed from the group or team, unless they currently have an assignment to another access package that includes that same group or team.

+You can select any [Azure AD security group or Microsoft 365 Group](../fundamentals/active-directory-groups-create-azure-portal.md). Administrators can add any group to a catalog; catalog owners can add any group to the catalog if they're owner of the group. Keep the following Azure AD constraints in mind when selecting a group:

+- Azure AD can't change the membership of a group that was synchronized from Windows Server Active Directory using Azure AD Connect, or that was created in Exchange Online as a distribution group.

+- The membership of dynamic groups can't be updated by adding or removing a member, so dynamic group memberships aren't suitable for use with entitlement management.

+- Microsoft 365 groups have additional constraints, described in the [overview of Microsoft 365 Groups for administrators](/microsoft-365/admin/create-groups/office-365-groups), including a limit of 100 owners per group, limits on how many members can access Group conversations concurrently, and 7000 groups per member.

+1. On the **Add resource roles to access package** page, select **Groups and Teams** to open the Select groups pane.

+1. Select **Select**.

+ Once you select the group or team, the **Sub type** column lists one of the following subtypes:

+ | Microsoft 365 | Microsoft 365 Group that isn't Teams-enabled. Used for collaboration between users, both inside and outside your company. |

+1. Select **Add**.

+ Any users with existing assignments to the access package will automatically become members of this group or team when it's added.

+You can have Azure AD automatically assign users access to an Azure AD enterprise application, including both SaaS applications and your organization's applications integrated with Azure AD, when a user is assigned an access package. For applications that integrate with Azure AD through federated single sign-on, Azure AD issues federation tokens for users assigned to the application.

+Applications can have multiple app roles defined in their manifest. When you add an application to an access package, if that application has more than one app role, you need to specify the appropriate role for those users in each access package. If you're developing applications, you can read more about how those roles are added to your applications in [How to: Configure the role claim issued in the SAML token for enterprise applications](../develop/active-directory-enterprise-app-role-management.md).

+- When a user's access package assignment expires, their access is removed from the application, unless they have an assignment to another access package that includes that application role.

+- Applications may also have groups assigned to their app roles as well. You can choose to add a group in place of an application role in an access package, however then the application won't be visible to the user as part of the access package in the My Access portal.

+- Azure portal may also show service principals for services that can't be selected as applications. In particular, **Exchange Online** and **SharePoint Online** are services, not applications that have resource roles in the directory, so they can't be included in an access package. Instead, use group-based licensing to establish an appropriate license for a user who needs access to those services.

+- Applications that only support Personal Microsoft Account users for authentication, and don't support organizational accounts in your directory, don't have application roles and can't be added to access package catalogs.

+1. On the **Add resource roles to access package** page, select **Applications** to open the Select applications pane.

+1. Select **Select**.

+1. Select **Add**.

+ Any users with existing assignments to the access package will automatically be given access to this application when it's added.

+Azure AD can automatically assign users access to a SharePoint Online site or SharePoint Online site collection when they're assigned an access package.

+1. On the **Add resource roles to access package** page, select **SharePoint sites** to open the Select SharePoint Online sites pane.

+1. Select **Select**.

+1. Select **Add**.

+ Any users with existing assignments to the access package will automatically be given access to this SharePoint Online site when it's added.

+1. [List the accessPackageResources in the catalog](/graph/api/entitlementmanagement-list-accesspackagecatalogs?tabs=http&view=graph-rest-beta&preserve-view=true) and [create an accessPackageResourceRequest](/graph/api/entitlementmanagement-post-accesspackageresourcerequests?tabs=http&view=graph-rest-beta&preserve-view=true) for any resources that aren't yet in the catalog.

+1. In the Azure portal, select **Azure Active Directory** and then select **Identity Governance**.

+1. In the left menu, select **Access packages** and then open the access package.

+1. In the left menu, select **Resource roles**.

+1. Select the ellipsis (**...**) and then select **Remove resource role**.

+ Any users with existing assignments to the access package will automatically have their access revoked to this resource role when it's removed.

+In entitlement management, Azure AD processes bulk changes for assignment and resources in your access packages several times a day. So, if you make an assignment, or change the resource roles of your access package, it can take up to 24 hours for that change to be made in Azure AD, plus the amount of time it takes to propagate those changes to other Microsoft Online Services or connected SaaS applications. If your change affects just a few objects, the change will likely only take a few minutes to apply in Azure AD, after which other Azure AD components will then detect that change and update the SaaS applications. If your change affects thousands of objects, the change takes longer. For example, if you have an access package with 2 applications and 100 user assignments, and you decide to add a SharePoint site role to the access package, there may be a delay until all the users are part of that SharePoint site role. You can monitor the progress through the Azure AD audit log, the Azure AD provisioning log, and the SharePoint site audit logs.

+When you remove a member of a team, they're removed from the Microsoft 365 Group as well. Removal from the team's chat functionality might be delayed. For more information, see [Group membership](/microsoftteams/office-365-groups#group-membership).

+When a resource role is added to an access package by an admin, users who are in that resource role, but don't have assignments to the access package, will remain in the resource role, but won't be assigned to the access package. For example, if a user is a member of a group and then an access package is created and that group's member role is added to an access package, the user won't automatically receive an assignment to the access package.

+If you want the users to also be assigned to the access package, you can [directly assign users](entitlement-management-access-package-assignments.md#directly-assign-a-user) to an access package using the Azure portal, or in bulk via Graph or PowerShell. The users will then also receive access to the other resource roles in the access package. However, as those users already have access prior to being added to the access package, when their access package assignment is removed, they remain in the resource role. For example, if a user was a member of a group, and was assigned to an access package that included group membership for that group as a resource role, and then that user's access package assignment was removed, the user would retain their group membership.

+Most users in your directory can sign in to the My Access portal and automatically see a list of access packages they can request. However, for external business partner users that aren't yet in your directory, you'll need to send them a link that they can use to request an access package.

+1. In the Azure portal, select **Azure Active Directory** and then select **Identity Governance**.

+1. In the left menu, select **Access packages** and then open the access package.

+ It's important that you copy the entire My Access portal link when sending it to an internal business partner. This ensures that the partner gets access to your directory's portal to make their request. The link starts with `myaccess`, includes a directory hint, and ends with an access package ID. (For US Government, the domain in the My Access portal link will be `myaccess.microsoft.us`.)

+1. You may receive an email from Microsoft that asks you to review access. Locate the email to open the access review. Here's an example of an email requesting a review of access:

+1. Select the **Review access** link.

+1. Select **Access reviews** on the left navigation bar to see a list of pending access reviews assigned to you.

+1. Select the review that youΓÇÖd like to begin.

+1. Select **Yes** to keep your access or select **No** to remove your access.

+1. If you chose **Yes**, you may need to include a justification statement in the **Reason** box.

+1. Select **Submit**.

+A catalog is a container of resources and access packages. You create a catalog when you want to group related resources and access packages. An administrator can create a catalog. In addition, a user who has been delegated the [catalog creator](entitlement-management-delegate.md) role can create a catalog for resources that they own. A nonadministrator who creates the catalog becomes the first catalog owner. A catalog owner can add more users, groups of users, or application service principals as catalog owners.

+ Users see this information in an access package's details.

+1. Select **Select** to add these members.

+1. In the Azure portal, select **Azure Active Directory** and then select **Identity Governance**.

+1. In the left menu, in the **Entitlement management** section, select **Settings**.

+1. Select **Edit**.

+1. In the **Delegate entitlement management** section, select **Add catalog creators** to select the users or groups that you want to delegate this entitlement management role to.

+1. Select **Select**.

+1. Select **Save**.

+1. In the Azure portal, select **Azure Active Directory** and then select **Users**.

+1. In the left menu, select **User settings**.

+To retrieve a list of the users and groups assigned to the catalog creators role, the role with definition ID `ba92d953-d8e0-4e39-a797-0cbedb0a89e8`, use the Graph query

+To delegate the creation and management of access packages in a catalog, you add users to the access package manager role. Access package managers must be familiar with the need for users to request access to resources in a catalog. For example, if a catalog is used for a project, then a project lead might be an access package manager for that catalog. Access package managers can't add resources to a catalog, but they can manage the access packages and policies in a catalog. When delegating to an access package manager, that person can then be responsible for:

+- What roles a user has to the resources in a catalog

+- How long the project lasts

+1. In the Azure portal, select **Azure Active Directory** and then select **Identity Governance**.

+1. In the left menu, select **Catalogs** and then open the catalog you want to add administrators to.

+1. In the left menu, select **Roles and administrators**.

+1. Select **Add access package managers** to select the members for these roles.

+1. Select **Select** to add these members.

+1. In the Azure portal, select **Azure Active Directory** and then select **Identity Governance**.

+1. In the left menu, select **Catalogs** and then open the catalog you want to add administrators to.

+1. In the left menu, select **Roles and administrators**.

+1. Select **Remove**.

+As the IT administrator, Hana has contacts in each department--Mamta in Marketing, Mark in Finance, and Joe in Legal who are responsible for their department's resources and business critical content.

+Here's one way that Hana could delegate access governance to the marketing, finance, and legal departments.

+Entitlement management has the following roles, with permissions for administering entitlement management itself, that applies across all catalogs.

+For a role that is specific to a catalog, the `appScopeId` in the response indicates the catalog in which the user is assigned a role. This response only retrieves explicit assignments of that principal to role in entitlement management, it doesn't return results for a user who has access rights via a directory role, or through membership in a group assigned to a role.

+1. Select **Azure Active Directory** then select **Diagnostic settings** under Monitoring in the left navigation menu. Check if there's already a setting to send the audit logs to that workspace.

+1. If there isn't already a setting, select **Add diagnostic setting**. Use the instructions in [Integrate Azure AD logs with Azure Monitor logs](../reports-monitoring/howto-integrate-activity-logs-with-log-analytics.md#send-logs-to-azure-monitor) to send the Azure AD audit log to the Azure Monitor workspace.

+1. Select **Usage and estimated costs** and select **Data Retention**. Change the slider to the number of days you want to keep the data to meet your auditing requirements.

+ 1. Select **Azure Active Directory** then select **Workbooks**.

+ 1. Expand the section **Azure Active Directory Troubleshooting**, and select on **Archived Log Date Range**.

+1. In the Azure portal, select **Azure Active Directory** then select **Workbooks**. If you only have one subscription, move on to step 3.

+1. If you would like to see if there have been changes to application role assignments for an application that weren't due to access package assignments, such as by a global administrator directly assigning a user to an application role, then you can select the workbook named *Application role assignment activity*.

+1. In Azure Active Directory of the Azure portal, select **Logs** under the Monitoring section in the left navigation menu to create a new query page.

+1. Your workspace should be shown in the upper left of the query page. If you have multiple Azure Monitor workspaces, and the workspace you're using to store Azure AD audit events isn't shown, select **Select Scope**. Then, select the correct subscription and workspace.

+1. Then select **Run**.

+The table shows the Audit log events for entitlement management from the last hour by default. You can change the "Time range" setting to view older events. However, changing this setting will only show events that occurred after Azure AD was configured to send events to Azure Monitor.

+1. Then select **Add** to add a role assignment.

+ [Get-AzOperationalInsightsWorkspace](/powershell/module/Az.OperationalInsights/Get-AzOperationalInsightsWorkspace) operates in one subscription at a time. So, if you have multiple Azure subscriptions, you want to make sure you connect to the one that has the Log Analytics workspace with the Azure AD logs.

+| Expired | If no approvers approve a request within the approval request timeout, the request expires. To try again, the user has to resubmit their request. |

+| Access expired | User's access to the access package has expired. To get access again, the user has to submit a request. |

+The following table provides more detail about each of these email notifications. To manage these emails, you can use rules. For example, in Outlook, you can create rules to move the emails to a folder if the subject contains words from this table. The words are based on the default language settings of the tenant where the user is requesting access.

+| 2 | Action required: Approve or deny request by *[date]* | This email is sent to the first approver, if escalation is disabled, to take action. | First approver |

+| 3 | Reminder: Approve or deny the request by *[date]* for *[requestor]* | This reminder email is sent to the first approver, if escalation is disabled. The email asks them to take action if they haven't. | First approver |

+| 4 | Approve or deny the request by *[time]* on *[date]* | This email is sent to the first approver (if escalation is enabled) to take action. | First approver |

+| 5 | Action required reminder: Approve or deny the request by *[date]* for *[requestor]* | This reminder email is sent to the first approver, if escalation is enabled. The email asks them to take action if they haven't. | First approver |

+| 7 | Request approved for *[requestor]* to *[access_package]* | This email is sent to the first approver and stage-1 alternate approvers upon request completion. | First approver, stage-1 alternate approvers |

+| 8 | Request approved for *[requestor]* to *[access_package]* | This email is sent to the first approver and stage-1 alternate approvers of a multi-stage request when the stage-1 request is approved. | First approver, stage-1 alternate approvers |

+| 9 | Request denied to *[access_package]* | This email is sent to the requestor when their request is denied | Requestor |

+| 10 | Your request has expired for *[access_package]* | This email is sent to the requestor at the end of a single or multi-stage request. The email notifies the requestor that the request expired. | Requestor |

+| 11 | Action required: Approve or deny request by *[date]* | This email is sent to the second approver, if escalation is disabled, to take action. | Second approver |

+| 12 | Action required reminder: Approve or deny the request by *[date]* | This reminder email is sent to the second approver, if escalation is disabled. The notification asks them to take action if they haven't yet. | Second approver |

+| 13 | Action required: Approve or deny the request by *[date]* for *[requestor]* | This email is sent to second approver, if escalation is enabled, to take action. | Second approver |

+| 14 | Action required reminder: Approve or deny the request by *[date]* for *[requestor]* | This reminder email is sent to the second approver, if escalation is enabled. The notification asks them to take action if they haven't yet. | Second approver |

+| 15 | Action required: Approve or deny forwarded request by *[date]* | This email is sent to stage-2 alternate approvers, if escalation is enabled, to take action. | Stage-2 alternate approvers |

+| 16 | Request approved for *[requestor]* to *[access_package]* | This email is sent to the second approver and stage-2 alternate approvers upon approving the request. | Second approver, Stage-2 alternate approvers |

+| 18 | You now have access to *[access_package]* | This email is sent to the end users to start using their access. | Requestor |

+| 19 | Extend access for *[access_package]* by *[date]* | This email is sent to the end users before their access expires. | Requestor |

+When a requestor submits an access request for an access package configured to require approval, all approvers added to the policy receives an email notification with details of the request. The details in the email include: requestor's name organization, and business justification; and the requested access start and end date (if provided). The details will also include when the request was submitted and when the request will expire.

+The email includes a link approvers can select on to go to My Access to approve or deny the access request. Here's a sample email notification that is sent to an approver to complete an access request:

+Approvers can also receive a reminder email. The email asks the approver to make a decision on the request. Here's a sample email notification the approver receives to remind them to take action:

+If the alternate approvers setting is enabled and the request is still pending, it's forwarded. Alternate approvers receive an email to approve or deny the request. You can enable alternate approvers in stage-1 and stage-2. Here's a sample email of the notification the alternate approvers receive:

+ When an approver receives an access request submitted by a requestor, they can approve or deny the access request. The approver needs to add a business justification for their decision. Here's a sample email sent to the approvers

+When an access request is approved, and their access is provisioned, an email notification is sent to the requestor that they now have access to the access package. Here's a sample email notification that is sent to a requestor when they're granted access to an access package:

+When an access request is denied, an email notification is sent to the requestor. Here's a sample email notification that is sent to a requestor when their access request is denied:

+If multi-stage approval is enabled, at least one approver from each stage must approve the request, before the requestor can receive access.

+During stage-1, the first approver receives the access request email and makes a decision.

+After the first or alternate approvers approve the request in stage-1, stage-2 begins. During stage-2, the second approver receives the access request notification email. After the second approver or alternate approvers in stage-2 (if escalation is enabled) decide to approve or deny the request, notification emails are sent to the first and second approvers, and all alternate approvers in stage-1 and stage-2, as well as the requestor.

+Here's a sample email notification that is sent to a requestor when their access request has expired:

+The entitlement management reports and Azure AD audit log provide additional details about what resources users have access to. As an administrator, you can view the access packages and resource assignments for a user and view request logs for auditing purposes or determining the status of a user's request. This article describes how to use the entitlement management reports and Azure AD audit logs.

+1. Select **Azure Active Directory** and then select **Identity Governance**.

+1. In the left menu, select **Reports**.

+1. Select **Access packages for a user**.

+1. Select **Select users** to open the Select users pane.

+1. Find the user in the list and then select **Select**.

+1. If there are more than one resource roles or policies for an access package, select the resource roles or policies entry to see selection details.

+1. Select the **Assigned** tab to see a list of the access packages currently assigned to the user. When an access package is assigned to a user, it means that the user has access to all of the resource roles in the access package.

+This report enables you to list the resources currently assigned to a user in entitlement management. This report is for resources managed with entitlement management. The user might have access to other resources in your directory outside of entitlement management.

+1. Select **Azure Active Directory** and then select **Identity Governance**.

+1. In the left menu, select **Reports**.

+1. Select **Resource assignments for a user**.

+1. Select **Select users** to open the Select users pane.

+1. Find the user in the list and then select **Select**.

+ If a user got access to the same resource in two or more packages, you can select an arrow to see each package and policy.

+1. Select **Azure Active Directory** and then select **Audit logs**.

+1. Select **Apply**.

+1. To download the logs, select **Download**.

+Azure AD writes additional audit records while the request is in progress, including:

+| `EntitlementManagement` | `Auto approve access package assignment request` | Request doesn't require approval |

+| `EntitlementManagement` | `Ready to fulfill access package assignment request` |Request approved, or doesn't require approval |

+If access wasn't assigned, then Azure AD writes an audit record for the `EntitlementManagement` category with **Activity** either `Deny access package assignment request`, if the request was denied by an approver, or `Access package assignment request timed out (no approver action taken)`, if the request timed out before an approver could approve.

+Entitlement Management doesn't block outside updates to the access packageΓÇÖs resources, so the Entitlement Management UI wouldn't accurately display this change. Therefore, the userΓÇÖs assignment status would be shown as ΓÇ£DeliveredΓÇ¥ even though the user doesn't have access to the resources anymore. However, if the userΓÇÖs assignment is reprocessed, they'll be added to the access packageΓÇÖs resources again. Reprocessing ensures that the access package assignments are up to date, that users have access to necessary resources, and that assignments are accurately reflected in the UI.

+If you have users who are in the "Delivered" state but don't have access to resources that are a part of the access package, you'll likely need to reprocess the assignments to reassign those users to the access package's resources. Follow these steps to reprocess assignments for an existing access package:

+1. Select **Azure Active Directory**, and then select **Identity Governance**.

+1. In the left menu, select **Access packages** and then open the access package with the user assignment you want to reprocess.

+1. Underneath **Manage** on the left side, select **Assignments**.

+1. Select **Reprocess**.

+Scenario: In this scenario you learn how to use custom extensibility, and a Logic App, to automatically generate ServiceNow tickets for manual provisioning of users who have received assignments and need access to apps.

+>- **Starting with [Azure AD Connect 2.1.19.0](../hybrid/reference-connect-version-history.md#functional-changes-1), `employeeLeaveDateTime` is added to the default 'Out to Azure AD' rule, so steps 10-16 aren't required.**

+## How to verify these attribute values in Azure AD

+To review the values set on these properties on user objects in Azure AD, you can use the [Microsoft Graph PowerShell SDK](/powershell/microsoftgraph/installation?view=graph-powershell-1.0&preserve-view=true). For example:

+```PowerShell

+# Import Module

+Import-Module Microsoft.Graph.Users

+# Define the necessary scopes

+$Scopes =@("User.Read.All", "User-LifeCycleInfo.Read.All")

+# Connect using the scopes defined and select the Beta API Version

+Connect-MgGraph -Scopes $Scopes

+Select-MgProfile -Name beta

+# Query a user, using its user ID, and return the desired properties

+Get-MgUser -UserId "44198096-38ea-440d-9497-bb6b06bcaf9b" | Select-Object DisplayName, EmployeeLeaveDateTime

+```

+

+To link an Azure Logic App with a custom task extension, the following prerequisites must be available:

+- **No authorization** - With this choice no authorization will be granted, and you separately have to assign an application permission (LifecycleWorkflows.ReadWrite.All), or role assignment (Lifecycle Workflows Administrator). If an application is responding we don't recommend this option, as it isn't following the principle of least privilege. This option may also be used if responses are only provided on behalf of a user (LifecycleWorkflows.ReadWrite.All delegated permission AND Lifecycle Workflows Administrator role assignment)

+- **Existing application** - With this choice you're able to choose an existing application to respond. This can be a regular application and a system or user-assigned managed identity. For more information on managed identity types, see: [Managed identity types](../managed-identities-azure-resources/overview.md#managed-identity-types).

+Tasks within workflows can be added, edited, reordered, and removed at will. To edit the tasks of a workflow using the Azure portal, you complete the following steps:

+To edit the execution conditions of a workflow using the Azure portal, you do the following steps:

+1. On this screen, you're presented with **Trigger details**. Here we have a trigger type and attribute details. In the template you can edit the attribute details to define when a workflow is run in relation to the attribute value measured in days. This attribute value can be from 0 to 60 days.

+1. On this screen you can define rules for who the workflow runs. In the template **Scope type** is set as Rule-Based, and you define the rule using expressions on user properties. For more information on supported user properties. see: [supported queries on user properties](/graph/aad-advanced-queries#user-properties).

+1. On this page, you see a list of the workflow versions.

+- Follow the [Monitor changes to federation configuration](how-to-connect-monitor-federation-changes.md) to set up alerts to monitor changes to the trust established between your Idp and Azure AD.

+* If your proxy or firewall limit which URLs can be accessed, the URLs documented in [Office 365 URLs and IP address ranges](https://support.office.com/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2) must be opened. Also see [Safelist the Azure portal URLs on your firewall or proxy server](../../../azure-portal/azure-portal-safelist-urls.md).

+To enable the EnforceCloudPasswordPolicyForPasswordSyncedUsers feature, run the following command using the MSOnline PowerShell module as shown below. You would have to type yes for the Enable parameter as shown below:

+* Changing a group member will lead to a re-evaluation of all group members. For example, if you have a group with 50K members and you only update 1 member, this will trigger a synchronization of all 50K members.

+Azure AD Connect Health ADDS and ADFS Health Agents (version 3.2.2256.26, Download Center Only)

+You can download the latest version of Azure AD Connect 2.0 from the [Microsoft Download Center](https://www.microsoft.com/download/details.aspx?id=47594). See the [release notes for the latest V2.0 release](reference-connect-version-history.md#21200).\

+ - We fixed a bug where the new employeeLeaveDateTime attribute wasn't syncing correctly in version 2.1.19.0. Note that if the incorrect attribute was already used in a rule, then the rule must be updated with the new attribute and any objects in the AAD connector space that have the incorrect attribute must be removed with the "Remove-ADSyncCSObject" cmdlet, and then a full sync cycle must be run.

+ - We have removed the public preview functionality for the Admin Agent from Azure AD Connect. We won't provide this functionality going forward.

+ - We fixed a bug where the tooltip of the "Help" button isn't accessible through keyboard if navigated with arrow keys.

+ - We fixed a bug in Sync Service Manager's About dialog where the Screen reader isn't announcing the information about the data appearing under the "About" dialog box.

+ - We fixed a bug where the Management Agent Name wasn't mentioned in logs when an error occurred while validating MA Name.

+ - We fixed several accessibility issues with the keyboard navigation and custom control type fixes. The Tooltip of the "help" button isn't collapsing by pressing "Esc" key. There was an Illogical keyboard focus on the User Sign In radio buttons and there was an invalid control type on the help popups.

+ - Fixed an issue where some sync rule functions weren't parsing surrogate pairs properly.

+ - Fixed an issue where, under certain circumstances, the sync service wouldn't start due to a model db corruption. You can read more about the model db corruption issue in [this article](/troubleshoot/azure/active-directory/resolve-model-database-corruption-sqllocaldb)

+We fixed a bug that occurred when a domain was renamed and Password Hash Sync failed with an error that indicated "a specified cast isn't valid" in the Event log. This regression is from earlier builds.

+We fixed a bug that occurred when a domain was renamed and Password Hash Sync failed with an error that indicated "a specified cast isn't valid" in the Event log. This regression is from earlier builds.

+ Title: 'Decommissioning Azure AD Connect V1'

+description: This article describes Azure AD Connect V1 decommissioning and how to migrate to V2.

+documentationcenter: ''

+editor: ''

+ na

+# Decommission Azure AD Connect V1

+The one-year advanced notice of Azure AD Connect V1's retirement was announced in August 2021. As of August 31, 2022, all V1 versions went out of support and were subject to stop working unexpectedly at any point.

+On **October 1, 2023**, Azure AD cloud services will stop accepting connections from Azure AD Connect V1 servers, and identities will no longer synchronize.

+If you are still using Azure AD Connect V1 you must take action immediately.

+>[!IMPORTANT]

+>Azure AD Connect V1 will stop working on October 1st 2023. You need to migrate to cloud sync or connect sync V2.

+## Migrate to cloud sync

+Before moving to Azure AD Connect V2, you should see if cloud sync is right for you instead. Cloud sync uses a light-weight provisioning agent and is fully configurable through the portal. To choose the best sync tool for your situation, use the [Wizard to evaluate sync options.](https://aka.ms/EvaluateSyncOptions)

+Based on your environment and needs, you may qualify for moving to cloud sync. For a comparison of cloud sync and connect sync, see [Comparison between cloud sync and connect sync](cloud-sync/what-is-cloud-sync.md#comparison-between-azure-ad-connect-and-cloud-sync). To learn more, read [What is cloud sync?](cloud-sync/what-is-cloud-sync.md) and [What is the provisioning agent?](cloud-sync/what-is-provisioning-agent.md)

+## Migrating to Azure AD Connect V2

+If you aren't yet eligible to move to cloud sync, use this table for more information on migrating to V2.

+|Title|Description|

+|--|--|

+|[Information on deprecation](connect/deprecated-azure-ad-connect.md)|Information on Azure AD Connect V1 deprecation|

+|[What is Azure AD Connect V2?](connect/whatis-azure-ad-connect-v2.md)|Information on the latest version of Azure AD Connect|

+|[Upgrading from a previous version](connect/how-to-upgrade-previous-version.md)|Information on moving from one version of Azure AD Connect to another

+## Frequently asked questions

+## Next steps

+- [What is Azure AD Connect V2?](whatis-azure-ad-connect-v2.md)

+- [Azure AD Cloud Sync](../cloud-sync/what-is-cloud-sync.md)

+- [Azure AD Connect version history](reference-connect-version-history.md)

+- Cross-tenant synchronization is supported within the commercial cloud and Azure Government.

+- Cross-tenant synchronization isn't supported within the Azure China cloud.

+## Manage role settings using Microsoft Graph

+To manage role settings for groups using PIM APIs in Microsoft Graph, use the [unifiedRoleManagementPolicy resource type and its related methods](/graph/api/resources/unifiedrolemanagementpolicy).

+In Microsoft Graph, role settings are referred to as rules and they're assigned to groups through container policies. You can retrieve all policies that are scoped to a group and for each policy, retrieve the associated collection of rules by using an `$expand` query parameter. The syntax for the request is as follows:

+```http

+GET https://graph.microsoft.com/beta/policies/roleManagementPolicies?$filter=scopeId eq '{groupId}' and scopeType eq 'Group'&$expand=rules

+```

+For more information about managing role settings through PIM APIs in Microsoft Graph, see [Role settings and PIM](/graph/api/resources/privilegedidentitymanagement-for-groups-api-overview#policy-settings-in-pim-for-groups). For examples of updating rules, see [Update rules in PIM using Microsoft Graph](/graph/how-to-pim-update-rules).

+## Manage role settings using Microsoft Graph

+To manage settings for Azure AD roles using PIM APIs in Microsoft Graph, use the [unifiedRoleManagementPolicy resource type and related methods](/graph/api/resources/unifiedrolemanagementpolicy).

+In Microsoft Graph, role settings are referred to as rules and they're assigned to Azure AD roles through container policies. Each Azure AD role is assigned a specific policy object. You can retrieve all policies that are scoped to Azure AD roles and for each policy, retrieve the associated collection of rules by using an `$expand` query parameter. The syntax for the request is as follows:

+For more information about managing role settings through PIM APIs in Microsoft Graph, see [Role settings and PIM](/graph/api/resources/privilegedidentitymanagementv3-overview#role-settings-and-pim). For examples of updating rules, see [Update rules in PIM using Microsoft Graph](/graph/how-to-pim-update-rules).

+> [GitHub Enterprise Managed User (EMU)](https://docs.github.com/enterprise-cloud@latest/admin/authentication/managing-your-enterprise-users-with-your-identity-provider/about-enterprise-managed-users) is a different type of [GitHub Enteprise Account](https://docs.github.com/enterprise-cloud@latest/admin/overview/about-enterprise-accounts). If you haven't specifically requested EMU instance, you have a standard GitHub Enterprise Account. In that case, please refer to [the documentation](./github-provisioning-tutorial.md) to configure user provisioning in your non-EMU organisation. User provisioning is not supported for [standard GitHub Enteprise Accounts](https://docs.github.com/enterprise-cloud@latest/admin/overview/about-enterprise-accounts), but is supported for organisations under standard GitHub Enterprise Account.

+ b. In the **Logout URL** textbox, paste the value: `https://login.microsoftonline.com/common/wsfederation?wa=wsignout1.0`.

+A persistent volume represents a piece of storage provisioned for use with Kubernetes pods. You can use a persistent volume with one or many pods, and you can provision it dynamically or statically. This article shows you how to dynamically create persistent volumes with Azure Disks in an Azure Kubernetes Service (AKS) cluster.

+* Work with a static PV by creating one or more Azure managed disks or use an existing one and attach it to a pod.

+|resourceGroup | Specify the resource group for the Azure Disks | Existing resource group name | No | If empty, driver uses the same resource group name as current AKS cluster|

+|DiskIOPSReadWrite | [UltraSSD disk][ultra-ssd-disks] IOPS Capability (minimum: 2 IOPS/GiB) | 100~160000 | No | `500`|

+Storage classes define how a unit of storage is dynamically created with a persistent volume. For more information on Kubernetes storage classes, see [Kubernetes storage classes][kubernetes-storage-classes].

+Each AKS cluster includes four precreated storage classes, two of them configured to work with Azure Disks:

+ * Standard SSDs backs Standard storage and delivers cost-effective storage while still delivering reliable performance.

+2. The *managed-csi-premium* storage class provisions a premium Azure Disk.

+ * SSD-based high-performance, low-latency disks back Premium disks. They're ideal for VMs running production workloads. When you use the Azure Disk CSI driver on AKS, you can also use the `managed-csi` storage class, which is backed by Standard SSD locally redundant storage (LRS).

+You can see the precreated storage classes using the [`kubectl get sc`][kubectl-get] command. The following example shows the precreated storage classes available within an AKS cluster:

+A persistent volume claim (PVC) automatically provisions storage based on a storage class. In this case, a PVC can use one of the precreated storage classes to create a standard or premium Azure managed disk.

+1. Create a file named `azure-pvc.yaml` and copy in the following manifest. The claim requests a disk named `azure-managed-disk` that's *5 GB* in size with *ReadWriteOnce* access. The *managed-csi* storage class is specified as the storage class.

+2. Create the persistent volume claim using the [`kubectl apply`][kubectl-apply] command and specify your *azure-pvc.yaml* file.

+After you create the persistent volume claim, you must verify it has a status of `Pending`. The `Pending` status indicates it's ready to be used by a pod.

+1. Verify the status of the PVC using the `kubectl describe pvc` command.

+ ```bash

+ kubectl describe pvc azure-managed-disk

+ ```

+ The output of the command resembles the following condensed example:

+ ```output

+ Name: azure-managed-disk

+ Namespace: default

+ StorageClass: managed-csi

+ Status: Pending

+ [...]

+ ```

+2. Create a file named `azure-pvc-disk.yaml` and copy in the following manifest. This manifest creates a basic NGINX pod that uses the persistent volume claim named *azure-managed-disk* to mount the Azure Disk at the path `/mnt/azure`. For Windows Server containers, specify a *mountPath* using the Windows path convention, such as *'D:'*.

+3. Create the pod using the [`kubectl apply`][kubectl-apply] command.

+4. You now have a running pod with your Azure Disk mounted in the `/mnt/azure` directory. Check the pod configuration using the [`kubectl describe`][kubectl-describe] command.

+When you create an Azure disk for use with AKS, you can create the disk resource in the **node** resource group. This approach allows the AKS cluster to access and manage the disk resource. If you instead create the disk in a separate resource group, you must grant the Azure Kubernetes Service (AKS) managed identity for your cluster the `Contributor` role to the disk's resource group.

+1. Identify the resource group name using the [`az aks show`][az-aks-show] command and add the `--query nodeResourceGroup` parameter.

+ # Output

+2. Create a disk using the [`az disk create`][az-disk-create] command. Specify the node resource group name and a name for the disk resource, such as *myAKSDisk*. The following example creates a *20*GiB disk, and outputs the ID of the disk after it's created. If you need to create a disk for use with Windows Server containers, add the `--os-type windows` parameter to correctly format the disk.

+ The disk resource ID is displayed once the command has successfully completed, as shown in the following example output. You use the disk ID to mount the disk in the next section.

+1. Create a *pv-azuredisk.yaml* file with a *PersistentVolume*. Update `volumeHandle` with disk resource ID from the previous step.

+2. Create a *pvc-azuredisk.yaml* file with a *PersistentVolumeClaim* that uses the *PersistentVolume*.

+3. Create the *PersistentVolume* and *PersistentVolumeClaim* using the [`kubectl apply`][kubectl-apply] command and reference the two YAML files you created.

+4. Verify your *PersistentVolumeClaim* is created and bound to the *PersistentVolume* using the `kubectl get pvc` command.

+5. Create a *azure-disk-pod.yaml* file to reference your *PersistentVolumeClaim*.

+6. Apply the configuration and mount the volume using the [`kubectl apply`][kubectl-apply] command.

+## Clean up resources

+When you're done with the resources created in this article, you can remove them using the `kubectl delete` command.

+```bash

+# Remove the pod

+kubectl delete -f azure-pvc-disk.yaml

+# Remove the persistent volume claim

+kubectl delete -f azure-pvc.yaml

+```

+* To learn how to use CSI driver for Azure Disks storage, see [Use Azure Disks storage with CSI driver][azure-disks-storage-csi].

+* For associated best practices, see [Best practices for storage and backups in AKS][operator-best-practices-storage].

+ ```azurecli

+ # Install aks-preview

+ az extension add --name aks-preview

+ # Update aks-preview

+ az extension update --name aks-preview

+ ```

+ ```azurecli

+ az feature register --namespace Microsoft.ContainerService --name DisableWindowsOutboundNATPreview

+ ```

+ ```azurecli

+ az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/DisableWindowsOutboundNATPreview')].{Name:name,State:properties.state}"

+ ```

+ ```azurecli

+ az provider register --namespace Microsoft.ContainerService

+ ```

+Node Autodrain is a best effort service and cannot be guaranteed to operate perfectly in all scenarios

+* For information about command-line deployment, see [Enabling Command-line or Continuous Delivery of Azure WebJobs](https://azure.microsoft.com/blog/enabling-command-line-or-continuous-delivery-of-azure-webjobs/).

+> [Learn more about the WebJobs SDK](webjobs-sdk-how-to.md)

+az appconfig kv import --label test --content-type "application/vnd.microsoft.appconfig.keyvaultref+json;charset=utf-8" --name <your store name> --source file --path keyvault-refs.json --format json

+## June 2023

+Azure Active Directory for authentication and role-based access control are available across regions that support Azure Cache for Redis.

+ connection="<CONNECTION_SETTING>")

+ Title: How to accelerate your journey to FedRAMP compliance with Azure

+description: Provides an overview of resources for Development, Automation, and Advisory partners to help them accelerate their path to ATO with Azure.

+# FedRAMP compliance program overview

+Accelerating your path to the US Federal Risk and Authorization Management Program (FedRAMP) compliance in Azure is a focused program that provides learning resources and implementation tools. The goal of the program is education and support during the scoping and implementation of your project. Moreover, Microsoft works with key assessment and automation partners to share reference architectures and solutions that can help you meet your compliance needs.

+As a partner who provides a service in this field, you can publish your offering in the marketplace that expands the reach of your service.

+## Customers

+US Government agencies and many other organizations rely on commercial software companies to achieve their missions. FedRAMP was established to provide a standardized approach for assessing, monitoring, and authorizing cloud computing products and services. This approach uses a ΓÇ£do once, use many timesΓÇ¥ framework that saves cost, time, and resources required to conduct individual agency security assessments. FedRAMP is based on the National Institute of Standards and Technology (NIST) SP 800-53 standard, augmented by FedRAMP controls and control enhancements.

+- A Provisional Authority to Operate (P-ATO) issued by the FedRAMP Joint Authorization Board (JAB)

+- An agency Authority to Operate (ATO)

+A FedRAMP P-ATO is an initial approval of the cloud service provider (CSP) authorization package by the JAB. An agency can rely on P-ATO to grant an ATO for the acquisition and use of the cloud service within their agency. The JAB consists of the Chief Information Officers (CIOs) from the US Department of Defense (DoD), Department of Homeland Security (DHS), and General Services Administration (GSA), supported by designated technical representatives (TRs) from their respective member organizations. A P-ATO means that the JAB has reviewed the cloud serviceΓÇÖs authorization package and provided a provisional approval for federal agencies to use when granting an ATO for a cloud services offering.

+As part of the agency authorization process, a CSP works directly with the agency sponsor who reviews the cloud serviceΓÇÖs security package. After completing a security assessment, the head of an agency (or their designee) can grant an ATO.

+Consequently, an ISV can choose to go for a JAB authorization, which grants a generalized authorization to its solution and can be used with multiple agencies. This process tends to be longer. They can also choose to go for an agency ATO, which is specific to the Government customer they're serving. This customer acts as the sponsor and may even have ΓÇ£reciprocityΓÇ¥ with other agencies, which allows for a faster, smoother adoption of the companyΓÇÖs solution with a different customer.

+Microsoft is able to scale through its partners. Scale is what allows us to create a more predictable, cost-effective, and speedy delivery. These concerns are also common with pursuing an ATO. We're focused on enabling two main kinds of partnerships:

+- **Advisory:** enables partners to create offerings based on Azure that guide a customer through individual steps or the entire ATO process. These partners offer consulting services bundled with some automated solutions that add value to what Azure Compliance Launchpad provides. They can usually be contracted directly, by reference, or via Microsoft Azure Marketplace.

+ - Foundational partners, which enable integration of third party solutions with Azure and help you achieve / meet controls from your FedRAMP package. These partners are part of our recommended reference architectures.

+ - True automation partners that help automate certain aspects of the ATO journey such as the FedRAMP System Security Plan (SSP) generation, self-healing, alerts, and monitoring.

+> [!NOTE]

+> Partners are asked to publish their solutions to Azure Marketplace. See the following steps for guidance.

+1. Join the Partner Network ΓÇô ItΓÇÖs a requirement for publishing but easy to sign up. For instructions, see [Create a Partner Center account and enroll in the commercial marketplace](../../marketplace/create-account.md#create-a-partner-center-account-and-enroll-in-the-commercial-marketplace).

+2. Enable your partner center account as Publisher / Developer for Marketplace by following the instructions in [Create a commercial marketplace account in Partner Center](../../marketplace/create-account.md).

+3. With an enabled Partner Center Account, publish your listing as a SaaS application as explained in [Create a SaaS offer](../../marketplace/create-new-saas-offer.md).

+For a list of existing Azure Marketplace offerings in this space, visit [Azure Marketplace](https://aka.ms/azclmarketplace).

+## More resources

+> [!NOTE]

+> The information provided here will allow you to sign up and learn about the FedRAMP compliance program. The program is designed to help Azure and Azure Government customers successfully prepare their environments for authorization and request a FedRAMP ATO. This information does not constitute an offer of any kind, and submitting the following forms in no way guarantees participation in the program. Currently, the program details shared with partners and customers are notional and subject to change without notice.

+- [FedRAMP training resources](https://www.fedramp.gov/training/).

+- [FedRAMP documents and templates](https://www.fedramp.gov/documents-templates/) to help you with program requirements.

+- Get familiar with the [FedRAMP Marketplace](https://marketplace.fedramp.gov/#/products).

+- Learn more about [Azure Government compliance](../documentation-government-plan-compliance.md).

+Review the [Publishing guide by offer type](/partner-center/marketplace/publisher-guide-by-offer-type) for further tips and troubleshooting. If you're still facing issues, open a ticket in Partner Center.

+Before being able to apply for CSP or any other programs that run under the Microsoft Partner Network, you need to obtain a Microsoft Partner Network ID (MPN ID). For more information, visit the [Microsoft Partner Network](https://partner.microsoft.com/cloud-solution-provider/get-started) page. After you fulfill the basic requirement of becoming a Microsoft Partner, you're ready to become a Government CSP. To initiate your application, see [Sell cloud services to the US government](https://partner.microsoft.com/membership/cloud-solution-provider/cloud-for-government).

+[Azure Government](./documentation-government-welcome.md) is a physically isolated instance of Azure that delivers services with world-class security and compliance critical to the US government. These services maintain FedRAMP and DoD authorizations, CJIS state-level agreements, support for IRS 1075, ability to sign a HIPAA Business Associate Agreement, and others. For more information, see [Azure Government compliance](./documentation-government-plan-compliance.md). Azure Government provides an extra layer of protection to customers through contractual commitments regarding storage of customer data in the United States. Moreover, Azure Government limits potential access to systems processing customer data to screened US persons. Most US government agencies and their partners are best aligned with Azure Government.

+The process begins with a request for an Azure Government tenant. For more information and to begin the validation, see [Microsoft Azure Government trial](https://azure.microsoft.com/global-infrastructure/government/request/?ReqType=CSP). Once complete, you should receive a tenant to activate your enrollment in the Cloud Solution Provider program for the US government. The following validation steps are in place:

+- Be enrolled in the [Microsoft Cloud Partner Program](/partner-center/mpn-overview) (have an MCPP ID).

+- Verification of business engagements with government customers (for example, proof of services rendered to government agencies, statements of work, evidence of being part of GSA Schedule).

+Once you have the credentials described previously, navigate to [Partner Center for Microsoft US Government Cloud](/partner-center/partner-center-for-microsoft-us-govt-cloud) to apply for the CSP Government reseller program. It takes 5-6 days to process the application, and, once approved, you should receive an email to log in to [Partner Center](https://partner.microsoft.com/dashboard/home) to accept the Terms and Conditions. For more information, see [Partner Center documentation](/partner-center/overview).

+> Terms and Conditions aren't negotiable for the Cloud Solution Provider program. If you wish to discuss customer terms that you have in place for your Commercial agreement, contact your Microsoft account representative.

+- Verification of an active enrollment in the Advanced Support for Partners program or Premier Support for Partners program. More information is available from [Compare partner support plans](https://partner.microsoft.com/support/partnersupport).

+After the validation has been completed and terms have been signed, you're ready to transact. For more information on billing, see [Azure plan](/partner-center/azure-plan-lp).

+- If you're still unclear about CSP or are looking to apply for the commercial side of the program, see [Enroll in the CSP program](/partner-center/enrolling-in-the-csp-program).

+- If you're interested in Office 365 GCC for CSP, which is transacted via the CSP for Commercial platform, see [Sell Office 365 Government GCC for CSP subscriptions to qualified customers](/partner-center/csp-gcc-overview).

+Once you have onboarded and are ready to create your first customer, make sure to review [Resources for building your Government CSP practice](https://devblogs.microsoft.com/azuregov/resources-for-building-your-government-csp-practice/). To review further documentation, visit the [FAQ](/partner-center/faq-for-us-govt-cloud). For all other questions, open a ticket in Partner Center.

+To help you navigate export control rules, Microsoft has published the [Microsoft Azure Export Controls](https://aka.ms/Azure-Export-Paper) whitepaper. It describes US export controls particularly as they apply to software and technical data, reviews potential sources of export control risks, and offers specific guidance to help you assess your obligations under these controls. Extra information is available from the Cloud Export FAQ, which is accessible from Frequently Asked Questions on [Exporting Microsoft products](https://www.microsoft.com/exporting).

+- [Exporting Microsoft products](https://www.microsoft.com/exporting)

+- [Azure FedRAMP compliance offering](/azure/compliance/offerings/offering-fedramp)

+| | Azure Stack HCI | X | | |

+| Azure Stack HCI | X | X | |

+| May 2023 | **Windows** <ul><li>Enable Large Event support for all regions.</li><li>Update to TroubleShooter 1.4.0.</li><li>Fixed issue when Event Log subscription become invalid; will resubscribe.</li><li>AMA: Fixed issue with Large Event sending too large data. Also affecting Custom Log.</li></ul> **Linux** <ul><li>Support for CIS and SELinux [hardening](https://learn.microsoft.com/azure/azure-monitor/agents/agents-overview#linux-hardening-standards)</li><li>Include Ubuntu 22.04 (jammy) in azure-mdsd package publishing</li><li>Move storage SDK patch to build container</li><li>Add system telegraf counters to AMA</li><li>Drop msgpack and syslog data if not configured in active configuration</li><li>Limit the events sent to Public ingestion pipeline</li><li>**Fixes** <ul><li>Fix mdsd crash in init when in persistent mode </li><li>Remove FdClosers from ProtocolListeners to avoid a race condition</li><li>Fix sed regex special character escaping issue in rpm macro for Centos 7.3.Maipo</li><li>Fix latency and future timestamp issue for 3P</li><li>Install AMA syslog configs only if customer is opted in for syslog in DCR</li><li>Fix heartbeat time check</li><li>Skip unnecessary cleanup in fatal signal handler</li><li>Fix case where fast-forwarding may cause intervals to be skipped</li><li>Fix comma separated custom log paths with fluent</li></ul></li><ul> | 1.16.0 | 1.26.2 |

+| Apr 2023 | **Windows** <ul><li>AMA: Enable Large Event support based on Region.</li><li>AMA: Upgrade to FluentBit version 2.0.9</li><li>Update Troubleshooter to 1.3.1</li><li>Update ME version to 2.2023.331.1521</li><li>Updating package version for AzSecPack 4.26 release</li></ul>|1.15.0.0| Coming soon|

+| Oct 2022 | **Windows** <ul><li>Increased reliability of data uploads</li><li>Data quality improvements</li></ul> **Linux** <ul><li>Support for `http_proxy` and `https_proxy` environment variables for [network proxy configurations](./azure-monitor-agent-data-collection-endpoint.md#proxy-configuration) for the agent</li><li>[Text logs](./data-collection-text-log.md) <ul><li>Network proxy support enabled</li><li>Fixed missing `_ResourceId`</li><li>Increased maximum line size support to 1MB</li></ul></li><li>Support ingestion of syslog events whose timestamp is in the future</li><li>Performance improvements</li><li>Fixed `diskio` metrics instance name dimension to use the disk mount path(s) instead of the device name(s)</li><li>Fixed world writable file issue to lock down write access to certain agent logs and configuration files stored locally on the machine</li></ul> | 1.10.0.0 | 1.24.2 |

+| June 2022 | Bug fixes with user assigned identity support, and reliability improvements | 1.6.0.0 | None |

+| February 2022 | <ul><li>Bug fixes for the AMA Client installer (private preview)</li><li>Versioning fix to reflect appropriate Windows major/minor/hotfix versions</li><li>Internal test improvement on Linux</li></ul> | 1.2.0.0 | 1.15.3 |

+3. Run the Troubleshooter: > AgentTroubleshooter --ama

+ - **Value:** "Evaluation windowStartTime: \${data.context.condition.windowStartTime}. windowEndTime: \${data.context.condition.windowEndTime}"

+ - **Value:** "\${data.context.condition.allOf[0].metricName} \${data.context.condition.allOf[0].operator} \${data.context.condition.allOf[0].threshold} \${data.essentials.monitorCondition}. The value is \${data.context.condition.allOf[0].metricValue}"

+> The `"Severity"` field value changes if you've [switched to the current scheduledQueryRules API](./alerts-log-api-switch.md) from the [legacy Log Analytics Alert API](./api-alerts.md).

+> The `ScheduledQueryRules` PowerShell cmdlets can only manage rules created in [this version of the Scheduled Query Rules API](/rest/api/monitor/scheduledqueryrule-2018-04-16/scheduled-query-rules). Log alert rules created by using the legacy [Log Analytics Alert API](./api-alerts.md) can only be managed by using PowerShell after you [switch to the Scheduled Query Rules API](./alerts-log-api-switch.md).

+ **Startup Command** doesn't honor `JAVA_OPTS` for JavaSE or `CATALINA_OPTS` for Tomcat.

+ If you don't use **Startup Command**, create a new environment variable, `JAVA_OPTS` for JavaSE or `CATALINA_OPTS` for Tomcat, with the value

+> If you set the `JAVA_OPTS` for JavaSE or `CATALINA_OPTS` for Tomcat environment variable, you will have to disable Application Insights in the portal. Alternatively, if you prefer to enable Application Insights from the portal, make sure that you don't set the `JAVA_OPTS` for JavaSE or `CATALINA_OPTS` for Tomcat variable in App Service configurations settings.

+> Using both a connection string and instrumentation key isn't recommended. Whichever was set last takes precedence. Also, using both could lead to [missing data](#missing-data).

+ var telemetryProcessorChainBuilder = telemetryConfiguration.DefaultTelemetrySink.TelemetryProcessorChainBuilder;

+ // Using adaptive sampling

+ telemetryProcessorChainBuilder.UseAdaptiveSampling(maxTelemetryItemsPerSecond: 5);

+ // Alternately, the following configures adaptive sampling with 5 items per second, and also excludes DependencyTelemetry from being subject to sampling:

+ // telemetryProcessorChainBuilder.UseAdaptiveSampling(maxTelemetryItemsPerSecond:5, excludedTypes: "Dependency");

+ EnableAdaptiveSampling = false,

+using Microsoft.ApplicationInsights.AspNetCore.Extensions;

+var builder = WebApplication.CreateBuilder(args);

+builder.Services.Configure<TelemetryConfiguration>(telemetryConfiguration =>

+ var telemetryProcessorChainBuilder = telemetryConfiguration.DefaultTelemetrySink.TelemetryProcessorChainBuilder;

+ // Using adaptive sampling

+ telemetryProcessorChainBuilder.UseAdaptiveSampling(maxTelemetryItemsPerSecond: 5);

+ // Alternately, the following configures adaptive sampling with 5 items per second, and also excludes DependencyTelemetry from being subject to sampling:

+ // telemetryProcessorChainBuilder.UseAdaptiveSampling(maxTelemetryItemsPerSecond:5, excludedTypes: "Dependency");

+});

+builder.Services.AddApplicationInsightsTelemetry(new ApplicationInsightsServiceOptions

+{

+ EnableAdaptiveSampling = false,

+});

+var app = builder.Build();

+## Data considerations

+Data stored in the Azure Monitor Workspace is handled in accordance with all standards described in the [Azure Trust Center](https://www.microsoft.com/en-us/trust-center?rtc=1). Several considerations exist specific to data stored in the Azure Monitor Workspace:

+- Data is physically stored in the same region that the Azure Monitor Workspace is provisioned in

+- Data is encrypted at rest using a Microsoft-managed key

+- Data is retained for 18 months

+- For details about the Azure Monitor managed service for Prometheus' support of PII/EUII data, please see details [here](./prometheus-metrics-overview.md)

+>

+> Azure Managed Prometheus support starts from KEDA v2.10. If you have an older version of KEDA installed, you must upgrade in order to work with Azure Managed Prometheus.

+Date list was last updated: 06/01/2023.

+<!--Gen Date: Thu Jun 01 2023 09:57:38 GMT+0300 (Israel Daylight Time)-->

+ - `kube_pod_container_status_waiting_reason`

+ - `kube_node_status_allocatable`

+ - `kube_pod_owner`

+ - `kube_replicaset_owner`

+ - `kube_node_status_capacity`

+ - `kube_pod_info`

+ - `kube_statefulset_status_replicas_ready`

+ - `kube_statefulset_status_replicas`

+ - `kube_statefulset_status_replicas_updated`

+ - `kube_job_status_start_time`

+ - `kube_job_failed`

+ - `kube_horizontalpodautoscaler_status_desired_replicas`

+ - `kube_horizontalpodautoscaler_status_current_replicas`

+ - `kube_horizontalpodautoscaler_spec_min_replicas`

+ - `kube_horizontalpodautoscaler_spec_max_replicas`

+ - `kube_node_status_condition`

+ - `kube_node_spec_taint`

+In the Azure portal, navigate to your Azure Monitor Workspace. Go to `Metrics` and verify that the metrics `Active Time Series % Utilization` and `Events Per Minute Ingested % Utilization` are below 100%.

+<!--Gen Date: Thu Jun 01 2023 09:57:38 GMT+0300 (Israel Daylight Time)-->

+ | Communication Services | [ACSCallAutomationIncomingOperations](/azure/azure-monitor/reference/tables/ACSCallAutomationIncomingOperations)<br>[ACSCallAutomationMediaSummary](/azure/azure-monitor/reference/tables/ACSCallAutomationMediaSummary)<br>[ACSCallRecordingIncomingOperations](/azure/azure-monitor/reference/tables/ACSCallRecordingIncomingOperations)<br>[ACSCallRecordingSummary](/azure/azure-monitor/reference/tables/ACSCallRecordingSummary)<br>[ACSRoomsIncomingOperations](/azure/azure-monitor/reference/tables/acsroomsincomingoperations) |

+ | Data Transfer | [DataTransferOperations](/azure/azure-monitor/reference/tables/DataTransferOperations) |

+ :::image type="content" source="./media/private-link-security/ampls-find-1c.png" lightbox="./media/private-link-security/ampls-find-1c.png" alt-text="Screenshot showing finding Azure Monitor Private Link Scope.":::

+Connect Azure Monitor resources like Log Analytics workspaces, Application Insights components, and [data collection endpoints](../essentials/data-collection-endpoint-overview.md)) to your Azure Monitor Private Link Scope (AMPLS).

+ :::image type="content" source="./media/private-link-security/ampls-select-private-endpoint-connect-3.png" lightbox="./media/private-link-security/ampls-select-private-endpoint-connect-3.png" alt-text="Screenshot that shows Private Endpoint connections.":::

+1. On the **Basics** tab, select the **Subscription** and **Resource group**

+1. Enter the **Name** of the endpoint, and **Network Interface Name**

+1. Select the **Region** the private endpoint should live in. The region must be the same region as the virtual network to which you connect it.

+ :::image type="content" source="./media/private-link-security/create-private-endpoint-basics.png" alt-text="A screenshot showing the create private endpoint basics tab." lightbox="./media/private-link-security/create-private-endpoint-basics.png":::

+1. On the **Resource** tab,select the *Subscription* that contains your Azure Monitor Private Link Scope resource.

+1. For **Resource type**, select *Microsoft.insights/privateLinkScopes*.

+1. From the **Resource** dropdown, select the Private Link Scope you created earlier.

+1. Select **Next: Virtual Network**.

+ :::image type="content" source="./media/private-link-security/create-private-endpoint-resource.png" alt-text="Screenshot that shows the Create a private endpoint page in the Azure portal with the Resource tab selected." lightbox="./media/private-link-security/create-private-endpoint-resource.png":::

+1. On the **Virtual Network** tab, select the **Virtual network** and **Subnet** that you want to connect to your Azure Monitor resources.

+1. For **Network policy for private endpoints**, select **edit** if you want to apply network security groups or Route tables to the subnet that contains the private endpoint.

+ In **Edit subnet network policy**, select the checkboxes next to **Network security groups** and **Route tables**, and select **Save**. For more information, see [Manage network policies for private endpoints](../../private-link/disable-private-endpoint-network-policy.md).

+1. For **Private IP configuration**, by default, **Dynamically allocate IP address** is selected. If you want to assign a static IP address, select **Statically allocate IP address**. Then enter a name and private IP.

+ Optionally, you can select or create an **Application security group**. You can use application security groups to group virtual machines and define network security policies based on those groups.

+1. Select **Next: DNS**.

+ :::image type="content" source="./media/private-link-security/create-private-endpoint-virtual-network.png" alt-text="Screenshot that shows the Create a private endpoint page in the Azure portal with the Virtual Network tab selected." lightbox="./media/private-link-security/create-private-endpoint-virtual-network.png":::

+1. On the **DNS** tab, select **Yes** for **Integrate with private DNS zone**, and let it automatically create a new private DNS zone. The actual DNS zones might be different from what's shown in the following screenshot.

+ > [!NOTE]

+ > If you select **No** and prefer to manage DNS records manually, first finish setting up your private link. Include this private endpoint and the AMPLS configuration. Then, configure your DNS according to the instructions in [Azure private endpoint DNS configuration](../../private-link/private-endpoint-dns.md). Make sure not to create empty records as preparation for your private link setup. The DNS records you create can override existing settings and affect your connectivity with Azure Monitor.

+1. Select **Next: Tags**, then select **Review + create**.

+ :::image type="content" source="./media/private-link-security/create-private-endpoint-dns.png" alt-text="Screenshot that shows the Create a private endpoint page in the Azure portal with the DNS tab selected." lightbox="./media/private-link-security/create-private-endpoint-dns.png":::

+1. On the **Review + create** , once the validation passes select **Create**.

+## AzAcSnap command won't run

+In some cases AzAcSnap won't start due to the user's environment.

+### Failed to create CoreCLR

+AzAcSnap is written in .NET and the CoreCLR is an execution engine for .NET apps, performing functions such as IL byte code loading, compilation to machine code and garbage collection. In this case there is an environmental problem blocking the CoreCLR engine from starting.

+A common cause is limited permissions or environmental setup for the AzAcSnap operating system user, usually 'azacsnap'.

+The error `Failed to create CoreCLR, HRESULT: 0x80004005` can be caused by lack of write access for the azacsnap user to the system's `TMPDIR`.

+> [!NOTE]

+> All command lines starting with `#` are commands run as `root`, all command lines starting with `>` are run as `azacsnap` user.

+Check the `/tmp` ownership and permissions (note in this example only the `root` user can read and write to `/tmp`):

+```bash

+# ls -ld /tmp

+drwx 9 root root 8192 Mar 31 10:50 /tmp

+```

+A typical `/tmp` has the following permissions, which would allow the azacsnap user to run the azacsnap command:

+```bash

+# ls -ld /tmp

+drwxrwxrwt 9 root root 8192 Mar 31 10:51 /tmp

+```

+If it's not possible to change the `/tmp` directory permissions, then create a user specific `TMPDIR`.

+

+Make a `TMPDIR` for the `azacsnap` user:

+```bash

+> mkdir /home/azacsnap/_tmp

+> export TMPDIR=/home/azacsnap/_tmp

+> azacsnap -c about

+```

+```output

+

+

+ WKO0XXXXXXXXXXXNW

+ Wk,.,oxxxxxxxxxxx0W

+ 0;.'.;dxxxxxxxxxxxKW

+ Xl'''.'cdxxxxxxxxxdkX

+ Wx,''''.,lxxxxdxdddddON

+ 0:''''''.;oxdddddddddxKW

+ Xl''''''''':dddddddddddkX

+ Wx,''''''''':ddddddddddddON

+ O:''''''''',xKxddddddoddod0W

+ Xl''''''''''oNW0dooooooooooxX

+ Wx,,,,,,'','c0WWNkoooooooooookN

+ WO:',,,,,,,,;cxxxxooooooooooooo0W

+ Xl,,,,,,,;;;;;;;;;;:llooooooooldX

+ Nx,,,,,,,,,,:c;;;;;;;;coooollllllkN

+ WO:,,,,,,,,,;kXkl:;;;;,;lolllllllloOW

+ Xl,,,,,,,,,,dN WNOl:;;;;:lllllllllldK

+ 0c,;;;;,,,;lK NOo:;;:clllllllllo0W

+ WK000000000N NK000KKKKKKKKKKXW

+

+

+ Azure Application Consistent Snapshot Tool

+ AzAcSnap 7a (Build: 1AA8343)

+```

+> [!IMPORTANT]

+> Changing the user's `TMPDIR` would need to be made permanent by changing the user's profile (e.g. `$HOME/.bashrc` or `$HOME/.bash_profile`). There would also be a need to clean-up the `TMPDIR` on system reboot, this is typically automatic for `/tmp`.

+```bash

+hdbsql -n 172.18.18.50 -i 00 -U AZACSNAP "select version from sys.m_database"

+```

+```output

+* -10104: Invalid value for KEY (AZACSNAP)

+```

+

+ >[!IMPORTANT]

+ >You should enable Continuous Availability for Citrix App Layering, SQL Server, and [FSLogix user profile containers](../virtual-desktop/create-fslogix-profile-container.md). Using SMB Continuous Availability shares for workloads other than Citrix App Layering, SQL Server, and FSLogix user profile containers is *not* supported. This feature is currently supported on Windows SQL Server. Linux SQL Server is not currently supported. If you are using a non-administrator (domain) account to install SQL Server, ensure that the account has the required security privilege assigned. If the domain account does not have the required security privilege (`SeSecurityPrivilege`), and the privilege cannot be set at the domain level, you can grant the privilege to the account by using the **Security privilege users** field of Active Directory connections. See [Create an Active Directory connection](create-active-directory-connections.md#create-an-active-directory-connection).

+* [Oracle On Azure IaaS Recommended Practices For Success](https://github.com/Azure/Oracle-Workloads-for-Azure/blob/main/Oracle%20on%20Azure%20IaaS%20Recommended%20Practices%20for%20Success.pdf)

+* [Oracle Databases on Microsoft Azure Using Azure NetApp Files](https://www.netapp.com/media/17105-tr4780.pdf)

+> You should enable Continuous Availability for [Citrix App Layering](https://docs.citrix.com/en-us/citrix-app-layering/4.html), SQL Server, and [FSLogix user profile containers](../virtual-desktop/create-fslogix-profile-container.md). Using SMB Continuous Availability shares for any other workload is not supported. This feature is currently supported on Windows SQL Server. Linux SQL Server is not currently supported.

+2. Select the SMB volume that you want to have SMB CA enabled. Then select **Edit**.

+ Title: Integrate - Build a real-time collaborative whiteboard using Azure Web PubSub and deploy it to Azure App Service

+description: A how-to guide about how to use Azure Web PubSub to enable real-time collaboration on a digital whiteboard and deploy as a Web App using Azure App Service

+# How-to: build a real-time collaborative whiteboard using Azure Web PubSub and deploy it to Azure App Service

+A new class of applications is reimagining what modern work could be. While [Microsoft Word](https://www.microsoft.com/microsoft-365/word) brings editors together, [Figma](https://www.figma.com) gathers up designers on the same creative endeavor. This class of applications builds on a user experience that makes us feel connected with our remote collaborators. From a technical point of view, user's activities need to be synchronized across users' screens at a low latency.

+## Overview

+In this how-to guide, we take a cloud-native approach and use Azure services to build a real-time collaborative whiteboard and we deploy the project as a Web App to Azure App Service. The whiteboard app is accessible in the browser and allows anyone can draw on the same canvas.

+> [!div class="nextstepaction"]

+> [Check out live whiteboard demo](https://azure.github.io/azure-webpubsub/demos/whiteboard)

+### Architecture

+|Azure service name | Purpose | Benefits |

+|-|-||

+|[Azure App Service](https://learn.microsoft.com/azure/app-service/) | Provides the hosting environment for the backend application, which is built with [Express](https://expressjs.com/) | Fully managed environment for application backends, with no need to worry about infrastructure where the code runs

+|[Azure Web PubSub](https://learn.microsoft.com/azure/azure-web-pubsub/overview) | Provides low-latency, bi-directional data exchange channel between the backend application and clients | Drastically reduces server load by freeing server from managing persistent WebSocket connections and scales to 100 K concurrent client connections with just one resource

+## Prerequisites

+You can find detailed explanation of the [data flow](#data-flow) at the end of this how-to guide as we're going to focus on building and deploying the whiteboard app first.

+

+In order to follow the step-by-step guide, you need

+> [!div class="checklist"]

+> * An [Azure](https://portal.azure.com/) account. If you don't have an Azure subscription, create an [Azure free account](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio) before you begin.

+> * [Azure CLI](/cli/azure/install-azure-cli) (version 2.29.0 or higher) or [Azure Cloud Shell](../cloud-shell/quickstart.md) to manage Azure resources.

+## Create Azure resources using Azure CLI

+### 1. Sign in

+1. Sign in to Azure CLI by running the following command.

+ ```azurecli-interactive

+ az login

+ ```

+1. Create a resource group on Azure.

+ ```azurecli-interactive

+ az group create \

+ --location "westus" \

+ --name "whiteboard-group"

+ ```

+### 2. Create a Web App resource

+1. Create a free App Service plan.

+ ```azurecli-interactive

+ az appservice plan create \

+ --resource-group "whiteboard-group" \

+ --name "demo" \

+ --sku FREE

+ --is-linux

+ ```

+1. Create a Web App resource

+ ```azurecli-interactive

+ az webapp create \

+ --resource-group "whiteboard-group" \

+ --name "whiteboard-app" \

+ --plan "demo" \

+ --runtime "NODE:18-lts"

+ ```

+### 3. Create a Web PubSub resource

+1. Create a Web PubSub resource.

+ ```azurecli-interactive

+ az webpubsub create \

+ --name "whiteboard-app" \

+ --resource-group "whiteboard-group" \

+ --location "westus" \

+ --sku Free_F1

+ ```

+1. Show and store the value of `primaryConnectionString` somewhere for later use.

+ ```azurecli-interactive

+ az webpubsub key show \

+ --name "whiteboard-app" \

+ --resource-group "whiteboard-group"

+ ```

+## Get the application code

+Run the following command to get a copy of the application code. You can find detailed explanation of the [data flow](#data-flow) at the end of this how-to guide.

+```bash

+git clone https://github.com/Azure/awps-webapp-sample.git

+```

+## Deploy the application to App Service

+1. App Service supports many deployment workflows. For this guide, we're going to deploy a ZIP package. Run the following commands to prepare the ZIP.

+ ```bash

+ npm install

+ npm run build

+ zip -r app.zip *

+ ```

+2. Use the following command to deploy it to Azure App Service.

+ ```azurecli-interactive

+ az webapp deployment source config-zip \

+ --resource-group "whiteboard-group" \

+ --name "whiteboard-app" \

+ --src app.zip

+ ```

+3. Set Azure Web PubSub connection string in the application settings. Use the value of `primaryConnectionString` you stored from an earlier step.

+ ```azurecli-interactive

+ az webapp config appsettings set \

+ --resource-group "whiteboard-group" \

+ --name "whiteboard-app" \

+ --setting Web_PubSub_ConnectionString="<primaryConnectionString>"

+ ```

+## Configure upstream server to handle events coming from Web PubSub

+Whenever a client sends a message to Web PubSub service, the service sends an HTTP request to an endpoint you specify. This mechanism is what your backend server uses to further process messages, for example, if you can persist messages in a database of choice.

+As is with HTTP requests, Web PubSub service needs to know where to locate your application server. Since the backend application is now deployed to App Service, we get a publically accessible domain name for it.

+1. Show and store the value of `name` somewhere.

+ ```azurecli-interactive

+ az webapp config hostname list \

+ --resource-group "whiteboard-group"

+ --webapp-name "whiteboard-app"

+ ```

+1. The endpoint we decided to expose on the backend server is [`/eventhandler`](https://github.com/Azure/awps-webapp-sample/blob/main/whiteboard/server.js#L17) and the `hub` name for whiteboard app [`"sample_draw"`](https://github.com/Azure/awps-webapp-sample/blob/main/whiteboard/server.js#l14)

+ ```azurecli-interactive

+ az webpubsub hub create \

+ --resource-group "whiteboard-group" \

+ --name "whiteboard-app" \

+ --hub-name "sample_draw" \

+ --event-handler url-template="https://<Replace with the hostname of your Web App resource>/eventhandler" user-event-pattern="*" system-event="connected" system-event="disconnected"

+ ```

+> [!IMPORTANT]

+> `url-template` has three parts: protocol + hostname + path, which in our case is `https://<The hostname of your Web App resource>/eventhandler`.

+## View the whiteboard app in a browser

+Now head over to your browser and visit your deployed Web App. It's recommended to have multiple browser tabs open so that you can experience the real-time collaborative aspect of the app. Or better, share the link with a colleague or friend.

+## Data flow

+### Overview

+The data flow section dives deeper into how the whiteboard app is built. The whiteboard app has two transport methods.

+- HTTP service written as an Express app and hosted on App Service.

+- WebSocket connections managed by Azure Web PubSub.

+By using Azure Web PubSub to manage WebSocket connections, the load on the Web App is reduced. Apart from authenticating the client and serving images, the Web App isn't involved synchronizing drawing activities. A client's drawing activities are directly sent to Web PubSub and broadcasted to all clients in a group.

+At any point in time, there maybe more than one client drawing. If the Web App were to manage WebSocket connections on its own, it needed to broadcast every drawing activity to all other clients. The huge traffic and processing are a large burden to the server.

+ :::column:::

+ The client, built with [Vue](https://vuejs.org/), makes an HTTP request for a Client Access Token to an endpoint `/negotiate`. The backend application is an [Express app](https://expressjs.com/) and hosted as a Web App using Azure App Service.

+ :::column-end:::

+ :::column:::

+ :::image type="content" source="./media/howto-integrate-app-service/dataflow-1.jpg" alt-text="Screenshot of step one of app data flow." lightbox="./media/howto-integrate-app-service/dataflow-1.jpg":::

+ :::column-end:::

+ :::column:::

+ When the backend application successfully [returns the Client Access Token](https://github.com/Azure/awps-webapp-sample/blob/main/whiteboard/server.js#L62) to the connecting client, the client uses it to establish a WebSocket connection with Azure Web PubSub.

+ :::column-end:::

+ :::column:::

+ :::image type="content" source="./media/howto-integrate-app-service/dataflow-2.jpg" alt-text="Screenshot of step two of app data flow." lightbox="./media/howto-integrate-app-service/dataflow-2.jpg":::

+ :::column-end:::

+ :::column:::

+ If the handshake with Azure Web PubSub is successful, the client is added to a group named `draw`, effectively subscribing to messages published to this group. Also, the client is given the permission to send messages to the [`draw` group](https://github.com/Azure/awps-webapp-sample/blob/main/whiteboard/server.js#L64).

+ :::column-end:::

+ :::column:::

+ :::image type="content" source="./media/howto-integrate-app-service/dataflow-3.jpg" alt-text="Screenshot of step three of app data flow." lightbox="./media/howto-integrate-app-service/dataflow-3.jpg":::

+ :::column-end:::

+> [!NOTE]

+> To keep this how-to guide focused, all connecting clients are added to the same group named `draw` and is given the permission to send messages to this group. To manage client connections at a granular level, see the full references of the APIs provided by Azure Web PubSub.

+ :::column:::

+ Azure Web PubSub notifies the backend application that a client has connected. The backend application handles the `onConnected` event by calling the `sendToAll()`, with a payload of the latest number of connected clients.

+ :::column-end:::

+ :::column:::

+ :::image type="content" source="./media/howto-integrate-app-service/dataflow-4.jpg" alt-text="Screenshot of step four of app data flow." lightbox="./media/howto-integrate-app-service/dataflow-4.jpg":::

+ :::column-end:::

+> [!NOTE]

+> It is important to note that if there are a large number of online users in the `draw` group, with **a single** network call from the backend application, all the online users will be notified that a new user has just joined. This drastically reduces the complexity and load of the backend application.

+ :::column:::

+ As soon as a client establishes a persistent connection with Web PubSub, it makes an HTTP request to the backend application to fetch the latest shape and background data at [`/diagram`](https://github.com/Azure/awps-webapp-sample/blob/main/whiteboard/server.js#L70). An HTTP service hosted on App Service can be combined with Web PubSub. App Service takes care serving HTTP endpoints, while Web PubSub takes care of managing WebSocket connections.

+ :::column-end:::

+ :::column:::

+ :::image type="content" source="./media/howto-integrate-app-service/dataflow-5.jpg" alt-text="Screenshot of step five of app data flow." lightbox="./media/howto-integrate-app-service/dataflow-5.jpg":::

+ :::column-end:::

+ :::column:::

+ Now that the clients and backend application have two ways to exchange data. One is the conventional HTTP request-response cycle and the other is the persistent, bi-directional channel through Web PubSub. The drawing actions, which originate from one user and need to be broadcasted to all users as soon as it takes place, are delivered through Web PubSub. It doesn't require involvement of the backend application.

+ :::column-end:::

+ :::column:::

+ :::image type="content" source="./media/howto-integrate-app-service/dataflow-6.jpg" alt-text="Screenshot of step six of app data flow." lightbox="./media/howto-integrate-app-service/dataflow-6.jpg":::

+ :::column-end:::

+## Clean up resources

+Although the application uses only the free tiers of both services, it's best practice to delete resources if you no longer need them. You can delete the resource group along with the resources in it using following command,

+```azurecli-interactive

+az group delete

+ --name "whiteboard-group"

+```

+## Next steps

+> [!div class="nextstepaction"]

+> [Check out more demos built with Web PubSub](https://azure.github.io/azure-webpubsub/demos/chat)

+ Title: Azure Web PubSub samples - app scenarios

+description: A list of code samples showing how Web PubSub is used in a wide variety of web applications

+zone_pivot_groups: azure-web-pubsub-samples-app-scenarios

+# Azure Web PubSub samples - app scenarios

+Bi-directional, low-latency and real-time data exchange between clients and server was a *nice-to-have* feature, but now end users expect this behavior **by default**. Azure Web PubSub is used in a wide range of industries, powering applications like

+- dashboard for real-time monitoring in finance, retail and manufacturing

+- cross-platform chat room in health care and social networking

+- competitive bidding in online auctions,

+- collaborative coauthoring in modern work applications

+- and a lot more

+Here's a list of code samples written by Azure Web PubSub team and the community. To have your project featured here, consider submitting a Pull Request.

+| App scenario | Industry |

+| | -- |

+| [Unity multiplayer gaming](https://github.com/Azure/azure-webpubsub/tree/main/samples/csharp/unity-multiplayer-sample) | Gaming |

+| [Chat app with persistent storage](https://github.com/Azure/azure-webpubsub/tree/main/samples/csharp/chatapp-withstorage) | Gaming |

+| App scenario | Industry |

+| | -- |

+| [Cross-platform chat](https://github.com/Azure/azure-webpubsub/blob/main/samples/csharp/chatapp/Startup.cs#L29) | Social |

+| [Collaborative code editor](https://github.com/Azure/azure-webpubsub/blob/main/samples/csharp/chatapp/Startup.cs#L29) | Modern work |

+| App scenario | Industry |

+| | -- |

+| [Chat app](https://github.com/Azure/azure-webpubsub/tree/main/samples/java/chatapp) | Social |

+| App scenario | Industry |

+| | -- |

+| [Chat app](https://github.com/Azure/azure-webpubsub/tree/main/samples/python/chatapp) | Social |

+ Title: Azure Web PubSub samples - authenticate and connect

+description: A list of code samples showing how to authenticate and connect to Web PubSub resource(s)

+zone_pivot_groups: azure-web-pubsub-samples-authenticate-and-connect

+# Azure Web PubSub samples - Authenticate and connect

+To make use of your Azure Web PubSub resource, you need to authenticate and connect to the service first. Azure Web PubSub service distinguishes two roles and they're given a different set of capabilities.

+

+## Client

+The client can be a browser, a mobile app, an IoT device or even an EV charging point as long as it supports WebSocket. A client is limited to publishing and subscribing to messages.

+## Application server

+While the client's role is often limited, the application server's role goes beyond simply receiving and publishing messages. Before a client tries to connect with your Web PubSub resource, it goes to the application server for a Client Access Token first. The token is used to establish a persistent WebSocket connection with your Web PubSub resource.

+| Use case | Description |

+| | -- |

+| [Using connection string](https://github.com/Azure/azure-webpubsub/blob/main/samples/csharp/chatapp/Startup.cs#L29) | Applies to application server only.

+| [Using Client Access Token](https://github.com/Azure/azure-webpubsub/blob/main/samples/csharp/chatapp/wwwroot/https://docsupdatetracker.net/index.html#L13) | Applies to client only. Client Access Token is generated on the application server.

+| [Using Azure Active Directory](https://github.com/Azure/azure-webpubsub/blob/main/samples/csharp/chatapp-aad/Startup.cs#L26) | Using Azure AD for authorization offers improved security and ease of use compared to Access Key authorization.

+| [Anonymous connection](https://github.com/Azure/azure-webpubsub/blob/main/samples/csharp/clientWithCert/client/Program.cs#L15) | Anonymous connection allows clients to connect with Azure Web PubSub directly without going to an application server for a Client Access Token first. This is useful for clients that have limited networking capabilities, like an EV charging point.

+| Use case | Description |

+| | -- |

+| [Using connection string](https://github.com/Azure/azure-webpubsub/blob/main/samples/javascript/chatapp/sdk/server.js#L9) | Applies to application server only.

+| [Using Client Access Token](https://github.com/Azure/azure-webpubsub/blob/main/samples/javascript/chatapp/sdk/src/index.js#L5) | Applies to client only. Client Access Token is generated on the application server.

+| [Using Azure Active Directory](https://github.com/Azure/azure-webpubsub/blob/main/samples/javascript/chatapp-aad/server.js#L24) | Using Azure AD for authorization offers improved security and ease of use compared to Access Key authorization.

+| Use case | Description |

+| | -- |

+| [Using connection string](https://github.com/Azure/azure-webpubsub/blob/eb60438ff9e0735d90a6e7e6370b9d38aa6bc730/samples/java/chatapp/src/main/java/com/webpubsub/tutorial/App.java#L21) | Applies to application server only.

+| [Using Client Access Token](https://github.com/Azure/azure-webpubsub/blob/eb60438ff9e0735d90a6e7e6370b9d38aa6bc730/samples/java/chatapp/src/main/resources/public/https://docsupdatetracker.net/index.html#L12) | Applies to client only. Client Access Token is generated on the application server.

+| [Using Azure Active Directory](https://github.com/Azure/azure-webpubsub/blob/eb60438ff9e0735d90a6e7e6370b9d38aa6bc730/samples/java/chatapp-aad/src/main/java/com/webpubsub/tutorial/App.java#L22) | Using Azure AD for authorization offers improved security and ease of use compared to Access Key authorization.

+| Use case | Description |

+| | -- |

+| [Using connection string](https://github.com/Azure/azure-webpubsub/blob/eb60438ff9e0735d90a6e7e6370b9d38aa6bc730/samples/python/chatapp/server.py#L19) | Applies to application server only.

+| [Using Client Access Token](https://github.com/Azure/azure-webpubsub/blob/eb60438ff9e0735d90a6e7e6370b9d38aa6bc730/samples/python/chatapp/public/https://docsupdatetracker.net/index.html#L13) | Applies to client only. Client Access Token is generated on the application server.

+| [Using Azure Active Directory](https://github.com/Azure/azure-webpubsub/blob/eb60438ff9e0735d90a6e7e6370b9d38aa6bc730/samples/python/chatapp-aad/server.py#L21) | Using Azure AD for authorization offers improved security and ease of use compared to Access Key authorization.

+ Title: Azure Web PubSub samples - platforms and frameworks

+description: A list of code samples showing how Web PubSub is used in a wide variety of platforms and frameworks

+# Azure Web PubSub samples - Platforms and frameworks

+As Azure Web PubSub is built on top of WebSocket, the only requirement is having the support of WebSocket, which is available in all modern browsers and major app development platforms.

+## Web front-end

+- [Blazor WebAssembly](https://github.com/Azure/azure-webpubsub/tree/main/samples/csharp/blazor-webassembly)

+- [React](https://github.com/Azure/azure-webpubsub/tree/main/samples/javascript/chatapp/react)

+- [Vue](https://github.com/Azure/azure-webpubsub/tree/main/samples/javascript/scoreboard)

+<!-- ## Cross-platform

+ :::column span="":::

+ [React native](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/storage/storage-blob/samples/v12/javascript/connectionStringAuth.js)

+ :::column-end:::

+## Gaming

+- [Unity](https://github.com/Azure/azure-webpubsub/tree/main/samples/csharp/unity-multiplayer-sample)

+<!-- ## Low-code / no-code platform

+ :::column span="":::

+ [Power Apps](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/storage/storage-blob/samples/v12/javascript/connectionStringAuth.js)

+ :::column-end:::

+ Title: Integrate - Create a chat app using Azure Web PubSub and deploy to Azure Static Web Apps

+Azure Web PubSub helps you build real-time messaging web applications using WebSocket. Azure Static Web Apps helps you build and deploy full-stack web apps automatically to Azure from a code repository. In this tutorial, you learn how to use Web PubSub and Static Web Apps together to build a real-time chat room application.

+More specifically, you learn how to:

+1. When a user signs in to the app, the Azure Functions `login` API is triggered to generate a Web PubSub service client connection URL.

+1. When a client sends a message to Azure Web PubSub service, the service responds with a user `message` event and the Functions `message` API is triggered to broadcast, the message to all the connected clients.

+This article uses a GitHub template repository to make it easy for you to get started. The template features a starter app that you deploy to Azure Static Web Apps.

+You're very close to complete. The last step is to configure Web PubSub so that client requests are transferred to your function APIs.

+In this quickstart, you learned how to develop and deploy a serverless chat application. Now, you can start building your own application.

+> [Have fun with playable demos](https://azure.github.io/azure-webpubsub/)

+| Azure Stack HCI 22H2 support | MABS V4 now supports protection of workloads running in Azure Stack HCI from V1 to 22H2. [Learn more](back-up-azure-stack-hyperconverged-infrastructure-virtual-machines.md). |

+For more information, see "NC2 on Azure Subscription and Billing" in [Nutanix Cloud Clusters on Azure Deployment and User Guide]

+(https://portal.nutanix.com/page/documents/details?targetId=Nutanix-Cloud-Clusters-Azure:Nutanix-Cloud-Clusters-Azure).

+gateway-external-api.cloud.nutanix.com.

+* Bastion only supports 50 requests, including creates and deletes, for shareable links at a time.

+ Title: Understand chaos engineering and resilience with Chaos Studio

+# Understand chaos engineering and resilience

+Before you start using Azure Chaos Studio, it's useful to understand the core site reliability engineering concepts being applied.

+It's never been easier to create large-scale, distributed applications. Infrastructure is hosted in the cloud, and programming language support is diverse. There are also many open-source and hosted components and services to build on.

+Unfortunately, there's no reliability guarantee for these underlying components and dependencies, or for systems built on them. Infrastructure can go offline, and service disruptions or outages can occur at any time. Minor disruptions in one area can be magnified and have longstanding side effects in another.

+Applications and services must plan for and accommodate issues like:

+- Service outages.

+- Disruptions to known and unknown dependencies.

+- Sudden unexpected load.

+- Latencies throughout the system.

+Applications and services must be designed to handle failure and be hardened against disruptions.

+Applications and services that deal with stresses and issues gracefully are *resilient*. Individual component reliability is good, but *resilience is a property of the entire system*. End-to-end system resilience must be validated in an integrated, production-like environment with the conditions and load that's faced in production.

+- **Chaos engineering**: The practice of subjecting applications and services to real-world stresses and failures. The goal is to build and validate resilience to unreliable conditions and missing dependencies.

+- **Fault injection**: The act of introducing an error to a system. You can use different faults, such as network latency or loss of access to storage, to target system components. You can create scenarios that an application or service must be able to handle or recover from.

+A chaos experiment is the application of faults individually, in parallel, or sequentially against one or more subscription resources or dependencies. The goal is to monitor system behavior and health so that you can act on any issues that arise.

+An experiment can represent a real-world scenario, such as a datacenter power outage or network latency to a DNS server. It can also be used to simulate edge conditions that occur. Examples are Black Friday shopping sprees or when concert tickets go on sale for a popular band.

+description: Understand the concept of a chaos experiment in Azure Chaos Studio. What are the parts of a chaos experiment? How can you create a chaos experiment?

+In Azure Chaos Studio, you create and run chaos experiments. A chaos experiment is an Azure resource that describes the faults that should be run and the resources those faults should be run against.

+An experiment is divided into two sections:

+- **Selectors**: Selectors are groups of target resources that have faults or other actions run against them. A selector allows you to logically group resources for reuse across multiple actions.

+ For example, you might have a selector named `AllNonProdEastUSVMs`, in which you've added all the nonproduction virtual machines in East US. You could then apply CPU pressure followed by virtual memory pressure to those virtual machines by referencing the selector.

+- **Logic**: The rest of the experiment describes how and when to run faults. An experiment is organized into *steps* that run one after the other. Each step has one or more *branches* that run at the same time. Steps and branches allow you to inject multiple faults across resources in your environment in parallel.

+ Each branch has one or more *actions*, which are either the faults you want to run or time delays. *Faults* are actions that cause some disruption. Most faults take one or more *parameters*, such as the duration to run the fault or the amount of stress to apply.

+

+Chaos experiments can target resources in a different subscription than the experiment if the subscription is within the same Azure tenant. Chaos experiments can target resources in a different region than the experiment if the region is a supported region for Chaos Studio.

+Now that you understand what a chaos experiment is you're ready to:

+The following table lists the supported resource types for faults, the target types, and suggested roles to use when you give an experiment permission to a resource of that type.

+| Resource type | Target name/type | Suggested role assignment |

+| Microsoft.DocumentDb/databaseAccounts (CosmosDB, service-direct) | Microsoft-CosmosDB | Azure Cosmos DB Operator |

+| Microsoft.KeyVault/vaults (service-direct) | Microsoft-KeyVault | Azure Key Vault Contributor |

+In Azure Chaos Studio, every activity that happens as part of an experiment is called an *action*. The most common type of action is a *fault*. This article describes actions and faults and the properties of each.

+An action is any activity that's orchestrated as part of a chaos experiment. Actions are organized into steps and branches, enabling actions to run either sequentially or in parallel. Every action has the following properties:

+* **Name**: The specific action that takes place. A name usually takes the form of a URN for the action, for example, `urn`.

+* **Type**: The way that the action executes. Actions can be either *continuous* or *discrete*. A continuous action runs nonstop over a period of time. An example is applying CPU pressure for 10 minutes. A discrete action occurs only once. An example is rebooting an Azure Cache for Redis instance.

+- **Faults**: This action causes a disruption in one or more resources.

+- **Time delays**: This action "waits" without affecting any resources. It's useful for pausing in between faults to wait for a system to be affected by the previous fault.

+Faults are the most common action in Chaos Studio. Faults cause a disruption in a system, allowing you to verify that the system effectively handles that disruption without affecting availability.

+Faults can:

+- Be destructive. For example, a fault can kill a process.

+- Apply pressure. For example, a fault can add virtual memory pressure.

+- Add latency.

+- Cause a configuration change.

+In addition to a name and type, faults might also have a *duration*, if continuous, and *parameters*. Parameters describe how the fault should be applied and are specific to the fault name. For example, a parameter for the Azure Cosmos DB failover fault is the read region that will be promoted to the write region during the write region failure. Some parameters are required while others are optional.

+Faults are either *agent-based* or *service-direct* depending on the target type. An agent-based fault requires the Chaos Studio agent to be installed on a virtual machine or virtual machine scale set. The agent is available for both Windows and Linux, but not all faults are available on both operating systems. For information on which faults are supported on each operating system, see [Chaos Studio fault and action library](chaos-studio-fault-library.md). Service-direct faults don't require any agent. They run directly against an Azure resource.

+Faults also include the name of the selector that describes the resources that the fault runs against. To learn more about selectors, see [Chaos experiments](chaos-studio-chaos-experiments.md). A fault can only affect a resource if the resource has been onboarded as a target and has the corresponding fault capability enabled on the resource.

+Now that you understand actions and faults you're ready to:

+description: Understand current limitations and known issues when you use Azure Chaos Studio.

+# Azure Chaos Studio Preview limitations and known issues

+## Limitations

+* The target resources must be in [one of the regions supported by the Azure Chaos Studio Preview](https://azure.microsoft.com/global-infrastructure/services/?products=chaos-studio).

+ * Regional endpoints to allowlist are listed in [Permissions and security in Azure Chaos Studio](chaos-studio-permissions-security.md#network-security).

+ * If you're sending telemetry data to Application Insights, the IPs in [IP addresses used by Azure Monitor](../azure-monitor/app/ip-addresses.md) are also required.

+* If you run an experiment that makes use of the Chaos Studio agent, the virtual machine must run one of the following operating systems:

+ * Red Hat Enterprise Linux 8.2, SUSE Enterprise Linux 15 SP2, CentOS 8.2, Debian 10 Buster (with unzip installation required), Oracle Linux 7.8, Ubuntu Server 16.04 LTS, and Ubuntu Server 18.04 LTS

+* The Chaos Studio agent isn't tested against custom Linux distributions or hardened Linux distributions (for example, FIPS or SELinux).

+ * **Windows:** Microsoft Edge, Google Chrome, and Firefox

+ * **MacOS:** Safari, Google Chrome, and Firefox

+When you pick target resources for an agent-based fault in the experiment designer, it's possible to select virtual machines or virtual machine scale sets with an operating system not supported by the fault selected.

+Get started creating and running chaos experiments to improve application resilience with Chaos Studio by using the following links:

+ Title: Create an experiment using an agent-based fault with Azure CLI

+description: Create an experiment that uses an agent-based fault and configure the chaos agent with the Azure CLI.

+# Create a chaos experiment that uses an agent-based fault with the Azure CLI

+You can use a chaos experiment to verify that your application is resilient to failures by causing those failures in a controlled environment. In this article, you cause a high CPU event on a Linux virtual machine (VM) by using a chaos experiment and Azure Chaos Studio Preview. Run this experiment to help you defend against an application from becoming resource starved.

+You can use these same steps to set up and run an experiment for any agent-based fault. An *agent-based* fault requires setup and installation of the chaos agent. A service-direct fault runs directly against an Azure resource without any need for instrumentation.

+- An Azure subscription. [!INCLUDE [quickstarts-free-trial-note](../../includes/quickstarts-free-trial-note.md)]

+- A virtual machine. If you don't have a VM, you can [create one](../virtual-machines/linux/quick-create-portal.md).

+- A network setup that permits you to [SSH into your VM](../virtual-machines/ssh-keys-portal.md).

+- A user-assigned managed identity. If you don't have a user-assigned managed identity, you can [create one](../active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities.md).

+## Open Azure Cloud Shell

+Azure Cloud Shell is a free interactive shell that you can use to run the steps in this article. It has common Azure tools preinstalled and configured to use with your account.

+To open Cloud Shell, select **Try it** in the upper-right corner of a code block. You can also open Cloud Shell in a separate browser tab by going to [Bash](https://shell.azure.com/bash). Select **Copy** to copy the blocks of code, paste it into Cloud Shell, and select **Enter** to run it.

+> These instructions use a Bash terminal in Cloud Shell. Some commands might not work as described if you run the CLI locally or in a PowerShell terminal.

+Before you set up Chaos Studio on the VM, assign a user-assigned managed identity to each VM or virtual machine scale set where you plan to install the agent. Use the `az vm identity assign` or `az vmss identity assign` command. Replace `$VM_RESOURCE_ID`/`$VMSS_RESOURCE_ID` with the resource ID of the VM you're adding as a chaos target. Replace `$MANAGED_IDENTITY_RESOURCE_ID` with the resource ID of the user-assigned managed identity.

+Virtual machine

+Virtual machine scale set

+Chaos Studio can't inject faults against a VM unless that VM was added to Chaos Studio first. To add a VM to Chaos Studio, create a [target and capabilities](chaos-studio-targets-capabilities.md) on the resource. Then you install the chaos agent.

+Virtual machines have two target types. One target type enables service-direct faults (where no agent is required). The other target type enables agent-based faults (which requires the installation of an agent). The chaos agent is an application installed on your VM as a [VM extension](../virtual-machines/extensions/overview.md). You use it to inject faults in the guest operating system.

+The Chaos Studio agent for Linux requires stress-ng. This open-source application can cause various stress events on a VM. To install stress-ng, [connect to your Linux VM](../virtual-machines/ssh-keys-portal.md). Then run the appropriate installation command for your package manager. For example:

+### Enable the chaos target and capabilities

+Next, set up a Microsoft-Agent target on each VM or virtual machine scale set that specifies the user-assigned managed identity that the agent uses to connect to Chaos Studio. In this example, we use one managed identity for all VMs. A target must be created via REST API. In this example, we use the `az rest` CLI command to execute the REST API calls.

+1. Modify the following JSON by replacing `$USER_IDENTITY_CLIENT_ID` with the client ID of your managed identity. You can find the client ID in the Azure portal overview of the user-assigned managed identity you created. Replace `$USER_IDENTITY_TENANT_ID` with your Azure tenant ID. You can find it in the Azure portal under **Azure Active Directory** under **Tenant information**. Save the JSON as a file in the same location where you're running the Azure CLI. In Cloud Shell, you can drag and drop the JSON file to upload it.

+1. Create the target by replacing `$RESOURCE_ID` with the resource ID of the target VM or virtual machine scale set. Replace `target.json` with the name of the JSON file you created in the previous step.

+1. Copy down the GUID for the **agentProfileId** returned by this command for use in a later step.

+1. Create the capabilities by replacing `$RESOURCE_ID` with the resource ID of the target VM or virtual machine scale set. Replace `$CAPABILITY` with the [name of the fault capability you're enabling](chaos-studio-fault-library.md).

+ For example, if you're enabling the CPU Pressure capability:

+The chaos agent is an application that runs in your VM or virtual machine scale set instances to execute agent-based faults. During installation, you configure:

+- The agent with the managed identity that the agent should use to authenticate to Chaos Studio.

+- The profile ID of the Microsoft-Agent target that you created.

+- Optionally, an Application Insights instrumentation key that enables the agent to send diagnostic events to Application Insights.

+1. Before you begin, make sure you have the following details:

+ * **agentProfileId**: The property returned when you create the target. If you don't have this property, you can run `az rest --method get --uri https://management.azure.com/$RESOURCE_ID/providers/Microsoft.Chaos/targets/Microsoft-Agent?api-version=2021-09-15-preview` and copy the `agentProfileId` property.

+ * **ClientId**: The client ID of the user-assigned managed identity used in the target. If you don't have this property, you can run `az rest --method get --uri https://management.azure.com/$RESOURCE_ID/providers/Microsoft.Chaos/targets/Microsoft-Agent?api-version=2021-09-15-preview` and copy the `clientId` property.

+ * **(Optionally) AppInsightsKey**: The instrumentation key for your Application Insights component, which you can find on the Application Insights page in the portal under **Essentials**.

+1. Install the Chaos Studio VM extension. Replace `$VM_RESOURCE_ID` with the resource ID of your VM or replace `$SUBSCRIPTION_ID`, `$RESOURCE_GROUP`, and `$VMSS_NAME` with those properties for your virtual machine scale set. Replace `$AGENT_PROFILE_ID` with the agent Profile ID. Replace `$USER_IDENTITY_CLIENT_ID` with the client ID of your managed identity. Replace `$APP_INSIGHTS_KEY` with your Application Insights instrumentation key. If you aren't using Application Insights, remove that key/value pair.

+ Windows

+ Linux

+ Windows

+ Linux

+1. If you're setting up a virtual machine scale set, verify that the instances were upgraded to the latest model. If needed, upgrade all instances in the model.

+After you've successfully deployed your VM, now you can create your experiment. A chaos experiment defines the actions you want to take against target resources. The actions are organized and run in sequential steps. The chaos experiment also defines the actions you want to take against branches, which run in parallel.

+1. Formulate your experiment JSON starting with the following JSON sample. Modify the JSON to correspond to the experiment you want to run by using the [Create Experiment API](/rest/api/chaosstudio/experiments/create-or-update) and the [fault library](chaos-studio-fault-library.md).

+ If you're running against a virtual machine scale set, modify the fault parameters to include the instance numbers to target:

+ You can identify scale set instance numbers in the Azure portal by going to your virtual machine scale set and selecting **Instances**. The instance name ends in the instance number.

+1. Create the experiment by using the Azure CLI. Replace `$SUBSCRIPTION_ID`, `$RESOURCE_GROUP`, and `$EXPERIMENT_NAME` with the properties for your experiment. Make sure you've saved and uploaded your experiment JSON. Update `experiment.json` with your JSON filename.

+ Each experiment creates a corresponding system-assigned managed identity. Note the principal ID for this identity in the response for the next step.

+## Give the experiment permission to your virtual machine

+When you create a chaos experiment, Chaos Studio creates a system-assigned managed identity that executes faults against your target resources. This identity must be given [appropriate permissions](chaos-studio-fault-providers.md) to the target resource for the experiment to run successfully. The Reader role is required for agent-based faults. Other roles that don't have */Read permission, such as Virtual Machine Contributor, won't grant appropriate permission for agent-based faults.

+Give the experiment access to your VM or virtual machine scale set by using the following command. Replace `$EXPERIMENT_PRINCIPAL_ID` with the principal ID from the previous step. Replace `$RESOURCE_ID` with the resource ID of the target VM or virtual machine scale set. Be sure to use the resource ID of the VM, not the resource ID of the chaos agent used in the experiment definition. Run this command for each VM or virtual machine scale set targeted in your experiment.

+You're now ready to run your experiment. To see the effect, we recommend that you open [an Azure Monitor metrics chart](../azure-monitor/essentials/tutorial-metrics.md) with your VM's CPU pressure in a separate browser tab.

+1. Start the experiment by using the Azure CLI. Replace `$SUBSCRIPTION_ID`, `$RESOURCE_GROUP`, and `$EXPERIMENT_NAME` with the properties for your experiment.

+1. The response includes a status URL that you can use to query experiment status as the experiment runs.

+Now that you've run an agent-based experiment, you're ready to:

+ Title: Create an experiment using an agent-based fault with the portal

+description: Create an experiment that uses an agent-based fault and configure the chaos agent with the portal.

+# Create a chaos experiment that uses an agent-based fault with the Azure portal

+You can use a chaos experiment to verify that your application is resilient to failures by causing those failures in a controlled environment. In this article, you cause a high CPU event on a Linux virtual machine (VM) by using a chaos experiment and Azure Chaos Studio Preview. Running this experiment can help you defend against an application from becoming resource starved.

+You can use these same steps to set up and run an experiment for any agent-based fault. An *agent-based* fault requires setup and installation of the chaos agent. A service-direct fault runs directly against an Azure resource without any need for instrumentation.

+- An Azure subscription. [!INCLUDE [quickstarts-free-trial-note](../../includes/quickstarts-free-trial-note.md)]

+- A Linux VM. If you don't have a VM, you can [create one](../virtual-machines/linux/quick-create-portal.md).

+- A network setup that permits you to [SSH into your VM](../virtual-machines/ssh-keys-portal.md).

+- A user-assigned managed identity *that was assigned to the target VM or virtual machine scale set*. If you don't have a user-assigned managed identity, you can [create one](../active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities.md).

+Chaos Studio can't inject faults against a VM unless that VM was added to Chaos Studio first. To add a VM to Chaos Studio, create a [target and capabilities](chaos-studio-targets-capabilities.md) on the resource. Then you install the chaos agent.

+Virtual machines have two target types. One target type enables service-direct faults (where no agent is required). Another target type enables agent-based faults (which requires the installation of an agent). The chaos agent is an application installed on your VM as a [VM extension](../virtual-machines/extensions/overview.md). You use it to inject faults in the guest operating system.

+The Chaos Studio agent for Linux requires stress-ng. This open-source application can cause various stress events on a VM. To install stress-ng, [connect to your Linux VM](../virtual-machines/ssh-keys-portal.md). Then run the appropriate installation command for your package manager. For example:

+Or:

+### Enable the chaos target, capabilities, and agent

+> Prior to finishing the next steps, you must [create a user-assigned managed identity](../active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities.md). Then you assign it to the target VM or virtual machine scale set.

+1. Search for **Chaos Studio (preview)** in the search bar.

+1. Select **Targets** and move to your VM.

+ 

+1. Select the checkbox next to your VM and select **Enable targets**. Then select **Enable agent-based targets** from the dropdown menu.

+ 

+1. Select the **Managed Identity** to use to authenticate the chaos agent and optionally enable Application Insights to see experiment events and agent logs.

+ 

+1. Select **Review + Enable** > **Enable**.

+ 

+1. After a few minutes, a notification appears that indicates that the resources selected were successfully enabled. The Azure portal adds the user-assigned identity to the VM. The portal enables the agent target and capabilities and installs the chaos agent as a VM extension.

+ 

+1. If you're enabling a virtual machine scale set, upgrade instances to the latest model by going to the virtual machine scale set resource pane. Select **Instances**, and then select all instances. Select **Upgrade** if you're not on the latest model.

+You've now successfully added your Linux VM to Chaos Studio. In the **Targets** view, you can also manage the capabilities enabled on this resource. Select the **Manage actions** link next to a resource to display the capabilities enabled for that resource.

+Now you can create your experiment. A chaos experiment defines the actions you want to take against target resources. The actions are organized and run in sequential steps. The chaos experiment also defines the actions you want to take against branches, which run in parallel.

+1. Select the **Experiments** tab in Chaos Studio. In this view, you can see and manage all your chaos experiments. Select **Add an experiment**.

+ 

+1. Fill in the **Subscription**, **Resource Group**, and **Location** where you want to deploy the chaos experiment. Give your experiment a name. Select **Next: Experiment designer**.

+ 

+1. You're now in the Chaos Studio experiment designer. You can build your experiment by adding steps, branches, and faults. Give a friendly name to your **Step** and **Branch**. Then select **Add fault**.

+ 

+1. Select **CPU Pressure** from the dropdown list. Fill in **Duration** with the number of minutes to apply pressure. Fill in **pressureLevel** with the amount of CPU pressure to apply. Leave **virtualMachineScaleSetInstances** blank. Select **Next: Target resources**.

+ 

+1. Select your VM and select **Next**.

+ 

+1. Verify that your experiment looks correct. Then select **Review + create** > **Create**.

+ 

+## Give the experiment permission to your virtual machine

+1. Go to your VM and select **Access control (IAM)**.

+ 

+1. Select **Add** > **Add role assignment**.

+ 

+1. Search for **Reader** and select the role. Select **Next**.

+ 

+1. Choose **Select members** and search for your experiment name. Select your experiment and choose **Select**. If there are multiple experiments in the same tenant with the same name, your experiment name is truncated with random characters added.

+ 

+1. Select **Review + assign** > **Review + assign**.

+You're now ready to run your experiment. To see the impact, we recommend that you open an [Azure Monitor metrics chart](../azure-monitor/essentials/tutorial-metrics.md) with your VM's CPU pressure in a separate browser tab.

+1. In the **Experiments** view, select your experiment. Select **Start** > **OK**.

+ 

+1. After the **Status** changes to *Running*, under **History**, select **Details** for the latest run to see details for the running experiment.

+Now that you've run an agent-based experiment, you're ready to:

+description: Create an experiment that uses dynamic targeting with the Azure CLI.

+You can use dynamic targeting in a chaos experiment to choose a set of targets to run an experiment against. In this article, we show you how to dynamically target virtual machine scale sets to shut down based on availability zone. Running this experiment can help you test failover to an Azure Virtual Machine Scale Sets instance in a different region if there's an outage.

+You can use these same steps to set up and run an experiment for any fault that supports dynamic targeting. Currently, only virtual machine scale set shutdown supports dynamic targeting.

+- An Azure subscription. [!INCLUDE [quickstarts-free-trial-note](../../includes/quickstarts-free-trial-note.md)]

+- An Azure Virtual Machine Scale Sets instance.

+## Open Azure Cloud Shell

+Azure Cloud Shell is a free interactive shell that you can use to run the steps in this article. It has common Azure tools preinstalled and configured to use with your account.

+To open Cloud Shell, select **Try it** in the upper-right corner of a code block. You can also open Cloud Shell in a separate browser tab by going to [Bash](https://shell.azure.com/bash). Select **Copy** to copy the blocks of code, paste it into Cloud Shell, and select **Enter** to run it.

+If you want to install and use the CLI locally, this tutorial requires Azure CLI version 2.0.30 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI]( /cli/azure/install-azure-cli).

+> These instructions use a Bash terminal in Cloud Shell. Some commands might not work as described if you're running the CLI locally or in a PowerShell terminal.

+Azure Chaos Studio Preview can't inject faults against a resource unless that resource was added to Chaos Studio first. To add a resource to Chaos Studio, create a [target and capabilities](chaos-studio-targets-capabilities.md) on the resource.

+Virtual Machine Scale Sets has only one target type (`Microsoft-VirtualMachineScaleSet`) and one capability (`shutdown`). Other resources might have up to two target types. One target type is for service-direct faults. Another target type is for agent-based faults. Other resources also might have many other capabilities.

+1. Create a [target for your virtual machine scale set](chaos-studio-fault-providers.md) resource. Replace `$RESOURCE_ID` with the resource ID of the virtual machine scale set you're adding:

+1. Create the capabilities on the virtual machine scale set target. Replace `$RESOURCE_ID` with the resource ID of the resource you're adding. Specify the `VirtualMachineScaleSet` target and the `Shutdown-2.0` capability.

+You've now successfully added your virtual machine scale set to Chaos Studio.

+Now you can create your experiment. A chaos experiment defines the actions you want to take against target resources. The actions are organized and run in sequential steps. The chaos experiment also defines the actions you want to take against branches, which run in parallel.

+1. Formulate your experiment JSON starting with the following [Virtual Machine Scale Sets Shutdown 2.0](chaos-studio-fault-library.md#version-20) JSON sample. Modify the JSON to correspond to the experiment you want to run by using the [Create Experiment API](/rest/api/chaosstudio/experiments/create-or-update) and the [fault library](chaos-studio-fault-library.md). At this time, dynamic targeting is only available with the Virtual Machine Scale Sets Shutdown 2.0 fault and can only filter on availability zones.

+ - Use the `filter` element to configure the list of Azure availability zones to filter targets by. If you don't provide a `filter`, the fault shuts down all instances in the virtual machine scale set.

+ - The experiment targets all Virtual Machine Scale Sets instances in the specified zones.

+1. Create the experiment by using the Azure CLI. Replace `$SUBSCRIPTION_ID`, `$RESOURCE_GROUP`, and `$EXPERIMENT_NAME` with the properties for your experiment. Make sure that you saved and uploaded your experiment JSON. Update `experiment.json` with your JSON filename.

+ Each experiment creates a corresponding system-assigned managed identity. Note the principal ID for this identity in the response for the next step.

+## Give experiment permission to your virtual machine scale sets

+Give the experiment access to your resources by using the following command. Replace `$EXPERIMENT_PRINCIPAL_ID` with the principal ID from the previous step. Replace `$RESOURCE_ID` with the resource ID of the target resource. Change the role to the appropriate [built-in role for that resource type](chaos-studio-fault-providers.md). Run this command for each resource targeted in your experiment.

+You're now ready to run your experiment. To see the effect, check the portal to see if your virtual machine scale sets targets are shut down. If they're shut down, check to see that the services running on your virtual machine scale sets are still running as expected.

+1. Start the experiment by using the Azure CLI. Replace `$SUBSCRIPTION_ID`, `$RESOURCE_GROUP`, and `$EXPERIMENT_NAME` with the properties for your experiment.

+1. The response includes a status URL that you can use to query experiment status as the experiment runs.

+Now that you've run a dynamically targeted virtual machine scale set shutdown experiment, you're ready to:

+- [Manage your experiment](chaos-studio-run-experiment.md)

+description: Use the Azure portal to create an experiment that uses dynamic targeting to select hosts in a zone.

+You can use dynamic targeting in a chaos experiment to choose a set of targets to run an experiment against, based on criteria evaluated at experiment runtime. This article shows how you can dynamically target a virtual machine scale set to shut down instances based on availability zone. Running this experiment can help you test failover to an Azure Virtual Machine Scale Sets instance in a different region if there's an outage.

+You can use these same steps to set up and run an experiment for any fault that supports dynamic targeting. Currently, only virtual machine scale set shutdown supports dynamic targeting.

+- An Azure subscription. [!INCLUDE [quickstarts-free-trial-note](../../includes/quickstarts-free-trial-note.md)]

+## Enable Chaos Studio on your virtual machine scale sets

+Azure Chaos Studio Preview can't inject faults against a resource until that resource is added to Chaos Studio. To add a resource to Chaos Studio, create a [target and capabilities](chaos-studio-targets-capabilities.md) on the resource.

+Virtual Machine Scale Sets has only one target type (`Microsoft-VirtualMachineScaleSet`) and one capability (`shutdown`). Other resources might have up to two target types. One target type is for service-direct faults. Another target type is for agent-based faults. Other resources also might have many other capabilities.

+1. Select **Targets** and find your virtual machine scale set resource.

+1. Select the virtual machine scale set resource and select **Enable targets** > **Enable service-direct targets**.

+ [ ](images/tutorial-dynamic-targets-enable.png#lightbox)

+1. Select **Review + Enable** > **Enable**.

+You've now successfully added your virtual machine scale set to Chaos Studio.

+Now you can create your experiment. A chaos experiment defines the actions you want to take against target resources. The actions are organized and run in sequential steps. The chaos experiment also defines the actions you want to take against branches, which run in parallel.

+1. In Chaos Studio, go to **Experiments** > **Create**.

+ [](images/tutorial-dynamic-targets-experiment-browse.png#lightbox)

+1. Add a name for your experiment that complies with resource naming guidelines. Select **Next: Experiment designer**.

+ [](images/tutorial-dynamic-targets-create-exp.png#lightbox)

+1. In **Step 1** and **Branch 1**, select **Add action** > **Add fault**.

+ [](images/tutorial-dynamic-targets-experiment-fault.png#lightbox)

+1. Select the **VMSS Shutdown (version 2.0)** fault. Select your desired duration and if you want the shutdown to be abrupt. Select **Next: Target resources**.

+ [](images/tutorial-dynamic-targets-fault-details.png#lightbox)

+1. Select the virtual machine scale set resource that you want to use in the experiment. Select **Next: Scope**.

+ [](images/tutorial-dynamic-targets-fault-resources.png#lightbox)

+1. In the **Zones** dropdown list, select the zone where you want virtual machines (VMs) in the Virtual Machine Scale Sets instance to be shut down. Select **Add**.

+ [](images/tutorial-dynamic-targets-fault-zones.png#lightbox)

+1. Select **Review + create** > **Create** to save the experiment.

+## Give the experiment permission to your virtual machine scale sets

+When you create a chaos experiment, Chaos Studio creates a system-assigned managed identity that executes faults against your target resources. This identity must be given [appropriate permissions](chaos-studio-fault-providers.md) to the target resource for the experiment to run successfully. To use these steps for any resource and target type, modify the role assignment in step 3 to match the [appropriate role for that resource and target type](chaos-studio-fault-providers.md).

+1. Go to your virtual machine scale set resource and select **Access control (IAM)** > **Add role assignment**.

+ [](images/tutorial-dynamic-targets-vmss-iam.png#lightbox)

+1. On the **Role** tab, select **Virtual Machine Contributor** and select **Next**.

+ [](images/tutorial-dynamic-targets-role-selection.png#lightbox)

+1. Choose **Select members** and search for your experiment name. Select your experiment and then choose **Select**. If there are multiple experiments in the same tenant with the same name, your experiment name is truncated with random characters added.

+ [](images/tutorial-dynamic-targets-role-assignment.png#lightbox)

+1. Select **Review + assign** > **Review + assign**.

+ [](images/tutorial-dynamic-targets-role-confirmation.png#lightbox)

+You're now ready to run your experiment.

+1. In **Chaos Studio**, go to the **Experiments** view, select your experiment, and select **Start experiment(s)**.

+ [](images/tutorial-dynamic-targets-start-experiment.png#lightbox)

+1. When the **Status** changes to *Running*, select **Details** for the latest run under **History** to see details for the running experiment. If any errors occur, you can view them in **Details**. Select a failed action and expand **Failed targets**.

+To see the effect, use a tool like **Azure Monitor** or the **Virtual Machine Scale Sets** section of the portal to check if your virtual machine scale set targets are shut down. If they're shut down, check to see that the services running on your virtual machine scale sets are still running as expected.

+[](images/tutorial-dynamic-targets-view-vmss.png#lightbox)

+> If your virtual machine scale set uses an autoscale policy, the policy provisions new VMs after this experiment shuts down existing VMs. To prevent this action, add a parallel branch in your experiment that includes the **Disable Autoscale** fault against the virtual machine scale set `microsoft.insights/autoscaleSettings` resource. Remember to add the `autoscaleSettings` resource as a target and assign the role.

+Now that you've run a dynamically targeted virtual machine scale set shutdown experiment, you're ready to:

+ Title: Create an experiment using a service-direct fault with Azure CLI

+description: Create an experiment that uses a service-direct fault with the Azure CLI.

+You can use a chaos experiment to verify that your application is resilient to failures by causing those failures in a controlled environment. In this article, you cause a multi-read, single-write Azure Cosmos DB failover by using a chaos experiment and Azure Chaos Studio Preview. Running this experiment can help you defend against data loss when a failover event occurs.

+You can use these same steps to set up and run an experiment for any service-direct fault. A *service-direct* fault runs directly against an Azure resource without any need for instrumentation, unlike agent-based faults, which require installation of the chaos agent.

+- An Azure subscription. [!INCLUDE [quickstarts-free-trial-note](../../includes/quickstarts-free-trial-note.md)]

+- An Azure Cosmos DB account. If you don't have an Azure Cosmos DB account, you can [create one](../cosmos-db/sql/create-cosmosdb-resources-portal.md).

+## Open Azure Cloud Shell

+Azure Cloud Shell is a free interactive shell that you can use to run the steps in this article. It has common Azure tools preinstalled and configured to use with your account.

+To open Cloud Shell, select **Try it** in the upper-right corner of a code block. You can also open Cloud Shell in a separate browser tab by going to [Bash](https://shell.azure.com/bash). Select **Copy** to copy the blocks of code, paste it into Cloud Shell, and select **Enter** to run it.

+If you want to install and use the CLI locally, this tutorial requires Azure CLI version 2.0.30 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI]( /cli/azure/install-azure-cli).

+> These instructions use a Bash terminal in Cloud Shell. Some commands might not work as described if you're running the CLI locally or in a PowerShell terminal.

+Chaos Studio can't inject faults against a resource unless that resource was added to Chaos Studio first. You add a resource to Chaos Studio by creating a [target and capabilities](chaos-studio-targets-capabilities.md) on the resource. Azure Cosmos DB accounts have only one target type (service-direct) and one capability (failover). Other resources might have up to two target types. One target type is for service-direct faults. Another target type is for agent-based faults. Other resources might have many other capabilities.

+1. Create a target by replacing `$RESOURCE_ID` with the resource ID of the resource you're adding. Replace `$TARGET_TYPE` with the [target type you're adding](chaos-studio-fault-providers.md):

+ For example, if you're adding a virtual machine as a service-direct target:

+1. Create the capabilities on the target by replacing `$RESOURCE_ID` with the resource ID of the resource you're adding. Replace `$TARGET_TYPE` with the [target type you're adding](chaos-studio-fault-providers.md). Replace `$CAPABILITY` with the [name of the fault capability you're enabling](chaos-studio-fault-library.md).

+ For example, if you're enabling the virtual machine shutdown capability:

+You've now successfully added your Azure Cosmos DB account to Chaos Studio.

+Now you can create your experiment. A chaos experiment defines the actions you want to take against target resources. The actions are organized and run in sequential steps. The chaos experiment also defines the actions you want to take against branches, which run in parallel.

+1. Formulate your experiment JSON starting with the following JSON sample. Modify the JSON to correspond to the experiment you want to run by using the [Create Experiment API](/rest/api/chaosstudio/experiments/create-or-update) and the [fault library](chaos-studio-fault-library.md).

+1. Create the experiment by using the Azure CLI. Replace `$SUBSCRIPTION_ID`, `$RESOURCE_GROUP`, and `$EXPERIMENT_NAME` with the properties for your experiment. Make sure that you've saved and uploaded your experiment JSON. Update `experiment.json` with your JSON filename.

+ Each experiment creates a corresponding system-assigned managed identity. Note the principal ID for this identity in the response for the next step.

+## Give the experiment permission to your Azure Cosmos DB account

+Give the experiment access to your resources by using the following command. Replace `$EXPERIMENT_PRINCIPAL_ID` with the principal ID from the previous step. Replace `$RESOURCE_ID` with the resource ID of the target resource. In this case, it's the Azure Cosmos DB instance resource ID. Change the role to the appropriate [built-in role for that resource type](chaos-studio-fault-providers.md). Run this command for each resource targeted in your experiment.

+You're now ready to run your experiment. To see the effect, we recommend that you open your Azure Cosmos DB account overview and go to **Replicate data globally** in a separate browser tab. Refresh periodically during the experiment to show the region swap.

+1. Start the experiment by using the Azure CLI. Replace `$SUBSCRIPTION_ID`, `$RESOURCE_GROUP`, and `$EXPERIMENT_NAME` with the properties for your experiment.

+1. The response includes a status URL that you can use to query experiment status as the experiment runs.

+Now that you've run an Azure Cosmos DB service-direct experiment, you're ready to:

+The background removal feature is available through the [Segment](https://centraluseuap.dev.cognitive.microsoft.com/docs/services/unified-vision-apis-public-preview-2023-02-01-preview/operations/63e6b6d9217d201194bbecbd) API (`imageanalysis:segment`). You can call this API through the REST API or the Vision SDK. See the [Background removal how-to guide](./how-to/background-removal.md) for more information.

+| [Speech to text](speech-container-stt.md) | Transcribes continuous real-time speech or batch audio recordings with intermediate results. | Latest: 3.14.0<br/><br/>For all supported versions and locales, see the [Microsoft Container Registry (MCR)](https://mcr.microsoft.com/product/azure-cognitive-services/speechservices/speech-to-text/tags) and [JSON tags](https://mcr.microsoft.com/v2/azure-cognitive-services/speechservices/speech-to-text/tags/list).|

+| [Custom speech to text](speech-container-cstt.md) | Using a custom model from the [Custom Speech portal](https://speech.microsoft.com/customspeech), transcribes continuous real-time speech or batch audio recordings into text with intermediate results. | Latest: 3.14.0<br/><br/>For all supported versions and locales, see the [Microsoft Container Registry (MCR)](https://mcr.microsoft.com/product/azure-cognitive-services/speechservices/custom-speech-to-text/tags) and [JSON tags](https://mcr.microsoft.com/v2/azure-cognitive-services/speechservices/speech-to-text/tags/list). |

+| [Neural text to speech](speech-container-ntts.md) | Converts text to natural-sounding speech by using deep neural network technology, which allows for more natural synthesized speech. | Latest: 2.13.0<br/><br/>For all supported versions and locales, see the [Microsoft Container Registry (MCR)](https://mcr.microsoft.com/product/azure-cognitive-services/speechservices/neural-text-to-speech/tags) and [JSON tags](https://mcr.microsoft.com/v2/azure-cognitive-services/speechservices/neural-text-to-speech/tags/list). |

+curl --location --request POST '<endpoint>/contentsafety/text/blocklists/<your_list_id>:addBlockItems?api-version=2023-04-30-preview' \

+| Base Model | Model(s) | Dimensions |

+||||

+| Ada | models ending in -001 (Version 1) | 1024 |

+| Ada | text-embedding-ada-002 (Version 2) | 1536 |

+| Babbage | | 2048 |

+| Curie | | 4096 |

+| Davinci | | 12288 |

+<br>¹ Currently, only version `0301` of this model is available.

+When defining additional safety and behavioral guardrails, itΓÇÖs helpful to first identify and prioritize [the harms](/legal/cognitive-services/openai/overview?context=/azure/cognitive-services/openai/context/context) youΓÇÖd like to address. Depending on the application, the sensitivity and severity of certain harms could be more important than others.

+| | Get associated ISO Country Code | ✔️ | ❌ | ❌ | ❌ |

+

+ 1. Assign value for `callKitRemoteInfo.displayNameForCallKit` to customize display name for call recipients and configure `CXHandle` value. This value specified in `displayNameForCallKit` is exactly how it shows up in the last dialed call log. in the last dialed call log.

+ 2. Assign the `cxHandle` value is what the application receives when user calls back on that contact

+ if `nil` is provided for `configureAudioSession` then SDK calls the default implementation in the SDK.

+ When the app receives incoming push notification payload, we need to call `handlePush` to process it. ACS Calling SDK will raise the `IncomingCall` event.

+ We can use `reportIncomingCall` to handle push notifications when the app is closed or otherwise.

+ CallClient.reportIncomingCall(with: callNotification, callKitOptions: callKitOptions) { (error) in

+ os_log("SDK couldn't handle push notification", log:self.log)

+ If you wish to integrate the CallKit within the app and not use the CallKit implementation in the SDK, refer to the quickstart sample [here](https://github.com/Azure-Samples/communication-services-ios-quickstarts/tree/main/add-video-calling).

+let outgoingAudioOptions = OutgoingAudioOptions()

+outgoingAudioOptions.muted = true

+let incomingAudioOptions = IncomingAudioOptions()

+incomingAudioOptions.muted = true

+var copyAcceptCallOptions = AcceptCallOptions()

+copyStartCallOptions.outgoingAudioOptions = outgoingAudioOptions

+copyStartCallOptions.incomingAudioOptions = incomingAudioOptions

+Muting speaker and microphone ensure that physical audio devices aren't used until the CallKit calls the `didActivateAudioSession` on `CXProviderDelegate`. Otherwise the call may get dropped or audio will not work.

+When `didActivateAudioSession` is when the audio streams should be started.

+ Task {

+ guard let activeCall = await self.callKitHelper.getActiveCall() else {

+ print("No active calls found when activating audio session !!")

+ return

+ }

+ try await startAudio(call: activeCall)

+ }

+}

+func provider(_ provider: CXProvider, didDeactivate audioSession: AVAudioSession) {

+ Task {

+ guard let activeCall = await self.callKitHelper.getActiveCall() else {

+ print("No active calls found when deactivating audio session !!")

+ return

+ }

+ try await stopAudio(call: activeCall)

+ }

+}

+private func stopAudio(call: Call) async throws {

+ try await self.callKitHelper.muteCall(callId: call.id, isMuted: true)

+ try await call.stopAudio(stream: call.activeOutgoingAudioStream)

+ try await call.stopAudio(stream: call.activeIncomingAudioStream)

+ try await call.muteIncomingAudio()

+}

+private func startAudio(call: Call) async throws {

+ try await call.startAudio(stream: LocalOutgoingAudioStream())

+ try await self.callKitHelper.muteCall(callId: call.id, isMuted: false)

+ try await call.startAudio(stream: RemoteIncomingAudioStream())

+ try await call.unmuteIncomingAudio()

+

+It's important to also mute the outgoing audio before stopping the audio in cases when CallKit does not invoke `didActivateAudioSession`. The user can then manually unmute the microphone.

+> [!NOTE]

+> Find the code for this quickstart on [GitHub](https://github.com/Azure/communication-preview/tree/master/samples/NumberLookup).

+- To enable Number Lookup service on your Azure Communication Services subscription, please complete this [form](https://forms.microsoft.com/pages/responsepage.aspx?id=v4j5cvGGr0GRqy180BHbR058xZQ9HIBLikwspEUN6t5URUVDTTdWMEg5VElQTFpaMVMyM085ODkwVS4u) for us to allow-list your subscription.

+dotnet add package Azure.Communication.PhoneNumbers --version 1.2.0-alpha.20230531.2

+internal class Program

+ static async Task Main(string[] args)

+ {

+ ...

+ }

+string? connectionString = Environment.GetEnvironmentVariable("COMMUNICATION_SERVICES_CONNECTION_STRING");

+## Sample code

+You can download the sample app from [GitHub](https://github.com/Azure/communication-preview/tree/master/samples/NumberLookup).

+> [!div class="nextstepaction"]

+> [Number Lookup Concept](../../concepts/numbers/number-lookup-concept.md)

+> [!div class="nextstepaction"]

+> [Number Lookup SDK](../../concepts/numbers/number-lookup-sdk.md)

+- [**Decentriq**](https://www.decentriq.com/) provides SaaS data cleanrooms built on confidential computing that enable secure data collaboration without sharing data. Data science cleanrooms allow flexible multi-party analysis, and no-code cleanrooms for media and advertising enable compliant audience activation and analytics based on first-party user data. Confidential cleanrooms are described in more detail in [this article on the Microsoft blog](https://techcommunity.microsoft.com/t5/azure-confidential-computing/confidential-data-clean-rooms-the-evolution-of-sensitive-data/ba-p/3273844).

+For details on health probes, refer to [Health probes in Azure Container Apps](./health-probes.md).

+In addition to resource consumption, Azure Container Apps also charges based on the number of HTTP requests received by your container app. Only requests that come from outside a Container Apps environment are billable.

+- The first 2 million requests in each subscription per calendar month are free.

+- [Health probe](./health-probes.md) requests are not billable.

+## General terms

+The `az containerapp up` command is a streamlined way to create and deploy container apps that primarily use default settings. However, you'll need to run other CLI commands to configure more advanced settings:

+- Dapr: [`az containerapp dapr enable`](/cli/azure/containerapp/dapr#az-containerapp-dapr-enable)

+- Secrets: [`az containerapp secret set`](/cli/azure/containerapp/secret#az-containerapp-secret-set)

+- Transport protocols: [`az containerapp ingress update`](/cli/azure/containerapp/ingress#az-containerapp-ingress-update)

+If you need to customize the Container Apps environment, first create the environment using the `az containerapp env create` command. If you don't provide an existing environment, the `up` command looks for one in your resource group and, if found, uses that environment. If not found, it creates an environment with a Log Analytics workspace.

+1. Enter a tag for the container. Accept the default, which is the project name with a run ID suffix.

+1. Select the Azure subscription that you want to use.

+1. Select **Linux** as the image base operating system (OS).

+Azure Container Apps implements container app versioning by creating revisions. A revision is an immutable snapshot of a container app version.

+Container Apps revisions help you manage the release of updates to your container app by creating a new revision each time you make a *revision-scope* change to your app. You can control which revisions are active, and the external traffic that is routed to each active revision.

+- Provisioning

+- Provisioned

+- Provisioning failed

+Revisions are fully functional after provisioning is complete. Use _running status_ to monitor the status of a revision.

+| Status | Description |

+|||

+| Running | The revision is running. There are no issues to report. |

+| Unhealthy | The revision isn't operating properly. Use the revision state details for details. Common issues include:<br>ΓÇó Container crashes<br>ΓÇó Resource quota exceeded<br>ΓÇó Image access issues, including [_ImagePullBackOff_ errors](/troubleshoot/azure/azure-kubernetes/cannot-pull-image-from-acr-to-aks-cluster) |

+| Failed | The revision isn't operating properly. Use the *revision state details* for more information. Common issues include:<br>ΓÇó Container crashes<br>ΓÇó Resource quota exceeded<br>ΓÇó Image access issues, including [_ImagePullBackOff_ errors](/troubleshoot/azure/azure-kubernetes/cannot-pull-image-from-acr-to-aks-cluster) |

+| Failed | Critical errors caused revisions to fail. The _running state_ provides details. Common causes include:<br>ΓÇó Termination<br>ΓÇó Exit code `137` |

+Inactive revisions remain in a list of up to 100 inactive revisions.

+|Machine Learning | [Deploy](../machine-learning/how-to-deploy-custom-container.md) or [train](../machine-learning/v1/how-to-train-with-custom-image.md) a model in a Machine Learning workspace using a custom Docker container image | Yes |

+| Maximum number of paths in a composite index| 100 |

+Vector search is a method that helps you find similar items based on their data characteristics rather than exact matches on a property field. This technique is useful in applications such as searching for similar text, finding related images, making recommendations, or even detecting anomalies. It works by taking the vector representations (lists of numbers) of your data that you have created using an ML model, or an embeddings API. Examples of embeddings APIs could be [Azure OpenAI Embeddings](https://learn.microsoft.com/azure/cognitive-services/openai/how-to/embeddings) or [Hugging Face on Azure](https://azure.microsoft.com/solutions/hugging-face-on-azure/). It then measures the distance between the data vectors and your query vector. The data vectors that are closest to your query vector are the ones that are found to be most similar semantically.

+ "sourceCollectionId": "mv-src",

+ "$databaseName/containers/$materializedViewName?api-version=2022-11-15-preview" \

+ "$databaseName/containers/$materializedViewName?api-version=2022-11-15-preview" \

+## [.NET](#tab/dotnet)

+ ```csharp

+ DefaultAzureCredential credential = new();

+ tokenCredential: credential

+ ```

+## [Go](#tab/go)

+1. To use `DefaultAzureCredential` in a Go application, install the `azidentity` module:

+ ```bash

+ go get -u github.com/Azure/azure-sdk-for-go/sdk/azidentity

+ ```

+1. At the top of your file, add the following code:

+ ```go

+ import (

+ "github.com/Azure/azure-sdk-for-go/sdk/azidentity"

+ )

+ ```

+1. Identify the locations in your code that create a `Client` instance to connect to Azure Cosmos DB. Update your code to match the following example:

+ ```go

+ cred, err := azidentity.NewDefaultAzureCredential(nil)

+ if err != nil {

+ // handle error

+ }

+ endpoint := os.Getenv("COSMOS_ENDPOINT")

+ client, err := azblob.NewClient(endpoint, cred, nil)

+ if err != nil {

+ // handle error

+ }

+ ```

+## [Java](#tab/java)

+1. To use `DefaultAzureCredential` in a Java application, install the `azure-identity` package via one of the following approaches:

+ 1. [Include the BOM file](/java/api/overview/azure/identity-readme?view=azure-java-stable&preserve-view=true#include-the-bom-file).

+ 1. [Include a direct dependency](/java/api/overview/azure/identity-readme?view=azure-java-stable&preserve-view=true#include-direct-dependency).

+1. At the top of your file, add the following code:

+ ```java

+ import com.azure.identity.DefaultAzureCredentialBuilder;

+ ```

+1. Identify the locations in your code that create a `CosmosClient` or `CosmosAsyncClient` object to connect to Azure Cosmos DB. Update your code to match the following example:

+ ```java

+ DefaultAzureCredential credential = new DefaultAzureCredentialBuilder()

+ .build();

+ String endpoint = System.getenv("COSMOS_ENDPOINT");

+

+ CosmosClient client = new CosmosClientBuilder()

+ .endpoint(endpoint)

+ .credential(credential)

+ .consistencyLevel(ConsistencyLevel.EVENTUAL)

+ .buildClient();

+ ```

+## [Node.js](#tab/nodejs)

+1. To use `DefaultAzureCredential` in a Node.js application, install the `@azure/identity` package:

+ ```bash

+ npm install --save @azure/identity

+ ```

+1. At the top of your file, add the following code:

+ ```nodejs

+ import { DefaultAzureCredential } from "@azure/identity";

+ ```

+1. Identify the locations in your code that create a `CosmosClient` object to connect to Azure Cosmos DB. Update your code to match the following example:

+ ```nodejs

+ const credential = new DefaultAzureCredential();

+ const endpoint = process.env.COSMOS_ENDPOINT;

+ const cosmosClient = new CosmosClient({

+ endpoint,

+ aadCredentials: credential

+ });

+ ```

+## [Python](#tab/python)

+1. To use `DefaultAzureCredential` in a Python application, install the `azure-identity` package:

+

+ ```bash

+ pip install azure-identity

+ ```

+1. At the top of your file, add the following code:

+ ```python

+ from azure.identity import DefaultAzureCredential

+ ```

+1. Identify the locations in your code that create a `BlobServiceClient` object to connect to Azure Blob Storage. Update your code to match the following example:

+ ```python

+ credential = DefaultAzureCredential()

+ endpoint = os.environ["COSMOS_ENDPOINT"]

+ client = CosmosClient(

+ url = endpoint,

+ credential = credential

+ )

+ ```

+* To learn more about .NET, see [Get started with .NET in 10 minutes](https://dotnet.microsoft.com/learn/dotnet/hello-world-tutorial/intro).

+> [!WARNING]

+> Please note that throughput control is not yet supported for gateway mode.

+> Currently, for [serverless Azure Cosmos DB accounts](../serverless.md), attempting to use `targetThroughputThreshold` to define a percentage will result in failure. You can only provide an absolute value for target throughput/RU using `targetThroughput`.

+> Please note that throughput control is not yet supported for gateway mode.

+> Currently, for [serverless Azure Cosmos DB accounts](../serverless.md), attempting to use `targetThroughputThreshold` to define a percentage will result in failure. You can only provide an absolute value for target throughput/RU using `spark.cosmos.throughputControl.targetThroughput`.

+## Verify SPN role assignments

+SPN role assignments are not visible in the Azure portal. You can view enrollment account role assignments, including the subscription creator role, with the [Billing Role Assignments - List By Enrollment Account - REST API (Azure Billing)](/rest/api/billing/2019-10-01-preview/billing-role-assignments/list-by-enrollment-account) API. Use the API to verify that the role assignment was successful.

+## Set alerts on utilization

+With reservation utilization alerts, you can promptly take remedial actions to ensure optimal utilization of your reservation purchases. To learn more, see [Reservation utilization alerts](../costs/reservation-utilization-alerts.md).

+ - China: https://www.microsoft.com/download/details.aspx?id=57062

+### Configure proxy server settings when using a private endpoint

+If your company's network architure involves the use of private endpoints and for security reasons, and your company's policy does not allow a direct internet connection from the VM hosting the Self Hosted Integration Runtime to the Azure Data Factory service URL, then you will need to allow bypass the ADF Service URL for full connectivity. The following procedure provides instructions for updating the diahost.exe.config file. You should also repeat these steps for the diawp.exe.config file.

+1. In File Explorer, make a safe copy of _C:\Program Files\Microsoft Integration Runtime\4.0\Shared\diahost.exe.config_ as a backup of the original file.

+1. Open Notepad running as administrator.

+1. In Notepad, open _C:\Program Files\Microsoft Integration Runtime\4.0\Shared\diahost.exe.config_.

+1. Find the default **system.net** tag as shown here:

+ ```xml

+ <system.net>

+ <defaultProxy useDefaultCredentials="true" />

+ </system.net>

+ ```

+ You can then add bypasslist details as shown in the following example:

+ ```xml

+ <system.net>

+ <defaultProxy>

+ <bypasslist>

+ <add address = "[adfresourcename].[adfresourcelocation].datafactory.azure.net" />

+ </bypasslist>

+ <proxy

+ usesystemdefault="True"

+ proxyaddress="http://proxy.domain.org:8888/"

+ bypassonlocal="True"

+ />

+ </defaultProxy>

+ </system.net>

+ ```

+## October 2022

+### Video summary

+> [!VIDEO https://www.youtube.com/embed?v=Ou90M59VQCA&list=PLt4mCx89QIGS1rQlNt2-7iuHHAKSomVLv&index=7]

+

+### Data flow

+- Export up to 1000 rows from data flow preview [Learn more](concepts-data-flow-debug-mode.md?tabs=data-factory#data-preview)

+- SQL CDC in Mapping Data Flows now available (Public Preview) [Learn more](connector-sql-server.md?tabs=data-factory#native-change-data-capture)

+- Unlock advanced analytics with Microsoft 365 Mapping Data Flow Connector [Learn more](https://devblogs.microsoft.com/microsoft365dev/scale-access-to-microsoft-365-data-with-microsoft-graph-data-connect/)

+- SAP Change Data Capture (CDC) in now generally available [Learn more](connector-sap-change-data-capture.md#transform-data-with-the-sap-cdc-connector)

+### Developer productivity

+- Now accepting community contributions to Template Gallery [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/introducing-azure-data-factory-community-templates/ba-p/3650989)

+- New design in Azure portal ΓÇô easily discover how to launch ADF Studio [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/improved-ui-for-launching-azure-data-factory-studio/ba-p/3659610)

+- Learning Center now available in the Azure Data Factory studio [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/introducing-the-learning-center-to-azure-data-factory-studio/ba-p/3660888)

+- One-click to try Azure Data Factory [Learn more](quickstart-get-started.md)

+### Orchestration

+- Granular billing view available for ADF ΓÇô see detailed billing information by pipeline (Public Preview) [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/granular-billing-for-azure-data-factory/ba-p/3654600)

+- Script activity execution timeout now configurable [Learn more](transform-data-using-script.md)

+### Region expansion

+Continued region expansion ΓÇô Qatar Central now supported [Learn more](https://azure.microsoft.com/explore/global-infrastructure/products-by-region/?products=data-factory)

+### Continuous integration and continuous deployment

+Exclude pipeline triggers that did not change in deployment now generally available [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/ci-cd-improvements-related-to-pipeline-triggers-deployment/ba-p/3605064)

+## September 2022

+### Video summary

+> [!VIDEO https://www.youtube.com/embed?v=Bh_VA8n-SL8&list=PLt4mCx89QIGS1rQlNt2-7iuHHAKSomVLv&index=6]

+### Data flow

+- Amazon S3 source connector added [Learn more](connector-amazon-simple-storage-service.md?tabs=data-factory)

+- Google Sheets REST-based connector added as Source (Preview) [Learn more](connector-google-sheets.md?tabs=data-factory)

+- Maximum column optimization in dataflow [Learn more](format-delimited-text.md#mapping-data-flow-properties)

+- SAP Change Data Capture capabilities in Mapping Data Flow (Preview) - Extract and transform data changes from SAP systems for a more efficient data refresh [Learn more](connector-sap-change-data-capture.md#transform-data-with-the-sap-cdc-connector)

+- Writing data to a lookup field via alternative keys supported in Dynamics 365/CRM connectors for mapping data flows [Learn more](connector-dynamics-crm-office-365.md?tabs=data-factory#writing-data-to-a-lookup-field-via-alternative-keys)

+### Data movement

+Support to convert Oracle NUMBER type to corresponding integer in source [Learn more](connector-oracle.md?tabs=data-factory#oracle-as-source)

+### Monitoring

+- Additional monitoring improvements in Azure Data Factory [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/further-adf-monitoring-improvements/ba-p/3607669)

+ - Monitoring loading improvements - pipeline re-run groupings data fetched only when expanded

+ - Pagination added to pipeline activity runs view to show all activity records in pipeline run

+ - Monitoring consumption improvement ΓÇô loading icon added to know when consumption report is fully calculated

+ - Additional sorting columns in monitoring ΓÇô sorting added for Pipeline name, Run End, and Status

+ - Time-zone settings now saved in monitoring

+- Gantt chart view now supported in IR monitoring [Learn more](monitor-integration-runtime.md)

+### Orchestration

+DELETE method in the Web activity now supports sending a body with HTTP request [Learn more](control-flow-web-activity.md#type-properties)

+### User interface

+- Native UI support of parameterization added for 6 additional linked services ΓÇô SAP ODP, ODBC, Microsoft Access, Informix, Snowflake, and DB2 [Learn more](parameterize-linked-services.md?tabs=data-factory#supported-linked-service-types)

+- Pipeline designer enhancements added in Studio Preview experience ΓÇô users can view workflow inside pipeline objects like For Each, If Then, etc. [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/azure-data-factory-updated-pipeline-designer/ba-p/3618755)

+## August 2022

+### Video summary

+> [!VIDEO https://www.youtube.com/embed?v=KCJ2F6Y_nfo&list=PLt4mCx89QIGS1rQlNt2-7iuHHAKSomVLv&index=5]

+### Data flow

+- Appfigures connector added as Source (Preview) [Learn more](connector-appfigures.md)

+- Cast transformation added ΓÇô visually convert data types [Learn more](data-flow-cast.md)

+- New UI for inline datasets - categories added to easily find data sources [Learn more](data-flow-source.md#inline-datasets)

+### Data movement

+Service principal authentication type added for Azure Blob storage [Learn more](connector-azure-blob-storage.md?tabs=data-factory#service-principal-authentication)

+### Developer productivity

+- Default activity time-out changed from 7 days to 12 hours [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/azure-data-factory-changing-default-pipeline-activity-timeout/ba-p/3598729)

+- New data factory creation experience - one click to have your factory ready within seconds [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/new-experience-for-creating-data-factory-within-seconds/ba-p/3561249)

+- Expression builder UI update ΓÇô categorical tabs added for easier use [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/coming-soon-to-adf-more-pipeline-expression-builder-ease-of-use/ba-p/3567196)

+### Continuous integration and continuous delivery (CI/CD)

+When CI/CD integrating ARM template, instead of turning off all triggers, it can exclude triggers that didn't change in deployment [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/ci-cd-improvements-related-to-pipeline-triggers-deployment/ba-p/3605064)

+## July 2022

+### Video summary

+> [!VIDEO https://www.youtube.com/embed?v=EOVVt4qYvZI&list=PLt4mCx89QIGS1rQlNt2-7iuHHAKSomVLv&index=4]

+### Data flow

+- Asana connector added as source [Learn more](connector-asana.md)

+- Three new data transformation functions now supported [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/3-new-data-transformation-functions-in-adf/ba-p/3582738)

+ - [collectUnique()](data-flow-expressions-usage.md#collectUnique) - Create a new collection of unique values in an array.

+ - [substringIndex()](data-flow-expressions-usage.md#substringIndex) - Extract the substring before n occurrences of a delimiter.

+ - [topN()](data-flow-expressions-usage.md#topN) - Return the top n results after sorting your data.

+- Refetch from source available in Refresh for data source change scenarios [Learn more](concepts-data-flow-debug-mode.md#data-preview)

+- User defined functions (GA) - Create reusable and customized expressions to avoid building complex logic over and over [Learn more](concepts-data-flow-udf.md) [Video](https://www.youtube.com/watch?v=ZFTVoe8eeOc&t=170s)

+- Easier configuration on data flow runtime - choose compute size among Small, Medium and Large to pre-configure all integration runtime settings [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/adf-makes-it-easy-to-select-azure-ir-size-for-data-flows/ba-p/3578033)

+### Continuous integration and continuous delivery (CI/CD)

+Include Global parameters supported in ARM template. [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/ci-cd-improvement-using-global-parameters-in-azure-data-factory/ba-p/3557265#M665)

+### Developer productivity

+Be a part of Azure Data Factory studio preview features - Experience the latest Azure Data Factory capabilities and be the first to share your feedback [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/introducing-the-azure-data-factory-studio-preview-experience/ba-p/3563880)

+## April 2023

+### Data flow

+Easily unroll multiple arrays in ADF data flows. ADF updated the **Flatten** transformation that now makes it super easy to unroll multiple arrays from a single **Flatten** transformation step. [Learn more](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/unroll-multiple-arrays-in-a-single-flatten-step-in-adf/ba-p/3802457)

+### Continuous integration and continuous deployment

+You can customize the commit message in Git mode now. Type in a detailed description about the changes you make, and we will save it to Git repository.

+### Connectors

+The Azure Blob Storage connector now supports anonymous authentication. [Learn more](connector-azure-blob-storage.md#anonymous-authentication)

+## March 2023

+### Connectors

+Azure Data Lake Storage Gen2 connector now supports shared access signature authentication. [Learn more](connector-azure-data-lake-storage.md#shared-access-signature-authentication)

+Security alerts are the notifications generated by Defender for Cloud's workload protection plans when threats are identified in your Azure, hybrid, or multicloud environments.

+| **Medium** | This is probably a suspicious activity might indicate that a resource is compromised. Defender for Cloud's confidence in the analytic or finding is medium and the confidence of the malicious intent is medium to high. These would usually be machine learning or anomaly based detections, for example a sign-in attempt from an unusual location. |

+> In the [incidents reference](incidents-reference.md), review the list of security incident that can be produced by incident correlation.

+|**Container with a miner image detected**<br>(VM_MinerInContainerImage) | Machine logs indicate execution of a Docker container that runs an image associated with a digital currency mining. | Execution | High |

+| **Suspected brute force attack using a valid user**<br>(SQL.DB_BruteForce<br>SQL.VM_BruteForce<br>SQL.DW_BruteForce<br>SQL.MI_BruteForce<br>Synapse.SQLPool_BruteForce) | A potential brute force attack has been detected on your resource. The attacker is using the valid user (username), which has permissions to log in. | PreAttack | High |

+| **Suspected brute force attack using a valid user**<br>(SQL.PostgreSQL_BruteForce<br>SQL.MariaDB_BruteForce<br>SQL.MySQL_BruteForce) | A potential brute force attack has been detected on your resource. The attacker is using the valid user (username), which has permissions to log in. | PreAttack | High |

+The following tables include the Defender for Servers security alerts [which have been deprecated in April, 2023 due to an improvement process](release-notes.md#deprecation-and-improvement-of-selected-alerts-for-windows-and-linux-servers).

+- **Exploitability information** - Each vulnerability report is searched through exploitability databases to assist our customers with determining actual risk associated with each reported vulnerability.

+When you use classic pipeline configuration, make sure you don't change artifact name. This can result not seeing the results for your project.

+Currently, OSS vulnerability findings are only available for GitHub repositories. Azure DevOps repositories will have the total exposed secrets, IaC misconfigurations, and code security findings available. It will show `N/A` for OSS vulnerabilities. You can learn more about how to [Review your findings](defender-for-devops-introduction.md).

+ Title: Reference table for all incidents in Microsoft Defender for Cloud

+description: This article lists the incidents visible in Microsoft Defender for Cloud

+# Incidents - a reference guide

+> [!NOTE]

+> For incidents that are in preview: [!INCLUDE [Legalese](../../includes/defender-for-cloud-preview-legal-text.md)]

+This article lists the incidents you might get from Microsoft Defender for Cloud and any Microsoft Defender plans you've enabled. The incidents shown in your environment depend on the resources and services you're protecting, and your customized configuration.

+A [security incident](alerts-overview.md#what-are-security-incidents) is a correlation of alerts with an attack story that share an entity. For example, Resource, IP Address, User or share a [kill chain](alerts-reference.md#intentions) patterns.

+You can select an incident to view all of the alerts that are related to the incident and get more information.

+Learn how to [manage security incidents](incidents.md#managing-security-incidents).

+> [!NOTE]

+> The same alert can exist as part of an incident, as well as to be visible as a standalone alert.

+## Security incident

+[Further details and notes](alerts-overview.md#what-are-security-incidents)

+| Alert | Description | Severity |

+|--|--|--|

+| **Security incident detected suspicious virtual machines activity** | This incident indicates suspicious activity on your virtual machines. Multiple alerts from different Defender for Cloud plans have been triggered revealing a similar pattern on your virtual machines. This might indicate a threat actor has gained unauthorized access to your environment and is attempting to compromise it. | Medium/High |

+| **Security incident detected suspicious source IP activity** | This incident indicates that suspicious activity has been detected on the same source IP. Multiple alerts from different Defender for Cloud plans have been triggered on the same IP address, which increases the fidelity of malicious activity in your environment. Suspicious activity on the same IP address might indicate that an attacker has gained unauthorized access to your environment and is attempting to compromise it. | Medium/High |

+| **Security incident detected on multiple resources** | This incident indicates that suspicious activity had been detected on your cloud resources. Multiple alerts from different Defender for Cloud plan have been triggered, revealing similar attack methods were performed on your cloud resources. This might indicate a threat actor has gained unauthorized access to your environment and is attempting to compromise it. | Medium/High |

+| **Security incident detected suspicious user activity (Preview)** | This incident indicates suspicious user operations in your environment. Multiple alerts from different Defender for Cloud plans have been triggered by this user, which increases the fidelity of malicious activity in your environment. While this activity may be legitimate, a threat actor might utilize such operations to compromise resources in your environment. This might indicate that the account is compromised and is being used with malicious intent. | High |

+| **Security incident detected suspicious service principal activity (Preview)** | This incident indicates suspicious service principal operations in your environment. Multiple alerts from different Defender for Cloud plans have been triggered by this service principal, which increases the fidelity of malicious activity in your environment. While this activity may be legitimate, a threat actor might utilize such operations to compromise resources in your environment. This might indicate that the service principal is compromised and is being used with malicious intent. | High |

+| **Security incident detected suspicious crypto mining activity (Preview)** | Scenario 1: This incident indicates that suspicious crypto mining activity has been detected following suspicious user or service principal activity. Multiple alerts from different Defender for Cloud plans have been triggered on the same resource, which increases the fidelity of malicious activity in your environment. Suspicious account activity might indicate a threat actor gained unauthorized access to your environment, and the succeeding crypto mining activity may suggest that they successfully compromised your resource and are using it for mining cryptocurrencies, which can lead to increased costs for your organization. <br><br> Scenario 2: This incident indicates that suspicious crypto mining activity has been detected following a brute force attack on the same virtual machine resource. Multiple alerts from different Defender for Cloud plans have been triggered on the same resource, which increases the fidelity of malicious activity in your environment. The brute force attack on the virtual machine might indicate that a threat actor is attempting to gain unauthorized access to your environment, and the succeeding crypto mining activity may suggest they successfully compromised your resource and using it for mining cryptocurrencies, which can lead to increased costs for your organization. | High |

+| **Security incident detected suspicious Key Vault activity (Preview)** | Scenario 1: This incident indicates that suspicious activity has been detected in your environment related to the usage of Key Vault. Multiple alerts from different Defender for Cloud plans have been triggered by this user or service principal, which increases the fidelity of malicious activity in your environment. Suspicious Key Vault activity might indicate that a threat actor is attempting to gain access to your sensitive data, such as keys, secrets, and certificates, and the account is compromised and is being used with malicious intent. <br><br> Scenario 2: This incident indicates that suspicious activity has been detected in your environment related to the usage of Key Vault. Multiple alerts from different Defender for Cloud plans have been triggered from the same IP address, which increases the fidelity of malicious activity in your environment. Suspicious Key Vault activity might indicate that a threat actor is attempting to gain access to your sensitive data, such as keys, secrets, and certificates, and the account is compromised and is being used with malicious intent. <br><br> Scenario 3: This incident indicates that suspicious activity has been detected in your environment related to the usage of Key Vault. Multiple alerts from different Defender for Cloud plans have been triggered on the same resource, which increases the fidelity of malicious activity in your environment. Suspicious Key Vault activity might indicate that a threat actor is attempting to gain access to your sensitive data, such as keys, secrets, and certificates, and the account is compromised and is being used with malicious intent. | High |

+| **Security incident detected suspicious SAS activity (Preview)** | This incident indicates that suspicious activity has been detected following the potential misuse of a SAS token. Multiple alerts from different Defender for Cloud plans have been triggered on the same resource, which increases the fidelity of malicious activity in your environment. The usage of a SAS token can indicate that a threat actor has gained unauthorized access to your storage account and is attempting to access or exfiltrate sensitive data. | High |

+| **Security incident detected anomalous geographical location activity (Preview)** | Scenario 1: This incident indicates that anomalous geographical location activity has been detected in your environment. Multiple alerts from different Defender for Cloud plans have been triggered on the same resource, which increases the fidelity of malicious activity in your environment. Suspicious activity originating from anomalous locations might indicate that a threat actor gained unauthorized access to your environment and is attempting to compromise it. <br><br> Scenario 2: This incident indicates that anomalous geographical location activity has been detected in your environment. Multiple alerts from different Defender for Cloud plans have been triggered from the same IP address, which increases the fidelity of malicious activity in your environment. Suspicious activity originating from anomalous locations might indicate that a threat actor gained unauthorized access to your environment and is attempting to compromise it. | High |

+| **Security incident detected suspicious IP activity (Preview)** | Scenario 1: This incident indicates that suspicious activity has been detected originating from a suspicious IP address. Multiple alerts from different Defender for Cloud plans have been triggered from the same IP address, which increases the fidelity of malicious activity in your environment. Suspicious activity originating from a suspicious IP address might indicate that an attacker has gained unauthorized access to your environment and is attempting to compromise it. <br><br> Scenario 2: This incident indicates that suspicious activity has been detected originating from a suspicious IP address. Multiple alerts from different Defender for Cloud plans have been triggered on the same user or service principal, which increases the fidelity of malicious activity in your environment. Suspicious activity originating from a suspicious IP address can indicate that an attacker has gained unauthorized access to your environment and is attempting to compromise it. | High |

+| **Security incident detected suspicious fileless attack activity (Preview)** | This incident indicates that a fileless attack toolkit has been detected on a virtual machine following a potential exploit attempt on the same resource. Multiple alerts from different Defender for Cloud plans have been triggered on the same virtual machine, which increases the fidelity of malicious activity in your environment. The presence of a fileless attack toolkit on the virtual machine might indicate that a threat actor has gained unauthorized access to your environment and is attempting to evade detection while carrying out further malicious activities. | High |

+| **Security incident detected suspicious DDOS activity (Preview)** | This incident indicates that suspicious Distributed Denial of Service (DDOS) activity has been detected in your environment. DDOS attacks are designed to overwhelm your network or application with a high volume of traffic, causing it to become unavailable to legitimate users. Multiple alerts from different Defender for Cloud plans have been triggered on the same IP address, which increases the fidelity of malicious activity in your environment. | High |

+| **Security incident detected suspicious data exfiltration activity (Preview)** | Scenario 1: This incident indicates that suspicious data exfiltration activity has been detected following suspicious user or service principal activity. Multiple alerts from different Defender for Cloud plans have been triggered on the same resource, which increases the fidelity of malicious activity in your environment. Suspicious account activity might indicate that a threat actor gained unauthorized access to your environment, and the succeeding data exfiltration activity may suggest that they are attempting to steal sensitive information. <br><br> Scenario 2: This incident indicates that suspicious data exfiltration activity has been detected following suspicious user or service principal activity. Multiple alerts from different Defender for Cloud plans have been triggered from the same IP address, which increases the fidelity of malicious activity in your environment. Suspicious account activity might indicate that a threat actor gained unauthorized access to your environment, and the succeeding data exfiltration activity may suggest that they are attempting to steal sensitive information. <br><br> Scenario 3: This incident indicates that suspicious data exfiltration activity has been detected following unusual password reset on a virtual machine. Multiple alerts from different Defender for Cloud plans have been triggered from the same IP address, which increases the fidelity of malicious activity in your environment. Suspicious account activity might indicate that a threat actor gained unauthorized access to your environment, and the succeeding data exfiltration activity may suggest that they are attempting to steal sensitive information. | High |

+| **Security incident detected suspicious API activity (Preview)** | This incident indicates that suspicious API activity has been detected. Multiple alerts from Defender for Cloud have been triggered on the same resource, which increases the fidelity of malicious activity in your environment. Suspicious API usage might indicate that a threat actor is attempting to access sensitive information or execute unauthorized actions. | High |

+| **Security incident detected suspicious Kubernetes cluster activity (Preview)** | This incident indicates that suspicious activity has been detected on your Kubernetes cluster following suspicious user activity. Multiple alerts from different Defender for Cloud plans have been triggered on the same cluster, which increases the fidelity of malicious activity in your environment. The suspicious activity on your Kubernetes cluster might indicate that a threat actor has gained unauthorized access to your environment and is attempting to compromise it. | High |

+| **Security incident detected suspicious storage activity (Preview)** | Scenario 1: This incident indicates that suspicious storage activity has been detected following suspicious user or service principal activity. Multiple alerts from different Defender for Cloud plans have been triggered on the same resource, which increases the fidelity of malicious activity in your environment. Suspicious account activity might indicate that a threat actor gained unauthorized access to your environment, and the succeeding suspicious storage activity may suggest they are attempting to access potentially sensitive data. <br><br> Scenario 2: This incident indicates that suspicious storage activity has been detected following suspicious user or service principal activity. Multiple alerts from different Defender for Cloud plans have been triggered from the same IP address, which increases the fidelity of malicious activity in your environment. Suspicious account activity might indicate that a threat actor gained unauthorized access to your environment, and the succeeding suspicious storage activity may suggest they are attempting to access potentially sensitive data. | High |

+| **Security incident detected suspicious Azure toolkit activity (Preview)** | This incident indicates that suspicious activity has been detected following the potential usage of an Azure toolkit. Multiple alerts from different Defender for Cloud plans have been triggered on the same user or service principal, which increases the fidelity of malicious activity in your environment. The usage of an Azure toolkit can indicate that an attacker has gained unauthorized access to your environment and is attempting to compromise it. | High |

+| **Security incident detected compromised machine** | This incident indicates suspicious activity on one or more of your virtual machines. Multiple alerts from different Defender for Cloud plans have been triggered in chronological order on the same resource, following the MITRE ATT&CK framework. This might indicate a threat actor has gained unauthorized access to your environment and successfully compromised this machine.| Medium/High |

+| **Security incident detected compromised machine with botnet communication** | This incident indicates suspicious botnet activity on your virtual machine. Multiple alerts from different Defender for Cloud plans have been triggered in chronological order on the same resource, following the MITRE ATT&CK framework. This might indicate a threat actor has gained unauthorized access to your environment and is attempting to compromise it. | Medium/High |

+| **Security incident detected compromised machines with botnet communication** | This incident indicates suspicious botnet activity on your virtual machines. Multiple alerts from different Defender for Cloud plans have been triggered in chronological order on the same resource, following the MITRE ATT&CK framework. This might indicate a threat actor has gained unauthorized access to your environment and is attempting to compromise it. | Medium/High |

+| **Security incident detected compromised machine with malicious outgoing activity** | This incident indicates suspicious outgoing activity on your virtual machine. Multiple alerts from different Defender for Cloud plans have been triggered in chronological order on the same resource, following the MITRE ATT&CK framework. This might indicate a threat actor has gained unauthorized access to your environment and is attempting to compromise it. | Medium/High |

+| **Security incident detected compromised machines** | This incident indicates suspicious activity on one or more of your virtual machines. Multiple alerts from different Defender for Cloud plans have been triggered in chronological order on the same resources, following the MITRE ATT&CK framework. This might indicate a threat actor has gained unauthorized access to your environment and successfully compromised these machines. | Medium/High |

+| **Security incident detected compromised machines with malicious outgoing activity** | This incident indicates suspicious outgoing activity from your virtual machines. Multiple alerts from different Defender for Cloud plans have been triggered in chronological order on the same resources, following the MITRE ATT&CK framework. This might indicate a threat actor has gained unauthorized access to your environment and is attempting to compromise it. | Medium/High |

+| **Security incident detected on multiple machines** | This incident indicates suspicious activity on one or more of your virtual machines. Multiple alerts from different Defender for Cloud plans have been triggered in chronological order on the same resource, following the MITRE ATT&CK framework. This might indicate a threat actor has gained unauthorized access to your environment and is attempting to compromise it. | Medium/High |

+| **Security incident with shared process detected** | Scenario 1: This incident indicates suspicious activity on your virtual machine. Multiple alerts from different Defender for Cloud plans have been triggered sharing the same process. This might indicate a threat actor has gained unauthorized access to your environment and is attempting to compromise it. <br><br> Scenario 2: This incident indicates suspicious activity on your virtual machines. Multiple alerts from different Defender for Cloud plans have been triggered sharing the same process. This might indicate a threat actor has gained unauthorized access to your environment and is attempting to compromise it. | Medium/High |

+## Next steps

+[Manage security incidents in Microsoft Defender for Cloud](incidents.md)

+- [Defender for DevOps Pull Request annotations in Azure DevOps repositories now includes Infrastructure as Code misconfigurations](#defender-for-devops-pull-request-annotations-in-azure-devops-repositories-now-includes-infrastructure-as-code-misconfigurations)

+### Defender for DevOps Pull Request annotations in Azure DevOps repositories now includes Infrastructure as Code misconfigurations

+Defender for DevOps has expanded its Pull Request (PR) annotation coverage in Azure DevOps to include Infrastructure as Code (IaC) misconfigurations that are detected in ARM and Bicep templates.

+Developers can now see annotations for IaC misconfigurations directly in their PRs. Developers can also remediate critical security issues before the infrastructure is provisioned into cloud workloads. To simplify remediation, developers are provided with a severity level, misconfiguration description, and remediation instructions within each annotation.

+Previously, coverage for Defender for DevOps PR annotations in Azure DevOps only included secrets.

+Learn more about [Defender for DevOps](defender-for-devops-introduction.md) and [Pull Request annotations](enable-pull-request-annotations.md).

+In order for Azure Digital Twins to send data to other Azure services via endpoints or data history, the receiving service must have either public network access enabled or the Trusted Microsoft Service option enabled. For data history, the data history connection must be configured with public network access enabled on the Event Hub and Azure Data Explorer instances. Once data history is configured, the Event Hub and Azure Data Explorer firewall and security settings will need to be configured manually.

+Azure Event Hubs already spreads the risk of catastrophic failures of individual machines or even complete racks across clusters that span multiple failure domains within a datacenter. It implements transparent failure detection and failover mechanisms such that the service will continue to operate within the assured service-levels and typically without noticeable interruptions in the event of such failures. If you create an Event Hubs namespace with [availability zones](../availability-zones/az-overview.md) enabled, you reduce the risk of outage further and enable high availablity. With availability zones, the outage risk is further spread across three physically separated facilities, and the service has enough capacity reserves to instantly cope with the complete, catastrophic loss of the entire facility.

+## Quotas and limits

+| Limit | Basic | Standard | Premium | Dedicated |

+| -- | -- | -- | -- | |

+| Size of compacted event hub | N/A | 1 GB per partition | 250 GB per partition | 250 GB per partition |

+For other quotas and limits, see [Event Hubs quotas and limits](event-hubs-quotas.md).

+## Quotas and limits

+| Limit | Basic | Standard | Premium | Dedicated |

+| -- | -- | -- | -- | |

+| Size of compacted event hub | N/A | 1 GB per partition | 250 GB per partition | 250 GB per partition |

+For other quotas and limits, see [Event Hubs quotas and limits](event-hubs-quotas.md).

+> [!TIP]

+> Remember that it may also be necessary to configure firewall rules on the client side to support the connection.

+|Inbound DNAT<br><br>(FTP client on Internet, server in VNet) |DNAT rule to configure:<br>- DNAT From Internet Source to VNet IP port 21<br><br>Network rule to configure:<br>- Allow **from** FTP server VNet IP **to** client Internet destination IP at destination client configured active ftp client port ranges |**Pre-Condition**:<br>Configure FTP server to accept data and control channels from different source IP addresses.<br><br>Tip: Azure Firewall supports limited number of DNAT rules. It is important to configure the FTP server to use a small port range on the Data channel.<br><br>DNAT Rules to configure:<br>- DNAT From Internet Source to VNet IP port 21<br>- DNAT From Internet Source to VNet IP \<Range of Data Ports> |

+ Title: Compare pricing between Azure Front Door tiers

+description: This article describes the billing model for Azure Front Door and compares the pricing for the Standard, Premium and (classic) tiers.

+# Compare pricing between Azure Front Door tiers

+> [!NOTE]

+> Prices shown in this article are examples and are for illustration purposes only. For pricing information according to your region, see the [Pricing page](https://azure.microsoft.com/pricing/details/frontdoor/)

+Azure Front Door has three tiers: Standard, Premium, and (classic). This article describes the billing model for Azure Front Door and compares the pricing for the Standard, Premium and (classic) tiers. When migrating from Azure Front Door (classic) to Standard or Premium, we recommend you do a cost analysis to understand the pricing differences between the tiers. We show you how to evaluate cost that you can apply your environment.

+## Pricing model comparison

+| Pricing dimensions | Standard | Premium | Classic |

+|--|--|--|--|

+| Base fees (per month) | $35 | $330 | N/A |

+| Outbound data transfer from edge location to client (per GB) | Varies by 8 zones | Same as standard | - Varies by 5 zones </br>- Higher unit rates when compared to Standard/Premium |

+| Outbound data transfer from edge to the origin (per GB) | Varies by 8 zones | Same as standard | Free |

+| Incoming requests from client to Front DoorΓÇÖs edge location (per 10,000 requests) | Varies by 8 zones | - Varies by 8 zones </br>- Higher unit rate than Standard | Free |

+| First 5 routing rules (per hour) | Free | Free | $0.03 |

+| Per additional routing rule (per hour) | Free | Free | $0.012 |

+| Inbound data transfer (per GB) | Free | Free | $0.01 |

+| Web Application Firewall custom rules | Free | Free | - $5/month/policy </br>- $1/month & $0.06 per million requests </br></br>For more information, see [Azure Web Application Firewall pricing](https://azure.microsoft.com/pricing/details/web-application-firewall/) |

+| Web Application Firewall managed rules | Free | Free | - $5/month/policy </br>- $20/month +$1/million requests </br></br>For more information, see [Azure Web Application Firewall pricing](https://azure.microsoft.com/pricing/details/web-application-firewall/) |

+| Data transfer from an origin in Azure data center to Front Door's edge location | Free | Free | See [Bandwidth pricing](https://azure.microsoft.com/pricing/details/bandwidth/) |

+| Private link to origin | Not supported | Free | Not supported |

+| First 100 custom domains per month | Free | Free | Free |

+| Per additional custom domain (per month) | Free | Free | $5 |

+## Cost assessment

+> [!NOTE]

+> Azure Front Door Standard and Premium has a lower total cost of ownership than Azure Front Door (classic). If you have a request heavy workload, it's recommended to estimate the impact of the request meter of the new tiers. If you have multiple instance of Azure Front Door, it's recommended to estimate the impact of the base fee of the new tiers.

+The following are general guidance for getting the right metrics to estimate the cost of the new tiers.

+1. Pull the invoice for the Azure Front Door (classic) profile to get the monthly charges.

+1. Compute the Azure Front Door Standard/Premium pricing using the following table:

+ | Azure Front Door Standard/Premium meter | How to calculate from Azure Front Door (classic) metrics |

+ |--|--|

+ | Base fee | - If you need managed WAF rules, bot protection, or Private Link: **$330/month** </br> - If you only need custom WAF rules: **$35/month** |

+ | Requests | **For Standard:** </br>1. Go to your Azure Front Door (classic) profile, select **Metrics** from under *Monitor* in the left side menu pane. </br>2. Select the **Request Count** from the *Metrics* drop-down menu. </br> 3. To view regional metrics, you can apply a split to the data by selecting **Client Country** or **Client Region**. </br> 4. If you select *Client Country*, you need to map them to the corresponding Azure Front Door pricing zone. </br> :::image type="content" source="./media/understanding-pricing/request-count.png" alt-text="Screenshot of the request count metric for Front Door (classic)." lightbox="./media/understanding-pricing/request-count.png"::: </br> **For Premium:** </br>You can look at the **Request Count** and the **WAF Request Count** metric in the Azure Front Door (classic) profile. </br> :::image type="content" source="./media/understanding-pricing/waf-request-count.png" alt-text="Screenshot of the Web Application Firewall request count metric for Front Door (classic)." lightbox="./media/understanding-pricing/waf-request-count.png"::: |

+ | Egress from Azure Front Door edge to client | You can obtain this data from your Azure Front Door (classic) invoice or from the **Billable Response Size** metric in the Azure Front Door (classic) profile. To get a more accurate estimation, apply split by *Client Count* or *Client Region*.</br> :::image type="content" source="./media/understanding-pricing/billable-response-size.png" alt-text="Screenshot of the billable response size metric for Front Door (classic)." lightbox="./media/understanding-pricing/billable-response-size.png"::: |

+ | Ingress from Azure Front Door edge to origin | You can obtain this data from your Azure Front Door (classic) invoice. Refer to the quantities for Data transfer from client to edge location as an estimation. |

+1. Go to the [pricing calculator](https://azure.microsoft.com/pricing/calculator/?service=frontdoor-standard-premium).

+1. Select the appropriate Azure Front Door tier and zone.

+1. Calculate the total cost for the Azure Front Door Standard/Premium profile from the metrics you obtained in the previous step.

+## Example scenarios

+Azure Front Door Standard/Premium cost less than Azure Front Door (classic) in the first three scenarios. However, in scenario 4 and 5 there are situations where Azure Front Door Standard/Premium can incur higher charges than Azure Front Door (classic). In these scenarios, you can use the cost assessment to estimate the cost of the new tiers.

+### Scenario 1: A static website with custom WAF rules

+* 10 routing rules are configured.

+* 20 TB of outbound data transfer from Azure Front Door edge to client.

+* 200 million requests from client to Azure Front Door edge. (Including 100 million custom WAF requests and 10 custom rules).

+* Traffic mostly originates from North America and Europe.

+| Cost dimensions | Azure Front Door (classic) | Azure Front Door Standard |

+|--|--|--|

+| Base fee | $0 | $35 |

+| Routing rules | $43.80 | $0 |

+| WAF policy and rule sets | $75 = $15 (WAF policy + custom WAF rules) + $60 (requests)| $0 |

+| Requests | $0 | $300 |

+| Egress from Azure Front Door edge to client | $3,200 | $1,475 |

+| Ingress from Azure Front Door edge to origin | $0 | $0 |

+| Total | ~$3,319 | $1,810 |

+Azure Front Door Standard is ~45% cheaper than Azure Front Door (classic) for static websites with custom WAF rules because of the lower egress cost and the free routing rules.

+### Scenario 2: A static website with managed WAF rules

+* 30 routing rules are configured.

+* 20 TB of outbound data transfer from Azure Front Door edge to client.

+* 200 million requests from client to Azure Front Door edge (Including 100 million managed WAF requests).

+* Traffic mostly originates from Asia Pacific (including Japan).

+| Cost dimensions | Azure Front Door (classic) | Azure Front Door Premium |

+|--|--|--|

+| Base fee | $0 | $330 |

+| Routing rules | $219 | $0 |

+| WAF policy and rule sets | $220 = $20 (managed WAF rules and ruleset) + $200 (requests) | $0 |

+| Requests | $0 | $336 |

+| Egress from Azure Front Door edge to client | $4,700 | $2,125 |

+| Ingress from Azure Front Door edge to origin | $0 | $0 |

+| Total | ~$5,139 | $2,791 |

+Azure Front Door Premium is ~45% cheaper than Azure Front Door (classic) for static websites with managed WAF rules because of the lower egress cost and the free routing rules.

+### Scenario 3: File downloads

+* Two routing rules are configured.

+* 150 TB of outbound data transfer from Azure Front Door edge to client.

+* 1.5 million requests from client to Azure Front Door edge.

+* Traffic mostly originates from India.

+| Cost dimensions | Azure Front Door (classic) | Azure Front Door Standard |

+|--|--|--|

+| Base fee | $0 | $35 |

+| Routing rules | $0 | $0 |

+| WAF policy and rule sets | $0 | $0 |

+| Requests | $0 | $1.62 |

+| Egress from Azure Front Door edge to client | $39,500 | $12,690 |

+| Ingress from Azure Front Door edge to origin | $0 | $0 |

+| Total | ~$39,500 | $12,727 |

+Azure Front Door Standard is ~68% cheaper than Azure Front Door (classic) for file downloads because of the lower egress cost.

+### Scenario 4: Request heavy scenario with WAF protection

+* A dynamic E-commerce website with 150 routing rules is configured.

+* 20 TB of outbound data transfer from Azure Front Door edge to client.

+* 5 billion requests with 10 TB of ingress.

+* 2.4 billion WAF requests (1.2 billion managed WAF rule requests and 1.2 billion custom WAF rule requests).

+| Cost dimensions | Azure Front Door (classic) | Azure Front Door Premium |

+|--|--|--|

+| Base fee | $0 | $330 |

+| Routing rules | $1,314 | $0 |

+| WAF policy and rule sets | $1840 = $20 (WAF policy) + $1820 (requests) | $0 |

+| Requests | $0 | $6,748 |

+| Egress from Azure Front Door edge to client | $3,200 | $1,475|

+| Ingress from Azure Front Door edge to origin | $100 | $200 |

+| Total | $6,454 | $8,753 |

+In this comparison, Azure Front Door Premium is ~35% more expensive than Azure Front Door (classic) because of the higher request cost and the base fee. If the cost increase is significant, reach out to your Microsoft sales representative to discuss options.

+### Scenario 5: Social media application with multiple Front Door (classic) profiles with WAF protection

+* The application is designed in a micro-services architecture with static and dynamic traffic. Each micro service component is deployed in a separate Azure Front Door (classic) profile. In total, there are 80 Azure Front Door (classic) profiles (30 dev/test, 50 production).

+* In each profile, there are 10 routing rules configured to route traffic to different backends based on the path.

+* There are two WAF policies with two rule sets to protect the application from top CVE attacks.

+* 50 million requests per month.

+* 50 TB of outbound data transfer from Azure Front Door edge to client. (20 million requests get blocked by WAF).

+* Traffic mostly originates from North America.

+| Cost dimensions | Azure Front Door (classic) | Azure Front Door Premium |

+|--|--|--|

+| Base fee | $0 | $26,400 = $330 x 80 profiles |

+| Routing rules | $7,008 | $0 |

+| WAF policy and rule sets | $60 = $40 (WAF policy) + $20 (requests) | $0 |

+| Requests | $0 | $75 |

+| Egress from Azure Front Door edge to client | $7,700 | $3,425|

+| Ingress from Azure Front Door edge to origin | $2 | $4 |

+| Total | $14,770 | $29,904 |

+In this comparison, Azure Front Door Premium is more than twice as expensive than Azure Front Door (classic) because of the higher base fee.

+#### Suggestion to reduce cost

+* Check if all 80 instances of Azure Front Door (classic) are required. Remove unnecessary resources, such as temporary testing environments.

+* Migrate your most important Front Door (classic) profiles to Azure Front Door Standard/Premium based on the necessity of the features.

+* If you have multiple Front Door (classic) profiles, consider consolidating them into a single Azure Front Door Standard/Premium profile. This change can reduce the base fee and the routing rule cost. The capability to consolidate multiple Azure Front Door (classic) profiles into a single Azure Front Door Standard/Premium profile will be available soon.

+## Next steps

+* Learn about how [settings are mapped](tier-mapping.md) from Azure Front Door (classic) to Azure Front Door Standard/Premium.

+* Learn about [Azure Front Door (classic) tier migration](tier-migration.md).

+* Learn how to [migrate from Azure Front Door (classic) to Azure Front Door Standard/Premium](migrate-tier.md).

+ Title: Overview of the MedTech service device mapping - Azure Health Data Services

+### IoT Edge has low message throughput when geographically distant from IoT Hub

+#### Symptoms

+Azure IoT Edge devices that are geographically distant from Azure IoT Hub have a lower than expected message throughput.

+#### Cause

+High latency between the device and IoT Hub can cause a lower than expected message throughput. IoT Edge uses a default message batch size of 10. This limits the number of messages that are sent in a single batch, which increases the number of round trips between the device and IoT Hub.

+#### Solution

+Try increasing the IoT Edge Hub **MaxUpstreamBatchSize** environment variable. This allows more messages to be sent in a single batch, which reduces the number of round trips between the device and IoT Hub.

+To set Azure Edge Hub environment variables in the Azure portal:

+1. Navigate to your IoT Hub and select **Devices** under the **Device management** menu.

+1. Select the IoT Edge device that you want to update.

+1. Select **Set Modules**.

+1. Select **Runtime Settings**.

+1. In the **Edge Hub** module settings tab, add the **MaxUpstreamBatchSize** environment variable as type **Number** with a value of **20**.

+1. Select **Apply**.

+## Throttling limits

+The following table shows the enforced throttles for operations that are available in all Device Update for IoT Hub tiers. Values refer to an individual Device Update instance.

+|Device Update service API | Throttling Rate |

+|-||

+|GetGroups |30/min|

+|GetGroupDetails| 30/min|

+|GetBestUpdates per group| 30/min|

+|GetUpdateCompliance per group| 30/min|

+|GetAllUpdateCompliance |30/min|

+|GetSubgroupUpdateCompliance| 30/min|

+|GetSubgroupBestUpdates| 30/min|

+|CreateOrUpdateDeployment| 7/min |

+|DeleteDeployment| 7/min |

+|ProcessSubgroupDeployment | 7/min|

+| Tuples | five-tuple | two-tuple | three-tuple |

+Azure Load Balancer uses a five-tuple hash based distribution mode by default.

+The five-tuple consists of:

+*Figure: Default five-tuple hash based distribution*

+Session persistence is also known session affinity, source IP affinity, or client IP affinity. This distribution mode uses a two-tuple (source IP and destination IP) or three-tuple (source IP, destination IP, and protocol type) hash to route to backend instances. When using session persistence, connections from the same client go to the same backend instance within the backend pool.

+* **Client IP (2-tuple)** - Specifies that successive requests from the same client IP address are handled by the same backend instance.

+* **Client IP and protocol (3-tuple)** - Specifies that successive requests from the same client IP address and protocol combination are handled by the same backend instance.

+The following figure illustrates a two-tuple configuration. Notice how the two-tuple runs through the load balancer to virtual machine 1 (VM1). VM1 is backed up by VM2 and VM3.

+Source IP affinity with client IP and protocol (source IP affinity three-tuple), solves an incompatibility between Azure Load Balancer and Remote Desktop Gateway (RD Gateway).

+## Symptom: VMs behind the Load Balancer aren't responding to health probes

+To resolve this issue, sign-in to the participating VMs, and check if the VM state is healthy, and can respond to **PsPing** or **TCPing** from another VM in the pool. If the VM is unhealthy, or is unable to respond to the probe, you must rectify the issue and get the VM back to a healthy state before it can participate in load balancing.

+### Cause 2: Load Balancer backend pool VM isn't listening on the probe port

+If the VM is healthy, but isn't responding to the probe, then one possible reason could be that the probe port isn't open on the participating VM, or the VM isn't listening on that port.

+1. Sign-in to the backend VM.

+2. Open a command prompt and run the following command to validate there's an application listening on the probe port:

+3. If the port state isn't listed as **LISTENING**, configure the proper port.

+4. Alternatively, select another port that is listed as LISTENING and update load balancer configuration accordingly.

+If the firewall on the VM is blocking the probe port, or one or more network security groups configured on the subnet or on the VM, isn't allowing the probe to reach the port, the VM is unable to respond to the health probe.

+1. If the firewall is enabled, check if it's configured to allow the probe port. If not, configure the firewall to allow traffic on the probe port, and test again.

+- Check to make sure your VM firewall isn't blocking probe traffic originating from IP address `168.63.129.16`

+If all the preceding causes seem to be validated and resolved correctly, and the backend VM still doesn't respond to the health probe, then manually test for connectivity, and collect some traces to understand the connectivity.

+ - If no incoming packets are observed on the backend pool VM, there's potentially a network security groups or UDR mis-configuration blocking the traffic.

+If the preceding steps don't resolve the issue, open a [support ticket](https://azure.microsoft.com/support/options/).

+ ResourceGroupName = 'CreateIntLBQS-rg'

+ ResourceGroupName = 'CreateIntLBQS-rg'

+> [!NOTE]

+> The public IP address is used by the NAT gateway to provide outbound connectivity for the virtual machines in the backend pool. This is recommended when you create an internal load balancer and need the backend pool resources to have outbound connectivity. For more information, see [NAT gateway](load-balancer-outbound-connections.md).

+ ResourceGroupName = 'CreateIntLBQS-rg'

+In this section, you create the two virtual machines for the backend pool of the load balancer.

+ ## Command to create network interface for VMs ##

+ $nic = @{

+ }

+ $nicVM = New-AzNetworkInterface @nic

+ ## Create a virtual machine configuration for VMs ##

+ $vmsz = @{

+ VMName = "myVM$i"

+ VMSize = 'Standard_DS1_v2'

+ }

+ $vmos = @{

+ ComputerName = "myVM$i"

+ Credential = $cred

+ }

+ $vmimage = @{

+ PublisherName = 'MicrosoftWindowsServer'

+ Offer = 'WindowsServer'

+ Skus = '2019-Datacenter'

+ Version = 'latest'

+ }

+ $vmConfig = New-AzVMConfig @vmsz `

+ | Set-AzVMOperatingSystem @vmos -Windows `

+ | Set-AzVMSourceImage @vmimage `

+ | Add-AzVMNetworkInterface -Id $nicVM.Id

+ ## Create the virtual machine for VMs ##

+ $vm = @{

+ ResourceGroupName = 'CreateIntLBQS-rg'

+ Location = 'eastus'

+ VM = $vmConfig

+ Zone = "$i"

+ }

+}

+New-AzVM @vm -asjob

+ $ext = @{

+ Publisher = 'Microsoft.Compute'

+ ExtensionType = 'CustomScriptExtension'

+ ExtensionName = 'IIS'

+ ResourceGroupName = 'CreateIntLBQS-rg'

+ VMName = "myVM$i"

+ Location = 'eastus'

+ TypeHandlerVersion = '1.8'

+ SettingString = '{"commandToExecute":"powershell Add-WindowsFeature Web-Server; powershell Add-Content -Path \"C:\\inetpub\\wwwroot\\Default.htm\" -Value $($env:computername)"}'

+ }

+ Set-AzVMExtension @ext -AsJob

+8. Enter the IP address from the previous step into the address bar of the browser. The custom IIS Web server page is displayed.

+ :::image type="content" source="./media/quickstart-load-balancer-standard-internal-portal/load-balancer-test.png" alt-text="Screenshot of web browser showing default web page for load balanced VM" border="true":::

+To see the load balancer distribute traffic across all three VMs, you can force-refresh your web browser from the test machine.

+ "tier": "Kubernetes",

+ "name": "K1",

+The preview SAP built-in connector trigger named **Register SAP RFC server for trigger** is available in the Azure portal, but the trigger currently can't receive calls from SAP when deployed in Azure. To fire the trigger, you can run the workflow locally in Visual Studio Code. For Visual Studio Code setup requirements and more information, see [Create a Standard logic app workflow in single-tenant Azure Logic Apps using Visual Studio Code](create-single-tenant-workflows-visual-studio-code.md). You must also set up the following environment variables on the computer where you install Visual Studio Code:

+

+ - **WEBSITE_PRIVATE_IP**: Set this environment variable value to **127.0.0.1** as the localhost address.

+ - **WEBSITE_PRIVATE_PORTS**: Set this environment variable value to two free and usable ports on your local computer, separating the values with a comma (**,**), for example, **8080,8088**.

+The preview SAP built-in connector trigger named **Register SAP RFC server for trigger** is available in the Azure portal, but the trigger currently can't receive calls from SAP when deployed in Azure. To fire the trigger, you can run the workflow locally in Visual Studio Code. For Visual Studio Code setup requirements and more information, see [Create a Standard logic app workflow in single-tenant Azure Logic Apps using Visual Studio Code](create-single-tenant-workflows-visual-studio-code.md). You must also set up the following environment variables on the computer where you install Visual Studio Code:

+

+ - **WEBSITE_PRIVATE_IP**: Set this environment variable value to **127.0.0.1** as the localhost address.

+ - **WEBSITE_PRIVATE_PORTS**: Set this environment variable value to two free and usable ports on your local computer, separating the values with a comma (**,**), for example, **8080,8088**.

+* [Create example workflows for common SAP scenarios](sap-create-example-scenario-workflows.md)

+* [How to select algorithms](../v1/how-to-select-algorithms.md)

+* [Tutorial: Build a model in designer to predict auto prices](../v1/tutorial-designer-automobile-price-train-score.md)

+* [How to select algorithms](../v1/how-to-select-algorithms.md)

+* [Tutorial: Build a model in designer to predict auto prices](../v1/tutorial-designer-automobile-price-train-score.md)

+`[key1] [operator1] [value1]; [key2] [operator1] [value2];`

+- Space will be ignored between keywords

+A common use of scoring is to return the output as part of a predictive web service. For more information, see [this tutorial](../v1/tutorial-designer-automobile-price-deploy.md) on how to deploy a real-time endpoint based on a pipeline in Azure Machine Learning designer.

+A common use of scoring is to return the output as part of a predictive web service. For more information, see [this tutorial](../v1/tutorial-designer-automobile-price-deploy.md) on how to deploy a real-time endpoint based on a pipeline in Azure Machine Learning designer.

+When you [create a real-time inference pipeline](../v1/tutorial-designer-automobile-price-deploy.md#create-a-real-time-inference-pipeline) from your training pipeline, the Web Service Input and Web Service Output components will be automatically added to show where user data enters the pipeline and where data is returned.

+After you submit the pipeline and the run finishes successfully, you can [deploy the real-time endpoint](../v1/tutorial-designer-automobile-price-deploy.md#deploy-the-real-time-endpoint).

+Learn more about [deploying the real-time endpoint](../v1/tutorial-designer-automobile-price-deploy.md#deploy-the-real-time-endpoint).

+* For more information on using the CLI, see [Use the CLI extension for Azure Machine Learning](how-to-configure-cli.md).

+* The [Azure Machine Learning Python SDK v1](/python/api/overview/azure/ml/install) or [Azure CLI extension for machine learning v1](./v1/reference-azure-machine-learning-cli.md).

+With the __new v2 API__, most operations use ARM. So enabling a private endpoint on your workspace doesn't provide the same level of network isolation. Operations that use ARM communicate over public networks, and include any metadata (such as your resource IDs) or parameters used by the operation. For example, the [create or update job](/rest/api/azureml/2023-04-01/jobs/create-or-update) api sends metadata, and [parameters](./reference-yaml-job-command.md).

+The Azure CLI [extension v1 for machine learning](./v1/reference-azure-machine-learning-cli.md) provides the [az ml workspace update](/cli/azure/ml(v1)/workspace#az-ml(v1)-workspace-update) command. To disable the parameter for a workspace, add the parameter `--v1-legacy-mode False`.

+type: azure_sql_db

+* The [Azure CLI extension for Machine Learning service (v2)](how-to-configure-cli.md), [Azure Machine Learning Python SDK](/python/api/overview/azure/ai-ml-readme), or the [Azure Machine Learning Visual Studio Code extension](how-to-setup-vs-code.md).

+This article is based on the [image_classification_keras_minist_convnet.ipynb](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/2e_image_classification_keras_minist_convnet/image_classification_keras_minist_convnet.ipynb) notebook found in the `sdk/python/jobs/pipelines/2e_image_classification_keras_minist_convnet` directory of the [Azure Machine Learning Examples](https://github.com/azure/azureml-examples) repository.

+- Train a neural network for image classification using training data

+For each component, you need to prepare the following:

+By using `command_component()` function as a decorator, you can easily define the component's interface, metadata and code to execute from a Python function. Each decorated Python function will be transformed into a single static specification (YAML) that the pipeline service can process.

+After defining the training function successfully, you can use `@command_component` in Azure Machine Learning SDK v2 to wrap your function as a component, which can be used in Azure Machine Learning pipelines.

+To create or overwrite a named compute resource, you'll use a PUT request. In the following, in addition to the now-familiar replacements of `YOUR-SUBSCRIPTION-ID`, `YOUR-RESOURCE-GROUP`, `YOUR-WORKSPACE-NAME`, and `YOUR-ACCESS-TOKEN`, replace `YOUR-COMPUTE-NAME`, and values for `location`, `vmSize`, `vmPriority`, `scaleSettings`, `adminUserName`, and `adminUserPassword`. As specified in the reference at [Machine Learning Compute - Create Or Update SDK Reference](/rest/api/azureml/2023-04-01/workspaces/create-or-update), the following command creates a dedicated, single-node Standard_D1 (a basic CPU compute resource) that will scale down after 30 minutes:

+- Explore [Azure Machine Learning with Jupyter notebooks](../machine-learning/samples-notebooks.md).

+> The compute cluster used to build Docker images needs to be able to access the package repositories that are used to train and deploy your models. You may need to add network security rules that allow access to public repos, [use private Python packages](concept-vulnerability-management.md#using-a-private-package-repository), or use [custom Docker images (SDK v1)](v1/how-to-train-with-custom-image.md?view=azureml-api-1&preserve-view=true) that already include the packages.

+* Azure Machine Learning designer: use the designer to train and deploy machine learning models without writing any code. Drag and drop datasets and components to create ML pipelines.

+Azure Container for PyTorch (ACPT) now includes **Nebula**, a fast, simple, disk-less, model-aware checkpoint tool. Nebula offers a simple, high-speed checkpointing solution for distributed large-scale model training jobs using PyTorch. By utilizing the latest distributed computing technologies, Nebula can reduce checkpoint times from hours to seconds - potentially saving 95% to 99.9% of time. Large-scale training jobs can greatly benefit from Nebula's performance.

+* An Azure Machine Learning script run configuration file. If you don't have one, you can follow [this resource](./v1/how-to-set-up-training-targets.md)

+- [Older versions of `azure-mgmt-authorization` package doesn't work with `AzureMLOnBehalfOfCredential`](#older-versions-of-azure-mgmt-authorization-package-doesnt-work-with-azuremlonbehalfofcredential)

+For more information, see [Permissions required for the `feature store materialization managed identity` role](how-to-setup-access-control-feature-store.md#permissions-required-for-the-feature-store-materialization-managed-identity-role).

+When the feature store is updated using the SDK/CLI, the update fails with the following error message:

+When you update the feature store using the same user-assigned managed identity as the materialization identity in the update request, while using the ARM ID in format (A), the update will fail with the error above.

+### Older versions of `azure-mgmt-authorization` package doesn't work with `AzureMLOnBehalfOfCredential`

+#### Symptom

+When you use the `setup_storage_uai` script provided in the *featurestore_sample* folder in the azureml-examples repository, the script fails with the error message:

+`AttributeError: 'AzureMLOnBehalfOfCredential' object has no attribute 'signed_session'`

+#### Solution:

+Check the version of the `azure-mgmt-authorization` package that is installed and make sure you're using a recent version, such as 3.0.0 or later. The old version, such as 0.61.0, doesn't work with `AzureMLOnBehalfOfCredential`.

+- [Can't find transformation class](#cant-find-transformation-class)

+Before you register a feature set into the feature store, define the feature set spec locally and run `<feature_set_spec>.to_spark_dataframe()` to validate it.

+When user runs `<feature_set_spec>.to_spark_dataframe()` , various schema validation failures may occur if the schema of the feature set dataframe isn't aligned with the definition in the feature set spec.

+- If the feature source data is of type ΓÇ£csvΓÇ¥, make sure the CSV files are generated with column headers.

+### Can't find transformation class

+- [[Observation Data isn't Joined with any feature values](#observation-data-isnt-joined-with-any-feature-values)]

+- [`generate_feature_retrieval_spec()` fails due to use of local feature set specification](#generate_feature_retrieval_spec-fails-due-to-use-of-local-feature-set-specification)

+- [`get_offline_features() query` takes a long time](#get_offline_features-query-takes-a-long-time)

+### Observation Data isn't joined with any feature values

+- format isn't correct.

+### `generate_feature_retrieval_spec()` fails due to use of local feature set specification

+#### Symptom:

+If you run the following python code to generate a feature retrieval spec on a given list of features.

+```python

+featurestore.generate_feature_retrieval_spec(feature_retrieval_spec_folder, features)

+```

+You receive the error:

+`AttributeError: 'FeatureSetSpec' object has no attribute 'id'`

+#### Solution:

+A feature retrieval spec can only be generated using feature sets registered in Feature Store. If the features list contains features defined by a local feature set specification, the `generate_feature_retrieval_spec()` fails with the error message above.

+To fix the issue:

+- Register the local feature set specification as feature set in the feature store

+- Get the registered the feature set

+- Create feature lists again using only features from registered feature sets

+- Generate the feature retrieval spec using the new features list

+### `get_offline_features() query` takes a long time

+#### Symptom:

+Running `get_offline_features` to generate training data using a few features from feature store takes a long time to finish.

+#### Solutions:

+Check the following configurations:

+- For each feature set used in the query, does it have `temporal_join_lookback` set in the feature set specification. Set its value to a smaller value.

+- If the size and timestamp window on the observation dataframe are large, configure the notebook session (or the job) to increase the size (memory and core) of driver and executor, and increase the number of executors.

+- [Streaming job results to notebook fails](#streaming-job-results-to-notebook-fails)

+- [Invalid Spark configuration](#invalid-spark-configuration)

+### Streaming job results to notebook fails

+#### Symptom:

+When using the feature store CRUD client to stream materialization job results to notebook using `fs_client.jobs.stream(ΓÇ£<job_id>ΓÇ¥)`, the SDK call fails with an error

+```

+HttpResponseError: (UserError) A job was found, but it is not supported in this API version and cannot be accessed.

+Code: UserError

+Message: A job was found, but it is not supported in this API version and cannot be accessed.

+```

+#### Solution:

+When the materialization job is created (for example, by a backfill call), it may take a few seconds for the job the to properly initialize. Run the `jobs.stream()` command again in a few seconds. The issue should be gone.

+### Invalid Spark configuration

+#### Symptom:

+A materialization job fails with the following error message:

+```python

+Synapse job submission failed due to invalid spark configuration request

+{

+"Message":"[..] Either the cores or memory of the driver, executors exceeded the SparkPool Node Size.\nRequested Driver Cores:[4]\nRequested Driver Memory:[36g]\nRequested Executor Cores:[4]\nRequested Executor Memory:[36g]\nSpark Pool Node Size:[small]\nSpark Pool Node Memory:[28]\nSpark Pool Node Cores:[4]"

+}

+```

+#### Solution:

+Update the `materialization_settings.spark_configuration{}` of the feature set. Make sure the following parameters are using memory size and number of cores less than what is provided by the instance type (defined by `materialization_settings.resource`)

+`spark.driver.cores`

+`spark.driver.memory`

+`spark.executor.cores`

+`spark.executor.memory`

+For example, on instance type *standard_e8s_v3*, the following spark configuration one of the valid options.

+

+```python

+transactions_fset_config.materialization_settings = MaterializationSettings(

+ offline_enabled=True,

+ resource = MaterializationComputeResource(instance_type="standard_e8s_v3"),

+ spark_configuration = {

+ "spark.driver.cores": 4,

+ "spark.driver.memory": "36g",

+ "spark.executor.cores": 4,

+ "spark.executor.memory": "36g",

+ "spark.executor.instances": 2

+ },

+ schedule = None,

+)

+fs_poller = fs_client.feature_sets.begin_create_or_update(transactions_fset_config)

+```

+ * Open the [Azure Machine Learning studio UI](https://ml.azure.com/) resource of your Azure Machine Learning workspace

+For an example of deploying a model as a web service, see [Tutorial: Train and deploy a model](tutorial-train-deploy-notebook.md).

+When you deploy a trained model in the designer, you can [deploy the model as a real-time endpoint](tutorial-designer-automobile-price-deploy.md). A real-time endpoint commonly receives a single request via the REST endpoint and returns a prediction in real-time. This is in contrast to batch processing, which processes multiple values at once and saves the results after completion to a datastore.

+The [Azure Machine Learning CLI v1](reference-azure-machine-learning-cli.md) is an extension to the Azure CLI, a cross-platform command-line interface for the Azure platform. This extension provides commands to automate your machine learning activities.

+ * [Start, monitor, and cancel training runs](how-to-track-monitor-analyze-runs.md)

+ * [Log metrics for training runs](how-to-log-view-metrics.md)

+ * [Track experiments with MLflow](how-to-use-mlflow.md)

+* For **DevOps** or **MLOps**, you can monitor information generated by models deployed as web services to identify problems with the deployments and gather data submitted to the service. For more information, see [Collect model data](how-to-enable-data-collection.md) and [Monitor with Application Insights](how-to-enable-app-insights.md).

+* [Tutorial: Train and deploy a model](tutorial-train-deploy-notebook.md)

+ + the [designer](tutorial-designer-automobile-price-train-score.md#import-data)

+A theme of the above steps is that your retraining should be automated, not ad hoc. [Azure Machine Learning pipelines](../concept-ml-pipelines.md) are a good answer for creating workflows relating to data preparation, training, validation, and deployment. Read [Retrain models with Azure Machine Learning designer](how-to-retrain-designer.md) to see how pipelines and the Azure Machine Learning designer fit into a retraining scenario.

+You can also deploy models directly in the designer to skip model registration and file download steps. This can be useful for rapid deployment. For more information see, [Deploy a model with the designer](tutorial-designer-automobile-price-deploy.md).

+* [Train a model in the designer](tutorial-designer-automobile-price-train-score.md)

+Learn the designer fundamentals with this [Tutorial: Predict automobile price with the designer](tutorial-designer-automobile-price-train-score.md).

+|Replication Lag|replication_lag|Seconds|Replication lag is the number of seconds the replica is behind in replaying the transactions received from the source server. This metric is calculated from "Seconds_behind_Master" from the command "SHOW SLAVE STATUS" and is available for replica servers only. For more information, see "[Monitor replication latency](../how-to-troubleshoot-replication-latency.md)"|

+- You can also monitor the replication latency by following the steps mentioned [here](../how-to-troubleshoot-replication-latency.md).

+- To troubleshoot high replication latency observed in Metrics, visit the [link](../how-to-troubleshoot-replication-latency.md#common-scenarios-for-high-replication-latency).

+ Title: Troubleshoot replication latency - Azure Database for MySQL - Flexible Server

+description: Learn how to troubleshoot replication latency by using Azure Database for MySQL - Flexible Server read replicas.

+keywords: mysql, troubleshoot, replication latency in seconds

+# Troubleshoot replication latency in Azure Database for MySQL - flexible Server

+The [read replica](concepts-read-replicas.md) feature allows you to replicate data from an Azure Database for MySQL server to a read-only replica server. You can scale out workloads by routing read and reporting queries from the application to replica servers. This setup reduces the pressure on the source server. It also improves overall performance and latency of the application as it scales.

+Replicas are updated asynchronously by using the MySQL engine's native binary log (binlog) file position-based replication technology. For more information, see [MySQL binlog file position-based replication configuration overview](https://dev.mysql.com/doc/refman/5.7/en/binlog-replication-configuration-overview.html).

+The replication lag on the secondary read replicas depends several factors. These factors include but aren't limited to:

+- Network latency.

+- Transaction volume on the source server.

+- Compute tier of the source server and secondary read replica server.

+- Queries running on the source server and secondary server.

+In this article, you learn how to troubleshoot replication latency in Azure Database for MySQL. You'll also understand some common causes of increased replication latency on replica servers.

+> [!NOTE]

+> This article contains references to the term *slave*, a term that Microsoft no longer uses. When the term is removed from the software, we'll remove it from this article.

+## Replication concepts

+When a binary log is enabled, the source server writes committed transactions into the binary log. The binary log is used for replication. It's turned on by default for all newly provisioned servers that support up to 16 TB of storage. On replica servers, two threads run on each replica server. One thread is the *IO thread*, and the other is the *SQL thread*:

+- The IO thread connects to the source server and requests updated binary logs. This thread receives the binary log updates. Those updates are saved on a replica server, in a local log called the *relay log*.

+- The SQL thread reads the relay log and then applies the data changes on replica servers.

+## Monitoring replication latency

+Azure Database for MySQL provides the metric for replication lag in seconds in [Azure Monitor](concepts-monitoring.md). This metric is available only on read replica servers. It's calculated by the seconds_behind_master metric that's available in MySQL.

+To understand the cause of increased replication latency, connect to the replica server by using [MySQL Workbench](connect-workbench.md) or [Azure Cloud Shell](https://shell.azure.com). Then run following command.

+> [!NOTE]

+> In your code, replace the example values with your replica server name and admin username. The admin username requires `@\<servername>` for Azure Database for MySQL.

+```azurecli-interactive

+mysql --host=myreplicademoserver.mysql.database.azure.com --user=myadmin@mydemoserver -p

+```

+Here's how the experience looks in the Cloud Shell terminal:

+```bash

+Requesting a Cloud Shell.Succeeded.

+Connecting terminal...

+Welcome to Azure Cloud Shell

+Type "az" to use Azure CLI

+Type "help" to learn about Cloud Shell

+user@Azure:~$mysql -h myreplicademoserver.mysql.database.azure.com -u myadmin@mydemoserver -p

+Enter password:

+Welcome to the MySQL monitor. Commands end with ; or \g.

+Your MySQL connection id is 64796

+Server version: 5.6.42.0 Source distribution

+Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

+Oracle is a registered trademark of Oracle Corporation and/or its

+affiliates. Other names may be trademarks of their respective

+owners.

+Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

+mysql>

+```

+In the same Cloud Shell terminal, run the following command:

+```sql

+mysql> SHOW SLAVE STATUS;

+```

+Here's a typical output:

+

+> [!div class="mx-imgBorder"]

+> :::image type="content" source="./media/how-to-troubleshoot-replication-latency/show-status.png" alt-text="Monitoring replication latency":::

+The output contains numerous information. Normally, you need to focus on only the rows that the following table describes.

+|Metric|Description|

+|||

+|Slave_IO_State| Represents the current status of the IO thread. Normally, the status is "Waiting for master to send event" if the source (master) server is synchronizing. A status such as "Connecting to master" indicates that the replica lost the connection to the source server. Make sure the source server is running, or check to see whether a firewall is blocking the connection.|

+|Master_Log_File| Represents the binary log file to which the source server is writing.|

+|Read_Master_Log_Pos| Indicates where the source server is writing in the binary log file.|

+|Relay_Master_Log_File| Represents the binary log file that the replica server is reading from the source server.|

+|Slave_IO_Running| Indicates whether the IO thread is running. The value should be `Yes`. If the value is `NO`, then the replication is likely broken.|

+|Slave_SQL_Running| Indicates whether the SQL thread is running. The value should be `Yes`. If the value is `NO`, then the replication is likely broken.|

+|Exec_Master_Log_Pos| Indicates the position of the Relay_Master_Log_File that the replica is applying. If there's latency, then this position sequence should be smaller than Read_Master_Log_Pos.|

+|Relay_Log_Space|Indicates the total combined size of all existing relay log files. You can check the upper limit size by querying `SHOW GLOBAL VARIABLES` like `relay_log_space_limit`.|

+|Seconds_Behind_Master| Displays replication latency in seconds.|

+|Last_IO_Errno|Displays the IO thread error code, if any. For more information about these codes, see the [MySQL server error message reference](https://dev.mysql.com/doc/mysql-errors/5.7/en/server-error-reference.html).|

+|Last_IO_Error| Displays the IO thread error message, if any.|

+|Last_SQL_Errno|Displays the SQL thread error code, if any. For more information about these codes, see the [MySQL server error message reference](https://dev.mysql.com/doc/mysql-errors/5.7/en/server-error-reference.html).|

+|Last_SQL_Error|Displays the SQL thread error message, if any.|

+|Slave_SQL_Running_State| Indicates the current SQL thread status. In this state, `System lock` is normal. It's also normal to see a status of `Waiting for dependent transaction to commit`. This status indicates that the replica is waiting for the source server to update committed transactions.|

+If Slave_IO_Running is `Yes` and Slave_SQL_Running is `Yes`, then the replication is running fine.

+Next, check Last_IO_Errno, Last_IO_Error, Last_SQL_Errno, and Last_SQL_Error. These fields display the error number and error message of the most-recent error that caused the SQL thread to stop. An error number of `0` and an empty message means there's no error. Investigate any nonzero error value by checking the error code in the [MySQL server error message reference](https://dev.mysql.com/doc/mysql-errors/5.7/en/server-error-reference.html).

+## Common scenarios for high replication latency

+The following sections address scenarios in which high replication latency is common.

+### Network latency or high CPU consumption on the source server

+If you see the following values, then replication latency is likely caused by high network latency or high CPU consumption on the source server.

+```bash

+Slave_IO_State: Waiting for master to send event

+Master_Log_File: the binary file sequence is larger then Relay_Master_Log_File, e.g. mysql-bin.00020

+Relay_Master_Log_File: the file sequence is smaller than Master_Log_File, e.g. mysql-bin.00010

+```

+In this case, the IO thread is running and is waiting on the source server. The source server has already written to binary log file number 20. The replica has received only up to file number 10. The primary factors for high replication latency in this scenario are network speed or high CPU utilization on the source server.

+In Azure, network latency within a region can typically be measured milliseconds. Across regions, latency ranges from milliseconds to seconds.

+In most cases, the connection delay between IO threads and the source server is caused by high CPU utilization on the source server. The IO threads are processed slowly. You can detect this problem by using Azure Monitor to check CPU utilization and the number of concurrent connections on the source server.

+If you don't see high CPU utilization on the source server, the problem might be network latency. If network latency is suddenly abnormally high, check the [Azure status page](https://azure.status.microsoft/status) for known issues or outages.

+### Heavy bursts of transactions on the source server

+If you see the following values, then a heavy burst of transactions on the source server is likely causing the replication latency.

+```bash

+Slave_IO_State: Waiting for the slave SQL thread to free enough relay log space

+Master_Log_File: the binary file sequence is larger then Relay_Master_Log_File, e.g. mysql-bin.00020

+Relay_Master_Log_File: the file sequence is smaller then Master_Log_File, e.g. mysql-bin.00010

+```

+The output shows that the replica can retrieve the binary log behind the source server. But the replica IO thread indicates that the relay log space is full already.

+Network speed isn't causing the delay. The replica is trying to catch up. But the updated binary log size exceeds the upper limit of the relay log space.

+To troubleshoot this issue, enable the [slow query log](concepts-server-logs.md) on the source server. Use slow query logs to identify long-running transactions on the source server. Then tune the identified queries to reduce the latency on the server.

+Replication latency of this sort is commonly caused by the data load on the source server. When source servers have weekly or monthly data loads, replication latency is unfortunately unavoidable. The replica servers eventually catch up after the data load on the source server finishes.

+### Slowness on the replica server

+If you observe the following values, then the problem might be on the replica server.

+```bash

+Slave_IO_State: Waiting for master to send event

+Master_Log_File: The binary log file sequence equals to Relay_Master_Log_File, e.g. mysql-bin.000191

+Read_Master_Log_Pos: The position of master server written to the above file is larger than Relay_Log_Pos, e.g. 103978138

+Relay_Master_Log_File: mysql-bin.000191

+Slave_IO_Running: Yes

+Slave_SQL_Running: Yes

+Exec_Master_Log_Pos: The position of slave reads from master binary log file is smaller than Read_Master_Log_Pos, e.g. 13468882

+Seconds_Behind_Master: There is latency and the value here is greater than 0

+```

+In this scenario, the output shows that both the IO thread and the SQL thread are running well. The replica reads the same binary log file that the source server writes. However, some latency on the replica server reflects the same transaction from the source server.

+The following sections describe common causes of this kind of latency.

+#### No primary key or unique key on a table

+Azure Database for MySQL uses row-based replication. The source server writes events to the binary log, recording changes in individual table rows. The SQL thread then replicates those changes to the corresponding table rows on the replica server. When a table lacks a primary key or unique key, the SQL thread scans all rows in the target table to apply the changes. This scan can cause replication latency.

+In MySQL, the primary key is an associated index that ensures fast query performance because it can't include NULL values. If you use the InnoDB storage engine, the table data is physically organized to do ultra-fast lookups and sorts based on the primary key.

+We recommend that you add a primary key on tables in the source server before you create the replica server. Add primary keys on the source server and then re-create read replicas to help improve replication latency.

+Use the following query to find out which tables are missing a primary key on the source server:

+```sql

+select tab.table_schema as database_name, tab.table_name

+from information_schema.tables tab left join

+information_schema.table_constraints tco

+on tab.table_schema = tco.table_schema

+and tab.table_name = tco.table_name

+and tco.constraint_type = 'PRIMARY KEY'

+where tco.constraint_type is null

+and tab.table_schema not in('mysql', 'information_schema', 'performance_schema', 'sys')

+and tab.table_type = 'BASE TABLE'

+order by tab.table_schema, tab.table_name;

+```

+#### Long-running queries on the replica server

+The workload on the replica server can make the SQL thread lag behind the IO thread. Long-running queries on the replica server are one of the common causes of high replication latency. To troubleshoot this problem, enable the [slow query log](concepts-server-logs.md) on the replica server.

+Slow queries can increase resource consumption or slow down the server so that the replica can't catch up with the source server. In this scenario, tune the slow queries. Faster queries prevent blockage of the SQL thread and improve replication latency significantly.

+#### DDL queries on the source server

+On the source server, a data definition language (DDL) command like [`ALTER TABLE`](https://dev.mysql.com/doc/refman/5.7/en/alter-table.html) can take a long time. While the DDL command is running, thousands of other queries might be running in parallel on the source server.

+When the DDL is replicated, to ensure database consistency, the MySQL engine runs the DDL in a single replication thread. During this task, all other replicated queries are blocked and must wait until the DDL operation finishes on the replica server. Even online DDL operations cause this delay. DDL operations increase replication latency.

+If you enabled the [slow query log](concepts-server-logs.md) on the source server, you can detect this latency problem by checking for a DDL command that ran on the source server. Through index dropping, renaming, and creating, you can use the INPLACE algorithm for the ALTER TABLE. You might need to copy the table data and rebuild the table.

+Typically, concurrent DML is supported for the INPLACE algorithm. But you can briefly take an exclusive metadata lock on the table when you prepare and run the operation. So for the CREATE INDEX statement, you can use the clauses ALGORITHM and LOCK to influence the method for table copying and the level of concurrency for reading and writing. You can still prevent DML operations by adding a FULLTEXT index or SPATIAL index.

+The following example creates an index by using ALGORITHM and LOCK clauses.

+```sql

+ALTER TABLE table_name ADD INDEX index_name (column), ALGORITHM=INPLACE, LOCK=NONE;

+```

+Unfortunately, for a DDL statement that requires a lock, you can't avoid replication latency. To reduce the potential effects, do these types of DDL operations during off-peak hours, for instance during the night.

+#### Downgraded replica server

+In Azure Database for MySQL, read replicas use the same server configuration as the source server. You can change the replica server configuration after it has been created.

+If the replica server is downgraded, the workload can consume more resources, which in turn can lead to replication latency. To detect this problem, use Azure Monitor to check the CPU and memory consumption of the replica server.

+In this scenario, we recommend that you keep the replica server's configuration at values equal to or greater than the values of the source server. This configuration allows the replica to keep up with the source server.

+#### Improving replication latency by tuning the source server parameters

+In Azure Database for MySQL, by default, replication is optimized to run with parallel threads on replicas. When high-concurrency workloads on the source server cause the replica server to fall behind, you can improve the replication latency by configuring the parameter binlog_group_commit_sync_delay on the source server.

+The binlog_group_commit_sync_delay parameter controls how many microseconds the binary log commit waits before synchronizing the binary log file. The benefit of this parameter is that instead of immediately applying every committed transaction, the source server sends the binary log updates in bulk. This delay reduces IO on the replica and helps improve performance.

+It might be useful to set the binlog_group_commit_sync_delay parameter to 1000 or so. Then monitor the replication latency. Set this parameter cautiously, and use it only for high-concurrency workloads.

+> [!IMPORTANT]

+> In replica server, binlog_group_commit_sync_delay parameter is recommended to be 0. This is recommended because unlike source server, the replica server won't have high-concurrency and increasing the value for binlog_group_commit_sync_delay on replica server could inadvertently cause replication lag to increase.

+For low-concurrency workloads that include many singleton transactions, the binlog_group_commit_sync_delay setting can increase latency. Latency can increase because the IO thread waits for bulk binary log updates even if only a few transactions are committed.

+## Advanced Troubleshooting Options

+If using the show slave status command doesn't provide enough information to troubleshoot replication latency, try viewing these additional options for learning about which processes are active or waiting.

+### View the threads table

+The [`performance_schema.threads`](https://dev.mysql.com/doc/refman/5.7/en/performance-schema-threads-table.html) table shows the process state. A process with the state Waiting for lock_type lock indicates that thereΓÇÖs a lock on one of the tables, preventing the replication thread from updating the table.

+```sql

+SELECT name, processlist_state, processlist_time FROM performance_schema.threads WHERE name LIKE '%slave%';

+```

+For more information, see [General Thread States](https://dev.mysql.com/doc/refman/5.7/en/general-thread-states.html).

+### View the replication_connection_status table

+The performance_schema.replication_connection_status table shows the current status of the replication I/O thread that handles the replica's connection to the source, and it changes more frequently. The table contains values that vary during the connection.

+```sql

+SELECT * FROM performance_schema.replication_connection_status;

+```

+### View the replication_applier_status_by_worker table

+The `performance_schema.replication_applier_status_by_worker` table shows the status of the worker threads, Last seen transaction along with last error number and message, which help you find the transaction having issue and identify the root cause.

+You can run the below commands in the Data-in replication to skip errors or transactions:

+`az_replication_skip_counter`

+or

+`az_replication_skip_gtid_transaction`

+```sql

+SELECT * FROM performance_schema.replication_applier_status_by_worker;

+```

+### View the SHOW RELAYLOG EVENTS statement

+The `show relaylog events` statement shows the events in the relay log of a replica.

+┬╖ For GITD based replication (Read replica), the statement shows GTID transaction and binlog file and its position, you can use mysqlbinlog to get contents and statements being run.

+┬╖ For MySQL binlog position replication (used for Data-in replication), it shows statements being run, which will help to know on which table transactions are being run

+### Check the InnoDB Standard Monitor and Lock Monitor Output

+You can also try checking the InnoDB Standard Monitor and Lock Monitor Output to help in resolving locks and deadlocks and minimize replication lag. The Lock Monitor is the same as the Standard Monitor except that it includes additional lock information. To view this additional lock and deadlock information, run the show engine innodb status\G command.

+## Next steps

+Check out the [MySQL binlog replication overview](https://dev.mysql.com/doc/refman/5.7/en/binlog-replication-configuration-overview.html).

+ Title: Check security rules using NSG diagnostics

+description: Use NSG diagnostics to check if traffic is allowed or denied by network security group rules or Azure Virtual Network Manager security admin rules.

+# Diagnose network security rules

+You can use [network security groups](../virtual-network/network-security-groups-overview.md) to filter and control inbound and outbound network traffic to and from your Azure resources. You can also use [Azure Virtual Network Manager](../virtual-network-manager/overview.md) to apply admin security rules to your Azure resources to control network traffic.

+In this article, you learn how to use Azure Network Watcher [NSG diagnostics](network-watcher-network-configuration-diagnostics-overview.md) to check and troubleshoot security rules applied to your Azure traffic. NSG diagnostics checks if the traffic is allowed or denied by applied security rules.

+The example in this article shows you how a misconfigured network security group can prevent you from using Azure Bastion to connect to a virtual machine.

+## Prerequisites

+# [**Portal**](#tab/portal)

+- An Azure account with an active subscription. [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+- Sign in to the [Azure portal](https://portal.azure.com/?WT.mc_id=A261C142F) with your Azure account.

+# [**PowerShell**](#tab/powershell)

+- An Azure account with an active subscription. [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+- Azure Cloud Shell or Azure PowerShell.

+ The steps in this article run the Azure PowerShell cmdlets interactively in [Azure Cloud Shell](/azure/cloud-shell/overview). To run the commands in the Cloud Shell, select **Open Cloudshell** at the upper-right corner of a code block. Select **Copy** to copy the code and then paste it into Cloud Shell to run it. You can also run the Cloud Shell from within the Azure portal.

+ You can also [install Azure PowerShell locally](/powershell/azure/install-azure-powershell) to run the cmdlets. If you run PowerShell locally, sign in to Azure using the [Connect-AzAccount](/powershell/module/az.accounts/connect-azaccount) cmdlet.

+# [**Azure CLI**](#tab/cli)

+- An Azure account with an active subscription. [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+- Azure Cloud Shell or Azure CLI.

+

+ The steps in this article run the Azure CLI commands interactively in [Azure Cloud Shell](/azure/cloud-shell/overview). To run the commands in the Cloud Shell, select **Open Cloudshell** at the upper-right corner of a code block. Select **Copy** to copy the code, and paste it into Cloud Shell to run it. You can also run the Cloud Shell from within the Azure portal.

+

+ You can also [install Azure CLI locally](/cli/azure/install-azure-cli) to run the commands. If you run Azure CLI locally, sign in to Azure using the [az login](/cli/azure/reference-index#az-login) command.

+## Create a virtual network and a Bastion host

+In this section, you create a virtual network with two subnets and an Azure Bastion host. The first subnet is used for the virtual machine, and the second subnet is used for the Bastion host. You also create a network security group and apply it to the first subnet.

+# [**Portal**](#tab/portal)

+1. In the search box at the top of the portal, enter *virtual networks*. Select **Virtual networks** in the search results.

+ :::image type="content" source="./media/diagnose-network-security-rules/portal-search.png" alt-text="Screenshot shows how to search for virtual networks in the Azure portal." lightbox="./media/diagnose-network-security-rules/portal-search.png":::

+1. Select **+ Create**. In **Create virtual network**, enter or select the following values in the **Basics** tab:

+ | Setting | Value |

+ | | |

+ | **Project Details** | |

+ | Subscription | Select your Azure subscription. |

+ | Resource Group | Select **Create new**. </br> Enter *myResourceGroup* in **Name**. </br> Select **OK**. |

+ | **Instance details** | |

+ | Virtual network name | Enter *myVNet*. |

+ | Region | Select **(US) East US**. |

+1. Select the **Security** tab, or select the **Next** button at the bottom of the page.

+1. Under **Azure Bastion**, select **Enable Azure Bastion** and accept the default values:

+ | Setting | Value |

+ | | |

+ | Azure Bastion host name | **myVNet-Bastion**. |

+ | Azure Bastion public IP Address | **(New) myVNet-bastion-publicIpAddress**. |

+1. Select the **IP Addresses** tab, or select **Next** button at the bottom of the page.

+1. Accept the default IP address space **10.0.0.0/16** and edit the default subnet by selecting the pencil icon. In the **Edit subnet** page, enter the following values:

+ | Setting | Value |

+ | | |

+ | **Subnet details** | |

+ | Name | Enter *mySubnet*. |

+ | **Security** | |

+ | Network security group | Select **Create new**. </br> Enter *mySubnet-nsg* in **Name**. </br> Select **OK**. |

+1. Select the **Review + create**.

+1. Review the settings, and then select **Create**.

+# [**PowerShell**](#tab/powershell)

+1. Create a resource group using [New-AzResourceGroup](/powershell/module/az.resources/new-azresourcegroup). An Azure resource group is a logical container into which Azure resources are deployed and managed.

+ ```azurepowershell-interactive

+ # Create a resource group.

+ New-AzResourceGroup -Name 'myResourceGroup' -Location 'eastus'

+ ```

+1. Create a default network security group using [New-AzNetworkSecurityGroup](/powershell/module/az.network/new-aznetworksecuritygroup).

+ ```azurepowershell-interactive

+ # Create a network security group.

+ $networkSecurityGroup = New-AzNetworkSecurityGroup -Name 'mySubnet-nsg' -ResourceGroupName 'myResourceGroup' -Location 'eastus'

+ ```

+1. Create a subnet configuration for the virtual machine subnet and the Bastion host subnet using [New-AzVirtualNetworkSubnetConfig](/powershell/module/az.network/new-azvirtualnetworksubnetconfig).

+ ```azurepowershell-interactive

+ # Create subnets configuration.

+ $firstSubnet = New-AzVirtualNetworkSubnetConfig -Name 'mySubnet' -AddressPrefix '10.0.0.0/24' -NetworkSecurityGroup $networkSecurityGroup

+ $secondSubnet = New-AzVirtualNetworkSubnetConfig -Name 'AzureBastionSubnet' -AddressPrefix '10.0.1.0/26'

+ ```

+1. Create a virtual network using [New-AzVirtualNetwork](/powershell/module/az.network/new-azvirtualnetwork).

+ ```azurepowershell-interactive

+ # Create a virtual network.

+ $vnet = New-AzVirtualNetwork -Name 'myVNet' -ResourceGroupName 'myResourceGroup' -Location 'eastus' -AddressPrefix '10.0.0.0/16' -Subnet $firstSubnet, $secondSubnet

+ ```

+1. Create the public IP address resource required for the Bastion host using [New-AzPublicIpAddress](/powershell/module/az.network/new-azpublicipaddress).

+ ```azurepowershell-interactive

+ # Create a public IP address for Azure Bastion.

+ New-AzPublicIpAddress -ResourceGroupName 'myResourceGroup' -Name 'myBastionIp' -Location 'eastus' -AllocationMethod 'Static' -Sku 'Standard'

+ ```

+1. Create the Bastion host using [New-AzBastion](/powershell/module/az.network/new-azbastion).

+ ```azurepowershell-interactive

+ # Create an Azure Bastion host.

+ New-AzBastion -ResourceGroupName 'myResourceGroup' -Name 'myVNet-Bastion' --PublicIpAddressRgName 'myResourceGroup' -PublicIpAddressName 'myBastionIp' -VirtualNetwork $vnet

+ ```

+# [**Azure CLI**](#tab/cli)

+1. Create a resource group using [az group create](/cli/azure/group#az-group-create). An Azure resource group is a logical container into which Azure resources are deployed and managed.

+ ```azurecli-interactive

+ # Create a resource group.

+ az group create --name 'myResourceGroup' --location 'eastus'

+ ```

+1. Create a default network security group using [az network nsg create](/cli/azure/network/nsg#az-network-nsg-create).

+ ```azurecli-interactive

+ # Create a network security group.

+ az network nsg create --name 'mySubnet-nsg' --resource-group 'myResourceGroup' --location 'eastus'

+ ```

+1. Create a virtual network using [az network vnet create](/cli/azure/network/vnet#az-network-vnet-create).

+ ```azurecli-interactive

+ az network vnet create --resource-group 'myResourceGroup' --name 'myVNet' --subnet-name 'mySubnet' --subnet-prefixes 10.0.0.0/24 --network-security-group 'mySubnet-nsg'

+ ```

+1. Create a subnet for Azure Bastion using [az network vnet subnet create](/cli/azure/network/vnet/subnet#az-network-vnet-subnet-create).

+ ```azurecli-interactive

+ # Create AzureBastionSubnet.

+ az network vnet subnet create --name 'AzureBastionSubnet' --resource-group 'myResourceGroup' --vnet-name 'myVNet' --address-prefixes '10.0.1.0/26'

+ ```

+1. Create a public IP address for the Bastion host using [az network public-ip create](/cli/azure/network/public-ip#az-network-public-ip-create).

+ ```azurecli-interactive

+ # Create a public IP address resource.

+ az network public-ip create --resource-group 'myResourceGroup' --name 'myBastionIp' --sku Standard

+ ```

+1. Create a Bastion host using [az network bastion create](/cli/azure/network/bastion#az-network-bastion-create).

+ ```azurecli-interactive

+ az network bastion create --name 'myVNet-Bastion' --public-ip-address 'myBastionIp' --resource-group 'myResourceGroup' --vnet-name 'myVNet'

+ ```

+## Create a virtual machine

+In this section, you create a virtual machine and a network security group applied to its network interface.

+# [**Portal**](#tab/portal)

+1. In the search box at the top of the portal, enter *virtual machines*. Select **Virtual machines** in the search results.

+1. Select **+ Create** and then select **Azure virtual machine**.

+1. In **Create a virtual machine**, enter or select the following values in the **Basics** tab:

+ | Setting | Value |

+ | | |

+ | **Project Details** | |

+ | Subscription | Select your Azure subscription. |

+ | Resource Group | Select **myResourceGroup**. |

+ | **Instance details** | |

+ | Virtual machine name | Enter *myVM*. |

+ | Region | Select **(US) East US**. |

+ | Availability Options | Select **No infrastructure redundancy required**. |

+ | Security type | Select **Standard**. |

+ | Image | Select **Windows Server 2022 Datacenter: Azure Edition - x64 Gen2**. |

+ | Size | Choose a size or leave the default setting. |

+ | **Administrator account** | |

+ | Username | Enter a username. |

+ | Password | Enter a password. |

+ | Confirm password | Reenter password. |

+1. Select the **Networking** tab, or select **Next: Disks**, then **Next: Networking**.

+1. In the Networking tab, enter or select the following values:

+ | Setting | Value |

+ | | |

+ | **Network interface** | |

+ | Virtual network | Select **myVNet**. |

+ | Subnet | Select **default**. |

+ | Public IP | Select **None**. |

+ | NIC network security group | Select **Basic**. |

+ | Public inbound ports | Select **None**. |

+1. Select **Review + create**.

+1. Review the settings, and then select **Create**.

+# [**PowerShell**](#tab/powershell)

+1. Create a default network security group using [New-AzNetworkSecurityGroup](/powershell/module/az.network/new-aznetworksecuritygroup).

+ ```azurepowershell-interactive

+ # Create

+ New-AzNetworkSecurityGroup -Name 'myVM-nsg' -ResourceGroupName 'myResourceGroup' -Location eastus

+ ```

+1. Create a virtual machine using [New-AzVM](/powershell/module/az.compute/new-azvm). When prompted, enter a username and password.

+ ```azurepowershell-interactive

+ # Create a virtual machine.

+ New-AzVm -ResourceGroupName 'myResourceGroup' -Name 'myVM' -Location 'eastus' -VirtualNetworkName 'myVNet' -SubnetName 'mySubnet' -SecurityGroupName 'myVM-nsg' -ImageName 'MicrosoftWindowsServer:WindowsServer:2022-Datacenter-azure-edition:latest'

+ ```

+# [**Azure CLI**](#tab/cli)

+1. Create a default network security group using [az network nsg create](/cli/azure/network/nsg#az-network-nsg-create).

+ ```azurecli-interactive

+ # Create a network security group for the network interface of the virtual machine.

+ az network nsg create --name 'myVM-nsg' --resource-group 'myResourceGroup' --location 'eastus'

+ ```

+1. Create a virtual machine using [az vm create](/cli/azure/vm#az-vm-create). When prompted, enter a username and password.

+ ```azurecli-interactive

+ # Create a virtual machine.

+ az vm create --resource-group 'myResourceGroup' --name 'myVM' --location 'eastus' --vnet-name 'myVNet' --subnet 'mySubnet' --public-ip-address '' --nsg 'myVM-nsg' --image 'Win2022AzureEditionCore'

+ ```

+## Add a security rule to the network security group

+In this section, you add a security rule to the network security group associated with the network interface of **myVM**. The rule denies any inbound traffic from the virtual network.

+# [**Portal**](#tab/portal)

+1. In the search box at the top of the portal, enter *network security groups*. Select **Network security groups** in the search results.

+1. From the list of network security groups, select **myVM-nsg**.

+1. Under **Settings**, select **Inbound security rules**.

+1. Select **+ Add**. In the Networking tab, enter or select the following values:

+ | Setting | Value |

+ | | |

+ | Source | Select **Service Tag**. |

+ | Source service tag | Select **VirtualNetwork**. |

+ | Source port ranges | Enter *. |

+ | Destination | Select **Any**. |

+ | Service | Select **Custom**. |

+ | Destination port ranges | Enter *. |

+ | Protocol | Select **Any**. |

+ | Action | Select **Deny**. |

+ | Priority | Enter *1000*. |

+ | Name | Enter *DenyVnetInBound*. |

+1. Select **Add**.

+# [**PowerShell**](#tab/powershell)

+Use [Add-AzNetworkSecurityRuleConfig](/powershell/module/az.network/add-aznetworksecurityruleconfig) to create a security rule that denies traffic from the virtual network. Then use [Set-AzNetworkSecurityGroup](/powershell/module/az.network/set-aznetworksecuritygroup) to update the network security group with the new security rule.

+```azurepowershell-interactive

+# Place the network security group configuration into a variable.

+$networkSecurityGroup = Get-AzNetworkSecurityGroup -Name 'myVM-nsg' -ResourceGroupName 'myResourceGroup'

+# Create a security rule.

+Add-AzNetworkSecurityRuleConfig -Name 'DenyVnetInBound' -NetworkSecurityGroup $networkSecurityGroup `

+-Access 'Deny' -Protocol '*' -Direction 'Inbound' -Priority '1000' `

+-SourceAddressPrefix 'virtualNetwork' -SourcePortRange '*' -DestinationAddressPrefix '*' -DestinationPortRange '*'

+# Updates the network security group.

+Set-AzNetworkSecurityGroup -NetworkSecurityGroup $networkSecurityGroup

+```

+# [**Azure CLI**](#tab/cli)

+Use [az network nsg rule create](/cli/azure/network/nsg/rule#az-network-nsg-rule-create) to add to the network security group a security rule that denies traffic from the virtual network.

+```azurecli-interactive

+# Add a security rule to the network security group.

+az network nsg rule create --name 'DenyVnetInBound' --resource-group 'myResourceGroup' --nsg-name 'myVM-nsg' --priority '1000' \

+--access 'Deny' --protocol '*' --direction 'Inbound' --source-address-prefixes 'virtualNetwork' --source-port-ranges '*' \

+--destination-address-prefixes '*' --destination-port-ranges '*'

+```

+> [!NOTE]

+> The **VirtualNetwork** service tag represents the address space of the virtual network, all connected on-premises address spaces, peered virtual networks, virtual networks connected to a virtual network gateway, the virtual IP address of the host, and address prefixes used on user-defined routes. For more information, see [Service tags](../virtual-network/service-tags-overview.md).

+## Check security rules applied to a virtual machine traffic

+Use NSG diagnostics to check the security rules applied to the traffic originated from the Bastion subnet to the virtual machine.

+# [**Portal**](#tab/portal)

+1. In the search box at the top of the portal, search for and select **Network Watcher**.

+1. Under **Network diagnostic tools**, select **NSG diagnostics**.

+1. On the **NSG diagnostics** page, enter or select the following values:

+ | Setting | Value |

+ | - | |

+ | Subscription | Select the Azure subscription that has the virtual machine that you want to test the connection with. |

+ | Resource group | Select the resource group that has the virtual machine that you want to test the connection with. |

+ | Supported resource type | Select **Virtual machine**. |

+ | Resource | Select the virtual machine that you want to test the connection with. |

+ | Protocol | Select **TCP**. Other available options are: **Any**, **UDP** and **ICMP**. |

+ | Direction | Select **Inbound**. Other available option is: **Outbound**. |

+ | Source type | Select **IPv4 address/CIDR**. Other available option is: **Service Tag**. |

+ | IPv4 address/CIDR | Enter *10.0.1.0/26*, which is the IP address range of the Bastion subnet. Acceptable values are: single IP address, multiple IP addresses, single IP prefix, multiple IP prefixes. |

+ | Destination IP address | Enter *10.0.0.4*, which is the IP address of **myVM**. |

+ | Destination port | Enter * to include all ports. |

+ :::image type="content" source="./media/diagnose-network-security-rules/nsg-diagnostics-vm-values.png" alt-text="Screenshot showing required values for NSG diagnostics to test inbound connections to a virtual machine in the Azure portal." lightbox="./media/diagnose-network-security-rules/nsg-diagnostics-vm-values.png":::

+1. Select **Check** to run the test. Once NSG diagnostics completes checking all security rules, it displays the result.

+ :::image type="content" source="./media/diagnose-network-security-rules/nsg-diagnostics-vm-test-result-denied.png" alt-text="Screenshot showing the result of inbound connections to the virtual machine as Denied." lightbox="./media/diagnose-network-security-rules/nsg-diagnostics-vm-test-result-denied.png":::

+ The result shows that there are three security rules assessed for the inbound connection from the Bastion subnet:

+ - **GlobalRules**: this security admin rule is applied at the virtual network level using Azure Virtual Network Manage. The rule allows inbound TCP traffic from the Bastion subnet to the virtual machine.

+ - **mySubnet-nsg**: this network security group is applied at the subnet level (subnet of the virtual machine). The rule allows inbound TCP traffic from the Bastion subnet to the virtual machine.

+ - **myVM-nsg**: this network security group is applied at the network interface (NIC) level. The rule denies inbound TCP traffic from the Bastion subnet to the virtual machine.

+1. Select **myVM-nsg** to see details about the security rules that this network security group has and which rule denied the traffic.

+ :::image type="content" source="./media/diagnose-network-security-rules/nsg-diagnostics-vm-test-result-denied-details.png" alt-text="Screenshot showing the details of the network security group that denied the traffic to the virtual machine." lightbox="./media/diagnose-network-security-rules/nsg-diagnostics-vm-test-result-denied-details.png":::

+ In **myVM-nsg** network security group, the security rule **DenyVnetInBound** denies any traffic coming from the address space of **VirtualNetwork** service tag to the virtual machine. The Bastion host uses IP addresses from **10.0.1.0/26**, which are included **VirtualNetwork** service tag, to connect to the virtual machine. Therefore, the connection from the Bastion host is denied by the **DenyVnetInBound** security rule.

+# [**PowerShell**](#tab/powershell)

+Use [Invoke-AzNetworkWatcherNetworkConfigurationDiagnostic](/powershell/module/az.network/invoke-aznetworkwatchernetworkconfigurationdiagnostic) to start the NSG diagnostics session.

+```azurepowershell-interactive

+# Create a profile for the diagnostic session.

+$profile = New-AzNetworkWatcherNetworkConfigurationDiagnosticProfile -Direction Inbound -Protocol Tcp -Source 10.0.1.0/26 -Destination 10.0.0.4 -DestinationPort *

+# Place the virtual machine configuration into a variable.

+$vm = Get-AzVM -Name 'myVM' -ResourceGroupName 'myResourceGroup'

+# Start the the NSG diagnostics session.

+Invoke-AzNetworkWatcherNetworkConfigurationDiagnostic -Location 'eastus' -TargetResourceId $vm.Id -Profile $profile | Format-List

+```

+Output similar to the following example output is returned:

+```output

+Results : {Microsoft.Azure.Commands.Network.Models.PSNetworkConfigurationDiagnosticResult}

+ResultsText : [

+ {

+ "Profile": {

+ "Direction": "Inbound",

+ "Protocol": "Tcp",

+ "Source": "10.0.1.0/26",

+ "Destination": "10.0.0.4",

+ "DestinationPort": "*"

+ },

+ "NetworkSecurityGroupResult": {

+ "SecurityRuleAccessResult": "Deny",

+ "EvaluatedNetworkSecurityGroups": [

+ {

+ "NetworkSecurityGroupId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/NetworkAdmin/providers/Microsoft.Network/networkManagers/GlobalRules",

+ "MatchedRule": {

+ "RuleName": "VirtualNetwork",

+ "Action": "Allow"

+ },

+ "RulesEvaluationResult": [

+ {

+ "Name": "VirtualNetwork",

+ "ProtocolMatched": true,

+ "SourceMatched": true,

+ "SourcePortMatched": true,

+ "DestinationMatched": true,

+ "DestinationPortMatched": true

+ }

+ ]

+ },

+ {

+ "NetworkSecurityGroupId": "/subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/myResourceGroup/providers/Microsoft.Network/networkSecurityGroups/mySubnet-nsg",

+ "MatchedRule": {

+ "RuleName": "DefaultRule_AllowVnetInBound",

+ "Action": "Allow"

+ },

+ "RulesEvaluationResult": [

+ {

+ "Name": "DefaultRule_AllowVnetInBound",

+ "ProtocolMatched": true,

+ "SourceMatched": true,

+ "SourcePortMatched": true,

+ "DestinationMatched": true,

+ "DestinationPortMatched": true

+ }

+ ]

+ },

+ {

+ "NetworkSecurityGroupId": "/subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/myResourceGroup/providers/Microsoft.Network/networkSecurityGroups/myVM-nsg",

+ "MatchedRule": {

+ "RuleName": "UserRule_DenyVnetInBound",

+ "Action": "Deny"

+ },

+ "RulesEvaluationResult": [

+ {

+ "Name": "UserRule_DenyVnetInBound",

+ "ProtocolMatched": true,

+ "SourceMatched": true,

+ "SourcePortMatched": true,

+ "DestinationMatched": true,

+ "DestinationPortMatched": true

+ }

+ ]

+ }

+ ]

+ }

+ }

+ ]

+```

+The result shows that there are three security rules assessed for the inbound connection from the Bastion subnet:

+- **GlobalRules**: this security admin rule is applied at the virtual network level using Azure Virtual Network Manage. The rule allows inbound TCP traffic from the Bastion subnet to the virtual machine.

+- **mySubnet-nsg**: this network security group is applied at the subnet level (subnet of the virtual machine). The rule allows inbound TCP traffic from the Bastion subnet to the virtual machine.

+- **myVM-nsg**: this network security group is applied at the network interface (NIC) level. The rule denies inbound TCP traffic from the Bastion subnet to the virtual machine.

+In **myVM-nsg** network security group, the security rule **DenyVnetInBound** denies any traffic coming from the address space of **VirtualNetwork** service tag to the virtual machine. The Bastion host uses IP addresses from **10.0.1.0/26**, which are included **VirtualNetwork** service tag, to connect to the virtual machine. Therefore, the connection from the Bastion host is denied by the **DenyVnetInBound** security rule.

+# [**Azure CLI**](#tab/cli)

+Use [az network watcher run-configuration-diagnostic](/cli/azure/network/watcher#az-network-watcher-run-configuration-diagnostic) to start the NSG diagnostics session.

+```azurecli-interactive

+# Start the the NSG diagnostics session.

+az network watcher run-configuration-diagnostic --resource 'myVM' --resource-group 'myResourceGroup' --resource-type 'virtualMachines' --direction 'Inbound' --protocol 'TCP' --source '10.0.1.0/26' --destination '10.0.0.4' --port '*'

+```

+Output similar to the following example output is returned:

+```output

+{

+ "results": [

+ {

+ "networkSecurityGroupResult": {

+ "evaluatedNetworkSecurityGroups": [

+ {

+ "appliedTo": "/subscriptions/abcdef01-2345-6789-0abc-def012345678/resourceGroups/myResourceGroup/providers/Microsoft.Network/virtualNetworks/myVNet",

+ "matchedRule": {

+ "action": "Allow",

+ "ruleName": "VirtualNetwork"

+ },

+ "networkSecurityGroupId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/NetworkAdmin/providers/Microsoft.Network/networkManagers/GlobalRules",

+ "rul