+- An API endpoint. You can create an API endpoint using one of our [samples](api-connector-samples.md#api-connector-rest-api-samples).

+1. Under **Azure services**, select **Azure AD B2C**.

+1. Select **API connectors**, and then select **New API connector**.

+ 

+1. Provide a display name for the call. For example, **Enrich token from external source**.

+1. Provide the **Endpoint URL** for the API call.

+1. Choose the **Authentication type** and configure the authentication information for calling your API. Learn how to [Secure your API Connector](secure-rest-api.md).

+ 

+1. Select **Save**.

+1. Under **Azure services**, select **Azure AD B2C**.

+1. Select **User flows**, and then select the user flow you want to add the API connector to.

+1. Select **API connectors**, and then select the API endpoint you want to invoke at the **Before sending the token (preview)** step in the user flow:

+ 

+1. Select **Save**.

+| \<extension\_{extensions-app-id}\_CustomAttribute> | \<attribute-type> | No | The claim does not need to contain `_<extensions-app-id>_`, it is *optional*. They can be returned in the token if selected as an **Application claim**. |

+### Using serverless cloud functions

+Serverless functions, like [HTTP triggers in Azure Functions](../azure-functions/functions-bindings-http-webhook-trigger.md), provide a way create API endpoints to use with the API connector. The serverless cloud function can also call and invoke other web APIs, data stores, and other cloud services for complex scenarios.

+### Using logging

+1. Under **Activities**, select **Audit logs**.

+1. Filter the list view: For **Date**, select the time interval you want, and for **Activity**, select **An API was called as part of a user flow**.

+1. Inspect individual logs. Each row represents an API connector attempting to be called during a user flow. If an API call fails and a retry occurs, it's still represented as a single row. The `numberOfAttempts` indicates the number of times your API was called. This value can be `1`or `2`. Other information about the API call is detailed in the logs.

+ 

+On the sign-up or sign-in page, Azure AD B2C presents a list of external identity providers the user can choose for sign-in. Once they select one of the external identity providers, they're taken (redirected) to the selected provider's website to complete the sign-in process. After the user successfully signs in, they're returned to Azure AD B2C for authentication of the account in your application.

+

+| Use Identity Protection and Conditional Access | Use these capabilities for significantly greater control over risky authentications and access policies. Azure AD B2C Premium P2 is required. [Learn more](conditional-access-identity-protection-overview.md). |

+|Tenant size | You need to plan with Azure AD B2C tenant size in mind. By default, Azure AD B2C tenant can accommodate 1.25 million objects (user accounts and applications). You can increase this limit to 5.25 million objects by adding a custom domain to your tenant, and verifying it. If you need a bigger tenant size, you need to contact [Support](find-help-open-support-ticket.md).|

+| [Microsoft Support](find-help-open-support-ticket.md) | File a support request for Azure AD B2C technical issues. Billing and subscription management support is provided at no cost. |

+This article describes how to enable custom domains in your redirect URLs for Azure Active Directory B2C (Azure AD B2C). By using a verified custom domain, you've benefits such as:

+- It provides a more seamless user experience. From the user's perspective, they remain in your domain during the sign in process rather than redirecting to the Azure AD B2C default domain *&lt;tenant-name&gt;.b2clogin.com*.

+- You increase the number of objects (user accounts and applications) you can create in your Azure AD B2C tenant from the default 1.25 million to 5.25 million.

+With Azure Active Directory B2C (Azure AD B2C) and solutions from software-vendor partners, customers can enable end-user identity verification and proofing for account registration. Identity verification and proofing can check documents, knowledge-based information, and liveness.

+## Architecture diagram

+The following architecture diagram illustrates the verification and proofing flow.

+ 

+1. User begins registration with a device.

+2. User enters information.

+3. Digital-risk score is assessed, then third-party identity proofing and identity validation occurs.

+4. Identity is validated or rejected.

+5. User attributes are passed to Azure Active Directory B2C.

+6. If user verification is successful, a user account is created in Azure AD B2C during sign-in.

+7. Based on the verification result, the user receives an access-approved or -denied message.

+## Software vendors and integration documentation

+Microsoft partners with independent software vendors (ISVs). Use the following table to locate an ISV and related integration documentation.

+| ISV logo | ISV link and description| Integration documentation|

+||||

+| |

+| |

+||

+||

+||

+| |

+| |

+## Resources

+- [Azure AD B2C custom policy overview](custom-policy-overview.md)

+- [Tutorial: Create user flows and custom policies in Azure Active Directory B2C](tutorial-create-user-flows.md?pivots=b2c-custom-policy&tabs=applications)

+Select and contact a partner from the previous table to get started on solution integration with Azure AD B2C. The partners have similar processes to contact them for a product demo.

+- To use MS Graph API, and interact with resources in your Azure AD B2C tenant, you need an application registration that grants the permissions to do so. Follow the steps in the [Register a Microsoft Graph application](microsoft-graph-get-started.md) article to create an application registration that your management application can use.

+## Tenant usage

+Use the [Get organization details](/graph/api/organization-get) API to get your directory size quota. You need to add the `$select` query parameter as shown in the following HTTP request:

+```http

+ GET https://graph.microsoft.com/v1.0/organization/organization-id?$select=directorySizeQuota

+```

+Replace `organization-id` with your organization or tenant ID.

+The response to the above request looks similar to the following JSON snippet:

+```json

+{

+ "directorySizeQuota": {

+ "used": 156,

+ "total": 1250000

+ }

+}

+```

+- [Tutorial: Create user flows and custom policies in Azure Active Directory B2C](./tutorial-create-user-flows.md?pivots=b2c-custom-policy)

+- [Visual Studio 2022](https://www.visualstudio.com/downloads/) with the **ASP.NET and web development** workload.

+1. For this quickstart, you run both the **TaskWebApp** and **TaskService** projects at the same time. Right-click the **B2C-WebAPI-DotNet** solution in Solution Explorer, and then select **Set StartUp Projects**.

+1. Select **Multiple startup projects** and change the **Action** for both projects to **Start**.

+1. Select **OK**.

+1. Press **F5** to debug both applications. Each application opens in its own browser tab:

+ 

+1. Azure AD B2C presents a sign-in page for a fictitious company called Fabrikam for the sample web application. To sign up using a social identity provider, select the button of the identity provider you want to use.

+ 

+1. Finish the sign-in process for the identity provider.

+ 

+1. Change your **Display name** or **City**, and then select **Continue** to update your profile.

+1. In the **New Item** text box, enter text. To call the Azure AD B2C protected web API that adds a to-do list item, select **Add**.

+ 

+|Total number of objects (user accounts and applications) per tenant (default limit)|1.25 million |

+|Total number of objects (user accounts and applications) per tenant (using a verified custom domain)|5.25 million |

+

+

+

+- When the recovery email prompt is **On**, a user signing up for the first time is prompted to verify a backup email. A user who hasn't provided a recovery email before is asked to verify a backup email during next sign-in.

+

+

+## Federated sign-in

+You can configure Azure AD B2C to allow users to sign in to your application with credentials from external social or enterprise identity providers (IdPs). Azure AD B2C supports many [external identity providers](add-identity-provider.md) and any identity provider that supports OAuth 1.0, OAuth 2.0, OpenID Connect, and SAML protocols.

+With external identity provider federation, you can offer your consumers the ability to sign in with their existing social or enterprise accounts, without having to create a new account just for your application.

+On the sign-up or sign-in page, Azure AD B2C presents a list of external identity providers the user can choose for sign-in. Once they select one of the external identity providers, they're redirected to the selected provider's website to complete the sign-in process. After the user successfully signs in, they're returned to Azure AD B2C for authentication of the account in your application.

+

+You can add identity providers that are supported by Azure Active Directory B2C (Azure AD B2C) to your [user flows](user-flow-overview.md) using the Azure portal. You can also add identity providers to your [custom policies](user-flow-overview.md).

+- [Configure your local account identity provider](identity-provider-local.md).

+

+## Get your tenant usage

+You can read your Azure AD B2C's total directory size, and how much of it is in use. To do so, follow the steps in [Get tenant usage by using Microsoft Graph API](microsoft-graph-operations.md#tenant-usage).

+Before you create your Azure AD B2C tenant, you need to take the following considerations into account:

+- You can create up to **20** tenants per subscription. This limit help protect against threats to your resources, such as denial-of-service attacks, and is enforced in both the Azure portal and the underlying tenant creation API. If you want to increase this limit, please contact [Microsoft Support](find-help-open-support-ticket.md).

+- By default, each tenant can accommodate a total of **1.25 million** objects (user accounts and applications), but you can increase this limit to **5.25 million** objects when you add and verify a custom domain. If you want to increase this limit, please contact [Microsoft Support](find-help-open-support-ticket.md). However, if you created your tenant before **September 2022**, this limit doesn't affect you, and your tenant will retain the size allocated to it at creation, that's, **50 million** objects.

+- If you want to reuse a tenant name that you previously tried to delete, but you see the error "Already in use by another directory" when you enter the domain name, you'll need to [follow these steps to fully delete the tenant first](./faq.yml?tabs=app-reg-ga#how-do-i-delete-my-azure-ad-b2c-tenant-). A role of at least *Subscription Administrator* is required. After deleting the tenant, you might also need to sign out and sign back in before you can reuse the domain name.

+1. Select **Create a new Azure AD B2C Tenant**.

+> [!NOTE]

+> Before you start the migration, make sure your Azure AD B2C tenant's unused quota can accommodate all the users you expect to migrate. Learn how to [Get your tenant usage](microsoft-graph-operations.md#tenant-usage). If you need to increase your tenant's quota limit, contact [Microsoft Support](find-help-open-support-ticket.md).

+> 2 application segment per complex application are supported for [Microsoft Azure AD premium subscription](https://azure.microsoft.com/pricing/details/active-directory). Licence requirement for more than 2 application segments per complex application to be announced soon.

+ Title: Securely integrate Azure Logic Apps with on-premises APIs using Azure Active Directory Application Proxy

+description: Azure Active Directory's Application Proxy lets cloud-native logic apps securely access on-premises APIs to bridge your workload.

+# Securely integrate Azure Logic Apps with on-premises APIs using Azure Active Directory Application Proxy

+Azure Logic Apps is a service allowing easy creation of managed workflows in a no-code environment that can integrate with various external services and systems. This can help automate a wide range of business processes, such as data integration, data processing, and event-driven scenarios.

+While Logic Apps easily integrate with other public and cloud-based services, the need may arise to utilize Logic Apps with protected, on-premises applications and services without exposing the service to the public via port forwarding or a traditional reverse proxy.

+This article describes the steps necessary to utilize the Azure AD Application Proxy solution to provide secure access to a Logic App, while protecting the internal application from unwanted actors. The process and end result is similar to [Access on-premises APIs with Azure Active Directory Application Proxy](./application-proxy-secure-api-access.md) with special attention paid to utilizing the API from within a Logic App.

+## Overview

+The following diagram shows a traditional way to publish on-premises APIs for access from Azure Logic Apps. This approach requires opening incoming TCP ports 80 and/or 443 to the API service.

+

+The following diagram shows how you can use Azure AD Application Proxy to securely publish APIs for use with Logic Apps (or other Azure Cloud services) without opening any incoming ports:

+

+The Azure AD App Proxy and associated connector facilitate secure authorization and integration to your on-premises services without additional configuration to your network security infrastructure.

+## Prerequisites

+To follow this tutorial, you will need:

+- Admin access to an Azure directory, with an account that can create and register apps

+- The *Logic App Contributor* role (or higher) in an active tenant

+- Azure Application Proxy connector deployed and an application configured as detailed in [Add an on-premises app - Application Proxy in Azure Active Directory](./application-proxy-add-on-premises-application.md)

+> [!NOTE]

+> While granting a user entitlement and testing the sign on is recommended, it is not required for this guide.

+## Configure the Application Access

+When a new Enterprise Application is created, a matching App Registration is also created. The App Registration allows configuration of secure programmatic access using certificates, secrets, or federated credentials. For integration with a Logic App, we will need to configure a client secret key, and configure the API permissions.

+1. From the Azure portal, open **Azure Active Directory**

+2. Select the **App Registrations** menu item from the navigation pane

+ 

+3. From the *App Registrations* window, select the **All applications** tab option

+4. Navigate to the application with a matching name to your deployed App Proxy application. For example, if you deployed *Sample App 1* as an Enterprise Application, click the **Sample App 1** registration item

+ > [!NOTE]

+ > If an associated application cannot be found, it may have not been automatically created or may have been deleted. A registration can be created using the **New Registration** button.

+5. From the *Sample App 1* detail page, take note of the *Application (client) ID* and *Directory (tenant) ID* fields. These will be used later.

+ 

+6. Select the **API permissions** menu item from the navigation pane

+ 

+7. From the *API permissions* page:

+ 1. Click the **Add a permission** button

+ 2. In the *Request API permissions* pop-up:

+ 1. Select the **APIs my organization uses** tab

+ 2. Search for your app by name (e.g. *Sample App 1*) and select the item

+ 3. Ensure *Delegated Permissions* is **selected**, then **check** the box for *user_impersonation*

+ 4. Click **Add permissions**

+ 3. Verify the configured permission appears

+ 

+8. Select the **Certificates & secrets** menu item from the navigation pane

+ 

+9. From the *Certificates & secrets* page:

+ 1. Select the **Client secrets** tab item

+ 2. Click the **New client secret** button

+ 3. From the *Add a client secret* pop-up:

+ 1. Enter a **Description** and desired expiration

+ 2. Click **Add**

+ 4. Verify the new client secret appears

+ 5. Click the **Copy** button for the *Value* of the newly created secret. Save this securely for use later, this value is only shown one time.

+ 

+## Configure the Logic App

+1. From the Logic App, open the **Designer** view

+2. Select a desired trigger (if prompted)

+3. Add a new step and select the **HTTP** operation

+ 

+4. In the operation details:

+ 1. *Method*: Select the desired HTTP method to be sent to the internal API

+ 2. *URI*: Fill in with the *public* FQDN of your application registered in Azure AD, along with the additional URI required for API access (e.g. *sampleapp1.msappproxy.net/api/1/status*)

+ > [!NOTE]

+ > Specific values for API will depend on your internal application. Refer to your application's documentation for more information.

+ 3. *Headers*: Enter any desired headers to be sent to the internal API

+ 4. *Queries*: Enter any desired queries to be sent to the internal API

+ 5. *Body*: Enter any desired body contents to be sent to the internal API

+ 6. *Cookie*: Enter any desired cookie(s) to be sent to the internal API

+ 7. Click *Add new parameter*, then check *Authentication*

+ 8. From the *Authentication type*, select *Active Directory OAuth*

+ 9. For the authentication, fill the following details:

+ 1. *Authority*: Enter *https://login.windows.net*

+ 2. *Tenant*: Enter the **Directory (tenant) ID** noted in *Configure the Application Access*

+ 3. *Audience*: Enter the *public* FQDN of your application registered in Azure AD (e.g. *sampleapp1.msappproxy.net*)

+ 4. *Client ID*: Enter the **Application (client) ID** noted in *Configure the Application Access*

+ 5. *Credential Type*: **Secret**

+ 6. *Secret*: Enter the **secret value** noted in *Configure the Application Access*

+ 

+5. Save the logic app and test with your trigger

+## Caveats

+- APIs that require authentication/authorization require special handling when using this method. Since Azure Active Directory OAuth is being used for access, the requests sent already contain an *Authorization* field that cannot also be utilized by the internal API (unless SSO is configured). As a workaround, some applications offer authentication or authorization that uses methods other than an *Authorization* header. For example, GitLab allows for a header titled *PRIVATE-TOKEN*, and Atlassian JIRA allows for requesting a Cookie that can be used in later requests

+- While the Logic App HTTP action shows cleartext values, it is highly recommended to store the App Registration Secret Key in Azure Key Vault for secure retrieval and use.

+## See Also

+- [How to configure an Application Proxy application](./application-proxy-config-how-to.md)

+- [Access on-premises APIs with Azure Active Directory Application Proxy](./application-proxy-secure-api-access.md)

+- [Common scenarios, examples, tutorials, and walkthroughs for Azure Logic Apps](../../logic-apps/logic-apps-examples-and-scenarios.md)

+## Determine OATH token registration type in mysecurityinfo

+Users can manage and add OATH token registrations by accessing https://aka.ms/mysecurityinfo or by selecting Security info from My Account. Specific icons are used to differentiate whether the OATH token registration is hardware or software based.

+OATH token registration type | Icon

+ |

+OATH software token | <img width="63" alt="Software OATH token" src="media/concept-authentication-methods/software-oath-token-icon.png">

+OATH hardware token | <img width="63" alt="Hardware OATH token" src="media/concept-authentication-methods/hardware-oath-token-icon.png">

+| **AuthenticationThrottled** | Too many attempts by user in a short period of time. Throttling. | Microsoft may limit repeated authentication attempts that are performed by the same user in a short period of time. This limitation does not apply to the Microsoft Authenticator or verification code. If you have hit these limits, you can use the Authenticator App, verification code or try to sign in again in a few minutes. |

+| **AuthenticationMethodLimitReached** | Authentication Method Limit Reached. Throttling. | Microsoft may limit repeated authentication attempts that are performed by the same user using the same authentication method type in a short period of time, specifically Voice call or SMS. This limitation does not apply to the Microsoft Authenticator or verification code. If you have hit these limits, you can use the Authenticator App, verification code or try to sign in again in a few minutes.|

+Permissions Management is a cloud infrastructure entitlement management (CIEM) solution that provides comprehensive visibility into permissions assigned to all identities. For example, over-privileged workload and user identities, actions, and resources across multicloud infrastructures in Microsoft Azure, Amazon Web Services (AWS), and Google Cloud Platform (GCP). Permissions Management detects, automatically right-sizes, and continuously monitors unused and excessive permissions. It deepens the Zero Trust security strategy by augmenting the least privilege access principle.

+Yes, non-Azure customers can use our solution. Permissions Management is a multicloud solution so even customers who have no subscription to Azure can benefit from it.

+Permissions Management complements Azure AD PIM. Azure AD PIM provides just-in-time access for admin roles in Azure (as well as Microsoft Online Services and apps that use groups), while Permissions Management allows multicloud discovery, remediation, and monitoring of privileged access across Azure, AWS, and GCP.

+If a customer initiates a free Permissions Management 45-day trial, but does not follow up and convert to a paid license within 45 days of the free trial expiration, we will delete all collected data on or just before 45 days.

+If a customer decides to discontinue licensing the service, we will also delete all previously collected data within 45 days of license termination.

+Yes, as of July 1st, 2022, new customers must acquire a free 45-day trial license or a paid license to use the service. You can enable a trial here: [https://aka.ms/TryPermissionsManagement](https://aka.ms/TryPermissionsManagement) or you can directly purchase resource-based licenses here: [https://aka.ms/BuyPermissionsManagement](https://aka.ms/BuyPermissionsManagement)

+After October 1st you will need to move over to use the newly released version of the service and enable a 45-day trial or purchase licenses to continue using the service.

+This article describes how to generate and download the **Permissions analytics report** in Permissions Management for AWS, Azure, and GCP. You can generate the report in Excel format, and also as a PDF.

+1. Select **Permissions Analytics Report** from the list. o download the report, select the down arrow to the right of the report name, or from the ellipses **(...)** menu, select **Download**.

+You can use the cloud sync attribute mapping feature to map attributes between your on-premises user or group objects and the objects in Azure AD.

+ :::image type="content" source="media/how-to-attribute-mapping/new-ux-mapping-1.png" alt-text="Screenshot of new UX screen attribute mapping." lightbox="media/how-to-attribute-mapping/new-ux-mapping-1.png":::

+Along with the type property, attribute mappings support certain attributes. These attributes will depend on the type of mapping you have selected. The following sections describe the supported attribute mappings for each of the individual types. The following type of attribute mapping is available.

+- Direct

+- Constant

+- Expression

+To use attribute mapping, follow these steps:

+ 1. In the Azure portal, select **Azure Active Directory**.

+ 2. On the left, select **Azure AD Connect**.

+ 3. On the left, select **Cloud sync**.

+

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-1.png" alt-text="Screenshot of new UX screen." lightbox="media/how-to-on-demand-provision/new-ux-1.png":::

+ 4. Under **Configuration**, select your configuration.

+ 5. On the left, select **Attribute mapping**.

+ 6. At the top, ensure that you have the correct object type selected. That is, user, group, or contact.

+ 7. Click **Add attribute mapping**.

+ :::image type="content" source="media/how-to-attribute-mapping/new-ux-mapping-3.png" alt-text="Screenshot of adding an attribute mapping." lightbox="media/how-to-attribute-mapping/new-ux-mapping-3.png":::

+ 8. Select the mapping type. This can be one of the following:

+

+ 9. Depending on what you have selected in the previous step, different options will be available for filling in.

+ 10. Select when to apply this mapping, and then select **Apply**.

+ :::image type="content" source="media/how-to-attribute-mapping/new-ux-mapping-4.png" alt-text="Screenshot of saving an attribute mapping." lightbox="media/how-to-attribute-mapping/new-ux-mapping-4.png":::

+ 11. Back on the **Attribute mappings** screen, you should see your new attribute mapping.

+ 12. Select **Save schema**. You will be notified that once you save the schema, a synchronization will occur. Click **OK**.

+ :::image type="content" source="media/how-to-attribute-mapping/new-ux-mapping-5.png" alt-text="Screenshot of saving schema." lightbox="media/how-to-attribute-mapping/new-ux-mapping-5.png":::

+ 13. Once the save is successful you will see a notification on the right.

+ :::image type="content" source="media/how-to-attribute-mapping/new-ux-mapping-6.png" alt-text="Screenshot of successful schema save." lightbox="media/how-to-attribute-mapping/new-ux-mapping-6.png":::

+ 1. In the Azure portal, select **Azure Active Directory**.

+ 2. On the left, select **Azure AD Connect**.

+ 3. On the left, select **Cloud sync**.

+ 4. Under **Configuration**, select your configuration.

+ 5. On the left, select **Provision on demand**.

+ 6. Enter the distinguished name of a user and select the **Provision** button.

+

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-2.png" alt-text="Screenshot of user distinguished name." lightbox="media/how-to-on-demand-provision/new-ux-2.png":::

+ 7. After provisioning finishes, a success screen appears with four green check marks. Any errors appear to the left.

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-3.png" alt-text="Screenshot of on-demand success." lightbox="media/how-to-on-demand-provision/new-ux-3.png":::

+The following document will guide you through configuring Azure AD Connect cloud sync.

+The following documentation demonstrates the new guided user experience for Azure AD Connect cloud sync. If you are not seeing the images below, you need to select the **Preview features** at the top. You can select this again to revert back to the old experience.

+ :::image type="content" source="media/how-to-configure/new-ux-configure-19.png" alt-text="Screenshot of enable preview features." lightbox="media/how-to-configure/new-ux-configure-19.png":::

+For additional information and an example of how to configure cloud sync, see the video below.

+ 1. In the Azure portal, select **Azure Active Directory**.

+ 2. On the left, select **Azure AD Connect**.

+ 3. On the left, select **Cloud sync**.

+

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-1.png" alt-text="Screenshot of new UX screen." lightbox="media/how-to-on-demand-provision/new-ux-1.png":::

+ :::image type="content" source="media/how-to-configure/new-ux-configure-1.png" alt-text="Screenshot of adding a configuration." lightbox="media/how-to-configure/new-ux-configure-1.png":::

+ :::image type="content" source="media/how-to-configure/new-ux-configure-2.png" alt-text="Screenshot of a new configuration." lightbox="media/how-to-configure/new-ux-configure-2.png":::

+ 6. The **Get started** screen will open. From here, you can continue configuring cloud sync.

+ :::image type="content" source="media/how-to-configure/new-ux-configure-3.png" alt-text="Screenshot of the getting started screen." lightbox="media/how-to-configure/new-ux-configure-3.png":::

+ 7. The configuration is split in to the following 5 sections.

+|Section|Description|

+|--|--|

+|1. Add [scoping filters](#scope-provisioning-to-specific-users-and-groups)|Use this section to define what objects appear in Azure AD|

+|2. Map [attributes](#attribute-mapping)|Use this section to map attributes between your on-premises users/groups with Azure AD objects|

+|3. [Test](#on-demand-provisioning)|Test your configuration before deploying it|

+|4. View [default properties](#accidental-deletions-and-email-notifications)|View the default setting prior to enabling them and make changes where appropriate|

+|5. Enable [your configuration](#enable-your-configuration)|Once ready, enable the configuration and users/groups will begin synchronizing|

+You can scope the agent to synchronize specific users and groups by using on-premises Active Directory groups or organizational units.

+ :::image type="content" source="media/how-to-configure/new-ux-configure-4.png" alt-text="Screenshot of scoping filters icon." lightbox="media/how-to-configure/new-ux-configure-4.png":::

+You can't configure groups and organizational units within a configuration.

+ 1. On the **Getting started** configuration screen. Click either **Add scoping filters** next to the **Add scoping filters** icon or on the click **Scoping filters** on the left under **Manage**.

+ :::image type="content" source="media/how-to-configure/new-ux-configure-5.png" alt-text="Screenshot of scoping filters." lightbox="media/how-to-configure/new-ux-configure-5.png":::

+ 2. Select the scoping filter. The filter can be one of the following:

+ - **All users**: Scopes the configuration to apply to all users that are being synchronized.

+ - **Selected security groups**: Scopes the configuration to apply to specific security groups.

+ - **Selected organizational units**: Scopes the configuration to apply to specific OUs.

+ 3. For security groups and organizational units, supply the appropriate distinguished name and click **Add**.

+ 4. Once your scoping filters are configured, click **Save**.

+ 5. After saving, you should see a message telling you what you still need to do to configure cloud sync. You can click the link to continue.

+ :::image type="content" source="media/how-to-configure/new-ux-configure-16.png" alt-text="Screenshot of the nudge for scoping filters." lightbox="media/how-to-configure/new-ux-configure-16.png":::

+ 7. Once you've changed the scope, you should [restart provisioning](#restart-provisioning) to initiate an immediate synchronization of the changes.

+Azure AD Connect cloud sync allows you to easily map attributes between your on-premises user/group objects and the objects in Azure AD.

+You can customize the default attribute-mappings according to your business needs. So, you can change or delete existing attribute-mappings, or create new attribute-mappings.

+After saving, you should see a message telling you what you still need to do to configure cloud sync. You can click the link to continue.

+ :::image type="content" source="media/how-to-configure/new-ux-configure-17.png" alt-text="Screenshot of the nudge for attribute filters." lightbox="media/how-to-configure/new-ux-configure-17.png":::

+For more information, see [attribute mapping](how-to-attribute-mapping.md).

+Azure AD Connect cloud sync allows you to test configuration changes, by applying these changes to a single user or group.

+You can use this to validate and verify that the changes made to the configuration were applied properly and are being correctly synchronized to Azure AD.

+After testing, you should see a message telling you what you still need to do to configure cloud sync. You can click the link to continue.

+ :::image type="content" source="media/how-to-configure/new-ux-configure-18.png" alt-text="Screenshot of the nudge for testing." lightbox="media/how-to-configure/new-ux-configure-18.png":::

+For more information, see [on-demand provisioning](how-to-on-demand-provision.md).

+## Accidental deletions and email notifications

+The default properties section provides information on accidental deletions and email notifications.

+The accidental delete feature is designed to protect you from accidental configuration changes and changes to your on-premises directory that would affect many users and groups.

+This feature allows you to:

+Click the **pencil** next to **Basics** to change the defaults in a configuration.

+## Enable your configuration

+Once you've finalized and tested your configuration, you can enable it.

+Click **Enable configuration** to enable it.

+If you don't want to wait for the next scheduled run, trigger the provisioning run by using the **Restart sync** button.

+ 2. On the left, select **Azure AD Connect**.

+ 3. On the left, select **Cloud sync**.

+ :::image type="content" source="media/how-to-configure/new-ux-configure-14.png" alt-text="Screenshot of restarting sync." lightbox="media/how-to-configure/new-ux-configure-14.png":::

+ 5. At the top, select **Restart sync**.

+ 2. On the left, select **Azure AD Connect**.

+ 3. On the left, select **Cloud sync**.

+ :::image type="content" source="media/how-to-configure/new-ux-configure-15.png" alt-text="Screenshot of deletion." lightbox="media/how-to-configure/new-ux-configure-15.png":::

+ 5. At the top of the configuration screen, select **Delete configuration**.

+ 1. In the Azure portal, select **Azure Active Directory**.

+ 2. On the left, select **Azure AD Connect**.

+ 3. On the left, select **Cloud sync**.

+

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-1.png" alt-text="Screenshot of new UX screen." lightbox="media/how-to-on-demand-provision/new-ux-1.png":::

+ 4. Under **Configuration**, select your configuration.

+ 5. On the left, select **Provision on demand**.

+ 6. Enter the distinguished name of a user and select the **Provision** button.

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-2.png" alt-text="Screenshot of user distinguished name." lightbox="media/how-to-on-demand-provision/new-ux-2.png":::

+ 7. After provisioning finishes, a success screen appears with four green check marks. Any errors appear to the left.

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-3.png" alt-text="Screenshot of on-demand success." lightbox="media/how-to-on-demand-provision/new-ux-3.png":::

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-4.png" alt-text="Screenshot of import user." lightbox="media/how-to-on-demand-provision/new-ux-4.png":::

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-5.png" alt-text="Screenshot of scope determination." lightbox="media/how-to-on-demand-provision/new-ux-5.png":::

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-6.png" alt-text="Screenshot of matching user." lightbox="media/how-to-on-demand-provision/new-ux-6.png":::

+ :::image type="content" source="media/how-to-on-demand-provision/new-ux-7.png" alt-text="Screenshot of perform action." lightbox="media/how-to-on-demand-provision/new-ux-7.png":::

+ Title: 'How to use single sign-on with cloud sync'

+description: This article describes how to install and use single sign-on with cloud sync.

+# Using single sign-on with cloud sync

+|HybridSynchronizationActiveDirectoryInternalServerError|Error Message: We were unable to process this request at this point. If this issue persists, please contact support and provide the following job identifier: AD2AADProvisioning.30b500eaf9c643b2b78804e80c1421fe.5c291d3c-d29f-4570-9d6b-f0c2fa3d5926. Additional details: Processing of the HTTP request resulted in an exception. |Couldn't process the parameters received in SCIM request to a Search request.|Please see the HTTP response returned by the 'Response' property of this exception for details.|

+|HybridIdentityServiceNoAgentsAssigned|Error Message: We're unable to find an active agent for the domain you're trying to sync. Please check to see if the agents have been removed. If so, re-install the agent again.|There are no agents running. Probably agents have been removed. Register a new agent.|"In this case, you won't see any agent assigned to the domain in portal.|

+|HybridIdentityServiceNoActiveAgents|Error Message: We're unable to find an active agent for the domain you're trying to sync. Please check to see if the agent is running by going to the server, where the agent is installed, and check to see if "Microsoft Azure AD Cloud Sync Agent" under Services is running.|"Agents aren't listening to the ServiceBus endpoint. [The agent is behind a firewall that doesn't allow connections to service bus](../app-proxy/application-proxy-configure-connectors-with-proxy-servers.md#use-the-outbound-proxy-server)|

+|HybridIdentityServiceInvalidResource|Error Message: We were unable to process this request at this point. If this issue persists, please contact support and provide the following job identifier: AD2AADProvisioning.3a2a0d8418f34f54a03da5b70b1f7b0c.d583d090-9cd3-4d0a-aee6-8d666658c3e9. Additional details: There seems to be an issue with your cloud sync setup. Please re-register your cloud sync agent on your on-premises AD domain and restart configuration from Azure portal.|The resource name must be set so HIS knows which agent to contact.|Please re-register your cloud sync agent on your on-premises AD domain and restart configuration from Azure portal.|

+|HybridIdentityServiceAgentSignalingError|Error Message: We were unable to process this request at this point. If this issue persists, please contact support and provide the following job identifier: AD2AADProvisioning.92d2e8750f37407fa2301c9e52ad7e9b.efb835ef-62e8-42e3-b495-18d5272eb3f9. Additional details: We were unable to process this request at this point. If this issue persists, please contact support with Job ID (from status pane of your configuration).|Service Bus isn't able to send a message to the agent. Could be an outage in service bus, or the agent isn't responsive.|If this issue persists, please contact support with Job ID (from status pane of your configuration).|

+|AzureActiveDirectoryInvalidCredential|Error Message: We found an issue with the service account that is used to run Azure AD Connect Cloud Sync. You can repair the cloud service account by following the instructions at [here](./how-to-troubleshoot.md). If the error persists, please contact support with Job ID (from status pane of your configuration). Additional Error Details: CredentialsInvalid AADSTS50034: The user account {EmailHidden} doesn't exist in the skydrive365.onmicrosoft.com directory. To sign into this application, the account must be added to the directory. Trace ID: 14b63033-3bc9-4bd4-b871-5eb4b3500200 Correlation ID: 57d93ed1-be4d-483c-997c-a3b6f03deb00 Timestamp: 2021-01-12 21:08:29Z |This error is thrown when the sync service account ADToAADSyncServiceAccount doesn't exist in the tenant. It can be due to accidental deletion of the account.|Use [Repair-AADCloudSyncToolsAccount](reference-powershell.md#repair-aadcloudsynctoolsaccount) to fix the service account.|

+|AzureActiveDirectoryExpiredCredentials|Error Message: We were unable to process this request at this point. If this issue persists, please contact support with Job ID (from status pane of your configuration). Additional Error Details: CredentialsExpired AADSTS50055: The password is expired. Trace ID: 989b1841-dbe5-49c9-ab6c-9aa25f7b0e00 Correlation ID: 1c69b196-1c3a-4381-9187-c84747807155 Timestamp: 2021-01-12 20:59:31Z | Response status code doesn't indicate success: 401 (Unauthorized).<br> Azure AD Sync service account credentials are expired.|You can repair the cloud service account by following the instructions at https://go.microsoft.com/fwlink/?linkid=2150988. If the error persists, please contact support with Job ID (from status pane of your configuration). Additional Error Details: Your administrative Azure Active Directory tenant credentials were exchanged for an OAuth token that has since expired."|

+- A computer with [Hyper-V](/windows-server/virtualization/hyper-v/hyper-v-technology-overview) installed. It's suggested to do this on either a [Windows 10](/virtualization/hyper-v-on-windows/about/supported-guest-os) or a [Windows Server 2016](/windows-server/virtualization/hyper-v/supported-windows-guest-operating-systems-for-hyper-v-on-windows) computer.

+3. You'll be prompted to ΓÇÿPress any key to boot from CD or DVDΓÇÖ. Go ahead and do so.

+Now that you have an Azure AD tenant, you'll create a global administrator account. To create the global administrator account do the following.

+3. Provide a name and username for this user. This will be your Global Admin for the tenant. You'll also want to change the **Directory role** to **Global administrator.** You can also show the temporary password. When you're done, select **Create**.</br>

+5. Change the password for the global administrator to something that you'll remember.

+3. You'll be prompted to ΓÇÿPress any key to boot from CD or DVDΓÇÖ. Go ahead and do so.

+ Title: Tutorial - Integrate an existing forest and a new forest with a single Azure AD tenant using Azure AD Connect cloud sync.

+# Integrate an existing forest and a new forest with a single Azure AD tenant

+In this scenario, there's an existing forest synced using Azure AD Connect sync to an Azure AD tenant. And you have a new forest that you want to sync to the same Azure AD tenant. You'll set up cloud sync for the new forest.

+### In the Azure Active Directory admin center

+1. Create a cloud-only global administrator account on your Azure AD tenant. This way, you can manage the configuration of your tenant should your on-premises services fail or become unavailable. Learn about [adding a cloud-only global administrator account](../fundamentals/add-users-azure-active-directory.md). Completing this step is critical to ensure that you don't get locked out of your tenant.

+2. Add one or more [custom domain names](../fundamentals/add-custom-domain.md) to your Azure AD tenant. Your users can sign in with one of these domain names.

+1. Identify a domain-joined host server running Windows Server 2012 R2 or greater with minimum of 4-GB RAM and .NET 4.7.1+ runtime

+2. If there's a firewall between your servers and Azure AD, configure the following items:

+ | **80** | Downloads the certificate revocation lists (CRLs) while validating the TLS/SSL certificate |

+ | **443** | Handles all outbound communication with the service |

+ | **8080** (optional) | Agents report their status every 10 minutes over port 8080, if port 443 is unavailable. This status is displayed on the Azure AD portal. |

+ - If your firewall or proxy allows you to specify safe suffixes, then add connections to **\*.msappproxy.net** and **\*.servicebus.windows.net**. If not, allow access to the [Azure datacenter IP ranges](https://www.microsoft.com/download/details.aspx?id=41653), which are updated weekly.

+ - For certificate validation, unblock the following URLs: **mscrl.microsoft.com:80**, **crl.microsoft.com:80**, **ocsp.msocsp.com:80**, and **www\.microsoft.com:80**. Since these URLs are used for certificate validation with other Microsoft products, you may already have these URLs unblocked.

+If you're using the [Basic AD and Azure environment](tutorial-basic-ad-azure.md) tutorial, it would be DC1. To install the agent, follow these steps:

+ Use the following steps to configure provisioning

+2. Select **Azure Active Directory**

+3. Select **Azure AD Connect**

+4. Select **Manage cloud sync**

+ 

+5. Select **New Configuration**

+ 

+7. On the configuration screen, enter a **Notification email**, move the selector to **Enable** and select **Save**.

+ 

+ 

+## Verify users are created and synchronization is occurring

+You'll now verify that the users that you had in our on-premises directory have been synchronized and now exist in our Azure AD tenant. This process may take a few hours to complete. To verify users are synchronized, do the following:

+1. Browse to the [Azure portal](https://portal.azure.com) and sign in with an account that has an Azure subscription.

+2. On the left, select **Azure Active Directory**

+3. Under **Manage**, select **Users**.

+4. Verify that you see the new users in our tenant

+## Test signing in with one of our users

+1. Browse to [https://myapps.microsoft.com](https://myapps.microsoft.com)

+2. Sign in with a user account that was created in our new tenant. You'll need to sign in using the following format: (user@domain.onmicrosoft.com). Use the same password that the user uses to sign in on-premises.

+ 

+ Title: Tutorial - Pilot Azure AD Connect cloud sync for an existing synced AD forest

+description: Learn how to pilot cloud sync for a test Active Directory forest that is already synced using Azure Active Directory (Azure AD) Connect sync.

+# Pilot cloud sync for an existing synced AD forest

+This tutorial walks you through piloting cloud sync for a test Active Directory forest that is already synced using Azure Active Directory (Azure AD) Connect sync.

+Before you try this tutorial, consider the following items:

+1. Ensure that you're familiar with basics of cloud sync.

+1. Ensure that you're running Azure AD Connect sync version 1.4.32.0 or later and have configured the sync rules as documented.

+1. When piloting, you'll be removing a test OU or group from Azure AD Connect sync scope. Moving objects out of scope leads to deletion of those objects in Azure AD.

+ - User objects, the objects in Azure AD are soft-deleted and can be restored.

+ - Group objects, the objects in Azure AD are hard-deleted and can't be restored.

+ A new link type has been introduced in Azure AD Connect sync, which will prevent the deletion in a piloting scenario.

+1. Ensure that the objects in the pilot scope have ms-ds-consistencyGUID populated so cloud sync hard matches the objects.

+ > Azure AD Connect sync does not populate *ms-ds-consistencyGUID* by default for group objects.

+1. This configuration is for advanced scenarios. Ensure that you follow the steps documented in this tutorial precisely.

+The following are prerequisites required for completing this tutorial

+- A test environment with Azure AD Connect sync version 1.4.32.0 or later

+- An OU or group that is in scope of sync and can be used the pilot. We recommend starting with a small set of objects.

+- A server running Windows Server 2012 R2 or later that will host the provisioning agent.

+- Source anchor for Azure AD Connect sync should be either *objectGuid* or *ms-ds-consistencyGUID*

+## Update Azure AD Connect

+As a minimum, you should have [Azure AD connect](https://www.microsoft.com/download/details.aspx?id=47594) 1.4.32.0. To update Azure AD Connect sync, complete the steps in [Azure AD Connect: Upgrade to the latest version](../hybrid/how-to-upgrade-previous-version.md).

+Azure AD Connect sync synchronizes changes occurring in your on-premises directory using a scheduler. In order to modify and add custom rules, you want to disable the scheduler so that synchronizations won't run while you're working making the changes. To stop the scheduler, use the following steps:

+1. On the server that is running Azure AD Connect sync open PowerShell with Administrative Privileges.

+2. Run `Stop-ADSyncSyncCycle`. Hit Enter.

+3. Run `Set-ADSyncScheduler -SyncCycleEnabled $false`.

+>If you are running your own custom scheduler for Azure AD Connect sync, then please disable the scheduler.

+## Create custom user inbound rule

+ 1. Launch the synchronization editor from the application menu in desktop as shown below:

+ 

+ 2. Select **Inbound** from the drop-down list for Direction and select **Add new rule**.

+ 

+ 3. On the **Description** page, enter the following and select **Next**:

+ - **Name:** Give the rule a meaningful name

+ - **Description:** Add a meaningful description

+ - **Connected System:** Choose the AD connector that you're writing the custom sync rule for

+ - **Connected System Object Type:** User

+ - **Metaverse Object Type:** Person

+ - **Link Type:** Join

+ - **Precedence:** Provide a value that is unique in the system

+ - **Tag:** Leave this empty

+ 4. On the **Scoping filter** page, enter the OU or security group that you want the pilot based off. To filter on OU, add the OU portion of the distinguished name. This rule will be applied to all users who are in that OU. So, if DN ends with "OU=CPUsers,DC=contoso,DC=com, you would add this filter. Then select **Next**.

+ |Scoping OU|DN|ENDSWITH|Distinguished name of the OU.|

+ |Scoping group||ISMEMBEROF|Distinguished name of the security group.|

+ 

+ 5. On the **Join** rules page, select **Next**.

+ 6. On the **Transformations** page, add a Constant transformation: flow True to cloudNoFlow attribute. Select **Add**.

+Same steps need to be followed for all object types (user, group and contact). Repeat steps per configured AD Connector / per AD forest.

+## Create custom user outbound rule

+ 1. Select **Outbound** from the drop-down list for Direction and select **Add rule**.

+ 

+ 2. On the **Description** page, enter the following and select **Next**:

+ - **Name:** Give the rule a meaningful name

+ - **Description:** Add a meaningful description

+ - **Connected System:** Choose the Azure AD connector that you're writing the custom sync rule for

+ - **Connected System Object Type:** User

+ - **Metaverse Object Type:** Person

+ - **Link Type:** JoinNoFlow

+ - **Precedence:** Provide a value that is unique in the system<br>

+ - **Tag:** Leave this empty

+ 

+ 3. On the **Scoping filter** page, choose **cloudNoFlow** equal **True**. Then select **Next**.

+ 4. On the **Join** rules page, select **Next**.

+ 5. On the **Transformations** page, select **Add**.

+Same steps need to be followed for all object types (user, group and contact).

+If you're using the [Basic AD and Azure environment](tutorial-basic-ad-azure.md) tutorial, it would be CP1. To install the agent, follow these steps:

+## Verify agent installation

+Use the following steps to configure provisioning:

+1. Sign-in to the Azure AD portal.

+2. Select **Azure Active Directory**

+3. Select **Azure AD Connect**

+4. Select **Manage cloud sync**

+ 

+5. Select **New Configuration**

+ 

+6. On the configuration screen, enter a **Notification email**, move the selector to **Enable** and select **Save**.

+ 

+7. Under **Configure**, select **All users** to change the scope of the configuration rule.

+ 

+8. On the right, change the scope to include the specific OU you created "OU=CPUsers,DC=contoso,DC=com".

+ 

+9. Select **Done** and **Save**.

+10. The scope should now be set to one organizational unit.

+ 

+## Verify users are provisioned by cloud sync

+You'll now verify that the users that you had in our on-premises directory have been synchronized and now exist in out Azure AD tenant. This process may take a few hours to complete. To verify users are provisioning by cloud sync, follow these steps:

+1. Browse to the [Azure portal](https://portal.azure.com) and sign in with an account that has an Azure subscription.

+2. On the left, select **Azure Active Directory**

+3. Select on **Azure AD Connect**

+4. Select on **Manage cloud sync**

+5. Select on **Logs** button

+6. Search for a username to confirm that the user is provisioned by cloud sync

+Azure AD Connect sync synchronizes changes occurring in your on-premises directory using a scheduler. Now that you've modified the rules, you can restart the scheduler. Use the following steps:

+1. On the server that is running Azure AD Connect sync open PowerShell with Administrative Privileges

+2. Run `Set-ADSyncScheduler -SyncCycleEnabled $true`.

+3. Run `Start-ADSyncSyncCycle`, then press <kbd>Enter</kbd>.

+> If you are running your own custom scheduler for Azure AD Connect sync, then please enable the scheduler.

+Once the scheduler is enabled, Azure AD Connect will stop exporting any changes on objects with `cloudNoFlow=true` in the metaverse, unless any reference attribute (such as `manager`) is being updated. In case there's any reference attribute update on the object, Azure AD Connect will ignore the `cloudNoFlow` signal and export all updates on the object.

+## Something went wrong

+In case the pilot doesn't work as expected, you can go back to the Azure AD Connect sync setup by following the steps below:

+1. Disable provisioning configuration in the Azure portal.

+2. Disable all the custom sync rules created for Cloud Provisioning using the Sync Rule Editor tool. Disabling should cause full sync on all the connectors.

+description: This topic describes the pre-requisites and the hardware requirements cloud sync.

+This tutorial walks you through creating a hybrid identity environment using Azure Active Directory (Azure AD) Connect cloud sync.

+1. Create a cloud-only global administrator account on your Azure AD tenant. This way, you can manage the configuration of your tenant should your on-premises services fail or become unavailable. Learn about [adding a cloud-only global administrator account](../fundamentals/add-users-azure-active-directory.md). Completing this step is critical to ensure that you don't get locked out of your tenant.

+2. Add one or more [custom domain names](../fundamentals/add-custom-domain.md) to your Azure AD tenant. Your users can sign in with one of these domain names.

+1. Identify a domain-joined host server running Windows Server 2016 or greater with minimum of 4-GB RAM and .NET 4.7.1+ runtime

+2. If there's a firewall between your servers and Azure AD, configure the following items:

+ | **80** | Downloads the certificate revocation lists (CRLs) while validating the TLS/SSL certificate |

+ | **443** | Handles all outbound communication with the service |

+ | **8080** (optional) | Agents report their status every 10 minutes over port 8080, if port 443 is unavailable. This status is displayed on the Azure AD portal. |

+ - If your firewall or proxy allows you to specify safe suffixes, then add connections t to **\*.msappproxy.net** and **\*.servicebus.windows.net**. If not, allow access to the [Azure datacenter IP ranges](https://www.microsoft.com/download/details.aspx?id=41653), which are updated weekly.

+ - For certificate validation, unblock the following URLs: **mscrl.microsoft.com:80**, **crl.microsoft.com:80**, **ocsp.msocsp.com:80**, and **www\.microsoft.com:80**. Since these URLs are used for certificate validation with other Microsoft products, you may already have these URLs unblocked.

+If you're using the [Basic AD and Azure environment](tutorial-basic-ad-azure.md) tutorial, it would be DC1. To install the agent, follow these steps:

+Use the following steps to configure and start the provisioning:

+1. Sign in to the Azure AD portal.

+1. Select **Azure Active Directory**

+1. Select **Azure AD Connect**

+1. Select **Manage cloud sync**

+ 

+1. Select **New Configuration**

+

+ [](media/tutorial-single-forest/configure-1.png#lightbox)

+1. On the configuration screen, enter a **Notification email**, move the selector to **Enable** and select **Save**.

+ [](media/how-to-configure/configure-2.png#lightbox)

+1. The configuration status should now be **Healthy**.

+ [](media/how-to-configure/manage-4.png#lightbox)

+## Verify users are created and synchronization is occurring

+You'll now verify that the users that you had in your on-premises directory have been synchronized and now exist in your Azure AD tenant. The sync operation may take a few hours to complete. To verify users are synchronized, follow these steps:

+1. Browse to the [Azure portal](https://portal.azure.com) and sign in with an account that has an Azure subscription.

+2. On the left, select **Azure Active Directory**

+3. Under **Manage**, select **Users**.

+4. Verify that the new users appear in your tenant

+1. Browse to [https://myapps.microsoft.com](https://myapps.microsoft.com)

+1. Sign in with a user account that was created in your tenant. You'll need to sign in using the following format: (user@domain.onmicrosoft.com). Use the same password that the user uses to sign in on-premises.

+ 

+You've now successfully configured a hybrid identity environment using Azure AD Connect cloud sync.

+- [What is Azure AD Connect cloud provisioning?](what-is-cloud-sync.md)

+- v2.0 for applications that support consumer accounts. The following example shows a v2.0 token (this token example won't validate because the keys have rotated prior to publication and personal information has been removed):

+description: Describes how to mark an app as publisher verified. When an application is marked as publisher verified, it means that the publisher (application developer) has verified the authenticity of their organization using a Microsoft Partner Network (MPN) account that has completed the verification process and has associated this MPN account with that application registration.

+- Provision your app using the Azure portal. For more information, see the instructions for creating an app in [the Android tutorial](./tutorial-v2-android.md#create-a-project)

+- Integrate your application with the [MSAL for Android](https://github.com/AzureAD/microsoft-authentication-library-for-android)

+## Methods for SSO

+- Through a [broker application](#sso-through-brokered-authentication)

+- Through the [system browser](#sso-through-system-browser)

+ It's recommended to use a broker application for benefits like device-wide SSO, account management, and conditional access. However, it requires your users to download additional applications.

+We recommend that you use one of Microsoft's authentication brokers to participate in device-wide SSO and to meet organizational Conditional Access policies. Integrating with a broker provides the following benefits:

+- Device SSO

+ - via Android AccountManager & Account Settings

+The following diagram illustrates the relationship between your app, the MSAL, and Microsoft's authentication brokers.

+When a broker is installed on a device, all subsequent interactive token requests (calls to `acquireToken()`) are handled by the broker rather than locally by MSAL. Any SSO state previously available to MSAL isn't available to the broker. As a result, the user will need to authenticate again, or select an account from the existing list of accounts known to the device.

+If there's only one broker hosting app installed, and it's removed, then the user will need to sign in again. Uninstalling the active broker removes the account and associated tokens from the device.

+If Intune Company Portal is installed and is operating as the active broker, and Microsoft Authenticator is also installed, then if the Intune Company Portal (active broker) is uninstalled the user will need to sign in again. Once they sign in again, the Microsoft Authenticator app becomes the active broker.

+Once you've generated a signature hash with _keytool_, use the Azure portal to generate the redirect URI:

+1. Sign in to the <a href="https://portal.azure.com/" target="_blank">Azure portal</a>.

+1. If you have access to multiple tenants, use the **Directories + subscriptions** filter :::image type="icon" source="/azure/active-directory/develop/media/common/portal-directory-subscription-filter.png" border="false"::: in the top menu to switch to the tenant in which you registered your application.

+1. Search for and select **Azure Active Directory**.

+1. Under **Manage**, select **App registrations**.

+1. Under **Manage**, select **App registrations**, then select your application.

+1. Under **Manage**, select **Authentication** > **Add a platform** > **Android**.

+1. In the settings on your Android device, look for a newly created account corresponding to the account that you authenticated with. The account should be of type _Work account_.

+Android applications have the option to use the WebView, system browser, or Chrome Custom Tabs for authentication user experience. If the application isn't using brokered authentication, it will need to use the system browser rather than the native webview in order to achieve SSO.

+MSAL supports authorization using a `WebView`, or the system browser. The image below shows how it looks using the `WebView`, or the system browser with CustomTabs or without CustomTabs:

+### SSO implications

+If the application uses MSAL with a broker like Microsoft Authenticator or Intune Company Portal, then users can have SSO experience across applications if they have an active sign-in with one of the apps.

+When using the in-app `WebView`, the user signs in directly to the app. The tokens are kept inside the sandbox of the app and aren't available outside the app's cookie jar. As a result, the user can't have SSO experience across applications unless the apps integrate with the Authenticator or Company Portal.

+Use this approach to provide SSO experience through the device's browser. MSAL uses a shared cookie jar, which allows other native apps or web apps to achieve SSO on the device by using the persist session cookie set by MSAL.

+If there are no browser packages on the device, MSAL uses the in-app `WebView`. If the device default setting isn't changed, the same browser should be launched for each sign-in to ensure SSO experience.

+| Device | Built-in Browser | Chrome | Opera | Microsoft Edge | UC Browser | Firefox |

+| - | :--: | -: | -: | -: | -: | -: |

+| Nexus 4 (API 17) | pass | pass | not applicable | not applicable | not applicable | not applicable |

+| Samsung S7 (API 25) | pass¹ | pass | pass | pass | fail | pass |

+| Huawei (API 26) | pass² | pass | fail | pass | pass | pass |

+| Vivo (API 26) | pass | pass | pass | pass | pass | fail |

+| Pixel 2 (API 26) | pass | pass | pass | pass | fail | pass |

+| Oppo | pass | not applicable³ | not applicable | not applicable | not applicable | not applicable |

+| OnePlus (API 25) | pass | pass | pass | pass | fail | pass |

+| Nexus (API 28) | pass | pass | pass | pass | fail | pass |

+| MI | pass | pass | pass | pass | fail | pass |

+[Shared device mode for Android devices](msal-android-shared-devices.md) allows you to configure an Android device so that it can be easily shared by multiple employees.

+## Example: Securing a SPA with ADAL.js vs. MSAL.js

+Publisher verification gives app users and organization admins information about the authenticity of the developer's organization, who publishes an app that integrates with the Microsoft identity platform.

+An app that's publisher verified means that the app's publisher (app developer) has verified the authenticity of their organization with Microsoft. Verifying an app includes using a Microsoft Partner Network (MPN) account that's been [verified](/partner-center/verification-responses) and associating the MPN account with an app registration.

+description: Learn about the differences between the Microsoft Authentication Library for Objective-C (MSAL for iOS and macOS) and Azure AD Authentication Library for Objective-C (ADAL.ObjC) and how to migrate between them.

+# Using redirect URIs with the Microsoft Authentication Library (MSAL) for iOS and macOS

+The MSAL requires that the redirect URI be registered with the Azure AD app in a specific format. MSAL uses a default redirect URI, if you don't specify one. The format is `msauth.[Your_Bundle_Id]://auth`.

+However, you may need to change the redirect URI for advanced scenarios, as described in the following section.

+### Cross-app single sign-on (SSO)

+For the Microsoft identity platform to share tokens across apps, each app needs to have the same client ID or application ID. The client ID is the unique identifier provided when you registered your app in the Azure portal (not the application bundle ID that you register per app with Apple).

+- Client ID: `ABCDE-12345`

+- RedirectUris: `msauth.com.contoso.app1://auth`, `msauth.com.contoso.app2://auth`, `msauth.com.contoso.app3://auth`

+When migrating code that used the Azure Active Directory Authentication Library (ADAL) to MSAL, you may already have a redirect URI configured for your app. You can continue using the same redirect URI as long as your ADAL app was configured to support brokered scenarios and your redirect URI satisfies the MSAL redirect URI format requirements.

+- The MSAL redirect URI must be in the form `<scheme>://host`

+ Where `<scheme>` is a unique string that identifies your app. It's primarily based on the Bundle Identifier of your application to guarantee uniqueness. For example, if your app's Bundle ID is `com.contoso.myapp`, your redirect URI would be in the form: `msauth.com.contoso.myapp://auth`.

+ If you're migrating from ADAL, your redirect URI will likely have this format: `<scheme>://[Your_Bundle_Id]`, where `scheme` is a unique string. The format will continue to work when you use MSAL.

+- `<scheme>` must be registered in your app's Info.plist under `CFBundleURLTypes > CFBundleURLSchemes`. In this example, Info.plist has been opened as source code:

+ ```xml

+ <key>CFBundleURLTypes</key>

+ <array>

+ <dict>

+ <key>CFBundleURLSchemes</key>

+ <array>

+ <string>msauth.[BUNDLE_ID]</string>

+ </array>

+ </dict>

+ </array>

+ ```

+- If you want to use universal links as a redirect URI, the `<scheme>` must be `https` and doesn't need to be declared in `CFBundleURLSchemes`. Instead, configure the app and domain per Apple's instructions at [Universal Links for Developers](https://developer.apple.com/ios/universal-links/) and call the `handleMSALResponse:sourceApplication:` method of `MSALPublicClientApplication` when your application is opened through a universal link.

+To use a custom redirect URI, pass the `redirectUri` parameter to `MSALPublicClientApplicationConfig` and pass that object to `MSALPublicClientApplication` when you initialize the object. If the redirect URI is invalid, the initializer will return `nil` and set the `redirectURIError`with additional information. For example:

+ // continue on with application

+}

+ return [MSALPublicClientApplication handleMSALResponse:url

+| Microsoft Teams Premium | Microsoft_Teams_Premium | 989a1621-93bc-4be0-835c-fe30171d6463 | MICROSOFT_ECDN (85704d55-2e73-47ee-93b4-4b8ea14db92b)<br/>TEAMSPRO_MGMT (0504111f-feb8-4a3c-992a-70280f9a2869)<br/>TEAMSPRO_CUST (cc8c0802-a325-43df-8cba-995d0c6cb373)<br/>TEAMSPRO_PROTECTION (f8b44f54-18bb-46a3-9658-44ab58712968)<br/>TEAMSPRO_VIRTUALAPPT (9104f592-f2a7-4f77-904c-ca5a5715883f)<br/>MCO_VIRTUAL_APPT (711413d0-b36e-4cd4-93db-0a50a4ab7ea3)<br/>TEAMSPRO_WEBINAR (78b58230-ec7e-4309-913c-93a45cc4735b) | Microsoft eCDN (85704d55-2e73-47ee-93b4-4b8ea14db92b)<br/>Microsoft Teams Premium Intelligent (0504111f-feb8-4a3c-992a-70280f9a2869)<br/>Microsoft Teams Premium Personalized (cc8c0802-a325-43df-8cba-995d0c6cb373)<br/>Microsoft Teams Premium Secure (f8b44f54-18bb-46a3-9658-44ab58712968)<br/>Microsoft Teams Premium Virtual Appointment (9104f592-f2a7-4f77-904c-ca5a5715883f)<br/>Microsoft Teams Premium Virtual Appointments (711413d0-b36e-4cd4-93db-0a50a4ab7ea3)<br/>Microsoft Teams Premium Webinar (78b58230-ec7e-4309-913c-93a45cc4735b) |

+**What permissions are required to configure a SAML/Ws-Fed identity provider?**

+You need to be an [External Identity Provider Administrator](../roles/permissions-reference.md#external-identity-provider-administrator) or a [Global Administrator](../roles/permissions-reference.md#global-administrator) in your Azure AD tenant to configure a SAML/Ws-Fed identity provider.

+1. Sign in to the [Azure portal](https://portal.azure.com/) as an External Identity Provider Administrator or a Global Administrator.

+2. In the left pane, select **Azure Active Directory**.

+3. Select **External Identities** > **All identity providers**.

+4. Select **New SAML/WS-Fed IdP**.

+1. Sign in to the [Azure portal](https://portal.azure.com) as an External Identity Provider Administrator or a Global Administrator.

+2. In the left pane, select **Azure Active Directory**.

+3. Select **External Identities**.

+4. Select **All identity providers**.

+5. Under **SAML/WS-Fed identity providers**, scroll to an identity provider in the list or use the search box.

+6. To update the certificate or modify configuration details:

+1. Sign in to the [Azure portal](https://portal.azure.com) as an External Identity Provider Administrator or a Global Administrator.

+1. Sign in to the [Azure portal](https://portal.azure.com) as an External Identity Provider Administrator or a Global Administrator.

+2. In the left pane, select **Azure Active Directory**.

+3. Select **External Identities**.

+4. Select **All identity providers**, and then select the **Google** button.

+5. Enter the client ID and client secret you obtained earlier. Select **Save**:

+To configure federation with Google, Facebook, or a SAML/Ws-Fed identity provider, you'll need to be an [External Identity Provider Administrator](../roles/permissions-reference.md#external-identity-provider-administrator) or a [Global Administrator](../roles/permissions-reference.md#global-administrator) in your Azure AD tenant.

+ Title: Leave an organization as a guest user

+As an Azure Active Directory (Azure AD) B2B collaboration or B2B direct connect user, you can leave an organization at any time if you no longer need to use apps from that organization, or maintain any association.

+## Before you begin

+You can usually leave an organization on your own without having to contact an administrator. However, in some cases this option won't be available and you'll need to contact your tenant admin, who can delete your account in the external organization. This article is intended for administrators. If you're a user looking for information about how to manage and leave an organization, see the [Manage organizations article.](https://support.microsoft.com/account-billing/manage-organizations-for-a-work-or-school-account-in-the-my-account-portal-a9b65a70-fec5-4a1a-8e00-09f99ebdea17)

+ - If you're using a personal account or email one-time passcode, you'll need to use a My Account URL that includes your tenant name or tenant ID.

+ For example:

+ https://myaccount.microsoft.com?tenantId=wingtiptoys.onmicrosoft.com

+ or

+ https://myaccount.microsoft.com?tenantId=ab123456-cd12-ef12-gh12-ijk123456789.

+Permanent deletion can be initiated by the admin, or it happens at the end of the soft deletion period. Permanent deletion can take up to an extra 30 days for data removal.

+For B2B direct connect users, data removal begins as soon as the user selects **Leave** in the confirmation message and can take up to 30 days to complete.

+- Learn more about [user deletion](/compliance/regulatory/gdpr-dsr-azure#step-5-delete) and about how to delete a user's data when there's [no account in the Azure tenant](/compliance/regulatory/gdpr-dsr-azure#delete-a-users-data-when-there-is-no-account-in-the-azure-tenant).

+- For more information about GDPR, see the GDPR section of the [Service Trust portal](https://servicetrust.microsoft.com/ViewPage/GDPRGetStarted).

+ :::image type="content" source="media/self-service-sign-up-add-api-connector/api-connector-new.png" alt-text="Screenshot of adding a new API connector to External Identities.":::

+ :::image type="content" source="media/self-service-sign-up-add-api-connector/api-connector-config.png" alt-text="Screenshot of configuring an API connector.":::

+The exact claims sent to the API depend on which information is provided by the identity provider. 'email' is always sent.

+The exact claims sent to the API depend on which information is collected from the user or is provided by the identity provider.

+ When the API responds with a validation-error response, the user flow stays on the attribute collection page, and a `userMessage` is displayed to the user. The user can then edit and resubmit the form. This type of response can be used for input validation.

+| \<extension\_{extensions-app-id}\_CustomAttribute> | \<attribute-type> | No | The claim doesn't need to contain `_<extensions-app-id>_`, it's *optional*. Returned values can overwrite values collected from a user. |

+ "userMessage": "There was an error with your request. Please try again or contact support.",

+ * If using a serverless function or scalable web service, use a hosting plan that keeps the API "awake" or "warm" in production. For Azure Functions, it's recommended to use at minimum the [Premium plan](../../azure-functions/functions-scale.md#overview-of-plans)

+ * [Microsoft Defender for Cloud Apps overview](/defender-cloud-apps/what-is-defender-for-cloud-apps)

+* [Block OneDrive use from Office](/office365/troubleshoot/group-policy/block-onedrive-use-from-office)

+ * See, [Passwordless authentication options for Azure AD](../authentication/concept-authentication-passwordless.md)

+ * See, [What is SSO in Azure AD?](../manage-apps/what-is-single-sign-on.md)

+ * See, [What is federation with Azure AD?](../hybrid/whatis-fed.md)

+ * See, [Remote access to on-premises applications through Azure AD Application Proxy](../app-proxy/application-proxy.md)

+ * See, [Azure Active Directory Seamless SSO: Technical deep dive](../hybrid/how-to-connect-sso-how-it-works.md)

+ * See, [What are access reviews?](../governance/access-reviews-overview.md)

+Learn more: [Dynamic membership rules for groups in Azure Active Directory](../enterprise-users/groups-dynamic-membership.md)

+Learn more: [Include the right stakeholders](active-directory-deployment-plans.md)

+Developers can use the platform for both internal-use apps and customer facing apps, and there are other benefits that come with using the platform. [Microsoft Authentication Libraries (MSAL)](../develop/msal-overview.md), which is part of the platform, allows developers to enable modern experiences like multi-factor authentication and the use of security keys to access their apps without needing to implement it themselves. Additionally, apps integrated with the Microsoft identity platform can access [Microsoft Graph](/graph/overview) - a unified API endpoint providing the Azure AD data that describes the patterns of productivity, identity, and security in an organization. Developers can use this information to implement features that increase productivity for your users. For example, by identifying the people the user has been interacting with recently and surfacing them in the app's UI.

+## Governance benefits

+## Collaboration methods

+## Azure AD B2B benefits

+## December 2022

+### General Availability - Risk-based Conditional Access for workload identities

+**Type:** New feature

+**Service category:** Conditional Access

+**Product capability:** Identity Security & Protection

+Customers can now bring one of the most powerful forms of access control in the industry to workload identities. Conditional Access supports risk-based policies for workload identities. Organizations can block sign-in attempts when Identity Protection detects compromised apps or services. For more information, see: [Create a risk-based Conditional Access policy](../conditional-access/workload-identity.md#create-a-risk-based-conditional-access-policy).

+### General Availability - API to recover accidentally deleted Service Principals

+**Type:** New feature

+**Service category:** Enterprise Apps

+**Product capability:** Identity Lifecycle Management

+Restore a recently deleted application, group, servicePrincipal, administrative unit, or user object from deleted items. If an item was accidentally deleted, you can fully restore the item. This isn't applicable to security groups, which are deleted permanently. A recently deleted item will remain available for up to 30 days. After 30 days, the item is permanently deleted. For more information, see: [servicePrincipal resource type](/graph/api/resources/serviceprincipal).

+### General Availability - Using Staged rollout to test Cert Based Authentication (CBA)

+**Type:** New feature

+**Service category:** Authentications (Logins)

+**Product capability:** Identity Security & Protection

+We're excited to announce the general availability of hybrid cloud Kerberos trust, a new Windows Hello for Business deployment model to enable a password-less sign-in experience. With this new model, weΓÇÖve made Windows Hello for Business much easier to deploy than the existing key trust and certificate trust deployment models by removing the need for maintaining complicated public key infrastructure (PKI), and Azure Active Directory (AD) Connect synchronization wait times. For more information, see: [Migrate to cloud authentication using Staged Rollout](../hybrid/how-to-connect-staged-rollout.md).

+### General Availability - Windows Hello for Business, cloud Kerberos trust deployment

+### General Availability - Expression builder with Application Provisioning

+### General Availability - SSPR writeback is now available for disconnected forests using Azure AD Connect Cloud sync

+### General Availability - Prevent accidental deletions

+### General Availability - Create group in administrative unit

+### General Availability - Number matching for Microsoft Authenticator notifications

+### General Availability - Additional context in Microsoft Authenticator notifications

+ >[!IMPORTANT]

+ > If you change the suffix in Active Directory, add and verify a matching custom domain name in Azure AD.

+ > [Add your custom domain name using the Azure Active Directory portal](../fundamentals/add-custom-domain.md)

+ >[!NOTE]

+ > Define a process for when you update a User Principal Name (UPN) of a user, or for your organization.

+Use Teams Meeting Notes to take and share notes.

+## 19 January 2023

+**Agent Update**

+- Azure AD Connect Health agent for Azure AD Connect (version 3.2.2188.23)

+ - We fixed a bug where, under certain circumstances, Azure AD Connect sync errors were not getting uploaded or shown in the portal.

+Learn more about how to [integrate your on-premises identities with Azure AD](whatis-hybrid-identity.md).

+- [Azure AD Connect cloud sync](../cloud-sync/what-is-cloud-sync.md) -a new Microsoft agent designed to meet and accomplish your hybrid identity goals. It is provides a light-weight inter -directory provisioning experience between Active Directory and Azure AD.

+If you have verified that the application isn't in your tenant, proceed with any of the following ways to add the enterprise application to your tenant.

+- The client ID (also called appId in Microsoft Graph) of the multi-tenant application.

+You can use an API client such as [Graph Explorer](https://aka.ms/ge) to work with Microsoft Graph.

+1. Grant the client app the *Application.ReadWrite.All* permission.

+1. To create the enterprise application, run the following query. The appId is the client ID of the application.

+ POST https://graph.microsoft.com/v1.0/servicePrincipals

+ Content-type: application/json

+

+

+ ```

+1. To delete the enterprise application you created, run the query.

+ DELETE https://graph.microsoft.com/v1.0/servicePrincipals(appId='fc876dd1-6bcb-4304-b9b6-18ddf1526b62')

+1. To get the list of service principals in your tenant, run the following query.

+ GET https://graph.microsoft.com/v1.0/servicePrincipals

+ DELETE https://graph.microsoft.com/v1.0/servicePrincipals/{servicePrincipal-id}

+- Sign in to the Azure portal as a global administrator, application administrator or cloud application administrator for your directory.

+1. Select **App launchers**.

+2. Select **Settings**.

+3. For **Users can only see Microsoft 365 apps in the Microsoft 365 portal**, select **Yes**.

+4. Select **Save**.

+Learn more: [Zero Trust security](../../security/fundamentals/zero-trust.md)

+Your solution can enable the customer to use SSO and Azure Active Directory features, even unsupported applications. To allow access with legacy protocols, your application calls Azure AD to authenticate the user and apply [Azure AD Conditional Access policies](../conditional-access/overview.md). Enable this integration from your console. Create a SAML or an OIDC application registration between your solution and Azure AD.

+Customers and partners can use the Microsoft Graph API to create or apply per application [Conditional Access policies](../conditional-access/overview.md). For partners, customers can apply these policies from your solution without using the Azure portal. There are two options to apply Azure AD Conditional Access policies:

+- [Assign the application to a Conditional Access policy](#use-a-conditional-access-policy)

+- [Create a new Conditional Access policy and assign the application to it](#create-a-new-conditional-access-policy)

+ * [Tutorial: Integrate Zscaler Private Access with Azure AD](../saas-apps/zscalerprivateaccess-tutorial.md)

+ Title: Secure hybrid access, protect legacy apps with Azure Active Directory

+description: Find partner solutions to integrate your legacy on-premises, public cloud, or private cloud applications with Azure AD.

+# Secure hybrid access: Protect legacy apps with Azure Active Directory

+In this article, learn to protect your on-premises and cloud legacy authentication applications by connecting them to Azure Active Directory (Azure AD).

+* **[Application Proxy](#secure-hybrid-access-with-application-proxy)**:

+ * [Remote access to on-premises applications through Azure AD Application Proxy](../app-proxy/application-proxy.md)

+ * Protect users, apps, and data in the cloud and on-premises

+ * [Use it to publish on-premises web applications externally](../app-proxy/what-is-application-proxy.md)

+

+* **[Secure hybrid access through Azure AD partner integrations](#partner-integrations-for-apps-on-premises-and-legacy-authentication)**:

+ * [Pre-built solutions](#secure-hybrid-access-through-azure-ad-partner-integrations)

+ * [Apply Conditional Access policies per application](secure-hybrid-access-integrations.md#apply-conditional-access-policies)

+

+In addition to Application Proxy, you can strengthen your security posture with [Azure AD Conditional Access](../conditional-access/overview.md) and [Identity Protection](../identity-protection/overview-identity-protection.md).

+## Single sign-on and multi-factor authentication

+With Azure AD as an identity provider (IdP), you can use modern authentication and authorization methods like [single sign-on (SSO)](what-is-single-sign-on.md) and [Azure AD Multi-Factor Authentication (MFA)](../authentication/concept-mfa-howitworks.md) to secure legacy, on-premises applications.

+## Secure hybrid access with Application Proxy

+Use Application Proxy to protect users, apps, and data in the cloud, and on premises. Use this tool for secure remote access to on-premises web applications. Users donΓÇÖt need to use a virtual private network (VPN); they connect to applications from devices with SSO.

+Learn more:

+* [Remote access to on-premises applications through Azure AD Application Proxy](../app-proxy/application-proxy.md)

+* [Tutorial: Add an on-premises application for remote access through Application Proxy in Azure AD](../app-proxy/application-proxy-add-on-premises-application.md)

+* [How to configure SSO to an Application Proxy application](../app-proxy/application-proxy-config-sso-how-to.md)

+* [Using Azure AD Application Proxy to publish on-premises apps for remote users](../app-proxy/what-is-application-proxy.md)

+### Application publishing and access management

+Use Application Proxy remote access as a service to publish applications to users outside the corporate network. Help improve your cloud access management without requiring modification to your on-premises applications. Plan an [Azure AD Application Proxy deployment](../app-proxy/application-proxy-deployment-plan.md).

+## Partner integrations for apps: on-premises and legacy authentication

+Microsoft partners with various companies that deliver pre-built solutions for on-premises applications, and applications that use legacy authentication. The following diagram illustrates a user flow from sign-in to secure access to apps and data.

+ 

+### Secure hybrid access through Azure AD partner integrations

+The following partners offer solutions to support [Conditional Access policies per application](secure-hybrid-access-integrations.md#apply-conditional-access-policies). Use the tables in the following sections to learn about the partners and Azure AD integration documentation.

+|Partner|Integration documentation|

+|||

+|Akamai Technologies|[Tutorial: Azure AD SSO integration with Akamai](../saas-apps/akamai-tutorial.md)|

+|Citrix Systems, Inc.|[Tutorial: Azure AD SSO integration with Citrix ADC SAML Connector for Azure AD (Kerberos-based authentication)](../saas-apps/citrix-netscaler-tutorial.md)|

+|Datawiza|[Tutorial: Configure Secure Hybrid Access with Azure AD and Datawiza](datawiza-with-azure-ad.md)|

+|F5, Inc.|[Integrate F5 BIG-IP with Azure AD](f5-aad-integration.md)</br>[Tutorial: Configure F5 BIG-IP SSL-VPN for Azure AD SSO](f5-aad-password-less-vpn.md)|

+|Progress Software Corporation, Progress Kemp|[Tutorial: Azure AD SSO integration with Kemp LoadMaster Azure AD integration](../saas-apps/kemp-tutorial.md)|

+|Perimeter 81 Ltd.|[Tutorial: Azure AD SSO integration with Perimeter 81](../saas-apps/perimeter-81-tutorial.md)|

+|Silverfort|[Tutorial: Configure Secure Hybrid Access with Azure AD and Silverfort](silverfort-azure-ad-integration.md)|

+|Strata Identity, Inc.|[Integrate Azure AD SSO with Maverics Identity Orchestrator SAML Connector](../saas-apps/maverics-identity-orchestrator-saml-connector-tutorial.md)|

+#### Partners with pre-built solutions and integration documentation

+|Partner|Integration documentation|

+|||

+|Amazon Web Service, Inc.|[Tutorial: Azure AD SSO integration with AWS ClientVPN](../saas-apps/aws-clientvpn-tutorial.md)|

+|Check Point Software Technologies Ltd.|[Tutorial: Azure AD single SSO integration with Check Point Remote Secure Access VPN](../saas-apps/check-point-remote-access-vpn-tutorial.md)|

+|Cisco Systems, Inc.|[Tutorial: Azure AD SSO integration with Cisco AnyConnect](../saas-apps/cisco-anyconnect.md)|

+|Cloudflare, Inc.|[Tutorial: Configure Cloudflare with Azure AD for secure hybrid access](cloudflare-azure-ad-integration.md)|

+|Fortinet, Inc.|[Tutorial: Azure AD SSO integration with FortiGate SSL VPN](../saas-apps/fortigate-ssl-vpn-tutorial.md)|

+|Palo Alto Networks|[Tutorial: Azure AD SSO integration with Palo Alto Networks Admin UI](../saas-apps/paloaltoadmin-tutorial.md)|

+|Pulse Secure|[Tutorial: Azure AD SSO integration with Pulse Connect Secure (PCS)](../saas-apps/pulse-secure-pcs-tutorial.md)</br>[Tutorial: Azure AD SSO integration with Pulse Secure Virtual Traffic Manager](../saas-apps/pulse-secure-virtual-traffic-manager-tutorial.md)|

+|Zscaler, Inc.|[Tutorial: Integrate Zscaler Private Access with Azure AD](../saas-apps/zscalerprivateaccess-tutorial.md)|

+## Next steps

+Select a partner in the tables mentioned to learn how to integrate their solution with Azure AD.

+* [Analyze Azure AD activity logs with Azure Monitor logs](../reports-monitoring/howto-analyze-activity-logs-log-analytics.md).

+If using passwords, make sure the accounts have strong passwords that do not expire. Ideally, the passwords should be at least 16 characters long and randomly generated.

+ Title: Azure AD SSO integration with Radancy's Employee Referrals

+description: Learn how to configure single sign-on between Azure Active Directory and Radancy's Employee Referrals.

+# Azure AD SSO integration with Radancy's Employee Referrals

+In this tutorial, you'll learn how to integrate Radancy's Employee Referrals with Azure Active Directory (Azure AD). When you integrate Radancy's Employee Referrals with Azure AD, you can:

+* Control in Azure AD who has access to Radancy's Employee Referrals.

+* Enable your users to be automatically signed-in to Radancy's Employee Referrals with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+## Prerequisites

+To get started, you need the following items:

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* Radancy's Employee Referrals single sign-on (SSO) enabled subscription.

+## Scenario description

+In this tutorial, you configure and test Azure AD SSO in a test environment.

+* Radancy's Employee Referrals supports **SP and IDP** initiated SSO.

+* Radancy's Employee Referrals supports **Just In Time** user provisioning.

+## Add Radancy's Employee Referrals from the gallery

+To configure the integration of Radancy's Employee Referrals into Azure AD, you need to add Radancy's Employee Referrals from the gallery to your list of managed SaaS apps.

+1. Sign in to the Azure portal using either a work or school account, or a personal Microsoft account.

+1. On the left navigation pane, select the **Azure Active Directory** service.

+1. Navigate to **Enterprise Applications** and then select **All Applications**.

+1. To add new application, select **New application**.

+1. In the **Add from the gallery** section, type **Radancy's Employee Referrals** in the search box.

+1. Select **Radancy's Employee Referrals** from results panel and then add the app. Wait a few seconds while the app is added to your tenant.

+Alternatively, you can also use the [Enterprise App Configuration Wizard](https://portal.office.com/AdminPortal/home?Q=Docs#/azureadappintegration). In this wizard, you can add an application to your tenant, add users/groups to the app, assign roles, and walk through the SSO configuration as well. [Learn more about Microsoft 365 wizards.](/microsoft-365/admin/misc/azure-ad-setup-guides)

+## Configure and test Azure AD SSO for Radancy's Employee Referrals

+Configure and test Azure AD SSO with Radancy's Employee Referrals using a test user called **B.Simon**. For SSO to work, you need to establish a link relationship between an Azure AD user and the related user in Radancy's Employee Referrals.

+To configure and test Azure AD SSO with Radancy's Employee Referrals, perform the following steps:

+1. **[Configure Azure AD SSO](#configure-azure-ad-sso)** - to enable your users to use this feature.

+ 1. **[Create an Azure AD test user](#create-an-azure-ad-test-user)** - to test Azure AD single sign-on with B.Simon.

+ 1. **[Assign the Azure AD test user](#assign-the-azure-ad-test-user)** - to enable B.Simon to use Azure AD single sign-on.

+1. **[Configure Radancy's Employee Referrals SSO](#configure-radancys-employee-referrals-sso)** - to configure the single sign-on settings on application side.

+ 1. **[Create Radancy's Employee Referrals test user](#create-radancys-employee-referrals-test-user)** - to have a counterpart of B.Simon in Radancy's Employee Referrals that are linked to the Azure AD representation of user.

+1. **[Test SSO](#test-sso)** - to verify whether the configuration works.

+## Configure Azure AD SSO

+Follow these steps to enable Azure AD SSO in the Azure portal.

+1. In the Azure portal, on the **Radancy's Employee Referrals** application integration page, find the **Manage** section and select **single sign-on**.

+1. On the **Select a single sign-on method** page, select **SAML**.

+1. On the **Set up single sign-on with SAML** page, click the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+1. On the **Basic SAML Configuration** section, perform the following steps:

+ a. In the **Identifier** text box, type a URL using the following pattern:

+ `https://<company-domain>.auth.1brd.com/saml/sp`

+ b. In the **Reply URL** text box, type a URL using the following pattern:

+ `https://<company-domain>.auth.1brd.com/saml/callback`

+1. Perform the following step if you wish to configure the application in **SP** initiated mode:

+ In the **Sign-on URL** text box, type a URL using the following pattern:

+ `https://<company-domain>.1brd.com/login`

+ > [!NOTE]

+ > These values are not real. Update these values with the actual Identifier, Reply URL and Sign-on URL. Contact [Radancy's Employee Referrals Client support team](mailto:support@firstbird.com) to get these values. You can also refer to the patterns shown in the **Basic SAML Configuration** section in the Azure portal.

+1. Radancy's Employee Referrals application expects the SAML assertions in a specific format, which requires you to add custom attribute mappings to your SAML token attributes configuration. The following screenshot shows the list of default attributes.

+ 

+1. In addition to above, Radancy's Employee Referrals application expects few more attributes to be passed back in SAML response, which are shown below. These attributes are also pre populated but you can review them as per your requirement.

+ | Name | Source Attribute|

+ | | |

+ | first_name | user.givenname |

+ | last_name | user.surname |

+ | email | user.mail |

+1. On the **Set up single sign-on with SAML** page, in the **SAML Signing Certificate** section, find **Federation Metadata XML** and select **Download** to download the certificate and save it on your computer.

+ 

+1. On the **Set up Radancy's Employee Referrals** section, copy the appropriate URL(s) based on your requirement.

+ 

+### Create an Azure AD test user

+In this section, you'll create a test user in the Azure portal called B.Simon.

+1. From the left pane in the Azure portal, select **Azure Active Directory**, select **Users**, and then select **All users**.

+1. Select **New user** at the top of the screen.

+1. In the **User** properties, follow these steps:

+ 1. In the **Name** field, enter `B.Simon`.

+ 1. In the **User name** field, enter the username@companydomain.extension. For example, `B.Simon@contoso.com`.

+ 1. Select the **Show password** check box, and then write down the value that's displayed in the **Password** box.

+ 1. Click **Create**.

+### Assign the Azure AD test user

+In this section, you'll enable B.Simon to use Azure single sign-on by granting access to Radancy's Employee Referrals.

+1. In the Azure portal, select **Enterprise Applications**, and then select **All applications**.

+1. In the applications list, select **Radancy's Employee Referrals**.

+1. In the app's overview page, find the **Manage** section and select **Users and groups**.

+1. Select **Add user**, then select **Users and groups** in the **Add Assignment** dialog.

+1. In the **Users and groups** dialog, select **B.Simon** from the Users list, then click the **Select** button at the bottom of the screen.

+1. If you're expecting any role value in the SAML assertion, in the **Select Role** dialog, select the appropriate role for the user from the list and then click the **Select** button at the bottom of the screen.

+1. In the **Add Assignment** dialog, click the **Assign** button.

+## Configure Radancy's Employee Referrals SSO

+1. Log in to the Radancy's Employee Referrals website as an administrator.

+1. Navigate to **Account Preferences** > **Authentication** > **Single Sign-On**.

+1. In the **SAML IdP Metadata Configuration** section, perform the following steps:

+

+ 

+ 1. In the **Entity ID** textbox, paste the **Azure AD Identifier** value, which you've copied from the Azure portal.

+ 1. In the **SSO-service URL** textbox, paste the **Login URL** value, which you've copied from the Azure portal.

+ 1. In the **Signing certificate** textbox, paste the **Federation Metadata XML** file, which you've downloaded from the Azure portal.

+ 1. **Save configuration** and verify the setup.

+ > [!NOTE]

+ > You need to have the SSO option included in your contract.

+### Create Radancy's Employee Referrals test user

+In this section, a user called B.Simon is created in Radancy's Employee Referrals. Radancy's Employee Referrals supports just-in-time user provisioning, which is enabled by default. There is no action item for you in this section. If a user doesn't already exist in Radancy's Employee Referrals, a new one is created after authentication.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+#### SP initiated:

+* Click on **Test this application** in Azure portal. This will redirect to Radancy's Employee Referrals Sign-on URL where you can initiate the login flow.

+* Go to Radancy's Employee Referrals Sign-on URL directly and initiate the login flow from there.

+#### IDP initiated:

+* Click on **Test this application** in Azure portal and you should be automatically signed in to the Radancy's Employee Referrals for which you set up the SSO.

+You can also use Microsoft My Apps to test the application in any mode. When you click the Radancy's Employee Referrals tile in the My Apps, if configured in SP mode you would be redirected to the application sign-on page for initiating the login flow and if configured in IDP mode, you should be automatically signed in to the Radancy's Employee Referrals for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Next steps

+Once you configure Radancy's Employee Referrals you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+[Microsoft Graph API](/graph/overview) REST calls are the most common way to programmatically access Azure AD. This API-based access requires an Azure AD identity with the necessary authorizations and scope. With the Graph API, you can integrate Microsoft's and other tools. Follow the principles outlined in this article when you're performing the integration.

+In today's world of interconnected infrastructures, compliance with governmental and industry frameworks and standards is often mandatory. Microsoft engages with governments, regulators, and standards bodies to understand and meet compliance requirements for Azure. There are [90 Azure compliance certifications](../../compliance/index.yml), which include many for various countries/regions. Azure has 35 compliance offerings for key industries including,

+* If you use Azure Files, you can mount the file share as a volume into the new cluster. See [Mount Static Azure Files as a Volume](./azure-csi-files-storage-provision.md#mount-file-share-as-a-persistent-volume).

+* If you use Azure Managed Disks, you can only mount the disk if unattached to any VM. See [Mount Static Azure Disk as a Volume](./azure-csi-disk-storage-provision.md#mount-disk-as-a-volume).

+1. Update pod specifications to [use existing volumes](./azure-disk-csi.md) rather than PersistentVolumeClaims (static provisioning).

+When you mount Azure Blob storage as a file system into a container or pod, it enables you to use blob storage with a number of applications that work massive amounts of unstructured data. For example:

+You're prompted to confirm there isn't an open-source Blob CSI driver installed. After you confirm, it may take several minutes to complete this action. Once it's complete, you should see in the output the status of enabling the driver on your cluster. The following example resembles the section indicating the results of the previous command:

+- To learn how to set up a static or dynamic persistent volume, see [Create and use a volume with Azure Blob storage][azure-csi-blob-storage-provision].

+[azure-csi-blob-storage-provision]: azure-csi-blob-storage-provision.md

+ Title: Create a persistent volume with Azure Blob storage in Azure Kubernetes Service (AKS)

+description: Learn how to create a static or dynamic persistent volume with Azure Blob storage for use with multiple concurrent pods in Azure Kubernetes Service (AKS)

+# Create and use a volume with Azure Blob storage in Azure Kubernetes Service (AKS)

+Container-based applications often need to access and persist data in an external data volume. If multiple pods need concurrent access to the same storage volume, you can use Azure Blob storage to connect using [blobfuse][blobfuse-overview] or [Network File System][nfs-overview] (NFS).

+This article shows you how to:

+* Work with a dynamic persistent volume (PV) by installing the Container Storage Interface (CSI) driver and dynamically creating an Azure Blob storage container to attach to a pod.

+* Work with a static PV by creating an Azure Blob storage container, or use an existing one and attach it to a pod.

+For more information on Kubernetes volumes, see [Storage options for applications in AKS][concepts-storage].

+## Before you begin

+- If you don't have a storage account that supports the NFS v3 protocol, review [NFS v3 support with Azure Blob storage][azure-blob-storage-nfs-support].

+- [Enable the Blob storage CSI driver][enable-blob-csi-driver] on your AKS cluster.

+## Dynamically provision a volume

+This section provides guidance for cluster administrators who want to provision one or more persistent volumes that include details of Blob storage for use by a workload. A persistent volume claim (PVC) uses the storage class object to dynamically provision an Azure Blob storage container.

+### Dynamic provisioning parameters

+|Name | Description | Example | Mandatory | Default value|

+| | | | | |

+|skuName | Specify an Azure storage account type (alias: `storageAccountType`). | `Standard_LRS`, `Premium_LRS`, `Standard_GRS`, `Standard_RAGRS` | No | `Standard_LRS`|

+|location | Specify an Azure location. | `eastus` | No | If empty, driver will use the same location name as current cluster.|

+|resourceGroup | Specify an Azure resource group name. | myResourceGroup | No | If empty, driver will use the same resource group name as current cluster.|

+|storageAccount | Specify an Azure storage account name.| storageAccountName | - No for blobfuse mount </br> - Yes for NFSv3 mount. | - For blobfuse mount: if empty, driver finds a suitable storage account that matches `skuName` in the same resource group. If a storage account name is provided, storage account must exist. </br> - For NFSv3 mount, storage account name must be provided.|

+|protocol | Specify blobfuse mount or NFSv3 mount. | `fuse`, `nfs` | No | `fuse`|

+|containerName | Specify the existing container (directory) name. | container | No | If empty, driver creates a new container name, starting with `pvc-fuse` for blobfuse or `pvc-nfs` for NFS v3. |

+|containerNamePrefix | Specify Azure storage directory prefix created by driver. | my |Can only contain lowercase letters, numbers, hyphens, and length should be fewer than 21 characters. | No |

+|server | Specify Azure storage account domain name. | Existing storage account DNS domain name, for example `<storage-account>.privatelink.blob.core.windows.net`. | No | If empty, driver uses default `<storage-account>.blob.core.windows.net` or other sovereign cloud storage account DNS domain name.|

+|allowBlobPublicAccess | Allow or disallow public access to all blobs or containers for storage account created by driver. | `true`,`false` | No | `false`|

+|storageEndpointSuffix | Specify Azure storage endpoint suffix. | `core.windows.net` | No | If empty, driver will use default storage endpoint suffix according to cloud environment.|

+|tags | [Tags][az-tags] would be created in new storage account. | Tag format: 'foo=aaa,bar=bbb' | No | ""|

+|matchTags | Match tags when driver tries to find a suitable storage account. | `true`,`false` | No | `false`|

+| | **Following parameters are only for blobfuse** | | | |

+|subscriptionID | Specify Azure subscription ID where blob storage directory will be created. | Azure subscription ID | No | If not empty, `resourceGroup` must be provided.|

+|storeAccountKey | Specify store account key to Kubernetes secret. <br><br> Note: <br> `false` means driver uses kubelet identity to get account key. | `true`,`false` | No | `true`|

+|secretName | Specify secret name to store account key. | | No |

+|secretNamespace | Specify the namespace of secret to store account key. | `default`,`kube-system`, etc. | No | pvc namespace |

+|isHnsEnabled | Enable `Hierarchical namespace` for Azure Data Lake storage account. | `true`,`false` | No | `false`|

+| | **Following parameters are only for NFS protocol** | | | |

+|mountPermissions | Specify mounted folder permissions. |The default is `0777`. If set to `0`, driver won't perform `chmod` after mount. | `0777` | No |

+### Create a persistent volume claim using built-in storage class

+A persistent volume claim (PVC) uses the storage class object to dynamically provision an Azure Blob storage container. The following YAML can be used to create a persistent volume claim 5 GB in size with *ReadWriteMany* access, using the built-in storage class. For more information on access modes, see the [Kubernetes persistent volume][kubernetes-volumes] documentation.

+1. Create a file named `blob-nfs-pvc.yaml` and copy in the following YAML.

+ ```yml

+ apiVersion: v1

+ kind: PersistentVolumeClaim

+ metadata:

+ name: azure-blob-storage

+ annotations:

+ volume.beta.kubernetes.io/storage-class: azureblob-nfs-premium

+ spec:

+ accessModes:

+ - ReadWriteMany

+ storageClassName: my-blobstorage

+ resources:

+ requests:

+ storage: 5Gi

+ ```

+2. Create the persistent volume claim with the [kubectl create][kubectl-create] command:

+ ```bash

+ kubectl create -f blob-nfs-pvc.yaml

+ ```

+Once completed, the Blob storage container will be created. You can use the [kubectl get][kubectl-get] command to view the status of the PVC:

+```bash

+kubectl get pvc azure-blob-storage

+```

+The output of the command resembles the following example:

+```bash

+NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

+azure-blob-storage Bound pvc-b88e36c5-c518-4d38-a5ee-337a7dda0a68 5Gi RWX azureblob-nfs-premium 92m

+```

+#### Use the persistent volume claim

+The following YAML creates a pod that uses the persistent volume claim **azure-blob-storage** to mount the Azure Blob storage at the `/mnt/blob' path.

+1. Create a file named `blob-nfs-pv`, and copy in the following YAML. Make sure that the **claimName** matches the PVC created in the previous step.

+ ```yml

+ kind: Pod

+ apiVersion: v1

+ metadata:

+ name: mypod

+ spec:

+ containers:

+ - name: mypod

+ image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine

+ resources:

+ requests:

+ cpu: 100m

+ memory: 128Mi

+ limits:

+ cpu: 250m

+ memory: 256Mi

+ volumeMounts:

+ - mountPath: "/mnt/blob"

+ name: volume

+ volumes:

+ - name: volume

+ persistentVolumeClaim:

+ claimName: azure-blob-storage

+ ```

+2. Create the pod with the [kubectl apply][kubectl-apply] command:

+ ```bash

+ kubectl apply -f blob-nfs-pv.yaml

+ ```

+3. After the pod is in the running state, run the following command to create a new file called `test.txt`.

+ ```bash

+ kubectl exec mypod -- touch /mnt/blob/test.txt

+ ```

+4. To validate the disk is correctly mounted, run the following command, and verify you see the `test.txt` file in the output:

+ ```bash

+ kubectl exec mypod -- ls /mnt/blob

+ ```

+ The output of the command resembles the following example:

+ ```bash

+ test.txt

+ ```

+### Create a custom storage class

+The default storage classes suit the most common scenarios, but not all. For some cases, you might want to have your own storage class customized with your own parameters. To demonstrate, two examples are shown. One based on using the NFS protocol, and the other using blobfuse.

+#### Storage class using NFS protocol

+In this example, the following manifest configures mounting a Blob storage container using the NFS protocol. Use it to add the *tags* parameter.

+1. Create a file named `blob-nfs-sc.yaml`, and paste the following example manifest:

+ ```yml

+ apiVersion: storage.k8s.io/v1

+ kind: StorageClass

+ metadata:

+ name: azureblob-nfs-premium

+ provisioner: blob.csi.azure.com

+ parameters:

+ protocol: nfs

+ tags: environment=Development

+ volumeBindingMode: Immediate

+ ```

+2. Create the storage class with the [kubectl apply][kubectl-apply] command:

+ ```bash

+ kubectl apply -f blob-nfs-sc.yaml

+ ```

+ The output of the command resembles the following example:

+ ```bash

+ storageclass.storage.k8s.io/blob-nfs-premium created

+ ```

+#### Storage class using blobfuse

+In this example, the following manifest configures using blobfuse and mounts a Blob storage container. Use it to update the *skuName* parameter.

+1. Create a file named `blobfuse-sc.yaml`, and paste the following example manifest:

+ ```yml

+ apiVersion: storage.k8s.io/v1

+ kind: StorageClass

+ metadata:

+ name: azureblob-fuse-premium

+ provisioner: blob.csi.azure.com

+ parameters:

+ skuName: Standard_GRS # available values: Standard_LRS, Premium_LRS, Standard_GRS, Standard_RAGRS

+ reclaimPolicy: Delete

+ volumeBindingMode: Immediate

+ allowVolumeExpansion: true

+ mountOptions:

+ - -o allow_other

+ - --file-cache-timeout-in-seconds=120

+ - --use-attr-cache=true

+ - --cancel-list-on-mount-seconds=10 # prevent billing charges on mounting

+ - -o attr_timeout=120

+ - -o entry_timeout=120

+ - -o negative_timeout=120

+ - --log-level=LOG_WARNING # LOG_WARNING, LOG_INFO, LOG_DEBUG

+ - --cache-size-mb=1000 # Default will be 80% of available memory, eviction will happen beyond that.

+ ```

+2. Create the storage class with the [kubectl apply][kubectl-apply] command:

+ ```bash

+ kubectl apply -f blobfuse-sc.yaml

+ ```

+ The output of the command resembles the following example:

+ ```bash

+ storageclass.storage.k8s.io/blob-fuse-premium created

+ ```

+## Statically provision a volume

+This section provides guidance for cluster administrators who want to create one or more persistent volumes that include details of Blob storage for use by a workload.

+### Static provisioning parameters

+|Name | Description | Example | Mandatory | Default value|

+| | | | | |

+|volumeHandle | Specify a value the driver can use to uniquely identify the storage blob container in the cluster. | A recommended way to produce a unique value is to combine the globally unique storage account name and container name: `{account-name}_{container-name}`.<br> Note: The `#` character is reserved for internal use and can't be used in a volume handle. | Yes ||

+|volumeAttributes.resourceGroup | Specify Azure resource group name. | myResourceGroup | No | If empty, driver uses the same resource group name as current cluster.|

+|volumeAttributes.storageAccount | Specify an existing Azure storage account name. | storageAccountName | Yes ||

+|volumeAttributes.containerName | Specify existing container name. | container | Yes ||

+|volumeAttributes.protocol | Specify blobfuse mount or NFS v3 mount. | `fuse`, `nfs` | No | `fuse`|

+| | **Following parameters are only for blobfuse** | | | |

+|volumeAttributes.secretName | Secret name that stores storage account name and key (only applies for SMB).| | No ||

+|volumeAttributes.secretNamespace | Specify namespace of secret to store account key. | `default` | No | Pvc namespace|

+|nodeStageSecretRef.name | Specify secret name that stores one of the following:<br> `azurestorageaccountkey`<br>`azurestorageaccountsastoken`<br>`msisecret`<br>`azurestoragespnclientsecret`. | |Existing Kubernetes secret name | No |

+|nodeStageSecretRef.namespace | Specify the namespace of secret. | Kubernetes namespace | Yes ||

+| | **Following parameters are only for NFS protocol** | | | |

+|volumeAttributes.mountPermissions | Specify mounted folder permissions. | `0777` | No ||

+| | **Following parameters are only for NFS VNet setting** | | | |

+|vnetResourceGroup | Specify VNet resource group hosting virtual network. | myResourceGroup | No | If empty, driver uses the `vnetResourceGroup` value specified in the Azure cloud config file.|

+|vnetName | Specify the virtual network name. | aksVNet | No | If empty, driver uses the `vnetName` value specified in the Azure cloud config file.|

+|subnetName | Specify the existing subnet name of the agent node. | aksSubnet | No | If empty, driver uses the `subnetName` value in Azure cloud config file. |

+| | **Following parameters are only for feature: blobfuse<br> [Managed Identity and Service Principal Name authentication](https://github.com/Azure/azure-storage-fuse#environment-variables)** | | | |

+|volumeAttributes.AzureStorageAuthType | Specify the authentication type. | `Key`, `SAS`, `MSI`, `SPN` | No | `Key`|

+|volumeAttributes.AzureStorageIdentityClientID | Specify the Identity Client ID. | | No ||

+|volumeAttributes.AzureStorageIdentityObjectID | Specify the Identity Object ID. | | No ||

+|volumeAttributes.AzureStorageIdentityResourceID | Specify the Identity Resource ID. | | No ||

+|volumeAttributes.MSIEndpoint | Specify the MSI endpoint. | | No ||

+|volumeAttributes.AzureStorageSPNClientID | Specify the Azure Service Principal Name (SPN) Client ID. | | No ||

+|volumeAttributes.AzureStorageSPNTenantID | Specify the Azure SPN Tenant ID. | | No ||

+|volumeAttributes.AzureStorageAADEndpoint | Specify the Azure Active Directory (Azure AD) endpoint. | | No ||

+| | **Following parameters are only for feature: blobfuse read account key or SAS token from key vault** | | | |

+|volumeAttributes.keyVaultURL | Specify Azure Key Vault DNS name. | {vault-name}.vault.azure.net | No ||

+|volumeAttributes.keyVaultSecretName | Specify Azure Key Vault secret name. | Existing Azure Key Vault secret name. | No ||

+|volumeAttributes.keyVaultSecretVersion | Azure Key Vault secret version. | Existing version | No |If empty, driver uses current version.|

+### Create a Blob storage container

+When you create an Azure Blob storage resource for use with AKS, you can create the resource in the node resource group. This approach allows the AKS cluster to access and manage the blob storage resource. If instead you create the blob storage resource in a separate resource group, you must grant the Azure Kubernetes Service managed identity for your cluster the [Contributor][rbac-contributor-role] role to the blob storage resource group.

+For this article, create the container in the node resource group. First, get the resource group name with the [az aks show][az-aks-show] command and add the `--query nodeResourceGroup` query parameter. The following example gets the node resource group for the AKS cluster named **myAKSCluster** in the resource group named **myResourceGroup**:

+```azurecli

+az aks show --resource-group myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv

+```

+The output of the command resembles the following example:

+```azurecli

+MC_myResourceGroup_myAKSCluster_eastus

+```

+Next, create a container for storing blobs following the steps in the [Manage blob storage][manage-blob-storage] to authorize access and then create the container.

+### Mount volume

+In this section, you mount the persistent volume using the NFS protocol or Blobfuse.

+#### [Mount volume using NFS protocol](#tab/mount-nfs)

+Mounting Blob storage using the NFS v3 protocol doesn't authenticate using an account key. Your AKS cluster needs to reside in the same or peered virtual network as the agent node. The only way to secure the data in your storage account is by using a virtual network and other network security settings. For more information on how to set up NFS access to your storage account, see [Mount Blob Storage by using the Network File System (NFS) 3.0 protocol](../storage/blobs/network-file-system-protocol-support-how-to.md).

+The following example demonstrates how to mount a Blob storage container as a persistent volume using the NFS protocol.

+1. Create a file named `pv-blob-nfs.yaml` and copy in the following YAML. Under `storageClass`, update `resourceGroup`, `storageAccount`, and `containerName`.

+ > [!NOTE]

+ > `volumeHandle` value should be a unique volumeID for every identical storage blob container in the cluster.

+ > The character `#` is reserved for internal use and cannot be used.

+ ```yml

+ apiVersion: v1

+ kind: PersistentVolume

+ metadata:

+ name: pv-blob

+ spec:

+ capacity:

+ storage: 1Pi

+ accessModes:

+ - ReadWriteMany

+ persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion

+ storageClassName: azureblob-nfs-premium

+ csi:

+ driver: blob.csi.azure.com

+ readOnly: false

+ # make sure volumeid is unique for every identical storage blob container in the cluster

+ # character `#` is reserved for internal use and cannot be used in volumehandle

+ volumeHandle: unique-volumeid

+ volumeAttributes:

+ resourceGroup: resourceGroupName

+ storageAccount: storageAccountName

+ containerName: containerName

+ protocol: nfs

+ ```

+ > [!NOTE]

+ > While the [Kubernetes API](https://github.com/kubernetes/kubernetes/blob/release-1.26/pkg/apis/core/types.go#L303-L306) **capacity** attribute is mandatory, this value isn't used by the Azure Blob storage CSI driver because you can flexibly write data until you reach your storage account's capacity limit. The value of the `capacity` attribute is used only for size matching between *PersistentVolumes* and *PersistenVolumeClaims*. We recommend using a fictitious high value. The pod sees a mounted volume with a fictitious size of 5 Petabytes.

+2. Run the following command to create the persistent volume using the [kubectl create][kubectl-create] command referencing the YAML file created earlier:

+ ```bash

+ kubectl create -f pv-blob-nfs.yaml

+ ```

+3. Create a `pvc-blob-nfs.yaml` file with a *PersistentVolumeClaim*. For example:

+ ```yml

+ kind: PersistentVolumeClaim

+ apiVersion: v1

+ metadata:

+ name: pvc-blob

+ spec:

+ accessModes:

+ - ReadWriteMany

+ resources:

+ requests:

+ storage: 10Gi

+ volumeName: pv-blob

+ storageClassName: azureblob-nfs-premium

+ ```

+4. Run the following command to create the persistent volume claim using the [kubectl create][kubectl-create] command referencing the YAML file created earlier:

+ ```bash

+ kubectl create -f pvc-blob-nfs.yaml

+ ```

+#### [Mount volume using Blobfuse](#tab/mount-blobfuse)

+Kubernetes needs credentials to access the Blob storage container created earlier, which is either an Azure access key or SAS tokens. These credentials are stored in a Kubernetes secret, which is referenced when you create a Kubernetes pod.

+1. Use the `kubectl create secret command` to create the secret. You can authenticate using a [Kubernetes secret][kubernetes-secret] or [shared access signature][sas-tokens] (SAS) tokens.

+ # [Secret](#tab/secret)

+ The following example creates a [Secret object][kubernetes-secret] named *azure-secret* and populates the *azurestorageaccountname* and *azurestorageaccountkey*. You need to provide the account name and key from an existing Azure storage account.

+ ```bash

+ kubectl create secret generic azure-secret --from-literal azurestorageaccountname=NAME --from-literal azurestorageaccountkey="KEY" --type=Opaque

+ ```

+ # [SAS tokens](#tab/sas-tokens)

+ The following example creates a [Secret object][kubernets-secret] named *azure-sas-token* and populates the *azurestorageaccountname* and *azurestorageaccountsastoken*. You need to provide the account name and shared access signature from an existing Azure storage account.

+ ```bash

+ kubectl create secret generic azure-sas-token --from-literal azurestorageaccountname=NAME --from-literal azurestorageaccountsastoken

+ ="sastoken" --type=Opaque

+ ```

+

+2. Create a `pv-blobfuse.yaml` file. Under `volumeAttributes`, update `containerName`. Under `nodeStateSecretRef`, update `name` with the name of the Secret object created earlier. For example:

+ > [!NOTE]

+ > `volumeHandle` value should be a unique volumeID for every identical storage blob container in the cluster.

+ > The character `#` is reserved for internal use and cannot be used.

+ ```yml

+ apiVersion: v1

+ kind: PersistentVolume

+ metadata:

+ name: pv-blob

+ spec:

+ capacity:

+ storage: 10Gi

+ accessModes:

+ - ReadWriteMany

+ persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion

+ storageClassName: azureblob-fuse-premium

+ mountOptions:

+ - -o allow_other

+ - --file-cache-timeout-in-seconds=120

+ csi:

+ driver: blob.csi.azure.com

+ readOnly: false

+ # volumeid has to be unique for every identical storage blob container in the cluster

+ # character `#` is reserved for internal use and cannot be used in volumehandle

+ volumeHandle: unique-volumeid

+ volumeAttributes:

+ containerName: containerName

+ nodeStageSecretRef:

+ name: azure-secret

+ namespace: default

+ ```

+3. Run the following command to create the persistent volume using the [kubectl create][kubectl-create] command referencing the YAML file created earlier:

+ ```bash

+ kubectl create -f pv-blobfuse.yaml

+ ```

+4. Create a `pvc-blobfuse.yaml` file with a *PersistentVolume*. For example:

+ ```yml

+ apiVersion: v1

+ kind: PersistentVolumeClaim

+ metadata:

+ name: pvc-blob

+ spec:

+ accessModes:

+ - ReadWriteMany

+ resources:

+ requests:

+ storage: 10Gi

+ volumeName: pv-blob

+ storageClassName: azureblob-fuse-premium

+ ```

+5. Run the following command to create the persistent volume claim using the [kubectl create][kubectl-create] command referencing the YAML file created earlier:

+ ```bash

+ kubectl create -f pvc-blobfuse.yaml

+ ```

+### Use the persistent volume

+The following YAML creates a pod that uses the persistent volume or persistent volume claim named **pvc-blob** created earlier, to mount the Azure Blob storage at the `/mnt/blob` path.

+1. Create a file named `nginx-pod-blob.yaml`, and copy in the following YAML. Make sure that the **claimName** matches the PVC created in the previous step when creating a persistent volume for NFS or Blobfuse.

+ ```yml

+ kind: Pod

+ apiVersion: v1

+ metadata:

+ name: nginx-blob

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine

+ name: nginx-blob

+ volumeMounts:

+ - name: blob01

+ mountPath: "/mnt/blob"

+ volumes:

+ - name: blob01

+ persistentVolumeClaim:

+ claimName: pvc-blob

+ ```

+2. Run the following command to create the pod and mount the PVC using the [kubectl create][kubectl-create] command referencing the YAML file created earlier:

+ ```bash

+ kubectl create -f nginx-pod-blob.yaml

+ ```

+3. Run the following command to create an interactive shell session with the pod to verify the Blob storage mounted:

+ ```bash

+ kubectl exec -it nginx-blob -- df -h

+ ```

+ The output from the command resembles the following example:

+ ```bash

+ Filesystem Size Used Avail Use% Mounted on

+ ...

+ blobfuse 14G 41M 13G 1% /mnt/blob

+ ...

+ ```

+## Next steps

+- To learn how to use CSI driver for Azure Blob storage, see [Use Azure Blob storage with CSI driver][azure-blob-storage-csi].

+- For associated best practices, see [Best practices for storage and backups in AKS][operator-best-practices-storage].

+<!-- LINKS - external -->

+[kubernetes-volumes]: https://kubernetes.io/docs/concepts/storage/volumes/

+[blobfuse-overview]: https://github.com/Azure/azure-storage-fuse

+[nfs-overview]: https://en.wikipedia.org/wiki/Network_File_System

+[kubectl-apply]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#apply

+[kubectl-get]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get

+[kubernetes-secret]: https://kubernetes.io/docs/concepts/configuration/secret/

+[kubectl-create]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#create

+<!-- LINKS - internal -->

+[operator-best-practices-storage]: operator-best-practices-storage.md

+[concepts-storage]: concepts-storage.md

+[azure-blob-storage-csi]: azure-blob-csi.md

+[azure-blob-storage-nfs-support]: ../storage/blobs/network-file-system-protocol-support.md

+[enable-blob-csi-driver]: azure-blob-csi.md#before-you-begin

+[az-tags]: ../azure-resource-manager/management/tag-resources.md

+[sas-tokens]: ../storage/common/storage-sas-overview.md

+ Title: Create a persistent volume with Azure Disks in Azure Kubernetes Service (AKS)

+description: Learn how to create a static or dynamic persistent volume with Azure Disks for use with multiple concurrent pods in Azure Kubernetes Service (AKS)

+# Create and use a volume with Azure Disks in Azure Kubernetes Service (AKS)

+A persistent volume represents a piece of storage that has been provisioned for use with Kubernetes pods. A persistent volume can be used by one or many pods, and can be dynamically or statically provisioned. This article shows you how to dynamically create persistent volumes with Azure Disks for use by a single pod in an Azure Kubernetes Service (AKS) cluster.

+> [!NOTE]

+> An Azure disk can only be mounted with *Access mode* type *ReadWriteOnce*, which makes it available to one node in AKS. If you need to share a persistent volume across multiple nodes, use [Azure Files][azure-files-pvc].

+This article shows you how to:

+* Work with a dynamic persistent volume (PV) by installing the Container Storage Interface (CSI) driver and dynamically creating one or more Azure managed disk to attach to a pod.

+* Work with a static PV by creating one or more Azure managed disk, or use an existing one and attach it to a pod.

+For more information on Kubernetes volumes, see [Storage options for applications in AKS][concepts-storage].

+## Before you begin

+- An Azure [storage account][azure-storage-account].

+- The Azure CLI version 2.0.59 or later installed and configured. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].

+The Azure Disks CSI driver has a limit of 32 volumes per node. The volume count changes based on the size of the node/node pool. Run the [kubectl get][kubectl-get] command to determine the number of volumes that can be allocated per node:

+```console

+kubectl get CSINode <nodename> -o yaml

+```

+## Dynamically provision a volume

+This section provides guidance for cluster administrators who want to provision one or more persistent volumes that include details of Azure Disk storage for use by a workload. A persistent volume claim (PVC) uses the storage class object to dynamically provision an Azure Disk storage container.

+### Dynamic provisioning parameters

+|Name | Meaning | Available Value | Mandatory | Default value

+| | | | |

+|skuName | Azure Disks storage account type (alias: `storageAccountType`)| `Standard_LRS`, `Premium_LRS`, `StandardSSD_LRS`, `PremiumV2_LRS`, `UltraSSD_LRS`, `Premium_ZRS`, `StandardSSD_ZRS` | No | `StandardSSD_LRS`|

+|fsType | File System Type | `ext4`, `ext3`, `ext2`, `xfs`, `btrfs` for Linux, `ntfs` for Windows | No | `ext4` for Linux, `ntfs` for Windows|

+|cachingMode | [Azure Data Disk Host Cache Setting][disk-host-cache-setting] | `None`, `ReadOnly`, `ReadWrite` | No | `ReadOnly`|

+|location | Specify Azure region where Azure Disks will be created | `eastus`, `westus`, etc. | No | If empty, driver will use the same location name as current AKS cluster|

+|resourceGroup | Specify the resource group where the Azure Disks will be created | Existing resource group name | No | If empty, driver will use the same resource group name as current AKS cluster|

+|DiskIOPSReadWrite | [UltraSSD disk][ultra-ssd-disks] IOPS Capability (minimum: 2 IOPS/GiB ) | 100~160000 | No | `500`|

+|DiskMBpsReadWrite | [UltraSSD disk][ultra-ssd-disks] Throughput Capability(minimum: 0.032/GiB) | 1~2000 | No | `100`|

+|LogicalSectorSize | Logical sector size in bytes for ultra disk. Supported values are 512 ad 4096. 4096 is the default. | `512`, `4096` | No | `4096`|

+|tags | Azure Disk [tags][azure-tags] | Tag format: `key1=val1,key2=val2` | No | ""|

+|diskEncryptionSetID | ResourceId of the disk encryption set to use for [enabling encryption at rest][disk-encryption] | format: `/subscriptions/{subs-id}/resourceGroups/{rg-name}/providers/Microsoft.Compute/diskEncryptionSets/{diskEncryptionSet-name}` | No | ""|

+|diskEncryptionType | Encryption type of the disk encryption set. | `EncryptionAtRestWithCustomerKey`(by default), `EncryptionAtRestWithPlatformAndCustomerKeys` | No | ""|

+|writeAcceleratorEnabled | [Write Accelerator on Azure Disks][azure-disk-write-accelerator] | `true`, `false` | No | ""|

+|networkAccessPolicy | NetworkAccessPolicy property to prevent generation of the SAS URI for a disk or a snapshot | `AllowAll`, `DenyAll`, `AllowPrivate` | No | `AllowAll`|

+|diskAccessID | Azure Resource ID of the DiskAccess resource to use private endpoints on disks | | No | ``|

+|enableBursting | [Enable on-demand bursting][on-demand-bursting] beyond the provisioned performance target of the disk. On-demand bursting should only be applied to Premium disk and when the disk size > 512 GB. Ultra and shared disk isn't supported. Bursting is disabled by default. | `true`, `false` | No | `false`|

+|useragent | User agent used for [customer usage attribution][customer-usage-attribution] | | No | Generated Useragent formatted `driverName/driverVersion compiler/version (OS-ARCH)`|

+|enableAsyncAttach | Allow multiple disk attach operations (in batch) on one node in parallel.<br> While this parameter can speed up disk attachment, you may encounter Azure API throttling limit when there are large number of volume attachments. | `true`, `false` | No | `false`|

+|subscriptionID | Specify Azure subscription ID where the Azure Disks is created. | Azure subscription ID | No | If not empty, `resourceGroup` must be provided.|

+| | **Following parameters are only for v2** | | | |

+| enableAsyncAttach | The v2 driver uses a different strategy to manage Azure API throttling and ignores this parameter. | | No | |

+| maxShares | The total number of shared disk mounts allowed for the disk. Setting the value to 2 or more enables attachment replicas. | Supported values depend on the disk size. See [Share an Azure managed disk][share-azure-managed-disk] for supported values. | No | 1 |

+| maxMountReplicaCount | The number of replicas attachments to maintain. | This value must be in the range `[0..(maxShares - 1)]` | No | If `accessMode` is `ReadWriteMany`, the default is `0`. Otherwise, the default is `maxShares - 1` |

+### Built-in storage classes

+A storage class is used to define how a unit of storage is dynamically created with a persistent volume. For more information on Kubernetes storage classes, see [Kubernetes Storage Classes][kubernetes-storage-classes].

+Each AKS cluster includes four pre-created storage classes, two of them configured to work with Azure Disks:

+* The *default* storage class provisions a standard SSD Azure Disk.

+ * Standard storage is backed by Standard SSDs and delivers cost-effective storage while still delivering reliable performance.

+* The *managed-csi-premium* storage class provisions a premium Azure Disk.

+ * Premium disks are backed by SSD-based high-performance, low-latency disk. Perfect for VMs running production workload. If the AKS nodes in your cluster use premium storage, select the *managed-premium* class.

+

+If you use one of the default storage classes, you can't update the volume size after the storage class is created. To be able to update the volume size after a storage class is created, add the line `allowVolumeExpansion: true` to one of the default storage classes, or you can create your own custom storage class. It's not supported to reduce the size of a PVC (to prevent data loss). You can edit an existing storage class by using the `kubectl edit sc` command.

+For example, if you want to use a disk of size 4 TiB, you must create a storage class that defines `cachingmode: None` because [disk caching isn't supported for disks 4 TiB and larger][disk-host-cache-setting].

+For more information about storage classes and creating your own storage class, see [Storage options for applications in AKS][storage-class-concepts].

+Use the [kubectl get sc][kubectl-get] command to see the pre-created storage classes. The following example shows the pre-create storage classes available within an AKS cluster:

+```bash

+kubectl get sc

+```

+The output of the command resembles the following example:

+```console

+NAME PROVISIONER AGE

+default (default) disk.csi.azure.com 1h

+managed-csi disk.csi.azure.com 1h

+```

+> [!NOTE]

+> Persistent volume claims are specified in GiB but Azure managed disks are billed by SKU for a specific size. These SKUs range from 32GiB for S4 or P4 disks to 32TiB for S80 or P80 disks (in preview). The throughput and IOPS performance of a Premium managed disk depends on the both the SKU and the instance size of the nodes in the AKS cluster. For more information, see [Pricing and performance of managed disks][managed-disk-pricing-performance].

+### Create a persistent volume claim

+A persistent volume claim (PVC) is used to automatically provision storage based on a storage class. In this case, a PVC can use one of the pre-created storage classes to create a standard or premium Azure managed disk.

+Create a file named `azure-pvc.yaml`, and copy in the following manifest. The claim requests a disk named `azure-managed-disk` that is *5 GB* in size with *ReadWriteOnce* access. The *managed-csi* storage class is specified as the storage class.

+```yaml

+apiVersion: v1

+kind: PersistentVolumeClaim

+metadata:

+ name: azure-managed-disk

+spec:

+ accessModes:

+ - ReadWriteOnce

+ storageClassName: managed-csi

+ resources:

+ requests:

+ storage: 5Gi

+```

+> [!TIP]

+> To create a disk that uses premium storage, use `storageClassName: managed-csi-premium` rather than *managed-csi*.

+Create the persistent volume claim with the [kubectl apply][kubectl-apply] command and specify your *azure-pvc.yaml* file:

+```bash

+kubectl apply -f azure-pvc.yaml

+```

+The output of the command resembles the following example:

+```console

+persistentvolumeclaim/azure-managed-disk created

+```

+### Use the persistent volume

+Once the persistent volume claim has been created and the disk successfully provisioned, a pod can be created with access to the disk. The following manifest creates a basic NGINX pod that uses the persistent volume claim named *azure-managed-disk* to mount the Azure Disk at the path `/mnt/azure`. For Windows Server containers, specify a *mountPath* using the Windows path convention, such as *'D:'*.

+Create a file named `azure-pvc-disk.yaml`, and copy in the following manifest.

+```yaml

+kind: Pod

+apiVersion: v1

+metadata:

+ name: mypod

+spec:

+ containers:

+ - name: mypod

+ image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine

+ resources:

+ requests:

+ cpu: 100m

+ memory: 128Mi

+ limits:

+ cpu: 250m

+ memory: 256Mi

+ volumeMounts:

+ - mountPath: "/mnt/azure"

+ name: volume

+ volumes:

+ - name: volume

+ persistentVolumeClaim:

+ claimName: azure-managed-disk

+```

+Create the pod with the [kubectl apply][kubectl-apply] command, as shown in the following example:

+```console

+kubectl apply -f azure-pvc-disk.yaml

+```

+The output of the command resembles the following example:

+```console

+pod/mypod created

+```

+You now have a running pod with your Azure Disk mounted in the `/mnt/azure` directory. This configuration can be seen when inspecting your pod using the [kubectl describe][kubectl-describe] command, as shown in the following condensed example:

+```bash

+kubectl describe pod mypod

+```

+The output of the command resembles the following example:

+```console

+[...]

+Volumes:

+ volume:

+ Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

+ ClaimName: azure-managed-disk

+ ReadOnly: false

+ default-token-smm2n:

+ Type: Secret (a volume populated by a Secret)

+ SecretName: default-token-smm2n

+ Optional: false

+[...]

+Events:

+ Type Reason Age From Message

+ - - - -

+ Normal Scheduled 2m default-scheduler Successfully assigned mypod to aks-nodepool1-79590246-0

+ Normal SuccessfulMountVolume 2m kubelet, aks-nodepool1-79590246-0 MountVolume.SetUp succeeded for volume "default-token-smm2n"

+ Normal SuccessfulMountVolume 1m kubelet, aks-nodepool1-79590246-0 MountVolume.SetUp succeeded for volume "pvc-faf0f176-8b8d-11e8-923b-deb28c58d242"

+[...]

+```

+### Use Azure ultra disks

+To use Azure ultra disk, see [Use ultra disks on Azure Kubernetes Service (AKS)][use-ultra-disks].

+### Back up a persistent volume

+To back up the data in your persistent volume, take a snapshot of the managed disk for the volume. You can then use this snapshot to create a restored disk and attach to pods as a means of restoring the data.

+First, get the volume name with the [kubectl get][kubectl-get] command, such as for the PVC named *azure-managed-disk*:

+```bash

+kubectl get pvc azure-managed-disk

+```

+The output of the command resembles the following example:

+```console

+NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

+azure-managed-disk Bound pvc-faf0f176-8b8d-11e8-923b-deb28c58d242 5Gi RWO managed-premium 3m

+```

+This volume name forms the underlying Azure disk name. Query for the disk ID with [az disk list][az-disk-list] and provide your PVC volume name, as shown in the following example:

+```azurecli

+az disk list --query '[].id | [?contains(@,`pvc-faf0f176-8b8d-11e8-923b-deb28c58d242`)]' -o tsv

+/subscriptions/<guid>/resourceGroups/MC_MYRESOURCEGROUP_MYAKSCLUSTER_EASTUS/providers/MicrosoftCompute/disks/kubernetes-dynamic-pvc-faf0f176-8b8d-11e8-923b-deb28c58d242

+```

+Use the disk ID to create a snapshot disk with [az snapshot create][az-snapshot-create]. The following example creates a snapshot named *pvcSnapshot* in the same resource group as the AKS cluster *MC_myResourceGroup_myAKSCluster_eastus*. You may encounter permission issues if you create snapshots and restore disks in resource groups that the AKS cluster doesn't have access to.

+```azurecli

+az snapshot create \

+ --resource-group MC_myResourceGroup_myAKSCluster_eastus \

+ --name pvcSnapshot \

+ --source /subscriptions/<guid>/resourceGroups/MC_myResourceGroup_myAKSCluster_eastus/providers/MicrosoftCompute/disks/kubernetes-dynamic-pvc-faf0f176-8b8d-11e8-923b-deb28c58d242

+```

+Depending on the amount of data on your disk, it may take a few minutes to create the snapshot.

+### Restore and use a snapshot

+To restore the disk and use it with a Kubernetes pod, use the snapshot as a source when you create a disk with [az disk create][az-disk-create]. This operation preserves the original resource if you then need to access the original data snapshot. The following example creates a disk named *pvcRestored* from the snapshot named *pvcSnapshot*:

+```azurecli

+az disk create --resource-group MC_myResourceGroup_myAKSCluster_eastus --name pvcRestored --source pvcSnapshot

+```

+To use the restored disk with a pod, specify the ID of the disk in the manifest. Get the disk ID with the [az disk show][az-disk-show] command. The following example gets the disk ID for *pvcRestored* created in the previous step:

+```azurecli

+az disk show --resource-group MC_myResourceGroup_myAKSCluster_eastus --name pvcRestored --query id -o tsv

+```

+Create a pod manifest named `azure-restored.yaml` and specify the disk URI obtained in the previous step. The following example creates a basic NGINX web server, with the restored disk mounted as a volume at */mnt/azure*:

+```yaml

+kind: Pod

+apiVersion: v1

+metadata:

+ name: mypodrestored

+spec:

+ containers:

+ - name: mypodrestored

+ image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine

+ resources:

+ requests:

+ cpu: 100m

+ memory: 128Mi

+ limits:

+ cpu: 250m

+ memory: 256Mi

+ volumeMounts:

+ - mountPath: "/mnt/azure"

+ name: volume

+ volumes:

+ - name: volume

+ azureDisk:

+ kind: Managed

+ diskName: pvcRestored

+ diskURI: /subscriptions/<guid>/resourceGroups/MC_myResourceGroupAKS_myAKSCluster_eastus/providers/Microsoft.Compute/disks/pvcRestored

+```

+Create the pod with the [kubectl apply][kubectl-apply] command, as shown in the following example:

+```bash

+kubectl apply -f azure-restored.yaml

+```

+The output of the command resembles the following example:

+```console

+pod/mypodrestored created

+```

+You can use `kubectl describe pod mypodrestored` to view details of the pod, such as the following condensed example that shows the volume information:

+```bash

+kubectl describe pod mypodrestored

+```

+The output of the command resembles the following example:

+```console

+[...]

+Volumes:

+ volume:

+ Type: AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)

+ DiskName: pvcRestored

+ DiskURI: /subscriptions/19da35d3-9a1a-4f3b-9b9c-3c56ef409565/resourceGroups/MC_myResourceGroupAKS_myAKSCluster_eastus/providers/Microsoft.Compute/disks/pvcRestored

+ Kind: Managed

+ FSType: ext4

+ CachingMode: ReadWrite

+ ReadOnly: false

+[...]

+```

+### Using Azure tags

+For more information on using Azure tags, see [Use Azure tags in Azure Kubernetes Service (AKS)][use-tags].

+## Statically provision a volume

+This section provides guidance for cluster administrators who want to create one or more persistent volumes that include details of Azure Disks storage for use by a workload.

+### Static provisioning parameters

+|Name | Meaning | Available Value | Mandatory | Default value|

+| | | | | |

+|volumeHandle| Azure disk URI | `/subscriptions/{sub-id}/resourcegroups/{group-name}/providers/microsoft.compute/disks/{disk-id}` | Yes | N/A|

+|volumeAttributes.fsType | File system type | `ext4`, `ext3`, `ext2`, `xfs`, `btrfs` for Linux, `ntfs` for Windows | No | `ext4` for Linux, `ntfs` for Windows |

+|volumeAttributes.partition | Partition number of the existing disk (only supported on Linux) | `1`, `2`, `3` | No | Empty (no partition) </br>- Make sure partition format is like `-part1` |

+|volumeAttributes.cachingMode | [Disk host cache setting][disk-host-cache-setting] | `None`, `ReadOnly`, `ReadWrite` | No | `ReadOnly`|

+### Create an Azure disk

+When you create an Azure disk for use with AKS, you can create the disk resource in the **node** resource group. This approach allows the AKS cluster to access and manage the disk resource. If instead you created the disk in a separate resource group, you must grant the Azure Kubernetes Service (AKS) managed identity for your cluster the `Contributor` role to the disk's resource group. In this exercise, you're going to create the disk in the same resource group as your cluster.

+1. Identify the resource group name using the [az aks show][az-aks-show] command and add the `--query nodeResourceGroup` parameter. The following example gets the node resource group for the AKS cluster name *myAKSCluster* in the resource group name *myResourceGroup*:

+ ```azurecli-interactive

+ az aks show --resource-group myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv

+

+ MC_myResourceGroup_myAKSCluster_eastus

+ ```

+2. Create a disk using the [az disk create][az-disk-create] command. Specify the node resource group name obtained in the previous command, and then a name for the disk resource, such as *myAKSDisk*. The following example creates a *20*GiB disk, and outputs the ID of the disk after it's created. If you need to create a disk for use with Windows Server containers, add the `--os-type windows` parameter to correctly format the disk.

+ ```azurecli-interactive

+ az disk create \

+ --resource-group MC_myResourceGroup_myAKSCluster_eastus \

+ --name myAKSDisk \

+ --size-gb 20 \

+ --query id --output tsv

+ ```

+ > [!NOTE]

+ > Azure Disks are billed by SKU for a specific size. These SKUs range from 32GiB for S4 or P4 disks to 32TiB for S80 or P80 disks (in preview). The throughput and IOPS performance of a Premium managed disk depends on both the SKU and the instance size of the nodes in the AKS cluster. See [Pricing and Performance of Managed Disks][managed-disk-pricing-performance].

+ The disk resource ID is displayed once the command has successfully completed, as shown in the following example output. This disk ID is used to mount the disk in the next section.

+ ```console

+ /subscriptions/<subscriptionID>/resourceGroups/MC_myAKSCluster_myAKSCluster_eastus/providers/Microsoft.Compute/disks/myAKSDisk

+ ```

+### Mount disk as a volume

+1. Create a *pv-azuredisk.yaml* file with a *PersistentVolume*. Update `volumeHandle` with disk resource ID from the previous step. For example:

+ ```yaml

+ apiVersion: v1

+ kind: PersistentVolume

+ metadata:

+ name: pv-azuredisk

+ spec:

+ capacity:

+ storage: 20Gi

+ accessModes:

+ - ReadWriteOnce

+ persistentVolumeReclaimPolicy: Retain

+ storageClassName: managed-csi

+ csi:

+ driver: disk.csi.azure.com

+ readOnly: false

+ volumeHandle: /subscriptions/<subscriptionID>/resourceGroups/MC_myAKSCluster_myAKSCluster_eastus/providers/Microsoft.Compute/disks/myAKSDisk

+ volumeAttributes:

+ fsType: ext4

+ ```

+2. Create a *pvc-azuredisk.yaml* file with a *PersistentVolumeClaim* that uses the *PersistentVolume*. For example:

+ ```yaml

+ apiVersion: v1

+ kind: PersistentVolumeClaim

+ metadata:

+ name: pvc-azuredisk

+ spec:

+ accessModes:

+ - ReadWriteOnce

+ resources:

+ requests:

+ storage: 20Gi

+ volumeName: pv-azuredisk

+ storageClassName: managed-csi

+ ```

+3. Use the [kubectl apply][kubectl-apply] commands to create the *PersistentVolume* and *PersistentVolumeClaim*, referencing the two YAML files created earlier:

+ ```bash

+ kubectl apply -f pv-azuredisk.yaml

+ kubectl apply -f pvc-azuredisk.yaml

+ ```

+4. To verify your *PersistentVolumeClaim* is created and bound to the *PersistentVolume*, run the

+following command:

+ ```bash

+ kubectl get pvc pvc-azuredisk

+ ```

+ The output of the command resembles the following example:

+ ```console

+ NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

+ pvc-azuredisk Bound pv-azuredisk 20Gi RWO 5s

+ ```

+5. Create a *azure-disk-pod.yaml* file to reference your *PersistentVolumeClaim*. For example:

+ ```yaml

+ apiVersion: v1

+ kind: Pod

+ metadata:

+ name: mypod

+ spec:

+ nodeSelector:

+ kubernetes.io/os: linux

+ containers:

+ - image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine

+ name: mypod

+ resources:

+ requests:

+ cpu: 100m

+ memory: 128Mi

+ limits:

+ cpu: 250m

+ memory: 256Mi

+ volumeMounts:

+ - name: azure

+ mountPath: /mnt/azure

+ volumes:

+ - name: azure

+ persistentVolumeClaim:

+ claimName: pvc-azuredisk

+ ```

+6. Run the [kubectl apply][kubectl-apply] command to apply the configuration and mount the volume, referencing the YAML

+configuration file created in the previous steps:

+ ```bash

+ kubectl apply -f azure-disk-pod.yaml

+ ```

+## Next steps

+- To learn how to use CSI driver for Azure Disks storage, see [Use Azure Disks storage with CSI driver][azure-disks-storage-csi].

+- For associated best practices, see [Best practices for storage and backups in AKS][operator-best-practices-storage].

+<!-- LINKS - external -->

+[access-modes]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes

+[kubectl-apply]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#apply

+[kubectl-get]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get

+[kubernetes-storage-classes]: https://kubernetes.io/docs/concepts/storage/storage-classes/

+[kubernetes-volumes]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/

+[managed-disk-pricing-performance]: https://azure.microsoft.com/pricing/details/managed-disks/

+[kubectl-describe]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe

+<!-- LINKS - internal -->

+[azure-storage-account]: ../storage/common/storage-introduction.md

+[azure-disks-storage-csi]: azure-disk-csi.md

+[azure-files-pvc]: azure-files-dynamic-pv.md

+[az-disk-list]: /cli/azure/disk#az_disk_list

+[az-snapshot-create]: /cli/azure/snapshot#az_snapshot_create

+[az-disk-create]: /cli/azure/disk#az_disk_create

+[az-disk-show]: /cli/azure/disk#az_disk_show

+[az-aks-show]: /cli/azure/aks#az-aks-show

+[install-azure-cli]: /cli/azure/install-azure-cli

+[operator-best-practices-storage]: operator-best-practices-storage.md

+[concepts-storage]: concepts-storage.md

+[storage-class-concepts]: concepts-storage.md#storage-classes

+[use-tags]: use-tags.md

+[share-azure-managed-disk]: ../virtual-machines/disks-shared.md

+[disk-host-cache-setting]: ../virtual-machines/windows/premium-storage-performance.md#disk-caching

+[use-ultra-disks]: use-ultra-disks.md

+[ultra-ssd-disks]: ../virtual-machines/linux/disks-ultra-ssd.md

+[azure-tags]: ../azure-resource-manager/management/tag-resources.md

+[disk-encryption]: ../virtual-machines/windows/disk-encryption.md

+[azure-disk-write-accelerator]: ../virtual-machines/windows/how-to-enable-write-accelerator.md

+[on-demand-bursting]: ../virtual-machines/disk-bursting.md

+[customer-usage-attribution]: ../marketplace/azure-partner-customer-usage-attribution.md

+ Title: Create a persistent volume with Azure Files in Azure Kubernetes Service (AKS)

+description: Learn how to create a static or dynamic persistent volume with Azure Files for use with multiple concurrent pods in Azure Kubernetes Service (AKS)

+# Create and use a volume with Azure Files in Azure Kubernetes Service (AKS)

+A persistent volume represents a piece of storage that has been provisioned for use with Kubernetes pods. A persistent volume can be used by one or many pods, and can be dynamically or statically provisioned. If multiple pods need concurrent access to the same storage volume, you can use Azure Files to connect using the [Server Message Block (SMB) protocol][smb-overview]. This article shows you how to dynamically create an Azure Files share for use by multiple pods in an Azure Kubernetes Service (AKS) cluster.

+This article shows you how to:

+* Work with a dynamic persistent volume (PV) by installing the Container Storage Interface (CSI) driver and dynamically creating one or more Azure file shares to attach to a pod.

+* Work with a static PV by creating one or more Azure file shares, or use an existing one and attach it to a pod.

+For more information on Kubernetes volumes, see [Storage options for applications in AKS][concepts-storage].

+## Before you begin

+- An Azure [storage account][azure-storage-account].

+- The Azure CLI version 2.0.59 or later installed and configured. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].

+## Dynamically provision a volume

+This section provides guidance for cluster administrators who want to provision one or more persistent volumes that include details of one or more shares on Azure Files for use by a workload. A persistent volume claim (PVC) uses the storage class object to dynamically provision an Azure Files file share.

+### Dynamic provisioning parameters

+|Name | Meaning | Available Value | Mandatory | Default value

+| | | | |

+|skuName | Azure Files storage account type (alias: `storageAccountType`)| `Standard_LRS`, `Standard_ZRS`, `Standard_GRS`, `Standard_RAGRS`, `Standard_RAGZRS`,`Premium_LRS`, `Premium_ZRS` | No | `StandardSSD_LRS`<br> Minimum file share size for Premium account type is 100 GB.<br> ZRS account type is supported in limited regions.<br> NFS file share only supports Premium account type.|

+|fsType | File System Type | `ext4`, `ext3`, `ext2`, `xfs`| Yes | `ext4` for Linux|

+|location | Specify Azure region where Azure storage account will be created. | For example, `eastus`. | No | If empty, driver uses the same location name as current AKS cluster.|

+|resourceGroup | Specify the resource group where the Azure Disks will be created | Existing resource group name | No | If empty, driver uses the same resource group name as current AKS cluster.|

+|shareName | Specify Azure file share name | Existing or new Azure file share name. | No | If empty, driver generates an Azure file share name. |

+|shareNamePrefix | Specify Azure file share name prefix created by driver. | Share name can only contain lowercase letters, numbers, hyphens, and length should be fewer than 21 characters. | No |

+|folderName | Specify folder name in Azure file share. | Existing folder name in Azure file share. | No | If folder name does not exist in file share, mount will fail. |

+|shareAccessTier | [Access tier for file share][storage-tiers] | General purpose v2 account can choose between `TransactionOptimized` (default), `Hot`, and `Cool`. Premium storage account type for file shares only. | No | Empty. Use default setting for different storage account types.|

+|accountAccessTier | [Access tier for storage account][access-tiers-overview] | Standard account can choose `Hot` or `Cool`, and Premium account can only choose `Premium`. | No | Empty. Use default setting for different storage account types. |

+|server | Specify Azure storage account server address | Existing server address, for example `accountname.privatelink.file.core.windows.net`. | No | If empty, driver uses default `accountname.file.core.windows.net` or other sovereign cloud account address. |

+|disableDeleteRetentionPolicy | Specify whether disable DeleteRetentionPolicy for storage account created by driver. | `true` or `false` | No | `false` |

+|allowBlobPublicAccess | Allow or disallow public access to all blobs or containers for storage account created by driver. | `true` or `false` | No | `false` |

+|requireInfraEncryption | Specify whether or not the service applies a secondary layer of encryption with platform managed keys for data at rest for storage account created by driver. | `true` or `false` | No | `false` |

+|storageEndpointSuffix | Specify Azure storage endpoint suffix. | `core.windows.net`, `core.chinacloudapi.cn`, etc. | No | If empty, driver uses default storage endpoint suffix according to cloud environment. For example, `core.windows.net`. |

+|tags | [Tags][tag-resources] are created in new storage account. | Tag format: 'foo=aaa,bar=bbb' | No | "" |

+|matchTags | Match tags when driver tries to find a suitable storage account. | `true` or `false` | No | `false` |

+| | **Following parameters are only for SMB protocol** | | |

+|subscriptionID | Specify Azure subscription ID where Azure file share is created. | Azure subscription ID | No | If not empty, `resourceGroup` must be provided. |

+|storeAccountKey | Specify whether to store account key to Kubernetes secret. | `true` or `false`<br>`false` means driver leverages kubelet identity to get account key. | No | `true` |

+|secretName | Specify secret name to store account key. | | No |

+|secretNamespace | Specify the namespace of secret to store account key. <br><br> **Note:** <br> If `secretNamespace` isn't specified, the secret is created in the same namespace as the pod. | `default`,`kube-system`, etc | No | Pvc namespace, for example `csi.storage.k8s.io/pvc/namespace` |

+|useDataPlaneAPI | Specify whether to use [data plane API][data-plane-api] for file share create/delete/resize. This could solve the SRP API throttling issue because the data plane API has almost no limit, while it would fail when there is firewall or Vnet setting on storage account. | `true` or `false` | No | `false` |

+| | **Following parameters are only for NFS protocol** | | |

+|rootSquashType | Specify root squashing behavior on the share. The default is `NoRootSquash` | `AllSquash`, `NoRootSquash`, `RootSquash` | No |

+|mountPermissions | Mounted folder permissions. The default is `0777`. If set to `0`, driver doesn't perform `chmod` after mount | `0777` | No |

+| | **Following parameters are only for VNet setting. For example, NFS, private end point** | | |

+|vnetResourceGroup | Specify VNet resource group where virtual network is defined. | Existing resource group name. | No | If empty, driver uses the `vnetResourceGroup` value in Azure cloud config file. |

+|vnetName | Virtual network name | Existing virtual network name. | No | If empty, driver uses the `vnetName` value in Azure cloud config file. |

+|subnetName | Subnet name | Existing subnet name of the agent node. | No | If empty, driver uses the `subnetName` value in Azure cloud config file. |

+|fsGroupChangePolicy | Indicates how volume's ownership is changed by the driver. Pod `securityContext.fsGroupChangePolicy` is ignored. | `OnRootMismatch` (default), `Always`, `None` | No | `OnRootMismatch`|

+### Create a storage class

+A storage class is used to define how an Azure file share is created. A storage account is automatically created in the [node resource group][node-resource-group] for use with the storage class to hold the Azure Files file share. Choose of the following [Azure storage redundancy][storage-skus] for *skuName*:

+* *Standard_LRS* - standard locally redundant storage (LRS)

+* *Standard_GRS* - standard geo-redundant storage (GRS)

+* *Standard_ZRS* - standard zone redundant storage (ZRS)

+* *Standard_RAGRS* - standard read-access geo-redundant storage (RA-GRS)

+* *Premium_LRS* - premium locally redundant storage (LRS)

+* *Premium_ZRS* - premium zone redundant storage (ZRS)

+> [!NOTE]

+> Minimum premium file share is 100GB.

+For more information on Kubernetes storage classes for Azure Files, see [Kubernetes Storage Classes][kubernetes-storage-classes].

+Create a file named `azure-file-sc.yaml` and copy in the following example manifest. For more information on *mountOptions*, see the [Mount options][mount-options] section.

+```yaml

+kind: StorageClass

+apiVersion: storage.k8s.io/v1

+metadata:

+ name: my-azurefile

+provisioner: file.csi.azure.com # replace with "kubernetes.io/azure-file" if aks version is less than 1.21

+allowVolumeExpansion: true

+mountOptions:

+ - dir_mode=0777

+ - file_mode=0777

+ - uid=0

+ - gid=0

+ - mfsymlinks

+ - cache=strict

+ - actimeo=30

+parameters:

+ skuName: Premium_LRS

+```

+Create the storage class with the [kubectl apply][kubectl-apply] command:

+```bash

+kubectl apply -f azure-file-sc.yaml

+```

+### Create a persistent volume claim

+A persistent volume claim (PVC) uses the storage class object to dynamically provision an Azure file share. The following YAML can be used to create a persistent volume claim *100 GB* in size with *ReadWriteMany* access. For more information on access modes, see the [Kubernetes persistent volume][access-modes] documentation.

+Now create a file named `azure-file-pvc.yaml` and copy in the following YAML. Make sure that the *storageClassName* matches the storage class created in the last step:

+```yaml

+apiVersion: v1

+kind: PersistentVolumeClaim

+metadata:

+ name: my-azurefile

+spec:

+ accessModes:

+ - ReadWriteMany

+ storageClassName: my-azurefile

+ resources:

+ requests:

+ storage: 100Gi

+```

+> [!NOTE]

+> If using the *Premium_LRS* sku for your storage class, the minimum value for *storage* must be *100Gi*.

+Create the persistent volume claim with the [kubectl apply][kubectl-apply] command:

+```bash

+kubectl apply -f azure-file-pvc.yaml

+```

+Once completed, the file share will be created. A Kubernetes secret is also created that includes connection information and credentials. You can use the [kubectl get][kubectl-get] command to view the status of the PVC:

+```bash

+kubectl get pvc my-azurefile

+```

+The output of the command resembles the following example:

+```console

+NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

+my-azurefile Bound pvc-8436e62e-a0d9-11e5-8521-5a8664dc0477 10Gi RWX my-azurefile 5m

+```

+### Use the persistent volume

+The following YAML creates a pod that uses the persistent volume claim *my-azurefile* to mount the Azure Files file share at the */mnt/azure* path. For Windows Server containers, specify a *mountPath* using the Windows path convention, such as *'D:'*.

+Create a file named `azure-pvc-files.yaml`, and copy in the following YAML. Make sure that the *claimName* matches the PVC created in the last step.

+```yaml

+kind: Pod

+apiVersion: v1

+metadata:

+ name: mypod

+spec:

+ containers:

+ - name: mypod

+ image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine

+ resources:

+ requests:

+ cpu: 100m

+ memory: 128Mi

+ limits:

+ cpu: 250m

+ memory: 256Mi

+ volumeMounts:

+ - mountPath: "/mnt/azure"

+ name: volume

+ volumes:

+ - name: volume

+ persistentVolumeClaim:

+ claimName: my-azurefile

+```

+Create the pod with the [kubectl apply][kubectl-apply] command.

+```bash

+kubectl apply -f azure-pvc-files.yaml

+```

+You now have a running pod with your Azure Files file share mounted in the */mnt/azure* directory. This configuration can be seen when inspecting your pod using the [kubectl describe][kubectl-describe] command. The following condensed example output shows the volume mounted in the container:

+```console

+Containers:

+ mypod:

+ Container ID: docker://053bc9c0df72232d755aa040bfba8b533fa696b123876108dec400e364d2523e

+ Image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine

+ Image ID: docker-pullable://nginx@sha256:d85914d547a6c92faa39ce7058bd7529baacab7e0cd4255442b04577c4d1f424

+ State: Running

+ Started: Fri, 01 Mar 2019 23:56:16 +0000

+ Ready: True

+ Mounts:

+ /mnt/azure from volume (rw)

+ /var/run/secrets/kubernetes.io/serviceaccount from default-token-8rv4z (ro)

+[...]

+Volumes:

+ volume:

+ Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

+ ClaimName: my-azurefile

+ ReadOnly: false

+[...]

+```

+### Mount options

+The default value for *fileMode* and *dirMode* is *0777* for Kubernetes version 1.13.0 and above. If dynamically creating the persistent volume with a storage class, mount options can be specified on the storage class object. The following example sets *0777*:

+```yaml

+kind: StorageClass

+apiVersion: storage.k8s.io/v1

+metadata:

+ name: my-azurefile

+provisioner: file.csi.azure.com # replace with "kubernetes.io/azure-file" if aks version is less than 1.21

+allowVolumeExpansion: true

+mountOptions:

+ - dir_mode=0777

+ - file_mode=0777

+ - uid=0

+ - gid=0

+ - mfsymlinks

+ - cache=strict

+ - actimeo=30

+parameters:

+ skuName: Premium_LRS

+```

+### Using Azure tags

+For more details on using Azure tags, see [Use Azure tags in Azure Kubernetes Service (AKS)][use-tags].

+## Statically provision a volume

+This section provides guidance for cluster administrators who want to create one or more persistent volumes that include details of an existing Azure Files share to use with a workload.

+### Static provisioning parameters

+|Name | Meaning | Available Value | Mandatory | Default value |

+| | | | | |

+|volumeAttributes.resourceGroup | Specify an Azure resource group name. | myResourceGroup | No | If empty, driver uses the same resource group name as current cluster. |

+|volumeAttributes.storageAccount | Specify an existing Azure storage account name. | storageAccountName | Yes ||

+|volumeAttributes.shareName | Specify an Azure file share name. | fileShareName | Yes ||

+|volumeAttributes.folderName | Specify a folder name in Azure file share. | folderName | No | If folder name doesn't exist in file share, mount would fail. |

+|volumeAttributes.protocol | Specify file share protocol. | `smb`, `nfs` | No | `smb` |

+|volumeAttributes.server | Specify Azure storage account server address | Existing server address, for example `accountname.privatelink.file.core.windows.net`. | No | If empty, driver uses default `accountname.file.core.windows.net` or other sovereign cloud account address. |

+| | **Following parameters are only for SMB protocol** | | | |

+|volumeAttributes.secretName | Specify a secret name that stores storage account name and key. | | No |

+|volumeAttributes.secretNamespace | Specify a secret namespace. | `default`,`kube-system`, etc. | No | PVC namespace (`csi.storage.k8s.io/pvc/namespace`) |

+|nodeStageSecretRef.name | Specify a secret name that stores storage account name and key. | Existing secret name | Yes ||

+|nodeStageSecretRef.namespace | Specify a secret namespace. | Kubernetes namespace | Yes ||

+| | **Following parameters are only for NFS protocol** | | | |

+|volumeAttributes.fsGroupChangePolicy | Indicates how a volumes ownership is changed by the driver. Pod `securityContext.fsGroupChangePolicy` is ignored. | `OnRootMismatch` (default), `Always`, `None` | No | `OnRootMismatch` |

+|volumeAttributes.mountPermissions | Specify mounted folder permissions. The default is `0777` | | No ||

+### Create an Azure file share

+Before you can use an Azure Files file share as a Kubernetes volume, you must create an Azure Storage account and the file share. In this article, you'll create the storage container in the node resource group.

+1. Get the resource group name with the [az aks show][az-aks-show] command and add the `--query nodeResourceGroup` query parameter. The following example gets the node resource group for the AKS cluster named **myAKSCluster** in the resource group named **myResourceGroup**.

+ ```azurecli

+ az aks show --resource-group myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv

+ ```

+ The output of the command resembles the following example:

+ ```azurecli

+ MC_myResourceGroup_myAKSCluster_eastus

+ ```

+2. The following command creates a storage account using the Standard_LRS SKU. Replace the following placeholders:

+ * `myAKSStorageAccount` with the name of the storage account

+ * `nodeResourceGroupName` with the name of the resource group that the AKS cluster nodes are hosted in

+ * `location` with the name of the region to create the resource in. It should be the same region as the AKS cluster nodes.

+ ```azurecli

+ az storage account create -n myAKSStorageAccount -g nodeResourceGroupName -l location --sku Standard_LRS

+ ```

+3. Run the following command to export the connection string as an environment variable. This is used when creating the Azure file share in a later step.

+ ```azurecli

+ export AZURE_STORAGE_CONNECTION_STRING=$(az storage account show-connection-string -n storageAccountName -g resourceGroupName -o tsv)

+ ```

+4. Create the file share using the [Az storage share create][az-storage-share-create] command. Replace the placeholder `shareName` with a name you want to use for the share.

+ ```azurecli

+ az storage share create -n shareName --connection-string $AZURE_STORAGE_CONNECTION_STRING

+ ```

+5. Run the following command to export the storage account key as an environment variable.

+ ```azurecli

+ STORAGE_KEY=$(az storage account keys list --resource-group $AKS_PERS_RESOURCE_GROUP --account-name $AKS_PERS_STORAGE_ACCOUNT_NAME --query "[0].value" -o tsv)

+ ```

+6. Run the following commands to echo the storage account name and key. Copy this information as these values are needed when you create the Kubernetes volume later in this article.

+ ```azurecli

+ echo Storage account name: $AKS_PERS_STORAGE_ACCOUNT_NAME

+ echo Storage account key: $STORAGE_KEY

+ ```

+### Create a Kubernetes secret

+Kubernetes needs credentials to access the file share created in the previous step. These credentials are stored in a [Kubernetes secret][kubernetes-secret], which is referenced when you create a Kubernetes pod.

+Use the `kubectl create secret` command to create the secret. The following example creates a secret named *azure-secret* and populates the *azurestorageaccountname* and *azurestorageaccountkey* from the previous step. To use an existing Azure storage account, provide the account name and key.

+```bash

+kubectl create secret generic azure-secret --from-literal=azurestorageaccountname=$AKS_PERS_STORAGE_ACCOUNT_NAME --from-literal=azurestorageaccountkey=$STORAGE_KEY

+```

+### Mount file share as an inline volume

+> [!NOTE]

+> Inline volume can only access secrets in the same namespace as the pod. To specify a different secret namespace, [please use the persistent volume example][persistent-volume-example] below instead.

+To mount the Azure Files file share into your pod, configure the volume in the container spec. Create a new file named `azure-files-pod.yaml` with the following contents. If you changed the name of the file share or secret name, update the *shareName* and *secretName*. If desired, update the `mountPath`, which is the path where the Files share is mounted in the pod. For Windows Server containers, specify a *mountPath* using the Windows path convention, such as *'D:'*.

+```yaml

+apiVersion: v1

+kind: Pod

+metadata:

+ name: mypod

+spec:

+ nodeSelector:

+ kubernetes.io/os: linux

+ containers:

+ - image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine

+ name: mypod

+ resources:

+ requests:

+ cpu: 100m

+ memory: 128Mi

+ limits:

+ cpu: 250m

+ memory: 256Mi

+ volumeMounts:

+ - name: azure

+ mountPath: /mnt/azure

+ volumes:

+ - name: azure

+ csi:

+ driver: file.csi.azure.com

+ readOnly: false

+ volumeAttributes:

+ secretName: azure-secret # required

+ shareName: aksshare # required

+ mountOptions: "dir_mode=0777,file_mode=0777,cache=strict,actimeo=30,nosharesock" # optional

+```

+Use the [kubectl apply][kubectl-apply] command to create the pod.

+```bash

+kubectl apply -f azure-files-pod.yaml

+```

+You now have a running pod with an Azure Files file share mounted at */mnt/azure*. You can verify the share is mounted successfully using the [kubectl describe][kubectl-describe] command:

+```bash

+kubectl describe pod mypod

+```

+### Mount file share as a persistent volume

+The following example demonstrates how to mount a file share as a persistent volume.

+1. Create a file named `azurefiles-pv.yaml` and copy in the following YAML. Under `csi`, update `resourceGroup`, `volumeHandle`, and `shareName`. For mount options, the default value for *fileMode* and *dirMode* is *0777*.

+ ```yaml

+ apiVersion: v1

+ kind: PersistentVolume

+ metadata:

+ name: azurefile

+ spec:

+ capacity:

+ storage: 5Gi

+ accessModes:

+ - ReadWriteMany

+ persistentVolumeReclaimPolicy: Retain

+ storageClassName: azurefile-csi

+ csi:

+ driver: file.csi.azure.com

+ readOnly: false

+ volumeHandle: unique-volumeid # make sure this volumeid is unique for every identical share in the cluster

+ volumeAttributes:

+ resourceGroup: resourceGroupName # optional, only set this when storage account is not in the same resource group as node

+ shareName: aksshare

+ nodeStageSecretRef:

+ name: azure-secret

+ namespace: default

+ mountOptions:

+ - dir_mode=0777

+ - file_mode=0777

+ - uid=0

+ - gid=0

+ - mfsymlinks

+ - cache=strict

+ - nosharesock

+ - nobrl

+ ```

+2. Run the following command to create the persistent volume using the [kubectl create][kubectl-create] command referencing the YAML file created earlier:

+ ```bash

+ kubectl create -f azurefiles-pv.yaml

+ ```

+3. Create a *azurefiles-mount-options-pvc.yaml* file with a *PersistentVolumeClaim* that uses the *PersistentVolume* and copy the following YAML.

+ ```yaml

+ apiVersion: v1

+ kind: PersistentVolumeClaim

+ metadata:

+ name: azurefile

+ spec:

+ accessModes:

+ - ReadWriteMany

+ storageClassName: azurefile-csi

+ volumeName: azurefile

+ resources:

+ requests:

+ storage: 5Gi

+ ```

+4. Use the `kubectl` commands to create the *PersistentVolumeClaim*.

+```bash

+kubectl apply -f azurefiles-mount-options-pvc.yaml

+```

+5. Verify your *PersistentVolumeClaim* is created and bound to the *PersistentVolume* by running the following command.

+ ```bash

+ kubectl get pvc azurefile

+ ```

+ The output from the command resembles the following example:

+ ```console

+ NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

+ azurefile Bound azurefile 5Gi RWX azurefile 5s

+ ```

+6. Update your container spec to reference your *PersistentVolumeClaim* and update your pod. For example:

+ ```yaml

+ ...

+ volumes:

+ - name: azure

+ persistentVolumeClaim:

+ claimName: azurefile

+ ```

+7. Because a pod spec can't be updated in place, use [kubectl delete][kubectl-delete] and [kubectl apply][kubectl-apply] commands to delete and then re-create the pod:

+ ```bash

+ kubectl delete pod mypod

+

+ kubectl apply -f azure-files-pod.yaml

+ ```

+## Next steps

+For Azure File CSI driver parameters, see [CSI driver parameters][CSI driver parameters].

+For associated best practices, see [Best practices for storage and backups in AKS][operator-best-practices-storage].

+<!-- LINKS - external -->

+[kubernetes-secret]: https://kubernetes.io/docs/concepts/configuration/secret/

+[smb-overview]: /windows/desktop/FileIO/microsoft-smb-protocol-and-cifs-protocol-overview

+[CSI driver parameters]: https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/docs/driver-parameters.md#static-provisionbring-your-own-file-share

+[kubernetes-storage-classes]: https://kubernetes.io/docs/concepts/storage/storage-classes/#azure-file

+[kubernetes-persistent-volume]: https://kubernetes.io/docs/concepts/storage/persistent-volumes

+[kubectl-apply]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#apply

+[kubectl-get]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get

+[kubectl-create]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#create

+[data-plane-api]: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

+[kubectl-describe]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe

+[kubectl-delete]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#delete

+<!-- LINKS - internal -->

+[azure-storage-account]: ../storage/common/storage-introduction.md

+[install-azure-cli]: /cli/azure/install-azure-cli

+[operator-best-practices-storage]: operator-best-practices-storage.md

+[concepts-storage]: concepts-storage.md

+[persistent-volume-example]: #mount-file-share-as-a-persistent-volume

+[use-tags]: use-tags.md

+[node-resource-group]: faq.md#why-are-two-resource-groups-created-with-aks

+[storage-skus]: ../storage/common/storage-redundancy.md

+[mount-options]: #mount-options

+[az-aks-show]: /cli/azure/aks#az-aks-show

+[az-storage-share-create]: /cli/azure/storage/share#az-storage-share-create

+[storage-tiers]: ../storage/files/storage-files-planning.md#storage-tiers

+[access-tiers-overview]: ../storage/blobs/access-tiers-overview.md

+[tag-resources]: ../azure-resource-manager/management/tag-resources.md

+> [!NOTE]

+> Depending on the VM SKU that's being used, the Azure Disks CSI driver might have a per-node volume limit. For some powerful VMs (for example, 16 cores), the limit is 64 volumes per node. To identify the limit per VM SKU, review the **Max data disks** column for each VM SKU offered. For a list of VM SKUs offered and their corresponding detailed capacity limits, see [General purpose virtual machine sizes][general-purpose-machine-sizes].

+|diskEncryptionType | Encryption type of the disk encryption set. | `EncryptionAtRestWithCustomerKey`(by default), `EncryptionAtRestWithPlatformAndCustomerKeys` | No | ""|

+|diskAccessID | Azure Resource ID of the DiskAccess resource to use private endpoints on disks | | No | ``|

+|subscriptionID | Specify Azure subscription ID where the Azure Disks is created. | Azure subscription ID | No | If not empty, `resourceGroup` must be provided.|

+A [persistent volume](concepts-storage.md#persistent-volumes) (PV) represents a piece of storage that's provisioned for use with Kubernetes pods. A PV can be used by one or many pods and can be dynamically or statically provisioned. This article shows you how to dynamically create PVs with Azure disk for use by a single pod in an AKS cluster. For static provisioning, see [Create a static volume with Azure Disks](azure-csi-disk-storage-provision.md#statically-provision-a-volume).

+- To learn how to use CSI driver for Azure Blob storage, see [Use Azure Blob storage with CSI driver][azure-blob-csi].

+[general-purpose-machine-sizes]: ../virtual-machines/sizes-general.md

+|location | Specify Azure region where Azure storage account will be created. | For example, `eastus`. | No | If empty, driver uses the same location name as current AKS cluster.|

+|resourceGroup | Specify the resource group where the Azure Disks will be created. | Existing resource group name | No | If empty, driver uses the same resource group name as current AKS cluster.|

+|shareNamePrefix | Specify Azure file share name prefix created by driver. | Share name can only contain lowercase letters, numbers, hyphens, and length should be fewer than 21 characters. | No |

+|tags | [tags][tag-resources] are created in new storage account. | Tag format: 'foo=aaa,bar=bbb' | No | "" |

+|storeAccountKey | Specify whether to store account key to Kubernetes secret. | `true` or `false`<br>`false` means driver leverages kubelet identity to get account key. | No | `true` |

+A [persistent volume (PV)][persistent-volume] represents a piece of storage that's provisioned for use with Kubernetes pods. A PV can be used by one or many pods and can be dynamically or statically provisioned. If multiple pods need concurrent access to the same storage volume, you can use Azure Files to connect by using the [Server Message Block (SMB)][smb-overview] or [NFS protocol][nfs-overview]. This article shows you how to dynamically create an Azure Files share for use by multiple pods in an AKS cluster. For static provisioning, see [Manually create and use a volume with an Azure Files share][azure-files-storage-provision.md#statically-provision-a-volume].

+When you use storage CSI drivers on AKS, there are two more built-in `StorageClasses` that uses the Azure Files CSI storage drivers. The other CSI storage classes are created with the cluster alongside the in-tree default storage classes.

+In AKS, the built-in `azurefile-csi` storage class already supports expansion, so use the [PVC created earlier with this storage class](#dynamically-create-azure-files-pvs-by-using-the-built-in-storage-classes). The PVC requested a 100 GiB file share. We can confirm that by running:

+[access-tier-file-share]: ../storage/files/storage-files-planning#storage-tiers.md

+[access-tier-storage-account]: ../storage/blobs/access-tiers-overview.md

+[azure-tags]: ../azure-resource-manager/management/tag-resources.md

+## Limitation

+Certificate rotation is not supported for stopped AKS clusters.

+> Depending on the VM SKU that's being used, the Azure Disks CSI driver might have a per-node volume limit. For some powerful VMs (for example, 16 cores), the limit is 64 volumes per node. To identify the limit per VM SKU, review the **Max data disks** column for each VM SKU offered. For a list of VM SKUs offered and their corresponding detailed capacity limits, see [General purpose virtual machine sizes][general-purpose-machine-sizes].

+Unless you specify a StorageClass for a persistent volume, the default StorageClass will be used. Ensure volumes use the appropriate storage you need when requesting persistent volumes.

+> Starting with Kubernetes version 1.21, AKS only uses CSI drivers by default and CSI migration is enabled. While existing in-tree persistent volumes continue to function, starting with version 1.26, AKS will no longer support volumes created using in-tree driver and storage provisioned for files and disk.

+>

+> The `default` class will be the same as `managed-csi`.

+[general-purpose-machine-sizes]: ../virtual-machines/sizes-general.md

+ Title: Configure Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in Azure Kubernetes Service (AKS)

+description: Learn how to configure Azure CNI (advanced) networking for dynamic allocation of IPs and enhanced subnet support in Azure Kubernetes Service (AKS)

+# Configure Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in Azure Kubernetes Service (AKS)

+A drawback with the traditional CNI is the exhaustion of pod IP addresses as the AKS cluster grows, which results in the need to rebuild your entire cluster in a bigger subnet. The new dynamic IP allocation capability in Azure CNI solves this problem by allocating pod IPs from a subnet separate from the subnet hosting the AKS cluster.

+It offers the following benefits:

+* **Better IP utilization**: IPs are dynamically allocated to cluster Pods from the Pod subnet. This leads to better utilization of IPs in the cluster compared to the traditional CNI solution, which does static allocation of IPs for every node.

+* **Scalable and flexible**: Node and pod subnets can be scaled independently. A single pod subnet can be shared across multiple node pools of a cluster or across multiple AKS clusters deployed in the same VNet. You can also configure a separate pod subnet for a node pool.

+* **High performance**: Since pod are assigned VNet IPs, they have direct connectivity to other cluster pod and resources in the VNet. The solution supports very large clusters without any degradation in performance.

+* **Separate VNet policies for pods**: Since pods have a separate subnet, you can configure separate VNet policies for them that are different from node policies. This enables many useful scenarios such as allowing internet connectivity only for pods and not for nodes, fixing the source IP for pod in a node pool using a VNet Network NAT, and using NSGs to filter traffic between node pools.

+* **Kubernetes network policies**: Both the Azure Network Policies and Calico work with this new solution.

+This article shows you how to use Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in AKS.

+## Prerequisites

+> [!NOTE]

+> When using dynamic allocation of IPs, exposing an application as a Private Link Service using a Kubernetes Load Balancer Service isn't supported.

+* Review the [prerequisites](/configure-azure-cni.md#prerequisites) for configuring basic Azure CNI networking in AKS, as the same prerequisites apply to this article.

+* Review the [deployment parameters](/configure-azure-cni.md#deployment-parameters) for configuring basic Azure CNI networking in AKS, as the same parameters apply.

+* Only linux node clusters and node pools are supported.

+* AKS Engine and DIY clusters aren't supported.

+* Azure CLI version `2.37.0` or later.

+## Plan IP addressing

+Planning your IP addressing is much simpler with this feature. Since the nodes and pods scale independently, their address spaces can also be planned separately. Since pod subnets can be configured to the granularity of a node pool, you can always add a new subnet when you add a node pool. The system pods in a cluster/node pool also receive IPs from the pod subnet, so this behavior needs to be accounted for.

+IPs are allocated to nodes in batches of 16. Pod subnet IP allocation should be planned with a minimum of 16 IPs per node in the cluster; nodes will request 16 IPs on startup and will request another batch of 16 any time there are <8 IPs unallocated in their allotment.

+The planning of IPs for Kubernetes services and Docker bridge remain unchanged.

+## Maximum pods per node in a cluster with dynamic allocation of IPs and enhanced subnet support

+The pods per node values when using Azure CNI with dynamic allocation of IPs slightly differ from the traditional CNI behavior:

+|CNI|Default|Configurable at deployment|

+|--| :--: |--|

+|Traditional Azure CNI|30|Yes (up to 250)|

+|Azure CNI with dynamic allocation of IPs|250|Yes (up to 250)|

+All other guidance related to configuring the maximum pods per node remains the same.

+## Deployment parameters

+The [deployment parameters](/configure-azure-cni.md#deployment-parameters) for configuring basic Azure CNI networking in AKS are all valid, with two exceptions:

+* The **subnet** parameter now refers to the subnet related to the cluster's nodes.

+* An additional parameter **pod subnet** is used to specify the subnet whose IP addresses will be dynamically allocated to pods.

+## Configure networking with dynamic allocation of IPs and enhanced subnet support - Azure CLI

+Using dynamic allocation of IPs and enhanced subnet support in your cluster is similar to the default method for configuring a cluster Azure CNI. The following example walks through creating a new virtual network with a subnet for nodes and a subnet for pods, and creating a cluster that uses Azure CNI with dynamic allocation of IPs and enhanced subnet support. Be sure to replace variables such as `$subscription` with your own values.

+Create the virtual network with two subnets.

+```azurecli-interactive

+resourceGroup="myResourceGroup"

+vnet="myVirtualNetwork"

+location="westcentralus"

+# Create the resource group

+az group create --name $resourceGroup --location $location

+# Create our two subnet network

+az network vnet create -g $resourceGroup --location $location --name $vnet --address-prefixes 10.0.0.0/8 -o none

+az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name nodesubnet --address-prefixes 10.240.0.0/16 -o none

+az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name podsubnet --address-prefixes 10.241.0.0/16 -o none

+```

+Create the cluster, referencing the node subnet using `--vnet-subnet-id` and the pod subnet using `--pod-subnet-id`.

+```azurecli-interactive

+clusterName="myAKSCluster"

+subscription="aaaaaaa-aaaaa-aaaaaa-aaaa"

+az aks create -n $clusterName -g $resourceGroup -l $location \

+ --max-pods 250 \

+ --node-count 2 \

+ --network-plugin azure \

+ --vnet-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/nodesubnet \

+ --pod-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/podsubnet

+```

+### Adding node pool

+When adding node pool, reference the node subnet using `--vnet-subnet-id` and the pod subnet using `--pod-subnet-id`. The following example creates two new subnets that are then referenced in the creation of a new node pool:

+```azurecli-interactive

+az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name node2subnet --address-prefixes 10.242.0.0/16 -o none

+az network vnet subnet create -g $resourceGroup --vnet-name $vnet --name pod2subnet --address-prefixes 10.243.0.0/16 -o none

+az aks nodepool add --cluster-name $clusterName -g $resourceGroup -n newnodepool \

+ --max-pods 250 \

+ --node-count 2 \

+ --vnet-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/node2subnet \

+ --pod-subnet-id /subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Network/virtualNetworks/$vnet/subnets/pod2subnet \

+ --no-wait

+```

+## Dynamic allocation of IP addresses and enhanced subnet support FAQs

+* **Can I assign multiple pod subnets to a cluster/node pool?**

+ Only one subnet can be assigned to a cluster or node pool. However, multiple clusters or node pools can share a single subnet.

+* **Can I assign Pod subnets from a different VNet altogether?**

+ No, the pod subnet should be from the same VNet as the cluster.

+* **Can some node pools in a cluster use the traditional CNI while others use the new CNI?**

+ The entire cluster should use only one type of CNI.

+## Next steps

+Learn more about networking in AKS in the following articles:

+* [Use a static IP address with the Azure Kubernetes Service (AKS) load balancer](static-ip.md)

+* [Use an internal load balancer with Azure Kubernetes Service (AKS)](internal-lb.md)

+* [Create a basic ingress controller with external network connectivity][aks-ingress-basic]

+* [Enable the HTTP application routing add-on][aks-http-app-routing]

+* [Create an ingress controller that uses an internal, private network and IP address][aks-ingress-internal]

+* [Create an ingress controller with a dynamic public IP and configure Let's Encrypt to automatically generate TLS certificates][aks-ingress-tls]

+* [Create an ingress controller with a static public IP and configure Let's Encrypt to automatically generate TLS certificates][aks-ingress-static-tls]

+<!-- LINKS - Internal -->

+[aks-ingress-basic]: ingress-basic.md

+[aks-ingress-tls]: ingress-tls.md

+[aks-ingress-static-tls]: ingress-static-ip.md

+[aks-http-app-routing]: http-application-routing.md

+[aks-ingress-internal]: ingress-internal-ip.md

+By default, AKS clusters use [kubenet][kubenet] and create a virtual network and subnet. With *kubenet*, nodes get an IP address from a virtual network subnet. Network address translation (NAT) is then configured on the nodes, and pods receive an IP address "hidden" behind the node IP. This approach reduces the number of IP addresses that you need to reserve in your network space for pods to use.

+With [Azure Container Networking Interface (CNI)][cni-networking], every pod gets an IP address from the subnet and can be accessed directly. These IP addresses must be unique across your network space and must be planned in advance. Each node has a configuration parameter for the maximum number of pods that it supports. The equivalent number of IP addresses per node are then reserved up front for that node. This approach requires more planning, and often leads to IP address exhaustion or the need to rebuild clusters in a larger subnet as your application demands grow.

+This article shows you how to use Azure CNI networking to create and use a virtual network subnet for an AKS cluster. For more information on network options and considerations, see [Network concepts for Kubernetes and AKS][aks-network-concepts].

+## Monitor IP subnet usage

+1. Download or grep the file named container-azm-ms-agentconfig.yaml from [GitHub][github].

+2. Find azure_subnet_ip_usage in integrations. Set `enabled` to `true`.

+3. Save the file.

+* **Can I deploy VMs in my cluster subnet?**

+* **What source IP do external systems see for traffic that originates in an Azure CNI-enabled pod?**

+* **Can I configure per-pod network policies?**

+* **Is the maximum number of pods deployable to a node configurable?**

+* **How do I configure additional properties for the subnet that I created during AKS cluster creation? For example, service endpoints.**

+* **Can I use a different subnet within my cluster virtual network for the *Kubernetes service address range*?**

+To configure Azure CNI networking with dynamic IP allocation and enhanced subnet support, see [Configure Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in AKS](/configure-azure-cni-dynamic-ip-allocation.md).

+ Title: Migrate from in-tree storage class to CSI drivers on Azure Kubernetes Service (AKS)

+description: Learn how to migrate from in-tree persistent volume to the Container Storage Interface (CSI) driver in an Azure Kubernetes Service (AKS) cluster.

+# Migrate from in-tree storage class to CSI drivers on Azure Kubernetes Service (AKS)

+The implementation of the [Container Storage Interface (CSI) driver][csi-driver-overview] was introduced in Azure Kubernetes Service (AKS) starting with version 1.21. By adopting and using CSI as the standard, your existing stateful workloads using in-tree Persistent Volumes (PVs) should be migrated or upgraded to use the CSI driver.

+To make this process as simple as possible, and to ensure no data loss, this article provides different migration options. These options include scripts to help ensure a smooth migration from in-tree to Azure Disks and Azure Files CSI drivers.

+## Before you begin

+* The Azure CLI version 2.37.0 or later. Run `az --version` to find the version, and run `az upgrade` to upgrade the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].

+* Kubectl and cluster administrators have access to create, get, list, delete access to a PVC or PV, volume snapshot, or volume snapshot content. For an Azure Active Directory (Azure AD) RBAC enabled cluster, you're a member of the [Azure Kubernetes Service RBAC Cluster Admin][aks-rbac-cluster-admin-role] role.

+## Migrate Disk volumes

+Migration from in-tree to CSI is supported using two migration options:

+* Create a static volume

+* Create a dynamic volume

+### Create a static volume

+Using this option, you create a PV by statically assigning `claimRef` to a new PVC that you'll create later, and specify the `volumeName` for the *PersistentVolumeClaim*.

+The benefits of this approach are:

+* It's simple and can be automated.

+* No need to clean up original configuration using in-tree storage class.

+* Low risk as you're only performing a logical deletion of Kubernetes PV/PVC, the actual physical data isn't deleted.

+* No extra costs as the result of not having to create more objects such as disk, snapshots, etc.

+The following are important considerations to evaluate:

+* Transition to static volumes from original dynamic-style volumes requires constructing and managing PV objects manually for all options.

+* Potential application downtime when redeploying the new application with reference to the new PVC object.

+#### Migration

+1. Update the existing PV `ReclaimPolicy` from **Delete** to **Retain** by running the following command:

+ ```bash

+ kubectl patch pv pvName -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

+ ```

+ Replace **pvName** with the name of your selected PersistentVolume. Alternatively, if you want to update the reclaimPolicy for multiple PVs, create a file named **patchReclaimPVs.sh** and copy in the following code.

+ ```bash

+ # Patch the Persistent Volume in case ReclaimPolicy is Delete

+ #!/bin/sh

+ namespace=$1

+ i=1

+ for pvc in $(kubectl get pvc -n $namespace | awk '{ print $1}'); do

+ # Ignore first record as it contains header

+ if [ $i -eq 1 ]; then

+ i=$((i + 1))

+ else

+ pv="$(kubectl get pvc $pvc -n $namespace -o jsonpath='{.spec.volumeName}')"

+ reclaimPolicy="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')"

+ echo "Reclaim Policy for Persistent Volume $pv is $reclaimPolicy"

+ if [[ $reclaimPolicy == "Delete" ]]; then

+ echo "Updating ReclaimPolicy for $pv to Retain"

+ kubectl patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

+ fi

+ fi

+ done

+ ```

+ Execute the script with the `namespace` parameter to specify the cluster namespace `./PatchReclaimPolicy.sh <namespace>`.

+2. Get a list of all of the PVCs in namespace sorted by **creationTimestamp** by running the following command. Set the namespace using the `--namespace` argument along with the actual cluster namespace.

+ ```bash

+ kubectl get pvc -n <namespace> --sort-by=.metadata.creationTimestamp -o custom-columns=NAME:.metadata.name,CreationTime:.metadata.creationTimestamp,StorageClass:.spec.storageClassName,Size:.spec.resources.requests.storage

+ ```

+ This step is helpful if you have a large number of PVs that need to be migrated, and you want to migrate a few at a time. Running this command enables you to identify which PVCs were created in a given time frame. When you run the *CreatePV.sh* script, two of the parameters are start time and end time that enable you to only migrate the PVCs during that period of time.

+3. Create a file named **CreatePV.sh** and copy in the following code. The script does the following:

+ * Creates a new PersistentVolume with name `existing-pv-csi` for all PersistentVolumes in namespaces for storage class `storageClassName`.

+ * Configure new PVC name as `existing-pvc-csi`.

+ * Updates the application (deployment/StatefulSet) to refer to new PVC.

+ * Creates a new PVC with the PV name you specify.

+ ```bash

+ #!/bin/sh

+ #kubectl get pvc -n <namespace> --sort-by=.metadata.creationTimestamp -o custom-columns=NAME:.metadata.name,CreationTime:.metadata.creationTimestamp,StorageClass:.spec.storageClassName,Size:.spec.resources.requests.storage

+ # TimeFormat 2022-04-20T13:19:56Z

+ namespace=$1

+ fileName=$(date +%Y%m%d%H%M)-$namespace

+ existingStorageClass=$2

+ storageClassNew=$3

+ starttimestamp=$4

+ endtimestamp=$5

+ i=1

+ for pvc in $(kubectl get pvc -n $namespace | awk '{ print $1}'); do

+ # Ignore first record as it contains header

+ if [ $i -eq 1 ]; then

+ i=$((i + 1))

+ else

+ pvcCreationTime=$(kubectl get pvc $pvc -n $namespace -o jsonpath='{.metadata.creationTimestamp}')

+ if [[ $pvcCreationTime > $starttimestamp ]]; then

+ if [[ $endtimestamp > $pvcCreationTime ]]; then

+ pv="$(kubectl get pvc $pvc -n $namespace -o jsonpath='{.spec.volumeName}')"

+ reclaimPolicy="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')"

+ storageClass="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.storageClassName}')"

+ echo $pvc

+ reclaimPolicy="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')"

+ if [[ $reclaimPolicy == "Retain" ]]; then

+ if [[ $storageClass == $existingStorageClass ]]; then

+ storageSize="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.capacity.storage}')"

+ skuName="$(kubectl get storageClass $storageClass -o jsonpath='{.reclaimPolicy}')"

+ diskURI="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.azureDisk.diskURI}')"

+ persistentVolumeReclaimPolicy="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')"

+

+ cat >$pvc-csi.yaml <<EOF

+ apiVersion: v1

+ kind: PersistentVolume

+ metadata:

+ annotations:

+ pv.kubernetes.io/provisioned-by: disk.csi.azure.com

+ name: $pv-csi

+ spec:

+ accessModes:

+ - ReadWriteOnce

+ capacity:

+ storage: $storageSize

+ claimRef:

+ apiVersion: v1

+ kind: PersistentVolumeClaim

+ name: $pvc-csi

+ namespace: $namespace

+ csi:

+ driver: disk.csi.azure.com

+ volumeAttributes:

+ csi.storage.k8s.io/pv/name: $pv-csi

+ csi.storage.k8s.io/pvc/name: $pvc-csi

+ csi.storage.k8s.io/pvc/namespace: $namespace

+ requestedsizegib: "$storageSize"

+ skuname: $skuName

+ volumeHandle: $diskURI

+ persistentVolumeReclaimPolicy: $persistentVolumeReclaimPolicy

+ storageClassName: $storageClassNew

+

+ apiVersion: v1

+ kind: PersistentVolumeClaim

+ metadata:

+ name: $pvc-csi

+ namespace: $namespace

+ spec:

+ accessModes:

+ - ReadWriteOnce

+ storageClassName: $storageClassNew

+ resources:

+ requests:

+ storage: $storageSize

+ volumeName: $pv-csi

+ EOF

+ kubectl apply -f $pvc-csi.yaml

+ line="PVC:$pvc,PV:$pv,StorageClassTarget:$storageClassNew"

+ printf '%s\n' "$line" >>$fileName

+ fi

+ fi

+ fi

+ fi

+ fi

+ done

+ ```

+4. To create a new PersistentVolume for all PersistentVolumes in the namespace, execute the script **CreatePV.sh** with the following parameters:

+ * `namespace` - The cluster namespace

+ * `sourceStorageClass` - The in-tree storage driver-based StorageClass

+ * `targetCSIStorageClass` - The CSI storage driver-based StorageClass, which can be either one of the default storage classes that have the provisioner set to **disk.csi.azure.com** or **file.csi.azure.com**. Or you can create a custom storage class as long as it is set to either one of those two provisioners.

+ * `volumeSnapshotClass` - Name of the volume snapshot class. For example, `custom-disk-snapshot-sc`.

+ * `startTimeStamp` - Provide a start time in the format **yyyy-mm-ddthh:mm:ssz**.

+ * `endTimeStamp` - Provide an end time in the format **yyyy-mm-ddthh:mm:ssz**.

+ ```bash

+ ./CreatePV.sh <namespace> <sourceIntreeStorageClass> <targetCSIStorageClass> <startTimestamp> <endTimestamp>

+ ```

+5. Update your application to use the new PVC.

+### Create a dynamic volume

+Using this option, you dynamically create a Persistent Volume from a Persistent Volume Claim.

+The benefits of this approach are:

+* It's less risky because all new objects are created while retaining other copies with snapshots.

+* No need to construct PVs separately and add volume name in PVC manifest.

+The following are important considerations to evaluate:

+* While this approach is less risky, it does create multiple objects that will increase your storage costs.

+* During creation of the new volume(s), your application is unavailable.

+* Deletion steps should be performed with caution. Temporary [resource locks][azure-resource-locks] can be applied to your resource group until migration is completed and your application is successfully verified.

+* Perform data validation/verification as new disks are created from snapshots.

+#### Migration

+Before proceeding, verify the following:

+* For specific workloads where data is written to memory before being written to disk, the application should be stopped and to allow in-memory data to be flushed to disk.

+* `VolumeSnapshot` class should exist as shown in the following example YAML:

+ ```yml

+ apiVersion: snapshot.storage.k8s.io/v1

+ kind: VolumeSnapshotClass

+ metadata:

+ name: custom-disk-snapshot-sc

+ driver: disk.csi.azure.com

+ deletionPolicy: Delete

+ parameters:

+ incremental: "false"

+ ```

+1. Get list of all the PVCs in a specified namespace sorted by *creationTimestamp* by running the following command. Set the namespace using the `--namespace` argument along with the actual cluster namespace.

+ ```bash

+ kubectl get pvc --namespace <namespace> --sort-by=.metadata.creationTimestamp -o custom-columns=NAME:.metadata.name,CreationTime:.metadata.creationTimestamp,StorageClass:.spec.storageClassName,Size:.spec.resources.requests.storage

+ ```

+ This step is helpful if you have a large number of PVs that need to be migrated, and you want to migrate a few at a time. Running this command enables you to identify which PVCs were created in a given time frame. When you run the *MigrateCSI.sh* script, two of the parameters are start time and end time that enable you to only migrate the PVCs during that period of time.

+2. Create a file named **MigrateToCSI.sh** and copy in the following code. The script does the following:

+ * Creates a full disk snapshot using the Azure CLI

+ * Creates `VolumesnapshotContent`

+ * Creates `VolumeSnapshot`

+ * Creates a new PVC from `VolumeSnapshot`

+ * Creates a new file with the filename `<namespace>-timestamp`, which contains a list of all old resources that needs to be cleaned up.

+ ```bash

+ #!/bin/sh

+ #kubectl get pvc -n <namespace> --sort-by=.metadata.creationTimestamp -o custom-columns=NAME:.metadata.name,CreationTime:.metadata.creationTimestamp,StorageClass:.spec.storageClassName,Size:.spec.resources.requests.storage

+ # TimeFormat 2022-04-20T13:19:56Z

+ namespace=$1

+ fileName=$namespace-$(date +%Y%m%d%H%M)

+ existingStorageClass=$2

+ storageClassNew=$3

+ volumestorageClass=$4

+ starttimestamp=$5

+ endtimestamp=$6

+ i=1

+ for pvc in $(kubectl get pvc -n $namespace | awk '{ print $1}'); do

+ # Ignore first record as it contains header

+ if [ $i -eq 1 ]; then

+ i=$((i + 1))

+ else

+ pvcCreationTime=$(kubectl get pvc $pvc -n $namespace -o jsonpath='{.metadata.creationTimestamp}')

+ if [[ $pvcCreationTime > $starttimestamp ]]; then

+ if [[ $endtimestamp > $pvcCreationTime ]]; then

+ pv="$(kubectl get pvc $pvc -n $namespace -o jsonpath='{.spec.volumeName}')"

+ reclaimPolicy="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')"

+ storageClass="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.storageClassName}')"

+ echo $pvc

+ reclaimPolicy="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')"

+ if [[ $storageClass == $existingStorageClass ]]; then

+ storageSize="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.capacity.storage}')"

+ skuName="$(kubectl get storageClass $storageClass -o jsonpath='{.reclaimPolicy}')"

+ diskURI="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.azureDisk.diskURI}')"

+ targetResourceGroup="$(cut -d'/' -f5 <<<"$diskURI")"

+ echo $diskURI

+ echo $targetResourceGroup

+ persistentVolumeReclaimPolicy="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')"

+ az snapshot create --resource-group $targetResourceGroup --name $pvc-$fileName --source "$diskURI"

+ snapshotPath=$(az snapshot list --resource-group $targetResourceGroup --query "[?name == '$pvc-$fileName'].id | [0]")

+ snapshotHandle=$(echo "$snapshotPath" | tr -d '"')

+ echo $snapshotHandle

+ sleep 10

+ # Create Restore File

+ cat <<EOF >$pvc-csi.yml

+ apiVersion: snapshot.storage.k8s.io/v1

+ kind: VolumeSnapshotContent

+ metadata:

+ name: $pvc-$fileName

+ spec:

+ deletionPolicy: 'Delete'

+ driver: 'disk.csi.azure.com'

+ volumeSnapshotClassName: $volumestorageClass

+ source:

+ snapshotHandle: $snapshotHandle

+ volumeSnapshotRef:

+ apiVersion: snapshot.storage.k8s.io/v1

+ kind: VolumeSnapshot

+ name: $pvc-$fileName

+ namespace: $1

+

+ apiVersion: snapshot.storage.k8s.io/v1

+ kind: VolumeSnapshot

+ metadata:

+ name: $pvc-$fileName

+ namespace: $1

+ spec:

+ volumeSnapshotClassName: $volumestorageClass

+ source:

+ volumeSnapshotContentName: $pvc-$fileName

+

+ apiVersion: v1

+ kind: PersistentVolumeClaim

+ metadata:

+ name: csi-$pvc

+ namespace: $1

+ spec:

+ accessModes:

+ - ReadWriteOnce

+ storageClassName: $storageClassNew

+ resources:

+ requests:

+ storage: $storageSize

+ dataSource:

+ name: $pvc-$fileName

+ kind: VolumeSnapshot

+ apiGroup: snapshot.storage.k8s.io

+

+ EOF

+ kubectl create -f $pvc-csi.yml

+ line="OLDPVC:$pvc,OLDPV:$pv,VolumeSnapshotContent:volumeSnapshotContent-$fileName,VolumeSnapshot:volumesnapshot$fileName,OLDdisk:$diskURI"

+ printf '%s\n' "$line" >>$fileName

+ fi

+ fi

+ fi

+ fi

+ done

+ ```

+3. To migrate the disk volumes, execute the script **MigrateToCSI.sh** with the following parameters:

+ * `namespace` - The cluster namespace

+ * `sourceStorageClass` - The in-tree storage driver-based StorageClass

+ * `targetCSIStorageClass` - The CSI storage driver-based StorageClass

+ * `volumeSnapshotClass` - Name of the volume snapshot class. For example, `custom-disk-snapshot-sc`.

+ * `startTimeStamp` - Provide a start time in the format **yyyy-mm-ddthh:mm:ssz**.

+ * `endTimeStamp` - Provide an end time in the format **yyyy-mm-ddthh:mm:ssz**.

+ ```bash

+ ./MigrateToCSI.sh <namespace> <sourceStorageClass> <TargetCSIstorageClass> <VolumeSnapshotClass> <startTimestamp> <endTimestamp>

+ ```

+4. Update your application to use the new PVC.

+5. Manually delete the older resources including in-tree PVC/PV, VolumeSnapshot, and VolumeSnapshotContent. Otherwise, maintaining the in-tree PVC/PC and snapshot objects will generate more cost.

+## Migrate File share volumes

+Migration from in-tree to CSI is supported by creating a static volume.

+### Migration

+1. Update the existing PV `ReclaimPolicy` from **Delete** to **Retain** by running the following command:

+ ```bash

+ kubectl patch pv pvName -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

+ ```

+ Replace **pvName** with the name of your selected PersistentVolume. Alternatively, if you want to update the reclaimPolicy for multiple PVs, create a file named **patchReclaimPVs.sh** and copy in the following code.

+ ```bash

+ # Patch the Persistent Volume in case ReclaimPolicy is Delete

+ #!/bin/sh

+ namespace=$1

+ i=1

+ for pvc in $(kubectl get pvc -n $namespace | awk '{ print $1}'); do

+ # Ignore first record as it contains header

+ if [ $i -eq 1 ]; then

+ i=$((i + 1))

+ else

+ pv="$(kubectl get pvc $pvc -n $namespace -o jsonpath='{.spec.volumeName}')"

+ reclaimPolicy="$(kubectl get pv $pv -n $namespace -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')"

+ echo "Reclaim Policy for Persistent Volume $pv is $reclaimPolicy"

+ if [[ $reclaimPolicy == "Delete" ]]; then

+ echo "Updating ReclaimPolicy for $pv to Retain"

+ kubectl patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

+ fi

+ fi

+ done

+ ```

+ Execute the script with the `namespace` parameter to specify the cluster namespace `./PatchReclaimPolicy.sh <namespace>`.

+2. Create a new Storage Class with the provisioner set to `file.csi.azure.com`, or you can use one of the default StorageClasses with the CSI file provisioner.

+3. Get the `secretName` and `shareName` from the existing *PersistentVolumes* by running the following command:

+ ```bash

+ kubectl describe pv pvName

+ ```

+4. Create a new PV using the new StorageClass, and the `shareName` and `secretName` from the in-tree PV. Create a file named *azurefile-mount-pv.yaml* and copy in the following code. Under `csi`, update `resourceGroup`, `volumeHandle`, and `shareName`. For mount options, the default value for *fileMode* and *dirMode* is *0777*.

+ The default value for `fileMode` and `dirMode` is **0777**.

+ ```yml

+ apiVersion: v1

+ kind: PersistentVolume

+ metadata:

+ name: azurefile

+ spec:

+ capacity:

+ storage: 5Gi

+ accessModes:

+ - ReadWriteMany

+ persistentVolumeReclaimPolicy: Retain

+ storageClassName: azurefile-csi

+ csi:

+ driver: file.csi.azure.com

+ readOnly: false

+ volumeHandle: unique-volumeid # make sure volumeid is unique for every identical share in the cluster

+ volumeAttributes:

+ resourceGroup: EXISTING_RESOURCE_GROUP_NAME # optional, only set this when storage account is not in the same resource group as the cluster nodes

+ shareName: aksshare

+ nodeStageSecretRef:

+ name: azure-secret

+ namespace: default

+ mountOptions:

+ - dir_mode=0777

+ - file_mode=0777

+ - uid=0

+ - gid=0

+ - mfsymlinks

+ - cache=strict

+ - nosharesock

+ - nobrl

+ ```

+5. Create a file named *azurefile-mount-pvc.yaml* file with a *PersistentVolumeClaim* that uses the *PersistentVolume* using the following code.

+ ```yml

+ apiVersion: v1

+ kind: PersistentVolumeClaim

+ metadata:

+ name: azurefile

+ spec:

+ accessModes:

+ - ReadWriteMany

+ storageClassName: azurefile-csi

+ volumeName: azurefile

+ resources:

+ requests:

+ storage: 5Gi

+ ```

+6. Use the `kubectl` command to create the *PersistentVolume*.

+ ```bash

+ kubectl apply -f azurefile-mount-pv.yaml

+ ```

+7. Use the `kubectl` command to create the *PersistentVolumeClaim*.

+ ```bash

+ kubectl apply -f azurefile-mount-pvc.yaml

+ ```

+8. Verify your *PersistentVolumeClaim* is created and bound to the *PersistentVolume* by running the following command.

+ ```bash

+ kubectl get pvc azurefile

+ ```

+ The output resembles the following:

+ ```output

+ NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

+ azurefile Bound azurefile 5Gi RWX azurefile 5s

+ ```

+9. Update your container spec to reference your *PersistentVolumeClaim* and update your pod. For example, copy the following code and create a file named *azure-files-pod.yaml*.

+ ```yml

+ ...

+ volumes:

+ - name: azure

+ persistentVolumeClaim:

+ claimName: azurefile

+ ```

+10. The pod spec can't be updated in place. Use the following `kubectl` commands to delete and then re-create the pod.

+ ```bash

+ kubectl delete pod mypod

+ ```

+ ```bash

+ kubectl apply -f azure-files-pod.yaml

+ ```

+## Next steps

+For more about storage best practices, see [Best practices for storage and backups in Azure Kubernetes Service][aks-storage-backups-best-practices].

+<!-- LINKS - internal -->

+[install-azure-cli]: /cli/azure/install-azure-cli

+[aks-rbac-cluster-admin-role]: manage-azure-rbac.md#create-role-assignments-for-users-to-access-cluster

+[azure-resource-locks]: /azure/azure-resource-manager/management/lock-resources

+[csi-driver-overview]: csi-storage-drivers.md

+[aks-storage-backups-best-practices]: operator-best-practices-storage.md

+> Starting with Kubernetes version 1.26, in-tree persistent volume types *kubernetes.io/azure-disk* and *kubernetes.io/azure-file* are deprecated and will no longer be supported. Removing these drivers following their deprecation is not planned, however you should migrate to the corresponding CSI drivers *disks.csi.azure.com* and *file.csi.azure.com*. To review the migration options for your storage classes and upgrade your cluster to use Azure Disks and Azure Files CSI drivers, see [Migrate from in-tree to CSI drivers][migrate-from-in-tree-to-csi-drivers].

+It may take several minutes to complete this action. Once it's complete, you should see in the output the status of enabling the driver on your cluster. The following example resembles the section indicating the results when enabling the Blob storage CSI driver:

+```output

+"storageProfile": {

+ "blobCsiDriver": {

+ "enabled": true

+ },

+```

+## Disable CSI storage drivers on a new or existing cluster

+To disable CSI storage drivers on a new cluster, include one of the following parameters depending on the storage system:

+* `--disable-disk-driver` allows you to disable the [Azure Disks CSI driver][azure-disk-csi].

+* `--disable-file-driver` allows you to disable the [Azure Files CSI driver][azure-files-csi].

+* `--disable-blob-driver` allows you to disable the [Azure Blob storage CSI driver][azure-blob-csi].

+* `--disable-snapshot-controller` allows you to disable the [snapshot controller][snapshot-controller].

+```azurecli

+az aks create -n myAKSCluster -g myResourceGroup --disable-disk-driver --disable-file-driver --disable-blob-driver --disable-snapshot-controller

+To disable CSI storage drivers on an existing cluster, use one of the parameters listed earlier depending on the storage system:

+```azurecli

+az aks update -n myAKSCluster -g myResourceGroup --disable-disk-driver --disable-file-driver --disable-blob-driver --disable-snapshot-controller

+```

+## Migrate custom in-tree storage classes to CSI

+If you've created in-tree driver storage classes, those storage classes continue to work since CSI migration is turned on after upgrading your cluster to 1.21.x. If you want to use CSI features you'll need to perform the migration.

+To review the migration options for your storage classes and upgrade your cluster to use Azure Disks and Azure Files CSI drivers, see [Migrate from in-tree to CSI drivers][migrate-from-in-tree-csi-drivers].

+- For more information on CSI migration, see [Kubernetes in-tree to CSI Volume Migration][csi-migration-community].

+[azure-disk-static-mount]: azure-csi-disk-storage-provision.md#mount-disk-as-a-volume

+[azure-file-static-mount]: azure-csi-files-storage-provision.md#mount-file-share-as-a-persistent-volume

+[azure-files-csi]: azure-files-csi.md

+[migrate-from-in-tree-csi-drivers]: csi-migrate-in-tree-volumes.md

+[aks-vnet-subnet]: configure-kubenet.md#create-a-virtual-network-and-subnet

+[unique-subnet]: use-multiple-node-pools.md#add-a-node-pool-with-a-unique-subnet

+[azure-disk]: ./azure-disk-csi.md

+[azure-files]: ./azure-files-csi.md

+* Use Azure CNI with Dynamic IP allocation for optimum IP utilization, and scale up to 50k application pods per cluster with one routable IP per pod. For more information, see [Configure Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in AKS][Configure Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in Azure Kubernetes Service (AKS)].

+[Configure Azure CNI networking for dynamic allocation of IPs and enhanced subnet support in Azure Kubernetes Service (AKS)]: configure-azure-cni-dynamic-ip-allocation.md

+[dynamic-disks]: azure-disk-csi.md

+[dynamic-files]: azure-files-csi.md

+> AKS follows 12 months of support for a GA Kubernetes version. To read more about our support policy for Kubernetes versioning, please read our [FAQ](https://learn.microsoft.com/azure/aks/supported-kubernetes-versions?tabs=azure-cli#faq).

+For the past release history, see [Kubernetes](https://en.wikipedia.org/wiki/Kubernetes#History).

+| 1.22 | Aug-04-21 | Sept 2021 | Dec 2021 | Dec 2022 |

+| 1.23 | Dec 2021 | Jan 2022 | Apr 2022 | Apr 2023 |

+| 1.24 | Apr-22-22 | May 2022 | Jul 2022 | Jul 2023

+| 1.25 | Aug 2022 | Oct 2022 | Dec 2022 | Dec 2023

+| 1.26 | Dec 2022 | Jan 2023 | Feb 2023 | Feb 2024

+| 1.27 | Apr 2023 | May 2023 | Jun 2023 | Jun 2024

+[operator-guide-patching]: /azure/architecture/operator-guides/aks/aks-upgrade-practices#considerations

+- The supported size range for ultra disks is between 100 and 1500.

+## Create a new cluster that can use ultra disks

+Create an AKS cluster that is able to leverage Azure ultra Disks by using the following CLI commands. Use the `--enable-ultra-ssd` flag to set the `EnableUltraSSD` feature.

+Create an AKS-managed Azure AD cluster with support for ultra disks.

+az aks create -g MyResourceGroup -n myAKSCluster -l westus2 --node-vm-size Standard_D2s_v3 --zones 1 2 --node-count 2 --enable-ultra-ssd

+## Enable ultra disks on an existing cluster

+To use ultra disks in our deployments or stateful sets you can use a [storage class for dynamic provisioning][azure-disk-volume].

+kubectl apply -f azure-ultra-disk-sc.yaml

+```

+The output from the command resembles the following example:

+```console

+kubectl apply -f azure-ultra-disk-pvc.yaml

+```

+The output from the command resembles the following example:

+```console

+kubectl apply -f nginx-ultra.yaml

+```

+The output from the command resembles the following example:

+```console

+kubectl describe pod nginx-ultra

+[azure-disk-volume]: azure-disk-csi.md

+[azure-files-pvc]: azure-files-csi.md

+This article provides an overview of Vertical Pod Autoscaler (VPA) (preview) in Azure Kubernetes Service (AKS), which is based on the open source [Kubernetes](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler) version. When configured, it automatically sets resource requests and limits on containers per workload based on past usage. VPA makes certain pods are scheduled onto nodes that have the required CPU and memory resources.

+1. Wait for the vpa-updater to launch a new hamster pod, which should take a few minutes. You can monitor the pods using the [kubectl get][kubectl-get] command.

+## Metrics server VPA throttling

+With AKS clusters version 1.24 and higher, vertical pod autoscaling is enabled for the metrics server. VPA enables you to adjust the resource limit when the metrics server is experiencing consistent CPU and memory resource constraints.

+If the metrics server throttling rate is high and the memory usage of its two pods are unbalanced, this indicates the metrics server requires more resources than the default values specified.

+To update the coefficient values, create a ConfigMap in the overlay *kube-system* namespace to override the values in the metrics server specification. Perform the following steps to update the metrics server.

+1. Create a ConfigMap file named *metrics-server-config.yaml* and copy in the following manifest.

+ ```yml

+ apiVersion: v1

+ kind: ConfigMap

+ metadata:

+ name: metrics-server-config

+ namespace: kube-system

+ labels:

+ kubernetes.io/cluster-service: "true"

+ addonmanager.kubernetes.io/mode: EnsureExists

+ data:

+ NannyConfiguration: |-

+ apiVersion: nannyconfig/v1alpha1

+ kind: NannyConfiguration

+ baseCPU: 100m

+ cpuPerNode: 1m

+ baseMemory: 100Mi

+ memoryPerNode: 8Mi

+ ```

+ In this ConfigMap example, it changes the resource limit and request to the following:

+ * cpu: (100+1n) millicore

+ * memory: (100+8n) mebibyte

+ Where *n* is the number of nodes.

+2. Create the ConfigMap using the [kubectl apply][kubectl-apply] command and specify the name of your YAML manifest:

+ ```bash

+ kubectl apply -f metrics-server-config.yaml

+ ```

+Be cautious of the *baseCPU*, *cpuPerNode*, *baseMemory*, and the *memoryPerNode* as the ConfigMap won't be validated by AKS. As a recommended practice, increase the value gradually to avoid unnecessary resource consumption. Proactively monitor resource usage when updating or creating the ConfigMap. A large number of resource requests could negatively impact the node.

+To rapidly scale application workloads in an AKS cluster, you can use virtual nodes. With virtual nodes, you have quick provisioning of pods, and only pay per second for their execution time. You don't need to wait for Kubernetes cluster autoscaler to deploy VM compute nodes to run more pods. Virtual nodes are only supported with Linux pods and nodes.

+The virtual nodes add on for AKS is based on the open source project [Virtual Kubelet][virtual-kubelet-repo].

+This article gives you an overview of the region availability and networking requirements for using virtual nodes, and the known limitations.

+All regions, where ACI supports VNET SKUs, are supported for virtual nodes deployments. For more information, see [Resource availability for Azure Container Instances in Azure regions](../container-instances/container-instances-region-availability.md).

+For available CPU and memory SKUs in each region, review [Azure Container Instances Resource availability for Azure Container Instances in Azure regions - Linux container groups](../container-instances/container-instances-region-availability.md#linux-container-groups)

+Virtual nodes enable network communication between pods that run in Azure Container Instances (ACI) and the AKS cluster. To support this communication, a virtual network subnet is created and delegated permissions are assigned. Virtual nodes only work with AKS clusters created using *advanced* networking (Azure CNI). By default, AKS clusters are created with *basic* networking (kubenet).

+Virtual nodes functionality is heavily dependent on ACI's feature set. In addition to the [quotas and limits for Azure Container Instances](../container-instances/container-instances-quotas.md), the following scenarios aren't supported with virtual nodes:

+* [DaemonSets](concepts-clusters-workloads.md#statefulsets-and-daemonsets) won't deploy pods to the virtual nodes

+* Volume mounting Azure Files share support [General-purpose V2](../storage/common/storage-account-overview.md#types-of-storage-accounts) and [General-purpose V1](../storage/common/storage-account-overview.md#types-of-storage-accounts). Follow the instructions for mounting [a volume with Azure Files share](azure-files-csi.md).

+* Using IPv6 isn't supported.

+- [Check out the Autoscale sample for virtual nodes][virtual-node-autoscale]

+> [!NOTE]

+> All requests to the API Management gateway, including those rejected by policy configurations, count toward configured rate limits, quotas, and billing limits if applied in the service tier.

+> * For the current supported features of the self-hosted gateway, ensure that you have upgraded to the latest major version of the self-hosted gateway [container image](self-hosted-gateway-overview.md#container-images).

+Managed and self-hosted gateways support all available [policies](api-management-policies.md) in policy definitions with the following exceptions.

+| Policy | Managed (Dedicated) | Managed (Consumption) | Self-hosted¹ |

+| [Dapr integration](api-management-policies.md#dapr-integration-policies) | ❌ | ❌ | ✔️ |

+| [Quota and rate limit](api-management-policies.md#access-restriction-policies) | Γ£ö∩╕Å | Γ£ö∩╕Ų | Γ£ö∩╕ų

+¹ Configured policies that aren't supported by the self-hosted gateway are skipped during policy execution.<br/>

+² The rate limit by key and quota by key policies aren't available in the Consumption tier.<br/>

+³ [!INCLUDE [api-management-self-hosted-gateway-rate-limit](../../includes/api-management-self-hosted-gateway-rate-limit.md)] [Learn more](how-to-self-hosted-gateway-on-kubernetes-in-production.md#request-throttling)

+## Request throttling

+Request throttling in a self-hosted gateway can be enabled by using the API Management [rate-limit](rate-limit-policy.md) or [rate-limit-by-key](rate-limit-by-key-policy.md) policy. Configure rate limit counts to synchronize among gateway instances across cluster nodes by exposing the following ports in the Kubernetes deployment for instance discovery:

+* Port 4290 (UDP), for the rate limiting synchronization

+* Port 4291 (UDP), for sending heartbeats to other instances

+> [!NOTE]

+> [!INCLUDE [api-management-self-hosted-gateway-rate-limit](../../includes/api-management-self-hosted-gateway-rate-limit.md)]

+| fragment-id | A string. Specifies the identifier (name) of a policy fragment created in the API Management instance. | Yes | N/A |

+### Usage notes

+- [Policy expressions](api-management-policy-expressions.md) can't be used in attribute values for this policy.

+### Usage notes

+* [!INCLUDE [api-management-self-hosted-gateway-rate-limit](../../includes/api-management-self-hosted-gateway-rate-limit.md)] [Learn more](how-to-self-hosted-gateway-on-kubernetes-in-production.md#request-throttling)

+* [!INCLUDE [api-management-self-hosted-gateway-rate-limit](../../includes/api-management-self-hosted-gateway-rate-limit.md)] [Learn more](how-to-self-hosted-gateway-on-kubernetes-in-production.md#request-throttling)

+ The domain mismatch between the browser app (`http://localhost:5000`) and remote resource (`http://<app_name>.azurewebsites.net`) is recognized by your browser as a cross-origin resource request. Also, the fact that your REST API the App Service app is not sending the `Access-Control-Allow-Origin` header, the browser has prevented cross-domain content from loading.

+You can add multiple allowed origins by running the command multiple times or by adding a comma-separate list in `--allowed-origins`. To allow all origins, use `--allowed-origins '*'`.

+## Frequently asked questions

+- [App Service CORS vs. your CORS](#app-service-cors-vs-your-cors)

+- [How do I set allowed origins to a wildcard subdomain?](#how-do-i-set-allowed-origins-to-a-wildcard-subdomain)

+- [How do I enable the ACCESS-CONTROL-ALLOW-CREDENTIALS header on the response?](#how-do-i-enable-the-access-control-allow-credentials-header-on-the-response)

+#### App Service CORS vs. your CORS

+#### How do I set allowed origins to a wildcard subdomain?

+A wildcard subdomain like `*.contoso.com` is more restrictive than the wildcard origin `*`. However, the app's CORS management page in the Azure portal doesn't let you set a wildcard subdomain as an allowed origin. However, you can do it using the Azure CLI, like so:

+```azurecli-interactive

+az webapp cors add --resource-group <group-name> --name <app-name> --allowed-origins 'https://*.contoso.com'

+```

+#### How do I enable the ACCESS-CONTROL-ALLOW-CREDENTIALS header on the response?

+If your app requires credentials such as cookies or authentication tokens to be sent, the browser may require the `ACCESS-CONTROL-ALLOW-CREDENTIALS` header on the response. To enable this in App Service, set `properties.cors.supportCredentials` to `true`.

+```azurecli-interactive

+az resource update --name web --resource-group <group-name> \

+ --namespace Microsoft.Web --resource-type config \

+ --parent sites/<app-name> --set properties.cors.supportCredentials=true

+```

+This operation is not allowed when allowed origins include the wildcard origin `'*'`. Specifying `AllowAnyOrigin` and `AllowCredentials` is an insecure configuration and can result in cross-site request forgery. To allow credentials, try replacing the wildcard origin with [wildcard subdomains](#how-do-i-set-allowed-origins-to-a-wildcard-subdomain).

+ | **Deployment slot setting** | When checked, the storage mount settings also apply to deployment slots.|

+ | **Deployment slot setting** | When checked, the storage mount settings also apply to deployment slots.|

+ | **Deployment slot setting** | When checked, the storage mount settings also apply to deployment slots.|

+- Azure Storage mounts can be configured as a virtual directory to serve static content. To configure the virtual directory, in the left navigation click **Configuration** > **Path Mappings** > **New Virtual Application or Directory**. Set the **Physical path** to the **Mount path** defined on the Azure Storage mount.

+ Title: Configure gateway-required virtual network integration for your app

+description: Integrate your app in Azure App Service with Azure virtual networks using gateway-required virtual network integration.

+# Configure gateway-required virtual network integration

+Gateway-required virtual network integration supports connecting to a virtual network in another region or to a classic virtual network. Gateway-required virtual network integration only works for Windows plans. We recommend using [regional virtual network integration](./overview-vnet-integration.md) to integrate with virtual networks.

+Gateway-required virtual network integration:

+* Enables an app to connect to only one virtual network at a time.

+* Enables up to five virtual networks to be integrated within an App Service plan.

+* Allows the same virtual network to be used by multiple apps in an App Service plan without affecting the total number that can be used by an App Service plan. If you have six apps using the same virtual network in the same App Service plan that counts as one virtual network being used.

+* SLA on the gateway can affect the overall [SLA](https://azure.microsoft.com/support/legal/sla/).

+* Enables your apps to use the DNS that the virtual network is configured with.

+* Requires a virtual network route-based gateway configured with an SSTP point-to-site VPN before it can be connected to an app.

+You can't use gateway-required virtual network integration:

+* With a virtual network connected with ExpressRoute.

+* From a Linux app.

+* From a [Windows container](./quickstart-custom-container.md).

+* To access service endpoint-secured resources.

+* To resolve App Settings referencing a network protected Key Vault.

+* With a coexistence gateway that supports both ExpressRoute and point-to-site or site-to-site VPNs.

+[Regional virtual network integration](./overview-vnet-integration.md) mitigates the above mentioned limitations.

+## Set up a gateway in your Azure virtual network

+To create a gateway:

+1. [Create the VPN gateway and subnet](../vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal.md#creategw). Select a route-based VPN type.

+1. [Set the point-to-site addresses](../vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal.md#addresspool). If the gateway isn't in the basic SKU, then IKEV2 must be disabled in the point-to-site configuration and SSTP must be selected. The point-to-site address space must be in the RFC 1918 address blocks 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16.

+If you create the gateway for use with gateway-required virtual network integration, you don't need to upload a certificate. Creating the gateway can take 30 minutes. You won't be able to integrate your app with your virtual network until the gateway is created.

+## How gateway-required virtual network integration works

+Gateway-required virtual network integration is built on top of point-to-site VPN technology. Point-to-site VPNs limit network access to the virtual machine that hosts the app. Apps are restricted to send traffic out to the internet only through hybrid connections or through virtual network integration. When your app is configured with the portal to use gateway-required virtual network integration, a complex negotiation is managed on your behalf to create and assign certificates on the gateway and the application side. The result is that the workers used to host your apps can directly connect to the virtual network gateway in the selected virtual network.

+## Access on-premises resources

+Apps can access on-premises resources by integrating with virtual networks that have site-to-site connections. If you use gateway-required virtual network integration, update your on-premises VPN gateway routes with your point-to-site address blocks. When the site-to-site VPN is first set up, the scripts used to configure it should set up routes properly. If you add the point-to-site addresses after you create your site-to-site VPN, you need to update the routes manually. Details on how to do that varies per gateway and aren't described here.

+BGP routes from on-premises won't be propagated automatically into App Service. You need to manually propagate them on the point-to-site configuration using the steps in this document [Advertise custom routes for P2S VPN clients](../vpn-gateway/vpn-gateway-p2s-advertise-custom-routes.md).

+> [!NOTE]

+> The gateway-required virtual network integration feature doesn't integrate an app with a virtual network that has an ExpressRoute gateway. Even if the ExpressRoute gateway is configured in [coexistence mode](../expressroute/expressroute-howto-coexist-resource-manager.md), the virtual network integration doesn't work. If you need to access resources through an ExpressRoute connection, use the regional virtual network integration feature or an [App Service Environment](./environment/intro.md), which runs in your virtual network.

+## Peering

+If you use gateway-required virtual network integration with peering, you need to configure a few more items. To configure peering to work with your app:

+1. Add a peering connection on the virtual network your app connects to. When you add the peering connection, enable **Allow virtual network access** and select **Allow forwarded traffic** and **Allow gateway transit**.

+1. Add a peering connection on the virtual network that's being peered to the virtual network you're connected to. When you add the peering connection on the destination virtual network, enable **Allow virtual network access** and select **Allow forwarded traffic** and **Allow remote gateways**.

+1. Go to **App Service plan** > **Networking** > **VNet integration** in the portal. Select the virtual network your app connects to. Under the routing section, add the address range of the virtual network that's peered with the virtual network your app is connected to.

+## Manage virtual network integration

+Connecting and disconnecting with a virtual network is at an app level. Operations that can affect virtual network integration across multiple apps are at the App Service plan level. From the app > **Networking** > **VNet integration** portal, you can get details on your virtual network. You can see similar information at the App Service plan level in the **App Service plan** > **Networking** > **VNet integration** portal.

+The only operation you can take in the app view of your virtual network integration instance is to disconnect your app from the virtual network it's currently connected to. To disconnect your app from a virtual network, select **Disconnect**. Your app is restarted when you disconnect from a virtual network. Disconnecting doesn't change your virtual network. The subnet or gateway isn't removed. If you then want to delete your virtual network, first disconnect your app from the virtual network and delete the resources in it, such as gateways.

+The App Service plan virtual network integration UI shows you all the virtual network integrations used by the apps in your App Service plan. To see details on each virtual network, select the virtual network you're interested in. There are two actions you can perform here for gateway-required virtual network integration:

+* **Sync network**: The sync network operation is used only for the gateway-required virtual network integration feature. Performing a sync network operation ensures that your certificates and network information are in sync. If you add or change the DNS of your virtual network, perform a sync network operation. This operation restarts any apps that use this virtual network. This operation won't work if you're using an app and a virtual network belonging to different subscriptions.

+* **Add routes**: Adding routes drives outbound traffic into your virtual network.

+The private IP assigned to the instance is exposed via the environment variable WEBSITE_PRIVATE_IP. Kudu console UI also shows the list of environment variables available to the web app. This IP is an IP from the address range of the point-to-site address pool configured on the virtual network gateway. This IP will be used by the web app to connect to the resources through the Azure virtual network.

+> [!NOTE]

+> The value of WEBSITE_PRIVATE_IP is bound to change. However, it will be an IP within the address range of the point-to-site address range, so you'll need to allow access from the entire address range.

+>

+## Gateway-required virtual network integration routing

+The routes that are defined in your virtual network are used to direct traffic into your virtual network from your app. To send more outbound traffic into the virtual network, add those address blocks here. This capability only works with gateway-required virtual network integration. Route tables don't affect your app traffic when you use gateway-required virtual network integration.

+## Gateway-required virtual network integration certificates

+When gateway-required virtual network integration is enabled, there's a required exchange of certificates to ensure the security of the connection. Along with the certificates are the DNS configuration, routes, and other similar things that describe the network.

+If certificates or network information is changed, select **Sync Network**. When you select **Sync Network**, you cause a brief outage in connectivity between your app and your virtual network. Your app isn't restarted, but the loss of connectivity could cause your site to not function properly.

+## Pricing details

+Three charges are related to the use of the gateway-required virtual network integration feature:

+* **App Service plan pricing tier charges**: Your apps need to be in a Basic, Standard, Premium, Premium v2, or Premium v3 App Service plan. For more information on those costs, see [App Service pricing](https://azure.microsoft.com/pricing/details/app-service/).

+* **Data transfer costs**: There's a charge for data egress, even if the virtual network is in the same datacenter. Those charges are described in [Data transfer pricing details](https://azure.microsoft.com/pricing/details/data-transfers/).

+* **VPN gateway costs**: There's a cost to the virtual network gateway that's required for the point-to-site VPN. For more information, see [VPN gateway pricing](https://azure.microsoft.com/pricing/details/vpn-gateway/).

+## Troubleshooting

+Many things can prevent your app from reaching a specific host and port. Most of the time it's one of these things:

+* **A firewall is in the way.** If you have a firewall in the way, you hit the TCP timeout. The TCP timeout is 21 seconds in this case. Use the **tcpping** tool to test connectivity. TCP timeouts can be caused by many things beyond firewalls, but start there.

+* **DNS isn't accessible.** The DNS timeout is 3 seconds per DNS server. If you have two DNS servers, the timeout is 6 seconds. Use nameresolver to see if DNS is working. You can't use nslookup, because that doesn't use the DNS your virtual network is configured with. If inaccessible, you could have a firewall or NSG blocking access to DNS or it could be down.

+If those items don't answer your problems, look first for things like:

+* Is the point-to-site address range in the RFC 1918 ranges (10.0.0.0-10.255.255.255 / 172.16.0.0-172.31.255.255 / 192.168.0.0-192.168.255.255)?

+* Does the gateway show as being up in the portal? If your gateway is down, then bring it back up.

+* Do certificates show as being in sync, or do you suspect that the network configuration was changed? If your certificates are out of sync or you suspect that a change was made to your virtual network configuration that wasn't synced with your ASPs, select **Sync Network**.

+* If you're going across a VPN, is the on-premises gateway configured to route traffic back up to Azure? If you can reach endpoints in your virtual network but not on-premises, check your routes.

+* Are you trying to use a coexistence gateway that supports both point to site and ExpressRoute? Coexistence gateways aren't supported with virtual network integration.

+Debugging networking issues is a challenge because you can't see what's blocking access to a specific host:port combination. Some causes include:

+* You have a firewall up on your host that prevents access to the application port from your point-to-site IP range. Crossing subnets often requires public access.

+* Your target host is down.

+* Your application is down.

+* You had the wrong IP or hostname.

+* Your application is listening on a different port than what you expected. You can match your process ID with the listening port by using "netstat -aon" on the endpoint host.

+* Your network security groups are configured in such a manner that they prevent access to your application host and port from your point-to-site IP range.

+You don't know what address your app actually uses. It could be any address in the point-to-site address range, so you need to allow access from the entire address range.

+More debug steps include:

+* Connect to a VM in your virtual network and attempt to reach your resource host:port from there. To test for TCP access, use the PowerShell command **Test-NetConnection**. The syntax is:

+

+```powershell

+Test-NetConnection hostname [optional: -Port]

+```

+* Bring up an application on a VM and test access to that host and port from the console from your app by using **tcpping**.

+### On-premises resources

+If your app can't reach a resource on-premises, check if you can reach the resource from your virtual network. Use the **Test-NetConnection** PowerShell command to check for TCP access. If your VM can't reach your on-premises resource, your VPN or ExpressRoute connection might not be configured properly.

+If your virtual network-hosted VM can reach your on-premises system but your app can't, the cause is likely one of the following reasons:

+* Your routes aren't configured with your subnet or point-to-site address ranges in your on-premises gateway.

+* Your network security groups are blocking access for your point-to-site IP range.

+* Your on-premises firewalls are blocking traffic from your point-to-site IP range.

+* You're trying to reach a non-RFC 1918 address by using the regional virtual network integration feature.

+For more information, see [virtual network integration troubleshooting guide](/troubleshoot/azure/app-service/troubleshoot-vnet-integration-apps).

+az appservice ase list-addresses -n --name $ASE_NAME -g $RESOURCE_GROUP_NAME --query allowNewPrivateEndpointConnections

+az appservice ase update --name $ASE_NAME -g $RESOURCE_GROUP_NAME --allow-incoming-ftp-connections true

+az appservice ase list-addresses -n --name $ASE_NAME -g $RESOURCE_GROUP_NAME --query ftpEnabled

+az appservice ase update --name $ASE_NAME -g $RESOURCE_GROUP_NAME --allow-remote-debugging true

+az appservice ase list-addresses -n --name $ASE_NAME -g $RESOURCE_GROUP_NAME --query remoteDebugEnabled

+> [Create an App Service Environment from a template](./how-to-create-from-template.md)

+ Title: Create an ASE with Azure Resource Manager

+> [!IMPORTANT]

+> This article is about App Service Environment v2 which is used with Isolated App Service plans. [App Service Environment v2 will be retired on 31 August 2024](https://azure.microsoft.com/updates/app-service-environment-v1-and-v2-retirement-announcement/). There's a new version of App Service Environment that is easier to use and runs on more powerful infrastructure. To learn more about the new version, start with the [Introduction to the App Service Environment](overview.md). If you're currently using App Service Environment v2, please follow the steps in [this article](migration-alternatives.md) to migrate to the new version.

+>

+When you create an ASE in the Azure portal, you can create your virtual network at the same time or choose a pre-existing virtual network to deploy into.

+To automate your ASE creation, follow they guidelines in the following sections. If you're creating an ILB ASEv2 with custom dnsSuffix (for example, `internal-contoso.com`), there are a few more things to do.

+A Resource Manager template that creates an ASE and its associated parameters file is available on GitHub for [ASEv2][quickstartasev2create].

+If you want to make an ASE, use this Resource Manager template [ASEv2][quickstartilbasecreate] example. Most of the parameters in the *azuredeploy.parameters.json* file are common to the creation of ILB ASEs and External ASEs. The following list calls out parameters of special note, or that's unique, when you create an ILB ASE with an existing subnet.

+### Parameters

+* *aseName*: This parameter defines a unique ASE name.

+With a valid TLS/SSL certificate in hand, two more preparatory steps are needed. Convert/save the TLS/SSL certificate as a .pfx file. Remember that the .pfx file must include all intermediate and root certificates. Secure it with a password.

+* *certificateThumbprint*: The certificate's thumbprint. If you retrieve this value from PowerShell (for example, `$certificate.Thumbprint` from the earlier code snippet), you can use the value as is. If you copy the value from the Windows certificate dialog box, remember to strip out the extraneous spaces. The *certificateThumbprint* should look something like AF3143EB61D43F6727842115BB7F17BBCECAECAE.

+It takes roughly 40 minutes per ASE front end to apply the change. For example, for a default-sized ASE that uses two front ends, the template takes around 1 hour and 20 minutes to complete. While the template is running, the ASE can't scale.

+[ConfigureSSL]: ../../app-service/configure-ssl-certificate.md

+ Title: Create an App Service Environment (ASE) v3 with Azure Resource Manager

+description: Learn how to create an external or ILB App Service Environment v3 by using an Azure Resource Manager template.

+# Create an App Service Environment by using an Azure Resource Manager template

+App Service Environment can be created using an Azure Resource Manager template allowing you to do repeatable deployment.

+> [!NOTE]

+> This article is about App Service Environment v3, which is used with isolated v2 App Service plans.

+## Overview

+Azure App Service Environment can be created with an internet-accessible endpoint or an endpoint on an internal address in an Azure Virtual Network. When created with an internal endpoint, that endpoint is provided by an Azure component called an internal load balancer (ILB). The App Service Environment on an internal IP address is called an ILB ASE. The App Service Environment with a public endpoint is called an External ASE.

+An ASE can be created by using the Azure portal or an Azure Resource Manager template. This article walks through the steps and syntax you need to create an External ASE or ILB ASE with Resource Manager templates. Learn [how to create an App Service Environment in Azure portal](./creation.md).

+When you create an App Service Environment in the Azure portal, you can create your virtual network at the same time or choose a pre-existing virtual network to deploy into.

+When you create an App Service Environment from a template, you must start with:

+* An Azure Virtual Network.

+* A subnet in that virtual network. We recommend a subnet size of `/24` with 256 addresses to accommodate future growth and scaling needs. After the App Service Environment is created, you can't change the size.

+* The location you want to deploy into.

+## Configuring the App Service Environment

+The basic Resource Manager template that creates an App Service Environment looks like this:

+```json

+{

+ "type": "Microsoft.Web/hostingEnvironments",

+ "apiVersion": "2022-03-01",

+ "name": "[parameters('aseName')]",

+ "location": "[resourceGroup().location]",

+ "kind": "ASEV3",

+ "properties": {

+ "internalLoadBalancingMode": "Web, Publishing",

+ "virtualNetwork": {

+ "id": "[parameters('subnetResourceId')]"

+ },

+ "networkingConfiguration": { },

+ "customDnsSuffixConfiguration": { }

+ },

+ "identity": {

+ "type": "SystemAssigned"

+ }

+}

+```

+In addition to the core properties, there are other configuration options that you can use to configure your App Service Environment.

+* *name*: Required. This parameter defines a unique App Service Environment name.

+* *virtualNetwork -> id*: Required. Specifies the resource ID of the subnet. Subnet must be empty and delegated to Microsoft.Web/hostingEnvironments

+* *internalLoadBalancingMode*: Required. In most cases, set this property to "Web, Publishing", which means both HTTP/HTTPS traffic and FTP traffic is on an internal VIP (Internal Load Balancer). If this property is set to "None", all traffic remains on the public VIP (External Load Balancer).

+* *zoneRedundant*: Optional. Defines with true/false if the App Service Environment will be deployed into Availability Zones (AZ). For more information, see [zone redundancy](./zone-redundancy.md).

+* *dedicatedHostCount*: Optional. In most cases, set this property to 0 or left out. You can set it to 2 if you want to deploy your App Service Environment with physical hardware isolation on dedicated hosts.

+* *upgradePreference*: Optional. Defines if upgrade is started automatically or a 15 day windows to start the deployment is given. Valid values are "None", "Early", "Late", "Manual". More information [about upgrade preference](./how-to-upgrade-preference.md).

+* *clusterSettings*: Optional. For more information, see [cluster settings](./app-service-app-service-environment-custom-settings.md).

+* *networkingConfiguration -> allowNewPrivateEndpointConnections*: Optional. For more information, see [networking configuration](./configure-network-settings.md#allow-new-private-endpoint-connections).

+* *networkingConfiguration -> remoteDebugEnabled*: Optional. For more information, see [networking configuration](./configure-network-settings.md#remote-debugging-access).

+* *networkingConfiguration -> ftpEnabled*: Optional. For more information, see [networking configuration](./configure-network-settings.md#ftp-access).

+* *networkingConfiguration -> inboundIpAddressOverride*: Optional. Allow you to create an App Service Environment with your own Azure Public IP address (specify the resource ID) or define a static IP for ILB deployments. This setting can't be changed after the App Service Environment is created.

+* *customDnsSuffixConfiguration*: Optional. Allows you to specify a custom domain suffix for the App Service Environment. Requires a valid certificate from a Key Vault and access using a Managed Identity. For more information about the specific parameters, see [configuration custom domain suffix](./how-to-custom-domain-suffix.md).

+### Deploying the App Service Environment

+After creating the ARM template, for example named *azuredeploy.json* and optionally a parameters file for example named *azuredeploy.parameters.json*, you can create the App Service Environment by using the Azure CLI code snippet. Change the file paths to match the Resource Manager template-file locations on your machine. Remember to supply your own value for the resource group name:

+```azurecli

+templatePath="PATH/azuredeploy.json"

+parameterPath="PATH/azuredeploy.parameters.json"

+az deployment group create --resource-group "YOUR-RG-NAME-HERE" --template-file $templatePath --parameters $parameterPath

+```

+It takes about two hours for the App Service Environment to be created.

+## Next steps

+> [!div class="nextstepaction"]

+> [Using an App Service Environment v3](./using.md)

+> [!div class="nextstepaction"]

+> [App Service Environment v3 Networking](./networking.md)

+> [!div class="nextstepaction"]

+> [Certificates in App Service Environment v3](./overview-certificates.md)

+If the certificate used for the custom domain suffix contains a Subject Alternate Name (SAN) entry for **.scm.CUSTOM-DOMAIN*, the scm site will then also be reachable from *APP-NAME.scm.CUSTOM-DOMAIN*. You can only access scm over custom domain using basic authentication. Single sign-on is only possible with the default root domain.

+Unlike earlier versions, the FTPS endpoints for your App Services on your App Service Environment v3 can only be reached using the default domain suffix.

+- Access the FTPS endpoint using a custom domain suffix.

+- **On-behalf-of (OBO)** - Make delegated access to remote resources on behalf of the user. With Azure Active Directory as the authentication provider, your App Service app can perform delegated sign-in to a remote service, such as [Microsoft Graph](/graph/overview) or a remote API app in App Service. For an end-to-end tutorial of this approach, see [Authenticate and authorize users end-to-end in Azure App Service](tutorial-auth-aad.md).

+# <a name="regional-virtual-network-integration"></a>Integrate your app with an Azure virtual network

+This article describes the Azure App Service virtual network integration feature and how to set it up with apps in [App Service](./overview.md). With [Azure virtual networks](../virtual-network/virtual-networks-overview.md), you can place many of your Azure resources in a non-internet-routable network. The App Service virtual network integration feature enables your apps to access resources in or through a virtual network.

+>[!NOTE]

+> Information about Gateway-required virtual network integration has [moved to a new location](./configure-gateway-required-vnet-integration.md).

+* The dedicated compute pricing tiers, which include the Basic, Standard, Premium, Premium v2, and Premium v3.

+* The App Service Environment, which deploys directly into your virtual network with dedicated supporting infrastructure and is using the Isolated and Isolated v2 pricing tiers.

+The virtual network integration feature is used in Azure App Service dedicated compute pricing tiers. If your app is in an [App Service Environment](./environment/overview.md), it's already integrated with a virtual network and doesn't require you to configure virtual network integration feature to reach resources in the same virtual network. For more information on all the networking features, see [App Service networking features](./networking-features.md).

+Virtual network integration gives your app access to resources in your virtual network, but it doesn't grant inbound private access to your app from the virtual network. Private site access refers to making an app accessible only from a private network, such as from within an Azure virtual network. Virtual network integration is used only to make outbound calls from your app into your virtual network. Refer to [private endpoint](./networking/private-endpoint.md) for inbound private access.

+The virtual network integration feature:

+* Requires a [supported Basic or Standard](./overview-vnet-integration.md#limitations), Premium, Premium v2, Premium v3, or Elastic Premium App Service pricing tier.

+* Supports TCP and UDP.

+* Works with App Service apps, function apps and Logic apps.

+There are some things that virtual network integration doesn't support, like:

+* Mounting a drive.

+* Windows Server Active Directory domain join.

+* NetBIOS.

+Virtual network integration supports connecting to a virtual network in the same region. Using virtual network integration enables your app to access:

+When you use virtual network integration, you can use the following Azure networking features:

+* **NAT gateway**: You can use [NAT gateway](./networking/nat-gateway-integration.md) to get a dedicated outbound IP and mitigate SNAT port exhaustion.

+Learn [how to enable virtual network integration](./configure-vnet-integration-enable.md).

+## <a name="how-regional-virtual-network-integration-works"></a> How virtual network integration works

+Apps in App Service are hosted on worker roles. Virtual network integration works by mounting virtual interfaces to the worker roles with addresses in the delegated subnet. Because the from address is in your virtual network, it can access most things in or through your virtual network like a VM in your virtual network would.

+When virtual network integration is enabled, your app makes outbound calls through your virtual network. The outbound addresses that are listed in the app properties portal are the addresses still used by your app. However, if your outbound call is to a virtual machine or private endpoint in the integration virtual network or peered virtual network, the outbound address will be an address from the integration subnet. The private IP assigned to an instance is exposed via the environment variable, WEBSITE_PRIVATE_IP.

+When all traffic routing is enabled, all outbound traffic is sent into your virtual network. If all traffic routing isn't enabled, only private traffic (RFC1918) and service endpoints configured on the integration subnet will be sent into the virtual network. Outbound traffic to the internet will be routed directly from the app.

+The feature supports two virtual interfaces per worker. Two virtual interfaces per worker mean two virtual network integrations per App Service plan. The apps in the same App Service plan can only use one of the virtual network integrations to a specific subnet. If you need an app to connect to more virtual networks or more subnets in the same virtual network, you need to create another App Service plan. The virtual interfaces used aren't resources customers have direct access to.

+## Subnet requirements

+Virtual network integration depends on a dedicated subnet. When you create a subnet, the Azure subnet consumes five IPs from the start. One address is used from the integration subnet for each plan instance. If you scale your app to four instances, then four addresses are used.

+When you scale up or down in size, the required address space is doubled for a short period of time. The scale operation affects the real, available supported instances for a given subnet size. The following table shows both the maximum available addresses per CIDR block and the effect the available addresses has on horizontal scale.

+Because subnet size can't be changed after assignment, use a subnet that's large enough to accommodate whatever scale your app might reach. To avoid any issues with subnet capacity, use a `/26` with 64 addresses. When you're creating subnets in Azure portal as part of integrating with the virtual network, a minimum size of /27 is required. If the subnet already exists before integrating through the portal, you can use a /28 subnet.

+## Permissions

+You must have at least the following Role-based access control permissions on the subnet or at a higher level to configure virtual network integration through Azure portal, CLI or when setting the `virtualNetworkSubnetId` site property directly:

+## Routes

+You can control what traffic goes through the virtual network integration. There are three types of routing to consider when you configure virtual network integration. [Application routing](#application-routing) defines what traffic is routed from your app and into the virtual network. [Configuration routing](#configuration-routing) affects operations that happen before or during startup of your app. Examples are container image pull and [app settings with Key Vault reference](./app-service-key-vault-references.md). [Network routing](#network-routing) is the ability to handle how both app and configuration traffic are routed from your virtual network and out.

+### Application routing

+> Outbound SMTP connectivity (port 25) is supported for App Service when the SMTP traffic is routed through the virtual network integration. The supportability is determined by a setting on the subscription where the virtual network is deployed. For virtual networks/subnets created before 1. August 2022 you need to initiate a temporary configuration change to the virtual network/subnet for the setting to be synchronized from the subscription. An example could be to add a temporary subnet, associate/dissociate an NSG temporarily or configure a service endpoint temporarily. For more information, see [Troubleshoot outbound SMTP connectivity problems in Azure](../virtual-network/troubleshoot-outbound-smtp-connectivity.md).

+### Configuration routing

+#### Content share

+#### Container image pull

+#### App settings using Key Vault references

+### Network routing

+Route tables and network security groups only apply to traffic routed through the virtual network integration. See [application routing](#application-routing) and [configuration routing](#configuration-routing) for details. Routes won't apply to replies from inbound app requests and inbound rules in an NSG don't apply to your app. Virtual network integration affects only outbound traffic from your app. To control inbound traffic to your app, use the [access restrictions](./overview-access-restrictions.md) feature or [private endpoints](./networking/private-endpoint.md).

+When configuring network security groups or route tables that applies to outbound traffic, you must make sure you consider your application dependencies. Application dependencies include endpoints that your app needs during runtime. Besides APIs and services the app is calling, these endpoints could also be derived endpoints like certificate revocation list (CRL) check endpoints and identity/authentication endpoint, for example Azure Active Directory. If you're using [continuous deployment in App Service](./deploy-continuous-deployment.md), you might also need to allow endpoints depending on type and language. Specifically for [Linux continuous deployment](https://github.com/microsoft/Oryx/blob/main/doc/hosts/appservice.md#network-dependencies), you'll need to allow `oryx-cdn.microsoft.io:443`.

+## Service endpoints

+Virtual network integration enables you to reach Azure services that are secured with service endpoints. To access a service endpoint-secured service, follow these steps:

+1. Configure virtual network integration with your web app to connect to a specific subnet for integration.

+## Private endpoints

+## Azure DNS private zones

+## Limitations

+There are some limitations with using virtual network integration:

+* You can't have more than two virtual network integrations per App Service plan. Multiple apps in the same App Service plan can use the same virtual network integration. Currently you can only configure the first integration through Azure portal. The second integration must be created using Azure Resource Manager templates or Azure CLI commands.

+* You can't change the subscription of an app or a plan while there's an app that's using virtual network integration.

+## Access on-premises resources

+No extra configuration is required for the virtual network integration feature to reach through your virtual network to on-premises resources. You simply need to connect your virtual network to on-premises resources by using ExpressRoute or a site-to-site VPN.

+## Peering

+If you use peering with virtual network integration, you don't need to do any more configuration.

+In the app view of your virtual network integration instance, you can disconnect your app from the virtual network and you can configure application routing. To disconnect your app from a virtual network, select **Disconnect**. Your app is restarted when you disconnect from a virtual network. Disconnecting doesn't change your virtual network. The subnet isn't removed. If you then want to delete your virtual network, first disconnect your app from the virtual network.

+The private IP assigned to the instance is exposed via the environment variable WEBSITE_PRIVATE_IP. Kudu console UI also shows the list of environment variables available to the web app. This IP is assigned from the address range of the integrated subnet. This IP will be used by the web app to connect to the resources through the Azure virtual network.

+> The value of WEBSITE_PRIVATE_IP is bound to change. However, it will be an IP within the address range of the integration subnet, so you'll need to allow access from the entire address range.

+## Pricing details

+The virtual network integration feature has no extra charge for use beyond the App Service plan pricing tier charges.

+## Troubleshooting

+The feature is easy to set up, but that doesn't mean your experience will be problem free. If you encounter problems accessing your desired endpoint, there are various steps you can take depending on what you are observing. For more information, see [virtual network integration troubleshooting guide](/troubleshoot/azure/app-service/troubleshoot-vnet-integration-apps).

+> [!NOTE]

+> * Virtual network integration isn't supported for Docker Compose scenarios in App Service.

+> * Access restrictions does not apply to traffic coming through a private endpoint.

+### Deleting the App Service plan or app before disconnecting the network integration

+If you deleted the app or the App Service plan without disconnecting the virtual network integration first, you won't be able to do any update/delete operations on the virtual network or subnet that was used for the integration with the deleted resource. A subnet delegation 'Microsoft.Web/serverFarms' will remain assigned to your subnet and will prevent the update/delete operations.

+In order to do update/delete the subnet or virtual network again, you need to re-create the virtual network integration, and then disconnect it:

+1. Re-create the App Service plan and app (it's mandatory to use the exact same web app name as before).

+1. Navigate to **Networking** on the app in Azure portal and configure the virtual network integration.

+1. After the virtual network integration is configured, select the 'Disconnect' button.

+1. Delete the App Service plan or app.

+1. Update/Delete the subnet or virtual network.

+If you still encounter issues with the virtual network integration after following these steps, contact Microsoft Support.

+> Azure DDoS Protection incurs a cost when you use the Standard SKU. Overages charges only apply if more than 100 public IPs are protected in the tenant. Ensure you delete the resources in this tutorial if you aren't using the resources in the future. For information about pricing, see [Azure DDoS Protection Pricing]( https://azure.microsoft.com/pricing/details/ddos-protection/). For more information about Azure DDoS protection, see [What is Azure DDoS Protection?](../ddos-protection/ddos-protection-overview.md).

+<!-- markdownlint-disable MD033 -->

+| **Layout**| &bullet; `latest` </br> &bullet; `2.1-preview`| `docker pull mcr.microsoft.com/azure-cognitive-services/form-recognizer/layout`|

+| **Business Card** | &bullet; `latest` </br> &bullet; `2.1-preview` |`docker pull mcr.microsoft.com/azure-cognitive-services/form-recognizer/businesscard` |

+| **ID Document** | &bullet; `latest` </br> &bullet; `2.1-preview`| `docker pull mcr.microsoft.com/azure-cognitive-services/form-recognizer/id-document`|

+| **Receipt**| &bullet; `latest` </br> &bullet; `2.1-preview`| `docker pull mcr.microsoft.com/azure-cognitive-services/form-recognizer/receipt` |

+| **Invoice**| &bullet; `latest` </br> &bullet; `2.1-preview`|`docker pull mcr.microsoft.com/azure-cognitive-services/form-recognizer/invoice` |

+| **Custom API** | &bullet; `latest` </br> &bullet; `2.1-preview`| `docker pull mcr.microsoft.com/azure-cognitive-services/form-recognizer/custom-api`|

+| **Custom Supervised**| &bullet; `latest` </br> &bullet; `2.1-preview`|`docker pull mcr.microsoft.com/azure-cognitive-services/form-recognizer/custom-supervised` |

+The following code sample is a self-contained `docker compose` example to run the Form Recognizer Layout container. With `docker compose`, you use a YAML file to configure your application's services. Then, with `docker-compose up` command, you create and start all the services from your configuration. Enter {FORM_RECOGNIZER_ENDPOINT_URI} and {{FORM_RECOGNIZER_KEY} values for your Layout container instance.

+The following code sample is a self-contained `docker compose` example to run Form Recognizer Business Card and Read containers together. With `docker compose`, you use a YAML file to configure your application's services. Then, with `docker-compose up` command, you create and start all the services from your configuration. Enter {FORM_RECOGNIZER_ENDPOINT_URI} and {FORM_RECOGNIZER_KEY} values for your Business Card container instance. Enter {COMPUTER_VISION_ENDPOINT_URI} and {COMPUTER_VISION_KEY} for your Computer Vision Read container.

+The following code sample is a self-contained `docker compose` example to run Form Recognizer ID Document and Read containers together. With `docker compose`, you use a YAML file to configure your application's services. Then, with `docker-compose up` command, you create and start all the services from your configuration. Enter {FORM_RECOGNIZER_ENDPOINT_URI} and {FORM_RECOGNIZER_KEY} values for your ID document container. Enter {COMPUTER_VISION_ENDPOINT_URI} and {COMPUTER_VISION_KEY} values for your Computer Vision Read container.

+The following code sample is a self-contained `docker compose` example to run Form Recognizer Invoice and Layout containers together. With `docker compose`, you use a YAML file to configure your application's services. Then, with `docker-compose up` command, you create and start all the services from your configuration. Enter {FORM_RECOGNIZER_ENDPOINT_URI} and {FORM_RECOGNIZER_KEY} values for your Invoice and Layout containers.

+The following code sample is a self-contained `docker compose` example to run Form Recognizer Receipt and Read containers together. With `docker compose`, you use a YAML file to configure your application's services. Then, with `docker-compose up` command, you create and start all the services from your configuration. Enter {FORM_RECOGNIZER_ENDPOINT_URI} and {FORM_RECOGNIZER_KEY} values for your Receipt container. Enter {COMPUTER_VISION_ENDPOINT_URI} and {COMPUTER_VISION_KEY} values for your Computer Vision Read container.

+In addition to the [prerequisites](#prerequisites), you'll need to do the following to process a custom document:

+ 1. Copy the file path in a convenient location, such as *Microsoft Notepad*. You'll need to add it to your **.env** file.

+ 1. Copy the file path in a convenient location, such as *Microsoft Notepad*. You'll need to add it to your **.env** file.

+* Gather a set of at least six forms of the same type. You'll use this data to train the model and test a form. You can use a [sample data set](https://go.microsoft.com/fwlink/?linkid=2090451) (download and extract *sample_data.zip*). Download the training files to the **shared** folder you created.

+2. The following code sample is a self-contained `docker compose` example to run Form Recognizer Layout, Label Tool, Custom API, and Custom Supervised containers together. With `docker compose`, you use a YAML file to configure your application's services. Then, with `docker-compose up` command, you create and start all the services from your configuration.

+## The Sample Labeling tool and Azure Container Instances (ACI)

+To learn how to use the Sample Labeling tool with an Azure Container Instance, *see*, [Deploy the Sample Labeling tool](../deploy-label-tool.md#deploy-with-azure-container-instances-aci).

+* You can open your favorite web browser and navigate to the external IP address and exposed port of the container in question. Use the listed request URLs to validate the container is running. The listed example request URLs are `http://localhost:5000`, but your specific container may vary. Keep in mind that you're navigating to your container's **External IP address** and exposed port.

+Azure Cognitive Services containers aren't licensed to run without being connected to the metering / billing endpoint. Containers must be enabled to always communicate billing information with the billing endpoint. Cognitive Services containers don't send customer data, such as the image or text that's being analyzed, to Microsoft.

+* An **Azure data science VM** for [**Windows**](/azure/machine-learning/data-science-virtual-machine/provision-vm) or [**Linux/Ubuntu**](/azure/machine-learning/data-science-virtual-machine/dsvm-ubuntu-intro) to optionally deploy a data science VM in the virtual network to test the secure connections being established.

+ If you want to alert on a status or runbook that isn't shown in the dropdown, click the **Add custom value** option next to the dimension. This action opens a dialog that allows you to specify a custom value, which hasn't emitted for that dimension recently. If you enter a value that doesn't exist for a property your alert won't be triggered. For more information, see [Job statuses](automation-runbook-execution.md#job-statuses).

+## Update Log Analytics agent to latest version

+Azure Automation [Agent-based User Hybrid Runbook Worker](automation-hybrid-runbook-worker.md) (V1) requires the [Log Analytics agent](../azure-monitor/agents/log-analytics-agent.md) (also known as MMA agent) during the installation of the Hybrid Worker. We recommend you to update the Log Analytics agent to the latest version to reduce security vulnerabilities and benefit from bug fixes.

+Log Analytics agent versions prior to [10.20.18053 (bundle) and 1.0.18053.0 (extension)](../virtual-machines/extensions/oms-windows.md#agent-and-vm-extension-version) use an older method of certificate handling, and hence it is **not recommended**. Hybrid Workers on the outdated agents will not be able to connect to Azure, and Azure Automation jobs executed by these Hybrid Workers will stop.

+You must update the Log Analytics agent to the latest version by following the below steps:

+1. Check the current version of the Log Analytics agent for your Windows Hybrid Worker: Go to the installation path - *C:\ProgramFiles\Microsoft Monitoring Agent\Agent* and right-click *HealthService.exe* to check **Properties**. The field **Product version** provides the version number of the Log Analytics agent.

+2. If your Log Analytics agent version is prior to [10.20.18053 (bundle) and 1.0.18053.0 (extension)](../virtual-machines/extensions/oms-windows.md#agent-and-vm-extension-version), upgrade to the latest version of the Windows Log Analytics agent, following these [guidelines](../azure-monitor/agents/agent-manage.md).

+> [!NOTE]

+> Any Azure Automation jobs running on the Hybrid Worker during the upgrade process might stop. Ensure that there arenΓÇÖt any jobs running or scheduled during the Log Analytics agent upgrade.

+## Update Log Analytics agent to latest version

+For Change Tracking & Inventory, machines use the [Log Analytics agent](../../azure-monitor/agents/log-analytics-agent.md) to collect data about changes to installed software, Windows services, Windows registry and files, and Linux daemons on monitored servers. Soon, Azure will no longer accept connections from older versions of the Windows Log Analytics (LA) agent, also known as the Windows Microsoft Monitoring Agent (MMA), that uses an older method for certificate handling. We recommend to upgrade your agent to the latest version as soon as possible.

+[Agents that are on version - 10.20.18053 (bundle) and 1.0.18053.0 (extension)](../../virtual-machines/extensions/oms-windows.md#agent-and-vm-extension-version) or newer aren't affected in response to this change. If youΓÇÖre on an agent prior to that, your agent will be unable to connect, and the Change Tracking & Inventory pipeline & downstream activities can stop. You can check the current LA agent version in HeartBeat table within your LA Workspace.

+Ensure to upgrade to the latest version of the Windows Log Analytics agent (MMA) following these [guidelines](../../azure-monitor/agents/agent-manage.md).

+## Connect to Azure Account

+To view all the resources within your Automation account, you must connect to your Azure account. Follow the steps to connect to Azure from Visual Studio Code:

+1. You can sign in to Azure from the Azure Automation extension or the Command Palette.

+ - To sign in from the Azure Automation extension: select **Sign in to Azure**.

+

+ Or

+ - To sign in from the Command Palette: from the menu bar, go to **View > Command Palette** and enter **Azure:Sign-in**.

+1. Follow the sign in instructions to sign in to Azure.

+ After you're connected, you will find the Azure account name on the status bar of Visual Studio Code.

+## Select subscriptions

+When you sign in for the first time, the extension loads only the default subscription resources and Automation accounts. To add or remove subscriptions, follow these steps:

+1. You can use the Command Palette or the window footer to start the subscription command.

+ - To sign in from Command Palette - from the menu bar, go to **View > Command Palette** and enter **Azure: Select Subscriptions**.

+

+ Or

+ - To sign in from window footer - In the window footer, select the segment that matches **Azure: your account**.

+1. Use the filter to find the subscriptions by name.

+1. Check or uncheck each subscription to add or remove them from the list of subscriptions shown by Azure Automation extension.

+1. Select **OK** after you have completed adding or removing the subscriptions.

+## Update Windows Log Analytics agent to latest version

+Update Management requiresΓÇ»[Log Analytics agent](../../azure-monitor/agents/log-analytics-agent.md)ΓÇ» for its functioning. We recommend you to update Windows Log Analytics agent (also known as Windows Microsoft Monitoring Agent (MMA)) to the latest version to reduce security vulnerabilities and benefit from bug fixes. Log Analytics agent versions prior to [10.20.18053 (bundle) and 1.0.18053.0 (extension)](../../virtual-machines/extensions/oms-windows.md#agent-and-vm-extension-version) use an older method of certificate handling and hence it is not recommended. Older Windows Log Analytics agents would not be able to connect to Azure and Update Management would stop working on them.

+You must update Log Analytics agent to the latest version, by following below steps:ΓÇ»

+Check the current version of Log Analytics agent for your machine:  Go to the installation path - *C:\ProgramFiles\Microsoft Monitoring Agent\Agent* and right-click on *HealthService.exe* to check **Properties**. In the **Details** tab, the field **Product version** provides version number of the Log Analytics agent.

+If your Log Analytics agent version is prior to [10.20.18053 (bundle) and 1.0.18053.0 (extension)](../../virtual-machines/extensions/oms-windows.md#agent-and-vm-extension-version), upgrade to the latest version of the Windows Log Analytics agent, following these [guidelines](../../azure-monitor/agents/agent-manage.md).ΓÇ»

+>[!NOTE]

+> During the upgrade process, update management schedules might fail. Ensure to do this when there is no planned schedule.

+## Additional network requirements