+In the sample folder, open the *authConfig.js* file. This file contains information about your Azure AD B2C identity provider. The web API app uses this information to validate the access token that the web app passes as a bearer token. Update the following properties of the app settings:

+| `AADB2C90289` | We encountered an 'invalid_client' error connecting to the identity provider. Please try again later. | Make sure the application secret is correct or it hasn't expired. Learn how to [Register apps](register-apps.md).|

+When you use the above elements, you need add them to your **UserJourneyBehaviors** element in the order specified in the table. For example, the **JourneyInsights** element must be added before (above) the **ScriptExecution** element.

+

+|Number of API connectors per tenant |20 |

+1. For **User attributes and token claims**, choose the claims and attributes that you want to collect and send from the user during sign-up. For example, select **Show more**, and then choose attributes and claims for **Country/Region**, **Display Name**, and **Postal Code**. Select **OK**.

+The Microsoft identity platform supports the [device authorization grant](https://tools.ietf.org/html/rfc8628), which allows users to sign in to input-constrained devices such as a smart TV, IoT device, or a printer. To enable this flow, the device has the user visit a webpage in a browser on another device to sign in. Once the user signs in, the device is able to get access tokens and refresh tokens as needed.

+This article describes how to program directly against the protocol in your application. When possible, we recommend you use the supported Microsoft Authentication Libraries (MSAL) instead to [acquire tokens and call secured web APIs](authentication-flows-app-scenarios.md#scenarios-and-supported-authentication-flows). You can refer to [sample apps that use MSAL](sample-v2-code.md) for examples.

+The entire device code flow is shown in the following diagram. Each step is explained throughout this article.

+The client must first check with the authentication server for a device and user code that's used to initiate authentication. The client collects this request from the `/devicecode` endpoint. In the request, the client should also include the permissions it needs to acquire from the user.

+From the moment the request is sent, the user has 15 minutes to sign in. This is the default value for `expires_in`. The request should only be made when the user has indicated they're ready to sign in.

+| `tenant` | Required | Can be `/common`, `/consumers`, or `/organizations`. It can also be the directory tenant that you want to request permission from in GUID or friendly name format. |

+After receiving the `user_code` and `verification_uri`, the client displays these to the user, instructing them to use their mobile phone or PC browser to sign in.

+If the user authenticates with a personal account, using `/common` or `/consumers`, they'll be asked to sign in again in order to transfer authentication state to the device. This is because the device is unable to access the user's cookies. They'll also be asked to consent to the permissions requested by the client. This however doesn't apply to work or school accounts used to authenticate.

+grant_type=urn:ietf:params:oauth:grant-type:device_code&client_id=6731de76-14a6-49ae-97bc-6eba6914391e&device_code=GMMhmHCXhWEzkobqIHGG_EnNYYsAkukHspeYUk9E8...

+The device code flow is a polling protocol so errors served to the client must be expected prior to completion of user authentication.

+| `authorization_declined` | The end user denied the authorization request.| Stop polling and revert to an unauthenticated state. |

+| `expired_token` | Value of `expires_in` has been exceeded and authentication is no longer possible with `device_code`. | Stop polling and revert to an unauthenticated state. |

+| `scope` | Space separated strings | If an access token was returned, this lists the scopes in which the access token is valid for. |

+| `expires_in`| int | Number of seconds the included access token is valid for. |

+A domain name is an important part of the identifier for resources in many Azure Active Directory (Azure AD) deployments. It is part of a user name or email address for a user, part of the address for a group, and is sometimes part of the app ID URI for an application. A resource in Azure AD can include a domain name that's owned by the Azure AD organization (sometimes called a tenant) that contains the resource. Only a Global Administrator can manage domains in Azure AD.

+If you change the DNS registrars, there are no other configuration tasks in Azure AD. You can continue using the domain name with Azure AD without interruption. If you use your custom domain name with Microsoft 365, Intune, or other services that rely on custom domain names in Azure AD, see the documentation for those services.

+## ForceDelete option

+## Best Practices for Domain Hygiene

+Use a reputable registrar that provides ample notifications for domain name changes, registration expiry, a grace period for expired domains, and maintains high security standards for controlling who has access to your domain name configuration and TXT records.

+Keep your domain names current with your Registrar, and verify TXT records for accuracy.

+* If you purposefully are expiring your domain name or turning over ownership to someone else (separately from your Azure AD tenant), you should delete it from your Azure AD tenant prior to expiring or transferring.

+* If you do allow your domain name to expire, if you are able to reactivate it/regain control of it, carefully review all TXT records with the registrar to ensure no tampering of your domain name took place.

+* If you can't reactivate or regain control of your domain name immediately, you should delete it from your Azure AD tenant. Dom't readd/re-verify until you are able to resolve ownership of the domain name and verify the full TXT record for correctness.

+>[!NOTE]

+> Microsoft will not allow a domain name to be verified with more than Azure AD tenant. Once you delete a domain name from your tenant, you will not be able to re-add/re-verify it with your Azure AD tenant if it is subsequently added and verified with another Azure AD tenant.

+## Frequently asked questions

+**A:** Today, certain groups like Mail-Enabled Security groups and distributed lists are provisioned by Exchange and need to be manually cleaned up in [Exchange Admin Center (EAC)](https://outlook.office365.com/ecp/). There may be lingering ProxyAddresses, which rely on the custom domain name and will need to be updated manually to another domain name.

+**A:** You can't reference the custom domain name you are trying to delete in your user account name. Ensure that the Global Administrator account is using the initial default domain name (.onmicrosoft.com) such as admin@contoso.onmicrosoft.com. Sign in with a different Global Administrator account that such as admin@contoso.onmicrosoft.com or another custom domain name like ΓÇ£fabrikam.comΓÇ¥ where the account is admin@fabrikam.com.

+**A:** The delete domain operation is an asynchronous background task that renames all references to the domain name. It may take up to 24 hours to complete. If domain deletion fails, ensure that you donΓÇÖt have:

+* The domain to be removed the set as the Primary domain of your organization

+Also note that the ForceDelete option won't work if the domain uses Federated authentication type. In that case the users/groups on the domain must be renamed or removed using the on-premises Active Directory before reattempting the domain removal.

+If you find that any of the conditions havenΓÇÖt been met, manually clean up the references, and try to delete the domain again.

+> For Windows 10, the correct format of the deviceOSVersion attribute is as follows: (device.deviceOSVersion -startsWith "10.0.1"). The formatting can be validated with the Get-MgDevice PowerShell cmdlet:

+> ```

+> Get-MgDevice -Search "displayName:YourMachineNameHere" -ConsistencyLevel eventual | Select-Object -ExpandProperty 'OperatingSystemVersion'

+> ```

+# Customer intent: As a tenant administrator, I want to enable B2B user access to on-premises apps.

+You must do the following:

+ - A PowerShell script, which is a more lightweight solution that doesn't require MIM.

+

+You can use MIM 2016 Service Pack 1, and the MIM management agent for Microsoft Graph to create the guest user objects in the on-premises directory. To learn more, see [Azure AD business-to-business (B2B) collaboration with Microsoft Identity Manager (MIM) 2016 SP1 with Azure Application Proxy](/microsoft-identity-manager/microsoft-identity-manager-2016-graph-b2b-scenario).

+- [Grant local users access to cloud apps](hybrid-on-premises-to-cloud.md)

+- [Azure Active Directory B2B collaboration for hybrid organizations](hybrid-organizations.md)

+- For an overview of Azure AD Connect, see [Integrate your on-premises directories with Azure Active Directory](../hybrid/whatis-hybrid-identity.md).

+description: Give locally managed external partners access to both local and cloud resources using the same credentials with Azure AD B2B collaboration.

+# Customer intent: As a tenant administrator, I want to enable locally-managed external partners' access to both local and cloud resources via the Azure AD B2B collaboration.

+# Grant locally managed partner accounts access to cloud resources using Azure AD B2B collaboration

+If you create accounts for your external partners in your on-premises directory (for example, you create an account with a sign-in name of "msullivan" for an external user named Maria Sullivan in your partners.contoso.com domain), you can now sync these accounts to the cloud. Specifically, you can use [Azure AD Connect](../hybrid/whatis-azure-ad-connect.md) to sync the partner accounts to the cloud, which creates a user account with UserType = Guest. This enables your partner users to access cloud resources using the same credentials as their local accounts, without giving them more access than they need.

+ For this exercise, we're adding "MDM policy - West" to the "MDM policy - All org" group. The "MDM - policy - West" group will have the same access as the "MDM policy - All org" group.

+You can remove an existing Security group from another Security group; however, removing the group also removes any inherited access for its members.

+Set-ADSyncRestrictedPermissions -ADConnectorAccountDN 'CN=ADConnectorAccount,OU=Users,DC=Contoso,DC=com' -Credential $credential

+## Use Microsoft Graph to configure application properties

+You can also configure properties of both app registrations and enterprise applications (service principals) through Microsoft Graph. These can include basic properties, permissions, and role assignments. For more information, see [Create and manage an Azure AD application using Microsoft Graph](/graph/tutorial-applications-basics#configure-other-basic-properties-for-your-app).

+- A tenant [configured](/azure/active-directory/verifiable-credentials/verifiable-credentials-configure-tenant)

+- [Request Service REST API issuance specification](issuance-request-api.md)

+You need to establish an authentication mechanism when using [Azure Container Registry (ACR)][acr-intro] with Azure Kubernetes Service (AKS). This operation is implemented as part of the Azure CLI, Azure PowerShell, and Azure portal experiences by granting the required permissions to your ACR. This article provides examples for configuring authentication between these Azure services.

+You can set up the AKS to ACR integration using the Azure CLI or Azure PowerShell. The AKS to ACR integration assigns the [**AcrPull** role][acr-pull] to the [Azure Active Directory (Azure AD) **managed identity**][aad-identity] associated with your AKS cluster.

+> This article covers automatic authentication between AKS and ACR. If you need to pull an image from a private external registry, use an [image pull secret][image-pull-secret].

+* You need to have the [**Owner**][rbac-owner], [**Azure account administrator**][rbac-classic], or [**Azure co-administrator**][rbac-classic] role on your **Azure subscription**.

+ * To avoid needing one of these roles, you can instead use an existing managed identity to authenticate ACR from AKS. For more information, see [Use an Azure managed identity to authenticate to an ACR](../container-registry/container-registry-authentication-managed-identity.md).

+* If you're using Azure CLI, this article requires that you're running Azure CLI version 2.7.0 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+* If you're using Azure PowerShell, this article requires that you're running Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+## Create a new AKS cluster with ACR integration

+You can set up AKS and ACR integration during the creation of your AKS cluster. To allow an AKS cluster to interact with ACR, an Azure AD managed identity is used.

+### Create an ACR

+If you don't already have an ACR, create one using the following command.

+#### [Azure CLI](#tab/azure-cli)

+```azurecli

+# Set this variable to the name of your ACR. The name must be globally unique.

+MYACR=myContainerRegistry

+az acr create -n $MYACR -g myContainerRegistryResourceGroup --sku basic

+```

+#### [Azure PowerShell](#tab/azure-powershell)

+```azurepowershell

+# Set this variable to the name of your ACR. The name must be globally unique.

+$MYACR = 'myContainerRegistry'

+New-AzContainerRegistry -Name $MYACR -ResourceGroupName myContainerRegistryResourceGroup -Sku Basic

+```

+### Create a new AKS cluster and integrate with an existing ACR

+If you already have an ACR, use the following command to create a new AKS cluster with ACR integration. This command allows you to authorize an existing ACR in your subscription and configures the appropriate **AcrPull** role for the managed identity. Supply valid values for your parameters below.

+#### [Azure CLI](#tab/azure-cli)

+# Set this variable to the name of your ACR. The name must be globally unique.

+# Create an AKS cluster with ACR integration.

+Alternatively, you can specify the ACR name using an ACR resource ID using the following format:

+> If you're using an ACR located in a different subscription from your AKS cluster, use the ACR *resource ID* when attaching or detaching from the cluster.

+>

+> ```azurecli

+> az aks create -n myAKSCluster -g myResourceGroup --generate-ssh-keys --attach-acr /subscriptions/<subscription-id>/resourceGroups/myContainerRegistryResourceGroup/providers/Microsoft.ContainerRegistry/registries/myContainerRegistry

+> ```

+#### [Azure PowerShell](#tab/azure-powershell)

+# Set this variable to the name of your ACR. The name must be globally unique.

+# Create an AKS cluster with ACR integration.

+### Attach an ACR to an AKS cluster

+#### [Azure CLI](#tab/azure-cli)

+Integrate an existing ACR with an existing AKS cluster using the [`--attach-acr` parameter][cli-param] and valid values for **acr-name** or **acr-resource-id**.

+# Attach using acr-name

+# Attach using acr-resource-id

+> The `az aks update --attach-acr` command uses the permissions of the user running the command to create the ACR role assignment. This role is assigned to the [kubelet][kubelet] managed identity. For more information on AKS managed identities, see [Summary of managed identities][summary-msi].

+#### [Azure PowerShell](#tab/azure-powershell)

+Integrate an existing ACR with an existing AKS cluster using the [`-AcrNameToAttach` parameter][ps-attach] and valid values for **acr-name**.

+```azurepowershell

+Set-AzAksCluster -Name myAKSCluster -ResourceGroupName myResourceGroup -AcrNameToAttach <acr-name>

+> [!NOTE]

+> Running the `Set-AzAksCluster -AcrNameToAttach` cmdlet uses the permissions of the user running the command to create the role ACR assignment. This role is assigned to the [kubelet][kubelet] managed identity. For more information on AKS managed identities, see [Summary of managed identities][summary-msi].

+### Detach an ACR from an AKS cluster

+#### [Azure CLI](#tab/azure-cli)

+Remove the integration between an ACR and an AKS cluster using the [`--detach-acr` parameter][cli-param] and valid values for **acr-name** or **acr-resource-id**.

+```azurecli

+# Detach using acr-name

+az aks update -n myAKSCluster -g myResourceGroup --detach-acr <acr-name>

+# Detach using acr-resource-id

+az aks update -n myAKSCluster -g myResourceGroup --detach-acr <acr-resource-id>

+#### [Azure PowerShell](#tab/azure-powershell)

+Remove the integration between an ACR and an AKS cluster using the [`-AcrNameToDetach` parameter][ps-detach] and valid values for **acr-name**.

+Run the following command to import an image from Docker Hub into your ACR.

+#### [Azure CLI](#tab/azure-cli)

+#### [Azure PowerShell](#tab/azure-powershell)

+Ensure you have the proper AKS credentials.

+#### [Azure CLI](#tab/azure-cli)

+#### [Azure PowerShell](#tab/azure-powershell)

+Create a file called **acr-nginx.yaml** using the sample YAML below. Replace **acr-name** with the name of your ACR.

+After creating the file, run the following deployment in your AKS cluster.

+You can monitor the deployment by running `kubectl get pods`.

+The output should show two running pods.

+* Run the [`az aks check-acr`](/cli/azure/aks#az-aks-check-acr) command to validate that the registry is accessible from the AKS cluster.

+* Learn more about [ACR monitoring](../container-registry/monitor-service.md).

+* Learn more about [ACR health](../container-registry/container-registry-check-health.md).

+[image-pull-secret]: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

+[acr-pull]: ../role-based-access-control/built-in-roles.md#acrpull

+[azure-cli-install]: /cli/azure/install-azure-cli

+[azure-powershell-install]: /powershell/azure/install-az-ps

+[acr-intro]: ../container-registry/container-registry-intro.md

+[aad-identity]: ../active-directory/managed-identities-azure-resources/overview.md

+[rbac-owner]: ../role-based-access-control/built-in-roles.md#owner

+[rbac-classic]: ../role-based-access-control/rbac-and-directory-admin-roles.md#classic-subscription-administrator-roles

+[kubelet]: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

+[ps-detach]: /powershell/module/az.aks/set-azakscluster#-acrnametodetach

+[cli-param]: /cli/azure/aks#az-aks-update-optional-parameters

+[ps-attach]: /powershell/module/az.aks/set-azakscluster#-acrnametoattach

+| `.metadata.name` | Specifies the name of the deployment. This file will run the *nginx* image from Docker Hub. |

+To create an AKS cluster with a new Managed NAT Gateway, use `--outbound-type managedNATGateway` as well as `--nat-gateway-managed-outbound-ip-count` and `--nat-gateway-idle-timeout` when running `az aks create`. The following example creates a *myresourcegroup* resource group, then creates a *natcluster* AKS cluster in *myresourcegroup* with a Managed NAT Gateway, two outbound IPs, and an idle timeout of 4 minutes.

+ --nat-gateway-idle-timeout 4

+## Supported container image sizes

+AKS doesn't set a limit on the container image size. However, it's important to understand that the larger the container image, the higher the memory demand. This could potentially exceed resource limits or the overall available memory of worker nodes. By default, memory for VM size Standard_DS2_v2 for an AKS cluster is set to 7 GiB.

+When a container image is very large (1 TiB or more), kubelet might not be able to pull it from your container registry to a node due to lack of disk space.

+* Qualys, Trivy, and Microsoft Defender for Containers are the only vulnerability scanning tools that support Mariner today.

+ - powershell.exe

+## Additional validations (optional)

+The steps defined above allow you to authenticate incoming requests for your Azure AD tenant. This allows anyone within the tenant to access the application, which is fine for many applications. However, some applications need to restrict access further by making authorization decisions. Your application code is often the best place to handle custom authorization logic. However, for common scenarios, the platform provides built-in checks that you can use to limit access.

+This section shows how to enable built-in checks using the [App Service authentication V2 API](./configure-authentication-api-version.md). Currently, the only way to configure these built-in checks is via [Azure Resource Manager templates](/azure/templates/microsoft.web/sites/config-authsettingsv2) or the [REST API](/rest/api/appservice/web-apps/update-auth-settings-v2).

+Within the API object, the Azure Active Directory identity provider configuration has a `valdation` section that can include a `defaultAuthorizationPolicy` object as in the following structure:

+```json

+{

+ "validation": {

+ "defaultAuthorizationPolicy": {

+ "allowedApplications": [],

+ "allowedPrincipals": {

+ "identities": []

+ }

+ }

+ }

+}

+```

+| Property | Description |

+||-|

+| `defaultAuthorizationPolicy` | A grouping of requirements that must be met in order to access the app. Access is granted based on a logical `AND` over each of its configured properties. When `allowedApplications` and `allowedPrincipals` are both configured, the incoming request must satisfy both requirements in order to be accepted. |

+| `allowedApplications` | An allowlist of string application **client IDs** representing the client resource that is calling into the app. When this property is configured as a nonempty array, only tokens obtained by an application specified in the list will be accepted.<br/><br/>This policy evaluates the `appid` or `azp` claim of the incoming token, which must be an access token. See the [Microsoft Identity Platform claims reference]. |

+| `allowedPrincipals` | A grouping of checks that determine if the principal represented by the incoming request may access the app. Satisfaction of `allowedPrincipals` is based on a logical `OR` over its configured properties. |

+| `identities` (under `allowedPrincipals`) | An allowlist of string **object IDs** representing users or applications that have access. When this property is configured as a nonempty array, the `allowedPrincipals` requirement can be satisfied if the user or application represented by the request is specified in the list.<br/><br/>This policy evaluates the `oid` claim of the incoming token. See the [Microsoft Identity Platform claims reference]. |

+Requests that fail these built-in checks are given an HTTP `403 Forbidden` response.

+[Microsoft Identity Platform claims reference]: ../active-directory/develop/access-tokens.md#payload-claims

+|clientIP | IP of the immediate client of Application Gateway. If another proxy fronts your application gateway, this displays the IP of that fronting proxy. |

+#### For Application Gateway Standard and WAF SKU (v1)

+|Value |Description |

+|||

+|instanceId | Application Gateway instance that served the request. |

+|clientIP | Originating IP for the request. |

+|clientPort | Originating port for the request. |

+|httpMethod | HTTP method used by the request. |

+|requestUri | URI of the received request. |

+|RequestQuery | **Server-Routed**: Backend pool instance that was sent the request.</br>**X-AzureApplicationGateway-LOG-ID**: Correlation ID used for the request. It can be used to troubleshoot traffic issues on the backend servers. </br>**SERVER-STATUS**: HTTP response code that Application Gateway received from the back end. |

+|UserAgent | User agent from the HTTP request header. |

+|httpStatus | HTTP status code returned to the client from Application Gateway. |

+|httpVersion | HTTP version of the request. |

+|receivedBytes | Size of packet received, in bytes. |

+|sentBytes| Size of packet sent, in bytes.|

+|timeTaken| Length of time (in milliseconds) that it takes for a request to be processed and its response to be sent. This is calculated as the interval from the time when Application Gateway receives the first byte of an HTTP request to the time when the response send operation finishes. It's important to note that the Time-Taken field usually includes the time that the request and response packets are traveling over the network. |

+|sslEnabled| Whether communication to the backend pools used TLS/SSL. Valid values are on and off.|

+|host| The hostname with which the request has been sent to the backend server. If backend hostname is being overridden, this name will reflect that.|

+|originalHost| The hostname with which the request was received by the Application Gateway from the client.|

+```json

+{

+ "resourceId": "/SUBSCRIPTIONS/{subscriptionId}/RESOURCEGROUPS/PEERINGTEST/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/{applicationGatewayName}",

+ "operationName": "ApplicationGatewayAccess",

+ "time": "2017-04-26T19:27:38Z",

+ "category": "ApplicationGatewayAccessLog",

+ "properties": {

+ "instanceId": "ApplicationGatewayRole_IN_0",

+ "clientIP": "191.96.249.97",

+ "clientPort": 46886,

+ "httpMethod": "GET",

+ "requestUri": "/phpmyadmin/scripts/setup.php",

+ "requestQuery": "X-AzureApplicationGateway-CACHE-HIT=0&SERVER-ROUTED=10.4.0.4&X-AzureApplicationGateway-LOG-ID=874f1f0f-6807-41c9-b7bc-f3cfa74aa0b1&SERVER-STATUS=404",

+ "userAgent": "-",

+ "httpStatus": 404,

+ "httpVersion": "HTTP/1.0",

+ "receivedBytes": 65,

+ "sentBytes": 553,

+ "timeTaken": 205,

+ "sslEnabled": "off",

+ "host": "www.contoso.com",

+ "originalHost": "www.contoso.com"

+ }

+}

+```

+Azure Automanage currently supports the following SDKs:

+- [CSharp](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/automanage/Azure.ResourceManager.Automanage)

+Here's a list of a few of the primary operations the SDKs provide:

+- Create Best Practices profile assignments

+- Create custom profile assignments

+* Add references to your App Configuration data in the *Application settings* of your App Service or Azure Functions. App Configuration offers tools to [export a collection of key-values as references](howto-import-export-data.md#export-data-to-azure-app-service) at once. For more information, see [Use App Configuration references for App Service and Azure Functions](../app-service/app-service-configuration-references.md).

+* [Export your App Configuration data](howto-import-export-data.md#export-data-to-azure-app-service) to the *Application settings* of your App Service or Azure Functions without selecting the option of export-as-reference. Export your data again every time you make new changes in App Configuration if you like your application to pick up the change.

+1. The service patches the replica node first.

+1. The patched replica cooperatively promotes itself to primary. This promotion is considered a planned failover.

+1. The former primary node reboots to take the new changes and comes back up as a replica node.

+Azure Cache for Redis supports upgrading the version of your Azure Cache for Redis from Redis 4 to Redis 6. Upgrading is similar to regular monthly maintenance. Upgrading follows the same pattern as maintenance: First, the Redis version on the replica node is updated, followed by an update to the primary node. Your client application should treat the upgrade operation exactly like a planned maintenance event.

+As a precautionary step, we recommend exporting the data from your existing Redis 4 cache and testing your client application with a Redis 6 cache in a lower environment before upgrading.

+- When you upgrade a cache in the Basic tier, it is unavailable for several minutes and results in data loss.

+If your project takes a dependency on the Application Insights SDK to do manual telemetry tracking, you may experience strange behavior if your sampling configuration differs from the sampling configuration in your function app. In such cases, use the same sampling configuration as the function app. For more information, see [Sampling in Application Insights](../azure-monitor/app/sampling.md).

+The following table details the supported features of Application Insights available for monitoring your function apps:

+| Azure Functions runtime version | 1.x | 2.x+ |

+|--|::|::|

+| | | |

+| **Automatic collection of** | | |

+| &bull; Requests | Γ£ô | Γ£ô |

+| &bull; Exceptions | Γ£ô | Γ£ô |

+| &bull; Performance Counters | Γ£ô | Γ£ô |

+| &bull; Dependencies | | |

+| &nbsp;&nbsp;&nbsp;&mdash; HTTP | | Γ£ô |

+| &nbsp;&nbsp;&nbsp;&mdash; Service Bus| | Γ£ô |

+| &nbsp;&nbsp;&nbsp;&mdash; Event Hubs | | Γ£ô |

+| &nbsp;&nbsp;&nbsp;&mdash; SQL\* | | Γ£ô |

+| | | |

+| **Supported features** | | |

+| &bull; QuickPulse/LiveMetrics | Yes | Yes |

+| &nbsp;&nbsp;&nbsp;&mdash; Secure Control Channel | | Yes |

+| &bull; Sampling | Yes | Yes |

+| &bull; Heartbeats | | Yes |

+| | | |

+| **Correlation** | | |

+| &bull; Service Bus | | Yes |

+| &bull; Event Hubs | | Yes |

+| | | |

+| **Configurable** | | |

+| &bull;[Fully configurable](#custom-telemetry-data) | | Yes |

+\* To enable the collection of SQL query string text, see [Enable SQL query collection](./configure-monitoring.md#enable-sql-query-collection).

+### Performance Counters

+Automatic collection of Performance Counters isn't supported when running on Linux.

+ 1. Open Azure portal > select your virtual machine > Open **Settings** : **Extensions + applications** from the pane on the left > 'AzureMonitorLinuxAgent'should show up with Status: 'Provisioning succeeded'

+ 2. Open Azure portal > select your data collection rule > Open **Configuration** : **Resources** from the pane on the left > You should see the virtual machine listed here.

+ 1. Open Azure portal > select your Arc-enabled server > Open **Settings** : **Extensions** from the pane on the left > 'AzureMonitorWindowsAgent'should show up with Status: 'Succeeded'

+ 3. Open Azure portal > select your data collection rule > Open **Configuration** : **Resources** from the pane on the left > You should see the Arc-enabled server listed here

+ 1. Open Azure portal > select your virtual machine > Open **Settings** : **Extensions + applications** from the pane on the left > 'AzureMonitorWindowsAgent'should show up with Status: 'Provisioning succeeded'

+ 3. Open Azure portal > select your data collection rule > Open **Configuration** : **Resources** from the pane on the left > You should see the virtual machine listed here

+1. From the pane on the left, navigate to the **Diagnostic Settings** for the virtual machine

+2. Select the **Logs** tab.

+ :::image type="content" source="./media/alerts-logic-apps/select-action-groups.png" alt-text="A screenshot showing the actions tab of the create rules page and the select action groups pane.":::

+⁽¹⁾ Name of rule as appears in smart detection Settings pane

+You can access the detections issued by smart detection both from the emails you receive, and from the smart detection pane.

+* **The smart detection pane** in Application Insights. Select **Smart detection** under the **Investigate** menu to see a list of recent detections.

+Configuring email notifications for a specific smart detection rule can be done by opening the smart detection **Settings** pane and selecting the rule, which will open the **Edit rule** pane.

+ Otherwise, open the Performance pane in Application Insights. You'll find there [Profiler](../profiler/profiler.md) data. If exceptions are thrown, you can also try the [snapshot debugger](../snapshot-debugger/snapshot-debugger.md).

+- The dashboard pin :::image type="content" source="media/app-map/image-10.png" alt-text="A screenshot displaying the dashboard pin button."::: is located next to the title bar of the Application Map pane. This button pins the map to a dashboard, along with the filters applied to it. This action can be useful for filters that are frequently interesting. As an example, the user can pin a map with "Error connector" filter applied to it, and the dashboard view will only show nodes that have errors in their HTTP calls.

+By default, the metrics are aggregated across all host machines from which the metrics were collected. To view the metrics per host, in the Chart details pane, turn on Grouping and then choose to group by CollectD-Host.

+HTTP requests data appears on the overview pane. (If it isn't there, wait a few seconds and then click Refresh.)

+Now that you've configured your project to send traces to Application Insights, you can view and search these traces in the Application Insights portal, in the [Search][diagnostic] pane.

+When a threshold is reached, a profile of the configured type and duration is gathered and uploaded. This profile is then visible within the performance pane of the associated Application Insights Portal UI.

+ :::image type="content" source="./media/java-standalone-profiler/performance-blade.png" alt-text="Screenshot of the link to open performance pane." lightbox="media/java-standalone-profiler/performance-blade.png":::

+ :::image type="content" source="./media/java-standalone-profiler/profiler-button.png" alt-text="Screenshot of the Profiler button from the Performance pane." lightbox="media/java-standalone-profiler/profiler-button.png":::

+Navigate to the functions app Overview pane and go to configurations. Under Application Settings, click "+ New application setting".

+> If you're not sure where to set the sampling rate, start at 5% (i.e., 0.05 sampling ratio) and adjust the rate based on the accuracy of the operations shown in the failures and performance panes. A higher rate generally results in higher accuracy. However, ANY sampling will affect accuracy so we recommend alerting on [OpenTelemetry metrics](#metrics), which are unaffected by sampling.

+In the past, the application monitoring telemetry data model in Application Insights was solely based on a small number of predefined types of events, such as requests, exceptions, dependency calls, page views, etc. Developers can use the SDK to either emit these events manually (by writing code that explicitly invokes the SDK) or they can rely on the automatic collection of events from auto-instrumentation. In either case, the Application Insights backend stores all collected events as logs, and the Application Insights panes in the Azure portal act as an analytical and diagnostic tool for visualizing event-based data from logs.

+

+To address the problems introduced by sampling pre-aggregated metrics are used in the SDKs. Additional details about these metrics, log-based and pre-aggregated, can be referenced in [Azure Application Insights - Azure Monitor | Microsoft Docs](./pre-aggregated-metrics-log-metrics.md#sdk-supported-pre-aggregated-metrics-table). Relevant properties of the logged data are identified and statistics extracted before sampling occurs. To avoid resource and cost issues, metrics are aggregated. The resulting aggregate data is represented by only a few metric telemetry items per minute, instead of potentially thousands of event telemetry items. These metrics calculate the 25 requests from the example and send a metric to the MDM account reporting ΓÇ£this web app processed 25 requestsΓÇ¥, but the sent request telemetry record will have an `itemCount` of 100. These pre-aggregated metrics report the correct numbers and can be relied upon when sampling affects the log-based queries results. They can be viewed on the Metrics pane of the Application Insights portal.

+Return to your application pane in the [Azure portal](https://portal.azure.com).

+See [Query Basic Logs in Azure Monitor](.//logs/basic-logs-query.md) for information on query limitations. See [Configure Basic Logs in Azure Monitor](logs/basic-logs-configure.md) for more information about Basic Logs.

+You can save on data ingestion costs by configuring certain tables in your Log Analytics workspace that you primarily use for debugging, troubleshooting, and auditing as Basic Logs. For more information, including the limitations of Basic Logs, see [Configure Basic Logs](../best-practices-cost.md#configure-basic-logs). ContainerLogV2 is the configured version of Basic Logs that Container Insights uses. ContainerLogV2 includes verbose text-based log records.

+### Onboarding from the Azure Arc-enabled Kubernetes resource pane

+2. From the resource pane on the left, select the 'Insights' item under the 'Monitoring' section.

+2. From the resource pane on the left, select the 'Extensions' item under the 'Settings' section.

+1. In the Azure portal, go to the **Storage accounts** resource pane. Select **Keys**, and take note of the storage account name and storage account key. You need this information in later steps.

+1. On the **Monitor** pane on the left, select **Metrics**.

+> [!NOTE]

+> NumActiveWorkers is supported only if YARN is installed, and the Resource Manager is running.

+- At the end of the commitment period, the workspace retains the selected commitment tier, and the workspace can be moved to Pay-As-You-Go or to a lower commitment tier at any time.

+- If a workspace is inadvertently moved into a commitment tier, contact Microsoft Support to reset the commitment period so you can move back to the Pay-As-You-Go pricing tier.

+When a Log Analytics workspace is linked to a dedicated cluster, new data ingested to the workspace, is routed to the cluster while existing data remains in the existing Log Analytics cluster. If the dedicated cluster is configured with customer-managed keys (CMK), new ingested data is encrypted with your key. The system abstracts the data location, you can query data as usual while the system performs cross-cluster queries in the background.

+A cluster can be linked to up to 1,000 workspaces. Linked workspaces can be located in the same region as the cluster. A workspace can't be linked to a cluster more than twice a month, to prevent data fragmentation.

+You need 'write' permissions to both the workspace and the cluster resource for workspace link operation:

+Other than the billing aspects, configuration of linked workspace remain, including data retention settings.

+- *Cluster* (default) -- billing is attributed to the Cluster resource

+- *Workspaces* -- billing is attributed to linked workspaces proportionally. When data volume from all linked workspaces is below Commitment Tier level, the bill for the remaining volume is attributed to the cluster

+You can unlink a workspace from a cluster at any time. The workspace pricing tier is changed to per-GB, data ingested to cluster before the unlink operation remains in the cluster, and new data to workspace get ingested to Log Analytics. You can query data as usual and the service performs cross-cluster queries seamlessly. If cluster was configured with Customer-managed key (CMK), data remains encrypted with your key and accessible, while your key and permissions to Key Vault remain.

+> [!NOT]

+> There is a limit of two link operations for a specific workspace within a month to prevent data distribution across clusters. Contact support if you reach limit.

+You need to have *write* permissions on the cluster resource.

+When deleting a cluster, you're losing access to all data in cluster, ingested from workspaces that are linked to it, or were linked previously. This operation isn't reversible. If you delete your cluster while workspaces are linked, the workspaces get automatically unlinked from the cluster before the delete, and new data to workspaces gets ingested to Log Analytics. If workspace data retention is longer than the period it was linked to the cluster, you can query workspace for the time range before the link to cluster and after the unlink, and the service performs cross-cluster queries seamlessly.

+> [!NOTE]

+> - There is a limit of seven clusters per subscription, five active plus two if were deleted in past 14 days.

+> - Cluster's name remain reserved for 14 days after deletion, and can't be used for creating a new cluster. Deleted cluster's name is released and can be reused after 14 days.

+**Example 1: Grant a user access to log data from their resources.**

+**Example 2: Grant a user access to log data from their resources and configure their resources to send logs to the workspace.**

+**Example 3: Grant a user access to log data from their resources without being able to read security events and send data.**

+**Example 4: Grant a user access to log data from their resources and read all Azure AD sign-in and read Update Management solution log data from the workspace.**

+* Unlock your cloud skills with more [Learn modules](/training/azure/).

+Returns the tenant of the user.

+There are two Azure build-in roles defined for template spec:

+- [Template Spec Reader](../../role-based-access-control//built-in-roles.md#template-spec-reader)

+- [Template Spec Contributor](../../role-based-access-control//built-in-roles.md#template-spec-contributor)

+In addition, you also need the permissions for deploying a Bicep file. See [Deploy - CLI](./deploy-cli.md#required-permissions) or [Deploy - PowerShell](./deploy-powershell.md#required-permissions).

+After you've created the template spec, users with the [Template Specs Reader](#required-permissions) role can deploy it. In addition, you also need the permissions for deploying an ARM template. See [Deploy - CLI](./deploy-cli.md#required-permissions) or [Deploy - PowerShell](./deploy-powershell.md#required-permissions).

+Returns the tenant of the user.

+description: Learn how to create forms that are displayed in the Azure portal forms. Use the form to deploying a template spec

+You can create a form that appears in the Azure portal to assist users in deploying a [template spec](template-specs.md). The form allows users to enter values that are passed as parameters to the template spec.

+The following screenshot shows a form opened in the Azure portal.

+ "defaultValue": "standard",

+ "standard",

+ "premium"

+ "type": "secureString",

+ "apiVersion": "2022-07-01",

+ "apiVersion": "2022-07-01",

+ "name": "[format('{0}/{1}', parameters('keyVaultName'), parameters('secretName'))]",

+The Azure portal provides a sandbox for creating and previewing forms. This sandbox can render a form from an existing ARM template. You'll use this default form to get started with creating a form for your template spec. For more information about the form structure, see [FormViewType](https://github.com/Azure/portaldocs/blob/main/portal-sdk/generated/dx-view-formViewType.md).

+ :::image type="content" source="./media/template-specs-create-portal-forms/deploy-template-spec-config.png" alt-text="Screenshot of form view sandbox.":::

+1. In **Package Type**, select **CustomTemplate**. Make sure you select the package type before specify deployment template.

+1. In **Deployment template (optional)**, select the key vault template you saved locally. When prompted if you want to overwrite current changes, select **Yes**. The autogenerated form is displayed in the code window. The form is editable from the portal. To customize the form, see [customize form](#customize-form).

+ If you look closely into the autogenerated form, the default title is called **Test Form View**, and there's only one step called **basics** defined.

+ ```json

+ {

+ "$schema": "https://schema.management.azure.com/schemas/2021-09-09/uiFormDefinition.schema.json",

+ "view": {

+ "kind": "Form",

+ "properties": {

+ "title": "Test Form View",

+ "steps": [

+ {

+ "name": "basics",

+ "label": "Basics",

+ "elements": [

+ ...

+ ]

+ }

+ ]

+ },

+ "outputs": {

+ ...

+ }

+ }

+ }

+ ```

+1. To see that works it without any modifications, select **Preview**.

+ :::image type="content" source="./media/template-specs-create-portal-forms/view-portal-basic.png" alt-text="Screenshot of the generated basic form.":::

+ The sandbox displays the form. It has fields for selecting a subscription, resource group, and region. It also fields for all of the parameters from the template.

+ Most of the fields are text boxes, but some fields are specific for the type of parameter. When your template includes allowed values for a parameter, the autogenerated form uses a drop-down element. The drop-down element is pre-populated with the allowed values.

+ In between the title and **Project details**, there are no tabs because the default form only has one step defined. In the **Customize form** section, you'll break the parameters into multiple tabs.

+ > [!WARNING]

+ > Don't select **Create** as it will launch a real deployment. You'll have a chance to deploy the template spec later in this tutorial.

+1. To exit from the preview, select **Cancel**.

+ ::: code language="json" source="~/azure-docs-json-samples/azure-resource-manager/ui-forms/keyvaultform.json" range="1-6" highlight="6" :::

+1. Your default form had all of the fields for your template combined into one step called **Basics**. To help users to understand the values they're providing, divide the form into steps. Each step contains fields related to a logical part of the solution to deploy.

+ Find the step labeled **Basics**. You'll keep this step but add steps below it. The new steps will focus on configuring the key vault, setting user permissions, and specifying the secret. Make sure you add a comma after the basics step.

+ ::: code language="json" source="~/azure-docs-json-samples/azure-resource-manager/ui-forms/steps.json" highlight="15-32" :::

+ > [!IMPORTANT]

+ > Properties in the form are case-sensitive. Make sure you use the casing shown in the examples.

+ :::image type="content" source="./media/template-specs-create-portal-forms/view-steps.png" alt-text="Screenshot of form steps.":::

+ ```json

+ {

+ "name": "secret",

+ "label": "Secret",

+ "elements": [

+ {

+ "name": "secretName",

+ "type": "Microsoft.Common.TextBox",

+ "label": "Secret Name",

+ "defaultValue": "",

+ "toolTip": "Specifies the name of the secret that you want to create.",

+ "constraints": {

+ "required": true,

+ "regex": "",

+ "validationMessage": ""

+ },

+ "visible": true

+ },

+ {

+ "name": "secretValue",

+ "type": "Microsoft.Common.PasswordBox",

+ "label": {

+ "password": "Secret Value",

+ "confirmPassword": "Confirm password"

+ },

+ "toolTip": "Specifies the value of the secret that you want to create.",

+ "constraints": {

+ "required": true,

+ "regex": "",

+ "validationMessage": ""

+ },

+ "options": {

+ "hideConfirmation": true

+ },

+ "visible": true

+ }

+ ]

+ }

+ ```

+ ```json

+ "outputs": {

+ "parameters": {

+ ...

+ "secretName": "[steps('secret').secretName]",

+ "secretValue": "[steps('secret').secretValue]"

+ }

+ ```

+ ::: code language="json" source="~/azure-docs-json-samples/azure-resource-manager/ui-forms/keyvaultform.json" :::

+1. Save this file locally with the name **keyvaultform.json**.

+## Required permissions

+There are two Azure build-in roles defined for template spec:

+- [Template Spec Reader](../../role-based-access-control//built-in-roles.md#template-spec-reader)

+- [Template Spec Contributor](../../role-based-access-control//built-in-roles.md#template-spec-contributor)

+In addition, you also need the permissions for deploying a Bicep file. See [Deploy - CLI](./deploy-cli.md#required-permissions) or [Deploy - PowerShell](./deploy-powershell.md#required-permissions).

+After you've created the template spec, users with the [template spec reader](#required-permissions) role can deploy it. In addition, you also need the permissions for deploying an ARM template. See [Deploy - CLI](./deploy-cli.md#required-permissions) or [Deploy - PowerShell](./deploy-powershell.md#required-permissions).

+The logic apps that you create in this article, contain one flow per app. The second section (**Create a new logic app of type consumption**) explains how to connect the two. The second flow stands alone and is triggered by the first one (the section with the callback URL).

+This section describes how to set up the first ("file upload") flow. The first flow is triggered when a blob is added or modified in an Azure Storage account. It uploads the new file to Azure Video Indexer with a callback URL to send a notification once the indexing operation completes.

+1. Create the <a href="https://portal.azure.com/#create/Microsoft.LogicApp" target="_blank">Logic App</a>. We create a Logic App in the same region as the Azure Video Indexer region (recommended but not required). We call the logic app `UploadIndexVideosApp`.

+ > Before moving to the next step, set up the right permission between the Logic app and the Azure Video Indexer account.

+## Create a new logic app of type consumption

+ Title: Send your Azure VMware Solution logs to Log Analytics

+description: Learn about sending logs to log analytics.

+# Send your Azure VMware Solution logs to Log Analytics

+This article shows you how to send Azure VMware Solution logs to Azure Monitor Log Analytics. You can send logs from your AVS private cloud to your Log Analytics workspace, allowing you to take advantage of the Log Analytics feature set, including:

+ΓÇó Powerful querying capabilities with Kusto Query Language (KQL)

+ΓÇó Interactive report-creation capability based on your data, using Workbooks

+...without having to get your logs out of the Microsoft ecosystem!

+In the rest of this article, weΓÇÖll show you how easy it is to make this happen.

+## How to set up Log Analytics

+A Log Analytics workspace:

+ΓÇó Contains your AVS private cloud logs.

+ΓÇó Is the workspace from which you can take desired actions, such as querying for logs.

+In this section, youΓÇÖll:

+ΓÇó Configure a Log Analytics workspace

+ΓÇó Create a diagnostic setting in your private cloud to send your logs to this workspace

+### Create a resource

+1. In the Azure portal, go to **Create a resource**.

+2. Search for ΓÇ£Log Analytics WorkspaceΓÇ¥ and click **Create** -> **Log Analytics Workspace**.

+### Set up your workspace

+1. Enter the Subscription you intend to use, the Resource Group thatΓÇÖll house this workspace. Give it a name and select a region.

+1. Click **Review** + **Create**.

+### Add a diagnostic setting

+Next, we add a diagnostic setting in your AVS private cloud, so it knows where to send your logs to.

+1. Click your AVS private cloud.

+Go to Diagnostic settings on the left-hand menu under Monitoring.

+Select **Add diagnostic setting**.

+2. Give your diagnostic setting a name.

+Select the log categories you are interested in sending to your Log Analytics workspace.

+3. Make sure to select the checkbox next to **Send to Log Analytics workspace**.

+Select the Subscription your Log Analytics workspace lives in and the Log Analytics workspace.

+Click **Save** on the top left.

+At this point, your Log Analytics workspace has been successfully configured to receive logs from your AVS private cloud.

+## Search and analyze logs using Kusto

+Now that youΓÇÖve successfully configured your logs to go to your Log Analytics workspace, you can use that data to gain meaningful insights with Log AnalyticsΓÇÖ search feature.

+Log Analytics uses a language called the Kusto Query Language (or Kusto) to search through your logs.

+For more information, see

+[Data analysis in Azure Data Explorer with Kusto Query Language](/training/paths/data-analysis-data-explorer-kusto-query-language/).

+- You want to build your own subprotocol or use existing advanced sub-protocols over WebSocket (for example, [GraphQL subscriptions over WebSocket](https://github.com/Azure/azure-webpubsub/tree/main/experimental/sdk/webpubsub-graphql-subscribe)).

+**Cross Zonal Restore** | Allows you to restore Azure Virtual Machines or disks pinned to any zone to different available zones (as per the Azure RBAC capabilities) from restore points. <br><br> You can trigger Cross Zonal Restore for managed virtual machines only. <br><br> Cross Zonal Restore is supported for [Restore with Managed System Identities (MSI)](#restore-vms-with-managed-identities). <br><br> Cross Zonal Restore supports restore of an Azure zone pinned/non-zone pinned VM from a vault with Zonal-redundant storage (ZRS) enabled. Learn [how to set Storage Redundancy](backup-create-rs-vault.md#set-storage-redundancy). <br><br> It's supported to restore an Azure zone pinned VM only from a [vault with Cross Region Restore (CRR)](backup-create-rs-vault.md#set-storage-redundancy) (if the secondary region supports zones) or Zone Redundant Storage (ZRS) enabled. <br><br> Cross Zonal Restore is supported from [secondary regions](#restore-in-secondary-region). <br><br> It's unsupported from [snapshots](backup-azure-vms-introduction.md#snapshot-creation) restore point. <br><br> It's unsupported for [Encrypted Azure VMs and Trusted Launch VMs](backup-azure-vms-introduction.md#encryption-of-azure-vm-backups).

+1. Choose the required zone from the **Availability Zone** drop-down list to restore an Azure VM pinned to any zone to a different zone.

+ Azure Backup now supports Cross Zonal Restore (CZR), you can now restore an Azure VM from the default zone to any available zones. Default zone is the zone in which Azure VM is running.

+ The following screenshot lists all zones that enable you to restore Azure VM to another zone.

+ :::image type="content" source="./media/backup-azure-arm-restore-vms/azure-virtual-machine-cross-zonal-restore.png" alt-text="Screenshot showing you how to select an available zone for VM restore.":::

+ >[!Note]

+ >Azure Backup supports CZR only for vaults with ZRS or CRR redundancy.

+1. Choose the required zone from the **Availability Zone** drop-down list to restore the VM disks to a different zone.

+ Azure Backup now supports Cross Zonal Restore (CZR). Like Azure VM, you can now restore Azure VM disks from the default zone to any available zones. Default zone is the zone in which the VM disks reside.

+ >[!Note]

+ >Azure Backup supports CZR only for vaults with ZRS or CRR redundancy.

+ Title: Overview of Backup center for Azure Backup

+# Overview of Backup center for Azure Backup

+Backup center provides a *single unified management experience* in Azure for enterprises to govern, monitor, operate, and analyze backups at scale. So, it's consistent with Azure's native management experiences.

+In this article, you'll learn about:

+> [!div class="checklist"]

+> - Key benefits

+> - Supported scenarios

+> - Get started

+> - Access community resources on Community Hub

+## Key benefits

+- **Single pane of glass to manage backups**: Backup center is designed to function well across a large and distributed Azure environment. You can use Backup center to efficiently manage backups spanning multiple workload types, vaults, subscriptions, regions, and [Azure Lighthouse](../lighthouse/overview.md) tenants.

+- **Datasource-centric management**: Backup center provides views and filters that are centered on the datasources that you're backing up (for example, VMs and databases). This allows a resource owner or a backup admin to monitor and operate backups of items without needing to focus on which vault an item is backed up to. A key feature of this design is the ability to filter views by datasource-specific properties, such as datasource subscription, datasource resource group, and datasource tags. For example, if your organization follows a practice of assigning different tags to VMs belonging to different departments, you can use Backup center to filter backup information based on the tags of the underlying VMs being backed up without needing to focus on the tag of the vault.

+- **Connected experiences**: Backup center provides native integrations to existing Azure services that enable management at scale. For example, Backup center uses the [Azure Policy](../governance/policy/overview.md) experience to help you govern your backups. It also leverages [Azure workbooks](../azure-monitor/visualize/workbooks-overview.md) and [Azure Monitor Logs](../azure-monitor/logs/data-platform-logs.md) to help you view detailed reports on backups. So, you don't need to learn any new principles to use the varied features that the Backup center offers. You can also [discover community resources from the Backup center](#access-community-resources-on-community-hub).

+Backup center is currently supported for:

+- Azure VM backup

+- SQL in Azure VM backup

+- SAP HANA on Azure VM backup

+- Azure Files backup

+- Azure Blobs backup

+- Azure Managed Disks backup

+- Azure Database for PostgreSQL Server backup

+Learn more about [supported and unsupported scenarios](backup-center-support-matrix.md).

+On the **Overview** blade, two tiles appear ΓÇô **Jobs** and **Backup instances**.

+On the **Jobs** tile, you get a summarized view of all backup and restore related jobs that were triggered across your backup estate in the last 24 hours.

+- You can view information on the number of jobs that have completed, failed, and are in-progress.

+- Select any of the numbers in this tile allows you to view more information on jobs for a particular datasource type, operation type, and status.

+On the **Backup Instances** tile, you get a summarized view of all backup instances across your backup estate. For example, you can see the number of backup instances that are in soft-deleted state compared to the number of instances that are still configured for protection.

+- Select any of the numbers in this tile allows you to view more information on backup instances for a particular datasource type and protection state.

+- You can also view all backup instances whose underlying datasource isn't found (the datasource might be deleted, or you may not have access to the datasource).

+See the [next steps](#next-steps) to understand the different capabilities that Backup center provides, and how you can use these capabilities to manage your backup estate efficiently.

+## Access community resources on Community Hub

+You can use Backup center to access various community resources useful for a backup admin or operator.

+To access the Community Hub, navigate to the Backup center in the Azure portal and select the **Community** menu item.

+Some of the resources available via the Community Hub are:

+- **Microsoft Q&A**: You can use this forum to ask and discover questions about various product features and obtain guidance from the community.

+- **Feature Requests**: You can navigate to UserVoice and file feature requests.

+- **Samples for automated deployments**: Using the Community Hub, you can discover sample Azure Resource Manager (ARM) templates and Azure Policies that you can use out of the box. You can also find sample PowerShell Scripts, CLI commands, and Microsoft Database Backup scripts.

+Back up the VM to a different region or subscription |Not supported.<br><br>For successful backup, virtual machines must be in the same subscription as the vault for backup.

+Multiple Backups Per Day | Supported (in preview), using *Enhanced policy* (in preview). <br><br> For hourly backup, the minimum RPO is 4 hours and the maximum is 24 hours. You can set the backup schedule to 4, 6, 8, 12, and 24 hours respectively. Learn how to [back up an Azure VM using Enhanced policy](backup-azure-vms-enhanced-policy.md).

+<a name="backup-azure-cross-zonal-restore">Restore across zone</a> | [Cross Zonal Restore](backup-azure-arm-restore-vms.md#restore-options) is now supported in Azure VMs.

+Restore of Zone-pinned VMs | Supported (where [availability zones](https://azure.microsoft.com/global-infrastructure/availability-zones/) are available).<br/><br/>Azure Backup now supports [restoring Azure VMs to a any available zones](backup-azure-arm-restore-vms.md#restore-options) other that the zone that's pinned in VMs. This enables you to restore VMs when the primary zone is unavailable.d

+<a name="tvm-backup">Trusted Launch VM</a> | Backup supported. <br><br> Backup of Trusted Launch VM is supported through [Enhanced policy](backup-azure-vms-enhanced-policy.md). You can enable backup through [Recovery Services vault](./backup-azure-arm-vms-prepare.md), [VM Manage blade](./backup-during-vm-creation.md#start-a-backup-after-creating-the-vm), and [Create VM blade](backup-during-vm-creation.md#create-a-vm-with-backup-configured). <br><br> **Feature details** <br><br> - Backup is supported in all regions where Trusted Launch VM is available. <br><br> - Configurations of Backup, Alerts, and Monitoring for Trusted Launch VM are currently not supported through Backup center. <br><br> - Migration of an existing [Generation 2](../virtual-machines/generation-2.md) VM (protected with Azure Backup) to Trusted Launch VM is currently not supported. Learn how to [create a Trusted Launch VM](../virtual-machines/trusted-launch-portal.md?tabs=portal#deploy-a-trusted-launch-vm).

+**Limitations**

+* This feature is not supported on Cloud Shell.

+* Shareable Links isn't currently supported for peered VNets that aren't in the same subscription.

+* Shareable Links isn't currently supported for peered VNets that aren't in the same region.

+* Shareable Links isn't supported for national clouds during preview.

+ Title: Batch synthesis API (Preview) for text to speech - Speech service

+description: Learn how to use the batch synthesis API for asynchronous synthesis of long-form text to speech.

+# Batch synthesis API (Preview) for text to speech

+The Batch synthesis API (Preview) can synthesize a large volume of text input (long and short) asynchronously. Publishers and audio content platforms can create long audio content in a batch. For example: audio books, news articles, and documents. The batch synthesis API can create synthesized audio longer than 10 minutes.

+> [!IMPORTANT]

+> The Batch synthesis API is currently in public preview. Once it's generally available, the Long Audio API will be deprecated. For more information, see [Migrate to batch synthesis API](migrate-to-batch-synthesis.md).

+The batch synthesis API is asynchronous and doesn't return synthesized audio in real time. You submit text files to be synthesized, poll for the status, and download the audio output when the status indicates success. The text inputs must be plain text or [Speech Synthesis Markup Language (SSML)](speech-synthesis-markup.md) text.

+This diagram provides a high-level overview of the workflow.

+

+> [!TIP]

+> You can also use the [Speech SDK](speech-sdk.md) to create synthesized audio longer than 10 minutes by iterating over the text and synthesizing it in chunks.

+You can use the following REST API operations for batch synthesis:

+| Operation | Method | REST API call |

+| - | -- | |

+| Create batch synthesis | `POST` | texttospeech/3.1-preview1/batchsynthesis |

+| Get batch synthesis | `GET` | texttospeech/3.1-preview1/batchsynthesis/{id} |

+| List batch synthesis | `GET` | texttospeech/3.1-preview1/batchsynthesis |

+| Delete batch synthesis | `DELETE` | texttospeech/3.1-preview1/batchsynthesis/{id} |

+## Create batch synthesis

+To submit a batch synthesis request, construct the HTTP POST request body according to the following instructions:

+- Set the required `textType` property.

+- If the `textType` property is set to "PlainText", then you must also set the `voice` property in the `synthesisConfig`. In the example below, the `textType` is set to "SSML", so the `speechSynthesis` isn't set.

+- Set the required `displayName` property. Choose a name that you can refer to later. The display name doesn't have to be unique.

+- Optionally you can set the `description`, `timeToLive`, and other properties. For more information, see [batch synthesis properties](#batch-synthesis-properties).

+> [!NOTE]

+> The maximum JSON payload size that will be accepted is 500 kilobytes. Each Speech resource can have up to 200 batch synthesis jobs that are running concurrently.

+Make an HTTP POST request using the URI as shown in the following example. Replace `YourSpeechKey` with your Speech resource key, replace `YourSpeechRegion` with your Speech resource region, and set the request body properties as previously described.

+```azurecli-interactive

+curl -v -X POST -H "Ocp-Apim-Subscription-Key: YourSpeechKey" -H "Content-Type: application/json" -d '{

+ "displayName": "batch synthesis sample",

+ "description": "my ssml test",

+ "textType": "SSML",

+ "inputs": [

+ {

+ "text": "<speak version='\''1.0'\'' xml:lang='\''en-US'\''>

+ <voice xml:lang='\''en-US'\'' xml:gender='\''Female'\'' name='\''en-US-JennyNeural'\''>

+ The rainbow has seven colors.

+ </voice>

+ </speak>",

+ },

+ ],

+ "properties": {

+ "outputFormat": "riff-24khz-16bit-mono-pcm",

+ "wordBoundaryEnabled": false,

+ "sentenceBoundaryEnabled": false,

+ "concatenateResult": false,

+ "decompressOutputFiles": false

+ },

+}' "https://YourSpeechRegion.customvoice.api.speech.microsoft.com/api/texttospeech/3.1-preview1/batchsynthesis"

+```

+You should receive a response body in the following format:

+```json

+{

+ "textType": "SSML",

+ "synthesisConfig": {},

+ "customVoices": {},

+ "properties": {

+ "timeToLive": "P31D",

+ "outputFormat": "riff-24khz-16bit-mono-pcm",

+ "concatenateResult": false,

+ "decompressOutputFiles": false,

+ "wordBoundaryEnabled": false,

+ "sentenceBoundaryEnabled": false

+ },

+ "lastActionDateTime": "2022-11-16T15:07:04.121Z",

+ "status": "NotStarted",

+ "id": "1e2e0fe8-e403-417c-a382-b55eb2ea943d",

+ "createdDateTime": "2022-11-16T15:07:04.121Z",

+ "displayName": "batch synthesis sample",

+ "description": "my ssml test"

+}

+```

+The `status` property should progress from `NotStarted` status, to `Running`, and finally to `Succeeded` or `Failed`. You can call the [GET batch synthesis API](#get-batch-synthesis) periodically until the returned status is `Succeeded` or `Failed`.

+## Get batch synthesis

+To get the status of the batch synthesis job, make an HTTP GET request using the URI as shown in the following example. Replace `YourSynthesisId` with your batch synthesis ID, replace `YourSpeechKey` with your Speech resource key, and replace `YourSpeechRegion` with your Speech resource region.

+```azurecli-interactive

+curl -v -X GET "https://YourSpeechRegion.customvoice.api.speech.microsoft.com/api/texttospeech/3.1-preview1/batchsynthesis/YourSynthesisId" -H "Ocp-Apim-Subscription-Key: YourSpeechKey"

+```

+You should receive a response body in the following format:

+```json

+{

+ "textType": "SSML",

+ "synthesisConfig": {},

+ "customVoices": {},

+ "properties": {

+ "audioSize": 100000,

+ "durationInTicks": 31250000,

+ "succeededAudioCount": 1,

+ "failedAudioCount": 0,

+ "duration": "PT3.125S",

+ "billingDetails": {

+ "customNeural": 0,

+ "neural": 33

+ },

+ "timeToLive": "P31D",

+ "outputFormat": "riff-24khz-16bit-mono-pcm",

+ "concatenateResult": false,

+ "decompressOutputFiles": false,

+ "wordBoundaryEnabled": false,

+ "sentenceBoundaryEnabled": false

+ },

+ "outputs": {

+ "result": "https://cvoiceprodeus.blob.core.windows.net/batch-synthesis-output/41b83de2-380d-45dc-91af-722b68cfdc8e/results.zip?SAS_Token"

+ },

+ "lastActionDateTime": "2022-11-05T14:00:32.523Z",

+ "status": "Succeeded",

+ "id": "41b83de2-380d-45dc-91af-722b68cfdc8e",

+ "createdDateTime": "2022-11-05T14:00:31.523Z",

+ "displayName": "batch synthesis sample",

+ "description": "my test"

+ }

+```

+From `outputs.result`, you can download a ZIP file that contains the audio (such as `0001.wav`), summary, and debug details. For more information, see [batch synthesis results](#batch-synthesis-results).

+## List batch synthesis

+To list all batch synthesis jobs for the Speech resource, make an HTTP GET request using the URI as shown in the following example. Replace `YourSpeechKey` with your Speech resource key and replace `YourSpeechRegion` with your Speech resource region. Optionally, you can set the `skip` and `top` (page size) query parameters in URL. The default value for `skip` is 0 and the default value for `top` is 100.

+```azurecli-interactive

+curl -v -X GET "https://YourSpeechRegion.customvoice.api.speech.microsoft.com/api/texttospeech/3.1-preview1/batchsynthesis?skip=0&top=2" -H "Ocp-Apim-Subscription-Key: YourSpeechKey"

+```

+You should receive a response body in the following format:

+```json

+{

+ "values": [

+ {

+ "textType": "SSML",

+ "synthesisConfig": {},

+ "customVoices": {},

+ "properties": {

+ "audioSize": 100000,

+ "durationInTicks": 31250000,

+ "succeededAudioCount": 1,

+ "failedAudioCount": 0,

+ "duration": "PT3.125S",

+ "billingDetails": {

+ "customNeural": 0,

+ "neural": 33

+ },

+ "timeToLive": "P31D",

+ "outputFormat": "riff-24khz-16bit-mono-pcm",

+ "concatenateResult": false,

+ "decompressOutputFiles": false,

+ "wordBoundaryEnabled": false,

+ "sentenceBoundaryEnabled": false

+ },

+ "outputs": {

+ "result": "https://cvoiceprodeus.blob.core.windows.net/batch-synthesis-output/41b83de2-380d-45dc-91af-722b68cfdc8e/results.zip?SAS_Token"

+ },

+ "lastActionDateTime": "2022-11-05T14:00:32.523Z",

+ "status": "Succeeded",

+ "id": "41b83de2-380d-45dc-91af-722b68cfdc8e",

+ "createdDateTime": "2022-11-05T14:00:31.523Z",

+ "displayName": "batch synthesis sample",

+ "description": "my test"

+ }

+ {

+ "textType": "PlainText",

+ "synthesisConfig": {

+ "voice": "en-US-JennyNeural",

+ "style": "chat",

+ "rate": "+30.00%",

+ "pitch": "x-high",

+ "volume": "80"

+ },

+ "customVoices": {},

+ "properties": {

+ "audioSize": 79384,

+ "durationInTicks": 24800000,

+ "succeededAudioCount": 1,

+ "failedAudioCount": 0,

+ "duration": "PT2.48S",

+ "billingDetails": {

+ "customNeural": 0,

+ "neural": 33

+ },

+ "timeToLive": "P31D",

+ "outputFormat": "riff-24khz-16bit-mono-pcm",

+ "concatenateResult": false,

+ "decompressOutputFiles": false,

+ "wordBoundaryEnabled": false,

+ "sentenceBoundaryEnabled": false

+ },

+ "outputs": {

+ "result": "https://cvoiceprodeus.blob.core.windows.net/batch-synthesis-output/38e249bf-2607-4236-930b-82f6724048d8/results.zip?SAS_Token"

+ },

+ "lastActionDateTime": "2022-11-05T18:52:23.210Z",

+ "status": "Succeeded",

+ "id": "38e249bf-2607-4236-930b-82f6724048d8",

+ "createdDateTime": "2022-11-05T18:52:22.807Z",

+ "displayName": "batch synthesis sample",

+ "description": "my test"

+ },

+ ],

+ // The next page link of the list of batch synthesis.

+ "@nextLink": "https://{region}.customvoice.api.speech.microsoft.com/api/texttospeech/3.1-preview1/batchsynthesis?skip=0&top=2"

+}

+```

+From `outputs.result`, you can download a ZIP file that contains the audio (such as `0001.wav`), summary, and debug details. For more information, see [batch synthesis results](#batch-synthesis-results).

+The `values` property in the json response lists your synthesis requests. The list is paginated, with a maximum page size of 100. The `"@nextLink"` property is provided as needed to get the next page of the paginated list.

+## Delete batch synthesis

+Delete the batch synthesis job history after you retrieved the audio output results. The Speech service will keep each synthesis history for up to 31 days, or the duration of the request `timeToLive` property, whichever comes sooner. The date and time of automatic deletion (for synthesis jobs with a status of "Succeeded" or "Failed") is equal to the `lastActionDateTime` + `timeToLive` properties.

+To delete a batch synthesis job, make an HTTP DELETE request using the URI as shown in the following example. Replace `YourSynthesisId` with your batch synthesis ID, replace `YourSpeechKey` with your Speech resource key, and replace `YourSpeechRegion` with your Speech resource region.

+```azurecli-interactive

+curl -v -X DELETE "https://YourSpeechRegion.customvoice.api.speech.microsoft.com/api/texttospeech/3.1-preview1/batchsynthesis/YourSynthesisId" -H "Ocp-Apim-Subscription-Key: YourSpeechKey"

+```

+The response headers will include `HTTP/1.1 204 No Content` if the delete request was successful.

+## Batch synthesis results

+After you [get a batch synthesis job](#get-batch-synthesis) with `status` of "Succeeded", you can download the audio output results. Use the URL from the `outputs.result` property of the [get batch synthesis](#get-batch-synthesis) response.

+To get the batch synthesis results file, make an HTTP GET request using the URI as shown in the following example. Replace `YourOutputsResultUrl` with the URL from the `outputs.result` property of the [get batch synthesis](#get-batch-synthesis) response. Replace `YourSpeechKey` with your Speech resource key.

+```azurecli-interactive

+curl -v -X GET "YourOutputsResultUrl" -H "Ocp-Apim-Subscription-Key: YourSpeechKey" > results.zip

+```

+The results are in a ZIP file that contains the audio (such as `0001.wav`), summary, and debug details. The numbered prefix of each filename (shown below as `[nnnn]`) is in the same order as the text inputs used when you created the batch synthesis.

+> [!NOTE]

+> The `[nnnn].debug.json` file contains the synthesis result ID and other information that might help with troubleshooting. The properties that it contains might change, so you shouldn't take any dependencies on the JSON format.

+The summary file contains the synthesis results for each text input. Here's an example `summary.json` file:

+```json

+{

+ "jobID": "41b83de2-380d-45dc-91af-722b68cfdc8e",

+ "status": "Succeeded",

+ "results": [

+ {

+ "texts": [

+ "<speak version='1.0' xml:lang='en-US'>\n\t\t\t\t<voice xml:lang='en-US' xml:gender='Female' name='en-US-JennyNeural'>\n\t\t\t\t\tThe rainbow has seven colors.\n\t\t\t\t</voice>\n\t\t\t</speak>"

+ ],

+ "status": "Succeeded",

+ "billingDetails": {

+ "CustomNeural": "0",

+ "Neural": "33"

+ },

+ "audioFileName": "0001.wav",

+ "properties": {

+ "audioSize": "100000",

+ "duration": "PT3.1S",

+ "durationInTicks": "31250000"

+ }

+ }

+ ]

+}

+```

+If sentence boundary data was requested (`"sentenceBoundaryEnabled": true`), then a corresponding `[nnnn].sentence.json` file will be included in the results. Likewise, if word boundary data was requested (`"wordBoundaryEnabled": true`), then a corresponding `[nnnn].word.json` file will be included in the results.

+Here's an example word data file with both audio offset and duration in milliseconds:

+```json

+[

+ {

+ "Text": "the",

+ "AudioOffset": 38,

+ "Duration": 153

+ },

+ {

+ "Text": "rainbow",

+ "AudioOffset": 201,

+ "Duration": 326

+ },

+ {

+ "Text": "has",

+ "AudioOffset": 567,

+ "Duration": 96

+ },

+ {

+ "Text": "seven",

+ "AudioOffset": 673,

+ "Duration": 96

+ },

+ {

+ "Text": "colors",

+ "AudioOffset": 778,

+ "Duration": 451

+ },

+]

+```

+## Batch synthesis properties

+Batch synthesis properties are described in the following table.

+| Property | Description |

+|-|-|

+|`createdDateTime`|The date and time when the batch synthesis job was created.<br/><br/>This property is read-only.|

+|`customProperties`|A custom set of optional batch synthesis configuration settings.<br/><br/>This property is stored for your convenience to associate the synthesis jobs that you created with the synthesis jobs that you get or list. This property is stored, but isn't used by the Speech service.<br/><br/>You can specify up to 10 custom properties as key and value pairs. The maximum allowed key length is 64 characters, and the maximum allowed value length is 256 characters.|

+|`customVoices`|The map of a custom voice name and its deployment ID.<br/><br/>For example: `"customVoices": {"your-custom-voice-name": "502ac834-6537-4bc3-9fd6-140114daa66d"}`<br/><br/>You can use the voice name in your `synthesisConfig.voice` (when the `textType` is set to `"PlainText"`) or within the SSML text of `inputs` (when the `textType` is set to `"SSML"`).<br/><br/>This property is required to use a custom voice. If you try to use a custom voice that isn't defined here, the service returns an error.|

+|`description`|The description of the batch synthesis.<br/><br/>This property is optional.|

+|`displayName`|The name of the batch synthesis. Choose a name that you can refer to later. The display name doesn't have to be unique.<br/><br/>This property is required.|

+|`id`|The batch synthesis job ID.<br/><br/>This property is read-only.|

+|`inputs`|The plain text or SSML to be synthesized.<br/><br/>When the `textType` is set to `"PlainText"`, provide plain text as shown here: `"inputs": [{"text": "The rainbow has seven colors."}]`. When the `textType` is set to `"SSML"`, provide text in the [Speech Synthesis Markup Language (SSML)](speech-synthesis-markup.md) as shown here: `"inputs": [{"text": "<speak version='\''1.0'\'' xml:lang='\''en-US'\''><voice xml:lang='\''en-US'\'' xml:gender='\''Female'\'' name='\''en-US-JennyNeural'\''>The rainbow has seven colors.</voice></speak>"}]`.<br/><br/>Include up to 1,000 text objects if you want multiple audio output files. Here's example input text that should be synthesized to two audio output files: `"inputs": [{"text": "synthesize this to a file"},{"text": "synthesize this to another file"}]`. However, if the `properties.concatenateResult` property is set to `true`, then each synthesized result will be written to the same audio output file.<br/><br/>You don't need separate text inputs for new paragraphs. Within any of the (up to 1,000) text inputs, you can specify new paragraphs using the "\r\n" (newline) string. Here's example input text with two paragraphs that should be synthesized to the same audio output file: `"inputs": [{"text": "synthesize this to a file\r\nsynthesize this to another paragraph in the same file"}]`<br/><br/>There are no paragraph limits, but keep in mind that the maximum JSON payload size (including all text inputs and other properties) that will be accepted is 500 kilobytes.<br/><br/>This property is required when you create a new batch synthesis job. This property isn't included in the response when you get the synthesis job.|

+|`lastActionDateTime`|The most recent date and time when the `status` property value changed.<br/><br/>This property is read-only.|

+|`outputs.result`|The location of the batch synthesis result files with audio output and logs.<br/><br/>This property is read-only.|

+|`properties`|A defined set of optional batch synthesis configuration settings.|

+|`properties.audioSize`|The audio output size in bytes.<br/><br/>This property is read-only.|

+|`properties.billingDetails`|The number of words that were processed and billed by `customNeural` versus `neural` (prebuilt) voices.<br/><br/>This property is read-only.|

+|`properties.concatenateResult`|Determines whether to concatenate the result. This optional `bool` value ("true" or "false") is "false" by default.|

+|`properties.decompressOutputFiles`|Determines whether to unzip the synthesis result files in the destination container. This property can only be set when the `destinationContainerUrl` property is set or BYOS (Bring Your Own Storage) is configured for the Speech resource. This optional `bool` value ("true" or "false") is "false" by default.|

+|`properties.destinationContainerUrl`|The batch synthesis results can be stored in a writable Azure container. If you don't specify a container URI with [shared access signatures (SAS)](../../storage/common/storage-sas-overview.md) token, the Speech service stores the results in a container managed by Microsoft. SAS with stored access policies isn't supported. When the synthesis job is deleted, the result data is also deleted.<br/><br/>This optional property isn't included in the response when you get the synthesis job.|

+|`properties.duration`|The audio output duration. The value is an ISO 8601 encoded duration.<br/><br/>This property is read-only.|

+|`properties.durationInTicks`|The audio output duration in ticks.<br/><br/>This property is read-only.|

+|`properties.failedAudioCount`|The count of batch synthesis inputs to audio output failed.<br/><br/>This property is read-only.|

+|`properties.outputFormat`|The audio output format.<br/><br/>For information about the accepted values, see [audio output formats](rest-text-to-speech.md#audio-outputs). The default output format is `riff-24khz-16bit-mono-pcm`.|

+|`properties.sentenceBoundaryEnabled`|Determines whether to generate sentence boundary data. This optional `bool` value ("true" or "false") is "false" by default.<br/><br/>If sentence boundary data is requested, then a corresponding `[nnnn].sentence.json` file will be included in the results data ZIP file.|

+|`properties.succeededAudioCount`|The count of batch synthesis inputs to audio output succeeded.<br/><br/>This property is read-only.|

+|`properties.timeToLive`|A duration after the synthesis job is created, when the synthesis results will be automatically deleted. The value is an ISO 8601 encoded duration. For example, specify `PT12H` for 12 hours. This optional setting is `P31D` (31 days) by default. The maximum time to live is 31 days. The date and time of automatic deletion (for synthesis jobs with a status of "Succeeded" or "Failed") is equal to the `lastActionDateTime` + `timeToLive` properties.<br/><br/>Otherwise, you can call the [delete](#delete-batch-synthesis) synthesis method to remove the job sooner.|

+|`properties.wordBoundaryEnabled`|Determines whether to generate word boundary data. This optional `bool` value ("true" or "false") is "false" by default.<br/><br/>If word boundary data is requested, then a corresponding `[nnnn].word.json` file will be included in the results data ZIP file.|

+|`status`|The batch synthesis processing status.<br/><br/>The status should progress from "NotStarted" to "Running", and finally to either "Succeeded" or "Failed".<br/><br/>This property is read-only.|

+|`synthesisConfig`|The configuration settings to use for batch synthesis of plain text.<br/><br/>This property is only applicable when `textType` is set to `"PlainText"`.|

+|`synthesisConfig.pitch`|The pitch of the audio output.<br/><br/>For information about the accepted values, see the [adjust prosody](speech-synthesis-markup.md#adjust-prosody) table in the Speech Synthesis Markup Language (SSML) documentation. Invalid values are ignored.<br/><br/>This optional property is only applicable when `textType` is set to `"PlainText"`.|

+|`synthesisConfig.rate`|The rate of the audio output.<br/><br/>For information about the accepted values, see the [adjust prosody](speech-synthesis-markup.md#adjust-prosody) table in the Speech Synthesis Markup Language (SSML) documentation. Invalid values are ignored.<br/><br/>This optional property is only applicable when `textType` is set to `"PlainText"`.|

+|`synthesisConfig.style`|For some voices, you can adjust the speaking style to express different emotions like cheerfulness, empathy, and calm. You can optimize the voice for different scenarios like customer service, newscast, and voice assistant.<br/><br/>For information about the available styles per voice, see [voice styles and roles](language-support.md?tabs=stt-tts#voice-styles-and-roles).<br/><br/>This optional property is only applicable when `textType` is set to `"PlainText"`.|

+|`synthesisConfig.voice`|The voice that speaks the audio output.<br/><br/>For information about the available prebuilt neural voices, see [language and voice support](language-support.md?tabs=stt-tts). To use a custom voice, you must specify a valid custom voice and deployment ID mapping in the `customVoices` property.<br/><br/>This property is required when `textType` is set to `"PlainText"`.|

+|`synthesisConfig.volume`|The volume of the audio output.<br/><br/>For information about the accepted values, see the [adjust prosody](speech-synthesis-markup.md#adjust-prosody) table in the Speech Synthesis Markup Language (SSML) documentation. Invalid values are ignored.<br/><br/>This optional property is only applicable when `textType` is set to `"PlainText"`.|

+|`textType`|Indicates whether the `inputs` text property should be plain text or SSML. The possible case-insensitive values are "PlainText" and "SSML". When the `textType` is set to `"PlainText"`, you must also set the `synthesisConfig` voice property.<br/><br/>This property is required.|

+## HTTP status codes

+The section details the HTTP response codes and messages from the batch synthesis API.

+### HTTP 200 OK

+HTTP 200 OK indicates that the request was successful.

+### HTTP 201 Created

+HTTP 201 Created indicates that the create batch synthesis request (via HTTP POST) was successful.

+### HTTP 204 error

+An HTTP 204 error indicates that the request was successful, but the resource doesn't exist. For example:

+- You tried to get or delete a synthesis job that doesn't exist.

+- You successfully deleted a synthesis job.

+### HTTP 400 error

+Here are examples that can result in the 400 error:

+- The `outputFormat` is unsupported or invalid. Provide a valid format value, or leave `outputFormat` empty to use the default setting.

+- The number of requested text inputs exceeded the limit of 1,000.

+- The `top` query parameter exceeded the limit of 100.

+- You tried to use an invalid deployment ID or a custom voice that isn't successfully deployed. Make sure the Speech resource has access to the custom voice, and the custom voice is successfully deployed. You must also ensure that the mapping of `{"your-custom-voice-name": "your-deployment-ID"}` is correct in your batch synthesis request.

+- You tried to delete a batch synthesis job that hasn't started or hasn't completed running. You can only delete batch synthesis jobs that have a status of "Succeeded" or "Failed".

+- You tried to use a `F0` Speech resource, but the region only supports the `S0` (standard) Speech resource pricing tier.

+- You tried to create a new batch synthesis job that would exceed the limit of 200 active jobs. Each Speech resource can have up to 200 batch synthesis jobs that don't have a status of "Succeeded" or "Failed".

+### HTTP 404 error

+The specified entity can't be found. Make sure the synthesis ID is correct.

+### HTTP 429 error

+

+There are too many recent requests. Each client application can submit up to 50 requests per 5 seconds for each Speech resource. Reduce the number of requests per second.

+You can check the rate limit and quota remaining via the HTTP headers as shown in the following example:

+```http

+X-RateLimit-Limit: 50

+X-RateLimit-Remaining: 49

+X-RateLimit-Reset: 2022-11-11T01:49:43Z

+```

+### HTTP 500 error

+HTTP 500 Internal Server Error indicates that the request failed. The response body contains the error message.

+### HTTP error example

+Here's an example request that results in an HTTP 400 error, because the `top` query parameter is set to a value greater than 100.

+```console

+curl -v -X GET "https://YourSpeechRegion.customvoice.api.speech.microsoft.com/api/texttospeech/3.1-preview1/batchsynthesis?skip=0&top=200" -H "Ocp-Apim-Subscription-Key: YourSpeechKey"

+```

+In this case, the response headers will include `HTTP/1.1 400 Bad Request`.

+The response body will resemble the following JSON example:

+```json

+{

+ "code": "InvalidRequest",

+ "message": "The top parameter should not be greater than 100.",

+ "innerError": {

+ "code": "InvalidParameter",

+ "message": "The top parameter should not be greater than 100."

+ }

+}

+```

+## Next steps

+- [Speech Synthesis Markup Language (SSML)](speech-synthesis-markup.md)

+- [Text-to-speech quickstart](get-started-text-to-speech.md)

+- [Migrate to batch synthesis](migrate-to-batch-synthesis.md)

+|`timeToLive`|A duration after the transcription job is created, when the transcription results will be automatically deleted. The value is an ISO 8601 encoded duration. For example, specify `PT12H` for 12 hours. As an alternative, you can call [DeleteTranscription](https://eastus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0/operations/DeleteTranscription) regularly after you retrieve the transcription results.|

+|`duration`|The audio duration. The value is an ISO 8601 encoded duration.|

+|`offset`|The offset in audio of this phrase. The value is an ISO 8601 encoded duration.|

+|`timestamp`|The creation date and time of the transcription. The value is an ISO 8601 encoded timestamp.|

+- [Speech Synthesis Markup Language (SSML)](speech-synthesis-markup.md)

+- [Batch synthesis](batch-synthesis.md)

+- [Batch synthesis](batch-synthesis.md)

+zone_pivot_groups: acs-js-csharp-python

+ Title: Migrate to Batch synthesis API - Speech service

+description: This document helps developers migrate code from Long Audio REST API to Batch synthesis REST API.

+ms.devlang: csharp

+# Migrate code from Long Audio API to Batch synthesis API

+The [Batch synthesis API](batch-synthesis.md) (Preview) provides asynchronous synthesis of long-form text to speech. Benefits of upgrading from Long Audio API to Batch synthesis API, and details about how to do so, are described in the sections below.

+> [!IMPORTANT]

+> [Batch synthesis API](batch-synthesis.md) is currently in public preview. Once it's generally available, the Long Audio API will be deprecated.

+## Base path

+You must update the base path in your code from `/texttospeech/v3.0/longaudiosynthesis` to `/texttospeech/3.1-preview1/batchsynthesis`. For example, to list synthesis jobs for your Speech resource in the `eastus` region, use `https://eastus.customvoice.api.speech.microsoft.com/api/texttospeech/3.1-preview1/batchsynthesis` instead of `https://eastus.customvoice.api.speech.microsoft.com/api/texttospeech/v3.0/longaudiosynthesis`.

+## Regions and endpoints

+Batch synthesis API is available in all [Speech regions](regions.md).

+The Long Audio API is limited to the following regions:

+| Region | Endpoint |

+|--|-|

+| Australia East | `https://australiaeast.customvoice.api.speech.microsoft.com` |

+| East US | `https://eastus.customvoice.api.speech.microsoft.com` |

+| India Central | `https://centralindia.customvoice.api.speech.microsoft.com` |

+| South Central US | `https://southcentralus.customvoice.api.speech.microsoft.com` |

+| Southeast Asia | `https://southeastasia.customvoice.api.speech.microsoft.com` |

+| UK South | `https://uksouth.customvoice.api.speech.microsoft.com` |

+| West Europe | `https://westeurope.customvoice.api.speech.microsoft.com` |

+## Voices list

+Batch synthesis API supports all [text-to-speech voices and styles](language-support.md?tabs=stt-tts).

+The Long Audio API is limited to the set of voices returned by a GET request to `https://<endpoint>/api/texttospeech/v3.0/longaudiosynthesis/voices`.

+## Text inputs

+Batch synthesis text inputs are sent in a JSON payload of up to 500 kilobytes.

+Long Audio API text inputs are uploaded from a file that meets the following requirements:

+* One plain text (.txt) or SSML text (.txt) file encoded as [UTF-8 with Byte Order Mark (BOM)](https://www.w3.org/International/questions/qa-utf8-bom.en#bom). Don't use compressed files such as ZIP. If you have more than one input file, you must submit multiple requests.

+* Contains more than 400 characters for plain text or 400 [billable characters](./text-to-speech.md#pricing-note) for SSML text, and less than 10,000 paragraphs. For plain text, each paragraph is separated by a new line. For SSML text, each SSML piece is considered a paragraph. Separate SSML pieces by different paragraphs.

+With Batch synthesis API, you can use any of the [supported SSML elements](speech-synthesis-markup.md?tabs=csharp#supported-ssml-elements), including the `audio`, `mstts:backgroundaudio`, and `lexicon` elements. The `audio`, `mstts:backgroundaudio`, and `lexicon` elements aren't supported by Long Audio API.

+## Audio output formats

+Batch synthesis API supports all [text-to-speech audio output formats](rest-text-to-speech.md#audio-outputs).

+The Long Audio API is limited to the following set of audio output formats. The sample rate for long audio voices is 24kHz, not 48kHz. Other sample rates can be obtained through upsampling or downsampling when synthesizing.

+* riff-8khz-16bit-mono-pcm

+* riff-16khz-16bit-mono-pcm

+* riff-24khz-16bit-mono-pcm

+* riff-48khz-16bit-mono-pcm

+* audio-16khz-32kbitrate-mono-mp3

+* audio-16khz-64kbitrate-mono-mp3

+* audio-16khz-128kbitrate-mono-mp3

+* audio-24khz-48kbitrate-mono-mp3

+* audio-24khz-96kbitrate-mono-mp3

+* audio-24khz-160kbitrate-mono-mp3

+## Getting results

+With batch synthesis API, use the URL from the `outputs.result` property of the GET batch synthesis response. The [results](batch-synthesis.md#batch-synthesis-results) are in a ZIP file that contains the audio (such as `0001.wav`), summary, and debug details.

+Long Audio API text inputs and results are returned via two separate content URLs as shown in the following example. The one with `"kind": "LongAudioSynthesisScript"` is the input script submitted. The other one with `"kind": "LongAudioSynthesisResult"` is the result of this request. Both ZIP files can be downloaded from the URL in their `links.contentUrl` property.

+## Cleaning up resources

+Batch synthesis API supports up to 200 batch synthesis jobs that don't have a status of "Succeeded" or "Failed". The Speech service will keep each synthesis history for up to 31 days, or the duration of the request `timeToLive` property, whichever comes sooner. The date and time of automatic deletion (for synthesis jobs with a status of "Succeeded" or "Failed") is equal to the `lastActionDateTime` + `timeToLive` properties.

+The Long Audio API is limited to 20,000 requests for each Azure subscription account. The Speech service doesn't remove job history automatically. You must remove the previous job run history before making new requests that would otherwise exceed the limit.

+## Next steps

+- [Batch synthesis API](batch-synthesis.md)

+- [Speech Synthesis Markup Language (SSML)](speech-synthesis-markup.md)

+- [Text-to-speech quickstart](get-started-text-to-speech.md)

+- [Batch synthesis](batch-synthesis.md)

+### Text-to-speech quotas and limits per Speech resource

+| **Max number of transactions per certain time period** | | |

+| Max number of transactions per second (TPS) | Not available for F0 | See [General](#general) |

+| Max number of datasets | N/A | 500 |

+| Max number of simultaneous dataset uploads | N/A | 5 |

+| Max number of simultaneous model trainings | N/A | 3 |

+| Max number of custom endpoints | N/A | 50 |

+| Concurrent request limit for Custom Neural Voice | | |

+| `type` | Specifies where and how to add silence. The following silence types are supported:<br/><ul><li>`Leading` ΓÇô Additional silence at the beginning of the text. The value that you set is added to the natural silence before the start of text.</li><li>`Leading-exact` ΓÇô Silence at the beginning of the text. The value is an absolute silence length.</li><li>`Tailing` ΓÇô Additional silence at the end of text. The value that you set is added to the natural silence after the last word.</li><li>`Tailing-exact` ΓÇô Silence at the end of the text. The value is an absolute silence length.</li><li>`Sentenceboundary` ΓÇô Additional silence between adjacent sentences. The actual silence length for this type includes the natural silence after the last word in the previous sentence, the value you set for this type, and the natural silence before the starting word in the next sentence.</li><li>`Sentenceboundary-exact` ΓÇô Silence between adjacent sentences. The value is an absolute silence length.</li></ul><br/>An absolute silence type (with the `-exact` suffix) replaces any otherwise natural leading or trailing silence. Absolute silence types take precedence over the corresponding non-absolute type. For example, if you set both `Leading` and `Leading-exact` types, the `Leading-exact` type will take effect.| Required |

+| `Value` | Specifies the duration of a pause in seconds or milliseconds. This value should be set less than 5,000 ms. Examples of valid values are `2s` and `500ms`.| Required |

+> The `lexicon` element is not supported by the [Long Audio API](migrate-to-batch-synthesis.md#text-inputs). For long-form text-to-speech, use the [batch synthesis API](batch-synthesis.md) (Preview) instead.

+> The 'audio' element is not supported by the [Long Audio API](migrate-to-batch-synthesis.md#text-inputs). For long-form text-to-speech, use the [batch synthesis API](batch-synthesis.md) (Preview) instead.

+> The `mstts:backgroundaudio` element is not supported by the [Long Audio API](migrate-to-batch-synthesis.md#text-inputs). For long-form text-to-speech, use the [batch synthesis API](batch-synthesis.md) (Preview) instead.

+* **Asynchronous synthesis of long audio**: Use the [batch synthesis API](batch-synthesis.md) (Preview) to asynchronously synthesize text-to-speech files longer than 10 minutes (for example, audio books or lectures). Unlike synthesis performed via the Speech SDK or speech-to-text REST API, responses aren't returned in real time. The expectation is that requests are sent asynchronously, responses are polled for, and synthesized audio is downloaded when the service makes it available.

+ Title: Modifications to Translator Service

+description: Translator Service changes, modifications, and deprecations

+# Modifications to Translator Service

+Learn about Translator service changes, modification, and deprecations.

+> [!NOTE]

+> Looking for updates and preview announcements? Visit our [What's new](whats-new.md) page to stay up to date with release notes, feature enhancements, and our newest documentation.

+## November 2022

+### Changes to Translator `Usage` metrics

+> [!IMPORTANT]

+> **`Characters Translated`** and **`Characters Trained`** metrics are deprecated and have been removed from the Azure portal.

+|Deprecated metric| Current metric(s) | Description|

+||||

+|Characters Translated (Deprecated)</br></br></br></br>|**&bullet; Text Characters Translated**</br></br>**&bullet;Text Custom Characters Translated**| &bullet; Number of characters in incoming **text** translation request.</br></br> &bullet; Number of characters in incoming **custom** translation request. |

+|Characters Trained (Deprecated) | **&bullet; Text Trained Characters** | &bullet; Number of characters **trained** using text translation service.|

+* In 2021, two new metrics, **Text Characters Translated** and **Text Custom Characters Translated**, were added to help with granular metrics data service usage. These metrics replaced **Characters Translated** which provided combined usage data for the general and custom text translation service.

+* Similarly, the **Text Trained Characters** metric was added to replace the **Characters Trained** metric.

+* **Characters Trained** and **Characters Translated** metrics have had continued support in the Azure portal with the deprecated flag to allow migration to the current metrics. As of October 2022, Characters Trained and Characters Translated are no longer available in the Azure portal.

+Translator Service is a cloud-based neural machine translation service that is part of the [Azure Cognitive Services](../what-are-cognitive-services.md) family of REST APIs and can be used with any operating system. Translator powers many Microsoft products and services used by thousands of businesses worldwide to perform language translation and other language-related operations. In this overview, you'll learn how Translator can enable you to build intelligent, multi-language solutions for your applications across all [supported languages](./language-support.md).

+Bookmark this page to stay up to date with release notes, feature enhancements, and our newest documentation.

+1. **Define your schema**: Know your data and identify the [classes](glossary.md#class) you want differentiate between, to avoid ambiguity.

+## Category: Age

+ Age

+> [!NOTE]

+> Conversation PII now supports 40,000 characters as document size.

+ |Python | [1.0.0](https://pypi.org/project/azure-ai-language-conversations/1.1.0b2) |

+## November 2022

+* Conversational PII now supports up to 40,000 characters as document size.

+The **Call Automation** tab displays data about calls placed or answered using Call Automation SDK, like active call count, operations executed and errors encountered by your resource over time. You can also examine a particular call by looking at the sequence of operations taken on that call using the SDK:

+If you are looking to build your own Network Diagnostic Tool or to perform deeper integration of this tool into your application, you can leverage [pre-call diagnostic APIs](../voice-video-calling/pre-call-diagnostics.md) for the calling SDK.

+ Title: Add a bot to your chat app

+# Add a bot to your chat app

+> This functionality is in public preview.

+In this quickstart, you will learn how to build conversational AI experiences in a chat application using Azure Communication Services Chat messaging channel that is available under Azure Bot Services. This article will describe how to create a bot using BotFramework SDK and how to integrate this bot into any chat application that is built using Communication Services Chat SDK.

+You will learn how to:

+- [Create and deploy an Azure bot](#step-1create-and-deploy-an-azure-bot)

+- [Get an Azure Communication Services resource](#step-2get-an-azure-communication-services-resource)

+- [Enable Communication Services Chat channel for the bot](#step-3enable-azure-communication-services-chat-channel)

+- [Explore more features available for bot](#more-things-you-can-do-with-a-bot)

+- Latest version of .NET Core. For this tutorial, we have used [.NET Core 3.1](https://dotnet.microsoft.com/download/dotnet-core/3.1) (Make sure to install the version that corresponds with your visual studio instance, 32 vs 64 bit)

+## Step 1 - Create and deploy an Azure bot

+To use Azure Communication Services chat as a channel in Azure Bot Service, the first step is to deploy a bot. You can do so by following below steps:

+### Create an Azure bot service resource in Azure

+ Refer to the Azure Bot Service documentation on how to [create a bot](/azure/bot-service/abs-quickstart?tabs=userassigned).

+ For this example, we have selected a multitenant bot but if you wish to use single tenant or managed identity bots refer to [configuring single tenant and managed identity bots](#support-for-single-tenant-and-managed-identity-bots).

+ Fetch your Azure bot's [Microsoft App ID and secret](/azure/bot-service/abs-quickstart?tabs=userassigned#to-get-your-app-or-tenant-id) as you will need those values for configurations.

+### Create a Web App where the bot logic resides

+ You can check out some samples at [Bot Builder Samples](https://github.com/Microsoft/BotBuilder-Samples) and tweak them or use [Bot Builder SDK](/composer/introduction) to create one. One of the simplest samples is [Echo Bot](https://github.com/microsoft/BotBuilder-Samples/tree/main/samples/csharp_dotnetcore/02.echo-bot). Generally, the Azure Bot Service expects the Bot Application Web App Controller to expose an endpoint `/api/messages`, which handles all the messages reaching the bot. To create the bot application, you can either use Azure CLI to [create an App Service](/azure/bot-service/provision-app-service?tabs=singletenant%2Cexistingplan) or directly create from the portal using below steps.

+ 1. Select `Create a resource` and in the search box, search for web app and select `Web App`.

+ :::image type="content" source="./media/web-app.png" alt-text="Screenshot of creating a Web app resource in Azure portal.":::

+ :::image type="content" source="./media/web-app-create-options.png" alt-text="Screenshot of specifying Web App create options to set.":::

+ 3. Review your options and create the Web App and once it has been created, copy the hostname URL exposed by the Web App.

+ :::image type="content" source="./media/web-app-endpoint.png" alt-text="Diagram that shows how to copy the newly created Web App endpoint.":::

+Configure the Azure Bot you created with its Web App endpoint where the bot logic is located. To do this configuration, copy the hostname URL of the Web App from previous step and append it with `/api/messages`

+ :::image type="content" source="./media/smaller-bot-configure-with-endpoint.png" alt-text="Diagram that shows how to set bot messaging endpoint with the copied Web App endpoint." lightbox="./media/bot-configure-with-endpoint.png":::

+The final step would be to deploy the Web App we created. The Echo bot functionality is limited to echoing the user input. Here's how we deploy it to Azure Web App.

+ git clone https://github.com/Microsoft/BotBuilder-Samples.git

+ cd BotBuilder-Samples

+ 3. Go to the appsettings.json file inside the project and copy the [Microsoft application ID and secret](#get-bots-microsoftappid-and-microsoftapppassword) in their respective placeholders.

+ For deploying the bot, you can either use command line to [deploy an Azure bot](/azure/bot-service/provision-and-publish-a-bot?tabs=userassigned%2Ccsharp) or use Visual studio for C# bots as described below.

+ 1. Select the project to publish the Web App code to Azure. Choose the publish option in Visual Studio.

+ :::image type="content" source="./media/publish-app.png" alt-text="Screenshot of publishing your Web App from Visual Studio.":::

+ 2. Select New to create a new publishing profile, choose Azure as the target, and Azure App Service as the specific target.

+ :::image type="content" source="./media/select-azure-as-target.png" alt-text="Diagram that shows how to select Azure as target in a new publishing profile.":::

+ :::image type="content" source="./media/select-app-service.png" alt-text="Diagram that shows how to select specific target as Azure App Service.":::

+ 3. Lastly, the above option opens the deployment config. Choose the Web App we had created from the list of options it comes up with after signing into your Azure account. Once ready select `Finish` to complete the profile, and then select `Publish` to start the deployment.

+ :::image type="content" source="./media/smaller-deployment-config.png" alt-text="Screenshot of setting deployment config with the created Web App name." lightbox="./media/deployment-config.png":::

+Now that bot is created and deployed, you will need an Azure Communication Services resource, which you can use to configure the Azure Communication Services channel.

+1. Create an Azure Communication Services resource. For details, see [Create an Azure Communication Services resource](../../quickstarts/create-communication-resource.md).

+2. Create an Azure Communication Services User and issue a [User Access Token](../../quickstarts/access-tokens.md). Be sure to set the scope to **chat**, and **note the token string as well as the userId string**.

+## Step 3 - Enable Azure Communication Services Chat channel

+With the Azure Communication Services resource, you can set up the Azure Communication Services channel in Azure Bot to assign an Azure Communication Services User ID to a bot.

+1. Go to your Bot Services resource on Azure portal. Navigate to `Channels` configuration on the left pane and select `Azure Communications Services - Chat` channel from the list provided.

+ :::image type="content" source="./media/smaller-demoapp-launch-acs-chat.png" alt-text="Screenshot of launching Azure Communication Services Chat channel." lightbox="./media/demoapp-launch-acs-chat.png":::

+2. Select the connect button to see a list of ACS resources available under your subscriptions.

+ :::image type="content" source="./media/smaller-bot-connect-acs-chat-channel.png" alt-text="Diagram that shows how to connect an Azure Communication Service Resource to this bot." lightbox="./media/bot-connect-acs-chat-channel.png":::

+3. Once you have selected the required Azure Communication Services resource from the resources dropdown list, press the apply button.

+ :::image type="content" source="./media/smaller-bot-choose-resource.png" alt-text="Diagram that shows how to save the selected Azure Communication Service resource to create a new Azure Communication Services user ID." lightbox="./media/bot-choose-resource.png":::

+4. Once the provided resource details are verified, you will see the **bot's Azure Communication Services ID** assigned. With this ID, you can add the bot to the conversation whenever appropriate using Chat's AddParticipant API. Once the bot is added as participant to a chat, it will start receiving chat related activities, and can respond back in the chat thread.

+ :::image type="content" source="./media/smaller-acs-chat-channel-saved.png" alt-text="Screenshot of new Azure Communication Services user ID assigned to the bot." lightbox="./media/acs-chat-channel-saved.png":::

+Now that you have the bot's Azure Communication Services ID, you can create a chat thread with the bot as a participant.

+To create a chat client, you will use your Azure Communication Services endpoint and the access token that was generated as part of Step 2. You need to use the `CommunicationIdentityClient` class from the Identity SDK to create a user and issue a token to pass to your chat client.

+When creating the chat applications, you can also receive real-time notifications by subscribing to listen for new incoming messages using our JavaScript or mobile SDKs. An example using JavaScript SDK would be:

+### Clean up the chat thread

+Delete the thread when finished.

+```csharp

+chatClient.DeleteChatThread(threadId);

+```

+Follow these steps to deploy the chat application:

+2. Select the ChatQuickstart project and from the right-click menu, select Publish

+ :::image type="content" source="./media/deploy-chat-application.png" alt-text="Screenshot of deploying chat application to Azure from Visual Studio.":::

+## More things you can do with a bot

+In addition to sending a plain text message, a bot is also able to receive many other activities from the user through Azure Communications Services Chat channel including

+- Various attachments including Adaptive cards

+- Bot channel data

+Below are some samples to illustrate these features:

+ The current Echo Bot logic accepts input from the user and echoes it back. If you would like to add extra logic such as responding to a participant added Azure Communication Services event, copy the following code snippets and paste into the source file: [EchoBot.cs](https://github.com/microsoft/BotBuilder-Samples/blob/main/samples/csharp_dotnetcore/02.echo-bot/Bots/EchoBot.cs)

+Sending adaptive cards to the chat thread can help you increase engagement and efficiency and communicate with users in a variety of ways. You can send adaptive cards from a bot by adding them as bot activity attachments.

+On the Azure Communication Services User side, the Azure Communication Services Chat channel will add a field to the message's metadata that will indicate that this message has an attachment. The key in the metadata is `microsoft.azure.communication.chat.bot.contenttype`, which is set to the value `azurebotservice.adaptivecard`. Here is an example of the chat message that will be received:

+* ### Send a message from user to bot

+You can send a simple text message from user to bot just the same way you send a text message to another user.

+However, when sending a message carrying an attachment from a user to the bot, you will need to add this flag to the ACS Chat metadata `"microsoft.azure.communication.chat.bot.contenttype": "azurebotservice.adaptivecard"`. For sending an event activity from user to bot, you will need to add to ACS Chat metadata `"microsoft.azure.communication.chat.bot.contenttype": "azurebotservice.event"`. Below are sample formats for user to bot ACS Chat messages.

+ * #### Simple text message

+```json

+{

+ "content":"Simple text message",

+ "senderDisplayName":"Acs-Dev-Bot",

+ "metadata":{

+ "text":"random text",

+ "key1":"value1",

+ "key2":"{\r\n \"subkey1\": \"subValue1\"\r\n

+ "},

+ "messageType": "Text"

+}

+```

+ * #### Message with an attachment

+```json

+{

+ "content": "{

+ \"text\":\"sample text\",

+ \"attachments\": [{

+ \"contentType\":\"application/vnd.microsoft.card.adaptive\",

+ \"content\": { \"*adaptive card payload*\" }

+ }]

+ }",

+ "senderDisplayName": "Acs-Dev-Bot",

+ "metadata": {

+ "microsoft.azure.communication.chat.bot.contenttype": "azurebotservice.adaptivecard",

+ "text": "random text",

+ "key1": "value1",

+ "key2": "{\r\n \"subkey1\": \"subValue1\"\r\n}"

+ },

+ "messageType": "Text"

+}

+```

+ * #### Message with an event activity

+Event payload comprises all json fields in the message content except name field, which should contain the name of the event. Below event name `endOfConversation` with the payload `"{field1":"value1", "field2": { "nestedField":"nestedValue" }}` is sent to the bot.

+```json

+{

+ "content":"{

+ \"name\":\"endOfConversation\",

+ \"field1\":\"value1\",

+ \"field2\": {

+ \"nestedField\":\"nestedValue\"

+ }

+ }",

+ "senderDisplayName":"Acs-Dev-Bot",

+ "metadata":{

+ "microsoft.azure.communication.chat.bot.contenttype": "azurebotservice.event",

+ "text":"random text",

+ "key1":"value1",

+ "key2":"{\r\n \"subkey1\": \"subValue1\"\r\n}"

+ },

+ "messageType": "Text"

+}

+```

+> The metadata field `"microsoft.azure.communication.chat.bot.contenttype"` is only needed in user to bot direction. It is not needed in bot to user direction.

+## Supported bot activity fields

+### Bot to user flow

+#### Activities

+- Message activity

+- Typing activity

+#### Message activity fields

+- `Text`

+- `Attachments`

+- `AttachmentLayout`

+- `SuggestedActions`

+- `From.Name` (Converted to ACS SenderDisplayName)

+- `ChannelData` (Converted to ACS Chat Metadata. If any `ChannelData` mapping values are objects, then they'll be serialized in JSON format and sent as a string)

+### User to bot flow

+#### Activities and fields

+- Message activity

+ - `Id` (ACS Chat message ID)

+ - `TimeStamp`

+ - `Text`

+ - `Attachments`

+- Conversation update activity

+ - `MembersAdded`

+ - `MembersRemoved`

+ - `TopicName`

+- Message update activity

+ - `Id` (Updated ACS Chat message ID)

+ - `Text`

+ - `Attachments`

+- Message delete activity

+ - `Id` (Deleted ACS Chat message ID)

+- Event activity

+ - `Name`

+ - `Value`

+- Typing activity

+#### Other common fields

+- `Recipient.Id` and `Recipeint.Name` (ACS Chat user ID and display name)

+- `From.Id` and `From.Name` (ACS Chat user ID and display name)

+- `Conversation.Id` (ACS Chat thread ID)

+- `ChannelId` (AcsChat if empty)

+- `ChannelData` (ACS Chat message metadata)

+## Support for single tenant and managed identity bots

+ACS Chat channel supports single tenant and managed identity bots as well. Refer to [bot identity information](/azure/bot-service/bot-builder-authentication?tabs=userassigned%2Caadv2%2Ccsharp#bot-identity-information) to set up your bot web app.

+For managed identity bots, additionally, you might have to [update bot service identity](/azure/bot-service/bot-builder-authentication?tabs=userassigned%2Caadv2%2Ccsharp#to-update-your-app-service).

+## Bot handoff patterns

+Sometimes the bot wouldn't be able to understand or answer a question or a customer can request to be connected to a human agent. Then it will be necessary to handoff the chat thread from a bot to a human agent. In such cases, you can design your application to [transition conversation from bot to human](/azure/bot-service/bot-service-design-pattern-handoff-human).

+## Handling bot to bot communication

+ There may be certain usecases where two bots need to be added to the same thread. If this occurs, then the bots may start replying to each other's messages. If this scenario is not handled properly, the bots' automated interaction between themselves may result in an infinite loop of messages. This scenario is handled by Azure Communication Services Chat by throttling the requests which will result in the bot not being able to send and receive the messages. You can learn more about the [throttle limits](/azure/communication-services/concepts/service-limits#chat).

+## Troubleshooting

+### Chat channel cannot be added

+- Verify that in the ABS portal, Configuration -> Bot Messaging endpoint has been set correctly.

+### Bot gets a forbidden exception while replying to a message

+- Verify that bot's Microsoft App ID and secret are saved correctly in the bot configuration file uploaded to the webapp.

+### Bot is not able to be added as a participant

+- Verify that bot's ACS ID is being used correctly while sending a request to add bot to a chat thread.

+ Title: Azure Confidential virtual machine options on AMD processors

+* Currently Continuous backup mode and Synapse Link aren't supported in the same database account. Customers have to choose one of these two features and this decision can't be changed.

+ Title: Materialized views (preview)

+description: This documentation is provided as a resource for participants in the preview of Azure Cosmos DB Cassandra API Materialized View.

+# Materialized views in Azure Cosmos DB for Apache Cassandra (preview)

+> Materialized views in Azure Cosmos DB for Cassandra is currently in preview. You can enable this feature using the Azure portal. This preview of materialized views is provided without a service-level agreement. At this time, materialized views are not recommended for production workloads. Certain features of this preview may not be supported or may have constrained capabilities. For more information, see [supplemental terms of use for Microsoft Azure previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+Materialized views, when defined, help provide a means to efficiently query a base table (or container in Azure Cosmos DB) with filters that aren't primary keys. When users write to the base table, the materialized view is built automatically in the background. This view can have a different primary key for efficient lookups. The view will also only contain columns explicitly projected from the base table. This view will be a read-only table.

+You can query a column store without specifying a partition key by using secondary indexes. However, the query won't be effective for columns with high or low cardinality. The query could scan through all data for a small result set. Such queries end up being expensive as they end up inadvertently executing as a cross-partition query.

+With a materialized view, you can:

+- Use as lookup or mapping table to persist cross partition scans that would otherwise be expensive queries.

+- Provide a SQL-based conditional predicate to populate only certain columns and data that meet the pre-condition.

+- Create real-time views that simplify event-based scenarios that are commonly stored as separate collections using change feed triggers.

+## Benefits of materialized views

+Materialized views have many benefits that include, but aren't limited to:

+- You can implement server-side denormalization using materialized views. With server-side denormalization, you can avoid multiple independent tables and computationally complex denormalization in client applications.

+- Materialized views automatically updating views to keep them consistent with the base table. This automatic update abstracts the responsibilities of your client applications with would typically implement custom logic to perform dual writes to the base table and the view.

+- Materialized views optimize read performance by reading from a single view.

+- You can specify throughput for the materialized view independently.

+- You can configure a materialized view builder layer to map to your requirements to hydrate a view.

+- Materialized views improve write performance as write operations only need to be written to the base table.

+- Additionally, the Azure Cosmos DB implementation of materialized views is based on a pull model. This implementation doesn't affect write performance.

+## Get started with materialized views

+Create new API for Cassandra accounts by using the Azure CLI to enable the materialized views feature either with a native command or a REST API operation.

+### [Azure portal](#tab/azure-portal)

+1. Sign in to the [Azure portal](https://portal.azure.com/).

+1. Navigate to your API for Cassandra account.

+1. In the resource menu, select **Settings**.

+1. In the **Settings** section, select **Materialized View for Cassandra API (Preview)**.

+1. In the new dialog, select **Enable** to enable this feature for this account.

+ :::image type="content" source="media/materialized-views/enable-in-portal.png" lightbox="media/materialized-views/enable-in-portal.png" alt-text="Screenshot of the Materialized Views feature being enabled in the Azure portal.":::

+### [Azure CLI](#tab/azure-cli)

+1. Sign-in to the Azure CLI.

+ ```azurecli

+ az login

+ ```

+ > [!NOTE]

+ > If you do not have the Azure CLI installed, see [how to install the Azure CLI](/cli/azure/install-azure-cli).

+1. Install the [`cosmosdb-preview`](https://github.com/azure/azure-cli-extensions/tree/main/src/cosmosdb-preview) extension.

+ ```azurecli

+ az extension add \

+ --name cosmosdb-preview

+ ```

+1. Create shell variables for `accountName` and `resourceGroupName`.

+ ```azurecli

+ # Variable for resource group name

+ resourceGroupName="<resource-group-name>"

+

+ # Variable for account name

+ accountName="<account-name>"

+ ```

+1. Enable the preview materialized views feature for the account using [`az cosmosdb update`](/cli/azure/cosmosdb#az-cosmosdb-update).

+ ```azurecli

+ az cosmosdb update \

+ --resource-group $resourceGroupName \

+ --name $accountName \

+ --enable-materialized-views true \

+ --capabilities CassandraEnableMaterializedViews

+ ```

+### [REST API](#tab/rest-api)

+1. Sign-in to the Azure CLI.

+ ```azurecli

+ az login

+ ```

+ > [!NOTE]

+ > If you do not have the Azure CLI installed, see [how to install the Azure CLI](/cli/azure/install-azure-cli).

+1. Create shell variables for `accountName` and `resourceGroupName`.

+ ```azurecli

+ # Variable for resource group name

+ resourceGroupName="<resource-group-name>"

+

+ # Variable for account name

+ accountName="<account-name>"

+ ```

+1. Create a new JSON file with the capabilities manifest.

+ ```json

+ {

+ "properties": {

+ "capabilities": [

+ {

+ "name": "CassandraEnableMaterializedViews"

+ }

+ ],

+ "enableMaterializedViews": true

+ }

+ }

+ ```

+ > [!NOTE]

+ > In this example, we named the JSON file **capabilities.json**.

+1. Get the unique identifier for your existing account using [`az cosmosdb show`](/cli/azure/cosmosdb#az-cosmosdb-show).

+ ```azurecli

+ az cosmosdb show \

+ --resource-group $resourceGroupName \

+ --name $accountName \

+ --query id

+ ```

+ Store the unique identifier in a shell variable named `$uri`.

+ ```azurecli

+ uri=$(

+ az cosmosdb show \

+ --resource-group $resourceGroupName \

+ --name $accountName \

+ --query id \

+ --output tsv

+ )

+ ```

+1. Enable the preview materialized views feature for the account using the REST API and [`az rest`](/cli/azure/reference-index#az-rest) with an HTTP `PATCH` verb.

+ ```azurecli

+ az rest \

+ --method PATCH \

+ --uri "https://management.azure.com/$uri/?api-version=2021-11-15-preview" \

+ --body @capabilities.json

+ ```

+The API for Cassandra uses a materialized view builder compute layer to maintain the views.

+You get the flexibility to configure the view builder's compute instances based on your latency and lag requirements to hydrate the views. From a technical stand point, this compute layer helps manage connections between partitions in a more efficient manner even when the data size is large and the number of partitions is high.

+The compute containers are shared among all materialized views within an Azure Cosmos DB account. Each provisioned compute container spawns off multiple tasks that read the change feed from base table partitions and writes data to the target materialized view\[s\]. The compute container transforms the data per the materialized view definition for each materialized view in the account.

+## Create a materialized view builder

+Create a materialized view builder to automatically transform data and write to a materialized view.

+### [Azure portal](#tab/azure-portal)

+1. Sign in to the [Azure portal](https://portal.azure.com/).

+1. Navigate to your API for Cassandra account.

+1. In the resource menu, select **Materialized Views Builder**.

+1. On the **Materialized Views Builder** page, configure the SKU and number of instances for the builder.

+ > [!NOTE]

+ > This resource menu option and page will only appear when the Materialized Views feature is enabled for the account.

+1. Select **Save**.

+### [Azure CLI](#tab/azure-cli)

+1. Enable the materialized views builder for the account using [`az cosmosdb service create`](/cli/azure/cosmosdb/service#az-cosmosdb-service-create).

+ ```azurecli

+ az cosmosdb service create \

+ --resource-group $resourceGroupName \

+ --name materialized-views-builder \

+ --account-name $accountName \

+ --count 1 \

+ --kind MaterializedViewsBuilder \

+ --size Cosmos.D4s

+ ```

+### [REST API](#tab/rest-api)

+1. Create a new JSON file with the builder manifest.

+ ```json

+ {

+ "properties": {

+ "serviceType": "materializedViewsBuilder",

+ "instanceCount": 1,

+ "instanceSize": "Cosmos.D4s"

+ }

+ }

+ ```

+ > [!NOTE]

+ > In this example, we named the JSON file **builder.json**.

+1. Enable the materialized views builder for the account using the REST API and `az rest` with an HTTP `PUT` verb.

+ ```azurecli

+ az rest \

+ --method PUT \

+ --uri "https://management.azure.com/$uri/services/materializedViewsBuilder?api-version=2021-11-15-preview" \

+ --body @builder.json

+ ```

+1. Wait a couple of minutes and check the status using `az rest` again with the HTTP `GET` verb. The status in the output should now be `Running`:

+ ```azurecli

+ az rest \

+ --method GET \

+ --uri "https://management.azure.com/$uri/services/materializedViewsBuilder?api-version=2021-11-15-preview"

+ ```

+## Create a materialized view

+Once your account and Materialized View Builder is set up, you should be able to create Materialized views using CQLSH.

+> [!NOTE]

+> If you do not already have the standalone CQLSH tool installed, see [install the CQLSH Tool](support.md#cql-shell). You should also [update your connection string](manage-data-cqlsh.md#update-your-connection-string) in the tool.

+Here are a few sample commands to create a materialized view:

+1. First, create a **keyspace** name `uprofile`.

+ ```sql

+ CREATE KEYSPACE IF NOT EXISTS uprofile WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 1 };

+ ```

+1. Next, create a table named `user` within the keyspace.

+ ```sql

+ CREATE TABLE IF NOT EXISTS uprofile.USER (user_id INT PRIMARY KEY, user_name text, user_bcity text);

+ ```

+1. Now, create a materialized view named `user_by_bcity` within the same keyspace. Specify, using a query, how data is projected into the view from the base table.

+ ```sql

+ CREATE MATERIALIZED VIEW uprofile.user_by_bcity AS

+ SELECT

+ user_id,

+ user_name,

+ user_bcity

+ FROM

+ uprofile.USER

+ WHERE

+ user_id IS NOT NULL

+ AND user_bcity IS NOT NULL PRIMARY KEY (user_bcity, user_id);

+ ```

+1. Insert rows into the base table.

+ ```sql

+ INSERT INTO

+ uprofile.USER (user_id, user_name, user_bcity)

+ VALUES

+ (

+ 101, 'johnjoe', 'New York'

+ );

+

+ INSERT INTO

+ uprofile.USER (user_id, user_name, user_bcity)

+ VALUES

+ (

+ 102, 'james', 'New York'

+ );

+ ```

+1. Query the materialized view.

+ ```sql

+ SELECT * FROM user_by_bcity;

+ ```

+1. Observe the output from the materialized view.

+ ```output

+ user_bcity | user_id | user_name

+ ++--

+ New York | 101 | johnjoe

+ New York | 102 | james

+

+ (2 rows)

+ ```

+Optionally, you can also use the resource provider to create or update a materialized view.

+- [Create or Update a view in API for Cassandra](/rest/api/cosmos-db-resource-provider/2021-11-15-preview/cassandra-resources/create-update-cassandra-view)

+- [Get a view in API for Cassandra](/rest/api/cosmos-db-resource-provider/2021-11-15-preview/cassandra-resources/get-cassandra-view)

+- [List views in API for Cassandra](/rest/api/cosmos-db-resource-provider/2021-11-15-preview/cassandra-resources/list-cassandra-views)

+- [Delete a view in API for Cassandra](/rest/api/cosmos-db-resource-provider/2021-11-15-preview/cassandra-resources/delete-cassandra-view)

+- [Update the throughput of a view in API for Cassandra](/rest/api/cosmos-db-resource-provider/2021-11-15-preview/cassandra-resources/update-cassandra-view-throughput)

+## Current limitations

+There are a few limitations with the API for Cassandra's preview implementation of materialized views:

+- Materialized views can't be created on a table that existed before support for materialized views was enabled on the account. To use materialized views, create a new table after the feature is enabled.

+- For the materialized view definitionΓÇÖs `WHERE` clause, only `IS NOT NULL` filters are currently allowed.

+- After a materialized view is created against a base table, `ALTER TABLE ADD` operations aren't allowed on the base tableΓÇÖs schema. `ALTER TABLE APP` is allowed only if none of the materialized views have selected `*` in their definition.

+- There are limits on partition key size (**2 Kb**) and total length of clustering key size (**1 Kb**). If this size limit is exceeded, the responsible message will end up in poison message queue.

+- If a base table has user-defined types (UDTs) and materialized view definition has either `SELECT * FROM` or has the UDT in one of projected columns, UDT updates aren't permitted on the account.

+- Materialized views may become inconsistent with the base table for a few rows after automatic regional failover. To avoid this inconsistency, rebuild the materialized view after the failover.

+- Creating materialized view builder instances with **32 cores** isn't supported. If needed, you can create multiple builder instances with a smaller number of cores.

+In addition to the above limitations, consider the following extra limitations:

+- Availability zones

+ - Materialized views can't be enabled on an account that has availability zone enabled regions.

+ - Adding a new region with an availability zone isn't supported once `enableMaterializedViews` is set to true on the account.

+- Periodic backup and restore

+ - Materialized views aren't automatically restored with the restore process. You'll need to re-create the materialized views after the restore process is complete. Then, you should configure `enableMaterializedViews` on their restored account before creating the materialized views and builders again.

+- Apache Cassandra

+ - Defining conflict resolution policy on materialized views isn't allowed.

+ - Write operations aren't allowed on materialized views.

+ - Cross document queries and use of aggregate functions aren't supported on materialized views.

+ - A materialized view's schema can't be modified after creation.

+ - Deleting the base table isn't allowed if at least one materialized view is defined on it. All the views must first be deleted and then the base table can be deleted.

+ - Defining materialized views on containers with static columns isn't allowed.

+## Next steps

+> [!div class="nextstepaction"]

+> [Review frequently asked questions (FAQ) about materialized views in API for Cassandra](materialized-views-faq.yml)

+ Title: Get started with Azure Cosmos DB for MongoDB using .NET

+ms.devlang: csharp

+# Get started with Azure Cosmos DB for MongoDB using .NET

+- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free).

+- [.NET 6.0](https://dotnet.microsoft.com/download)

+- [Azure Command-Line Interface (CLI)](/cli/azure/) or [Azure PowerShell](/powershell/azure/)

+- [Azure Cosmos DB for MongoDB resource](quickstart-dotnet.md#create-an-azure-cosmos-db-account)

+1. Create a new .NET Core application in an empty folder using your preferred terminal. For this scenario, you'll use a console application. Use the [``dotnet new``](/dotnet/core/tools/dotnet-new) command to create and name the console app.

+- [Manage databases](how-to-dotnet-manage-databases.md)

+- [Manage collections](how-to-dotnet-manage-collections.md)

+- [Manage documents](how-to-dotnet-manage-documents.md)

+- [Use queries to find documents](how-to-dotnet-manage-queries.md)

+Now that you've connected to an API for MongoDB account, use the next guide to create and manage databases.

+ms.devlang: csharp

+- Keep collection names between 3 and 63 characters long

+- Collection names can only contain lowercase letters, numbers, or the dash (-) character.

+- Container names must start with a lowercase letter or number.

+- [MongoClient.Database.Collection](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html)

+- [MongoClient.Database.Collection](https://mongodb.github.io/node-mongodb-native/4.5/classes/Db.html#collection)

+- [MongoClient.Database.Collection.InsertOne](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#insertOne)

+- [MongoClient.Database.Collection.InsertMany](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#insertMany)

+- [MongoClient.Db.dropCollection](https://mongodb.github.io/node-mongodb-native/4.7/classes/Db.html#dropCollection)

+- [MongoClient.Database.Collection.indexes](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#indexes)

+ms.devlang: csharp

+- Keep database names between 3 and 63 characters long

+- Database names can only contain lowercase letters, numbers, or the dash (-) character.

+- Database names must start with a lowercase letter or number.

+- [MongoClient](https://mongodb.github.io/mongo-csharp-driver/2.16/apidocs/html/T_MongoDB_Driver_MongoClient.htm)

+- [MongoClient.Database](https://mongodb.github.io/mongo-csharp-driver/2.16/apidocs/html/T_MongoDB_Driver_MongoDatabase.htm)

+The following code snippet creates a new database by inserting a document into a collection. Remember, the database won't be created until it's needed for this type of operation.

+- [MongoClient.GetDatabase](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/M_MongoDB_Driver_MongoClient_GetDatabase.htm)

+- [MongoClient.Database.ListDatabaseNames](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/M_MongoDB_Driver_MongoClient_ListDatabaseNames_3.htm)

+A database is removed from the server using the `DropDatabase` method on the DB class.

+- [MongoClient.DropDatabase](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/M_MongoDB_Driver_MongoClient_DropDatabase_1.htm)

+ms.devlang: csharp

+- [MongoClient.Database.Collection.InsertOne](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/M_MongoDB_Driver_IMongoCollection_1_InsertOne_1.htm)

+- [MongoClient.Database.Collection.InsertMany](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/M_MongoDB_Driver_IMongoCollection_1_InsertMany_1.htm)

+To update a document, specify the query filter used to find the document along with a set of properties of the document that should be updated.

+- [MongoClient.Database.Collection.UpdateOne](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/M_MongoDB_Driver_IMongoCollection_1_UpdateOne_1.htm)

+- [MongoClient.Database.Collection.UpdateMany](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/M_MongoDB_Driver_IMongoCollection_1_UpdateMany_1.htm)

+You can perform several different types of operations at once with the **bulkWrite** operation. Learn more about how to [optimize bulk writes for Azure Cosmos DB](optimize-write-performance.md#tune-for-the-optimal-batch-size-and-thread-count).

+- [MongoClient.Database.Collection.BulkWrite](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/M_MongoDB_Driver_IMongoCollection_1_BulkWrite_1.htm)

+ - insertOne

+ - updateOne

+ - updateMany

+ - deleteOne

+

+ - deleteMany

+To delete documents, use a query to define how the documents are found.

+- [MongoClient.Database.Collection.DeleteOne](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/M_MongoDB_Driver_IMongoCollection_1_DeleteOne_1.htm)

+- [MongoClient.Database.Collection.DeleteMany](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/M_MongoDB_Driver_IMongoCollection_1_DeleteMany_1.htm)

+ms.devlang: csharp

+To find documents, use a query filter on the collection to define how the documents are found.

+- [MongoClient.Database.Collection.Find](https://www.mongodb.com/docs/manual/reference/method/db.collection.find/)

+- [FilterDefinition](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/T_MongoDB_Driver_FilterDefinition_1.htm)

+- [FilterDefinitionBuilder](https://mongodb.github.io/mongo-csharp-driver/2.17/apidocs/html/T_MongoDB_Driver_FilterDefinitionBuilder_1.htm)

+ Title: Get started with Azure Cosmos DB for MongoDB using JavaScript

+# Get started with Azure Cosmos DB for MongoDB using JavaScript

+- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free).

+- [Node.js LTS](https://nodejs.org/en/download/)

+- [Azure Command-Line Interface (CLI)](/cli/azure/) or [Azure PowerShell](/powershell/azure/)

+- [Azure Cosmos DB for MongoDB resource](quickstart-nodejs.md#create-an-azure-cosmos-db-account)

+1. Create a new JavaScript application in an empty folder using your preferred terminal. Use the [``npm init``](https://docs.npmjs.com/cli/v8/commands/npm-init) command to begin the prompts to create the `package.json` file. Accept the defaults for the prompts.

+1. Add the [MongoDB](https://www.npmjs.com/package/mongodb) npm package to the JavaScript project. Use the [``npm install package``](https://docs.npmjs.com/cli/v8/commands/npm-install) command specifying the name of the npm package. The `dotenv` package is used to read the environment variables from a `.env` file during local development.

+1. To run the app, use a terminal to navigate to the application directory and run the application.

+To connect with the MongoDB native driver to Azure Cosmos DB, create an instance of the [``MongoClient``](https://mongodb.github.io/node-mongodb-native/4.5/classes/MongoClient.html#connect) class. This class is the starting point to perform all operations against databases.

+1. Add dependencies to reference the MongoDB and DotEnv npm packages.

+1. Define a new instance of the ``MongoClient`` class using the constructor, and [``process.env.``](https://nodejs.org/dist/latest-v8.x/docs/api/process.html#process_process_env) to use the connection string.

+When your application is finished with the connection, remember to close it. The `.close()` call should be after all database calls are made.

+- [Manage databases](how-to-javascript-manage-databases.md)

+- [Manage collections](how-to-javascript-manage-collections.md)

+- [Manage documents](how-to-javascript-manage-documents.md)

+- [Use queries to find documents](how-to-javascript-manage-queries.md)

+Now that you've connected to an API for MongoDB account, use the next guide to create and manage databases.

+- Keep collection names between 3 and 63 characters long

+- Collection names can only contain lowercase letters, numbers, or the dash (-) character.

+- Container names must start with a lowercase letter or number.

+- [MongoClient.Db.Collection](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html)

+- [MongoClient.Db.Collection](https://mongodb.github.io/node-mongodb-native/4.5/classes/Db.html#collection)

+- [MongoClient.Db.Collection.insertOne](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#insertOne)

+- [MongoClient.Db.Collection.insertMany](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#insertMany)

+- [MongoClient.Db.dropCollection](https://mongodb.github.io/node-mongodb-native/4.7/classes/Db.html#dropCollection)

+- [MongoClient.Db.Collection.indexes](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#indexes)

+- [MongoDB](https://www.npmjs.com/package/mongodb)

+- Keep database names between 3 and 63 characters long

+- Database names can only contain lowercase letters, numbers, or the dash (-) character.

+- Database names must start with a lowercase letter or number.

+The database holds the collections and their documents. Use an instance of the `Db` class to access the databases on the server.

+- [MongoClient.Db](https://mongodb.github.io/node-mongodb-native/4.7/classes/Db.html)

+- [MongoClient.Db.Admin](https://mongodb.github.io/node-mongodb-native/4.7/classes/Admin.html)

+- [MongoClient.Db.Admin.listDatabases](https://mongodb.github.io/node-mongodb-native/4.7/classes/Db.html)

+- [MongoClient.Db.Admin.listDatabases](https://mongodb.github.io/node-mongodb-native/4.7/classes/Db.html)

+- [MongoClient.Db.listCollections](https://mongodb.github.io/node-mongodb-native/4.7/classes/Db.html#listCollections)

+- [MongoClient.Db.Collection](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html)

+- [MongoClient.Db.Collection.countDocuments](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#countDocuments)

+- [``MongoClient.Db``](https://mongodb.github.io/node-mongodb-native/4.5/classes/Db.html)

+A database is created when it's accessed. The most common way to access a new database is to add a document to a collection. In one line of code using chained objects, the database, collection, and doc are created.

+A database is removed from the server using the dropDatabase method on the DB class.

+- [DB.dropDatabase](https://mongodb.github.io/node-mongodb-native/4.7/classes/Db.html#dropDatabase)

+- [MongoClient.Db.Collection.insertOne](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#insertOne)

+- [MongoClient.Db.Collection.insertMany](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#insertMany)

+- [ObjectId](https://mongodb.github.io/node-mongodb-native/4.7/classes/ObjectId.html)

+To update a document, specify the query used to find the document along with a set of properties of the document that should be updated. You can choose to upsert the document, which inserts the document if it doesn't already exist.

+- [MongoClient.Db.Collection.updateOne](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#updateOne)

+- [MongoClient.Db.Collection.updateMany](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#updateMany)

+You can perform several operations at once with the **bulkWrite** operation. Learn more about how to [optimize bulk writes for Azure Cosmos DB](optimize-write-performance.md#tune-for-the-optimal-batch-size-and-thread-count).

+- [MongoClient.Db.Collection.bulkWrite](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#bulkWrite)

+ - insertOne

+ - updateOne

+ - updateMany

+ - deleteOne

+ - deleteMany

+To delete documents, use a query to define how the documents are found.

+- [MongoClient.Db.Collection.deleteOne](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#deleteOne)

+- [MongoClient.Db.Collection.deleteMany](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#deleteMany)

+To find documents, use a query to define how the documents are found.

+- [MongoClient.Db.Collection.findOne](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#findOne)

+- [MongoClient.Db.Collection.find](https://mongodb.github.io/node-mongodb-native/4.7/classes/Collection.html#find)

+- [FindCursor](https://mongodb.github.io/node-mongodb-native/4.7/classes/FindCursor.html)

+Aggregation pipelines are useful to isolate expensive query computation, transformations, and other processing on your Azure Cosmos DB server, instead of performing these operations on the client.

+For specific **aggregation pipeline support**, refer to the following:

+- [Version 4.2](feature-support-42.md#aggregation-pipeline)

+- [Version 4.0](feature-support-40.md#aggregation-pipeline)

+- [Version 3.6](feature-support-36.md#aggregation-pipeline)

+- [Version 3.2](feature-support-32.md#aggregation-pipeline)

+A pipeline is an array with a series of stages as JSON objects.

+- $match - find documents

+- $addFields - add field to cursor, usually from previous stage

+- $limit - limit the number of results returned in cursor

+- $project - pass along new or existing fields, can be computed fields

+- $group - group results by a field or fields in pipeline

+- $sort - sort results

+The pipeline is aggregated to produce an iterable cursor.

+Use a pipeline to keep data processing on the server before returning to the client.

+### Example product data

+Use the following [sample code](https://github.com/Azure-Samples/cosmos-db-mongodb-api-javascript-samples/blob/main/280-aggregation/average-price-in-each-product-subcategory.js) to report on average price in each product subcategory.

+Use the following [sample code](https://github.com/Azure-Samples/cosmos-db-mongodb-api-javascript-samples/blob/main/280-aggregation/bike-types-and-price-ranges.js) to report on the `Bikes` subcategory.

+ Title: Get started with Azure Cosmos DB for MongoDB and Python

+description: Get started developing a Python application that works with Azure Cosmos DB for MongoDB. This article helps you learn how to set up a project and configure access to an Azure Cosmos DB for MongoDB database.

+ms.devlang: python

+# Get started with Azure Cosmos DB for MongoDB and Python

+This article shows you how to connect to Azure Cosmos DB for MongoDB using the PyMongo driver package. Once connected, you can perform operations on databases, collections, and docs.

+> [!NOTE]

+> The [example code snippets](https://github.com/Azure-Samples/azure-cosmos-db-mongodb-python-getting-started) are available on GitHub as a Python project.

+This article shows you how to communicate with the Azure Cosmos DBΓÇÖs API for MongoDB by using one of the open-source MongoDB client drivers for Python, [PyMongo](https://www.mongodb.com/docs/drivers/pymongo/).

+## Prerequisites

+* An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free).

+* [Python 3.8+](https://www.python.org/downloads/)

+* [Azure Command-Line Interface (CLI)](/cli/azure/) or [Azure PowerShell](/powershell/azure/)

+* [Azure Cosmos DB for MongoDB resource](quickstart-python.md#create-an-azure-cosmos-db-account)

+## Create a new Python app

+1. Create a new empty folder using your preferred terminal and change directory to the folder.

+ > [!NOTE]

+ > If you just want the finished code, download or fork and clone the [example code snippets](https://github.com/Azure-Samples/azure-cosmos-db-mongodb-python-getting-started) repo that has the full example. You can also `git clone` the repo in Azure Cloud Shell to walk through the steps shown in this quickstart.

+2. Create a *requirements.txt* file that lists the [PyMongo](https://www.mongodb.com/docs/drivers/pymongo/) and [python-dotenv](https://pypi.org/project/python-dotenv/) packages. The `dotenv` package is used to read the environment variables from a `.env` file during local development.

+ ```text

+ # requirements.txt

+ pymongo

+ python-dotenv

+ ```

+3. Create a virtual environment and install the packages.

+ #### [Windows](#tab/venv-windows)

+

+ ```bash

+ # py -3 uses the global python interpreter. You can also use python3 -m venv .venv.

+ py -3 -m venv .venv

+ source .venv/Scripts/activate

+ pip install -r requirements.txt

+ ```

+

+ #### [Linux / macOS](#tab/venv-linux+macos)

+

+ ```bash

+ python3 -m venv .venv

+ source .venv/bin/activate

+ pip install -r requirements.txt

+ ```

+

+

+## Connect with PyMongo driver to Azure Cosmos DB for MongoDB

+To connect with the PyMongo driver to Azure Cosmos DB, create an instance of the [MongoClient](https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient) object. This class is the starting point to perform all operations against databases.

+The most common constructor for **MongoClient** requires just the `host` parameter, which in this article is set to the `COSMOS_CONNECTION_STRING` environment variable. There are other optional parameters and keyword parameters you can use in the constructor. Many of the optional parameters can also be specified with the `host` parameter. If the same option is passed in with `host` and as a parameter, the parameter takes precedence.

+Refer to the [Troubleshooting guide](error-codes-solutions.md) for connection issues.

+## Get resource name

+In the commands below, we show *msdocs-cosmos* as the resource group name. Change the name as appropriate for your situation.

+### [Azure CLI](#tab/azure-cli)

+### [PowerShell](#tab/azure-powershell)

+### [Portal](#tab/azure-portal)

+Skip this step and use the information for the portal in the next step.

+## Retrieve your connection string

+### [Azure CLI](#tab/azure-cli)

+### [PowerShell](#tab/azure-powershell)

+### [Portal](#tab/azure-portal)

+## Configure environment variables

+## Create MongoClient with connection string

+1. Add dependencies to reference the [PyMongo](https://www.mongodb.com/docs/drivers/pymongo/) and [python-dotenv](https://pypi.org/project/python-dotenv/) packages.

+ :::code language="python" source="~/azure-cosmos-db-mongodb-python-getting-started/101-client-connection-string/run.py" id="package_dependencies":::

+2. Define a new instance of the `MongoClient` class using the constructor and the connection string read from an environment variable.

+ :::code language="python" source="~/azure-cosmos-db-mongodb-python-getting-started/101-client-connection-string/run.py" id="client_credentials":::

+For more information on different ways to create a ``MongoClient`` instance, see [Making a Connection with MongoClient](https://pymongo.readthedocs.io/en/stable/tutorial.html#making-a-connection-with-mongoclient).

+## Close the MongoClient connection

+When your application is finished with the connection, remember to close it. That `.close()` call should be after all database calls are made.

+```python

+client.close()

+```

+## Use MongoDB client classes with Azure Cosmos DB for API for MongoDB

+Let's look at the hierarchy of resources in the API for MongoDB and the object model that's used to create and access these resources. The API for MongoDB creates resources in the following order:

+* [MongoClient](https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html) - The first step when working with PyMongo is to create a MongoClient to connect to Azure Cosmos DB's API for MongoDB. The client object is used to configure and execute requests against the service.

+* [Database](https://pymongo.readthedocs.io/en/stable/api/pymongo/database.html) - Azure Cosmos DB's API for MongoDB can support one or more independent databases.

+* [Collection](https://pymongo.readthedocs.io/en/stable/api/pymongo/database.html) - A database can contain one or more collections. A collection is a group of documents stored in MongoDB, and can be thought of as roughly the equivalent of a table in a relational database.

+* [Document](https://pymongo.readthedocs.io/en/stable/tutorial.html#documents) - A document is a set of key-value pairs. Documents have dynamic schema. Dynamic schema means that documents in the same collection don't need to have the same set of fields or structure. And common fields in a collection's documents may hold different types of data.

+To learn more about the hierarchy of entities, see the [Azure Cosmos DB resource model](../account-databases-containers-items.md) article.

+## See also

+- [PyPI Package](https://pypi.org/project/azure-cosmos/)

+- [API reference](https://www.mongodb.com/docs/drivers/python/)

+## Next steps

+Now that you've connected to an API for MongoDB account, use the next guide to create and manage databases.

+> [!div class="nextstepaction"]

+> [Create a database in Azure Cosmos DB for MongoDB using Python](how-to-python-manage-databases.md)

+ Title: Manage a MongoDB database using Python

+description: Learn how to manage your Azure Cosmos DB resource when it provides the API for MongoDB with a Python SDK.

+ms.devlang: python

+# Manage a MongoDB database using Python

+Your MongoDB server in Azure Cosmos DB is available from the common Python packages for MongoDB such as:

+* [PyMongo](https://www.mongodb.com/docs/drivers/pymongo/) for synchronous Python applications and used in this article.

+* [Motor](https://www.mongodb.com/docs/drivers/motor/) for asynchronous Python applications.

+> [!NOTE]

+> The [example code snippets](https://github.com/Azure-Samples/azure-cosmos-db-mongodb-python-getting-started) are available on GitHub as a Python project.

+## Name a database

+In Azure Cosmos DB, a database is analogous to a namespace. When you create a database, the database name forms a segment of the URI used to access the database resource and any child resources.

+Here are some quick rules when naming a database:

+* Keep database names between 3 and 63 characters long

+* Database names can only contain lowercase letters, numbers, or the dash (-) character.

+* Database names must start with a lowercase letter or number.

+Once created, the URI for a database is in this format:

+`https://<cosmos-account-name>.documents.azure.com/dbs/<database-name>`

+## Get database instance

+The database holds the collections and their documents. To access a database, use attribute style access or dictionary style access of the MongoClient. For more information, see [Getting a Database](https://pymongo.readthedocs.io/en/stable/tutorial.html#getting-a-database).

+The following code snippets assume you've already created your [client connection](how-to-python-get-started.md#create-mongoclient-with-connection-string) and that you [close your client connection](how-to-python-get-started.md#close-the-mongoclient-connection) after these code snippets.

+## Get server information

+Access server info with the [server_info](https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient.server_info) method of the MongoClient class. You don't need to specify the database name to get this information. The information returned is specific to MongoDB and doesn't represent the Azure Cosmos DB platform itself.

+You can also list databases using the [MongoClient.list_database_names](https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient.list_database_names) method and issue a [MongoDB command](https://www.mongodb.com/docs/manual/reference/command/nav-diagnostic/) to a database with the [MongoClient.db.command](https://pymongo.readthedocs.io/en/stable/api/pymongo/database.html#pymongo.database.Database.command) method.

+The preceding code snippet displays output similar to the following example console output:

+## Does database exist?

+The PyMongo driver for Python creates a database if it doesn't exist when you access it. However, we recommend that instead you use the [MongoDB extension commands](/azure/cosmos-db/mongodb/custom-commands) to manage data stored in Azure Cosmos DBΓÇÖs API for MongoDB. To create a new database if it doesn't exist, use the [create database extension](/azure/cosmos-db/mongodb/custom-commands#create-database) as shown in the following code snippet.

+To see if the database already exists before using it, get the list of current databases with the [list_database_names](https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient.list_database_names) method.

+The preceding code snippet displays output similar to the following example console output:

+## Get list of databases, collections, and document count

+When you manage your MongoDB server programmatically, it's helpful to know what databases and collections are on the server and how many documents in each collection. For more information, see:

+* [Getting a database](https://pymongo.readthedocs.io/en/stable/tutorial.html#getting-a-database)

+* [Getting a collection](https://pymongo.readthedocs.io/en/stable/tutorial.html#getting-a-collection)

+* [Counting documents](https://pymongo.readthedocs.io/en/stable/tutorial.html#counting)

+The preceding code snippet displays output similar to the following example console output:

+## Get database object instance

+If a database doesn't exist, the PyMongo driver for Python creates it when you access it. However, we recommend that instead you use the [MongoDB extension commands](/azure/cosmos-db/mongodb/custom-commands) to manage data stored in Azure Cosmos DBΓÇÖs API for MongoDB. The pattern is shown above in the section [Does database exist?](#does-database-exist).

+When working with PyMongo, you access databases using attribute style access on MongoClient instances. Once you have a database instance, you can use database level operations as shown below.

+```python

+collections = client[db].list_collection_names()

+```

+For an overview of working with databases using the PyMongo driver, see [Database level operations](https://pymongo.readthedocs.io/en/stable/api/pymongo/database.html#pymongo.database.Database).

+## Drop a database

+A database is removed from the server using the [drop_database](https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient.drop_database) method of the MongoClient.

+The preceding code snippet displays output similar to the following example console output:

+## See also

+- [Get started with Azure Cosmos DB for MongoDB and Python](how-to-python-get-started.md)

+ms.devlang: csharp

+- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free).

+- [.NET 6.0](https://dotnet.microsoft.com/download)

+- [Azure Command-Line Interface (CLI)](/cli/azure/install-azure-cli) or [Azure PowerShell](/powershell/azure/install-az-ps)

+- In a terminal or command window, run ``dotnet --list-sdks`` to check that .NET 6.x is one of the available versions.

+- Run ``az --version`` (Azure CLI) or ``Get-Module -ListAvailable AzureRM`` (Azure PowerShell) to check that you have the appropriate Azure command-line tools installed.

+- [``MongoClient``](https://mongodb.github.io/mongo-csharp-driver/2.16/apidocs/html/T_MongoDB_Driver_MongoClient.htm) - This class provides a client-side logical representation for the API for MongoDB layer on Azure Cosmos DB. The client object is used to configure and execute requests against the service.

+- [``MongoDatabase``](https://mongodb.github.io/mongo-csharp-driver/2.16/apidocs/html/T_MongoDB_Driver_MongoDatabase.htm) - This class is a reference to a database that may, or may not, exist in the service yet. The database is validated server-side when you attempt to access it or perform an operation against it.

+- [``Collection``](https://mongodb.github.io/mongo-csharp-driver/2.16/apidocs/html/T_MongoDB_Driver_MongoCollection.htm) - This class is a reference to a collection that also may not exist in the service yet. The collection is validated server-side when you attempt to work with it.

+- [Authenticate the client](#authenticate-the-client)

+- [Create a database](#create-a-database)

+- [Create a container](#create-a-collection)

+- [Create an item](#create-an-item)

+- [Get an item](#get-an-item)

+- [Query items](#query-items)

+ Title: Quickstart - Azure Cosmos DB for MongoDB driver for MongoDB

+description: Learn how to build a Node.js app to manage Azure Cosmos DB for MongoDB account resources and data in this quickstart.

+# Quickstart: Azure Cosmos DB for MongoDB driver for Node.js

+- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free).

+- [Node.js LTS](https://nodejs.org/en/download/)

+- [Azure Command-Line Interface (CLI)](/cli/azure/) or [Azure PowerShell](/powershell/azure/)

+- In a terminal or command window, run ``node --version`` to check that Node.js is one of the LTS versions.

+- Run ``az --version`` (Azure CLI) or ``Get-Module -ListAvailable AzureRM`` (Azure PowerShell) to check that you have the appropriate Azure command-line tools installed.

+- [``MongoClient``](https://mongodb.github.io/node-mongodb-native/4.5/classes/MongoClient.html) - This class provides a client-side logical representation for the API for MongoDB layer on Azure Cosmos DB. The client object is used to configure and execute requests against the service.

+- [``Db``](https://mongodb.github.io/node-mongodb-native/4.5/classes/Db.html) - This class is a reference to a database that may, or may not, exist in the service yet. The database is validated server-side when you attempt to access it or perform an operation against it.

+- [``Collection``](https://mongodb.github.io/node-mongodb-native/4.5/classes/Collection.html) - This class is a reference to a collection that also may not exist in the service yet. The collection is validated server-side when you attempt to work with it.

+- [Authenticate the client](#authenticate-the-client)

+- [Get database instance](#get-database-instance)

+- [Get collection instance](#get-collection-instance)

+- [Chained instances](#chained-instances)

+- [Create an index](#create-an-index)

+- [Create a doc](#create-a-doc)

+- [Get an doc](#get-a-doc)

+- [Query docs](#query-docs)

+- An _id property for the unique identifier of the product.

+- A *category* property. This property can be used as the logical partition key.

+- A *name* property.

+- An inventory *quantity* property.

+- A *sale* property, indicating whether the product is on sale.

+- If you get an error such as `The index path corresponding to the specified order-by item is excluded.`, make sure you [created the index](#create-an-index).

+- Keep container names between 3 and 63 characters long

+- Container names can only contain lowercase letters, numbers, or the dash (-) character.

+- Container names must start with a lowercase letter or number.

+- [``CreateContainerAsync``](#create-a-container-asynchronously)

+- [``CreateContainerIfNotExistsAsync``](#create-a-container-asynchronously-if-it-doesnt-already-exist)

+- Keep database names between 3 and 63 characters long

+- Database names can only contain lowercase letters, numbers, or the dash (-) character.

+- Database names must start with a lowercase letter or number.

+- [``CreateDatabaseAsync``](#create-a-database-asynchronously)

+- [``CreateDatabaseIfNotExistsAsync``](#create-a-database-asynchronously-if-it-doesnt-already-exist)

+- [``CreateItemAsync<>``](#create-an-item-asynchronously)

+- [``ReplaceItemAsync<>``](#replace-an-item-asynchronously)

+- [``UpsertItemAsync<>``](#create-or-replace-an-item-asynchronously)

+ Title: Get started with Azure Cosmos DB for NoSQL using .NET

+# Get started with Azure Cosmos DB for NoSQL using .NET

+- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free).

+- Azure Cosmos DB for NoSQL account. [Create a API for NoSQL account](how-to-create-account.md).

+- [.NET 6.0 or later](https://dotnet.microsoft.com/download)

+- [Azure Command-Line Interface (CLI)](/cli/azure/) or [Azure PowerShell](/powershell/azure/)

+To connect to the API for NoSQL of Azure Cosmos DB, create an instance of the [``CosmosClient``](/dotnet/api/microsoft.azure.cosmos.cosmosclient) class. This class is the starting point to perform all operations against databases. There are three core ways to connect to an API for NoSQL account using the **CosmosClient** class:

+- [Connect with a API for NoSQL endpoint and read/write key](#connect-with-an-endpoint-and-key)

+- [Connect with a API for NoSQL connection string](#connect-with-a-connection-string)

+- [Connect with Azure Active Directory](#connect-using-the-microsoft-identity-platform)

+- The API for NoSQL account, which is the unique top-level namespace for your Azure Cosmos DB data.

+- Databases, which organize the containers in your account.

+- Containers, which contain a set of individual items in your database.

+- Items, which represent a JSON document in your container.

+- [Package (NuGet)](https://www.nuget.org/packages/Microsoft.Azure.Cosmos)

+- [Samples](samples-dotnet.md)

+- [API reference](/dotnet/api/microsoft.azure.cosmos)

+- [Library source code](https://github.com/Azure/azure-cosmos-dotnet-v3)

+- [Give Feedback](https://github.com/Azure/azure-cosmos-dotnet-v3/issues)

+Now that you've connected to an API for NoSQL account, use the next guide to create and manage databases.

+- [``GetItemQueryIterator<>``](#query-items-using-a-sql-query-asynchronously)

+- [``GetItemLinqQueryable<>``](#query-items-using-linq-asynchronously)

+- [``ReadItemAsync<>``](#read-an-item-asynchronously)

+- [``ReadItemStreamAsync<>``](#read-an-item-as-a-stream-asynchronously)

+- [``ReadManyItemsAsync<>``](#read-multiple-items-asynchronously)

+ Title: Quickstart - Azure Cosmos DB for NoSQL client library for Node.js

+description: Learn how to build a Node.js app to manage Azure Cosmos DB for NoSQL account resources in this quickstart.

+# Quickstart - Azure Cosmos DB for NoSQL client library for Node.js

+- In a terminal or command window, run ``node --version`` to check that the Node.js version is one of the current long term support (LTS) versions.

+- Run ``az --version`` (Azure CLI) or ``Get-Module -ListAvailable AzureRM`` (Azure PowerShell) to check that you have the appropriate Azure command-line tools installed.

+Add the following variables to manage unique database and container names and the [**partition key (`pk`)**](../partitioning-overview.md).

+- [``CosmosClient``](/javascript/api/@azure/cosmos/cosmosclient) - This class provides a client-side logical representation for the Azure Cosmos DB service. The client object is used to configure and execute requests against the service.

+- [``Database``](/javascript/api/@azure/cosmos/database) - This class is a reference to a database that may, or may not, exist in the service yet. The database is validated server-side when you attempt to access it or perform an operation against it.

+- [``Container``](/javascript/api/@azure/cosmos/container) - This class is a reference to a container that also may not exist in the service yet. The container is validated server-side when you attempt to work with it.

+- [``SqlQuerySpec``](/javascript/api/@azure/cosmos/sqlqueryspec) - This interface represents a SQL query and any query parameters.

+- [``QueryIterator<>``](/javascript/api/@azure/cosmos/queryiterator) - This class represents an iterator that can track the current page of results and get a new page of results.

+- [``FeedResponse<>``](/javascript/api/@azure/cosmos/feedresponse) - This class represents a single page of responses from the iterator.

+- [Authenticate the client](#authenticate-the-client)

+- [Create a database](#create-a-database)

+- [Create a container](#create-a-container)

+- [Create an item](#create-an-item)

+- [Get an item](#get-an-item)

+- [Query items](#query-items)

+> [Tutorial: Build a Node.js console app](sql-api-nodejs-get-started.md)

+- An Azure account with an active subscription. Without a credit card or an Azure subscription, you can set up a free [Try Azure Cosmos DB account](https://aka.ms/trycosmosdb).

+- Azure Cosmos DB for NoSQL account. [Create a API for NoSQL account](how-to-create-account.md).

+- [.NET 6.0 or later](https://dotnet.microsoft.com/download)

+- [Azure Command-Line Interface (CLI)](/cli/azure/) or [Azure PowerShell](/powershell/azure/)

+ Title: Examples for Azure Cosmos DB for NoSQL SDK for JS

+ms.devlang: javascript

+# Examples for Azure Cosmos DB for NoSQL SDK for JS

+- Links to the tasks in each of the Node.js example project files.

+- Links to the related API reference content.

+## Prerequisites

+The [DatabaseManagement](https://github.com/Azure/azure-cosmos-js/blob/master/samples/DatabaseManagement.ts) file shows how to perform the CRUD operations on the database. To learn about the Azure Cosmos DB databases before running the following samples, see [Working with databases, containers, and items](../account-databases-containers-items.md) conceptual article.

+| [Create a database if it doesn't exist](https://github.com/Azure/azure-cosmos-js/blob/master/samples/DatabaseManagement.ts#L12-L14) |[Databases.createIfNotExists](/javascript/api/@azure/cosmos/databases#createifnotexists-databaserequest--requestoptions-) |

+The [ContainerManagement](https://github.com/Azure/azure-cosmos-js/blob/master/samples/ContainerManagement.ts) file shows how to perform the CRUD operations on the container. To learn about the Azure Cosmos DB collections before running the following samples, see [Working with databases, containers, and items](../account-databases-containers-items.md) conceptual article.

+| [Create a container if it doesn't exist](https://github.com/Azure/azure-cosmos-js/blob/master/samples/ContainerManagement.ts#L14-L15) |[Containers.createIfNotExists](/javascript/api/@azure/cosmos/containers#createifnotexists-containerrequest--requestoptions-) |

+The [ItemManagement](https://github.com/Azure/azure-cosmos-js/blob/master/samples/ItemManagement.ts) file shows how to perform the CRUD operations on the item. To learn about the Azure Cosmos DB documents before running the following samples, see [Working with databases, containers, and items](../account-databases-containers-items.md) conceptual article.

+| [Read item only if item has changed](https://github.com/Azure/azure-cosmos-js/blob/master/samples/ItemManagement.ts#L45-L56) |[Item.read](/javascript/api/%40azure/cosmos/item) - [RequestOptions.accessCondition](/javascript/api/%40azure/cosmos/requestoptions#accesscondition) |

+| [Replace item with conditional ETag check](https://github.com/Azure/azure-cosmos-js/blob/master/samples/ItemManagement.ts#L98-L135) |[Item.replace](/javascript/api/%40azure/cosmos/item) - [RequestOptions.accessCondition](/javascript/api/%40azure/cosmos/requestoptions#accesscondition) |

+The [IndexManagement](https://github.com/Azure/azure-cosmos-js/blob/master/samples/IndexManagement.ts) file shows how to manage indexing. To learn about indexing in Azure Cosmos DB before running the following samples, see [indexing policies](../index-policy.md), [indexing types](../index-overview.md#index-types), and [indexing paths](../index-policy.md#include-exclude-paths) conceptual articles.

+| [Create a container with default indexPolicy, then update the container online](https://github.com/Azure/azure-cosmos-js/blob/master/samples/IndexManagement.ts#L13-L15) |[Containers.create](/javascript/api/%40azure/cosmos/containers)

+The [index.ts](https://github.com/Azure/azure-cosmos-js/blob/master/samples/ServerSideScripts/index.ts) file of the [ServerSideScripts](https://github.com/Azure/azure-cosmos-js/tree/master/samples/ServerSideScripts) project shows how to perform the following tasks. To learn about Server-side programming in Azure Cosmos DB before running the following samples, see [Stored procedures, triggers, and user-defined functions](stored-procedures-triggers-udfs.md) conceptual article.

+- If all you know is the number of vCores and servers in your existing database cluster, see [estimating request units using vCores or vCPUs](../convert-vcore-to-request-unit.md)

+- If you know typical request rates for your current database workload, read about [estimating request units using Azure Cosmos DB capacity planner](estimate-ru-with-capacity-planner.md)

+ Title: Examples for Azure Cosmos DB for NoSQL SDK for Python

+# Examples for Azure Cosmos DB for NoSQL SDK for Python

+- [Create an instance of the ``TableEntity`` class](#use-a-built-in-class)

+- [Implement the ``ITableEntity`` interface](#implement-interface)

+- Keep table names between 3 and 63 characters long

+- Table names can only contain lowercase letters, numbers, or the dash (-) character.

+- Table names must start with a lowercase letter or number.

+- [``CreateAsync``](#create-a-table-asynchronously)

+- [``CreateIfNotExistsAsync``](#create-a-table-asynchronously-if-it-doesnt-already-exist)

+ Title: Get started with Azure Cosmos DB for Table using .NET

+# Get started with Azure Cosmos DB for Table using .NET

+- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free).

+- Azure Cosmos DB for Table account. [Create a API for Table account](how-to-create-account.md).

+- [.NET 6.0 or later](https://dotnet.microsoft.com/download)

+- [Azure Command-Line Interface (CLI)](/cli/azure/) or [Azure PowerShell](/powershell/azure/)

+- [Connect with a API for Table connection string](#connect-with-a-connection-string)

+- The API for Table account, which is the unique top-level namespace for your Azure Cosmos DB data.

+- Tables, which contain a set of individual items in your account.

+- Items, which represent an individual item in your table.

+- [Package (NuGet)](https://www.nuget.org/packages/Azure.Data.Tables/)

+- [Samples](samples-dotnet.md)

+- [API reference](/dotnet/api/azure.data.tables)

+- [Library source code](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/tables/Azure.Data.Tables)

+- [Give Feedback](https://github.com/Azure/azure-sdk-for-net/issues)

+- [Return a ``TableEntity`` object using ``GetEntityAsync<>``](#read-an-item-using-a-built-in-class)

+- [Return an object of your own type using ``GetEntityAsync<>``](#read-an-item-using-your-own-type)

+ms.devlang: csharp

+> Try Azure Cosmos DB supports global distribution in only the **East US**, **North Europe**, **Southeast Asia**, and **North Central US** regions. Azure support tickets can't be created for Try Azure Cosmos DB accounts. However, support is provided for subscribers with existing support plans.

+> Try Azure Cosmos DB supports global distribution in only the **East US**, **North Europe**, **Southeast Asia**, and **North Central US** regions. Azure support tickets can't be created for Try Azure Cosmos DB accounts. However, support is provided for subscribers with existing support plans.

+> Try Azure Cosmos DB supports global distribution in only the **East US**, **North Europe**, **Southeast Asia**, and **North Central US** regions. Azure support tickets can't be created for Try Azure Cosmos DB accounts. However, support is provided for subscribers with existing support plans.

+ Title: Group and allocate costs using tag inheritance

+description: This article explains how to group costs using tag inheritance.

+# Group and allocate costs using tag inheritance

+Azure tags are widely used to group costs to align with different business units, engineering environments, and cost departments. Tags provide the visibility needed for businesses to manage and allocate costs across the different groups.

+This article explains how to use the tag inheritance setting in Cost Management. When enabled, tag inheritance applies resource group and subscription tags to child resource usage records. You don't have to tag every resource or rely on resources that emit usage to have their own tags.

+Tag inheritance is available for customers with an Enterprise Account (EA) or a Microsoft Customer Agreement (MCA) account.

+## Required permissions

+- For subscriptions:

+ - Cost Management reader to view

+ - Cost Management Contributor to edit

+- For EA billing accounts:

+ - Enterprise Administrator (read-only) to view

+ - Enterprise Administrator to edit

+- For MCA billing profiles:

+ - Billing profile reader to view

+ - Billing profile contributor to edit

+## Enable tag inheritance

+You can enable the tag inheritance setting in the Azure portal. You apply the setting at the EA billing account, MCA billing profile, and subscription scopes. After the setting is enabled, all resource group and subscription tags are automatically applied to child resource usage records.

+To enable tag inheritance in the Azure portal:

+1. In the Azure portal, navigate to Cost Management.

+2. Select a scope.

+3. In the left menu under **Settings**, select either **Manage billing account** or **Manage subscription**, depending on your scope.

+4. Under **Tag inheritance**, select **Edit**.

+ :::image type="content" source="./media/enable-tag-inheritance/edit-tag-inheritance.png" alt-text="Screenshot showing the Edit option for Tag inheritance." lightbox="./media/enable-tag-inheritance/edit-tag-inheritance.png" :::

+5. In the Tag inheritance (Preview) window, select **Automatically apply subscription and resource group tags to new data**.

+ :::image type="content" source="./media/enable-tag-inheritance/automatically-apply-tags-new-usage-data.png" alt-text="Screenshot showing the Automatically apply subscription and resource group tags to new data option." lightbox="./media/enable-tag-inheritance/automatically-apply-tags-new-usage-data.png" :::

+Here's an example diagram showing how a tag is inherited.

+## Choose between resource and inherited tags

+When a resource tag matches the resource group or subscription tag being applied, the resource tag is applied to its usage record by default. You can change the default behavior to have the subscription or resource group tag override the resource tag.

+In the Tag inheritance window, select the **Use the subscription or resource group tag** option.

+Let's look at an example of how a resource tag gets applied. In the following diagram, resource 4 and resource group 2 have the same tag: *App*. Because the user chose to keep the resource tag, usage record 4 is updated with the resource tag value *E2E*.

+Let's look at another example where a resource tag gets overridden. In the following diagram, resource 4 and resource group 2 have the same tag: **App**. Because the user chose to use the resource group or subscription tag, usage record 4 is updated with the resource group tag value, which is *backend*.

+## Usage record updates

+After the tag inheritance setting is enabled, it takes about 8-24 hours for the child resource usage records to get updated with subscription and resource group tags. The usage records are updated for the current month using the existing subscription and resource group tags.

+For example, if the tag inheritance setting is enabled on October 20, child resource usage records are updated from October 1 using the tags that existed on October 20.

+Similarly, if the tag inheritance setting is disabled, the inherited tags will be removed from the usage records for the current month.

+> [!NOTE]

+> If there are purchases or resources that donΓÇÖt emit usage at a subscription scope, they will not have the subscription tags applied even if the setting is enabled.

+## View costs grouped by tags

+You can use cost analysis to view the costs grouped by tags.

+1. In the Azure portal, navigate to **Cost Management**.

+1. In the left menu, select **Cost Analysis**.

+1. Select a scope.

+1. In the **Group by** list, select the tag you want to view costs for.

+Here's an example showing costs for the *org* tag.

+You can also view the inherited tags by downloading your Azure usage. For more information, see [View and download your Azure usage and charges](../understand/download-azure-daily-usage.md).

+## Next steps

+- Learn how to [split shared costs](allocate-costs.md).

+- [Group and allocate costs using tag inheritance](enable-tag-inheritance.md) to apply resource group and subscription tags to child resource usage records. If you were using Azure policy to enforce tagging for cost reporting, consider enabling the tag inheritance setting for easier management and more flexibility.

+### API key generation

+Microsoft partners provide services that help customers achieve business and mission objectives using Microsoft products. When a partner acts on behalf of the customer to manage, configure, and support Azure services, the partner users will need access to the customerΓÇÖs environment. When partners use Partner Admin Link (PAL), they can associate their partner network ID with the credentials used for service delivery.

+However, if you're managing customer resources through Azure Lighthouse, you should create the link in your service provider tenant, using an account that has access to the customer resources. For more information, see [Link your partner ID to track your impact on delegated resources](../../lighthouse/how-to/partner-earned-credit.md).

+In order for Azure Lighthouse activities to be recognized, you need to associate your Partner ID with at least one user account that has access to each of your onboarded subscriptions. The association is needed in your service provider tenant rather than in each customer tenant. For simplicity, we recommend creating a service principal account in your tenant, associating it with your Partner ID, then granting it access to every customer you onboard with an [Azure built-in role that is eligible for partner earned credit](/partner-center/azure-roles-perms-pec). For more information, see [Link your partner ID to track your impact on delegated resources](../../lighthouse/how-to/partner-earned-credit.md).

+The PAL association to existing credentials provides no new customer data to Microsoft. It simply provides the information to Microsoft where a partner is actively involved in a customerΓÇÖs Azure environment. Microsoft can attribute influence and Azure consumed revenue from customer environment to partner organization based on the account's permissions (Azure role) and scope (Management Group, Subscription, Resource Group, Resource) provided to the partner by customer.

+PAL association only adds partnerΓÇÖs ID to the credential already provisioned and it doesn't alter any permissions (Azure role) or provide other Azure service data to partner or Microsoft.

+**What happens if the PAL identity is deleted?**

+If the partner network ID, also called MPN ID, is deleted, then all the recognition mechanisms including Azure Consumed Revenue (ACR) attribution stops working.

+ - A new billing account for a Microsoft Online Services Program can have a maximum of 5 subscriptions. However, subscriptions transferred to the new billing account don't count against the limit.

+|Storage account owned by a different tenant |One or more storage accounts were moved under a different tenant. Resolve the error and resume data copy, or skip to data erasure and complete the order.|**Storage accounts moved to a different tenant**<br>Storage accounts: &lt;*storage accounts list*&gt; were moved to a different tenant. Restore the account to the original tenant and then confirm to resume data copy.<br>[Learn more on how to move storage accounts](../storage/common/storage-account-move.md). |

+To learn more about implementation details such as supported operating systems, feature availability, outbound proxy, see [Defender for Containers feature availability](supported-machines-endpoint-solutions-clouds-containers.md).

+ Your physical media must have a minimum of 4-GB storage.

+#### Gateway checks

+Use the `route` command to show the gateway's IP address. For example:

+``` CLI

+<root@xsense:/# route -n

+Kernel IP routing table

+Destination Gateway Genmask Flags Metric Ref Use Iface

+0.0.0.0 172.18.0.1 0.0.0.0 UG 0 0 0 eth0

+172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

+>

+```

+Use the `arp -a` command to verify that there is a binding between the MAC address and the IP address of the default gateway. For example:

+``` CLI

+<root@xsense:/# arp -a

+cusalvtecca101-gi0-02-2851.network.microsoft.com (172.18.0.1) at 02:42:b0:3a:e8:b5 [ether] on eth0

+mariadb_22.2.6.27-r-c64cbca.iot_network_22.2.6.27-r-c64cbca (172.18.0.5) at 02:42:ac:12:00:05 [ether] on eth0

+redis_22.2.6.27-r-c64cbca.iot_network_22.2.6.27-r-c64cbca (172.18.0.3) at 02:42:ac:12:00:03 [ether] on eth0

+>

+```

+#### DNS checks

+Use the `cat /etc/resolv.conf` command to find the IP address that's configured for DNS traffic. For example:

+``` CLI

+<root@xsense:/# cat /etc/resolv.conf

+search reddog.microsoft.com

+nameserver 127.0.0.11

+options ndots:0

+>

+```

+Use the `host` command to resolve an FQDN. For example:

+``` CLI

+<root@xsense:/# host www.apple.com

+www.apple.com is an alias for www.apple.com.edgekey.net.

+www.apple.com.edgekey.net is an alias for www.apple.com.edgekey.net.globalredir.akadns.net.

+www.apple.com.edgekey.net.globalredir.akadns.net is an alias for e6858.dscx.akamaiedge.net.

+e6858.dscx.akamaiedge.net has address 72.246.148.202

+e6858.dscx.akamaiedge.net has IPv6 address 2a02:26f0:5700:1b4::1aca

+e6858.dscx.akamaiedge.net has IPv6 address 2a02:26f0:5700:182::1aca

+>

+```

+#### Firewall checks

+Use the `wget` command to verify that port 443 is open for communication. For example:

+``` CLI

+<root@xsense:/# wget https://www.apple.com

+--2022-11-09 11:21:15-- https://www.apple.com/

+Resolving www.apple.com (www.apple.com)... 72.246.148.202, 2a02:26f0:5700:1b4::1aca, 2a02:26f0:5700:182::1aca

+Connecting to www.apple.com (www.apple.com)|72.246.148.202|:443... connected.

+HTTP request sent, awaiting response... 200 OK

+Length: 99966 (98K) [text/html]

+Saving to: 'https://docsupdatetracker.net/index.html.1'

+https://docsupdatetracker.net/index.html.1 100%[===================>] 97.62K --.-KB/s in 0.02s

+2022-11-09 11:21:15 (5.88 MB/s) - 'https://docsupdatetracker.net/index.html.1' saved [99966/99966]

+>

+```

+ --catalog-item-name <catalog-item-name> --catalog-name <catalog-name>

+ :::image type="content" source="media/how-to-create-app-registration/client-secret.png" alt-text="Screenshot of the Azure portal showing an Azure AD app registration and a highlight around 'New client secret'." lightbox="media/how-to-create-app-registration/client-secret.png":::

+ :::image type="content" source="media/how-to-create-app-registration/add-client-secret.png" alt-text="Screenshot of the Azure portal while adding a client secret.":::

+ :::image type="content" source="media/how-to-create-app-registration/client-secret-value.png" alt-text="Screenshot of the Azure portal showing how to copy the client secret value." lightbox="media/how-to-create-app-registration/client-secret-value.png":::

+ :::image type="content" source="../../includes/role-based-access-control/media/add-role-assignment-page.png" alt-text="Screenshot of the 'Add role assignment' page." lightbox="../../includes/role-based-access-control/media/add-role-assignment-page.png":::

+ :::image type="content" source="media/how-to-create-app-registration/grant-admin-consent.png" alt-text="Screenshot of the Azure portal showing the 'Grant admin consent' button under API permissions." lightbox="media/how-to-create-app-registration/grant-admin-consent.png":::

+ - If consent was granted successfully, the entry for Azure Digital Twins should then show a **Status** value of **Granted for (your company)**

+ :::image type="content" source="media/how-to-create-app-registration/granted-admin-consent-done.png" alt-text="Screenshot of the Azure portal showing the admin consent granted for the company under API permissions." lightbox="media/how-to-create-app-registration/granted-admin-consent-done.png":::

+ :::image type="content" source="media/how-to-enable-private-link/create-private-endpoint-full.png" alt-text="Screenshot of the Azure portal showing the Create private endpoint page. It contains the fields described below." lightbox="media/how-to-enable-private-link/create-private-endpoint-full.png":::

+ :::image type="content" source="media/how-to-enable-private-link/network-flag-portal.png" alt-text="Screenshot of the Azure portal showing the Networking page for an Azure Digital Twins instance, highlighting how to toggle public access." lightbox="media/how-to-enable-private-link/network-flag-portal.png":::

+ :::image type="content" source="media/how-to-integrate-time-series-insights/create-event-hub-trigger-function.png" alt-text="Screenshot of Visual Studio to create a new Azure function of type event hub trigger." lightbox="media/how-to-integrate-time-series-insights/create-event-hub-trigger-function.png":::

+description: See how to use a system-assigned identity to forward events in Azure Digital Twins.

+This article describes how to use a [system-assigned identity for an Azure Digital Twins instance](concepts-security.md#managed-identity-for-accessing-other-resources) when forwarding events to supported routing destinations. Setting up a managed identity isn't required for routing, but it can help the instance to easily access other Azure AD-protected resources, such as [Event Hubs](../event-hubs/event-hubs-about.md), [Service Bus](../service-bus-messaging/service-bus-messaging-overview.md) destinations, and [Azure Storage Container](../storage/blobs/storage-blobs-introduction.md).

+## Create an Azure Digital Twins instance with a managed identity

+If you already have an Azure Digital Twins instance, ensure that you've enabled a [system-managed identity](how-to-set-up-instance-cli.md#enabledisable-system-managed-identity-for-the-instance) for it.

+If you don't have an Azure Digital Twins instance, follow the instructions in [Create the instance with a system-managed identity](how-to-set-up-instance-cli.md#create-the-instance-with-a-system-managed-identity) to create an Azure Digital Twins instance with a managed identity for the first time.

+Then, make sure you have *Azure Digital Twins Data Owner* role on the instance. You can find instructions in [Set up user access permissions](how-to-set-up-instance-cli.md#set-up-user-access-permissions).

+ :::image type="content" source="../../includes/role-based-access-control/media/add-role-assignment-page.png" alt-text="Screenshot of the 'Add role assignment' page for an Azure Digital Twins instance." lightbox="../../includes/role-based-access-control/media/add-role-assignment-page.png":::

+### Create the instance with a system-managed identity

+When you enable a [system-assigned identity](concepts-security.md#managed-identity-for-accessing-other-resources) on your Azure Digital Twins instance, Azure automatically creates an identity for it in [Azure Active Directory (Azure AD)](../active-directory/fundamentals/active-directory-whatis.md). That identity can then be used to authenticate to Azure Digital Twins endpoints for event forwarding. You can enable a system-managed identity for an Azure Digital Twins instance during instance creation, or [later on an existing instance](#enabledisable-system-managed-identity-for-the-instance).

+To create an Azure Digital Twins instance with system-assigned identity enabled, you can add an `--assign-identity` parameter to the `az dt create` command that's used to create the instance. (For more information about this command, see its [reference documentation](/cli/azure/dt#az-dt-create) or the [general instructions for setting up an Azure Digital Twins instance](how-to-set-up-instance-cli.md#create-the-azure-digital-twins-instance)).

+To create an instance with a system managed identity, add the `--assign-identity` parameter like this:

+```azurecli-interactive

+az dt create --dt-name <new-instance-name> --resource-group <resource-group> --assign-identity

+```

+## Enable/disable system-managed identity for the instance

+This section shows you how to add a system-managed identity to an existing Azure Digital Twins instance. You can also disable system-managed identity on an instance that has it already.

+The command to enable managed identity for an existing instance is the same `az dt create` command that's used to [create a new instance with a system managed identity](#create-the-instance-with-a-system-managed-identity). Instead of providing a new name of an instance to create, you can provide the name of an instance that already exists. Then, make sure to add the `--assign-identity` parameter.

+```azurecli-interactive

+az dt create --dt-name <name-of-existing-instance> --resource-group <resource-group> --assign-identity

+```

+To disable managed identity on an instance where it's currently enabled, use the following similar command to set `--assign-identity` to `false`.

+```azurecli-interactive

+az dt create --dt-name <name-of-existing-instance> --resource-group <resource-group> --assign-identity false

+```

+### Considerations for disabling system-managed identities

+It's important to consider the effects that any changes to the identity or its roles can have on the resources that use it. If you're [using managed identities with your Azure Digital Twins endpoints](how-to-route-with-managed-identity.md) or for [data history](how-to-use-data-history.md) and the identity is disabled, or a necessary role is removed from it, the endpoint or data history connection can become inaccessible and the flow of events will be disrupted.

+To continue using an endpoint that was set up with a managed identity that's now been disabled, you'll need to delete the endpoint and [re-create it](how-to-manage-routes.md#create-an-endpoint-for-azure-digital-twins) with a different authentication type. It may take up to an hour for events to resume delivery to the endpoint after this change.

+* **Advanced**: In this tab, you can enable a [system-managed identity](concepts-security.md#managed-identity-for-accessing-other-resources) for your instance. When this is enabled, Azure automatically creates an identity for the instance in [Azure Active Directory (Azure AD)](../active-directory/fundamentals/active-directory-whatis.md). That identity can then be used to authenticate to Azure Digital Twins endpoints for event forwarding. You can enable that system-managed identity here, or [later on an existing instance](#enabledisable-system-managed-identity-for-the-instance).

+## Enable/disable system-managed identity for the instance

+This section shows you how to add a system-managed identity to an existing Azure Digital Twins instance. You can also use this page to disable system-managed identity on an instance that has it already.

+Start by opening the [Azure portal](https://portal.azure.com) in a browser.

+1. Search for the name of your instance in the portal search bar, and select it to view its details.

+1. Select **Identity** in the left-hand menu.

+1. On this page, select the **On** option to turn on this feature.

+1. Select the **Save** button, and **Yes** to confirm.

+ :::image type="content" source="media/how-to-route-with-managed-identity/identity-digital-twins.png" alt-text="Screenshot of the Azure portal showing the Identity page for an Azure Digital Twins instance." lightbox="media/how-to-route-with-managed-identity/identity-digital-twins.png":::

+After the change is saved, more fields will appear on this page for the new identity's **Object ID** and **Permissions**.

+You can copy the **Object ID** from here if needed, and use the **Permissions** button to view the Azure roles that are assigned to the identity. To set up some roles, continue to the next section.

+### Considerations for disabling system-managed identities

+It's important to consider the effects that any changes to the identity or its roles can have on the resources that use it. If you're [using managed identities with your Azure Digital Twins endpoints](how-to-route-with-managed-identity.md) or for [data history](how-to-use-data-history.md) and the identity is disabled, or a necessary role is removed from it, the endpoint or data history connection can become inaccessible and the flow of events will be disrupted.

+If you already have an Azure Digital Twins instance, ensure that you've enabled a [system-managed identity](how-to-set-up-instance-cli.md#enabledisable-system-managed-identity-for-the-instance) for it.

+If you don't have an Azure Digital Twins instance, follow the instructions in [Create the instance with a system-managed identity](how-to-set-up-instance-cli.md#create-the-instance-with-a-system-managed-identity) to create an Azure Digital Twins instance with a managed identity for the first time.

+Then, make sure you have *Azure Digital Twins Data Owner* role on the instance. You can find instructions in [Set up user access permissions](how-to-set-up-instance-cli.md#set-up-user-access-permissions).

+ :::image type="content" source="media/how-to-use-postman-with-digital-twins/console-access-token.png" alt-text="Screenshot of the console showing the result of the az account get-access-token command. The accessToken field and its sample value is highlighted." lightbox="media/how-to-use-postman-with-digital-twins/console-access-token.png":::

+ :::image type="content" source="media/how-to-use-postman-with-digital-twins/swagger-raw.png" alt-text="Screenshot of the data plane digitaltwins.json file in GitHub. There is a highlight around the Raw button." lightbox="media/how-to-use-postman-with-digital-twins/swagger-raw.png":::

+ :::image type="content" source="media/how-to-use-postman-with-digital-twins/postman-import-collection-2.png" alt-text="Screenshot of Postman's 'Import' window, showing the file to import as a collection and the Import button." lightbox="media/how-to-use-postman-with-digital-twins/postman-import-collection-2.png":::

+Sdutil is a command line Python utility tool designed to easily interact with seismic store. The seismic store is a cloud-based solution designed to store and manage datasets of any size in the cloud by enabling a secure way to access them through a scoped authorization mechanism. Seismic Store overcomes the object size limitations imposed by a cloud provider by managing generic datasets as multi-independent objects. This provides a generic, reliable, and better performing solution to handle data in cloud storage.

+1. Ensure you have followed the [installation](#prerequisites) and [configuration](#configuration) steps from above. This includes downloading the SDUTIL source code, configuring your Python virtual environment, editing the `config.yaml` file and setting your three environment variables.

+An event domain is a management tool for large number of Event Grid topics related to the same application. You can think of it as a meta-topic that can have thousands of individual topics. It allows an event publisher to publish events to thousands of topics at the same time. Domains also give you authentication and authorization control over each topic so you can partition your tenants. This article describes how to use event domains to manage the flow of custom events to your various business organizations, customers, or applications. Use event domains to:

+* Manage your authentication and authorization.

+With a domain, you get fine grain authorization and authentication control over each topic via Azure role-based access control (Azure RBAC). You can use these roles to restrict each tenant in your application to only the topics you wish to grant them access to. Azure RBAC in event domains works the same way [managed access control](security-authorization.md) works in the rest of Event Grid and Azure. Use Azure RBAC to create and enforce custom role definitions in event domains.

+Event Grid has two built-in role definitions to make Azure RBAC easier for working with event domains. These roles are **EventGrid EventSubscription Contributor** and **EventGrid EventSubscription Reader**. You assign these roles to users who need to subscribe to topics in your event domain. You scope the role assignment to only the topic that users need to subscribe to. For information about these roles, see [Built-in roles for Event Grid](security-authorization.md#built-in-roles).

+Subscribing to events for a topic within an event domain is the same as [creating an Event Subscription on a custom topic](./custom-event-quickstart.md) or subscribing to an event from an Azure service.

+When you create an event domain, you're given a publishing endpoint similar to if you had created a topic in Event Grid. To publish events to any topic in an event domain, push the events to the domain's endpoint the [same way you would for a custom topic](./post-to-custom-topic.md). The only difference is that you must specify the topic you'd like the event to be delivered to. For example, publishing the following array of events would send event with `"id": "1111"` to topic `foo` while the event with `"id": "2222"` would be sent to topic `bar`:

+Event domains use the same [operations pricing](https://azure.microsoft.com/pricing/details/event-grid/) that all other features in Event Grid use. Operations work the same in event domains as they do in custom topics. Each ingress of an event to an event domain is an operation, and each delivery attempt for an event is an operation.

+To learn about setting up event domains, creating topics, creating event subscriptions, and publishing events, see [Manage event domains](./how-to-event-domains.md).

+This article provides the properties and schema for events in [Azure Key Vault](../key-vault/index.yml). For an introduction to event schemas, see [Azure Event Grid event schema](event-schema.md) and [Cloud event schema](cloud-event-schema.md).

+An event handler receives events from an event source via Event Grid, and processes those events. You can use instances of a few Azure services to handle events and **Azure Service Bus** is one of them. This article shows you how to use a Service Bus queue or topic as a handler for events from Event Grid.

+You can route events in Event Grid directly to Service Bus queues for use in buffering or command and control scenarios in enterprise applications.

+### Use Azure portal

+In the Azure portal, while creating an event subscription, select **Service Bus Queue** as the endpoint type and then click **select an endpoint** to choose a Service Bus queue.

+> [!NOTE]

+> Session enabled queues are not supported as event handlers for Azure Event Grid events

+

+### Use Azure CLI

+Use the [`az eventgrid event-subscription create`](/cli/azure/eventgrid/event-subscription) command with `--endpoint-type` set to `servicebusqueue` and `--endpoint` set to `/subscriptions/{AZURE SUBSCRIPTION}/resourceGroups/<RESOURCE GROUP NAME>/providers/Microsoft.ServiceBus/namespaces/<NAMESPACE NAME>/queues/<QUEUE NAME>`. Here's an example:

+You can also use the [`az eventgrid topic event-subscription`](/cli/azure/eventgrid/topic/event-subscription) command for custom topics, the [`az eventgrid system-topic event-subscription`](/cli/azure/eventgrid/system-topic/event-subscription) command for system topics, and the [`az eventgrid partner topic event-subscription create`](/cli/azure/eventgrid/partner/topic/event-subscription#az-eventgrid-partner-topic-event-subscription-create) command for partner topics.

+### Use Azure PowerShell

+Use the [New-AzEventGridSubscription](/powershell/module/az.eventgrid/new-azeventgridsubscription) command with `-EndpointType` set to `servicebusqueue` and `-Endpoint` set to `/subscriptions/{AZURE SUBSCRIPTION}/resourceGroups/<RESOURCE GROUP NAME>/providers/Microsoft.ServiceBus/namespaces/<NAMESPACE NAME>/queues/<QUEUE NAME>`. Here's an example:

+```azurepowershell-interactive

+New-AzEventGridSubscription -ResourceGroup MyResourceGroup `

+ -TopicName Topic1 `

+ -EndpointType servicebusqueue `

+ -Endpoint /subscriptions/{SubID}/resourceGroups/TestRG/providers/Microsoft.ServiceBus/namespaces/ns1/queues/queue1 `

+ -EventSubscriptionName EventSubscription1

+```

+You can also use the [`New-AzEventGridSystemTopicEventSubscription`](/powershell/module/az.eventgrid/new-azeventgridsystemtopiceventsubscription) command for system topics, and the [`New-AzEventGridPartnerTopicEventSubscription`](/powershell/module/az.eventgrid/new-azeventgridpartnertopiceventsubscription) command for partner topics.

+You can route events in Event Grid directly to Service Bus topics for command and control messaging scenarios.

+### Use Azure portal

+In the Azure portal, while creating an event subscription, select **Service Bus Topic** as the endpoint type and then click **select an endpoint** to choose a Service Bus topic.

+### Use Azure CLI

+Use the [`az eventgrid event-subscription create`](/cli/azure/eventgrid/event-subscription) command with `--endpoint-type` set to `servicebustopic` and `--endpoint` set to `/subscriptions/{AZURE SUBSCRIPTION}/resourceGroups/<RESOURCE GROUP NAME>/providers/Microsoft.ServiceBus/namespaces/<NAMESPACE NAME>/topics/<TOPIC NAME>`. Here's an example:

+You can also use the [`az eventgrid topic event-subscription`](/cli/azure/eventgrid/topic/event-subscription) command for custom topics, the [`az eventgrid system-topic event-subscription`](/cli/azure/eventgrid/system-topic/event-subscription) command for system topics, and the [`az eventgrid partner topic event-subscription create`](/cli/azure/eventgrid/partner/topic/event-subscription#az-eventgrid-partner-topic-event-subscription-create) command for partner topics.

+### Use Azure PowerShell

+Use the [New-AzEventGridSubscription](/powershell/module/az.eventgrid/new-azeventgridsubscription) command with `-EndpointType` set to `servicebustopic` and `-Endpoint` set to `/subscriptions/{AZURE SUBSCRIPTION}/resourceGroups/<RESOURCE GROUP NAME>/providers/Microsoft.ServiceBus/namespaces/<NAMESPACE NAME>/topics/<TOPIC NAME>`. Here's an example:

+```azurepowershell-interactive

+New-AzEventGridSubscription -ResourceGroup MyResourceGroup `

+ -TopicName Topic1 `

+ -EndpointType servicebustopic `

+ -Endpoint /subscriptions/{SubID}/resourceGroups/TestRG/providers/Microsoft.ServiceBus/namespaces/ns1/topics/topic1 `

+ -EventSubscriptionName EventSubscription1

+```

+You can also use the [`New-AzEventGridSystemTopicEventSubscription`](/powershell/module/az.eventgrid/new-azeventgridsystemtopiceventsubscription) command for system topics, and the [`New-AzEventGridPartnerTopicEventSubscription`](/powershell/module/az.eventgrid/new-azeventgridpartnertopiceventsubscription) command for partner topics.

+## Delivery properties

+Event subscriptions allow you to set up HTTP headers that are included in delivered events. This capability allows you to set custom headers that are required by a destination. You can set custom headers on the events that are delivered to Azure Service Bus queues and topics.

+Azure Service Bus supports the use of following message properties when sending single messages.

+| Header name | Header type |

+| :-- | :-- |

+| `MessageId` | Dynamic |

+| `PartitionKey` | Static or dynamic |

+| `SessionId` | Static or dynamic |

+| `CorrelationId` | Static or dynamic |

+| `Label` | Static or dynamic |

+| `ReplyTo` | Static or dynamic |

+| `ReplyToSessionId` | Static or dynamic |

+| `To` |Static or dynamic |

+| `ViaPartitionKey` | Static or dynamic |

+> [!NOTE]

+> - The default value of `MessageId` is the internal ID of the Event Grid event. You can override it. For example, `data.field`.

+> - You can only set either `SessionId` or `MessageId`.

+For more information, see [Custom delivery properties](delivery-properties.md).

+An event handler receives events from an event source via Event Grid, and processes those events. You can use any WebHook as an event handler for events forwarded by Event Grid. The WebHook doesn't need to be hosted in Azure to handle events. Event Grid supports only HTTPS Webhook endpoints. You can also use an Azure Automation workbook or an Azure logic app as an event handler via webhooks. This article provides you links to conceptual, quickstart, and tutorial articles that provide you with more information.

+> Even though you can use **Webhook** as an **endpoint type** to configure an Azure function as an event handler, use **Azure Function** as an endpoint type. For more information, see [Azure function as an event handler](handler-functions.md).

+| [Overview: receive events to an HTTP endpoint](receive-events.md) | Describes how to validate an HTTP endpoint to receive events from an event subscription, and receive and deserialize events. |

+When sending the HTTP POST to a custom topic, use the URI format: `https://<topic-endpoint>?api-version=2018-01-01`. For example, a valid URI is: `https://exampletopic.westus2-1.eventgrid.azure.net/api/events?api-version=2018-01-01`. To get the endpoint for a custom topic using Azure CLI, use:

+To get the endpoint for a custom topic using Azure PowerShell, use:

+In the request, include a header value named `aeg-sas-key` that contains a key for authentication. For example, a valid header value is `aeg-sas-key: xxxxxxxxxxxxxxxxxxxxxxx`. To get the key for a custom topic using Azure CLI, use:

+To get the key for a custom topic using PowerShell, use:

+For custom topics, the top-level data contains the same fields as standard resource-defined events. One of those properties is a `data` property that contains properties unique to the custom topic. As an event publisher, you determine properties for that data object. Here's the schema:

+For a description of these properties, see [Azure Event Grid event schema](event-schema.md). When posting events to an Event Grid topic, the array can have a total size of up to 1 MB. The maximum allowed size for an event is also 1 MB. Events over 64 KB are charged in 64-KB increments. When receiving events in a batch, the maximum allowed number of events is 5,000 per batch.

+- **URL filtering** - extends Azure FirewallΓÇÖs FQDN filtering capability to consider an entire URL along with any additional path. For example, `www.contoso.com/a/c` instead of `www.contoso.com`.

+Under the **Web Categories** tab in **Firewall Policy Settings**, you can request a category change if you:

+When your origin server processes a request, it sends data back to Front Door so that it can be returned to the client. This traffic is not billed by Front Door, even if the origin is in a different region to the Front Door edge location for the request.

+ :::image type="content" source="./media/apache-kafka-spark-structured-streaming-cosmosdb/hdi-custom-parameters-40.png" alt-text="HDInsight version 4.0 custom deployment values":::

+Data Lake Storage is optimized for event ingestion through Azure Event Hubs.

+ * Replace **\<cluster-type\>** with the type of the HDInsight cluster to create. Allowed strings include: `hadoop`, `interactivehive`, `hbase`, and `spark`.

+|IoT / Streaming|Kafka, Spark|

+|Cluster types|Hadoop, Spark, Confluent Kafka, Solr|

+|HBase as a platform|Applications can run on top of HBase by using it as a datastore. Examples include Phoenix, OpenTSDB, `Kiji`, and Titan. Applications can also integrate with HBase. Examples include: [Apache Hive](https://hive.apache.org/), Apache Pig, [Solr](https://lucene.apache.org/solr/), Apache Flume, [Apache Impala](https://impala.apache.org/), Apache Spark, `Ganglia`, and Apache Drill.|

+description: Learn how to use the Azure CLI to manage Azure HDInsight clusters. Cluster types include Apache Hadoop, Spark, HBase, Kafka, Interactive Query.

+Some commands may require you to specify the scheme as part of the URI when accessing a file. When using non-default storage (storage added as "additional" storage to the cluster), you must always use the scheme as part of the URI.

+* [Use MapReduce jobs with HDInsight](hadoop/hdinsight-use-mapreduce.md)

+|Stream processing|Kafka is often used with Spark for real-time stream processing. Kafka 0.10.0.0 (HDInsight version 3.5 and 3.6) introduced a streaming API that allows you to build streaming solutions without requiring Spark. For more information, see [Start with Apache Kafka on HDInsight](apache-kafka-get-started.md).|

+Kafka stream processing is often done using Apache Spark. Kafka version 1.1.0 (in HDInsight 3.5 and 3.6) introduced the Kafka Streams API. This API allows you to transform data streams between input and output topics.

+ $clusterType = "Hadoop" #Use any supported cluster type (Hadoop, HBase, etc.)

+> [Manage Apache Hadoop clusters in HDInsight by using Azure PowerShell](hdinsight-administer-use-powershell.md)

+> * IO Cache was supported till Spark 2.3 and will not be supported in Spark 2.4 (HDInsight 4.0) and Spark 3.1.2 (HDInsight 5.0)

+<!SMART Implementation Guide v1.0.0 is supported by Azure Health Data Services and Azure API Management (APIM). This is our recommended approach, as it enabled Health IT developers to comply with 21st Century Act Criterion §170.315(g)(10) Standardized API for patient and population services.

+One of the main purposes of the specifications is to describe how an application should discover authentication endpoints for an FHIR server and start an authentication sequence. SMART on FHIR uses parameter naming conventions that arenΓÇÖt immediately compatible with Azure Active Directory (Azure AD), the Azure Health Data Services (FHIR Service) has a built-in Azure AD SMART on FHIR proxy that enables a subset of the SMART on FHIR launch sequences. Specifically, the proxy enables the [EHR launch sequence](https://hl7.org/fhir/smart-app-launch/#ehr-launch-sequence).

+- [Register public client application in Azure AD]([https://learn.microsoft.com/azure/healthcare-apis/azure-api-for-fhir/register-public-azure-ad-client-app]

+The MedTech service may be used with devices created and managed through an [Azure IoT Hub](../../iot-hub/iot-concepts-and-iot-hub.md) for enhanced workflows and ease of use. This tutorial uses an Azure Resource Manager (ARM) template and a **Deploy to Azure** button to deploy a MedTech service using an Azure IoT Hub for device creation, management, and routing of device messages to the MedTech service device message event hub. The ARM template used in this article is available from the [Azure Quickstart Templates](/samples/azure/azure-quickstart-templates/iotconnectors-with-iothub/) site using the **azuredeploy.json** file located on [GitHub](https://github.com/azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.healthcareapis/workspaces/iotconnectors-with-iothub).

+ > To learn more about advanced metrics display and sharing options, see [Getting started with Azure Metrics Explorer](/azure/azure-monitor/essentials/metrics-getting-started)

+The root certificate authority (CA) is the [Baltimore CyberTrust Root](https://www.digicert.com/kb/digicert-root-certificates.htm) certificate. This root certificate is signed by DigiCert, and is widely trusted and stored in many operating systems. For example, both Ubuntu and Windows include it in the default certificate store.

+### NTLM Authentication

+IoT Edge does not currently support network proxies that use NTLM authentication. Users may consider bypassing the proxy by adding the required endpoints to the firewall allow-list.

+- [The requirements.txt can be downloaded from the Azure Python samples](https://github.com/Azure-Samples/azure-samples-python-management/blob/main/samples/labservices/requirements.txt)

+- [The requirements.txt can be downloaded from the Azure Python samples](https://github.com/Azure-Samples/azure-samples-python-management/blob/main/samples/labservices/requirements.txt)

+## 2022-11-08

+### Azure Machine Learning CLI (v2) v2.11.0

+- The CLI is depending on azure-ai-ml 1.1.0.

+- `az ml registry`

+ - Added `ml registry delete` command.

+ - Adjusted registry experimental tags and imports to avoid warning printouts for unrelated operations.

+- `az ml environment`

+ - Prevented registering an already existing environment that references conda file.

+- [Access data in a job](how-to-read-write-data-v2.md)

+- [Data administration](how-to-administrate-data-authentication.md#data-administration)

+ Title: Access data from Azure cloud storage during interactive development

+description: Access data from Azure cloud storage during interactive development

+#Customer intent: As a professional data scientist, I want to know how to build and deploy a model with Azure Machine Learning by using Python in a Jupyter Notebook.

+# Access data from Azure cloud storage during interactive development

+Typically the beginning of a machine learning project involves exploratory data analysis (EDA), data-preprocessing (cleaning, feature engineering), and building prototypes of ML models to validate hypotheses. This *prototyping* phase of the project is highly interactive in nature that lends itself to developing in a Jupyter notebook or an IDE with a *Python interactive console*. In this article you'll learn how to:

+> [!div class="checklist"]

+> * Access data from a Azure ML Datastores URI as if it were a file system.

+> * Materialize data into Pandas using `mltable` Python library.

+> * Materialize Azure ML data assets into Pandas using `mltable` Python library.

+> * Materialize data through an explicit download with the `azcopy` utility.

+## Prerequisites

+* An Azure Machine Learning workspace. For more information, see [Manage Azure Machine Learning workspaces in the portal or with the Python SDK (v2)](how-to-manage-workspace.md).

+* An Azure Machine Learning Datastore. For more information, see [Create datastores](how-to-datastore.md).

+> [!TIP]

+> The guidance in this article to access data during interactive development applies to any host that can run a Python session - for example: your local machine, a cloud VM, a GitHub Codespace, etc. We recommend using an Azure Machine Learning compute instance - a fully managed and pre-configured cloud workstation. For more information, see [Create and manage an Azure Machine Learning compute instance](how-to-create-manage-compute-instance.md).

+> [!IMPORTANT]

+> Ensure you have the latest `azure-fsspec` and `mltable` python libraries installed in your python environment:

+>

+> ```bash

+> pip install -U azureml-fsspec mltable

+> ```

+## Access data from a datastore URI, like a filesystem (preview)

+An Azure ML datastore is a *reference* to an *existing* storage account on Azure. The benefits of creating and using a datastore include:

+> [!div class="checklist"]

+> * A common and easy-to-use API to interact with different storage types (Blob/Files/ADLS).

+> * Easier to discover useful datastores when working as a team.

+> * Supports both credential-based (for example, SAS token) and identity-based (use Azure Active Directory or Manged identity) to access data.

+> * When using credential-based access, the connection information is secured so you don't expose keys in scripts.

+> * Browse data and copy-paste datastore URIs in the Studio UI.

+A *Datastore URI* is a Uniform Resource Identifier, which is a *reference* to a storage *location* (path) on your Azure storage account. The format of the datastore URI is:

+```python

+# AzureML workspace details:

+subscription = '<subscription_id>'

+resource_group = '<resource_group>'

+workspace = '<workspace>'

+datastore_name = '<datastore>'

+path_on_datastore '<path>'

+# long-form Datastore uri format:

+uri = f'azureml://subscriptions/{subscription}/resourcegroups/{resource_group}/workspaces/{workspace}/datastores/{datastore_name}/paths/{path_on_datastore}'.

+```

+These Datastore URIs are a known implementation of [Filesystem spec](https://filesystem-spec.readthedocs.io/latest/https://docsupdatetracker.net/index.html#) (`fsspec`): A unified pythonic interface to local, remote and embedded file systems and bytes storage.

+The Azure ML Datastore implementation of `fsspec` automatically handles credential/identity passthrough used by the Azure ML datastore. This means you don't need to expose account keys in your scripts or do additional sign-in procedures on a compute instance.

+For example, you can directly use Datastore URIs in Pandas - below is an example of reading a CSV file:

+```python

+import pandas as pd

+df = pd.read_csv("azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<workspace_name>/datastores/<datastore_name>/paths/<folder>/<filename>.csv")

+df.head()

+```

+> [!TIP]

+> Rather than remember the datastore URI format, you can copy-and-paste the datastore URI from the Studio UI by following these steps:

+> 1. Select **Data** from the left-hand menu followed by the **Datastores** tab.

+> 1. Select your datastore name and then **Browse**.

+> 1. Find the file/folder you want to read into pandas, select the elipsis (**...**) next to it. Select from the menu **Copy URI**. You can select the **Datastore URI** to copy into your notebook/script.

+> :::image type="content" source="media/how-to-access-data-ci/datastore_uri_copy.png" alt-text="Screenshot highlighting the copy of the datastore URI.":::

+You can also instantiate an Azure ML filesystem and do filesystem-like commands like `ls`, `glob`, `exists`, `open`, etc. The `open()` method will return a file-like object, which can be passed to any other library that expects to work with python files, or used by your own code as you would a normal python file object. These file-like objects respect the use of `with` contexts, for example:

+```python

+from azureml.fsspec import AzureMachineLearningFileSystem

+# instantiate file system using datastore URI

+fs = AzureMachineLearningFileSystem('azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<workspace_name>/datastores/<datastore_name>/paths/<folder>')

+# list files in the path

+fs.ls()

+# output example:

+# /datastore_name/folder/file1.csv

+# /datastore_name/folder/file2.csv

+# use an open context

+with fs.open('/datastore_name/folder/file1.csv') as f:

+ # do some process

+ process_file(f)

+```

+### Examples

+In this section we provide some examples of how to use Filesystem spec, for some common scenarios.

+#### Read a single CSV file into pandas

+If you have a *single* CSV file, then as outlined above you can read that into pandas with:

+```python

+import pandas as pd

+df = pd.read_csv("azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<workspace_name>/datastores/<datastore_name>/paths/<folder>/<filename>.csv")

+```

+#### Read a folder of CSV files into pandas

+The Pandas `read_csv()` method doesn't support reading a folder of CSV files. You need to glob csv paths and concatenate them to a data frame using Pandas `concat()` method. The code below demonstrates how to achieve this concatenation with the Azure ML filesystem:

+```python

+import pandas as pd

+from azureml.fsspec import AzureMachineLearningFileSystem

+# define the URI - update <> placeholders

+uri = 'azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<workspace_name>/datastores/<datastore_name>/paths/<folder>/*.csv'

+# create the filesystem

+fs = AzureMachineLearningFileSystem(uri)

+# append csv files in folder to a list

+dflist = []

+for path in fs.ls():

+ with fs.open(path) as f:

+ dflist.append(pd.read_csv(f))

+# concatenate data frames

+df = pd.concat(dflist)

+df.head()

+```

+#### Reading CSV files into Dask

+Below is an example of reading a CSV file into a Dask data frame:

+```python

+import dask.dd as dd

+df = dd.read_csv("azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<workspace_name>/datastores/<datastore_name>/paths/<folder>/<filename>.csv")

+df.head()

+```

+#### Read a folder of parquet files into pandas

+Parquet files are typically written to a folder as part of an ETL process, which can emit files pertaining to the ETL such as progress, commits, etc. Below is an example of files created from an ETL process (files beginning with `_`) to produce a parquet file of data.

+In these scenarios, you'll only want to read the parquet files in the folder and ignore the ETL process files. The code below shows how you can use glob patterns to read only parquet files in a folder:

+```python

+import pandas as pd

+from azureml.fsspec import AzureMachineLearningFileSystem

+# define the URI - update <> placeholders

+uri = 'azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<workspace_name>/datastores/<datastore_name>/paths/<folder>/*.parquet'

+# create the filesystem

+fs = AzureMachineLearningFileSystem(uri)

+# append csv files in folder to a list

+dflist = []

+for path in fs.ls():

+ with fs.open(path) as f:

+ dflist.append(pd.read_parquet(f))

+# concatenate data frames

+df = pd.concat(dflist)

+df.head()

+```

+#### Accessing data from your Azure Databricks filesystem (`dbfs`)

+Filesystem spec (`fsspec`) has a range of [known implementations](https://filesystem-spec.readthedocs.io/en/stable/_modules/https://docsupdatetracker.net/index.html), one of which is the Databricks Filesystem (`dbfs`).

+To access data from `dbfs` you will need:

+- **Instance name**, which is in the form of `adb-<some-number>.<two digits>.azuredatabricks.net`. You can glean this from the URL of your Azure Databricks workspace.

+- **Personal Access Token (PAT)**, for more information on creating a PAT, please see [Authentication using Azure Databricks personal access tokens](/azure/databricks/dev-tools/api/latest/authentication)

+Once you have these, you will need to create an environment variable on your compute instance for the PAT token:

+```bash

+export ADB_PAT=<pat_token>

+```

+You can then access data in Pandas using:

+```python

+import os

+import pandas as pd

+pat = os.getenv(ADB_PAT)

+path_on_dbfs = '<absolute_path_on_dbfs>' # e.g. /folder/subfolder/file.csv

+storage_options = {

+ 'instance':'adb-<some-number>.<two digits>.azuredatabricks.net',

+ 'token': pat

+}

+df = pd.read_csv(f'dbfs://{path_on_dbfs}', storage_options=storage_options)

+```

+#### Reading images with `pillow`

+```python

+from PIL import Image

+from azureml.fsspec import AzureMachineLearningFileSystem

+# define the URI - update <> placeholders

+uri = 'azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<workspace_name>/datastores/<datastore_name>/paths/<folder>/<image.jpeg>'

+# create the filesystem

+fs = AzureMachineLearningFileSystem(uri)

+with fs.open() as f:

+ img = Image.open(f)

+ img.show()

+```

+#### PyTorch custom dataset example

+In this example, you create a PyTorch custom dataset for processing images. The assumption is that an annotations file (in CSV format) exists that looks like:

+```text

+image_path, label

+0/image0.png, label0

+0/image1.png, label0

+1/image2.png, label1

+1/image3.png, label1

+2/image4.png, label2

+2/image5.png, label2

+```

+The images are stored in subfolders according to their label:

+```text

+/

+└── 📁images

+ ├── 📁0

+ │ ├── 📷image0.png

+ │ └── 📷image1.png

+ ├── 📁1

+ │ ├── 📷image2.png

+ │ └── 📷image3.png

+ └── 📁2

+ ├── 📷image4.png

+ └── 📷image5.png

+```

+A custom Dataset class in PyTorch must implement three functions: `__init__`, `__len__`, and `__getitem__`, which are implemented below:

+```python

+import os

+import pandas as pd

+from PIL import Image

+from torch.utils.data import Dataset

+class CustomImageDataset(Dataset):

+ def __init__(self, filesystem, annotations_file, img_dir, transform=None, target_transform=None):

+ self.fs = filesystem

+ f = filesystem.open(annotations_file)

+ self.img_labels = pd.read_csv(f)

+ f.close()

+ self.img_dir = img_dir

+ self.transform = transform

+ self.target_transform = target_transform

+ def __len__(self):

+ return len(self.img_labels)

+ def __getitem__(self, idx):

+ img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])

+ f = self.fs.open(img_path)

+ image = Image.open(f)

+ f.close()

+ label = self.img_labels.iloc[idx, 1]

+ if self.transform:

+ image = self.transform(image)

+ if self.target_transform:

+ label = self.target_transform(label)

+ return image, label

+```

+You can then instantiate the dataset using:

+```python

+from azureml.fsspec import AzureMachineLearningFileSystem

+from torch.utils.data import DataLoader

+# define the URI - update <> placeholders

+uri = 'azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<workspace_name>/datastores/<datastore_name>/paths/<folder>/'

+# create the filesystem

+fs = AzureMachineLearningFileSystem(uri)

+# create the dataset

+training_data = CustomImageDataset(

+ filesystem=fs,

+ annotations_file='<datastore_name>/<path>/annotations.csv',

+ img_dir='<datastore_name>/<path_to_images>/'

+)

+# Preparing your data for training with DataLoaders

+train_dataloader = DataLoader(training_data, batch_size=64, shuffle=True)

+```

+## Materialize data into Pandas using `mltable` library (preview)

+Another method for accessing data in cloud storage is to use the `mltable` library. The general format for reading data into pandas using `mltable` is:

+```python

+import mltable

+# define a path or folder or pattern

+path = {

+ 'file': '<supported_path>'

+ # alternatives

+ # 'folder': '<supported_path>'

+ # 'pattern': '<supported_path>'

+}

+# create an mltable from paths

+tbl = mltable.from_delimited_files(paths=[path])

+# alternatives

+# tbl = mltable.from_parquet_files(paths=[path])

+# tbl = mltable.from_json_lines_files(paths=[path])

+# tbl = mltable.from_delta_lake(paths=[path])

+# materialize to pandas

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+### Supported paths

+You'll notice the `mltable` library supports reading tabular data from different path types:

+|Location | Examples |

+|||

+|A path on your local computer | `./home/username/data/my_data` |

+|A path on a public http(s) server | `https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv` |

+|A path on Azure Storage | `wasbs://<container_name>@<account_name>.blob.core.windows.net/<path>` <br> `abfss://<file_system>@<account_name>.dfs.core.windows.net/<path>` |

+|A long-form Azure ML datastore | `azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<wsname>/datastores/<name>/paths/<path>` |

+> [!NOTE]

+> `mltable` does user credential passthrough for paths on Azure Storage and Azure ML datastores. If you do not have permission to the data on the underlying storage then you will not be able to access the data.

+### Files, folders and globs

+`mltable` supports reading from:

+- file(s), for example: `abfss://<file_system>@<account_name>.dfs.core.windows.net/my-csv.csv`

+- folder(s), for example `abfss://<file_system>@<account_name>.dfs.core.windows.net/my-folder/`

+- [glob](https://wikipedia.org/wiki/Glob_(programming)) pattern(s), for example `abfss://<file_system>@<account_name>.dfs.core.windows.net/my-folder/*.csv`

+- Or, a combination of files, folders, globbing patterns

+The flexibility of `mltable` allows you to materialize data into a single dataframe from a combination of local/cloud storage and combinations of files/folder/globs. For example:

+```python

+path1 = {

+ 'file': 'abfss://filesystem@account1.dfs.core.windows.net/my-csv.csv'

+}

+path2 = {

+ 'folder': './home/username/data/my_data'

+}

+path3 = {

+ 'pattern': 'abfss://filesystem@account2.dfs.core.windows.net/folder/*.csv'

+}

+tbl = mltable.from_delimited_files(paths=[path1, path2, path3])

+```

+### Supported file formats

+`mltable` supports the following file formats:

+- Delimited Text (for example: CSV files): `mltable.from_delimited_files(paths=[path])`

+- Parquet: `mltable.from_parquet_files(paths=[path])`

+- Delta: `mltable.from_delta_lake(paths=[path])`

+- JSON lines format: `mltable.from_json_lines_files(paths=[path])`

+### Examples

+#### Read a CSV file

+##### [ADLS gen2](#tab/adls)

+Update the placeholders (`<>`) in the code snippet with your details.

+```python

+import mltable

+path = {

+ 'file': 'abfss://<filesystem>@<account>.dfs.core.windows.net/<folder>/<file_name>.csv'

+}

+tbl = mltable.from_delimited_files(paths=[path])

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+##### [Blob storage](#tab/blob)

+Update the placeholders (`<>`) in the code snippet with your details.

+```python

+import mltable

+path = {

+ 'file': 'wasbs://<container>@<account>.blob.core.windows.net/<folder>/<file_name>.csv'

+}

+tbl = mltable.from_delimited_files(paths=[path])

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+##### [Azure ML Datastore](#tab/datastore)

+Update the placeholders (`<>`) in the code snippet with your details.

+```python

+import mltable

+path = {

+ 'file': 'azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<wsname>/datastores/<name>/paths/<folder>/<file>.csv'

+}

+tbl = mltable.from_delimited_files(paths=[path])

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+> [!TIP]

+> Rather than remember the datastore URI format, you can copy-and-paste the datastore URI from the Studio UI by following these steps:

+> 1. Select **Data** from the left-hand menu followed by the **Datastores** tab.

+> 1. Select your datastore name and then **Browse**.

+> 1. Find the file/folder you want to read into pandas, select the elipsis (**...**) next to it. Select from the menu **Copy URI**. You can select the **Datastore URI** to copy into your notebook/script.

+> :::image type="content" source="media/how-to-access-data-ci/datastore_uri_copy.png" alt-text="Screenshot highlighting the copy of the datastore URI.":::

+##### [HTTP Server](#tab/http)

+```python

+import mltable

+path = {

+ 'file': 'https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv'

+}

+tbl = mltable.from_delimited_files(paths=[path])

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+#### Read parquet files in a folder

+The example code below shows how `mltable` can use [glob](https://wikipedia.org/wiki/Glob_(programming)) patterns - such as wildcards - to ensure only the parquet files are read.

+##### [ADLS gen2](#tab/adls)

+Update the placeholders (`<>`) in the code snippet with your details.

+```python

+import mltable

+path = {

+ 'pattern': 'abfss://<filesystem>@<account>.dfs.core.windows.net/<folder>/*.parquet'

+}

+tbl = mltable.from_parquet_files(paths=[path])

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+##### [Blob storage](#tab/blob)

+Update the placeholders (`<>`) in the code snippet with your details.

+```python

+import mltable

+path = {

+ 'pattern': 'wasbs://<container>@<account>.blob.core.windows.net/<folder>/*.parquet'

+}

+tbl = mltable.from_delimited_files(paths=[path])

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+##### [Azure ML Datastore](#tab/datastore)

+Update the placeholders (`<>`) in the code snippet with your details.

+```python

+import mltable

+path = {

+ 'pattern': 'azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<wsname>/datastores/<name>/paths/<folder>/*.parquet'

+}

+tbl = mltable.from_parquet_files(paths=[path])

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+> [!TIP]

+> Rather than remember the datastore URI format, you can copy-and-paste the datastore URI from the Studio UI by following these steps:

+> 1. Select **Data** from the left-hand menu followed by the **Datastores** tab.

+> 1. Select your datastore name and then **Browse**.

+> 1. Find the file/folder you want to read into pandas, select the elipsis (**...**) next to it. Select from the menu **Copy URI**. You can select the **Datastore URI** to copy into your notebook/script.

+> :::image type="content" source="media/how-to-access-data-ci/datastore_uri_copy.png" alt-text="Screenshot highlighting the copy of the datastore URI.":::

+##### [HTTP Server](#tab/http)

+Update the placeholders (`<>`) in the code snippet with your details.

+> [!IMPORTANT]

+> To glob the pattern on a public HTTP server, there must be access at the **folder** level.

+```python

+import mltable

+path = {

+ 'pattern': '<https_address>/<folder>/*.parquet'

+}

+tbl = mltable.from_parquet_files(paths=[path])

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+### Reading data assets

+In this section, you'll learn how to access your Azure ML data assets into pandas.

+#### Table asset

+If you've previously created a Table asset in Azure ML (an `mltable`, or a V1 `TabularDataset`), you can load that into pandas using:

+```python

+import mltable

+from azure.ai.ml import MLClient

+from azure.identity import DefaultAzureCredential

+ml_client = MLClient.from_config(credential=DefaultAzureCredential())

+data_asset = ml_client.data.get(name="<name_of_asset>", version="<version>")

+tbl = mltable.load(f'azureml:/{data_asset.id}')

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+#### File asset

+If you've registered a File asset that you want to read into Pandas data frame - for example, a CSV file - you can achieve this using:

+```python

+import mltable

+from azure.ai.ml import MLClient

+from azure.identity import DefaultAzureCredential

+ml_client = MLClient.from_config(credential=DefaultAzureCredential())

+data_asset = ml_client.data.get(name="<name_of_asset>", version="<version>")

+path = {

+ 'file': data_asset.path

+}

+tbl = mltable.from_delimited_files(paths=[path])

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+#### Folder asset

+If you've registered a Folder asset (`uri_folder` or a V1 `FileDataset`) that you want to read into Pandas data frame - for example, a folder containing CSV file - you can achieve this using:

+```python

+import mltable

+from azure.ai.ml import MLClient

+from azure.identity import DefaultAzureCredential

+ml_client = MLClient.from_config(credential=DefaultAzureCredential())

+data_asset = ml_client.data.get(name="<name_of_asset>", version="<version>")

+path = {

+ 'folder': data_asset.path

+}

+tbl = mltable.from_delimited_files(paths=[path])

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+## A note on reading and processing large data volumes with Pandas

+> [!TIP]

+> Pandas is not designed to handle large datasets - you will only be able to process data that can fit into the memory of the compute instance.

+>

+> For large datasets we recommend that you use AzureML managed Spark, which provides the [PySpark Pandas API](https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/https://docsupdatetracker.net/index.html).

+You may wish to iterate quickly on a smaller subset of a large dataset before scaling up to a remote asynchronous job. `mltable` provides in-built functionality to get samples of large data using the [take_random_sample](/python/api/mltable/mltable.mltable.mltable#mltable-mltable-mltable-take-random-sample) method:

+```python

+import mltable

+path = {

+ 'file': 'https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv'

+}

+tbl = mltable.from_delimited_files(paths=[path])

+# take a random 30% sample of the data

+tbl = tbl.take_random_sample(probability=.3)

+df = tbl.to_pandas_dataframe()

+df.head()

+```

+You can also take subsets of large data by using:

+- [filter](/python/api/mltable/mltable.mltable.mltable#mltable-mltable-mltable-filter)

+- [keep_columns](/python/api/mltable/mltable.mltable.mltable#mltable-mltable-mltable-keep-columns)

+- [drop_columns](/python/api/mltable/mltable.mltable.mltable#mltable-mltable-mltable-drop-columns)

+## Downloading data using the `azcopy` utility

+You may want to download the data to the local SSD of your host (local machine, cloud VM, Azure ML Compute Instance) and use the local filesystem. You can do this with the `azcopy` utility, which is pre-installed on an Azure ML compute instance. If you are **not** using an Azure ML compute instance or a Data Science Virtual Machine (DSVM), you may need to install `azcopy`. For more information please read [azcopy](../storage/common/storage-ref-azcopy.md).

+> [!CAUTION]

+> We do not recommend downloading data in the `/home/azureuser/cloudfiles/code` location on a compute instance. This is designed to store notebook and code artifacts, **not** data. Reading data from this location will incur significant performance overhead when training. Instead we recommend storing your data in `home/azureuser`, which is the local SSD of the compute node.

+Open a terminal and create a new directory, for example:

+```bash

+mkdir /home/azureuser/data

+```

+Sign-in to azcopy using:

+```bash

+azcopy login

+```

+Next, you can copy data using a storage URI

+```bash

+SOURCE=https://<account_name>.blob.core.windows.net/<container>/<path>

+DEST=/home/azureuser/data

+azcopy cp $SOURCE $DEST

+```

+To integrate Git with your Azure Machine Learning workspace, see [Git integration for Azure Machine Learning](concept-train-model-git-integration.md).

+In this article, you learn how to create a data asset in Azure Machine Learning. By creating a data asset, you create a *reference* to the data source location, along with a copy of its metadata. Because the data remains in its existing location, you incur no extra storage cost, and don't risk the integrity of your data sources. You can create Data from Azure ML datastores, Azure Storage, public URLs, and local files.

+> If you just want to access your data in an interactive session (for example, a Notebook) or a job, you are **not** required to create a data asset first. Creating a data asset would be an unnecessary step for you.

+>

+> For more information about accessing your data in a notebook, please see [Access data from Azure cloud storage for interactive development](how-to-access-data-interactive.md).

+>

+> For more information about accessing your data - both local and cloud storage - in a job, please see [Access data in a job](how-to-read-write-data-v2.md).

+>

+> Creating data assets are useful when you want to:

+> - **Share and reuse** data with other members of your team so that they don't need to remember file locations in cloud storage.

+> - **Version** the metadata such as location, description and tags.

+

+1. Under **Assets** in the left navigation, select **Data**. On the Data assets tab, select Create

+1. Under **Assets** in the left navigation, select **Data**. On the Data assets tab, select Create

+ delimiter: ,

+The from_* methods do not materialize the data, but rather stores it as a transformation in the MLTable definition.

+1. Under **Assets** in the left navigation, select **Data**. On the Data assets tab, select Create

+ # Note that idle_time_before_shutdown has been deprecated.

+ # Note that idle_time_before_shutdown has been deprecated.

+* The example template may not always use the latest API version for Azure Machine Learning. Before using the template, we recommend modifying it to use the latest API versions. For information on the latest API versions for Azure Machine Learning, see the [Azure Machine Learning REST API](/rest/api/azureml/).

+ > [!TIP]

+ > Each Azure service has its own set of API versions. For information on the API for a specific service, check the service information in the [Azure REST API reference](/rest/api/azure/).

+ To update the API version, find the `"apiVersion": "YYYY-MM-DD"` entry for the resource type and update it to the latest version. The following example is an entry for Azure Machine Learning:

+ ```json

+ "type": "Microsoft.MachineLearningServices/workspaces",

+ "apiVersion": "2020-03-01",

+ ```

+[Use a compute instance terminal](how-to-access-terminal.md#git) to clone and manage Git repositories. To integrate Git with your Azure Machine Learning workspace, see [Git integration for Azure Machine Learning](concept-train-model-git-integration.md).

+|A path from a Model Asset in AzureML Workspace | `azureml:<model-name>:<version>`|

+|A path from a Model Asset in AzureML Registry | `azureml://registries/<registry-name>/models/<model-name>/versions/<version>`|

+ Title: Access data in a job

+# Access data in a job

+* Learn more about [Data in Azure Machine Learning](concept-data.md)

+You'll use data that is stored on a public blob as a [zip file](https://azuremlexamples.blob.core.windows.net/datasets/fowl_data.zip). This dataset consists of about 120 training images each for two classes (turkeys and chickens), with 100 validation images for each class. The images are a subset of the [Open Images v5 Dataset](https://storage.googleapis.com/openimages/web/https://docsupdatetracker.net/index.html). We'll download and extract the dataset as part of our training script `pytorch_train.py`.

+- [Reference architecture for distributed deep learning training in Azure](/azure/architecture/reference-architectures/ai/training-deep-learning)

+* [Subscription does not exist](#subscription-does-not-exist)

+#### Subscription does not exist

+The Azure subscription that is entered must be existing. This error occurs when we cannot find the Azure subscription that was referenced. This is likely due to a typo in the subscription ID. Please double-check that the subscription ID was correctly typed and that it is currently active.

+For more information about Azure subscriptions, refer to the [prerequisites section](#prerequisites).

+* [V2 - Command Job](/python/api/azure-ai-ml/azure.ai.ml#azure-ai-ml-command)

+> [!IMPORTANT]

+> The example templates may not always use the latest API version for Azure Machine Learning. Before using the template, we recommend modifying it to use the latest API versions. For information on the latest API versions for Azure Machine Learning, see the [Azure Machine Learning REST API](/rest/api/azureml/).

+>

+> Each Azure service has its own set of API versions. For information on the API for a specific service, check the service information in the [Azure REST API reference](/rest/api/azure/).

+>

+> To update the API version, find the `Microsoft.MachineLearningServices/<resource>` entry for the resource type and update it to the latest version. The following example is an entry for the Azure Machine Learning workspace that uses an API version of `2022-05-01`:

+>

+>```json

+>resource machineLearning 'Microsoft.MachineLearningServices/workspaces@2022-05-01' = {

+>```

+Deprecation is the removal of a VM offer or a subset of the offer from Azure Marketplace so that it is no longer available for customers to deploy additional instances. Reasons to deprecate may vary. Common examples are due to security issues or end of life. You can deprecate image versions, plans, or an entire VM offer:

+- If deprecating an offer, the offer will no longer be searchable in the marketplace upon scheduling the deprecation. This is to reduce the discoverability of the offer.

+- Existing virtual machine scale sets deployments cannot be scaled out. If deprecating a plan or offer, all existing scale sets deployments using any image within the plan or offer respectively cannot be scaled out.

+> Before you deprecate an offer, plan, or image, make sure you understand the current usage by reviewing the [Usage dashboard in commercial marketplace analytics](https://partner.microsoft.com/dashboard/insights/commercial-marketplace/analytics/usage). If usage is high, consider hiding the plan or offer to minimize discoverability within the commercial marketplace and steer new customers towards other available options. To hide an offer, select the **Hide plan** checkbox on the **Pricing and Availability** page of each individual plan in the offer.

+- Each plan must have at least one active image.

+- You must publish the offer after scheduling the deprecation of an image.

+- Images that are published only to preview and have never been published live can be deleted immediately when the offer is still in preview state.

+- You must publish the offer after restoring an image for it to become available to customers.

+1. > [!NOTE]

+ > If the image can no longer be restored, then no actions will be available.

+

+Keep the following things in mind when deprecating a plan:

+- You must publish the offer after scheduling the deprecation of a plan.

+- You must publish the offer after restoring a plan for it to become available to customers.

+- The deprecation will immediately be scheduled 90 days into the future upon confirmation and customers will be notified

+1. > [!NOTE]

+ > On the **Offer overview** page, under **Publish status**, the scheduled deprecation date is shown.

+

+| [**Consulting service**](./plan-consulting-service-offer.md) | Consulting services help to connect customers with services to support and extend their use of Azure, Dynamics 365, Microsoft 365 or Power Suite services.|

+**Agentless dependency analysis** | Domain or non-domain (local) account with administrative permissions | Sudo user account with permissions to execute ls and netstat commands. If you are providing a sudo user account, ensure that you have enabled **NOPASSWD** for the account to run the required commands without prompting for a password every time the sudo command is invoked. <br /><br /> Alternatively, you can create a user account that has the CAP_DAC_READ_SEARCH and CAP_SYS_PTRACE permissions on /bin/netstat and /bin/ls files, set using the following commands:<br /><code>sudo setcap CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE=ep /bin/ls<br /> sudo setcap CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE=ep /bin/netstat</code>

+This article answers common questions about discovery, assessment, and dependency analysis in Azure Migrate. If you've other questions, check these resources:

+You can discover up to 10,000 servers from VMware environment, up to 5,000 servers from Hyper-V environment, and up to 1000 physical servers by using a single appliance. If you've more servers, read about [scaling a Hyper-V assessment](scale-hyper-v-assessment.md), [scaling a VMware assessment](scale-vmware-assessment.md), or [scaling a physical server assessment](scale-physical-assessment.md).

+- Use assessment type **Azure SQL** when you want to assess your on-premises SQL Server in your VMware, Microsoft Hyper-V, and Physical/Bare metal environments as well as IaaS Servers of other public clouds such as AWS, GCP, etc. for migration to SQL Server on Azure VM or Azure SQL Database or Azure SQL Managed Instance. [Learn More](concepts-azure-sql-assessment-calculation.md).

+- You can use a common group with VMware machines only to run both types of assessments. If you're running AVS assessments in Azure Migrate for the first time, it's advisable to create a new group of VMware machines.

+For "Performance-based" assessment, the assessment report export says 'PercentageOfCoresUtilizedMissing' or 'PercentageOfMemoryUtilizedMissing' when the Azure Migrate appliance can't collect performance data for the on-premises servers. Check:

+- If the servers are powered on for the duration for which you're creating the assessment

+- If only memory counters are missing and you're trying to assess servers in Hyper-V environment. In this scenario, enable dynamic memory on the servers and 'Recalculate' the assessment to reflect the latest changes. The appliance can collect memory utilization values for severs in Hyper-V environment only when the server has dynamic memory enabled.

+- If the SQL Servers are powered on for the duration for which you're creating the assessment.

+- If Azure Migrate connection status for all SQL instances is 'Connected' in the discovered SQL instance section.

+## Why is confidence rating not available for Azure App Service assessments?

+Performance data isn't captured for Azure App Service assessment and hence you don't see confidence rating for this assessment type. Azure App Service assessment takes configuration data of web apps in to account while performing assessment calculation.

+- You didn't profile your environment for the duration for which you're creating the assessment. For example, if you're creating an assessment with performance duration set to one week, you need to wait for at least a week after you start the discovery for all the data points to get collected. If you can't wait for the duration, change the performance duration to a smaller period and **Recalculate** the assessment.

+- Assessment isn't able to collect the performance data for some or all the servers in the assessment period. For a high confidence rating, ensure that:

+ - For Azure SQL assessments, Azure Migrate connection status for all SQL instances is "Connected" in the discovered SQL instance section.

+- For Azure VM and AVS assessments, few servers were created after discovery had started. For example, if you're creating an assessment for the performance history of last one month, but few servers were created in the environment only a week ago. In this case, the performance data for the new servers won't be available for the entire duration and the confidence rating would be low. [Learn more](./concepts-assessment-calculation.md#confidence-ratings-performance-based).

+- For Azure SQL assessments, few SQL instances or databases were created after discovery had started. For example, if you're creating an assessment for the performance history of last one month, but few SQL instances or databases were created in the environment only a week ago. In this case, the performance data for the new servers won't be available for the entire duration and the confidence rating would be low. [Learn more](./concepts-azure-sql-assessment-calculation.md#confidence-ratings).

+- You've chosen an Azure region where a particular series isn't supported. Azure VM families shown in Azure VM assessment properties are dependent on the availability of the VM series in the chosen Azure location, storage type and Reserved Instance.

+- The VM series isn't support in the assessment and isn't in the consideration logic of the assessment. We currently don't support B-series burstable, accelerated and high performance SKU series. We are trying to keep the VM series updated, and the ones mentioned are on our roadmap.

+ To remediate this, select the total number of assessments to navigate to all the assessments and recalculate the Azure VM or AVS assessment. The discovery and assessment tool will then show the correct count for that assessment type.

+Discovery and assessment of SQL Server instances and databases running in your VMware, Microsoft Hyper-V, and Physical/Bare metal environments as well as IaaS Servers of other public clouds such as AWS, GCP, etc. is now in preview. Get started with [this tutorial](tutorial-discover-vmware.md). If you want to try out this feature in an existing project, ensure that you've completed the [prerequisites](how-to-discover-sql-existing-project.md) in this article.

+Discovery and assessment of .NET web apps running in your VMware environment is now in preview. Get started with [this tutorial](tutorial-discover-vmware.md). If you want to try out this feature in an existing project, ensure that you've completed the [prerequisites](how-to-discover-sql-existing-project.md) in this article.

+- Azure SQL assessment can only be done on servers running where SQL instances were discovered. If you don't see the servers and SQL instances that you wish to assess, wait for some time for the discovery, and then create the assessment.

+- If you're not able to see a previously created group while creating the assessment, remove any server without a SQL instance from the group.

+- If you're running Azure SQL assessments in Azure Migrate for the first time, it's advisable to create a new group of servers.

+- Azure App Service assessment can only be done on servers running where web server role was discovered. If you don't see the servers that you wish to assess, wait for some time for the discovery to get completed, and then create the assessment.

+- If you're not able to see a previously created group while creating the assessment, remove any non-VMware server or any server without a web app from the group.

+- If you're running Azure App Service assessments in Azure Migrate for the first time, it's advisable to create a new group of servers.

+This can happen when one or more technical checks fail for a given web app. You may select the readiness status for the web app to find out details and remediation for failed checks.

+- There are some discovery issues that you need to fix in **Errors and notifications**.

+You can create a single **Azure SQL** assessment consisting of desired SQL servers across VMware, Microsoft Hyper-V and Physical/Bare metal environments as well as IaaS Servers of other public clouds such as AWS, GCP, etc. A single assessment covers readiness, SKUs, estimated costs and migration blockers for all the available SQL migration targets in Azure - Azure SQL Managed Instance, Azure SQL Database and SQL Server on Azure VM. You can then compare the assessment output for the desired targets. [Learn More](./concepts-azure-sql-assessment-calculation.md)

+- If you're running AVS assessments in Azure Migrate for the first time, it's advisable to create a new group of VMware machines.

+No. Currently, both Azure Migrate and Azure Site Recovery don't support migration to Ultra disks. Find steps to deploy Ultra disk [here](../virtual-machines/disks-enable-ultra-ssd.md?tabs=azure-portal#deploy-an-ultra-disk)

+ - The readiness criteria is less stringent in import-based assessments on the boot type parameter. If the boot type isn't provided, it's assumed the machine has BIOS boot type, and the machine isn't marked as **Conditionally Ready**. In assessments with discovery source as appliance, the readiness is marked as **Conditionally Ready** if the boot type is missing. This difference in readiness calculation is because users may not have all information on the machines in the early stages of migration planning when import-based assessments are done.

+For machines imported via a CSV file, the default migration tool in an AVS assessment is unknown. Though, for VMware machines, it's recommended to use the VMware Hybrid Cloud Extension (HCX) solution. [Learn More](../azure-vmware/install-vmware-hcx.md).

+Support | This option is currently in preview, and is only available for servers in VMware environment. [Review](migrate-support-matrix-vmware.md#dependency-analysis-requirements-agentless) supported operating systems. | In General Availability (GA).

+- If you've machines that don't have internet connectivity, download and install the Log Analytics gateway on them.

+You can [visualize dependencies](./how-to-create-a-group.md#refine-a-group-with-dependency-mapping) for groups that have up to 10 servers. If you've a group that has more than 10 servers, we recommend that you split the group into smaller groups, and then visualize the dependencies.

+**Supported servers** | supported only for servers running SQL Server in your VMware, Microsoft Hyper-V, and Physical/Bare metal environments as well as IaaS Servers of other public clouds such as AWS, GCP, etc. You can discover up to 300 SQL Server instances or 6,000 SQL databases, whichever is less.

+**Supported SQL configuration** | Currently, only discovery for standalone SQL Server instances and corresponding databases is supported.<br /><br /> Identification of Failover Cluster and Always On availability groups isn't supported.

+**Supported SQL services** | Only SQL Server Database Engine is supported. <br /><br /> Discovery of SQL Server Reporting Services (SSRS), SQL Server Integration Services (SSIS), and SQL Server Analysis Services (SSAS) isn't supported.

+[Dependency analysis](concepts-dependency-visualization.md) helps you analyze the dependencies between the discovered servers, which can be easily visualized with a map view in Azure Migrate project and can be used to group related servers for migration to Azure. The following table summarizes the requirements for setting up agentless dependency analysis:

+**Linux server access** | Sudo user account with permissions to execute ls and netstat commands. If you're providing a sudo user account, ensure that you have enabled **NOPASSWD** for the account to run the required commands without prompting for a password every time sudo command is invoked. <br /><br /> Alternatively, you can create a user account that has the CAP_DAC_READ_SEARCH and CAP_SYS_PTRACE permissions on /bin/netstat and /bin/ls files, set using the following commands:<br /><code>sudo setcap CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE=ep /bin/ls<br /> sudo setcap CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE=ep /bin/netstat</code>

+**Costs** | The Service Map solution doesn't incur any charges for the first 180 days (from the day that you associate the Log Analytics workspace with the project)/<br/><br/> After 180 days, standard Log Analytics charges will apply.<br/><br/> Using any solution other than Service Map in the associated Log Analytics workspace will incur [standard charges](https://azure.microsoft.com/pricing/details/log-analytics/) for Log Analytics.<br/><br/> When the project is deleted, the workspace isn't deleted along with it. After deleting the project, Service Map usage isn't free, and each node will be charged as per the paid tier of Log Analytics workspace/<br/><br/>If you have projects that you created before Azure Migrate general availability (GA- 28 February 2018), you might have incurred additional Service Map charges. To ensure payment after 180 days only, we recommend that you create a new project, since existing workspaces before GA are still chargeable.

+**Management** | When you register agents to the workspace, you use the ID and key provided by the project.<br/><br/> You can use the Log Analytics workspace outside Azure Migrate.<br/><br/> If you delete the associated project, the workspace isn't deleted automatically. [Delete it manually](../azure-monitor/logs/manage-access.md).<br/><br/> Don't delete the workspace created by Azure Migrate, unless you delete the project. If you do, the dependency visualization functionality won't work as expected.

+## Permissions for Windows server

+> [!Note]

+> For Windows Server 2008 and 2008 R2, ensure that WMF 3.0 is installed on the servers.

+> To discover SQL Server databases on Windows Servers, both Windows and SQL Server authentication are supported. You can provide credentials of both authentication types in the appliance configuration manager. Azure Migrate requires a Windows user account that is a member of the sysadmin server role.

+## Permissions for Linux server

+For Linux servers, based on the features you want to perform, you can create a user account in one of two ways:

+### Option 1

+- You need a sudo user account on the servers that you want to discover. This account can be used to pull configuration and performance metadata, perform software inventory (discovery of installed applications) and enable agentless dependency analysis using SSH connectivity.

+- You need to enable sudo access for the commands listed [here](discovered-metadata.md#linux-server-metadata). In addition to these commands, the user account also needs to have permissions to execute ls and netstat commands to perform agentless dependency analysis.

+- Make sure that you have enabled **NOPASSWD** for the account to run the required commands without prompting for a password every time sudo command is invoked.

+ Operating system | Versions

+ Red Hat Enterprise Linux | 5.1, 5.3, 5.11, 6.x, 7.x, 8.x

+ Cent OS | 5.1, 5.9, 5.11, 6.x, 7.x, 8.x

+ Ubuntu | 12.04, 14.04, 16.04, 18.04, 20.04

+ Oracle Linux | 6.1, 6.7, 6.8, 6.9, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9, 8, 8.1, 8.3, 8.5

+ SUSE Linux | 10, 11 SP4, 12 SP1, 12 SP2, 12 SP3, 12 SP4, 15 SP2, 15 SP3

+ Debian | 7, 8, 9, 10, 11

+> [!Note]

+> If you want to perform software inventory (discovery of installed applications) and enable agentless dependency analysis on Linux servers, it recommended to use Option 1.

+### Option 2

+- If you can't provide root account or user account with sudo access, then you can set 'isSudo' registry key to value '0' in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AzureAppliance registry on appliance server and provide a non-root account with the required capabilities using the following commands:

+- To perform agentless dependency analysis on the server, ensure that you also set the required permissions on /bin/netstat and /bin/ls files by using the following commands:<br /><code>sudo setcap CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE=ep /bin/ls<br /> sudo setcap CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE=ep /bin/netstat</code>

+**Supported servers** | supported only for servers running SQL Server in your VMware, Microsoft Hyper-V, and Physical/Bare metal environments as well as IaaS Servers of other public clouds such as AWS, GCP, etc. You can discover up to 300 SQL Server instances or 6,000 SQL databases, whichever is less.

+**vCenter Server** | Servers that you want to discover and assess must be managed by vCenter Server version 7.0, 6.7, 6.5, 6.0, or 5.5.<br /><br /> Discovering servers by providing ESXi host details in the appliance currently isn't supported. <br /><br /> IPv6 addresses aren't supported for vCenter Server (for discovery and assessment of servers) and ESXi hosts (for replication of servers).

+**Supported servers** | supported only for servers running SQL Server in your VMware, Microsoft Hyper-V, and Physical/Bare metal environments as well as IaaS Servers of other public clouds such as AWS, GCP, etc. You can discover up to 300 SQL Server instances or 6,000 SQL databases, whichever is less.

+**Supported SQL configuration** | Currently, only discovery for standalone SQL Server instances and corresponding databases is supported.<br /><br /> Identification of Failover Cluster and Always On availability groups isn't supported.

+**Supported SQL services** | Only SQL Server Database Engine is supported. <br /><br /> Discovery of SQL Server Reporting Services (SSRS), SQL Server Integration Services (SSIS), and SQL Server Analysis Services (SSAS) isn't supported.

+[Dependency analysis](concepts-dependency-visualization.md) helps you analyze the dependencies between the discovered servers, which can be easily visualized with a map view in Azure Migrate project and can be used to group related servers for migration to Azure. The following table summarizes the requirements for setting up agentless dependency analysis:

+**Linux servers** | Red Hat Enterprise Linux 5.1, 5.3, 5.11, 6.x, 7.x, 8.x <br /> Cent OS 5.1, 5.9, 5.11, 6.x, 7.x, 8.x <br /> Ubuntu 12.04, 14.04, 16.04, 18.04, 20.04 <br /> OracleLinux 6.1, 6.7, 6.8, 6.9, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9, 8, 8.1, 8.3, 8.5 <br /> SUSE Linux 10, 11 SP4, 12 SP1, 12 SP2, 12 SP3, 12 SP4, 15 SP2, 15 SP3 <br /> Debian 7, 8, 9, 10, 11

+**Linux server access** | Sudo user account with permissions to execute ls and netstat commands. If you're providing a sudo user account, ensure that you have enabled **NOPASSWD** for the account to run the required commands without prompting for a password every time a sudo command is invoked. <br /><br /> Alternatively, you can create a user account that has the CAP_DAC_READ_SEARCH and CAP_SYS_PTRACE permissions on /bin/netstat and /bin/ls files, set using the following commands:<br /><code>sudo setcap CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE=ep /bin/ls<br /> sudo setcap CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE=ep /bin/netstat</code>

+**Cost** | The Service Map solution doesn't incur any charges for the first 180 days (from the day you associate the Log Analytics workspace with the project).<br /><br /> After 180 days, standard Log Analytics charges apply.<br /><br /> Using any solution other than Service Map in the associated Log Analytics workspace will incur [standard charges](https://azure.microsoft.com/pricing/details/log-analytics/) for Log Analytics.<br /><br /> When the project is deleted, the workspace isn't automatically deleted. After deleting the project, Service Map usage isn't free, and each node will be charged per the paid tier of Log Analytics workspace.<br /><br />If you have projects that you created before Azure Migrate general availability (February 28, 2018), you might have incurred additional Service Map charges. To ensure that you're charged only after 180 days, we recommend that you create a new project. Workspaces that were created before GA are still chargeable.

+ - For **Linux servers**, you need a root account on the Linux servers that you want to discover. Refer to the instructions in the [support matrix](migrate-support-matrix-physical.md#permissions-for-linux-server) for an alternative.

+ - If you are using a root user to discover your Linux servers, ensure root sign in is allowed on the servers.

+1. Select **Generate key** to start the creation of the required Azure resources. Do not close the Discover servers page during the creation of resources.

+In **2: Download Azure Migrate appliance**, select **Download**.

+ Alternately, you can open the app from the desktop by selecting the app shortcut.

+ :::image type="content" source="./media/tutorial-discover-vmware/device-code.png" alt-text="Screenshot that shows where to copy the device code and sign in.":::

+ > If you close the sign in tab accidentally without signing in, refresh the browser tab of the appliance configuration manager to display the device code and Copy code & Login button.

+ 5. After you successfully sign in, return to the browser tab that displays the appliance configuration manager. If the Azure user account that you used to sign in has the required permissions for the Azure resources that were created during key generation, appliance registration starts.

+1. In **Step 1: Provide credentials for discovery of Windows and Linux physical or virtual serversΓÇï**, select **Add credentials**.

+1. For Windows server, select the source type as **Windows Server**, specify a friendly name for credentials, add the username and password. Select **Save**.

+1. If you are using password-based authentication for Linux server, select the source type as **Linux Server (Password-based)**, specify a friendly name for credentials, add the username and password. Select **Save**.

+1. If you are using SSH key-based authentication for Linux server, you can select source type as **Linux Server (SSH key-based)**, specify a friendly name for credentials, add the username, browse, and select the SSH private key file. Select **Save**.

+1. If you want to add multiple credentials at once, select **Add more** to save and add more credentials. Multiple credentials are supported for physical servers discovery.

+1. In **Step 2:Provide physical or virtual server detailsΓÇï**, select **Add discovery source** to specify the server **IP address/FQDN** and the friendly name for credentials to connect to the server.

+ - If you choose **Add single item**, you can choose the OS type, specify friendly name for credentials, add server **IP address/FQDN**, and select **Save**.

+ - If you choose **Add multiple items**, you can add multiple records at once by specifying server **IP address/FQDN** with the friendly name for credentials in the text box. Verify** the added records and select **Save**.

+ - If you choose **Import CSV** _(selected by default)_, you can download a CSV template file, populate the file with the server **IP address/FQDN** and friendly name for credentials. You then import the file into the appliance, **verify** the records in the file and select **Save**.

+1. On selecting Save, appliance will try validating the connection to the servers added and show the **Validation status** in the table against each server.

+ - If validation fails for a server, review the error by selecting **Validation failed** in the Status column of the table. Fix the issue, and validate again.

+ - To remove a server, select **Delete**.

+Select **Start discovery**, to kick off discovery of the successfully validated servers. After the discovery has been successfully initiated, you can check the discovery status against each server in the table.

+2. In **Servers, databases and web apps** > **Azure Migrate: Discovery and assessment** page, select the icon that displays the count for **Discovered servers**.

+ * You need a root account on the Linux servers that you want to discover. If you aren't able to provide a root account, refer to the instructions in the [support matrix](migrate-support-matrix-physical.md#permissions-for-linux-server) for an alternative.

+ * If you're using a root user to discover your Linux servers, ensure root login is allowed on the servers.

+3. In **1:Generate project key**, provide a name for the Azure Migrate appliance that you'll set up for discovery of your GCP virtual servers. The name should be alphanumeric with 14 characters or fewer.

+4. Click **Generate key** to start the creation of the required Azure resources. Don't close the Discover servers page during the creation of resources.

+6. Copy the key as you'll need it to complete the registration of the appliance during its configuration.

+ :::image type="content" source="./media/tutorial-discover-vmware/device-code.png" alt-text="Screenshot that shows where to copy the device code and sign in.":::

+ > If you close the sign in tab accidentally without logging in, refresh the browser tab of the appliance configuration manager to display the device code and Copy code & Login button.

+ 5. After you successfully sign in, return to the browser tab that displays the appliance configuration manager. If the Azure user account that you used to sign in has the required permissions for the Azure resources that were created during key generation, appliance registration starts.

+1. In **Step 1: Provide credentials for discovery of Windows and Linux physical or virtual serversΓÇï**, select **Add credentials**.

+1. For Windows server, select the source type as **Windows Server**, specify a friendly name for credentials, add the username and password. Select **Save**.

+1. If you're using password-based authentication for Linux server, select the source type as **Linux Server (Password-based)**, specify a friendly name for credentials, add the username and password. Select **Save**.

+1. If you're using SSH key-based authentication for Linux server, you can select source type as **Linux Server (SSH key-based)**, specify a friendly name for credentials, add the username, browse and select the SSH private key file. Select **Save**.

+ - Currently Azure Migrate doesn't support passphrase-based SSH key. Use an SSH key without a passphrase.

+ - Currently Azure Migrate doesn't support SSH private key file generated by PuTTY.

+2. If you want to add multiple credentials at once, select **Add more** to save and add more credentials.

+3. In **Step 2:Provide physical or virtual server detailsΓÇï**, select **Add discovery source** to specify the server **IP address/FQDN** and the friendly name for credentials to connect to the server.

+4. You can either **Add single item** at a time or **Add multiple items** in one go. There's also an option to provide server details through **Import CSV**.

+ - If you choose **Add single item**, you can choose the OS type, specify friendly name for credentials, add server **IP address/FQDN** and select **Save**.

+ - If you choose **Add multiple items**, you can add multiple records at once by specifying server **IP address/FQDN** with the friendly name for credentials in the text box. Verify** the added records and select **Save**.

+ - If you choose **Import CSV** _(selected by default)_, you can download a CSV template file, populate the file with the server **IP address/FQDN** and friendly name for credentials. You then import the file into the appliance, **verify** the records in the file and select **Save**.

+ - To remove a server, select **Delete**.

+* For **Windows servers**, create an account (local or domain) that has administrator permissions on the servers. To discover SQL Server instances and databases, the Windows or SQL Server account must be a member of the sysadmin server role. Learn how to [assign the required role to the user account](/sql/relational-databases/security/authentication-access/server-level-roles).

+* For **Linux servers**, provide a sudo user account with permissions to execute ls and netstat commands or create a user account that has the CAP_DAC_READ_SEARCH and CAP_SYS_PTRACE permissions on /bin/netstat and /bin/ls files. If you're providing a sudo user account, ensure that you have enabled **NOPASSWD** for the account to run the required commands without prompting for a password every time sudo command is invoked.

+3. In **1:Generate project key**, provide a name for the Azure Migrate appliance that you'll set up for discovery of servers. The name should be alphanumeric with 14 characters or fewer.

+1. Select **Generate key** to start the creation of the required Azure resources. Don't close the Discover server page during the creation of resources.

+1. Copy the key as you'll need it to complete the registration of the appliance during its configuration.

+In **2: Download Azure Migrate appliance**, select the .VHD file and select **Download**.

+1. In **Step 1: Provide Hyper-V host credentials**, select **Add credentials** to specify a friendly name for credentials, add **Username** and **Password** for a Hyper-V host/cluster that the appliance will use to discover servers. select **Save**.

+1. If you want to add multiple credentials at once, select **Add more** to save and add more credentials. Multiple credentials are supported for discovery of servers in Hyper-V environment.

+1. In **Step 2: Provide Hyper-V host/cluster details**, select **Add discovery source** to specify the Hyper-V host/cluster **IP address/FQDN** and the friendly name for credentials to connect to the host/cluster.

+1. You can either **Add single item** at a time or **Add multiple items** in one go. There's also an option to provide Hyper-V host/cluster details through **Import CSV**.

+ - If you choose **Add single item**, you need to specify friendly name for credentials and Hyper-V host/cluster **IP address/FQDN**, and select **Save**.

+ - If you choose **Add multiple items** _(selected by default)_, you can add multiple records at once by specifying Hyper-V host/cluster **IP address/FQDN** with the friendly name for credentials in the text box. **Verify** the added records and select **Save**.

+ - If you choose **Import CSV**, you can download a CSV template file, populate the file with the Hyper-V host/cluster **IP address/FQDN** and friendly name for credentials. You then import the file into the appliance, **verify** the records in the file and select **Save**.

+ - To remove hosts or clusters, select **Delete**.

+Select **Start discovery**, to kick off server discovery from the successfully validated host(s)/cluster(s). After the discovery has been successfully initiated, you can check the discovery status against each host/cluster in the table.

+## Prepare Windows server

+For Windows servers, use a domain account for domain-joined servers, and a local account for servers that aren't domain-joined. The user account can be created in one of the two ways:

+> [!Note]

+> To discover SQL Server databases on Windows Servers, both Windows and SQL Server authentication are supported. You can provide credentials of both authentication types in the appliance configuration manager. Azure Migrate requires a Windows user account that is a member of the sysadmin server role.

+## Prepare Linux server

+For Linux servers, you can create a user account in one of two ways:

+- You need a sudo user account on the servers that you want to discover. This account can be used to pull configuration and performance metadata and perform software inventory (discovery of installed applications) and enable agentless dependency analysis using SSH connectivity.

+- You need to enable sudo access for the commands listed [here](discovered-metadata.md#linux-server-metadata). In addition to these commands, the user account also needs to have permissions to execute ls and netstat commands to perform agentless dependency analysis.

+- Make sure that you have enabled **NOPASSWD** for the account to run the required commands without prompting for a password every time sudo command is invoked.

+- The Linux OS distributions that are supported for discovery by Azure Migrate using an account with sudo access are listed [here](migrate-support-matrix-physical.md#option-1-1).

+- If you can't provide user account with sudo access, then you can set 'isSudo' registry key to value '0' in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AzureAppliance registry on the appliance server and provide a non-root account with the required capabilities using the following commands:

+ **Command** | **Purpose**

+ | |

+ setcap CAP_DAC_READ_SEARCH+eip /usr/sbin/fdisk <br></br> setcap CAP_DAC_READ_SEARCH+eip /sbin/fdisk _(if /usr/sbin/fdisk is not present)_ | To collect disk configuration data

+ setcap "cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_setuid,<br> cap_setpcap,cap_net_bind_service,cap_net_admin,cap_sys_chroot,cap_sys_admin,<br> cap_sys_resource,cap_audit_control,cap_setfcap=+eip" /sbin/lvm | To collect disk performance data

+ setcap CAP_DAC_READ_SEARCH+eip /usr/sbin/dmidecode | To collect BIOS serial number

+ chmod a+r /sys/class/dmi/id/product_uuid | To collect BIOS GUID

+- To perform agentless dependency analysis on the server, ensure that you also set the required permissions on /bin/netstat and /bin/ls files by using the following commands:<br /><code>sudo setcap CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE=ep /bin/ls<br /> sudo setcap CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE=ep /bin/netstat</code>

+3. In **1:Generate project key**, provide a name for the Azure Migrate appliance that you'll set up for discovery of physical or virtual servers. The name should be alphanumeric with 14 characters or fewer.

+1. Select **Generate key** to start the creation of the required Azure resources. Do not close the Discover servers page during the creation of resources.

+1. Copy the key as you'll need it to complete the registration of the appliance during its configuration.

+1. In **Step 1: Provide credentials for discovery of Windows and Linux physical or virtual serversΓÇï**, select **Add credentials**.

+1. For Windows server, select the source type as **Windows Server**, specify a friendly name for credentials, add the username and password. Select **Save**.

+1. If you're using password-based authentication for Linux server, select the source type as **Linux Server (Password-based)**, specify a friendly name for credentials, add the username and password. Select **Save**.

+1. If you're using SSH key-based authentication for Linux server, you can select source type as **Linux Server (SSH key-based)**, specify a friendly name for credentials, add the username, browse and select the SSH private key file. Select **Save**.

+ - Currently Azure Migrate doesn't support passphrase-based SSH key. Use an SSH key without a passphrase.

+ - Currently Azure Migrate doesn't support SSH private key file generated by PuTTY.

+1. If you want to add multiple credentials at once, select **Add more** to save and add more credentials. Multiple credentials are supported for physical servers discovery.

+1. In **Step 2:Provide physical or virtual server detailsΓÇï**, select **Add discovery source** to specify the server **IP address/FQDN** and the friendly name for credentials to connect to the server.

+1. You can either **Add single item** at a time or **Add multiple items** in one go. There's also an option to provide server details through **Import CSV**.

+ - If you choose **Add single item**, you can choose the OS type, specify friendly name for credentials, add server **IP address/FQDN** and select **Save**.

+ - If you choose **Add multiple items**, you can add multiple records at once by specifying server **IP address/FQDN** with the friendly name for credentials in the text box. **Verify** the added records and select **Save**.

+ - If you choose **Import CSV** _(selected by default)_, you can download a CSV template file, populate the file with the server **IP address/FQDN** and friendly name for credentials. You then import the file into the appliance, **verify** the records in the file and select **Save**.

+ - To remove a server, select **Delete**.

+select **Start discovery**, to kick off discovery of the successfully validated servers. After the discovery has been successfully initiated, you can check the discovery status against each server in the table.

+1. If **App registrations** is set to **No**, request the tenant, or global admin to assign the required permissions. Alternately, the tenant or global admin can assign the Application Developer role to an account to allow Azure AD app registration by users. [Learn more](../active-directory/fundamentals/active-directory-users-assign-role-azure-portal.md).

+* For **Windows servers** and web apps discovery, create an account (local or domain) that has administrator permissions on the servers. To discover SQL Server instances and databases, the Windows or SQL Server account must be a member of the sysadmin server role. Learn how to [assign the required role to the user account](/sql/relational-databases/security/authentication-access/server-level-roles).

+* For **Linux servers**, provide a sudo user account with permissions to execute ls and netstat commands or create a user account that has the CAP_DAC_READ_SEARCH and CAP_SYS_PTRACE permissions on /bin/netstat and /bin/ls files. If you're providing a sudo user account, ensure that you have enabled **NOPASSWD** for the account to run the required commands without prompting for a password every time sudo command is invoked.

+ :::image type="content" source="./media/tutorial-discover-vmware/device-code.png" alt-text="Screenshot that shows where to copy the device code and sign in.":::

+ > If you close the sign in tab accidentally without logging in, refresh the browser tab of the appliance configuration manager to display the device code and Copy code & Login button.

+ 5. After you successfully sign in, return to the browser tab that displays the appliance configuration manager. If the Azure user account that you used to sign in has the required permissions for the Azure resources that were created during key generation, appliance registration starts.

+ - If you want to add multiple credentials at once, select **Add more** to save and add more credentials. Multiple credentials are supported for discovery of servers across multiple vCenter Servers using a single appliance.

+ Select **Add more** to save the previous details and add more vCenter Server details. **You can add up to 10 vCenter Servers per appliance.**

+1. Select the discovered servers count to review the discovered inventory. You can filter the inventory by selecting the appliance name and selecting one or more vCenter Servers from the **Source** filter.

+ms.

+## Update (November 2022)

+- Support for providing a sudo user account to perform agentless dependency analysis on Linux servers running in VMware, Hyper-V, and Physical/other cloud environments.

+- Private preview: Plan replication. This is a new feature added to the Migration and Modernization tool of Azure Migrate. It helps to estimate the time and resources required for replication and migration of the discovered servers. This feature will help in planning the replication and migration schedule. Currently, the feature is available for VMware agentless migrations. To enroll for private preview, please fill this [form](https://forms.office.com/pages/responsepage.aspx?id=v4j5cvGGr0GRqy180BHbR2QMeqnlmM1An_w4S8FpvJ5UNEQ1VEgzNUpEUE0xODBHVUdVVUxGMUdVNS4u).

+ - *Right sized Lift and Shift* - Server to *SQL Server on Azure VM*. We recommend this when SQL Server credentials aren't available.

+- Migration of UEFI-based VMs and physical servers to Azure generation 2 VMs is now supported. With this release, Azure Migrate: Server Migration tool won't perform the conversion from Gen 2 VM to Gen 1 VM during migration.

+NGINX for Azure (preview) delivers secure and high performance applications using familiar and trusted load balancing solutions. Use NGINX for Azure (preview) as a reverse proxy within your Azure environment.

+ | AzureBastionSubnet address space | Enter **10.1.1.0/26** |

+ | Image | Select **Windows Server 2022 Datacenter - Gen2**. |

+ | Network policy for private endpoints | Select **edit** to apply Network security groups and/or Route tables to the subnet that contains the private endpoint. </br> In **Edit subnet network policy**, select the checkbox next to **Network security groups** and **Route Tables**. </br> Select **Save**. </br></br>For more information, see [Manage network policies for private endpoints](disable-private-endpoint-network-policy.md) |

+Use the virtual machine that you created earlier to connect to the web app across the private endpoint.

+| `insight.prod.ext.web.purview.azure.com` | A | \<account private endpoint IP address of Microsoft Purview> |

+Ensure you have the *Data Source Admin* permission as described [here](how-to-enable-data-use-management.md#configure-microsoft-purview-permissions-for-publishing-data-owner-policies)

+Ensure you have the *Data Source Admin* permission as described [here](how-to-enable-data-use-management.md#configure-microsoft-purview-permissions-for-publishing-data-owner-policies)

+ Title: Connect to and manage Azure Arc-enabled SQL Server

+description: This guide describes how to connect to Azure Arc-enabled SQL Server in Microsoft Purview, and use Microsoft Purview features to scan and manage your Azure Arc-enabled SQL Server source.

+# Connect to and manage Azure Arc-enabled SQL Server in Microsoft Purview (public preview)

+This article shows how to register an Azure Arc-enabled SQL Server instance. It also shows how to authenticate and interact with Azure Arc-enabled SQL Server in Microsoft Purview. For more information about Microsoft Purview, read the [introductory article](overview.md).

+|Metadata extraction|Full scan|Incremental scan|Scoped scan|Classification|Access policy|Lineage|Data sharing|

+\** Lineage is supported if the dataset is used as a source/sink in the [Azure Data Factory copy activity](how-to-link-azure-data-factory.md).

+The supported SQL Server versions are 2012 and later. SQL Server Express LocalDB is not supported.

+When you're scanning Azure Arc-enabled SQL Server, Microsoft Purview supports extracting the following technical metadata:

+- Instances

+- Databases

+- Schemas

+- Tables, including the columns

+- Views, including the columns

+When you're setting up a scan, you can choose to specify the database name to scan one database. You can further scope the scan by selecting tables and views as needed. The whole Azure Arc-enabled SQL Server instance will be scanned if you don't provide a database name.

+* Data Source Administrator and Data Reader permissions to register a source and manage it in the Microsoft Purview governance portal. See [Access control in the Microsoft Purview governance portal](catalog-permissions.md) for details.

+* The latest [self-hosted integration runtime](https://www.microsoft.com/download/details.aspx?id=39717). For more information, see [Create and manage a self-hosted integration runtime](manage-integration-runtimes.md).

+This section describes how to register an Azure Arc-enabled SQL Server instance in Microsoft Purview by using the [Microsoft Purview governance portal](https://web.purview.azure.com/).

+There are two ways to set up authentication for scanning Azure Arc-enabled SQL Server with a self-hosted integration runtime:

+- Windows authentication

+- SQL Server authentication

+To configure authentication for the SQL Server deployment:

+1. In SQL Server Management Studio (SSMS), go to **Server Properties**, and then select **Security** on the left pane.

+1. Under **Server authentication**:

+ - For Windows authentication, select either **Windows Authentication mode** or **SQL Server and Windows Authentication mode**.

+ - For SQL Server authentication, select **SQL Server and Windows Authentication mode**.

+ :::image type="content" source="media/register-scan-azure-arc-enabled-sql-server/enable-sql-server-authentication.png" alt-text="Screenshot that shows the Security page of the Server Properties window, with options for selecting authentication mode.":::

+A change to the server authentication requires you to restart the SQL Server instance and SQL Server Agent. In SSMS, go to the SQL Server instance and select **Restart** on the right-click options pane.

+#### Create a new login and user

+If you want to create a new login and user to scan your SQL Server instance, use the following steps.

+The account must have access to the master database, because `sys.databases` is in the master database. The Microsoft Purview scanner needs to enumerate `sys.databases` in order to find all the SQL databases on the server.

+> You can run all the following steps by using [this code](https://github.com/Azure/Purview-Samples/blob/master/TSQL-Code-Permissions/grant-access-to-on-prem-sql-databases.sql).

+1. Go to SSMS, connect to the server, and then select **Security** on the left pane.

+1. Select and hold (or right-click) **Login**, and then select **New login**. If Windows authentication is applied, select **Windows authentication**. If SQL Server authentication is applied, select **SQL Server authentication**.

+ :::image type="content" source="media/register-scan-azure-arc-enabled-sql-server/create-new-login-user.png" alt-text="Screenshot that shows selections for creating a new login and user.":::

+1. Select **Server Roles** on the left pane, and ensure that a public role is assigned.

+1. Select **User Mapping** on the left pane, select all the databases in the map, and then select the **db_datareader** database role.

+1. Select **OK** to save.

+1. If SQL Server authentication is applied, you must change your password as soon as you create a new login:

+ 1. Select and hold (or right-click) the user that you created, and then select **Properties**.

+ 1. Enter a new password and confirm it.

+ 1. Select the **Specify old password** checkbox and enter the old password.

+ 1. Select **OK**.

+ :::image type="content" source="media/register-scan-azure-arc-enabled-sql-server/change-password.png" alt-text="Screenshot that shows selections for changing a password.":::

+#### Store your SQL Server login password in a key vault and create a credential in Microsoft Purview

+1. Go to your key vault in the Azure portal. Select **Settings** > **Secrets**.

+1. Select **+ Generate/Import**. For **Name** and **Value**, enter the password from your SQL Server login.

+1. Select **Create**.

+1. If your key vault is not connected to Microsoft Purview yet, [create a new key vault connection](manage-credentials.md#create-azure-key-vaults-connections-in-your-microsoft-purview-account).

+1. [Create a new credential](manage-credentials.md#create-a-new-credential) by using the username and password to set up your scan.

+ Be sure to select the right authentication method when you're creating a new credential. If Windows authentication is applied, select **Windows authentication**. If SQL Server authentication is applied, select **SQL Server authentication**.

+1. Go to your Microsoft Purview account.

+1. Under **Sources and scanning** on the left pane, select **Integration runtimes**. Make sure that a self-hosted integration runtime is set up. If it isn't set up, [follow the steps to create a self-hosted integration runtime](manage-integration-runtimes.md) for scanning on an on-premises or Azure virtual machine that has access to your on-premises network.

+1. Select **Data Map** on the left pane.

+1. Select **Register**.

+1. Select **SQL Server on Azure Arc-enabled servers**, and then select **Continue**.

+ :::image type="content" source="media/register-scan-azure-arc-enabled-sql-server/set-up-azure-arc-enabled-sql-data-source.png" alt-text="Screenshot that shows selecting a SQL data source.":::

+1. Provide a friendly name, which is a short name that you can use to identify your server. Also provide the server endpoint.

+Use the following steps to scan Azure Arc-enabled SQL Server instances to automatically identify assets and classify your data. For more information about scanning in general, see [Scans and ingestion in Microsoft Purview](concept-scans-and-ingestion.md).

+To create and run a new scan:

+1. In the [Microsoft Purview governance portal](https://web.purview.azure.com/resource/), select the **Data Map** tab on the left pane.

+1. Select the Azure Arc-enabled SQL Server source that you registered.

+1. Select **New scan**.

+1. Select the credential to connect to your data source. Credentials are grouped and listed under the authentication methods.

+ :::image type="content" source="media/register-scan-azure-arc-enabled-sql-server/azure-arc-enabled-sql-set-up-scan-win-auth.png" alt-text="Screenshot that shows selecting a credential for a scan.":::

+ :::image type="content" source="media/register-scan-azure-arc-enabled-sql-server/azure-arc-enabled-sql-scope-your-scan.png" alt-text="Screenshot that shows selected assets for scoping a scan.":::

+1. Select a scan rule set. You can choose between the system default, existing custom rule sets, or creation of a new rule set inline.

+ :::image type="content" source="media/register-scan-azure-arc-enabled-sql-server/azure-arc-enabled-sql-scan-rule-set.png" alt-text="Screenshot that shows selecting a scan rule set.":::

+ :::image type="content" source="media/register-scan-azure-arc-enabled-sql-server/trigger-scan.png" alt-text="Screenshot that shows setting up a recurring scan trigger.":::

+1. Review your scan, and then select **Save and run**.

+- [Data Owner policies](concept-policies-data-owner.md) (preview)

+### Access policy prerequisites on Azure Arc-enabled SQL Server

+Before you can create policies, you must register the Azure Arc-enabled SQL Server data source with Microsoft Purview:

+1. Go to **Data map** on the left pane, select **Sources**, and then select **Register**. Enter **Azure Arc** in the search box and select **SQL Server on Azure Arc**. Then select **Continue**.

+ 

+1. For **Name**, enter a name for this registration. It's best practice to make the name of the registration the same as the server name in the next step.

+1. Select values for **Azure subscription**, **Server name**, and **Server endpoint**.

+1. For **Select a collection**, choose a collection to put this registration in.

+1. Enable **Data use management**. **Data use management** needs certain permissions and can affect the security of your data, because it delegates to certain Microsoft Purview roles to manage access to the data sources. Go through the secure practices related to **Data use management** in this guide: [Enable Data use management on your Microsoft Purview sources](./how-to-enable-data-use-management.md).

+1. After you enable **Data use management**, Microsoft Purview automatically captures the application ID of the app registration that's related to this Azure Arc-enabled SQL Server instance. Come back to this screen and select the refresh button, in case the association between Azure Arc-enabled SQL Server and the app registration changes in the future.

+1. Select **Register** or **Apply**.

+

+To create an access policy for Azure Arc-enabled SQL Server, follow these guides:

+* [DevOps policy on a single Azure Arc-enabled SQL Server instance](./how-to-policies-devops-arc-sql-server.md#create-a-new-devops-policy)

+* [Data owner policy on a single Azure Arc-enabled SQL Server instance](./how-to-policies-data-owner-arc-sql-server.md#create-and-publish-a-data-owner-policy)

+To create policies that cover all data sources inside a resource group or Azure subscription, see [Discover and govern multiple Azure sources in Microsoft Purview](register-scan-azure-multiple-sources.md#access-policy).

+Now that you've registered your source, use the following guides to learn more about Microsoft Purview and your data:

+- [Search the data catalog](how-to-search-catalog.md)

+To fetch policies via delta pull, send a `GET` request to /policyEvents as follows:

+| [Azure Container Apps](../container-apps/disaster-recovery.md) |  |

+| Support | Details|

+|-- | -|

+|Move support | Azure resources that are supported for a move with Resource Mover can be moved from any public region to another public region and within regions in China. Moving resources within Azure Gov is also supported (US DoD Central, US DoD East, US Gov Arizona, US Gov Texas, US Gov Virginia). US Sec East/West/West Central are not currently supported.|

+|Metadata support | Supported regions for storing metadata about machines to be moved include East US2, North Europe, Southeast Asia, Japan East, UK South, and Australia East as metadata regions. <br/><br/> Moving resources within the Azure China region is also supported with the metadata region China North2.|

+When a resource is selected for move, the corresponding resource group is added automatically for moving. This is so that the destination resource can be placed in a resource group. You can choose to customize and provide an existing resource group after it's added for move. Moving a resource group doesn't mean that all the resources in the source resource group will be moved.

+It's stored in an [Azure Cosmos DB](../cosmos-db/database-encryption-at-rest.md) database, and in [Azure Blob storage](../storage/common/storage-service-encryption.md), in a Microsoft subscription. Currently, metadata is stored in East US 2 and North Europe. We will expand this coverage to other regions. This doesn't restrict you from moving resources across any public region.

+There are a couple of reasons you might not have permission.

+|Possible cause | Recommendation|

+|-- | --|

+|You're not a *Contributor* and *User Access Administrator* (or *Owner*) when you add a resource for the first time. | Use an account with *Contributor* and *User Access Administrator* (or *Owner*) permissions for the subscription.|

+|The Resource Mover managed identity doesn't have the required role. | Add the 'Contributor' and 'User Access administrator' roles. |

+|The Resource Mover managed identity was reset to *None*. | Reenable a system-assigned identity in the move collection settings > **Identity**. Alternatively, in **Add Resources**, add the resource again, which does the same thing. |

+|The subscription was moved to a different tenant. | Disable and then enable managed identity for the move collection.|

+You can remove resources that you've added to the move list. The exact remove behavior depends on the resource state. [Learn more](remove-move-resources.md#vm-resource-state-after-removing).

+- **Align for services/features**: Move resources to take advantage of the services or features that are available in a specific region.

+- **Respond to deployment requirements**: Move resources that were deployed in error or move in response to capacity needs.

+- An easy way to identify dependencies across resources you want to move. This feature helps you to move related resources together so that everything works as expected in the target region after the move.

+- Testing. You can try out a move and then discard it if you don't want to do a full move.

+To move resources across regions, you select the resources that you want to move. Resource Mover validates those resources and resolves any dependencies they have on other resources. If there are dependencies, you have a couple of options:

+- Don't move the dependent resources but use equivalent resources in the target region instead.

+2. After the initial move, you can decide whether to commit and complete the move or discard the move.

+3. After the move is done you can decide whether you want to delete the resources in the source location.

+You can move resources across regions in the Resource Mover hub or from within a resource group. [Learn more](select-move-tool.md)

+- Encrypted Azure VMs and associated disks. This includes VMs with Azure disk encryption enabled and Azure VMs using default server-side encryption (both with platform-managed keys and customer-managed keys)

+- **Move resources across regions**: Move resources from within the Resource Mover hub, or a resource group.

+| Tool | When to use| Learn more |

+|-- | - | - |

+| **Move within resource group** | Move resources to a different resource group/subscription, or across regions.<br/><br/> If you move across regions, in the resource group you select the resources you want to move, and then you move to the Resource Mover hub, to verify dependencies and move the resources to the target region. | [Move resources to another resource group/subscription](../azure-resource-manager/management/move-resource-group-and-subscription.md).<br/><br/> [Move resources to another region from a resource group](move-region-within-resource-group.md). |

+| **Move from the Resource Mover hub** | Move resources across regions. <br/><br/> You can move to a target region, or to a specific availability zone, or availability set, within the target region. | [Move resources across regions in the Resource Mover hub](). |

+| **Move VMs with Site Recovery** | Use it for moving Azure VMs between government and public clouds.<br/><br/> Use if you want to move VMs between availability zones in the same region. |[Move resources between government/public clouds](../site-recovery/region-move-cross-geos.md), [Move resources to availability zones in the same region](../site-recovery/azure-to-azure-how-to-enable-zone-to-zone-disaster-recovery.md).|

+> * Prepare to move VMs and select resources in the source region that you want to move them from.

+| Requirement |Details |

+| | -|

+|**Subscription permissions** | Check to ensure that you have *Owner* access on the subscription that contains the resources you want to move.<br/><br/> *Why do I need Owner access?* The first time you add a resource for a specific source and destination pair in an Azure subscription, Resource Mover creates a [system-assigned managed identity](../active-directory/managed-identities-azure-resources/overview.md#managed-identity-types), formerly known as the Managed Service Identity (MSI). This identity is trusted by the subscription. Before you can create the identity and assign it the required roles (*Contributor* and *User access administrator* in the source subscription), the account you use to add resources needs *Owner* permissions in the subscription. For more information, see [Classic subscription administrator roles, Azure roles, and Azure AD roles](../role-based-access-control/rbac-and-directory-admin-roles.md#azure-roles).|

+| **VM support** | Check to ensure that the VMs you want to move are supported by doing the following:<li>[Verify](support-matrix-move-region-azure-vm.md#windows-vm-support) supported Windows VMs.<li>[Verify](support-matrix-move-region-azure-vm.md#linux-vm-support) supported Linux VMs and kernel versions.<li>Check supported [compute](support-matrix-move-region-azure-vm.md#supported-vm-compute-settings), [storage](support-matrix-move-region-azure-vm.md#supported-vm-storage-settings), and [networking](support-matrix-move-region-azure-vm.md#supported-vm-networking-settings) settings.|

+| **Key vault requirements (Azure Disk Encryption)** | If you have Azure Disk Encryption enabled for VMs, you need a key vault in both the source and destination regions. For more information, see [Create a key vault](../key-vault/general/quick-create-portal.md).<br/><br/> For the key vaults in the source and destination regions, you need these permissions:<li>Key permissions: Key Management Operations (Get, List) and Cryptographic Operations (Decrypt and Encrypt)<li>Secret permissions: Secret Management Operations (Get, List, and Set)<li>Certificate (List and Get)|

+| **Disk encryption set (server-side encryption with CMK)** | If you're using VMs with server-side encryption that uses a CMK, you need a disk encryption set in both the source and destination regions. For more information, see [Create a disk encryption set](../virtual-machines/disks-enable-customer-managed-keys-portal.md#set-up-your-disk-encryption-set).<br/><br/> Moving between regions isn't supported if you're using a hardware security module (HSM keys) for customer-managed keys.|

+| **Target region quota** | The subscription needs enough quota to create the resources you're moving in the target region. If it doesn't have a quota, [request additional limits](../azure-resource-manager/management/azure-subscription-service-limits.md).|

+| **Target region charges** | Verify the pricing and charges that are associated with the target region to which you're moving the VMs. Use the [pricing calculator](https://azure.microsoft.com/pricing/calculator/).|

+Keys <br></br> If you're using a KEK, you need these permissions in addition to the permissions for secrets. | *Get*, *Create*, and *Encrypt* <br></br> Select **Key Permissions** > **Key Management Operations**, and then select **Get** and **Create**. In **Cryptographic Operations**, select **Encrypt**.

+- The cache storage has a lock that must be deleted before it can be deleted.

+1. Locate the resources in the resource group ```RegionMoveRG-<sourceregion>-<target-region>```.

+1. Check to ensure that all the VMs and other source resources in the source region have been moved or deleted. This step ensures that no pending resources are using them.

+> Tutorials show the quickest path for trying out a scenario and use default options.

+| Requirement | Description |

+| | |

+| **Subscription permissions** | Check you have *Owner* access on the subscription containing the resources that you want to move<br/><br/> **Why do I need Owner access?** The first time you add a resource for a specific source and destination pair in an Azure subscription, Resource Mover creates a [system-assigned managed identity](../active-directory/managed-identities-azure-resources/overview.md#managed-identity-types) (formerly known as Managed Service Identify (MSI)) that's trusted by the subscription. To create the identity, and to assign it the required role (Contributor or User Access administrator in the source subscription), the account you use to add resources needs *Owner* permissions on the subscription. [Learn more](../role-based-access-control/rbac-and-directory-admin-roles.md#azure-roles) about Azure roles. |

+| **Resource Mover support** | [Review](common-questions.md) supported regions and other common questions.|

+| **VM support** | Check that any VMs you want to move are supported.<br/><br/> - [Verify](support-matrix-move-region-azure-vm.md#windows-vm-support) supported Windows VMs.<br/><br/> - [Verify](support-matrix-move-region-azure-vm.md#linux-vm-support) supported Linux VMs and kernel versions.<br/><br/> - Check supported [compute](support-matrix-move-region-azure-vm.md#supported-vm-compute-settings), [storage](support-matrix-move-region-azure-vm.md#supported-vm-storage-settings), and [networking](support-matrix-move-region-azure-vm.md#supported-vm-networking-settings) settings.|

+| **SQL support** | If you want to move SQL resources, review the [SQL requirements list](tutorial-move-region-sql.md#check-sql-requirements).|

+| **Destination subscription** | The subscription in the destination region needs enough quota to create the resources you're moving in the target region. If it doesn't have a quota, [request additional limits](../azure-resource-manager/management/azure-subscription-service-limits.md).|

+| **Destination region charges** | Verify pricing and charges associated with the target region to which you're moving VMs. Use the [pricing calculator](https://azure.microsoft.com/pricing/calculator/) to help you. |

+| **Operation** | **Portal** | **PowerShell** |

+| | | |

+| **Create a move collection** | A move collection (a list of all the resources you're moving) is created automatically. Required identity permissions are assigned in the backend by the portal. | You use PowerShell cmdlets to:<br/><br/> - Create a resource group for the move collection and specify the location for it.<br/><br/> - Assign a managed identity to the collection.<br/><br/> - Add resources to the collection.|

+| **Remove a move collection** | You can't directly remove a move collection in the portal. | You use a PowerShell cmdlet to remove a move collection.|

+| **Resource move operations**<br/><br/> (Prepare, initiate move, commit, etc.).| Single steps with automatic validation by Resource Mover. | PowerShell cmdlets to:<br/><br/> 1) Validate dependencies.<br/><br/> 2) Perform the move.|

+| **Delete source resources** | Directly in the Resource Mover portal. | PowerShell cmdlets at the resource-type level. |

+| **Setting** | **Value** |

+| | |

+| Subscription ID | subscription-id |

+| Source region | Central US |

+| Target region | West Central US |

+| Resource group (holding metadata for move collection) | RG-MoveCollection-demoRMS |

+| Move collection name | PS-centralus-westcentralus-demoRMS |

+| Resource group (source region) | PSDemoRM |

+| Resource group (target region) | PSDemoRM-target |

+| Resource Move service location | East US 2 |

+| IdentityType | SystemAssigned |

+| VM to move | PSDemoVM |

+The MoveCollection object stores metadata and configuration information about the resources you want to move. To set up a move collection, you do the following:

+2. Create the target resource settings object per the resource you're moving. In our case, it's a VM.

+- To move stateful resources such as Azure VMs and SQL databases, you might need to start replicating resources from the source to the destination region.

+ > You can provide the source resource ID instead of the resource name as the input parameters for the Prepare cmdlet, as well as in the Initiate Move and Commit cmdlets. To do this, run:

+After the initial move, you can decide whether you want to commit the move or discard it.

+> Tutorials show the quickest path for trying out a scenario and using default options.

+| Requirement | Description |

+| | |

+| **Resource Mover support** | [Review](common-questions.md) supported regions and other common questions. |

+| **Subscription permissions** | Check you have *Owner* access on the subscription containing the resources that you want to move<br/><br/> **Why do I need Owner access?** The first time you add a resource for a specific source and destination pair in an Azure subscription, Resource Mover creates a [system-assigned managed identity](../active-directory/managed-identities-azure-resources/overview.md#managed-identity-types) (formerly known as Managed Service Identify (MSI)) that's trusted by the subscription. To create the identity, and to assign it the required role (Contributor or User Access administrator in the source subscription), the account you use to add resources needs *Owner* permissions on the subscription. [Learn more](../role-based-access-control/rbac-and-directory-admin-roles.md#azure-roles) about Azure roles.|

+| **VM support** | Check that the VMs you want to move are supported.<br/><br/> - [Verify](support-matrix-move-region-azure-vm.md#windows-vm-support) supported Windows VMs.<br/><br/> - [Verify](support-matrix-move-region-azure-vm.md#linux-vm-support) supported Linux VMs and kernel versions.<br/><br/> - Check supported [compute](support-matrix-move-region-azure-vm.md#supported-vm-compute-settings), [storage](support-matrix-move-region-azure-vm.md#supported-vm-storage-settings), and [networking](support-matrix-move-region-azure-vm.md#supported-vm-networking-settings) settings.|

+| **Destination subscription** | The subscription in the destination region needs enough quota to create the resources you're moving in the target region. If it doesn't have a quota, [request additional limits](../azure-resource-manager/management/azure-subscription-service-limits.md).

+**Destination region charges** | Verify pricing and charges associated with the target region to which you're moving VMs. Use the [pricing calculator](https://azure.microsoft.com/pricing/calculator/) to help you.|

+Select the resources you want to move.

+> - If you want to remove a resource from a move collection, the method for doing that depends on where you are in the move process. [Learn more](remove-move-resources.md).

+ - Show all dependencies and iterates through all of the direct and indirect dependencies for a resource. For example, for a VM it shows the NIC, virtual network, network security groups (NSGs), etc.

+ - Show first-level dependencies only shows only direct dependencies. For example, for a VM it shows the NIC, but not the virtual network.

+> - After moving resources, they're in a *Commit move pending* state.

+After the initial move, you can decide whether you want to commit the move or discard it.

+- Automation rules can also be the mechanism by which you run a playbook in response to an **alert** *not associated with an incident*.

+1. If the health feature isnΓÇÖt enabled, [enable it](enable-monitoring.md).

+1. If the health feature isnΓÇÖt enabled, [enable it](enable-monitoring.md).

+ Title: Use Logstash to stream logs with pipeline transformations via DCR-based API

+description: Use Logstash to forward logs from external data sources into custom and standard tables in Microsoft Sentinel, and to configure the output with DCRs.

+# Use Logstash to stream logs with pipeline transformations via DCR-based API

+> [!IMPORTANT]

+> Data ingestion using the Logstash output plugin with Data Collection Rules (DCRs) is currently in public preview. This feature is provided without a service level agreement. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+Microsoft Sentinel's new Logstash output plugin supports pipeline transformations and advanced configuration via Data Collection Rules (DCRs). The plugin forwards any type of logs from external data sources into custom or standard tables in Microsoft Sentinel.

+In this article, you learn how to set up the new Logstash plugin to stream the data into Microsoft Sentinel using DCRs, with full control over the output schema. Learn how to **[deploy the plugin](#deploy-the-microsoft-sentinel-output-plugin-in-logstash)**.

+> [!NOTE]

+> A [previous version of the Logstash plugin](connect-logstash.md) allows you to connect data sources through Logstash via the Data Collection API.

+With the new plugin, you can:

+- Control the configuration of the column names and types.

+- Perform ingestion-time transformations like filtering or enrichment.

+- Ingest custom logs into a custom table, or ingest a Syslog input stream into the Microsoft Sentinel Syslog table.

+Ingestion into standard tables is limited only to [standard tables supported for custom logs ingestion](data-transformation.md#data-transformation-support-for-custom-data-connectors).

+To learn more about working with the Logstash data collection engine, see [Getting started with Logstash](https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html).

+## Overview

+### Architecture and background

+The Logstash engine is comprised of three components:

+- Input plugins: Customized collection of data from various sources.

+- Filter plugins: Manipulation and normalization of data according to specified criteria.

+- Output plugins: Customized sending of collected and processed data to various destinations.

+> [!NOTE]

+> - Microsoft supports only the Microsoft Sentinel-provided Logstash output plugin discussed here. The current plugin is named **[microsoft-sentinel-logstash-output-plugin](https://github.com/Azure/Azure-Sentinel/tree/master/DataConnectors/microsoft-sentinel-logstash-output-plugin)**, v1.0.0. You can [open a support ticket](https://portal.azure.com/#create/Microsoft.Support) for any issues regarding the output plugin.

+>

+> - Microsoft does not support third-party Logstash output plugins for Microsoft Sentinel, or any other Logstash plugin or component of any type.

+>

+> - See the [prerequisites](#prerequisites) for the pluginΓÇÖs Logstash version support.

+The Microsoft Sentinel output plugin for Logstash sends JSON-formatted data to your Log Analytics workspace, using the Log Analytics HTTP Data Collector REST API. The data is ingested into custom logs.

+- Learn more about the [Log Analytics REST API](/rest/api/loganalytics/create-request).

+- Learn more about [custom logs](../azure-monitor/agents/data-sources-custom-logs.md).

+## Deploy the Microsoft Sentinel output plugin in Logstash

+**To set up the plugin, follow these steps**:

+1. Review the [prerequisites](#prerequisites)

+1. [Install the plugin](#install-the-plugin)

+1. [Create a sample file](#create-a-sample-file)

+1. [Create the required DCR-related resources](#create-the-required-dcr-resources)

+1. [Configure Logstash configuration file](#configure-logstash-configuration-file)

+1. [Restart Logstash](#restart-logstash)

+1. [View inc oming logs in Microsoft Sentinel](#view-incoming-logs-in-microsoft-sentinel)

+1. [Monitor output plugin audit logs](#monitor-output-plugin-audit-logs)

+### Prerequisites

+- Install a supported version of Logstash. The plugin supports:

+ - Logstash version 7.0 to 7.17.6.

+ - Logstash version 8.0 to 8.4.2.

+

+ > [!NOTE]

+ > If you use Logstash 8, we recommended that you [disable ECS in the pipeline](https://www.elastic.co/guide/en/logstash/8.4/ecs-ls.html).

+- Verify that you have a Log Analytics workspace with at least contributor rights.

+- Verify that you have permissions to create DCR objects in the workspace.

+### Install the plugin

+The Microsoft Sentinel output plugin is available in the Logstash collection.

+- Follow the instructions in the Logstash [Working with plugins](https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html) document to install the **[microsoft-logstash-output-azure-loganalytics](https://github.com/Azure/Azure-Sentinel/tree/master/DataConnectors/microsoft-logstash-output-azure-loganalytics)** plugin.

+- If your Logstash system does not have Internet access, follow the instructions in the Logstash [Offline Plugin Management](https://www.elastic.co/guide/en/logstash/current/offline-plugins.html) document to prepare and use an offline plugin pack. (This will require you to build another Logstash system with Internet access.)

+

+### Create a sample file

+In this section, you create a sample file in one of these scenarios:

+- [Create a sample file for custom logs](#create-a-sample-file-for-custom-logs)

+- [Create a sample file to ingest logs into the Syslog table](#create-a-sample-file-to-ingest-logs-into-the-syslog-table)

+#### Create a sample file for custom logs

+

+In this scenario, you configure the Logstash input plugin to send events to Microsoft Sentinel. For this example, we use the generator input plugin to simulate events. You can use any other input plugin.

+In this example, the Logstash configuration file looks like this:

+```

+input {

+ generator {

+ lines => [

+ "This is a test log message"

+ ]

+ count => 10

+ }

+}

+```

+1. Copy the output plugin configuration below to your Logstash configuration file.

+ ```

+ output {

+ microsoft-sentinel-logstash-output-plugin {

+ create_sample_file => true

+ sample_file_path => "<enter the path to the file in which the sample data will be written>" #for example: "c:\\temp" (for windows) or "/tmp" for Linux.

+ }

+ }

+ ```

+1. To make sure that the referenced file path exists before creating the sample file, start Logstash.

+

+ The plugin writes ten records to a sample file named `sampleFile<epoch seconds>.json` in the configured path. For example: *c:\temp\sampleFile1648453501.json*.

+ Here is part of a sample file that the plugin creates:

+

+ ```

+ [

+ {

+ "host": "logstashMachine",

+ "sequence": 0,

+ "message": "This is a test log message",

+ "ls_timestamp": "2022-03-28T17:45:01.690Z",

+ "ls_version": "1"

+ },

+ {

+ "host": "logstashMachine",

+ "sequence": 1

+ ...

+

+ ]

+ ```

+ The plugin automatically adds these properties to every record:

+ - `ls_timestamp`: The time when the record is received from the input plugin

+ - `ls_version`: The Logstash pipeline version.

+

+ You can remove these fields when you [create the DCR](#create-the-required-dcr-resources).

+#### Create a sample file to ingest logs into the Syslog table

+In this scenario, you configure the Logstash input plugin to send syslog events to Microsoft Sentinel.

+1. If you don't already have syslog messages forwarded into your Logstash machine, you can use the logger command to generate messages. For example (for Linux):

+ ```

+ logger -p local4.warn --rfc3164 --tcp -t CEF: "0|Microsoft|Device|cef-test|example|data|1|here is some more data for the example" -P 514 -d -n 127.0.0.1

+ Here is an example for the Logstash input plugin:

+ input {

+ syslog {

+ port => 514

+ }

+ }