-| scope |Required |A space-separated list of scopes. The `openid` scope indicates a permission to sign in the user and get data about the user in the form of ID tokens. The `offline_access` scope is optional for web applications. It indicates that your application will need a *refresh token* for extended access to resources. The `https://{tenant-name}/{app-id-uri}/{scope}` indicates a permission to protected resources, such as a web API. For more information, see [Request an access token](access-tokens.md#scopes). |

-1. Select **OAuth consent screen** in the left menu, select **External**, and then select **Create**.

-Enter a **Name** for your application. Enter *b2clogin.com* in the **Authorized domains** section and select **Save**.

-## SAML entityID customization

-If you have multiple SAML applications that depend on different entityID values, you can override the `issueruri` value in your relying party file. To do this, copy the technical profile with the "Saml2AssertionIssuer" ID from the base file and override the `issueruri` value.

-> [!TIP]

-> Copy the `<ClaimsProviders>` section from the base and preserve these elements within the claims provider: `<DisplayName>Token Issuer</DisplayName>`, `<TechnicalProfile Id="Saml2AssertionIssuer">`, and `<DisplayName>Token Issuer</DisplayName>`.

-

-Example:

-```xml

- <ClaimsProviders>

- <ClaimsProvider>

- <DisplayName>Token Issuer</DisplayName>

- <TechnicalProfiles>

- <TechnicalProfile Id="Saml2AssertionIssuer">

- <DisplayName>Token Issuer</DisplayName>

- <Metadata>

- <Item Key="IssuerUri">customURI</Item>

- </Metadata>

- </TechnicalProfile>

- </TechnicalProfiles>

- </ClaimsProvider>

- </ClaimsProviders>

- <RelyingParty>

- <DefaultUserJourney ReferenceId="SignUpInSAML" />

- <TechnicalProfile Id="PolicyProfile">

- <DisplayName>PolicyProfile</DisplayName>

- <Protocol Name="SAML2" />

- <Metadata>

- …

-```

- * The reply URL must begin with the scheme `https`.

-2. ECMA Connector Host receives the GET request and queries its internal cache to see if the user exists and has based imported. This is done using the **query attribute**. The query attribute is defined in the object types page.

- [](.\media\on-premises-application-provisioning-architecture\match-2.png#lightbox)

-| `ctldl.windowsupdate.com` <br> `www.microsoft.com/pkiops` | 80/HTTP | The connector uses this URL during the registration process. |

-| ctldl.windowsupdate.com<br>www.microsoft.com/pkiops | 80/HTTP | The connector uses this URL during the registration process. |

- | AZURE_MFA_HOSTNAME | adnotifications.windowsazure.us |

- | AZURE_MFA_HOSTNAME | adnotifications.windowsazure.cn |

-Verify that https://adnotifications.windowsazure.com is reachable from the server running the NPS extension.

-| iOS | Microsoft Edge, Safari |

-> Edge 85+ requires the user to be signed in to the browser to properly pass device identity. Otherwise, it behaves like Chrome without the accounts extension. This sign-in might not occur automatically in a Hybrid Azure AD Join scenario.

-This rest of this article uses the Azure portal to configure and demonstrate Azure AD entitlement management. You can also follow a tutorial to [manage access to resources via Microsoft Graph](/graph/tutorial-access-package-api?toc=/azure/active-directory/governance/toc.json&bc=/azure/active-directory/governance/breadcrumb/toc.json) or [via PowerShell](/powershell/microsoftgraph/tutorial-entitlement-management?view=graph-powershell-beta).

-> | Create Azure AD Domain Services instance | [Application Administrator](../roles/permissions-reference.md#application-administrator) and [Groups Administrator](../roles/permissions-reference.md#groups-administrator)|[Domain Services Contributor](/azure/role-based-access-control/built-in-roles#domain-services-contributor) |

-To configure single sign-on on **Code42** side, you need to send the **App Federation Metadata Url** to [Code42 support team](mailto:idpsupport@code42.com). They set this setting to have the SAML SSO connection set properly on both sides.

-In this section, you create a user called B.Simon in Code42. Work with [Code42 support team](mailto:idpsupport@code42.com) to add the users in the Code42 platform. Users must be created and activated before you use single sign-on.

-To update the API server authorized IP ranges on an existing cluster, use [az aks update][az-aks-update] command and use the *`--api-server-authorized-ip-ranges`*,--load-balancer-outbound-ip-prefixes*, *`--load-balancer-outbound-ips`*, or--load-balancer-outbound-ip-prefixes* parameters.

- <domain to be rewritten>.com:53 {

- log

- errors

- rewrite stop {

- name regex (.*)\.<domain to be rewritten>.com {1}.default.svc.cluster.local

- answer name (.*)\.default\.svc\.cluster\.local {1}.<domain to be rewritten>.com

- }

- forward . /etc/resolv.conf # you can redirect this to a specific DNS server such as 10.0.0.10, but that server must be able to resolve the rewritten domain name

-}

-Add-ons provide extra capabilities for your AKS cluster and their installation and configuration is managed Azure. Use `az aks addon` to manage all add-ons for your cluster.

-[virtual-networks-name-resolution]: ../virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances.md#name-resolution-that-uses-your-own-dns-server

-* Fault domain count can only be 1.

-[az-vm-host-group-create]: /cli/azure/vm/host/group#az_vm_host_group_create

-If migration is supported for your App Service Environment, there are three ways to access the migration feature. These methods include a banner at the top of the Overview page, a new item in the left-hand side menu called **Migration (preview)**, and an info box on the **Configuration** page. Select any of these methods to move on to the next step in the migration process.

-If you don't see these elements, your App Service Environment isn't supported for migration at this time or your environment is in an unhealthy or suspended state (which blocks migration). If your environment [won't be supported for migration](migrate.md#supported-scenarios) or you want to migrate to App Service Environment v3 without using the migration feature, see the [manual migration options](migration-alternatives.md).

-Under **Generate new IP addresses**, confirm you understand the implications and start the process. This step will take about 15 minutes to complete. Don't scale or make changes to your existing App Service Environment during this time. If you may see a message a few minutes after starting this step asking you to refresh the page, select refresh as shown in the sample to allow your new IP addresses to appear.

-App Service Environment v3 requires the subnet it's in to have a single delegation of `Microsoft.Web/hostingEnvironments`. Previous versions didn't require this delegation. You'll need to confirm your subnet is delegated properly and update the delegation if needed before migrating. A link to your subnet is given so that you can confirm and update as needed.

-| [!INCLUDE [VS Code connect app to postgres step 3](<./includes/tutorial-python-postgresql-app/connect-postgres-to-app-visual-studio-code-3.md>)] | :::image type="content" source="./media/tutorial-python-postgresql-app/connect-app-to-database-settings-example-a-240px.png" lightbox="./media/tutorial-python-postgresql-app/connect-app-to-database-settings-example-a.png" alt-text="A screenshot showing adding setting name for app service to connect to Postgresql database in VS Code." ::: :::image type="content" source="./media/tutorial-python-postgresql-app/connect-app-to-database-settings-example-b-240px.png" lightbox="./media/tutorial-python-postgresql-app/connect-app-to-database-settings-example-b.png" alt-text="A screenshot showing adding setting value for app service to connect to Postgresql database in VS Code." ::: |

- const poller = await client.beginAnalyzeDocuments("prebuilt-document", formUrl);

- const poller = await client.beginAnalyzeDocuments("prebuilt-layout", formUrl);

- const poller = await client.beginAnalyzeDocuments(PrebuiltModels.Invoice, invoiceUrl);

-1. Select the **Kubernetes - Azure Arc** object name.

-1. Select **Settings** > **Extensions**. Select **Add**.

-1. Select **Azure Arc data controller**.

-1. Click **Create**.

-||Speech service|See [STT API docs](../cognitive-services/speech-service/rest-speech-to-text.md#regions-and-endpoints)|[Speech Studio](https://speech.azure.us/)</br></br>See [Speech service endpoints](../cognitive-services/Speech-Service/sovereign-clouds.md)</br></br>**Speech translation endpoints**</br>Virginia: `https://usgovvirginia.s2s.speech.azure.us`</br>Arizona: `https://usgovarizona.s2s.speech.azure.us`</br>||

-Microsoft is currently in the process of adding TomTom, Moovit, and AccuWeather to the Online Services Subcontractor List.

-> The alert recommendations feature is currently in preview and is only enabled for VMs.

-If you don't have any alert rules defined for the selected resource, you can enable our recommended out-of-the-box alert rules.

-> The alert recommendations feature is currently in preview and is only enabled for VMs.

-If you don't have any alerts defined for the selected resource, you can [create a new alert rule](alerts-log.md#create-a-new-log-alert-rule-in-the-azure-portal), or [enable recommended out-of-the-box alert rules in the Azure portal (preview)](alerts-log.md#enable-recommended-out-of-the-box-alert-rules-in-the-azure-portal-preview).

-> The alert recommendations feature is currently in preview and is only enabled for VMs.

-If you don't have any alerts defined for the selected resource, you can [create a new alert rule](alerts-log.md#create-a-new-log-alert-rule-in-the-azure-portal), or [enable recommended out-of-the-box alert rules in the Azure portal (preview)](alerts-log.md#enable-recommended-out-of-the-box-alert-rules-in-the-azure-portal-preview).

- }

-Customer scenarios where we visualize this having the most impact:

-Your connection string is displayed on the Overview blade of your Application Insights resource.

-In this example, this connection string specifies the endpoint suffix and the SDK will construct service endpoints.

-In this example, this connection string specifies explicit overrides for every service. The SDK will use the exact endpoints provided without modification.

-To get the value for `clientIdOfSPNOrMsi`, you can run the command `az aks show` as shown in the following example. If the `servicePrincipalProfile` object has a valid `clientid` value, you can use that. Otherwise, if it's set to `msi`, you need to pass in the client ID from `addonProfiles.omsagent.identity.clientId`.

-* If you move a volume to a capacity pool of a higher service level (for example, moving from *Standard* to *Premium* or *Ultra* service level), you must wait at least seven days before you can move that volume *again* to a capacity pool of a lower service level (for example, moving from *Ultra* to *Premium* or *Standard*).

-* [Capacity management FAQs](faq-capacity-management.md)

-|Supported Software and Services |Azure Device Update <br> [Azure IoT](https://azure.microsoft.com/overview/iot/) <br> [Azure IoT Hub](https://azure.microsoft.com/services/iot-hub/) <br> [Azure IoT Central](https://azure.microsoft.com/services/iot-central/) <br> [Azure IoT Edge](https://azure.microsoft.com/services/iot-edge/) and [Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/category/internet-of-things?page=1) <br> [Azure Container Registry](https://azure.microsoft.com/services/container-registry/) <br> [Azure Mariner OS with Connectivity](https://github.com/microsoft/CBL-Mariner) <br> [Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/) <br> [ONNX Runtime](https://www.onnxruntime.ai/) <br> [TensorFlow](https://www.tensorflow.org/) <br> [Azure Analysis Services](https://azure.microsoft.com/services/analysis-services/) <br> IoT Plug and Play <br> [Azure Device Provisioning Service (DPS)](../iot-dps/index.yml) <br> [Azure Cognitive Services](https://azure.microsoft.com/services/cognitive-services/) <br> [Power BI](https://powerbi.microsoft.com/) |

- 1. Copy the executable to the **$HOME/.azure/bin** directory on an air-gapped machine.

- 1. Copy the executable to the **$HOME/.azure/bin** directory on an air-gapped machine.

- 1. Copy the executable to the **%UserProfile%/.azure/bin** directory on an air-gapped machine.

-# Create a SignalR Service

-This script creates a new SignalR Service resource and a new resource group.

-```azurecli-interactive

-#!/bin/bash

-# Generate a unique suffix for the service name

-let randomNum=$RANDOM*$RANDOM

-# Generate a unique service and group name with the suffix

-SignalRName=SignalRTestSvc$randomNum

-#resource name must be lowercase

-mySignalRSvcName=${SignalRName}

-myResourceGroupName=$SignalRName"Group"

-# Create resource group

-az group create --name $myResourceGroupName --location eastus

-# Create the Azure SignalR Service resource

-az signalr create \

- --name $mySignalRSvcName \

- --resource-group $myResourceGroupName \

- --sku Standard_S1 \

- --unit-count 1 \

- --service-mode Default

-# Get the SignalR primary connection string

-primaryConnectionString=$(az signalr key list --name $mySignalRSvcName \

- --resource-group $myResourceGroupName --query primaryConnectionString -o tsv)

-echo "$primaryConnectionString"

-Make a note of the actual name generated for the new resource group. You will use that resource group name when you want to delete all group resources.

-## Script explanation

-If you choose to install and use the CLI locally, this article requires that you are running the Azure CLI version 2.0 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install the Azure CLI]( /cli/azure/install-azure-cli).

-## Sample script

-This script uses the *signalr* extension for the Azure CLI. Execute the following command to install the *signalr* extension for the Azure CLI before using this sample script:

-```azurecli-interactive

-#!/bin/bash

-#========================================================================

-#=== Update these values based on your desired deployment username ===

-#=== and password. ===

-#========================================================================

-deploymentUser=<Replace with your desired username>

-deploymentUserPassword=<Replace with your desired password>

-#========================================================================

-#=== Update these values based on your GitHub OAuth App registration. ===

-#========================================================================

-GitHubClientId=<Replace with your GitHub OAuth app Client ID>

-GitHubClientSecret=<Replace with your GitHub OAuth app Client Secret>

-# Generate a unique suffix for the service name

-let randomNum=$RANDOM*$RANDOM

-# Generate unique names for the SignalR service, resource group,

-# app service, and app service plan

-SignalRName=SignalRTestSvc$randomNum

-#resource name must be lowercase

-mySignalRSvcName=${SignalRName,,}

-myResourceGroupName=$SignalRName"Group"

-myWebAppName=SignalRTestWebApp$randomNum

-myAppSvcPlanName=$myAppSvcName"Plan"

-# Create resource group

-az group create --name $myResourceGroupName --location eastus

-# Create the Azure SignalR Service resource

-az signalr create \

- --name $mySignalRSvcName \

- --resource-group $myResourceGroupName \

- --sku Standard_S1 \

- --unit-count 1 \

- --service-mode Default

-# Create an App Service plan.

-az appservice plan create --name $myAppSvcPlanName --resource-group $myResourceGroupName --sku FREE

-# Create the Web App

-az webapp create --name $myWebAppName --resource-group $myResourceGroupName --plan $myAppSvcPlanName

-# Get the SignalR primary connection string

-primaryConnectionString=$(az signalr key list --name $mySignalRSvcName \

- --resource-group $myResourceGroupName --query primaryConnectionString -o tsv)

-#Add an app setting to the web app for the SignalR connection

-az webapp config appsettings set --name $myWebAppName --resource-group $myResourceGroupName \

- --settings "Azure:SignalR:ConnectionString=$primaryConnectionString"

-#Add app settings to use with GitHub authentication

-az webapp config appsettings set --name $myWebAppName --resource-group $myResourceGroupName \

- --settings "GitHubClientId=$GitHubClientId"

-az webapp config appsettings set --name $myWebAppName --resource-group $myResourceGroupName \

- --settings "GitHubClientSecret=$GitHubClientSecret"

-# Add the desired deployment user name and password

-az webapp deployment user set --user-name $deploymentUser --password $deploymentUserPassword

-# Configure Git deployment and note the deployment URL in the output

-az webapp deployment source config-local-git --name $myWebAppName --resource-group $myResourceGroupName \

- --query [url] -o tsv

-Make a note of the actual name generated for the new resource group. It will be shown in the output. You will use that resource group name when you want to delete all group resources.

-## Script explanation

-This script uses the *signalr* extension for the Azure CLI. Execute the following command to install the *signalr* extension for the Azure CLI before using this sample script:

-```azurecli-interactive

-#!/bin/bash

-# Generate a unique suffix for the service name

-let randomNum=$RANDOM*$RANDOM

-# Generate unique names for the SignalR service, resource group,

-# app service, and app service plan

-SignalRName=SignalRTestSvc$randomNum

-#resource name must be lowercase

-mySignalRSvcName=${SignalRName,,}

-myResourceGroupName=$SignalRName"Group"

-myWebAppName=SignalRTestWebApp$randomNum

-myAppSvcPlanName=$myAppSvcName"Plan"

-# Create resource group

-az group create --name $myResourceGroupName --location eastus

-# Create the Azure SignalR Service resource

-az signalr create \

- --name $mySignalRSvcName \

- --resource-group $myResourceGroupName \

- --sku Standard_S1 \

- --unit-count 1 \

- --service-mode Default

-# Create an App Service plan.

-az appservice plan create --name $myAppSvcPlanName --resource-group $myResourceGroupName --sku FREE

-# Create the Web App

-az webapp create --name $myWebAppName --resource-group $myResourceGroupName --plan $myAppSvcPlanName

-# Get the SignalR primary connection string

-primaryConnectionString=$(az signalr key list --name $mySignalRSvcName \

- --resource-group $myResourceGroupName --query primaryConnectionString -o tsv)

-#Add an app setting to the web app for the SignalR connection

-az webapp config appsettings set --name $myWebAppName --resource-group $myResourceGroupName \

- --settings "AzureSignalRConnectionString=$primaryConnectionString"

-Make a note of the actual name generated for the new resource group. It will be shown in the output. You will use that resource group name when you want to delete all group resources.

-## Script explanation

-For comprehensive details, see the other articles in this series: [Checklist](performance-guidelines-best-practices-checklist.md), [VM size](performance-guidelines-best-practices-vm-size.md), [Storage](performance-guidelines-best-practices-storage.md), [Security](security-considerations-best-practices.md), [HADR configuration](hadr-cluster-best-practices.md), [Collect baseline](performance-guidelines-best-practices-collect-baseline.md).

-Enable [SQL Assessment for SQL Server on Azure VMs](sql-assessment-for-sql-vm.md) and your SQL Server will be evaluated against known best practices and results shown on the [SQL VM management page](manage-sql-vm-portal.md) of the Azure portal.

-For video introductions and the latest features on Azure SQL VM optimization and management automation, review this video series from Data Exposed:

-The following is a quick checklist of VM size best practices for running your SQL Server on Azure VM:

-For comprehensive details, see the other articles in this series: [Checklist](performance-guidelines-best-practices-checklist.md), [Storage](performance-guidelines-best-practices-storage.md), [Security](security-considerations-best-practices.md), [HADR configuration](hadr-cluster-best-practices.md), [Collect baseline](performance-guidelines-best-practices-collect-baseline.md).

-Review the following checklist for a brief overview of the VM size best practices that the rest of the article covers in greater detail:

-To compare the VM size checklist with the others, see the comprehensive [Performance best practices checklist](performance-guidelines-best-practices-checklist.md).

-When you are creating a SQL Server on Azure VM, carefully consider the type of workload necessary. If you are migrating an existing environment, [collect a performance baseline](performance-guidelines-best-practices-collect-baseline.md) to determine your SQL Server on Azure VM requirements. If this is a new VM, then create your new SQL Server VM based on your vendor requirements.

-Start the development environments with the lower-tier D-Series, B-Series, or Av2-series and grow your environment over time.

-Use the SQL Server VM Azure Marketplace images with the storage configuration in the portal. This will make it easier to properly create the storage pools necessary to get the size, IOPS, and throughput required for your workloads. It is important to choose SQL Server VMs that support premium storage and premium storage caching. See the [storage](performance-guidelines-best-practices-storage.md) article to learn more.

-The recommended minimum for a production OLTP environment is 4 vCore, 32 GB of memory, and a memory-to-vCore ratio of 8. For new environments, start with 4 vCore machines and scale to 8, 16, 32 vCores or more when your data and compute requirements change. For OLTP throughput, target SQL Server VMs that have 5000 IOPS for every vCore.

-The [memory optimized virtual machine sizes](../../../virtual-machines/sizes-memory.md) are a primary target for SQL Server VMs and the recommended choice by Microsoft. The memory optimized virtual machines offer higher memory-to-vCore ratios and medium-to-large cache options.

-The [Edsv5-series](../../../virtual-machines/edv5-edsv5-series.md#edsv5-series) is designed for memory-intensive applications and is the VM series that Microsoft recommends for most SQL Server workloads. These VMs have a large local storage SSD capacity, up to 672 GiB of RAM, and the highest local and remote storage throughput currently available in Azure. There is a nearly consistent 8 GiB of memory per vCore across most of these virtual machines, which is ideal for most SQL Server workloads. These VMs offer the best price-performance for SQL Server workloads running on Azure virtual machines.

-The [general purpose virtual machine sizes](../../../virtual-machines/sizes-general.md) are designed to provide balanced memory-to-vCore ratios for smaller entry level workloads such as development and test, web servers, and smaller database servers.

-Because of the smaller memory-to-vCore ratios with the general purpose virtual machines, it is important to carefully monitor memory-based performance counters to ensure SQL Server is able to get the buffer cache memory it needs. See [memory performance baseline](performance-guidelines-best-practices-collect-baseline.md#memory) for more information.

-These machines are ideal for side-by-side SQL and app deployments that require fast access to temp storage and departmental relational databases. There is a standard memory-to-vCore ratio of 4 across all of the virtual machines in this series.

-The [burstable B-series](../../../virtual-machines/sizes-b-series-burstable.md) virtual machine sizes are ideal for workloads that do not need consistent performance such as proof of concept and very small application and development servers.

-This series is unique as the apps have the ability to **burst** during business hours with burstable credits varying based on machine size.

-> [!NOTE]

-These virtual machines are both good options for smaller development and test SQL Server machines.

-> [!NOTE]

-The [storage optimized VM sizes](../../../virtual-machines/sizes-storage.md) are for specific use cases. These virtual machines are specifically designed with optimized disk throughput and IO.

-The [Lsv2-series](../../../virtual-machines/lsv2-series.md) features high throughput, low latency, and local NVMe storage. The Lsv2-series VMs are optimized to use the local disk on the node attached directly to the VM rather than using durable data disks.

-The Lsv2 and Ls series support [premium storage](../../../virtual-machines/premium-storage-performance.md), but not premium storage caching. The creation of a local cache to increase IOPs is not supported.

-> Storing your data files on the ephemeral NVMe storage could result in data loss when the VM is deallocated.

-High performing SQL Server workloads often need larger amounts of memory, I/O, and throughput without the higher vCore counts.

-Most OLTP workloads are application databases driven by large numbers of smaller transactions. With OLTP workloads, only a small amount of the data is read or modified, but the volumes of transactions driven by user counts are much higher. It is important to have the SQL Server memory available to cache plans, store recently accessed data for performance, and ensure physical reads can be read into memory quickly.

-These OLTP environments need higher amounts of memory, fast storage, and the I/O bandwidth necessary to perform optimally.

-In order to maintain this level of performance without the higher SQL Server licensing costs, Azure offers VM sizes with [constrained vCPU counts](../../../virtual-machines/constrained-vcpu.md).

-These new VM sizes have a suffix that specifies the number of active vCPUs to make them easier to identify.

-> [!NOTE]

-> - Medium to large data warehouse workloads may still benefit from [constrained vCore VMs](../../../virtual-machines/constrained-vcpu.md), but data warehouse workloads are commonly characterized by fewer users and processes addressing larger amounts of data through query plans that run in parallel.

-> - The compute cost, which includes operating system licensing, will remain the same as the parent virtual machine.

-description: Learn about the key capabilities of Azure VMware Solution software-defined data centers and vSphere clusters.

-> Azure VMware Solution tenant admins must not edit or delete the above defined VMware vCenter alarms, as these are managed by the Azure VMware Solution control plane on vCenter. These alarms are used by Azure VMware Solution monitoring to trigger the Azure VMware Solution host remediation process.

-Private cloud vCenter and NSX-T configurations are on an hourly backup schedule. Backups are kept for three days. If you need to restore from a backup, open a [support request](https://rc.portal.azure.com/#create/Microsoft.Support) in the Azure portal to request restoration.

-Azure VMware Solution continuously monitors the health of both the underlay and the VMware components. When Azure VMware Solution detects a failure, it takes action to repair the failed components.

-[VCSA versions]: https://kb.vmware.com/s/article/2143838

-Local storage in each cluster host is used as part of a vSAN datastore. All diskgroups use an NVMe cache tier of 1.6 TB with the raw, per host, SSD-based capacity of 15.4 TB. The size of the raw capacity tier of a cluster is the per host capacity times the number of hosts. For example, a four host cluster provides 61.6-TB raw capacity in the vSAN capacity tier.

-Local storage in cluster hosts is used in cluster-wide vSAN datastore. All datastores are created as part of private cloud deployment and are available for use immediately. The **cloudadmin** user and all users assigned to the CloudAdmin role can manage datastores with these vSAN privileges:

-That default storage policy is set to RAID-1 (Mirroring), FTT-1, and thick provisioning. Unless you adjust the storage policy or apply a new policy, the cluster grows with this configuration. To set the storage policy, see [Configure storage policy](configure-storage-policy.md).

-In a three-host cluster, FTT-1 accommodates a single host's failure. Microsoft governs failures regularly and replaces the hardware when events are detected from an architecture perspective.

-|**Thin** | Consumes the space that it needs initially and grows to the data space demand used in the datastore. Outside the default (thick provision), you can create VMs with FTT-1 thin provisioning. For dedupe setup, use thin provisioning for your VM template. |

->:::image type="content" source="media/concepts/vsphere-vm-storage-policies-2.png" alt-text="Screenshot showing the RAID-5 FTT-1 and RAID-6 Ftt-2 options highlighed.":::

-vSAN datastores use data-at-rest encryption by default using keys stored in Azure Key Vault. The encryption solution is KMS-based and supports vCenter operations for key management. When a host is removed from a cluster, data on SSDs is invalidated immediately.

-| Azure Virtual Machines | Only monthly and yearly recovery points. Daily and weekly recovery points aren't supported. <br><br> Age >= 3 months in Vault-atandard tier <br><br> Retention left >= 6 months. <br><br> No active daily and weekly dependencies. |

->As shown in the screenshot above, the FQDNs depict `xxxxxxxx.<geo>.backup.windowsazure.com` and not `xxxxxxxx.privatelink.<geo>.backup. windowsazure.com`. In such cases, ensure you include (and if required, add) the `.privatelink.` according to the stated format.

-2. Execute **nslookup** on the backup URL (`xxxxxxxx.privatelink.<geo>.backup. windowsazure.com`) from your VM, to ensure connectivity. This should return the private IP assigned in your virtual network.

-You can review and test the detailed API, which is available as a [Swagger document](https://westus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0).

-For full details about the preceding calls, see our [Swagger document](https://westus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0). For the full sample shown here, go to [GitHub](https://aka.ms/csspeech/samples) in the `samples/batch` subdirectory.

-You can use [Speech-to-text REST API v3.0](rest-speech-to-text.md#speech-to-text-rest-api-v30) to automate any operations related to your custom models. In particular, you can use the REST API to upload a dataset.

-A dataset that you create by using the Speech-to-text REST API v3.0 will *not* be connected to any of the Speech Studio projects, unless you specify a special parameter in the request body (see the code block later in this section). Connection with a Speech Studio project is *not* required for any model customization operations, if you perform them by using the REST API.

-| 401 | Unauthorized | The request isn't authorized. Check to make sure your subscription key or [token](rest-speech-to-text.md#authentication) is valid and in the correct region. |

-[Online transcription](get-started-speech-to-text.md) and [Translation](speech-translation.md) use either the [Speech SDK](speech-sdk.md) or the [REST API for short audio](rest-speech-to-text.md#speech-to-text-rest-api-for-short-audio).

-In case of [Speech-to-text REST API for short audio](rest-speech-to-text.md#speech-to-text-rest-api-for-short-audio) you need to "inject" the session information in the requests. See details [below](#provide-session-id-using-rest-api-for-short-audio).

-Unlike Speech SDK, [Speech-to-text REST API for short audio](rest-speech-to-text.md#speech-to-text-rest-api-for-short-audio) does not automatically generate a Session ID. You need to generate it yourself and provide it within the REST request.

-[Batch transcription](batch-transcription.md) uses [Speech-to-text REST API v3.0](rest-speech-to-text.md#speech-to-text-rest-api-v30).

-> [!NOTE]

-Both, the Microsoft [Speech SDK](speech-sdk.md) and the [Speech-to-text REST API v3.0](rest-speech-to-text.md#speech-to-text-rest-api-v30), can be used to obtain transcripts. The decision does impact overall costs as it is explained in the guide.

-All entities from v2 can also be found in the v3 API under the same identity. Where the schema of a result has changed, (for example, transcriptions), the result of a GET in the v3 version of the API uses the v3 schema. The result of a GET in the v2 version of the API uses the same v2 schema. Newly created entities on v3 are **not** available in responses from v2 APIs.

-Endpoint host names have changed from `{region}.cris.ai` to `{region}.api.cognitive.microsoft.com`. Paths to the new endpoints no longer contain `api/` because it's part of the hostname. The [Swagger document](https://westus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0) lists valid regions and paths.

-Examine all features of these commonly used REST APIs provided by Speech

-* [Speech-to-text REST API](rest-speech-to-text.md)

-* [Swagger document](https://westus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0) for v3 of the REST API

-* For sample code to perform batch transcriptions, view the the [GitHub sample repository](https://aka.ms/csspeech/samples) in the `samples/batch` subdirectory.

-| | [Pronunciation assessment](./how-to-pronunciation-assessment.md) | Pronunciation assessment evaluates speech pronunciation and gives speakers feedback on the accuracy and fluency of spoken audio. With pronunciation assessment, language learners can practice, get instant feedback, and improve their pronunciation so that they can speak and present with confidence. | [Yes](./how-to-pronunciation-assessment.md) | [Yes](./rest-speech-to-text.md#pronunciation-assessment-parameters) |

-description: Learn how to use REST APIs to convert speech to text.

-# Speech-to-text REST APIs

-Speech-to-text has two REST APIs. Each API serves a special purpose and uses its own set of endpoints. In this article, you learn how to use those APIs, including authorization options, query options, how to structure a request, and how to interpret a response.

-## Speech-to-text REST API v3.0

-Speech-to-text REST API v3.0 is used for [Batch transcription](batch-transcription.md) and [Custom Speech](custom-speech-overview.md). v3.0 is a [successor of v2.0](./migrate-v2-to-v3.md). If you need to communicate with the online transcription via REST, use the [speech-to-text REST API for short audio](#speech-to-text-rest-api-for-short-audio).

-You can find the full speech-to-text REST API v3.0 reference on the [Microsoft developer portal](https://centralus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0).

-## Speech-to-text REST API for short audio

-As an alternative to the [Speech SDK](speech-sdk.md), the Speech service allows you to convert speech to text by using the [REST API for short audio](#speech-to-text-rest-api-for-short-audio).

-This API is very limited. Use it only in cases where you can't use the Speech SDK.

-Before you use the speech-to-text REST API for short audio, consider the following limitations:

-* Requests that use the REST API for short audio and transmit audio directly can contain no more than 60 seconds of audio.

-* The REST API for short audio returns only final results. It doesn't provide partial results.

-If sending longer audio is a requirement for your application, consider using the Speech SDK or [speech-to-text REST API v3.0](#speech-to-text-rest-api-v30).

-> [!TIP]

-> For Azure Government and Azure China endpoints, see [this article about sovereign clouds](sovereign-clouds.md).

-### Regions and endpoints

-The endpoint for the REST API for short audio has this format:

-```

-https://<REGION_IDENTIFIER>.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1

-```

-Replace `<REGION_IDENTIFIER>` with the identifier that matches the region of your subscription from this table:

-> [!NOTE]

-> You must append the language parameter to the URL to avoid receiving a 4xx HTTP error. For example, the language set to US English via the West US endpoint is: `https://westus.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=en-US`.

-### Query parameters

-These parameters might be included in the query string of the REST request:

-| Parameter | Description | Required or optional |

-|--|-||

-| `language` | Identifies the spoken language that's being recognized. See [Supported languages](language-support.md#speech-to-text). | Required |

-| `format` | Specifies the result format. Accepted values are `simple` and `detailed`. Simple results include `RecognitionStatus`, `DisplayText`, `Offset`, and `Duration`. Detailed responses include four different representations of display text. The default setting is `simple`. | Optional |

-| `profanity` | Specifies how to handle profanity in recognition results. Accepted values are: <br><br>`masked`, which replaces profanity with asterisks. <br>`removed`, which removes all profanity from the result. <br>`raw`, which includes profanity in the result. <br><br>The default setting is `masked`. | Optional |

-| `cid` | When you're using the [Custom Speech portal](./custom-speech-overview.md) to create custom models, you can take advantage of the **Endpoint ID** value from the **Deployment** page. Use the **Endpoint ID** value as the argument to the `cid` query string parameter. | Optional |

-### Request headers

-This table lists required and optional headers for speech-to-text requests:

-|Header| Description | Required or optional |

-||-||

-| `Ocp-Apim-Subscription-Key` | Your subscription key for the Speech service. | Either this header or `Authorization` is required. |

-| `Authorization` | An authorization token preceded by the word `Bearer`. For more information, see [Authentication](#authentication). | Either this header or `Ocp-Apim-Subscription-Key` is required. |

-| `Pronunciation-Assessment` | Specifies the parameters for showing pronunciation scores in recognition results. These scores assess the pronunciation quality of speech input, with indicators like accuracy, fluency, and completeness. <br><br>This parameter is a Base64-encoded JSON that contains multiple detailed parameters. To learn how to build this header, see [Pronunciation assessment parameters](#pronunciation-assessment-parameters). | Optional |

-| `Content-type` | Describes the format and codec of the provided audio data. Accepted values are `audio/wav; codecs=audio/pcm; samplerate=16000` and `audio/ogg; codecs=opus`. | Required |

-| `Transfer-Encoding` | Specifies that chunked audio data is being sent, rather than a single file. Use this header only if you're chunking audio data. | Optional |

-| `Expect` | If you're using chunked transfer, send `Expect: 100-continue`. The Speech service acknowledges the initial request and awaits additional data.| Required if you're sending chunked audio data. |

-| `Accept` | If provided, it must be `application/json`. The Speech service provides results in JSON. Some request frameworks provide an incompatible default value. It's good practice to always include `Accept`. | Optional, but recommended. |

-### Audio formats

-Audio is sent in the body of the HTTP `POST` request. It must be in one of the formats in this table:

-| Format | Codec | Bit rate | Sample rate |

-|--|-|-|--|

-| WAV | PCM | 256 kbps | 16 kHz, mono |

-| OGG | OPUS | 256 kpbs | 16 kHz, mono |

->[!NOTE]

->The preceding formats are supported through the REST API for short audio and WebSocket in the Speech service. The [Speech SDK](speech-sdk.md) supports the WAV format with PCM codec as well as [other formats](how-to-use-codec-compressed-audio-input-streams.md).

-### Pronunciation assessment parameters

-This table lists required and optional parameters for pronunciation assessment:

-| Parameter | Description | Required or optional |

-|--|-||

-| `ReferenceText` | The text that the pronunciation will be evaluated against. | Required |

-| `GradingSystem` | The point system for score calibration. The `FivePoint` system gives a 0-5 floating point score, and `HundredMark` gives a 0-100 floating point score. Default: `FivePoint`. | Optional |

-| `Granularity` | The evaluation granularity. Accepted values are:<br><br> `Phoneme`, which shows the score on the full-text, word, and phoneme levels.<br>`Word`, which shows the score on the full-text and word levels. <br>`FullText`, which shows the score on the full-text level only.<br><br> The default setting is `Phoneme`. | Optional |

-| `Dimension` | Defines the output criteria. Accepted values are:<br><br> `Basic`, which shows the accuracy score only. <br>`Comprehensive`, which shows scores on more dimensions (for example, fluency score and completeness score on the full-text level, and error type on the word level).<br><br> To see definitions of different score dimensions and word error types, see [Response parameters](#response-parameters). The default setting is `Basic`. | Optional |

-| `EnableMiscue` | Enables miscue calculation. With this parameter enabled, the pronounced words will be compared to the reference text. They'll be marked with omission or insertion based on the comparison. Accepted values are `False` and `True`. The default setting is `False`. | Optional |

-| `ScenarioId` | A GUID that indicates a customized point system. | Optional |

-Here's example JSON that contains the pronunciation assessment parameters:

-```json

-{

- "ReferenceText": "Good morning.",

- "GradingSystem": "HundredMark",

- "Granularity": "FullText",

- "Dimension": "Comprehensive"

-}

-```

-The following sample code shows how to build the pronunciation assessment parameters into the `Pronunciation-Assessment` header:

-```csharp

-var pronAssessmentParamsJson = $"{{\"ReferenceText\":\"Good morning.\",\"GradingSystem\":\"HundredMark\",\"Granularity\":\"FullText\",\"Dimension\":\"Comprehensive\"}}";

-var pronAssessmentParamsBytes = Encoding.UTF8.GetBytes(pronAssessmentParamsJson);

-var pronAssessmentHeader = Convert.ToBase64String(pronAssessmentParamsBytes);

-```

-We strongly recommend streaming (chunked) uploading while you're posting the audio data, which can significantly reduce the latency. To learn how to enable streaming, see the [sample code in various programming languages](https://github.com/Azure-Samples/Cognitive-Speech-TTS/tree/master/PronunciationAssessment).

->[!NOTE]

-> For more For more information, see [pronunciation assessment](how-to-pronunciation-assessment.md).

-### Sample request

-The following sample includes the host name and required headers. It's important to note that the service also expects audio data, which is not included in this sample. As mentioned earlier, chunking is recommended but not required.

-```HTTP

-POST speech/recognition/conversation/cognitiveservices/v1?language=en-US&format=detailed HTTP/1.1

-Accept: application/json;text/xml

-Content-Type: audio/wav; codecs=audio/pcm; samplerate=16000

-Ocp-Apim-Subscription-Key: YOUR_SUBSCRIPTION_KEY

-Host: westus.stt.speech.microsoft.com

-Transfer-Encoding: chunked

-Expect: 100-continue

-```

-To enable pronunciation assessment, you can add the following header. To learn how to build this header, see [Pronunciation assessment parameters](#pronunciation-assessment-parameters).

-```HTTP

-Pronunciation-Assessment: eyJSZWZlcm...

-```

-### HTTP status codes

-The HTTP status code for each response indicates success or common errors.

-| HTTP status code | Description | Possible reasons |

-||-|--|

-| 100 | Continue | The initial request has been accepted. Proceed with sending the rest of the data. (This code is used with chunked transfer.) |

-| 200 | OK | The request was successful. The response body is a JSON object. |

-| 400 | Bad request | The language code wasn't provided, the language isn't supported, or the audio file is invalid (for example). |

-| 401 | Unauthorized | A subscription key or an authorization token is invalid in the specified region, or an endpoint is invalid. |

-| 403 | Forbidden | A subscription key or authorization token is missing. |

-### Chunked transfer

-Chunked transfer (`Transfer-Encoding: chunked`) can help reduce recognition latency. It allows the Speech service to begin processing the audio file while it's transmitted. The REST API for short audio does not provide partial or interim results.

-The following code sample shows how to send audio in chunks. Only the first chunk should contain the audio file's header. `request` is an `HttpWebRequest` object that's connected to the appropriate REST endpoint. `audioFile` is the path to an audio file on disk.

-```csharp

-var request = (HttpWebRequest)HttpWebRequest.Create(requestUri);

-request.SendChunked = true;

-request.Accept = @"application/json;text/xml";

-request.Method = "POST";

-request.ProtocolVersion = HttpVersion.Version11;

-request.Host = host;

-request.ContentType = @"audio/wav; codecs=audio/pcm; samplerate=16000";

-request.Headers["Ocp-Apim-Subscription-Key"] = "YOUR_SUBSCRIPTION_KEY";

-request.AllowWriteStreamBuffering = false;

-using (var fs = new FileStream(audioFile, FileMode.Open, FileAccess.Read))

-{

- // Open a request stream and write 1,024-byte chunks in the stream one at a time.

- byte[] buffer = null;

- int bytesRead = 0;

- using (var requestStream = request.GetRequestStream())

- {

- // Read 1,024 raw bytes from the input audio file.

- buffer = new Byte[checked((uint)Math.Min(1024, (int)fs.Length))];

- while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) != 0)

- {

- requestStream.Write(buffer, 0, bytesRead);

- }

- requestStream.Flush();

- }

-}

-```

-### Response parameters

-Results are provided as JSON. The `simple` format includes the following top-level fields:

-| Parameter | Description |

-|--|--|

-|`RecognitionStatus`|Status, such as `Success` for successful recognition. See the next table.|

-|`DisplayText`|The recognized text after capitalization, punctuation, inverse text normalization, and profanity masking. Present only on success. Inverse text normalization is conversion of spoken text to shorter forms, such as 200 for "two hundred" or "Dr. Smith" for "doctor smith."|

-|`Offset`|The time (in 100-nanosecond units) at which the recognized speech begins in the audio stream.|

-|`Duration`|The duration (in 100-nanosecond units) of the recognized speech in the audio stream.|

-The `RecognitionStatus` field might contain these values:

-| Status | Description |

-|--|-|

-| `Success` | The recognition was successful, and the `DisplayText` field is present. |

-| `NoMatch` | Speech was detected in the audio stream, but no words from the target language were matched. This status usually means that the recognition language is different from the language that the user is speaking. |

-| `InitialSilenceTimeout` | The start of the audio stream contained only silence, and the service timed out while waiting for speech. |

-| `BabbleTimeout` | The start of the audio stream contained only noise, and the service timed out while waiting for speech. |

-| `Error` | The recognition service encountered an internal error and could not continue. Try again if possible. |

-> [!NOTE]

-> If the audio consists only of profanity, and the `profanity` query parameter is set to `remove`, the service does not return a speech result.

-The `detailed` format includes additional forms of recognized results.

-When you're using the `detailed` format, `DisplayText` is provided as `Display` for each result in the `NBest` list.

-The object in the `NBest` list can include:

-| Parameter | Description |

-|--|-|

-| `Confidence` | The confidence score of the entry, from 0.0 (no confidence) to 1.0 (full confidence). |

-| `Lexical` | The lexical form of the recognized text: the actual words recognized. |

-| `ITN` | The inverse-text-normalized (ITN) or canonical form of the recognized text, with phone numbers, numbers, abbreviations ("doctor smith" to "dr smith"), and other transformations applied. |

-| `MaskedITN` | The ITN form with profanity masking applied, if requested. |

-| `Display` | The display form of the recognized text, with punctuation and capitalization added. This parameter is the same as what `DisplayText` provides when the format is set to `simple`. |

-| `AccuracyScore` | Pronunciation accuracy of the speech. Accuracy indicates how closely the phonemes match a native speaker's pronunciation. The accuracy score at the word and full-text levels is aggregated from the accuracy score at the phoneme level. |

-| `FluencyScore` | Fluency of the provided speech. Fluency indicates how closely the speech matches a native speaker's use of silent breaks between words. |

-| `CompletenessScore` | Completeness of the speech, determined by calculating the ratio of pronounced words to reference text input. |

-| `PronScore` | Overall score that indicates the pronunciation quality of the provided speech. This score is aggregated from `AccuracyScore`, `FluencyScore`, and `CompletenessScore` with weight. |

-| `ErrorType` | Value that indicates whether a word is omitted, inserted, or badly pronounced, compared to `ReferenceText`. Possible values are `None` (meaning no error on this word), `Omission`, `Insertion`, and `Mispronunciation`. |

-### Sample responses

-Here's a typical response for `simple` recognition:

-```json

-{

- "RecognitionStatus": "Success",

- "DisplayText": "Remind me to buy 5 pencils.",

- "Offset": "1236645672289",

- "Duration": "1236645672289"

-}

-```

-Here's a typical response for `detailed` recognition:

-```json

-{

- "RecognitionStatus": "Success",

- "Offset": "1236645672289",

- "Duration": "1236645672289",

- "NBest": [

- {

- "Confidence": 0.9052885,

- "Display": "What's the weather like?",

- "ITN": "what's the weather like",

- "Lexical": "what's the weather like",

- "MaskedITN": "what's the weather like"

- },

- {

- "Confidence": 0.92459863,

- "Display": "what is the weather like",

- "ITN": "what is the weather like",

- "Lexical": "what is the weather like",

- "MaskedITN": "what is the weather like"

- }

- ]

-}

-```

-Here's a typical response for recognition with pronunciation assessment:

-```json

-{

- "RecognitionStatus": "Success",

- "Offset": "400000",

- "Duration": "11000000",

- "NBest": [

- {

- "Confidence" : "0.87",

- "Lexical" : "good morning",

- "ITN" : "good morning",

- "MaskedITN" : "good morning",

- "Display" : "Good morning.",

- "PronScore" : 84.4,

- "AccuracyScore" : 100.0,

- "FluencyScore" : 74.0,

- "CompletenessScore" : 100.0,

- "Words": [

- {

- "Word" : "Good",

- "AccuracyScore" : 100.0,

- "ErrorType" : "None",

- "Offset" : 500000,

- "Duration" : 2700000

- },

- {

- "Word" : "morning",

- "AccuracyScore" : 100.0,

- "ErrorType" : "None",

- "Offset" : 5300000,

- "Duration" : 900000

- }

- ]

- }

- ]

-}

-```

-| [Speech-to-text REST API v3.0](rest-speech-to-text.md#speech-to-text-rest-api-v30) | `https://<REGION_IDENTIFIER>.api.cognitive.microsoft.us/<URL_PATH>` |

-| [Speech-to-text REST API for short audio](rest-speech-to-text.md#speech-to-text-rest-api-for-short-audio) | `https://<REGION_IDENTIFIER>.stt.speech.azure.us/<URL_PATH>` |

-| [Speech-to-text REST API v3.0](rest-speech-to-text.md#speech-to-text-rest-api-v30) | `https://<REGION_IDENTIFIER>.api.cognitive.azure.cn/<URL_PATH>` |

-| [Speech-to-text REST API for short audio](rest-speech-to-text.md#speech-to-text-rest-api-for-short-audio) | `https://<REGION_IDENTIFIER>.stt.speech.azure.cn/<URL_PATH>` |

-Usually, Speech resources use [Cognitive Services regional endpoints](../cognitive-services-custom-subdomains.md#is-there-a-list-of-regional-endpoints) for communicating with the [Speech-to-text REST API v3.0](rest-speech-to-text.md#speech-to-text-rest-api-v30). These resources have the following naming format: <p/>`{region}.api.cognitive.microsoft.com`.

-The [Speech-to-text REST API for short audio](rest-speech-to-text.md#speech-to-text-rest-api-for-short-audio) and the [Text-to-speech REST API](rest-text-to-speech.md) use two types of endpoints:

-> When you're using the Speech-to-text REST API for short audio and Text-to-speech REST API in private endpoint scenarios, use a subscription key passed through the `Ocp-Apim-Subscription-Key` header. (See details for [Speech-to-text REST API for short audio](rest-speech-to-text.md#request-headers) and [Text-to-speech REST API](rest-text-to-speech.md#request-headers))

-In this case, usage of the Speech-to-text REST API for short audio and usage of the Text-to-speech REST API have no differences from the general case, with one exception. (See the following note.) You should use both APIs as described in the [speech-to-text REST API for short audio](rest-speech-to-text.md#speech-to-text-rest-api-for-short-audio) and [Text-to-speech REST API](rest-text-to-speech.md) documentation.

-> When you're using the Speech-to-text REST API for short audio and Text-to-speech REST API in custom domain scenarios, use a subscription key passed through the `Ocp-Apim-Subscription-Key` header. (See details for [Speech-to-text REST API for short audio](rest-speech-to-text.md#request-headers) and [Text-to-speech REST API](rest-text-to-speech.md#request-headers))

-You can use online transcription with the [Speech SDK](speech-sdk.md) or the [speech-to-text REST API for short audio](rest-speech-to-text.md#speech-to-text-rest-api-for-short-audio).

-| [Speech-to-text REST API V2.0 and v3.0](rest-speech-to-text.md#speech-to-text-rest-api-v30) limit | Not available for F0 | 300 requests per minute |

-To get started with speech-to-text, see the [quickstart](get-started-speech-to-text.md). Speech-to-text is available via the [Speech SDK](speech-sdk.md), the [REST API](rest-speech-to-text.md#pronunciation-assessment-parameters), and the [Speech CLI](spx-overview.md).

-description: The Swagger documentation can be used to auto-generate SDKs for a number of programming languages. All operations in our service are supported by Swagger

-# Swagger documentation

-> However only [Speech-to-text REST API v3.0](rest-speech-to-text.md#speech-to-text-rest-api-v30) and v2.0 are documented in the Swagger specification. See the documents referenced in the previous paragraph for the information on all other Speech Services REST APIs.

-* [Swagger: Speech-to-text REST API v3.0](https://westus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0)

-* [Speech-to-text REST API](rest-speech-to-text.md)

-Your [dataset is split](../how-to/train-model.md#data-split) into two parts: a set for training, and a set for testing. The training set while building the model and the testing set is used as a blind set to evaluate model performance after training is completed.

-You can train multiple models on the same dataset within the same project. After you have trained your model successfully, you can [view its evaluation](how-to/view-model-evaluation.md). You can [deploy and test](quickstart.md#deploy-your-model) your model within [Language studio](https://aka.ms/languageStudio). You can add or remove tags from your data and train a **new** model and test it as well. View [service limits](service-limits.md)to learn about maximum number of trained models with the same project. When you train a new model, your dataset is [split](how-to/train-model.md#data-split) randomly into training and testing sets. Because of this, there is no guarantee that the model evaluation is performed on the same test set, so results are not comparable. It is recommended that you develop your own test set and use it to evaluate both models so you can measure improvement.

-* When you train a new model your dataset is [split](how-to/train-model.md#data-split) randomly into training and testing sets, so there is no guarantee that the reflected model evaluation is on the same test set, so results are not comparable.

-Before creating a custom text classification model, you need to have tagged data first. If your data is not tagged already, you can tag it in the language studio. Tagged data informs the model how to interpret text, and is used for training and evaluation.

-<!--Tagging your data will let you [train your model](train-model.md), [evaluate it](train-model.md), and use it to [classify text](call-api.md).-->

-* Although we recommended having around 50 tagged files per class, there is no fixed number that can guarantee your model will perform the best, because model performance also depends on possible ambiguity in your [schema](design-schema.md), and the quality of your tagged data.

-3. You can find a list of all .txt files available in your projects to the left. You can select the file you want to start tagging or you can use the Back and Next button from the bottom of the page to navigate.

-6. Start tagging your files.

- * **Single label classification**: your file can only be tagged with one class, you can do so by checking one of the radio buttons next to the class you want to tag this file with.

- :::image type="content" source="../media/tag-single.png" alt-text="A screenshot showing the single label classification menu" lightbox="../media/tag-single.png":::

- * **Multi label classification**: your file can be tagged with multiple classes, you can do so by checking all applicable check boxes next to the classes you want to tag this file with.

- :::image type="content" source="../media/tag-multi.png" alt-text="A screenshot showing the multi label classification menu" lightbox="../media/tag-multi.png":::

-While tagging, your changes will be synced periodically, if they have not been saved yet you will find a warning at the top of your page. If you want to save manually, click on Save tags button at the top of the page.

-## Data split

-Before you start the training process, files in your dataset are divided into three groups at random:

-* The **training set** contains 80% of the files in your dataset. It is the main set that is used to train the model.

-* The **test set** contains 20% of the files available in your dataset. This set is used to provide an unbiased [evaluation](../how-to/view-model-evaluation.md) of the model. This set is not introduced to the model during training. The details of correct and incorrect predictions for this set are not shown so that you don't readjust your training data and alter the results.

-6. You can check the status of the training job in the same page. Only successfully completed tasks will generate models.

-* Your [training dataset](how-to/train-model.md#data-split) should include at least 10 files and no more than 1,000,000 files.

-Users of your Communication Services solution will be connecting to your services from their client devices. Communication between these devices and your servers is handled with **signaling**. For example: call initiation and real-time chat are supported by signaling between devices and your service. Most signaling traffic uses HTTPS REST, though in some scenarios, SIP can be used as a signaling traffic protocol. While this type of traffic is less sensitive to latency, low-latency signaling will the users of your solution a pleasant end-user experience.

-| UI Library| [npm](https://www.npmjs.com/package/@azure/communication-react) | - | - | - | - | - | [GitHub](https://github.com/Azure/communication-ui-library), [Storybook](https://azure.github.io/communication-ui-library/?path=/story/overview--page) |

-Check out the [Call Recoding Quickstart](../../quickstarts/voice-video-calling/call-recording-sample.md) to learn more.

-> Confidential virtual machines (confidential VMs) in Azure confidential computing is currently in PREVIEW.

-> See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

-# Use bulk executor Java library to perform bulk operations on Azure Cosmos DB data

-This tutorial provides instructions on using the Azure Cosmos DB's bulk executor Java library to import, and update Azure Cosmos DB documents. To learn about bulk executor library and how it helps you use massive throughput and storage, see [bulk executor Library overview](../bulk-executor-overview.md) article. In this tutorial, you build a Java application that generates random documents and they are bulk imported into an Azure Cosmos container. After importing, you will bulk update some properties of a document.

-> [!IMPORTANT]

-> The [Azure Cosmos DB Java V4 SDK](sql-api-sdk-java-v4.md) comes with the bulk executor library built-in to the SDK. If you are using an older version of Java SDK, it's recommended to [migrate to the latest version](migrate-java-v4-sdk.md). Azure Cosmos DB Java V4 SDK is the current recommended solution for Java bulk support. Currently, the bulk executor library is supported only by Azure Cosmos DB SQL API and Gremlin API accounts. To learn about using bulk executor .NET library with Gremlin API, see [perform bulk operations in Azure Cosmos DB Gremlin API](../graph/bulk-executor-graph-dotnet.md).

->

-* [Java Development Kit (JDK) 1.7+](/java/azure/jdk/)

-Now let's switch to working with code by downloading a sample Java application from GitHub. This application performs bulk operations on Azure Cosmos DB data. To clone the application, open a command prompt, navigate to the directory where you want to copy the application and run the following command:

- git clone https://github.com/Azure/azure-cosmosdb-bulkexecutor-java-getting-started.git

-The cloned repository contains two samples "bulkimport" and "bulkupdate" relative to the "\azure-cosmosdb-bulkexecutor-java-getting-started\samples\bulkexecutor-sample\src\main\java\com\microsoft\azure\cosmosdb\bulkexecutor" folder. The "bulkimport" application generates random documents and imports them to Azure Cosmos DB. The "bulkupdate" application updates some documents in Azure Cosmos DB. In the next sections, we will review the code in each of these sample apps.

-## Bulk import data to Azure Cosmos DB

-1. The Azure Cosmos DB's connection strings are read as arguments and assigned to variables defined in CmdLineConfiguration.java file.

-2. Next the DocumentClient object is initialized by using the following statements:

- ```java

- ConnectionPolicy connectionPolicy = new ConnectionPolicy();

- connectionPolicy.setMaxPoolSize(1000);

- DocumentClient client = new DocumentClient(

- HOST,

- MASTER_KEY,

- connectionPolicy,

- ConsistencyLevel.Session)

- ```

-3. The DocumentBulkExecutor object is initialized with a high retry value for wait time and throttled requests. And then they are set to 0 to pass congestion control to DocumentBulkExecutor for its lifetime.

- // Set client's retry options high for initialization

- client.getConnectionPolicy().getRetryOptions().setMaxRetryWaitTimeInSeconds(30);

- client.getConnectionPolicy().getRetryOptions().setMaxRetryAttemptsOnThrottledRequests(9);

- // Builder pattern

- Builder bulkExecutorBuilder = DocumentBulkExecutor.builder().from(

- client,

- DATABASE_NAME,

- COLLECTION_NAME,

- collection.getPartitionKey(),

- offerThroughput) // throughput you want to allocate for bulk import out of the container's total throughput

- // Instantiate DocumentBulkExecutor

- DocumentBulkExecutor bulkExecutor = bulkExecutorBuilder.build()

- // Set retries to 0 to pass complete control to bulk executor

- client.getConnectionPolicy().getRetryOptions().setMaxRetryWaitTimeInSeconds(0);

- client.getConnectionPolicy().getRetryOptions().setMaxRetryAttemptsOnThrottledRequests(0);

-4. Call the importAll API that generates random documents to bulk import into an Azure Cosmos container. You can configure the command-line configurations within the CmdLineConfiguration.java file.

- ```java

- BulkImportResponse bulkImportResponse = bulkExecutor.importAll(documents, false, true, null);

- ```

- The bulk import API accepts a collection of JSON-serialized documents and it has the following syntax, for more information, see the [API documentation](/java/api/com.microsoft.azure.documentdb.bulkexecutor):

- public BulkImportResponse importAll(

- Collection<String> documents,

- boolean isUpsert,

- boolean disableAutomaticIdGeneration,

- Integer maxConcurrencyPerPartitionRange) throws DocumentClientException;

- The importAll method accepts the following parameters:

-5. After you have the bulk import application ready, build the command-line tool from source by using the 'mvn clean package' command. This command generates a jar file in the target folder:

- ```

-* It is recommended to instantiate a single DocumentBulkExecutor object for the entire application within a single virtual machine that corresponds to a specific Azure Cosmos container.

-1. **The delegate:** The delegate is the code that defines what you, the developer, want to do with each batch of changes that the change feed processor reads.

-To further understand how these four elements of change feed processor work together, let's look at an example in the following diagram. The monitored container stores documents and uses 'City' as the partition key. We see that the partition key values are distributed in ranges that contain items.

-There are two compute instances and the change feed processor is assigning different ranges of partition key values to each instance to maximize compute distribution, each instance has a unique and different name.

-Each range is being read in parallel and its progress is maintained separately from other ranges in the lease container.

-A single change feed processor deployment unit consists of one or more compute instances with the same `processorName` and lease container configuration but different instance name each. You can have many deployment units where each one has a different business flow for the changes and each deployment unit consisting of one or more instances.

-Change feed read operations on the monitored container will consume RUs.

-Operations on the lease container consume RUs. The higher the number of instances using the same lease container, the higher the potential RU consumption will be. Remember to monitor your RU consumption on the leases container if you decide to scale and increment the number of instances.

-<tr><td>Flowlets General Availability (GA)</td><td>Flowlets is now generally available to create reusable portions of data flow logic that you can share in other pipelines as inline transformations. Flowlets enable ETL jobs to be composed of custom or common logic components.<br><a href="https://techcommunity.microsoft.com/t5/azure-data-factory-blog/flowlets-and-change-feed-now-ga-in-azure-data-factory/ba-p/3267450">Learn more</a></td></tr>

-<tr><td>SFTP connector for Mapping Data Flow</td><td>The SFTP connector is now available for Mapping Data Flows.<br><a href="connector-sftp.md?tabs=data-factory#mapping-data-flow-properties">Learn more</a></td></tr>

-1. Enter the unique identifier that you received on the **Password recovery** screen and select **Recover**. The `password_recovery.zip` file is downloaded.

-### Customer topics

-* To quickly get started using Event Grid, see [Create and route custom events with Azure Event Grid](custom-event-quickstart.md).

-1. On the **Event Grid Partner Registrations** page, select **+ Add** on the toolbar.

-# Tutorial: Automate resizing uploaded images using Event Grid

-This tutorial is part two of a series of Storage tutorials. It extends the [previous Storage tutorial][previous-tutorial] to add serverless automatic thumbnail generation using Azure Event Grid and Azure Functions. Event Grid enables [Azure Functions](../azure-functions/functions-overview.md) to respond to [Azure Blob storage](../storage/blobs/storage-blobs-introduction.md) events and generate thumbnails of uploaded images. An event subscription is created against the Blob storage create event. When a blob is added to a specific Blob storage container, a function endpoint is called. Data passed to the function binding from Event Grid is used to access the blob and generate the thumbnail image.

-The image resize function is triggered by HTTP requests sent to it from the Event Grid service. You tell Event Grid that you want to get these notifications at your function's URL by creating an event subscription. For this tutorial you subscribe to blob-created events.

-1. In the [Azure portal](https://portal.azure.com), at the top of the page search for and select `Function App` and choose the function app that you just created. Select **Functions** and choose the **Thumbnail** function.

- 1. For **Subject begins with**, enter the following value : **/blobServices/default/containers/images/**.

-1. Select **Create** to add the event subscription. This creates an event subscription that triggers the `Thumbnail` function when a blob is added to the `images` container. The function resizes the images and adds them to the `thumbnails` container.

-[previous-tutorial]: ../storage/blobs/storage-upload-process-images.md

- :::image type="content" source="./media/subscribe-to-partner-events/select-endpoint.png" lightbox="./media/subscribe-to-partner-events/select-endpoint.png" alt-text="Image showing the configuration of an endpoint for an event subscription.":::

-> You can download this quickstart as a sample from the [GitHub](https://github.com/Azure-Samples/azure-sdk-for-go-samples/tree/master/eventhubs), replace `EventHubConnectionString` and `EventHubName` strings with your event hub values, and run it. Alternatively, you can follow the steps in this tutorial to create your own.

-Samples for creating Storage artifacts with the Go SDK are available in the [Go samples repo](https://github.com/Azure-Samples/azure-sdk-for-go-samples/tree/master/storage) and in the sample corresponding to this tutorial.

-> To enable ExpressRoute Global Reach between [different geopolitical regions](expressroute-locations-providers.md#locations), your circuits must be **Premium SKU**.

-You've learned how to add additional storage accounts to an existing HDInsight cluster. For more information on script actions, see [Customize Linux-based HDInsight clusters using script action](hdinsight-hadoop-customize-cluster-linux.md)

-* [Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters](../hdinsight-hadoop-use-data-lake-storage-gen2.md)

-It is also possible to perform delete operations for devices using REST APIs. For more information, see [Service - Delete Device](/azure/iot-hub/iot-c-sdk-ref/iothub-registrymanager-h/iothubregistrymanager-deletedevice).

-It is also possible to perform export operations for devices using REST APIs. For more information, see [Service - Get Device](/azure/iot-hub/iot-c-sdk-ref/iothub-registrymanager-h/iothubregistrymanager-getdevice).

-Full documentation for Azure IoT Hub Service APIs is located at [IoT Hub Service APIs](/rest/api/iothub/service/configuration).

-* [400027 ConnectionForcefullyClosedOnNewConnection](iot-hub-troubleshoot-error-400027-connectionforcefullyclosedonnewconnection.md)

-If you're monitoring device connections with Event Hub, make sure you build in a way of filtering out the periodic disconnects due to SAS token renewal. For example, do not trigger actions based on disconnects as long as the disconnect event is followed by a connect event within a certain time span.

-description: Understand how to fix error 400027 ConnectionForcefullyClosedOnNewConnection

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 400027 ConnectionForcefullyClosedOnNewConnection errors.

-# 400027 ConnectionForcefullyClosedOnNewConnection

-This article describes the causes and solutions for **400027 ConnectionForcefullyClosedOnNewConnection** errors.

-## Symptoms

-Your device get disconnected with **Communication_Error** as **ConnectionStatusChangeReason** using .NET SDK and MQTT transport type.

-Your device-to-cloud twin operation (such as read or patch reported properties) or direct method invocation fails with the error code **400027**.

-## Cause

-Another client created a new connection to IoT Hub using the same identity, so IoT Hub closed the previous connection. IoT Hub doesn't allow more than one client to connect using the same identity.

-## Solution

-Ensure that each client connects to IoT Hub using its own identity.

-description: Understand how to fix error 401003 IoTHubUnauthorized

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 401003 IoTHubUnauthorized errors.

-# 401003 IoTHubUnauthorized

-This article describes the causes and solutions for **401003 IoTHubUnauthorized** errors.

-## Symptoms

-### Symptom 1

-In logs, you see a pattern of devices disconnecting with **401003 IoTHubUnauthorized**, followed by **404104 DeviceConnectionClosedRemotely**, and then successfully connecting shortly after.

-### Symptom 2

-Requests to IoT Hub fail with one of the following error messages:

-* Authorization header missing

-* IotHub '\*' does not contain the specified device '\*'

-* Authorization rule '\*' does not allow access for '\*'

-* Authentication failed for this device, renew token or certificate and reconnect

-* Thumbprint does not match configuration: Thumbprint: SHA1Hash=\*, SHA2Hash=\*; Configuration: PrimaryThumbprint=\*, SecondaryThumbprint=\*

-* Principal user@example.com is not authorized for GET on /exampleOperation due to no assigned permissions

-## Cause

-### Cause 1

-For MQTT, some SDKs rely on IoT Hub to issue the disconnect when the SAS token expires to know when to refresh it. So,

-1. The SAS token expires

-1. IoT Hub notices the expiration, and disconnects the device with **401003 IoTHubUnauthorized**

-1. The device completes the disconnection with **404104 DeviceConnectionClosedRemotely**

-1. The IoT SDK generates a new SAS token

-1. The device reconnects with IoT Hub successfully

-### Cause 2

-IoT Hub couldn't authenticate the auth header, rule, or key. This could be due to any of the reasons cited in the symptoms.

-## Solution

-### Solution 1

-No action needed if using IoT SDK for connection using the device connection string. IoT SDK regenerates the new token to reconnect on SAS token expiration.

-The default token lifespan is 60 minutes across SDKs; however, for some SDKs the token lifespan and the token renewal threshold is configurable. Additionally, the errors generated when a device disconnects and reconnects on token renewal differs for each SDK. To learn more, and for information about how to determine which SDK your device is using in logs, see [MQTT device disconnect behavior with Azure IoT SDKs](iot-hub-troubleshoot-connectivity.md#mqtt-device-disconnect-behavior-with-azure-iot-sdks).

-For device developers, if the volume of errors is a concern, switch to the C SDK, which renews the SAS token before expiration. For AMQP, the SAS token can refresh without disconnection.

-### Solution 2

-In general, the error message presented should explain how to fix the error. If for some reason you don't have access to the error message detail, make sure:

-## Next steps

-description: Understand how to fix error 403002 IoTHubQuotaExceeded

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 403002 IoTHubQuotaExceeded errors.

-# 403002 IoTHubQuotaExceeded

-This article describes the causes and solutions for **403002 IoTHubQuotaExceeded** errors.

-## Symptoms

-All requests to IoT Hub fail with the error **403002 IoTHubQuotaExceeded**. In Azure portal, the IoT hub device list doesn't load.

-## Cause

-The daily message quota for the IoT hub is exceeded.

-## Solution

-[Upgrade or increase the number of units on the IoT hub](iot-hub-upgrade.md) or wait for the next UTC day for the daily quota to refresh.

-## Next steps

-* To understand how operations are counted toward the quota, such as twin queries and direct methods, see [Understand IoT Hub pricing](iot-hub-devguide-pricing.md#charges-per-operation)

-* To set up monitoring for daily quota usage, set up an alert with the metric *Total number of messages used*. For step-by-step instructions, see [Set up metrics and alerts with IoT Hub](tutorial-use-metrics-and-diags.md#set-up-metrics)

-description: Understand how to fix error 403004 DeviceMaximumQueueDepthExceeded

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 403004 DeviceMaximumQueueDepthExceeded errors.

-# 403004 DeviceMaximumQueueDepthExceeded

-This article describes the causes and solutions for **403004 DeviceMaximumQueueDepthExceeded** errors.

-## Symptoms

-When trying to send a cloud-to-device message, the request fails with the error **403004** or **DeviceMaximumQueueDepthExceeded**.

-## Cause

-The underlying cause is that the number of messages enqueued for the device exceeds the [queue limit (50)](./iot-hub-devguide-quotas-throttling.md#other-limits).

-The most likely reason that you're running into this limit is because you're using HTTPS to receive the message, which leads to continuous polling using `ReceiveAsync`, resulting in IoT Hub throttling the request.

-## Solution

-The supported pattern for cloud-to-device messages with HTTPS is intermittently connected devices that check for messages infrequently (less than every 25 minutes). To reduce the likelihood of running into the queue limit, switch to AMQP or MQTT for cloud-to-device messages.

-Alternatively, enhance device side logic to complete, reject, or abandon queued messages quickly, shorten the time to live, or consider sending fewer messages. See [C2D message time to live](./iot-hub-devguide-messages-c2d.md#message-expiration-time-to-live).

-Lastly, consider using the [Purge Queue API](/azure/iot-hub/iot-c-sdk-ref/iothub-registrymanager-h/iothubregistrymanager-deletedevice) to periodically clean up pending messages before the limit is reached.

-description: Understand how to fix error 403006 DeviceMaximumActiveFileUploadLimitExceeded

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 403006 DeviceMaximumActiveFileUploadLimitExceeded errors.

-# 403006 DeviceMaximumActiveFileUploadLimitExceeded

-This article describes the causes and solutions for **403006 DeviceMaximumActiveFileUploadLimitExceeded** errors.

-## Symptoms

-Your file upload request fails with the error code **403006** and a message "Number of active file upload requests cannot exceed 10".

-## Cause

-Each device client is limited to [10 concurrent file uploads](./iot-hub-devguide-quotas-throttling.md#other-limits).

-You can easily exceed the limit if your device doesn't notify IoT Hub when file uploads are completed. This problem is commonly caused by an unreliable device side network.

-## Solution

-Ensure the device can promptly [notify IoT Hub file upload completion](./iot-hub-devguide-file-upload.md#device-notify-iot-hub-of-a-completed-file-upload). Then, try [reducing the SAS token TTL for file upload configuration](iot-hub-configure-file-upload.md).

-## Next steps

-To learn more about file uploads, see [Upload files with IoT Hub](./iot-hub-devguide-file-upload.md) and [Configure IoT Hub file uploads using the Azure portal](./iot-hub-configure-file-upload.md).

-description: Understand how to fix error 404001 DeviceNotFound

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 404001 DeviceNotFound errors.

-# 404001 DeviceNotFound

-This article describes the causes and solutions for **404001 DeviceNotFound** errors.

-## Symptoms

-During a cloud-to-device (C2D) communication, such as C2D message, twin update, or direct method, the operation fails with error **404001 DeviceNotFound**.

-## Cause

-The operation failed because the device cannot be found by IoT Hub. The device is either not registered or disabled.

-## Solution

-Register the device ID that you used, then try again.

-description: Understand how to fix error 404103 DeviceNotOnline

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 404103 DeviceNotOnline errors.

-# 404103 DeviceNotOnline

-This article describes the causes and solutions for **404103 DeviceNotOnline** errors.

-## Symptoms

-A direct method to a device fails with the error **404103 DeviceNotOnline** even if the device is online.

-## Cause

-If you know that the device is online and still get the error, it's likely because the direct method callback isn't registered on the device.

-## Solution

-To configure your device properly for direct method callbacks, see [Handle a direct method on a device](iot-hub-devguide-direct-methods.md#handle-a-direct-method-on-a-device).

-description: Understand how to fix error 404104 DeviceConnectionClosedRemotely

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 404104 DeviceConnectionClosedRemotely errors.

-# 404104 DeviceConnectionClosedRemotely

-This article describes the causes and solutions for **404104 DeviceConnectionClosedRemotely** errors.

-## Symptoms

-### Symptom 1

-Devices disconnect at a regular interval (every 65 minutes, for example) and you see **404104 DeviceConnectionClosedRemotely** in IoT Hub resource logs. Sometimes, you also see **401003 IoTHubUnauthorized** and a successful device connection event less than a minute later.

-### Symptom 2

-Devices disconnect randomly, and you see **404104 DeviceConnectionClosedRemotely** in IoT Hub resource logs.

-### Symptom 3

-Many devices disconnect at once, you see a dip in the [Connected devices (connectedDeviceCount) metric](monitor-iot-hub-reference.md), and there are more **404104 DeviceConnectionClosedRemotely** and [500xxx Internal errors](iot-hub-troubleshoot-error-500xxx-internal-errors.md) in Azure Monitor Logs than usual.

-## Causes

-### Cause 1

-The [SAS token used to connect to IoT Hub](iot-hub-dev-guide-sas.md#security-tokens) expired, which causes IoT Hub to disconnect the device. The connection is re-established when the token is refreshed by the device. For example, [the SAS token expires every hour by default for C SDK](https://github.com/Azure/azure-iot-sdk-c/blob/master/doc/connection_and_messaging_reliability.md#connection-authentication), which can lead to regular disconnects.

-To learn more, see [401003 IoTHubUnauthorized cause](iot-hub-troubleshoot-error-401003-iothubunauthorized.md#cause-1).

-### Cause 2

-Some possibilities include:

-### Cause 3

-IoT Hub might be experiencing a transient issue. See [IoT Hub internal server error cause](iot-hub-troubleshoot-error-500xxx-internal-errors.md#cause).

-## Solutions

-### Solution 1

-See [401003 IoTHubUnauthorized solution 1](iot-hub-troubleshoot-error-401003-iothubunauthorized.md#solution-1)

-### Solution 2

-### Solution 3

-See [solutions to IoT Hub internal server errors](iot-hub-troubleshoot-error-500xxx-internal-errors.md#solution).

-## Next steps

-We recommend using Azure IoT device SDKs to manage connections reliably. To learn more, see [Manage connectivity and reliable messaging by using Azure IoT Hub device SDKs](iot-hub-reliability-features-in-sdks.md)

-description: Understand how to fix error 409001 DeviceAlreadyExists

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 409001 DeviceAlreadyExists errors.

-# 409001 DeviceAlreadyExists

-This article describes the causes and solutions for **409001 DeviceAlreadyExists** errors.

-## Symptoms

-When trying to register a device in IoT Hub, the request fails with the error **409001 DeviceAlreadyExists**.

-## Cause

-There's already a device with the same device ID in the IoT hub.

-## Solution

-Use a different device ID and try again.

-description: Understand how to fix error 409002 LinkCreationConflict

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 409002 LinkCreationConflict errors.

-# 409002 LinkCreationConflict

-This article describes the causes and solutions for **409002 LinkCreationConflict** errors.

-## Symptoms

-You see the error **409002 LinkCreationConflict** in logs along with device disconnection or cloud-to-device message failure.

-<!-- When using AMQP? -->

-## Cause

-Generally, this error happens when IoT Hub detects a client has more than one connection. In fact, when a new connection request arrives for a device with an existing connection, IoT Hub closes the existing connection with this error.

-### Cause 1

-In the most common case, a separate issue (such as [404104 DeviceConnectionClosedRemotely](iot-hub-troubleshoot-error-404104-deviceconnectionclosedremotely.md)) causes the device to disconnect. The device tries to reestablish the connection immediately, but IoT Hub still considers the device connected. IoT Hub closes the previous connection and logs this error.

-### Cause 2

-Faulty device-side logic causes the device to establish the connection when one is already open.

-## Solution

-This error usually appears as a side effect of a different, transient issue, so look for other errors in the logs to troubleshoot further. Otherwise, make sure to issue a new connection request only if the connection drops.

-description: Understand how to fix error 412002 DeviceMessageLockLost

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 412002 DeviceMessageLockLost errors.

-# 412002 DeviceMessageLockLost

-This article describes the causes and solutions for **412002 DeviceMessageLockLost** errors.

-## Symptoms

-When trying to send a cloud-to-device message, the request fails with the error **412002 DeviceMessageLockLost**.

-## Cause

-When a device receives a cloud-to-device message from the queue (for example, using [`ReceiveAsync()`](/dotnet/api/microsoft.azure.devices.client.deviceclient.receiveasync)) the message is locked by IoT Hub for a lock timeout duration of one minute. If the device tries to complete the message after the lock timeout expires, IoT Hub throws this exception.

-## Solution

-If IoT Hub doesn't get the notification within the one-minute lock timeout duration, it sets the message back to *Enqueued* state. The device can attempt to receive the message again. To prevent the error from happening in the future, implement device side logic to complete the message within one minute of receiving the message. This one-minute time-out can't be changed.

-description: Understand how to fix error 429001 ThrottlingException

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 429001 ThrottlingException errors.

-# 429001 ThrottlingException

-This article describes the causes and solutions for **429001 ThrottlingException** errors.

-## Symptoms

-Your requests to IoT Hub fail with the error **429001 ThrottlingException**.

-## Cause

-IoT Hub [throttling limits](./iot-hub-devguide-quotas-throttling.md) have been exceeded for the requested operation.

-## Solution

-Check if you're hitting the throttling limit by comparing your *Telemetry message send attempts* metric against the limits specified above. You can also check the *Number of throttling errors* metric. For information about these metrics, see [Device telemetry metrics](monitor-iot-hub-reference.md#device-telemetry-metrics). For information about how use metrics to help you monitor your IoT hub, see [Monitor IoT Hub](monitor-iot-hub.md).

-IoT Hub returns 429 ThrottlingException only after the limit has been violated for too long a period. This is done so that your messages aren't dropped if your IoT hub gets burst traffic. In the meantime, IoT Hub processes the messages at the operation throttle rate, which might be slow if there's too much traffic in the backlog. To learn more, see [IoT Hub traffic shaping](./iot-hub-devguide-quotas-throttling.md#traffic-shaping).

-## Next steps

-Consider [scaling up your IoT Hub](./iot-hub-scaling.md) if you're running into quota or throttling limits.

-description: Understand how to fix 500xxx Internal errors

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 500xxx Internal errors.

-# 500xxx Internal errors

-This article describes the causes and solutions for **500xxx Internal errors**.

-## Symptoms

-Your request to IoT Hub fails with an error that begins with 500 and/or some sort of "server error". Some possibilities are:

-* **500001 ServerError**: IoT Hub ran into a server-side issue.

-* **500008 GenericTimeout**: IoT Hub couldn't complete the connection request before timing out.

-* **ServiceUnavailable (no error code)**: IoT Hub encountered an internal error.

-* **InternalServerError (no error code)**: IoT Hub encountered an internal error.

-## Cause

-There can be a number of causes for a 500xxx error response. In all cases, the issue is most likely transient. While the IoT Hub team works hard to maintain [the SLA](https://azure.microsoft.com/support/legal/sla/iot-hub/), small subsets of IoT Hub nodes can occasionally experience transient faults. When your device tries to connect to a node that's having issues, you receive this error.

-## Solution

-To mitigate 500xxx errors, issue a retry from the device. To [automatically manage retries](./iot-hub-reliability-features-in-sdks.md#connection-and-retry), make sure you use the latest version of the [Azure IoT SDKs](./iot-hub-devguide-sdks.md). For best practice on transient fault handling and retries, see [Transient fault handling](/azure/architecture/best-practices/transient-faults). If the problem persists, check [Resource Health](./iot-hub-azure-service-health-integration.md#check-health-of-an-iot-hub-with-azure-resource-health) and [Azure Status](https://status.azure.com/) to see if IoT Hub has a known problem. You can also use the [manual failover feature](./tutorial-manual-failover.md). If there are no known problems and the issue continues, [contact support](https://azure.microsoft.com/support/options/) for further investigation.

-description: Understand how to fix error 503003 PartitionNotFound

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 503003 PartitionNotFound errors.

-# 503003 PartitionNotFound

-This article describes the causes and solutions for **503003 PartitionNotFound** errors.

-## Symptoms

-Requests to IoT Hub fail with the error **503003 PartitionNotFound**.

-## Cause

-This error is internal to IoT Hub and is likely transient. See [IoT Hub internal server error cause](iot-hub-troubleshoot-error-500xxx-internal-errors.md#cause).

-## Solution

-See [solutions to IoT Hub internal server errors](iot-hub-troubleshoot-error-500xxx-internal-errors.md#solution).

-description: Understand how to fix error 504101 GatewayTimeout

-#Customer intent: As a developer or operator for Azure IoT Hub, I want to resolve 504101 GatewayTimeout errors.

-# 504101 GatewayTimeout

-This article describes the causes and solutions for **504101 GatewayTimeout** errors.

-## Symptoms

-When trying to invoke a direct method from IoT Hub to a device, the request fails with the error **504101 GatewayTimeout**.

-## Cause

-### Cause 1

-IoT Hub encountered an error and couldn't confirm if the direct method completed before timing out.

-### Cause 2

-When using an earlier version of the Azure IoT C# SDK (<1.19.0), the AMQP link between the device and IoT Hub can be dropped silently because of a bug.

-## Solution

-### Solution 1

-Issue a retry.

-### Solution 2

-Upgrade to the latest version of the Azure IOT C# SDK.

-The following example creates an HSM named **ContosoMHSM**, in the resource group **ContosoResourceGroup**, residing in the **West US 2** location, with **the current signed in user** as the only administrator.

-az keyvault create --hsm-name "ContosoMHSM2" --resource-group "ContosoResourceGroup" --location "westus2" --administrators $oid

-A resource group is a logical container into which Azure resources are deployed and managed. The following example creates a resource group named *ContosoResourceGroup* in the *westus2* location.

-az group create --name "ContosoResourceGroup" --location westus2

-The following example creates an HSM named **ContosoMHSM**, in the resource group **ContosoResourceGroup**, residing in the **West US 2** location, with **the current signed in user** as the only administrator, with **7 days retention period** for soft-delete. Read more about [Managed HSM soft-delete](soft-delete-overview.md)

-az keyvault create --hsm-name "ContosoMHSM" --resource-group "ContosoResourceGroup" --location "westus2" --administrators $oid --retention-days 7

-A resource group is a logical container into which Azure resources are deployed and managed. Use the Azure PowerShell [New-AzResourceGroup](/powershell/module/az.resources/new-azresourcegroup) cmdlet to create a resource group named *myResourceGroup* in the *westus2* location.

-New-AzResourceGroup -Name "myResourceGroup" -Location "westus2"

-New-AzKeyVaultManagedHsm -Name "<your-unique-managed-hsm-name>" -ResourceGroupName "myResourceGroup" -Location "westus2" -Administrator "<your-principal-ID>"

- - **Location**: Select a location. For example, **West US 2**.

-* If your logic app workflows run in single-tenant Azure Logic Apps, you need to find the fully qualified domain names (FQDNs) for your connections. For more information, review the corresponding sections in these topics:

- * [Firewall permissions for single tenant logic apps - Azure portal](create-single-tenant-workflows-azure-portal.md#firewall-setup)

- * [Firewall permissions for single tenant logic apps - Visual Studio Code](create-single-tenant-workflows-visual-studio-code.md#firewall-setup)

-* If your logic app workflows run in an [integration service environment (ISE)](connect-virtual-network-vnet-isolated-environment-overview.md), make sure that you [open these ports too](../logic-apps/connect-virtual-network-vnet-isolated-environment.md#network-ports-for-ise).

-* To help you simplify any security rules that you want to create, you can optionally use [service tags](../virtual-network/service-tags-overview.md) instead, rather than specify IP address prefixes for each region. These tags work across the regions where the Logic Apps service is available:

- * **LogicAppsManagement**: Represents the inbound IP address prefixes for the Logic Apps service.

- * **LogicApps**: Represents the outbound IP address prefixes for the Logic Apps service.

- * **AzureConnectors**: Represents the IP address prefixes for managed connectors that make inbound webhook callbacks to the Logic Apps service and outbound calls to their respective services, such as Azure Storage or Azure Event Hubs.

-> These tags work across the regions where the Logic Apps service is available.

-> The following connectors make inbound webhook callbacks to the Logic Apps service:

-> multi-tenant environment, the on-premises data gateway makes the call back to the Logic Apps service. In an ISE, the SAP

-> connector makes the call back to the Logic Apps service.

-> specify outbound managed connector IP address prefixes for each region. These tags work across the regions where the Logic Apps service is available.

-If your environment has strict network requirements or firewalls that limit traffic, you have to allow access for any trigger or action connections in your logic app workflows. To find the fully qualified domain names (FQDNs) for these connections, review the corresponding sections in these topics:

-* [Firewall permissions for single tenant logic apps - Visual Studio Code](create-single-tenant-workflows-visual-studio-code.md#firewall-setup)

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/deploy-managed-online-endpoint.sh" ID="get_logs" :::

- :::code language="yaml" source="~/azureml-examples-march-cli-preview/cli/endpoints/online/mlflow/create-endpoint.yaml":::

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/deploy-managed-online-endpoint-mlflow.sh" ID="create_endpoint":::

- :::code language="yaml" source="~/azureml-examples-march-cli-preview/cli/endpoints/online/mlflow/sklearn-deployment.yaml":::

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/deploy-managed-online-endpoint-mlflow.sh" ID="create_sklearn_deployment":::

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/deploy-triton-managed-online-endpoint.sh" ID="set_endpoint_name":::

- :::code language="yaml" source="~/azureml-examples-march-cli-preview/cli/endpoints/online/triton/single-model/create-managed-endpoint.yaml":::

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/deploy-triton-managed-online-endpoint.sh" ID="create_endpoint":::

- :::code language="yaml" source="~/azureml-examples-march-cli-preview/cli/endpoints/online/triton/single-model/create-managed-deployment.yaml":::

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/deploy-triton-managed-online-endpoint.sh" ID="create_deployment":::

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/deploy-triton-managed-online-endpoint.sh" ID="get_scoring_uri":::

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/deploy-triton-managed-online-endpoint.sh" ID="get_token":::

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/deploy-triton-managed-online-endpoint.sh" ID="check_scoring_of_model":::

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/batch-score.sh" ID="start_batch_scoring_job" :::

- :::code language="azurecli" source="~/azureml-examples-march-cli-preview/cli/batch-score.sh" ID="show_job_in_studio" :::

-#if WINDOWS_UWP

- // "HighLatencyAccuratePosition" - (Not yet implemented) Consumes more CPU

-var InstanceChangedHandler = new Windows.Foundation.TypedEventHandler<ObjectInstance, ObjectInstanceChangedEventArgs>((sender, args) =>

- ObjectInstanceState? state = sender.TryGetCurrentState();

- if (state.HasValue)

- // Process latest state via state.Value.

- // An object pose includes scale, rotation and translation, applied in

- sender.Dispose();

-});

-> You can also use the [JBoss EAP Operator](https://access.redhat.com/documentation/en-us/red_hat_jboss_enterprise_application_platform/7.4/html/getting_started_with_jboss_eap_for_openshift_container_platform/eap-operator-for-automating-application-deployment-on-openshift_default) to deploy this example, however, notice that the JBoss EAP Operator will deploy the application as `StatefulSets`. Use the JBoss EAP Operator if your application requires one or more one of the following.

-5. On the **Integration Runtime settings** page, follow the steps under the **Manual setup** section. You will have to download the integration runtime from the download site onto a VM or machine where you intend to run it.

-The default logon service account of self-hosted integration runtime is **NT SERVICE\DIAHostService**. You can see it in **Services -> Integration Runtime Service -> Properties -> Log on**.

-| `*.frontend.clouddatahub.net` | 443 | Required to connect to the Azure Purview service. Currently wildcard is required as there is no dedicated resource. |

-| `*.servicebus.windows.net` | 443 | Required for setting up scan on Azure Purview Studio. This endpoint is used for interactive authoring from UI, for example, test connection, browse folder list and table list to scope scan. Currently wildcard is required as there is no dedicated resource. |

-Depending on the sources you want to scan, you also need to allow additional domains and outbound ports for other Azure or external sources. A few examples are provided here:

-If your corporate network environment uses a proxy server to access the internet, configure the self-hosted integration runtime to use appropriate proxy settings. You can set the proxy during the initial registration phase or after it is being registered.

-When configured, the self-hosted integration runtime uses the proxy server to connect to the services which use HTTP or HTTPS protocol. This is why you select **Change link** during initial setup.

- The proxy tag allows additional properties to specify required settings like `scriptLocation`. See [\<proxy\> Element (Network Settings)](/dotnet/framework/configure-apps/file-schema/network/proxy-element-network-settings) for syntax.

-If you scan Parquet files using the self-hosted integration runtime with Azure Purview, you will need to install either the Java Runtime Environment or OpenJDK on your self-hosted IR machine.

-1. Allow Azure connections through the firewall.

-1. Install a Self-Hosted Integration Runtime and give it access through the firewall.

-1. You'll need a SQL login with at least `db_datareader` permissions to be able to access the information Azure Purview needs to scan the database. You can follow the instructions in [CREATE LOGIN](/sql/t-sql/statements/create-login-transact-sql?view=azuresqldb-current&preserve-view=true#examples-1) to create a sign in for Azure SQL Database. You'll need to save the **username** and **password** for the next steps.

- :::image type="content" source="media/register-scan-azure-sql-database/register-scan-azure-sql-db-key-vault.png" alt-text="Screenshot that shows the key vault to add a secret for for Service Principal.":::

- :::image type="content" source="media/register-scan-azure-sql-database/register-scan-azure-sql-db-credentials.png" alt-text="Screenshot that shows the key vault option to add a credentials for Service Principal.":::

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See our [Azure Purview Permissions page](catalog-permissions.md) for details.

-If your data store is publicly accessible, you can use the managed Azure integration runtime for scan without additional settings. Otherwise, if your data store limits access from on-premises network, private network or specific IPs, you need to configure a self-hosted integration runtime to connect to it:

-* Set up the latest [self-hosted integration runtime](https://www.microsoft.com/download/details.aspx?id=39717).

- For more information, see [the create and configure a self-hosted integration runtime guide](manage-integration-runtimes.md).

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-The supported IBM Db2 versions are Db2 for LUW 9.7 to 11.x. Db2 for z/OS (mainframe) and iSeries (AS/400) are not supported now.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See [Azure Purview Permissions page](catalog-permissions.md) for details.

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.html) is installed on the virtual machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-* Manually download a Db2 JDBC driver from [here](https://www.ibm.com/support/pages/db2-jdbc-driver-versions-and-downloads) onto your virtual machine where self-hosted integration runtime is running.

- > [!Note]

- > The driver should be accessible to all accounts in the VM. Do not install it in a user account.

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to create a self-hosted integration runtime.

- Usage of NOT and special characters are not acceptable.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See our [Azure Purview Permissions page](catalog-permissions.md) for details.

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the virtual machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up on the VM where erwin Mart instance is running. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to set up a self-hosted integration runtime.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See our [Azure Purview Permissions page](catalog-permissions.md) for details.

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the virtual machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-* Download and unzip BigQuery's JDBC driver on the machine where your self-hosted integration runtime is running. You can find the driver [here](https://cloud.google.com/bigquery/providers/simba-drivers).

- > [!Note]

- > The driver should be accessible to all accounts in the machine. Don't put it in a path under user account.

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md).

- Usage of NOT and special characters are not acceptable.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-* Ensure that [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the machine where the self-hosted integration runtime is running.

-* Ensure that Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the machine where the self-hosted integration runtime is running. If you don't have this update installed, [download it now](https://www.microsoft.com/download/details.aspx?id=30679).

-* Download the Hive Metastore database's JDBC driver on the machine where your self-hosted integration runtime is running. For example, if the database is *mssql*, download [Microsoft's JDBC driver for SQL Server](/sql/connect/jdbc/download-microsoft-jdbc-driver-for-sql-server). If you scan Azure Databricks's Hive Metastore, download the MariaDB Connector/J version 2.7.5 from [here](https://dlm.mariadb.com/1965742/Connectors/java/connector-java-2.7.5/mariadb-java-client-2.7.5.jar); version 3.0.3 is not supported.

- > [!Note]

- > The driver should be accessible to all accounts in the machine. Don't install it in a user account.

- Usage of `NOT` and special characters is not acceptable.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See our [Azure Purview Permissions page](catalog-permissions.md) for details.

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up on the VM where erwin Mart instance is running. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to set up a self-hosted integration runtime.

- 1. **Project filter** -- Scope your scan by providing a semicolon separated list of Looker projects. This option is used to select looks and dashboards by their parent project.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See [Azure Purview Permissions page](catalog-permissions.md) for details.

-If your data store is publicly accessible, you can use the managed Azure integration runtime for scan without additional settings. Otherwise, if your data store limits access from on-premises network, private network or specific IPs, you need to configure a self-hosted integration runtime to connect to it:

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.html) is installed on the machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to create a self-hosted integration runtime.

- Usage of NOT and special characters are not acceptable.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-The supported Oracle server versions are 6i to 19c. Proxy server is not supported when scanning Oracle source.

-Currently, the Oracle service name is not captured in the metadata or hierarchy.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See our [Azure Purview Permissions page](catalog-permissions.md) for details.

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the virtual machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-* Manually download an Oracle JDBC driver from [here](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html) onto your virtual machine where self-hosted integration runtime is running.

- > [!Note]

- > The driver should be accessible to all accounts in the VM. Do not install it in a user account.

-The user should have permission to create a session as well as role SELECT\_CATALOG\_ROLE assigned. Alternatively, the user may have SELECT permission granted for every individual system table that this connector queries metadata from:

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to create a self-hosted integration runtime.

- Usage of NOT and special characters are not acceptable.

- 1. **Stored procedure details**: Controls the amount of details imported from stored procedures:

- - None: Stored procedure details are not included.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See our [Azure Purview Permissions page](catalog-permissions.md) for details.

-If your data store is publicly accessible, you can use the managed Azure integration runtime for scan without additional settings. Otherwise, if your data store limits access from on-premises network, private network or specific IPs, you need to configure a self-hosted integration runtime to connect to it:

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.html) is installed on the machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to create a self-hosted integration runtime.

- Usage of NOT and special characters are not acceptable.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See our [Azure Purview Permissions page](catalog-permissions.md) for details.

-You can use the managed Azure integration runtime for scan - make sure to provide the security token to authenticate to Salesforce, learn more from the credential configuration in [Scan](#scan) section. Otherwise, if you want the scan to be initiated from a Salesforce trusted IP range for your organization, you can configure a self-hosted integration runtime to connect to it:

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.html) is installed on the machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-* Ensure the self-hosted integration runtime machine's IP is within the [trusted IP ranges for your organization](https://help.salesforce.com/s/articleView?id=sf.security_networkaccess.htm&type=5) set on Salesforce.

-In the event that users will be submitting Salesforce Documents, certain security settings must be configured to allow this access on Standard Objects and Custom Objects. To configure permissions:

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to create a self-hosted integration runtime.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-The supported SAP BW versions are 7.3 to 7.5. SAP BW4/HANA is not supported.

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-* The connector reads metadata from SAP using the [SAP Java Connector (JCo)](https://support.sap.com/en/product/connectors/jco.html) 3.0 API. Make sure the Java Connector is available on your machine where self-hosted integration runtime is installed. Make sure that you use the correct JCo distribution for your environment, and the **sapjco3.jar** and **sapjco3.dll** files are available.

- > [!Note]

- > The driver should be accessible to all accounts in the machine. Don't put it in a path under user account.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-* Ensure that [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the machine where the self-hosted integration runtime is running.

-* Ensure that Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the machine where the self-hosted integration runtime is running. If you don't have this update installed, [download it now](https://www.microsoft.com/download/details.aspx?id=30679).

-* Download the SAP HANA JDBC driver ([JAR ngdbc](https://mvnrepository.com/artifact/com.sap.cloud.db.jdbc/ngdbc)) on the machine where your self-hosted integration runtime is running.

- > [!Note]

- > The driver should be accessible to all accounts in the machine. Don't put it in a path under user account.

- Usage of NOT and special characters are not acceptable.

- 1. **Driver location**: Specify the path to the JDBC driver location in your machine where self-host integration runtime is running. This should be the path to valid JAR folder location. Do not include the name of the driver in the path.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See our [Azure Purview Permissions page](catalog-permissions.md) for details.

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the virtual machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-* Download the 64-bit [SAP Connector for Microsoft .NET 3.0](https://support.sap.com/en/product/connectors/msnet.html) from SAP\'s website and install it on the self-hosted integration runtime machine. During installation, make sure you select the **Install Assemblies to GAC** option in the **Optional setup steps** window.

- :::image type="content" source="media/register-scan-saps4hana-source/requirement.png" alt-text="pre-requisite" border="true":::

-* The connector reads metadata from SAP using the [SAP Java Connector (JCo)](https://support.sap.com/en/product/connectors/jco.html) 3.0 API. Make sure the Java Connector is available on your virtual machine where self-hosted integration runtime is installed. Make sure that you are using the correct JCo distribution for your environment. For example: on a Microsoft Windows machine, make sure the sapjco3.jar and sapjco3.dll files are available.

- > [!Note]

- > The driver should be accessible to all accounts in the VM. Do not install it in a user account.

-* Deploy the metadata extraction ABAP function module on the SAP server by following the steps mentioned in [ABAP functions deployment guide](abap-functions-deployment-guide.md). You will need an ABAP developer account to create the RFC function module on the SAP server. The user account requires sufficient permissions to connect to the SAP server and execute the following RFC function modules:

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to create a self-hosted integration runtime.

- 1. **Maximum memory available:** Maximum memory (in GB) available on the Self-hosted Integration Runtime machine to be used by scanning processes. This is dependent on the size of SAP ECC source to be scanned. It's recommended to provide large available memory e.g. 100.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See our [Azure Purview Permissions page](catalog-permissions.md) for details.

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the virtual machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-* Download the 64-bit [SAP Connector for Microsoft .NET 3.0](https://support.sap.com/en/product/connectors/msnet.html) from SAP\'s website and install it on the self-hosted integration runtime machine. During installation, make sure you select the **Install Assemblies to GAC** option in the **Optional setup steps** window.

- :::image type="content" source="media/register-scan-saps4hana-source/requirement.png" alt-text="pre-requisite" border="true":::

-* The connector reads metadata from SAP using the [SAP Java Connector (JCo)](https://support.sap.com/en/product/connectors/jco.html) 3.0 API. Hence make sure the Java Connector is available on your virtual machine where self-hosted integration runtime is installed. Make sure that you are using the correct JCo distribution for your environment. For example, on a Microsoft Windows machine, make sure the sapjco3.jar and sapjco3.dll files are available.

- > [!Note]

- >The driver should be accessible to all accounts in the VM. Do not install it in a user account.

-* Deploy the metadata extraction ABAP function module on the SAP server by following the steps mentioned in [ABAP functions deployment guide](abap-functions-deployment-guide.md). You will need an ABAP developer account to create the RFC function module on the SAP server. The user account requires sufficient permissions to connect to the SAP server and execute the following RFC function modules:

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to create a self-hosted integration runtime

- 1. **Maximum memory available:** Maximum memory (in GB) available on customer's VM to be used by scanning processes. This is dependent on the size of SAP S/4HANA source to be scanned. It's recommended to provide large available memory e.g. 100.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-If your data store is publicly accessible, you can use the managed Azure integration runtime for scan without additional settings. Otherwise, if your data store limits access from specific IPs, you need to configure a self-hosted integration runtime to connect to it:

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.html) is installed on the machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-1. In the Management Center, select Integration runtimes. Make sure a self-hosted integration runtime is set up. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to create a self-hosted integration runtime.

- Usage of NOT and special characters are not acceptable.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Azure Purview Studio. See our [Azure Purview Permissions page](catalog-permissions.md) for details.

-* Ensure [JDK 11](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) is installed on the virtual machine where the self-hosted integration runtime is installed.

-* Ensure Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the self-hosted integration runtime machine. If you don't have this update installed, [you can download it here](https://www.microsoft.com/download/details.aspx?id=30679).

-* You will have to manually download Teradata's JDBC Driver on your virtual machine where self-hosted integration runtime is running. The executable JAR file can be downloaded from the Teradata [website](https://downloads.teradata.com/).

- > [!Note]

- > The driver should be accessible to all accounts in the VM. Please do not install in a user account.

-1. In the Management Center, select **Integration runtimes**. Make sure a self-hosted integration runtime is set up. If it is not set up, use the steps mentioned [here](./manage-integration-runtimes.md) to set up a self-hosted integration runtime

- Usage of NOT and special characters are not acceptable

- 1. **Stored procedure details**: Controls the amount of details imported from stored procedures:

- - None: Stored procedure details are not included.

-Now that you have registered your source, follow the below guides to learn more about Azure Purview and your data.

-+ [Access tiers](../storage/blobs/access-tiers-overview.md) for Blob Storage include hot, cool, and archive. Only hot and cool can be accessed by search indexers.

-+ Blobs containing text. If blobs contain binary data or unstructured text, consider adding [AI enrichment](cognitive-search-concept-intro.md) for image and natural language processing. Blob content canΓÇÖt exceed the [indexer limits](search-limits-quotas-capacity.md#indexer-limits) for your search service tier.

-+ Read permissions on Azure Storage. A "full access" connection string includes a key that grants access to the content, but if you're using Azure roles instead, make sure the [search service managed identity](search-howto-managed-identities-data-sources.md) has **Storage Blob Data Reader** permissions.

-+ A REST client, such as [Postman](search-get-started-rest.md) or [Visual Studio Code with the extension for Azure Cognitive Search](search-get-started-vs-code.md) to send REST calls that create the data source, index, and indexer.

-Cognitive Search can use a system-assigned or user-assigned managed identity on outbound connections to Azure resources. A system managed identity is indicated when a connection string is the unique resource ID of an Azure AD-aware service or application. A user managed identity is specified through an "identity" property.

-| Scenario | System managed identity | User managed identity (preview) |

-| [Indexer connections to supported Azure data sources](search-indexer-overview.md) | Yes | Yes |

-| [Debug sessions (hosted in Azure Storage)](cognitive-search-debug-session.md) | Yes | No |

-| [Enrichment cache (hosted in Azure Storage)](search-howto-incremental-index.md)| Yes <sup>1,</sup> <sup>2</sup>| Yes |

-| [Knowledge Store (hosted in Azure Storage)](knowledge-store-create-rest.md) | Yes <sup>2</sup>| Yes |

-<sup>1</sup> The Import data wizard doesn't currently accept a managed identity connection string for enrichment cache, but after the wizard completes, you can update the connection string in indexer JSON definition to specify the managed identity, and then rerun the indexer.

-<sup>2</sup> If your indexer has an attached skillset that writes back to Azure Storage (for example, it creates a knowledge store or caches enriched content), a managed identity won't work if the storage account is behind a firewall or has IP restrictions. This is a known limitation that will be lifted when managed identity support for skillset scenarios becomes generally available. The solution is to use a full access connection string instead of a managed identity if Azure Storage is behind a firewall.

-Debug sessions, enrichment cache, and knowledge store are features that write to Blob Storage. Assign a managed identity to the **Storage Blob Data Contributor** role to support these features.

-Knowledge store will also write to Table Storage. Assign a managed identity to the **Storage Table Data Contributor** role to support table projections.

-You can use the Management REST API instead of the portal to assign a user managed identity. Be sure to use the [2021-04-01-preview management API](/rest/api/searchmanagement/2021-04-01-preview/services/create-or-update#searchcreateorupdateservicewithidentity) for this task.

-See [Create a search service with a system assigned managed identity (Azure PowerShell](search-manage-powershell.md#create-a-service-with-a-system-assigned-managed-identity).

-See [Create a search service with a system assigned managed identity (Azure CLI)](search-manage-azure-cli.md#create-a-service-with-a-system-assigned-managed-identity).

-## Create a user managed identity (preview)

->This feature is in public preview under [supplemental terms of use](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

- In the next several steps, you'll assign the user managed identity to your search service.

-You can use the Management REST API instead of the portal to assign a user managed identity. Be sure to use the [2021-04-01-preview management API](/rest/api/searchmanagement/2021-04-01-preview/services/create-or-update) for this task.

-+ For same-region connections to Azure Blob Storage or Azure Data Lake Storage Gen2, use the [trusted service exception](search-indexer-howto-access-trusted-service-exception.md) to admit requests.

-+ For all other resources and connections, [configure an IP firewall rule](search-indexer-howto-access-ip-restricted.md) that admits requests from Search. See [Indexer access to content protected by Azure network security features](search-indexer-securing-resources.md) for more detail.

-> If your indexer has an attached skillset that writes back to Azure Storage (for example, it creates a knowledge store or caches enriched content), a managed identity won't work if the storage account is behind a firewall or has IP restrictions. This is a known limitation that will be lifted when managed identity support for skillset scenarios becomes generally available. The solution is to use a full access connection string instead of a managed identity if Azure Storage is behind a firewall.

-## Prerequisites

-The storage account and the search service must be in different regions. If your setup doesn't permit this, try the [trusted service exception](search-indexer-howto-access-trusted-service-exception.md) or [resource instance rule](../storage/common/storage-network-security.md#grant-access-from-azure-resource-instances-preview).

-Indexers in an Azure Cognitive Search service that access blob data in Azure Storage accounts can make use of the [trusted service exception](../storage/common/storage-network-security.md#exceptions) capability to securely access data. This mechanism offers customers who are unable to grant [indexer access using IP firewall rules](search-indexer-howto-access-ip-restricted.md) a simple, secure, and free alternative for accessing data in storage accounts.

-## Prerequisites

-+ A search service with a [system-assigned managed identity](search-howto-managed-identities-data-sources.md).

-+ Azure Storage with the **Allow trusted Microsoft services to access this storage account** option.

-+ Content in Azure Blob Storage or Azure Data Lake Storage Gen2 (ADLS Gen2) that you want to index.

-> This capability is limited to blobs and ADLS Gen2 on Azure Storage. The trusted service exception is not supported for indexer connections to Azure Table Storage and Azure File Storage. It's also not currently supported for indexers that invoke skillsets that write to Azure Storage (knowledge store, enrichment cache, or debug sessions).

-1. On the **Identity** page, make sure that a system assigned identity is enabled. User-assigned managed identities, currently in preview, won't work for a trusted service connection.

-An indexer makes outbound calls in two situations:

-| Azure Storage (blobs, tables, ADLS Gen 2) | Data source |

-| Azure Storage (blobs, tables) | Skillsets (caching enriched documents, and storing knowledge store projections) |

-| Azure Functions | Attached to a skillset and used to host for custom web api skills |

-| Cognitive Services | Attached to a skillset and used to bill enrichment beyond the 20 free documents limit |

-> A Cognitive Service resource attached to a skillset is used for billing, based on the enrichments performed and written into the search index or a knowledge store. It isn't used for accessing the Cognitive Services APIs. Access from an indexer's enrichment pipeline to Cognitive Services APIs occurs via an internal secure communication channel, where data is strongly encrypted in transit and is never stored at rest.

-| Resource | IP Restriction | Private endpoint |

-| Azure Storage for text-based indexing (blobs, tables, ADLS Gen 2) | Supported only if the storage account and search service are in different regions. | Supported |

-| Azure Storage for AI enrichment (caching, knowledge store, debug sessions) | Supported only if the storage account and search service are in different regions, and when the search service connects using a full access connection string. Managed identity is not currently supported for write back operations to an IP restricted storage account. | Unsupported |

-> [!NOTE]

-> In addition to the options listed above, for network-secured Azure Storage accounts, you can make Azure Cognitive Search a [trusted Microsoft service](../storage/common/storage-network-security.md#trusted-microsoft-services). This means that a specific search service can bypass virtual network or IP restrictions on the storage account and can access data in the storage account, if the appropriate role-based access control is enabled on the storage account. For more information, see [Indexer connections using the trusted service exception](search-indexer-howto-access-trusted-service-exception.md). This option can be utilized instead of the IP restriction route, in case either the storage account or the search service can't be moved to a different region.

-| Maximum simple fields per index&nbsp;<sup>2</sup> |1000 |100 |3000 |3000 |3000 |1000 |1000 |1000 |

-<sup>1</sup> Basic services created before December 2017 have lower limits (5 instead of 15) on indexes. Basic tier is the only SKU with a lower limit of 100 fields per index. You might find some variation in maximum limits for Basic if your service is provisioned on a more powerful cluster. The limits here represent the common denominator. Indexes built to the above specifications will be portable across service tiers in any region.

-<sup>2</sup> The upper limit on fields includes both first-level fields and nested subfields in a complex collection. For example, if an index contains 15 fields and has two complex collections with 5 subfields each, the field count of your index is 25. Indexes with a large fields collection can be slow. Limit fields to just those you need, and run indexing and query test to ensure performance is acceptable.

-Outbound requests made by an indexer are subject to the authentication protocols supported by the external service. A search service can be made a trusted service on Azure, connecting to other services using a system or user managed identity. For more information, see [Set up an indexer connection to a data source using a managed identity](search-howto-managed-identities-data-sources.md).

-+ Index configuration (specifically, whether you include suggesters)

-You can monitor index size in the Indexes tab in the Azure portal, or by issuing a [GET INDEX request](/rest/api/searchservice/get-index) against your search service.

-### Factors influencing index size

-Document composition and quantity will be determined by what you choose to import. Remember that a search index should only contain searchable content. If source documents include binary fields, you would generally omit those fields from the index schema (unless you are using AI enrichment to crack and analyze the content to create text searchable information.)

-Index configuration can include other components besides documents, such as suggesters, customer analyzers, scoring profiles, CORS settings, and encryption key information. From the above list, the only component that has the potential for impacting index size is suggesters. [**Suggesters**](index-add-suggesters.md) are constructs that support type-ahead or autocomplete queries. As such, when you include a suggester, the indexing process will create the data structures necessary for verbatim character matches. Suggesters are implemented at the field level, so choose only those fields that are reasonable for type-ahead.

-Field attributes are the third consideration of index size. Attributes determine behaviors. To support those behaviors, the indexing process will create the supporting data structures. For example, "searchable" invokes [full text search](search-lucene-query-architecture.md), which scans inverted indices for the tokenized term. In contrast, a "filterable" or "sortable" attribute supports iteration over unmodified strings.

-> [!IMPORTANT]

->

-> - The **automation rules** feature is currently in **PREVIEW**. See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

-Automation rules are a new concept in Microsoft Sentinel. This feature allows users to centrally manage the automation of incident handling. Besides letting you assign playbooks to incidents (not just to alerts as before), automation rules also allow you to automate responses for multiple analytics rules at once, automatically tag, assign, or close incidents without the need for playbooks, and control the order of actions that are executed. Automation rules will streamline automation use in Microsoft Sentinel and will enable you to simplify complex workflows for your incident orchestration processes.

-A playbook is a collection of these remediation actions that can be run from Microsoft Sentinel as a routine. A playbook can help [**automate and orchestrate your threat response**](tutorial-respond-threats-playbook.md); it can be run manually or set to run automatically in response to specific alerts or incidents, when triggered by an analytics rule or an automation rule, respectively.

- > [!IMPORTANT]

- >

- > The **incident trigger** feature for playbooks is currently in **PREVIEW**. See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

-Automation rules are a new concept in Microsoft Sentinel. This feature allows users to centrally manage the automation of incident handling. Besides letting you assign playbooks to incidents (not just to alerts as before), automation rules also allow you to automate responses for multiple analytics rules at once, automatically tag, assign, or close incidents without the need for playbooks, and control the order of actions that are executed. Automation rules will streamline automation use in Microsoft Sentinel and will enable you to simplify complex workflows for your incident orchestration processes.

-> [!IMPORTANT]

->

-> - The **automation rules** feature is currently in **PREVIEW**. See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

-Automation rules help you triage incidents in Microsoft Sentinel. You can use them to automatically assign incidents to the right personnel, close noisy incidents or known [false positives](false-positives.md), change their severity, and add tags. They are also the mechanism by which you can run playbooks in response to incidents.

-> [!IMPORTANT]

->

-> - **Automation rules**, and the use of the **incident trigger** for playbooks, are currently in **PREVIEW**. See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

-| Availability for read requests | At least 99.9% (99% for cool access tier) | At least 99.9% (99% for cool access tier) | At least 99.9% (99% for cool access tier) for GRS<br /><br />At least 99.99% (99.9% for cool access tier) for RA-GRS | At least 99.9% (99% for cool access tier) for GZRS<br /><br />At least 99.99% (99.9% for cool access tier) for RA-GZRS |

-| Availability for write requests | At least 99.9% (99% for cool access tier) | At least 99.9% (99% for cool access tier) | At least 99.9% (99% for cool access tier) | At least 99.9% (99% for cool access tier) |

-* **Storage cost:** When this migration starts copying files into an Azure file share, Azure Files storage is consumed and billed. Migrated backups will become [Azure file share snapshots](storage-snapshots-files.md). File share snapshots only consume storage capacity for the differences they contain.

-description: See how to create and use Azure file shares with the Azure portal, Azure CLI, or Azure PowerShell module. Create a storage account, create an Azure file share, and use your Azure file share.

-#Customer intent: As a < type of user >, I want < what? > so that < why? >.

-A resource group is a logical container into which Azure resources are deployed and managed. If you don't already have an Azure resource group, you can create a new one with the [New-AzResourceGroup](/powershell/module/az.resources/new-azresourcegroup) cmdlet. You need a resource group to create a storage account.

-This example creates a storage account using the [New-AzStorageAccount](/powershell/module/az.storage/new-azstorageaccount) cmdlet. The storage account is named *mystorageaccount\<random number>* and a reference to that storage account is stored in the variable **$storageAcct**. Storage account names must be unique, so use `Get-Random` to append a number to the name to make it unique.

-The following example creates a storage account using the [az storage account create](/cli/azure/storage/account) command. Storage account names must be unique, so use `$RANDOM` to append a number to the name to make it unique.

- -Path "myDirectory\"

-> Because Azure Files may be accessed over SMB, it is possible to write simple applications that access the Azure file share using the standard Python I/O classes and functions. This article will describe how to write apps that use the Azure Files Storage Python SDK, which uses the [Azure Files REST API](/rest/api/storageservices/file-service-rest-api) to talk to Azure Files.

-|  |**Qubole**<br>Qubole provides a cloud-native platform that enables users to conduct ETL, analytics, and AI/ML workloads. It supports different kinds of open-source engines - Apache Spark, TensorFlow, Presto, Airflow, Hadoop, Hive, and more. It provides easy-to-use end-user tools for data processing from SQL query tools, to notebooks, and dashboards that use powerful open-source engines.|[Product page](https://www.qubole.com/company/partners/partners-microsoft-azure/) |  |**SAS® Viya®**<br>SAS® Viya® is an AI, analytic, and data management solution running on a scalable, cloud-native architecture. It enables you to operationalize insights, empowering everyone – from data scientists to business users – to collaborate and realize innovative results faster. Using open source or SAS models, SAS® Viya® can be accessed through APIs or interactive interfaces to transform raw data into actions. |[Product page](https://www.sas.com/microsoft)<br> [Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/sas-institute-560503.sas-viya-saas?tab=Overview)<br> |

-description: Azure Synapse Dedicated SQL Pool Connector for Apache Spark to move data between the Synapse Serverless Spark Pool and the Synapse Dedicated SQL Pool.

-The Azure Synapse Dedicated SQL Pool Connector for Apache Spark in Azure Synapse Analytics enables efficient transfer of large data sets between the [Apache Spark runtime](../../synapse-analytics/spark/apache-spark-overview.md) and the [Dedicated SQL pool](../../synapse-analytics/sql-data-warehouse/sql-data-warehouse-overview-what-is.md). The connector is implemented using `Scala` language. The connector is shipped as a default library within Azure Synapse environment - workspace Notebook and Serverless Spark Pool runtime. To use the Connector with other notebook language choices, use the Spark magic command - `%%spark`.

-At a high-level, the connector provides the following capabilities:

-* Write to Azure Synapse Dedicated SQL Pool:

- * Ingest large volume data to Internal and External table types.

- * Supports following DataFrame save mode preferences:

- * Write to External Table type supports Parquet and Delimited Text file format (example - CSV).

- * To write data to internal tables, the connector now uses [COPY statement](../../synapse-analytics/sql-data-warehouse/quickstart-bulk-load-copy-tsql.md) instead of CETAS/CTAS approach.

- * Enhancements to optimize end-to-end write throughput performance.

- * Introduces an optional call-back handle (a Scala function argument) that clients can use to receive post-write metrics.

- * Few examples include - number of records, duration to complete certain action, and failure reason.

-* Read from Azure Synapse Dedicated SQL Pool:

- * Read large data sets from Synapse Dedicated SQL Pool Tables (Internal and External) and Views.

- * Comprehensive predicate push down support, where filters on DataFrame get mapped to corresponding SQL predicate push down.

-## Orchestration Approach

-

-

-## Pre-requisites

-This section details necessary pre-requisite steps include Azure Resource setup and Configurations including authentication and authorization requirements for using the Azure Synapse Dedicated SQL Pool Connector for Apache Spark.

-### Azure Resources

-Review and setup following dependent Azure Resources:

-* [Azure Data Lake Storage](../../storage/blobs/data-lake-storage-introduction.md) - used as the primary storage account for the Azure Synapse Workspace.

-* [Azure Synapse Workspace](../../synapse-analytics/get-started-create-workspace.md) - create notebooks, build and deploy DataFrame based ingress-egress workflows.

-* [Dedicated SQL Pool (formerly SQL DW)](../../synapse-analytics/sql-data-warehouse/sql-data-warehouse-overview-what-is.md) - provides enterprise Data Warehousing features.

-* [Azure Synapse Serverless Spark Pool](../../synapse-analytics/get-started-analyze-spark.md) - Spark runtime where the jobs are executed as Spark Applications.

-#### Prepare the Database

-Connect to the Synapse Dedicated SQL Pool database and run following setup statements:

-* Create a database user that is mapped to the Azure Active Directory User Identity used to sign in to the Azure Synapse Workspace.

-* Create schema in which tables will be defined, such that the Connector can successfully write-to and read-from respective tables.

-#### Azure Active Directory based Authentication

-Azure Active Directory based authentication is an integrated authentication approach. The user is required to successfully log in to the Azure Synapse Analytics Workspace.

-A basic authentication approach requires user to configure `username` and `password` options. Refer to the section - [Configuration Options](#configuration-options) to learn about relevant configuration parameters for reading from and writing to tables in Azure Synapse Dedicated SQL Pool.

-There are two ways to grant access permissions to Azure Data Lake Storage Gen2 - Storage Account:

-* Role based Access Control role - [Storage Blob Data Contributor role](../../role-based-access-control/built-in-roles.md#storage-blob-data-contributor)

- * Assigning the `Storage Blob Data Contributor Role` grants the User permissions to read, write and delete from the Azure Storage Blob Containers.

-* [Access Control Lists (ACL)](../../storage/blobs/data-lake-storage-access-control.md)

- * ACL approach allows for fine-grained controls over specific paths and/or files under a given folder.

- * ACL checks aren't enforced if the User is already granted permissions using RBAC approach.

- * Access Permissions (applied at a specific level or object).

- * Default Permissions (automatically applied for all child objects at the time of their creation).

- * Type of permissions include:

- * `Execute` enables ability to traverse or navigate the folder hierarchies.

- * `Read` enables ability to read.

- * `Write` enables ability to write.

- * It's important to configure ACLs such that the Connector can successfully write and read from the storage locations.

-To enable successful interaction with Azure Synapse Dedicated SQL Pool, following authorization is necessary unless you're a user also configured as an `Active Directory Admin` on the Dedicated SQL End Point:

-* Write Scenario

- * Configure required permissions described [here](../../synapse-analytics/sql-data-warehouse/quickstart-bulk-load-copy-tsql.md#set-up-the-required-permissions).

-* Read Scenario

- * Grant the user `db_exporter` using the system stored procedure `sp_addrolemember`.

-## Connector API Documentation

-Azure Synapse Dedicated SQL Pool Connector for Apache Spark - [API Documentation](https://synapsesql.blob.core.windows.net/docs/latest/scala/https://docsupdatetracker.net/index.html).

-### Configuration Options

-To successfully bootstrap and orchestrate the read or write operation, the Connector expects certain configuration parameters. The object definition - `com.microsoft.spark.sqlanalytics.utils.Constants` provides a list of standardized constants for each parameter key.

-Following table describes the essential configuration options that must be set for each usage scenario:

-|Usage Scenario| Options to configure |

-| Write using Azure AD based authentication | <ul><li>Azure Synapse Dedicated SQL End Point<ul><li>`Constants.SERVER`<ul><li>By default, the Connector will infer the Synapse Dedicated SQL End Point associated with the database name (from the three part table name argument to `synapsesql` method).</li><li>Alternatively, users can provide the `Constants.SERVER` option.</li></ul></ul></li><li>Azure Data Lake Storage (Gen 2) End Point - Staging Folders<ul><li>For Internal Table Type:<ul><li>Configure either `Constants.TEMP_FOLDER` or `Constants.DATASOURCE` option.</li><li>If user chose to provide `Constants.DATASOURCE` option, staging folder will be derived by using the `location` value on the DataSource.</li><li>If both are provided, then the `Constants.TEMP_FOLDER` option value will be used.</li><li>In the absence of a staging folder option, the Connector will derive one based on the runtime configuration - `spark.sqlanalyticsconnector.stagingdir.prefix`.</li></ul></li><li>For External Table Type:<ul><li>`Constants.DATASOURCE` is a required configuration option.</li><li>The storage path defined on the Data Source's `location` parameter will be used as the base path to establish final absolute path.</li><li>The base path is then appended with the value set on the `synapsesql` method's `location` argument, example `/<external_table_name>`.</li><li>If the `location` argument to `synapsesql` method isn't provided, then the connector will derive the location value as `<base_path>/dbName/schemaName/tableName`.</li></ul></li></ul></li></ul>|

-| Write using Basic Authentication | <ul><li>Azure Synapse Dedicated SQL End Point<ul><li>`Constants.SERVER` - Synapse Dedicated SQL Pool End Point (Server FQDN)</li><li>`Constants.USER` - SQL User Name.</li><li>`Constants.PASSWORD` - SQL User Password.</li><li>`Constants.STAGING_STORAGE_ACCOUNT_KEY` associated with Storage Account that hosts `Constants.TEMP_FOLDERS` (internal table types only) or `Constants.DATASOURCE`.</li></ul></li><li>Azure Data Lake Storage (Gen 2) End Point - Staging Folders<ul><li>SQL basic authentication credentials don't apply to access storage end points. Hence it's required that the workspace user identity is given relevant access permissions (reference the section - [Azure Data Lake Storage Gen2](#azure-data-lake-storage-gen2).</li></ul></li></ul>|

-|Read using Azure AD based authentication|<ul><li>Credentials are auto-mapped, and user isn't required to provide specific configuration options.</li><li>Three-part table name argument on `synapsesql` method is required to read from respective table in Azure Synapse Dedicated SQL Pool.</li></ul>|

-|Read using basic authentication|<ul><li>Azure Synapse Dedicated SQL End Point<ul><li>`Constants.SERVER` - Synapse Dedicated SQL Pool End Point (Server FQDN)</li><li>`Constants.USER` - SQL User Name.</li><li>`Constants.PASSWORD` - SQL User Password.</li></ul></li><li>Azure Data Lake Storage (Gen 2) End Point - Staging Folders<ul><li>`Constants.DATA_SOURCE` - Location setting from data source is used to stage extracted data from Azure Synapse Dedicated SQL End Point.</li></ul></li></ul>|

-## Code Templates

-#### Write Request - `synapsesql` Method Signature

-The method signature for the Connector version built for Spark 2.4.8 has one less argument, than that applied to the Spark 3.1.2 version. Following are the two method signatures:

-* Spark Pool Version 2.4.8

-```Scala

-synapsesql(tableName:String,

- tableType:String = Constants.INTERNAL,

- location:Option[String] = None):Unit

-```

-* Spark Pool Version 3.1.2

-```Scala

-synapsesql(tableName:String,

- tableType:String = Constants.INTERNAL,

- location:Option[String] = None,

- callBackHandle=Option[(Map[String, Any], Option[Throwable])=>Unit]):Unit

-```

-### Write using Azure AD based Authentication

-Following is a comprehensive code template that describes how to use the Connector for write scenarios:

-//Define read options for example, if reading from CSV source, configure header and delimiter options.

-//Define read configuration for the input CSV

-//Initialize DataFrame that reads CSV data from a given source

-//Setup and trigger the read DataFrame for write to Synapse Dedicated SQL Pool.

-//Fully qualified SQL Server DNS name can be obtained using one of the following methods:

-// 2. From Azure Portal, follow the bread-crumbs for <Portal_Home> -> <Resource_Group> -> <Dedicated SQL Pool> and then go to Connection Strings/JDBC tab.

-//Like-wise, if `Constants.TEMP_FOLDER` is not provided, the connector will use the runtime staging directory config (see section on Configuration Options for details).

-//Setup optional callback/feedback function that can receive post write metrics of the job performed.

-//Configure and submit the request to write to Synapse Dedicated SQL Pool (note - default SaveMode is set to ErrorIfExists)

-//Sample below is using AAD-based authentication approach; See further examples to leverage SQL Basic auth.

-//If write request has failed, raise an error and fail the Cell's execution.

-Following code snippet replaces the write definition described in the [Write using Azure AD based authentication](#write-using-azure-ad-based-authentication) section, to submit write request using SQL basic authentication approach:

-//Define write options to use SQL basic authentication

-//Configure and submit the request to write to Synapse Dedicated SQL Pool.

-In a basic authentication approach, in order to read data from a source storage path other configuration options are required. Following code snippet provides an example to read from a Azure Data Lake Storage Gen2 data source using Service Principal credentials:

-//Define data frame to interface with the data source

-#### DataFrame Write SaveMode Support

-Following SaveModes are supported when writing source data to a destination table in Azure Synapse Dedicated SQL Pool:

- * If destination table exists, then the write is aborted with an exception returned to the callee. Else, a new table is created with data from the staging folders.

- * If the destination table exists, then the write will ignore the write request without returning an error. Else, a new table is created with data from the staging folders.

- * If the destination table exists, then existing data in the destination is replaced with data from the staging folders. Else, a new table is created with data from the staging folders.

- * If the destination table exists, then the new data is appended to it. Else, a new table is created with data from the staging folders.

-#### Write Request Callback Handle

-The new write path API changes introduced an experimental feature to provide the client with a key->value map of post-write metrics. Keys for the metrics are defined in the new Object definition - `Constants.FeedbackConstants`. Metrics can be retrieved as a JSON string by passing in the callback handle (a `Scala Function`). Following is the function signature:

-//Function signature is expected to have two arguments - a `scala.collection.immutable.Map[String, Any]` and an Option[Throwable]

-//These arguments will have valid objects in either Success or Failure case. In case of Failure the second argument will be a `Some(Throwable)`.

-Following are some notable metrics (presented in camel case):

-Following is a sample JSON string with post-write metrics:

-#### Read Request - `synapsesql` Method Signature

-#### Read using Azure AD based authentication

-//Use case is to read data from an internal table in Synapse Dedicated SQL Pool DB

-//Azure Active Directory based authentication approach is preferred here.

-//Read from existing internal table

- //to `synapsesql` method is used to infer the Synapse Dedicated SQL End Point.

-//Show contents of the dataframe

-#### Read using basic authentication

-//Use case is to read data from an internal table in Synapse Dedicated SQL Pool DB

-//Azure Active Directory based authentication approach is preferred here.

-//Read from existing internal table

- //to `synapsesql` method is used to infer the Synapse Dedicated SQL End Point.

- //Set database user name

- //Set user's password to the database

- //Set name of the data source definition that is defined with database scoped credentials.

- //Column-pruning i.e., query select column values.

- //Push-down filter criteria that gets translated to SQL Push-down Predicates.

- //Fetch a sample of 10 records

-//Show contents of the dataframe

-### More Code Samples

-#### Using the Connector with Other language preferences

-Example that demonstrates how to use the Connector with `PySpark (Python)` language preference:

-//Code to write or read goes here (refer to the aforementioned code templates)

-#### Using materialized data across cells

-Spark DataFrame's `createOrReplaceTempView` can be used to access data fetched in another cell, by registering a temporary view.

-* Cell where data is fetched (say with Notebook language preference as `Scala`)

-```Scala

- //Necessary imports

- import org.apache.spark.sql.DataFrame

- import org.apache.spark.sql.SaveMode

- import com.microsoft.spark.sqlanalytics.utils.Constants

- import org.apache.spark.sql.SqlAnalyticsConnector._

- //Configure options and read from Synapse Dedicated SQL Pool.

- val readDF = spark.read.

- //Set Synapse Dedicated SQL End Point name.

- option(Constants.SERVER, "<synapse-dedicated-sql-end-point>.sql.azuresynapse.net").

- //Set database user name.

- option(Constants.USER, "<user_name>").

- //Set database user's password.

- option(Constants.PASSWORD, "<user_password>").

- //Set name of the data source definition that is defined with database scoped credentials.

- option(Constants.DATA_SOURCE,"<data_source_name>").

- //Set the three-part table name from which the read must be performed.

- synapsesql("<database_name>.<schema_name>.<table_name>").

- //Optional - specify number of records the DataFrame would read.

- limit(10)

- //Register the temporary view (scope - current active Spark Session)

- readDF.createOrReplaceTempView("<temporary_view_name>")

-```

-* Now, change the language preference on the Notebook to `PySpark (Python)` and fetch data from the registered view `<temporary_view_name>`

-## Response Handling

-Invoking `synapsesql` has two possible end states - Success or a Failed State. This section describes how to handle the request response for each scenario.

-### Read Request Response

-Upon completion, the read response snippet is displayed in the cell's output. Failure in the current cell will also cancel subsequent cell executions. Detailed error information is available in the Spark Application Logs.

-### Write Request Response

-By default, a write response is printed to the cell output. On failure, the current cell is marked as failed, and subsequent cell executions will be aborted. The other approach is to pass the [callback handle](#write-request-callback-handle) option to the `synapsesql` method. The callback handle will provide programmatic access to the write response.

-## Things to Note

-* When writing to the Azure Synapse Dedicated SQL Pool tables:

- * Column types are inferred from the DataFrame that would read data from source. String columns are mapped to `NVARCHAR(4000)`.

- * DataFrame's initial parallelism drives the data organization for the external table.

- * Column types are inferred from the DataFrame that would read data from source.

- * When writing large data sets, it's important to factor in the impact of [DWU Performance Level](../../synapse-analytics/sql-data-warehouse/quickstart-scale-compute-portal.md) setting that limits [transaction size](../../synapse-analytics/sql-data-warehouse/sql-data-warehouse-develop-transactions.md#transaction-size).

-* When reading from the Azure Synapse Dedicated SQL Pool tables:

- * Consider applying necessary filters on the DataFrame to take advantage of the Connector's column-pruning feature.

- * Read scenario doesn't support the `TOP(n-rows)` clause, when framing the `SELECT` query statements. The choice to limit data is to use the DataFrame's limit(.) clause.

- * Refer the example - [Using materialized data across cells](#using-materialized-data-across-cells) section.

-* Monitor [Azure Data Lake Storage Gen2](../../storage/blobs/data-lake-storage-best-practices.md) utilization trends to spot throttling behaviors that can [impact](../../storage/common/scalability-targets-standard-account.md) read and write performance.

- .format("cosmos.oltp")\

- .outputMode("append")\

- .option("spark.synapse.linkedService", "<enter linked service name>")\

- .option("spark.cosmos.container", "<enter container name>")\

- .option("spark.cosmos.connection.mode", "Gateway")\

- format("cosmos.oltp").

- outputMode("append").

- option("spark.synapse.linkedService", "<enter linked service name>").

- option("spark.cosmos.container", "<enter container name>").

- option("spark.cosmos.connection.mode", "Gateway").

-You can copy incremental snapshots to any region of your choice. Azure manages the copy process removing the maintenance overhead of managing the copy process by staging a storage account in the target region. Moreover, Azure ensures that only changes since the last snapshot in the target region are copied to the target region to reduce the data footprint, reducing the recovery point objective. You can check the progress of the copy so you can know when a target snapshot is ready to restore disks in the target region. Customers are charged only for the bandwidth cost of the data transfer across the region.

-sudo sed -i 's/# AutoUpdate.Enabled=n/AutoUpdate.Enabled=y/g' /etc/waagent.conf

-sudo sed -i 's/# AutoUpdate.Enabled=n/AutoUpdate.Enabled=y/g' /etc/waagent.conf

-For Oracle Linux, make sure that the `Addons` repository is enabled. Choose to edit the file `/etc/yum.repos.d/public-yum-ol6.repo`(Oracle Linux 6) or `/etc/yum.repos.d/public-yum-ol7.repo`(Oracle Linux), and change the line `enabled=0` to `enabled=1` under **[ol6_addons]** or **[ol7_addons]** in this file.

-> The source managed image must be of a supported OS and the image must same region as your Azure Image Builder template.

-Sets the source image an existing image version in an Azure Compute Gallery.

-> The source managed image must be of a supported OS and the image must same region as your Azure Image Builder template, if not, please replicate the image version to the Image Builder Template region.

-| [Memory optimized](sizes-memory.md) | Esv3, Ev3, Easv4, Eav4, Ev4, Esv4, Edv4, Edsv4, Ev5, Esv5, Edv5, Edsv5, Easv5, Eadsv5, Mv2, M, DSv2, Dv2 | High memory-to-CPU ratio. Great for relational database servers, medium to large caches, and in-memory analytics. |

-Other address spaces may work but may have undesirable side effects.