+* Deployments set to **Auto-update to default** automatically update to use the new default version.

+* Deployments set to **Upgrade when expired** automatically update when its current version is retired.

+### VersionUpgradeOption

+You can check what model upgrade options are set for previously deployed models in [Azure OpenAI Studio](https://oai.azure.com). Select **Deployments** > Under the deployment name column select one of the deployment names that are highlighted in blue > The **Properties** will contain a value for **Version update policy**.

+The corresponding property can also be accessed via [REST](../how-to/working-with-models.md#model-deployment-upgrade-configuration), [Azure PowerShell](/powershell/module/az.cognitiveservices/get-azcognitiveservicesaccountdeployment), and [Azure CLI](/cli/azure/cognitiveservices/account/deployment#az-cognitiveservices-account-deployment-show).

+|Option| Read | Update |

+||||

+| [REST](../how-to/working-with-models.md#model-deployment-upgrade-configuration) | Yes. If `versionUpgradeOption` is not returned it means it is `null` |Yes |

+| [Azure PowerShell](/powershell/module/az.cognitiveservices/get-azcognitiveservicesaccountdeployment) | Yes.`VersionUpgradeOption` can be checked for `$null`| Yes |

+| [Azure CLI](/cli/azure/cognitiveservices/account/deployment#az-cognitiveservices-account-deployment-show) | Yes. It shows `null` if `versionUpgradeOption` is not set.| *No.* It is currently not possible to update the version upgrade option.|

+> [!NOTE]

+> `null` is equivalent to `AutoUpgradeWhenExpired`.

+**Azure PowerShell**

+Review the Azure PowerShell [getting started guide](/powershell/azure/get-started-azureps) to install Azure PowerShell locally or you can use the [Azure Cloud Shell](/azure/cloud-shell/overview).

+The steps below demonstrate checking the `VersionUpgradeOption` option property as well as updating it:

+```powershell

+// Step 1: Get Deployment

+$deployment = Get-AzCognitiveServicesAccountDeployment -ResourceGroupName {ResourceGroupName} -AccountName {AccountName} -Name {DeploymentName}

+

+// Step 2: Show Deployment VersionUpgradeOption

+$deployment.Properties.VersionUpgradeOption

+

+// VersionUpgradeOption can be null - one way to check is

+$null -eq $deployment.Properties.VersionUpgradeOption

+

+// Step 3: Update Deployment VersionUpgradeOption

+$deployment.Properties.VersionUpgradeOption = "NoAutoUpgrade"

+New-AzCognitiveServicesAccountDeployment -ResourceGroupName {ResourceGroupName} -AccountName {AccountName} -Name {DeploymentName} -Properties $deployment.Properties -Sku $deployment.Sku

+

+// repeat step 1 and 2 to confirm the change.

+// If not sure about deployment name, use this command to show all deployments under an account

+Get-AzCognitiveServicesAccountDeployment -ResourceGroupName {ResourceGroupName} -AccountName {AccountName}

+```

+## Embeddings

+GPT-4 and GPT-4-32k models are now available to all Azure OpenAI Service customers. Availability varies by region. If you don't see GPT-4 in your region, please check back later.

+| Model ID | Max Request (tokens) | Training Data (up to) |

+| | :: | :: |

+| `gpt-4` (0314) | 8,192 | Sep 2021 |

+| `gpt-4-32k`(0314) | 32,768 | Sep 2021 |

+| `gpt-4` (0613) | 8,192 | Sep 2021 |

+| `gpt-4-32k` (0613) | 32,768 | Sep 2021 |

+> [!NOTE]

+> Any region where GPT-4 is listed as available will always have access to both the 8K and 32K versions of the model

+### GPT-4 model availability

+| Model Availability | gpt-4 (0314) | gpt-4 (0613) |

+||:|:|

+| Available to all subscriptions with Azure OpenAI access | | Canada East <br> Sweden Central <br> Switzerland North |

+| Available to subscriptions with current access to the model version in the region | East US <br> France Central <br> South Central US <br> UK South | Australia East <br> East US <br> East US 2 <br> France Central <br> Japan East <br> UK South |

+> [!NOTE]

+> Version `0301` of `gpt-35-turbo` will be retired no earlier than July 5, 2024. See [model updates](../how-to/working-with-models.md#model-updates) for model upgrade behavior.

+### GPT-3.5-Turbo model availability

+| Model ID | Model Availability | Max Request (tokens) | Training Data (up to) |

+| | -- |::|:-:|

+| `gpt-35-turbo`¹ (0301) | East US <br> France Central <br> South Central US <br> UK South <br> West Europe | 4096 | Sep 2021 |

+| `gpt-35-turbo` (0613) | Australia East <br> Canada East <br> East US <br> East US 2 <br> France Central <br> Japan East <br> North Central US <br> Sweden Central <br> Switzerland North <br> UK South | 4096 | Sep 2021 |

+| `gpt-35-turbo-16k` (0613) | Australia East <br> Canada East <br> East US <br> East US 2 <br> France Central <br> Japan East <br> North Central US <br> Sweden Central <br> Switzerland North<br> UK South | 16,384 | Sep 2021 |

+| `gpt-35-turbo-instruct` (0914) | East US <br> Sweden Central | 4097 |Sep 2021 |

+¹ This model will accept requests > 4096 tokens. It is not recommended to exceed the 4096 input token limit as the newer version of the model are capped at 4096 tokens. If you encounter issues when exceeding 4096 input tokens with this model this configuration is not officially supported.

+| Model ID | Model Availability | Max Request (tokens) | Training Data (up to) | Output Dimensions |

+||| ::|::|::|

+| `text-embedding-ada-002` (version 2) | Australia East <br> Canada East <br> East US <br> East US2 <br> France Central <br> Japan East <br> North Central US <br> South Central US <br> Switzerland North <br> UK South <br> West Europe |8,191 | Sep 2021 | 1536 |

+| `text-embedding-ada-002` (version 1) | East US <br> South Central US <br> West Europe |2,046 | Sep 2021 | 1536 |

+| Model ID | Feature Availability | Max Request (characters) |

+| | | :: |

+| dalle2 | East US | 1000 |

+| | | :: | :: |

+| `babbage-002` | North Central US <br> Sweden Central | 16,384 | Sep 2021 |

+| `davinci-002` | North Central US <br> Sweden Central | 16,384 | Sep 2021 |

+| `gpt-35-turbo` (0613) | North Central US <br> Sweden Central | 4096 | Sep 2021 |

+| Model ID | Model Availability | Max Request (audio file size) |

+| | | :: |

+| `whisper` | North Central US <br> West Europe | 25 MB |

+* [Vector search](/azure/search/vector-search-overview) using Ada [embedding](./understand-embeddings.md) models, available in [select regions](models.md#embeddings-models).

+| `embeddingDeploymentName` | string | Optional | null | The Ada embedding model deployment name within the same Azure OpenAI resource. Used instead of `embeddingEndpoint` and `embeddingKey` for [vector search](./concepts/use-your-data.md#search-options). Should only be used when both the `embeddingEndpoint` and `embeddingKey` parameters are defined. When this parameter is provided, Azure OpenAI on your data will use an internal call to evaluate the Ada embedding model, rather than calling the Azure OpenAI endpoint. This enables you to use vector search in private networks and private endpoints. Billing remains the same whether this parameter is defined or not. Available in regions where embedding models are [available](./concepts/models.md#embeddings-models) starting in API versions `2023-06-01-preview` and later.|

+### Azure CNI Overlay networking

+[Azure CNI Overlay][azure-cni-overlay] represents an evolution of Azure CNI, addressing scalability and planning challenges arising from the assignment of VNet IPs to pods. It achieves this by assigning private CIDR IPs to pods, which are separate from the VNet and can be reused across multiple clusters. Additionally, Azure CNI Overlay can scale beyond the 400 node limit enforced in Kubenet clusters. Azure CNI Overlay is the recommended option for most clusters.

+[Azure CNI Powered by Cilium][azure-cni-powered-by-cilium] uses [Cilium](https://cilium.io) to provide high-performance networking, observability, and network policy enforcement. It integrates natively with [Azure CNI Overlay][azure-cni-overlay] for scalable IP address management (IPAM)

+Additionally, Cilium enforces network policies by default, without requiring a separate network policy engine. Using eBPF programs and a more efficient API object structure, Azure CNI Powered by Cilium can scale beyond [Azure Network Policy Manager's limits of 250 nodes / 20K pod][use-network-policies].

+Azure CNI Powered by Cilium is the recommended option for clusters that require network policy enforcement.

+Kubernetes Event-driven Autoscaling (KEDA) is a single-purpose and lightweight component that strives to make application autoscaling simple and is a CNCF Graduate project.

+| Platform metrics | [Platform metrics](monitor-aks-reference.md#metrics) are automatically collected for AKS clusters at no cost. You can analyze these metrics with [metrics explorer](../azure-monitor/essentials/analyze-metrics.md) or use them for [metric alerts](../azure-monitor/alerts/alerts-types.md#metric-alerts). |

+| Container insights | Container insights collects various logs and performance data from a cluster including stdout/stderr streams and stores them in a [Log Analytics workspace](../azure-monitor/logs/log-analytics-workspace-overview.md) and [Azure Monitor Metrics](../azure-monitor/essentials/data-platform-metrics.md). Analyze this data with views and workbooks included with Container insights or with [Log Analytics](../azure-monitor/logs/log-analytics-overview.md) and [metrics explorer](../azure-monitor/essentials/analyze-metrics.md). |

+The **Monitoring** tab on the **Overview** page offers a quick way to get started viewing monitoring data in the Azure portal for each AKS cluster. This includes graphs with common metrics for the cluster separated by node pool. Click on any of these graphs to further analyze the data in [metrics explorer](../azure-monitor/essentials/analyze-metrics.md).

+1. Check if the Metrics in **Kubernetes / Networking** Grafana dashboard are visible. If metrics aren't shown, change time range to last 15 minutes in top right dropdown box.

+>[!WARNING]

+> File should only be named as **`prometheus-config`**. Do not add any extensions like .yaml or .txt.

+ ```azurecli-interactive

+1. Open `http://localhost:9090` in your browser and navigate to **Status** > **Targets**, verify if **cilium-pods** are present and state says up.

+1. Sign in to Azure Managed Grafana and import dashboard with the ID: [16611](https://grafana.com/grafana/dashboards/16611-cilium-metrics/). Also, select **Dashboards** from the left navigation menu, open **Kubernetes / Networking** dashboard under **Managed Prometheus** folder. Metrics should be visible in both these dashboards.

+Analysis Services provides metrics in Azure Metrics Explorer, a free tool in the portal, to help you monitor the performance and health of your servers. For example, monitor memory and CPU usage, number of client connections, and query resource consumption. Analysis Services uses the same monitoring framework as most other Azure services. To learn more, see [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md).

+[Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md)

+When you migrate to App Service Environment v3 from previous versions, there are scenarios that you should consider that can potentially reduce your monthly cost. In addition to the following scenarios, consider [reservations](../../cost-management-billing/reservations/reservation-discount-app-service.md#how-reservation-discounts-apply-to-isolated-v2-instances) and [savings plans](../../cost-management-billing/savings-plan/savings-plan-compute-overview.md) to further reduce your costs.

+|**4**|**Optimize your App Service plans**|Once your upgrade is complete, you can optimize the App Service plans for additional benefits.<br><br>Review the autoselected Isolated v2 SKU sizes and scale up or scale down your App Service plans as needed.<br><br>- [Scale down your App Service plans](../manage-scale-up.md)<br>- [App Service Environment post-migration scaling guidance](migrate.md#pricing)<br><br>Explore reserved instance pricing, savings plans, and check out the pricing estimates if needed.<br><br>- [App Service pricing page](https://azure.microsoft.com/pricing/details/app-service/windows/)<br>- [How reservation discounts apply to Isolated v2 instances](../../cost-management-billing/reservations/reservation-discount-app-service.md#how-reservation-discounts-apply-to-isolated-v2-instances)<br>- [Azure pricing calculator](https://azure.microsoft.com/pricing/calculator)|

+You can analyze metrics for *App Service* with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+You can analyze metrics for Azure Application Gateway with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+|CorrelationId| string| An ID provided by the client to correlate multiple requests.

+|HitCount| int |The number of requests that the record is associated with.

+You can analyze metrics for App Configuration with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool. For App Configuration, the following metrics are collected:

+ | summarize requestCount=sum(HitCount) by ClientIPAddress

+ | summarize requestCount=sum(HitCount) by StatusCode

+ | summarize requestcount=sum(HitCount) by Day

+|Request quota usage exceeded | RequestQuotaUsage >= 100 | The configuration store has exceeded the [request quota usage](./faq.yml#are-there-any-limits-on-the-number-of-requests-made-to-app-configuration). Upgrade to a standard tier store or follow the [best practices](./howto-best-practices.md#reduce-requests-made-to-app-configuration) to optimize your usage. |

+To prepare for this new offer, you need to plan and prepare to onboard your machines to Azure Arc-enabled servers through the installation of the [Azure Connected Machine agent](agent-overview.md) (version 1.34 or higher) and establishing a connection to Azure. Windows Server 2012 Extended Security Updates supports Windows Server 2012 and R2 Standard and Datacenter editions. Windows Server 2012 Storage is not supported.

+You can analyze metrics for *Azure Functions* with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+Using various authentication systems can be cumbersome and risky because it's difficult to manage credentials at scale. You can now choose to [opt out of local authentication](#disable-local-authentication) to ensure only telemetry exclusively authenticated by using [managed identities](../../active-directory/managed-identities-azure-resources/overview.md) and [Microsoft Entra ID](../../active-directory/fundamentals/active-directory-whatis.md) is ingested in your resource. This feature is a step to enhance the security and reliability of the telemetry used to make critical operational ([alerting](../alerts/alerts-overview.md#what-are-azure-monitor-alerts) and [autoscaling](../autoscale/autoscale-overview.md#overview-of-autoscale-in-azure)) and business decisions.

+> This document covers data ingestion into Application Insights using Microsoft Entra ID-based authentication. For information on querying data within Application Insights, see [Query Application Insights using Microsoft Entra authentication](./app-insights-azure-ad-api.md).

+The following preliminary steps are required to enable Microsoft Entra authenticated ingestion. You need to:

+- Be familiar with:

+ - [Managed identity](../../active-directory/managed-identities-azure-resources/overview.md).

+ - [Service principal](../../active-directory/develop/howto-create-service-principal-portal.md).

+ - [Assigning Azure roles](../../role-based-access-control/role-assignments-portal.md).

+- [Application Insights Java 2.x SDK](deprecated-java-2x.md#monitor-dependencies-caught-exceptions-and-method-execution-times-in-java-web-apps).<br />

+- On-by-default [autoinstrumentation/codeless monitoring](codeless-overview.md) (for languages) for Azure App Service, Azure Virtual Machines/Azure Virtual Machine Scale Sets, and Azure Functions.

+ - For system-assigned, use the default constructor without parameters.

+ - For user-assigned, provide the client ID to the constructor.

+ - Provide the tenant ID, client ID, and client secret to the constructor.

+> For more information about migrating from the `2.X` SDK to the `3.X` Java agent, see [Upgrading from Application Insights Java 2.x SDK](java-standalone-upgrade-from-2x.md).

+- For system-assigned identity:

+| App setting | Value |

+| -- | |

+| APPLICATIONINSIGHTS_AUTHENTICATION_STRING | `Authorization=AAD` |

+- For user-assigned identity:

+| App setting | Value |

+| - | -- |

+| APPLICATIONINSIGHTS_AUTHENTICATION_STRING | `Authorization=AAD;ClientId={Client id of the User-Assigned Identity}` |

+The `OpenCensus` Azure Monitor exporters support these authentication types. We recommend using managed identities in production environments.

+*InstrumentationKey={profile.InstrumentationKey};IngestionEndpoint={ingestionEndpoint};LiveEndpoint={liveDiagnosticsEndpoint};AADAudience={aadAudience}*

+`Failed to get AAD Token. Error message:`.

+- [Monitor your telemetry in the portal](overview-dashboard.md)

+- [Diagnose with Live Metrics Stream](live-stream.md)

+- [Query Application Insights using Microsoft Entra authentication](./app-insights-azure-ad-api.md)

+* Analyze metrics with [metrics explorer](../essentials/analyze-metrics.md).

+ 1. If [`customDataPrefix`](#customdataprefix) isn't declared in the advanced configuration, the `id` provided in the `data-id` is used as the `customEvent` name.

+ 1. If [`customDataPrefix`](#customdataprefix) is declared, the `id` provided in the `data-*-id`, which means it must start with `data` and end with `id`, is used as the `customEvent` name. For example, if the clicked HTML element has the attribute `"data-sample-id"="button1"`, then `"button1"` is the `customEvent` name.

+ 1. If the `data-id` or `data-*-id` attribute doesn't exist and if [`useDefaultContentNameOrId`](#icustomdatatags) is set to `true`, the clicked element's HTML attribute `id` or content name of the element is used as the `customEvent` name. If both `id` and the content name are present, precedence is given to `id`.

+### `contentName`

+If you have the [`contentName` callback function](#ivaluecallback) in advanced configuration defined, the `contentName` column of the `customEvent` is populated based on the following rules:

+- For a clicked HTML `<a>` element, the plugin attempts to collect the value of its innerText (text) attribute. If the plugin can't find this attribute, it attempts to collect the value of its innerHtml attribute.

+- For a clicked HTML `<img>` or `<area>` element, the plugin collects the value of its `alt` attribute.

+- For all other clicked HTML elements, `contentName` is populated based on the following rules, which are listed in order of precedence:

+ 1. The value of the `value` attribute for the element

+ 1. The value of the `name` attribute for the element

+ 1. The value of the `alt` attribute for the element

+ 1. The value of the innerText attribute for the element

+ 1. The value of the `id` attribute for the element

+The [`customDataPrefix` option in advanced configuration](#icustomdatatags) provides the user the ability to configure a data attribute prefix to help identify where heart is located within the individual's codebase. The prefix must always be lowercase and start with `data-`. For example:

+In HTML, the `data-*` global attributes are called custom data attributes that allow proprietary information to be exchanged between the HTML and its DOM representation by scripts. Older browsers like Internet Explorer and Safari drop attributes they don't understand, unless they start with `data-`.

+Use [Azure Monitor metrics explorer](../essentials/analyze-metrics.md) to plot a chart for the custom metric name `React Component Engaged Time (seconds)` and [split](../essentials/analyze-metrics.md#use-dimension-filters-and-splitting) this custom metric by `Component Name`.

+ npm install @opentelemetry/sdk-trace-base

+| Azure Monitor Metrics | Platform metrics will write to the Azure Monitor metrics database with no configuration. Access platform metrics from Metrics Explorer. | [Analyze metrics with Azure Monitor metrics explorer](essentials/analyze-metrics.md) <br>[Supported metrics with Azure Monitor](essentials/metrics-supported.md) |

+ Title: Analyze metrics with Azure Monitor metrics explorer

+description: Learn how to analyze metrics with Azure Monitor metrics explorer by creating metrics charts, setting chart dimensions, time ranges, aggregation, filters, splitting, and sharing.

+# Analyze metrics with Azure Monitor metrics explorer

+In Azure Monitor, [metrics](data-platform-metrics.md) are a series of measured values and counts that are collected and stored over time. Metrics can be standard (also called *platform*) or custom. The Azure platform provides standard metrics. These metrics reflect the health and usage statistics of your Azure resources.

+In addition to standard metrics, your application emits extra _custom_ performance indicators or business-related metrics. Custom metrics can be emitted by any application or Azure resource and collected by using [Azure Monitor Insights](../insights/insights-overview.md), agents running on virtual machines, or [OpenTelemetry](../app/opentelemetry-enable.md).

+Azure Monitor metrics explorer is a component of the Azure portal that helps you plot charts, visually correlate trends, and investigate spikes and dips in metrics values. You can use metrics explorer to investigate the health and utilization of your resources.

+Watch the following video for an overview of creating and working with metrics charts in Azure Monitor metrics explorer.

+> [!VIDEO https://www.microsoft.com/videoplayer/embed/RE4qO59]

+## Create a metric chart

+You can open metrics explorer from the **Azure Monitor overview** page, or from the **Monitoring** section of any resource. In the Azure portal, select **Metrics**.

+If you open metrics explorer from Azure Monitor, the **Select a scope** page opens. Set the **Subscription**, **Resource**, and region **Location** fields to the resource to explore. If you open metrics explorer for a specific resource, the scope is prepopulated with information about that resource.

+Here's a summary of configuration tasks for creating a chart to analyze metrics:

+- [Select your resource and metric](#set-the-resource-scope) to see the chart. You can choose to work with one or multiple resources and view a single or multiple metrics.

+- [Configure the time settings](#configure-the-time-range) that are relevant for your investigation. You can set the time granularity to allow for pan and zoom on your chart, and configure aggregations to show values like the maximum and minimum.

+- [Use dimension filters and splitting](#use-dimension-filters-and-splitting) to analyze which segments of the metric contribute to the overall metric value and identify possible outliers in the data.

+- Work with advanced settings to customize your chart. [Lock the y-axis range](#lock-the-y-axis-range) to identify small data variations that might have significant consequences. [Correlate metrics to logs](#correlate-metrics-to-logs) to diagnose the cause of anomalies in your chart.

+- [Configure alerts](../alerts/alerts-metric-overview.md) and [receive notifications](#set-up-alert-rules) when the metric value exceeds or drops below a threshold.

+- [Share your chart](#share-your-charts) or pin it to dashboards.

+## Set the resource scope

+The resource **scope picker** lets you scope your chart to view metrics for a single resource or for multiple resources. To view metrics across multiple resources, the resources must be within the same subscription and region location.

+> [!NOTE]

+> You must have _Monitoring Reader_ permission at the subscription level to visualize metrics across multiple resources, resource groups, or a subscription. For more information, see [Assign Azure roles in the Azure portal](../../role-based-access-control/role-assignments-portal.md).

+### Select a single resource

+1. Choose **Select a scope**.

+ :::image source="./media/analyze-metrics/scope-picker.png" alt-text="Screenshot that shows how to open the resource scope picker for metrics explorer.":::

+1. Use the scope picker to select the resources whose metrics you want to see. If you open metrics explorer for a specific resource, the scope should be populated.

+ For some resources, you can view only one resource's metrics at a time. On the **Resource types** menu, these resources are shown in the **All resource types** section.

+ :::image source="./media/analyze-metrics/single-resource-scope.png" alt-text="Screenshot that shows available resources in the scope picker." lightbox="./media/analyze-metrics/single-resource-scope.png":::

+1. Select a resource. The picker updates to show all subscriptions and resource groups that contain the selected resource.

+ :::image source="./media/analyze-metrics/available-single-resource.png" alt-text="Screenshot that shows a single resource." lightbox="./media/analyze-metrics/available-single-resource.png":::

+ > [!TIP]

+ > If you want the capability to view the metrics for multiple resources at the same time, or to view metrics across a subscription or resource group, select **Upvote**.

+1. When you're satisfied with your selection, select **Apply**.

+### Select multiple resources

+You can see which metrics can be queried across multiple resources at the top of the **Resource types** menu in the scope picker.

+1. To visualize metrics over multiple resources, start by selecting multiple resources within the resource scope picker.

+ :::image source="./media/analyze-metrics/select-multiple-resources.png" alt-text="Screenshot that shows how to select multiple resources in the resource scope picker.":::

+ The resources you select must be within the same resource type, location, and subscription. Resources that don't meet these criteria aren't selectable.

+1. Select **Apply**.

+### Select a resource group or subscription

+For types that are compatible with multiple resources, you can query for metrics across a subscription or multiple resource groups.

+1. Start by selecting a subscription or one or more resource groups.

+ :::image source="./media/analyze-metrics/query-across-multiple-resource-groups.png" alt-text="Screenshot that shows how to query across multiple resource groups.":::

+1. Select a resource type and location.

+ :::image source="./media/analyze-metrics/select-resource-group.png" alt-text="Screenshot that shows how to select resource groups in the resource scope picker.":::

+1. Expand the selected scopes to verify the resources your selections apply to.

+ :::image source="./media/analyze-metrics/verify-selected-resources.png" alt-text="Screenshot that shows the selected resources within the groups.":::

+1. Select **Apply**.

+## Configure the time range

+The **time picker** lets you configure the time range for your metric chart to view data that's relevant to your monitoring scenario. By default, the chart shows the most recent 24 hours of metrics data.

+> [!NOTE]

+> [Most metrics in Azure are stored for 93 days](../essentials/data-platform-metrics.md#retention-of-metrics). You can query no more than 30 days of data on any single chart. You can [pan](#pan-across-metrics-data) the chart to view the full retention. The 30-day limitation doesn't apply to [log-based metrics](../app/pre-aggregated-metrics-log-metrics.md#log-based-metrics).

+Use the time picker to change the **Time range** for your data, such as the last 12 hours or the last 30 days.

+In addition to changing the time range with the time picker, you can pan and zoom by using the controls in the chart area.

+### Pan across metrics data

+To pan, select the left and right arrows at the edge of the chart. The arrow control moves the selected time range back and forward by one half of the chart's time span. If you're viewing the past 24 hours, selecting the left arrow causes the time range to shift to span a day and a half to 12 hours ago.

+### Zoom into metrics data

+You can configure the _time granularity_ of the chart data to support zoom in and zoom out for the time range. Use the **time brush** to investigate an interesting area of the chart like a spike or a dip in the data. Select an area on the chart and the chart zooms in to show more detail for the selected area based on your granularity settings. If the time grain is set to **Automatic**, zooming selects a smaller time grain. The new time range applies to all charts in metrics explorer.

+## View multiple metric lines and charts

+You can create charts that plot multiple metric lines or show multiple metric charts at the same time. This functionality allows you to:

+- Correlate related metrics on the same graph to see how one value relates to another.

+- Display metrics that use different units of measure in close proximity.

+- Visually aggregate and compare metrics from multiple resources.

+Suppose you have five storage accounts and you want to know how much space they consume together. You can create a stacked area chart that shows the individual values and the sum of all the values at points in time.

+After you create a chart, select **Add metric** to add another metric to the same chart.

+### Add multiple charts

+Typically, your charts shouldn't mix metrics that use different units of measure. For example, avoid mixing one metric that uses milliseconds with another that uses kilobytes. Also avoid mixing metrics whose scales differ significantly. In these cases, consider using multiple charts instead.

+- To create another chart that uses a different metric, select **New chart**.

+- To reorder or delete multiple charts, select **More options** (...), and then select the **Move up**, **Move down**, or **Delete** action.

+ :::image source="./media/analyze-metrics/multiple-charts.png" alt-text="Screenshot that shows multiple charts." lightbox="./media/analyze-metrics/multiple-charts.png":::

+### Use different line colors

+Chart lines are automatically assigned a color from a default palette. To change the color of a chart line, select the colored bar in the legend that corresponds to the line on the chart. Use the **color picker** to select the line color.

+Customized colors are preserved when you pin the chart to a dashboard. The following section shows how to pin a chart.

+## Configure aggregation

+When you add a metric to a chart, metrics explorer applies a default aggregation. The default makes sense in basic scenarios, but you can use a different aggregation to gain more insights about the metric.

+Before you use different aggregations on a chart, you should understand how metrics explorer handles them. Metrics are a series of measurements (or "metric values") that are captured over a time period. When you plot a chart, the values of the selected metric are separately aggregated over the _time granularity_.

+You select the size of the time grain by using the time picker in metrics explorer. If you don't explicitly select the time grain, metrics explorer uses the currently selected time range by default. After metrics explorer determines the time grain, the metric values that it captures during each time grain are aggregated on the chart, one data point per time grain.

+Suppose a chart shows the *Server response time* metric. It uses the average aggregation over the time span of the last 24 hours.

+In this scenario, if you set the time granularity to 30 minutes, metrics explorer draws the chart from 48 aggregated data points. That is, it uses two data points per hour for 24 hours. The line chart connects 48 dots in the chart plot area. Each data point represents the average of all captured response times for server requests that occurred during each of the relevant 30-minute time periods. If you switch the time granularity to 15 minutes, you get 96 aggregated data points. That is, you get four data points per hour for 24 hours.

+Metrics explorer has five aggregation types:

+- **Sum**: The sum of all values captured during the aggregation interval. The sum aggregation is sometimes called the *total* aggregation.

+- **Count**: The number of measurements captured during the aggregation interval.

+ When the metric is always captured with the value of 1, the count aggregation is equal to the sum aggregation. This scenario is common when the metric tracks the count of distinct events and each measurement represents one event. The code emits a metric record every time a new request arrives.

+- **Average**: The average of the metric values captured during the aggregation interval.

+- **Min**: The smallest value captured during the aggregation interval.

+- **Max**: The largest value captured during the aggregation interval.

+Metrics explorer hides the aggregations that are irrelevant and can't be used.

+For more information about how metric aggregation works, see [Azure Monitor metrics aggregation and display explained](metrics-aggregation-explained.md).

+## Use dimension filters and splitting

+Filtering and splitting are powerful diagnostic tools for metrics that have dimensions. You can implement these options to analyze which segments of the metric contribute to the overall metric value and identify possible outliers in the metric data. These features show how various metric segments or dimensions affect the overall value of the metric.

+**Filtering** lets you choose which dimension values are included in the chart. You might want to show successful requests when you chart the *server response time* metric. You apply the filter on the *success of request* dimension.

+**Splitting** controls whether the chart displays separate lines for each value of a dimension or aggregates the values into a single line. Splitting allows you to visualize how different segments of the metric compare with each other. You can see one line for an average CPU usage across all server instances, or you can see separate lines for each server.

+> [!TIP]

+> To hide segments that are irrelevant for your scenario and to make your charts easier to read, use both filtering and splitting on the same dimension.

+### Add filters

+You can apply filters to charts whose metrics have dimensions. Consider a *Transaction count* metric that has a *Response type* dimension. This dimension indicates whether the response from transactions succeeded or failed. If you filter on this dimension, metrics explorer displays a chart line for only successful or only failed transactions.

+1. Above the chart, select **Add filter** to open the **filter picker**.

+1. Select a dimension from the **Property** dropdown list.

+ :::image type="content" source="./media/analyze-metrics/filter-property.png" alt-text="Screenshot that shows the dropdown list for filter properties in metrics explorer." lightbox="./media/analyze-metrics/filter-property.png":::

+1. Select the operator that you want to apply against the dimension (or _property_). The default operator is equals (`=`).

+

+ :::image type="content" source="./media/analyze-metrics/filter-operator.png" alt-text="Screenshot that shows the operator that you can use with the filter.":::

+1. Select which dimension values you want to apply to the filter when you're plotting the chart. This example shows filtering out the successful storage transactions.

+ :::image type="content" source="./media/analyze-metrics/filter-values.png" alt-text="Screenshot that shows the dropdown list for filter values in metrics explorer.":::

+1. After you select the filter values, click outside the **filter picker** to complete the action. The chart shows how many storage transactions have failed.

+ :::image type="content" source="./media/analyze-metrics/filtered-chart.png" alt-text="Screenshot that shows the successful filtered storage transactions in the updated chart in metrics explorer." lightbox="./media/analyze-metrics/filtered-chart.png":::

+1. Repeat these steps to apply multiple filters to the same charts.

+### Apply metric splitting

+You can split a metric by dimension to visualize how different segments of the metric compare. Splitting can also help you identify the outlying segments of a dimension.

+1. Above the chart, select **Apply splitting** to open the **segment picker**.

+1. Choose the dimensions to use to segment your chart.

+ :::image type="content" source="./media/analyze-metrics/apply-splitting.png" alt-text="Screenshot that shows the selected dimension on which to segment the chart for splitting.":::

+ The chart shows multiple lines with one line for each dimension segment.

+ :::image type="content" source="./media/analyze-metrics/segment-dimension.png" alt-text="Screenshot that shows multiple lines, one for each segment of dimension." lightbox="./media/analyze-metrics/segment-dimension.png":::

+1. Choose a limit on the number of values to display after you split by the selected dimension. The default limit is 10, as shown in the preceding chart. The range of the limit is 1 to 50.

+ :::image type="content" source="./media/analyze-metrics/segment-dimension-limit.png" alt-text="Screenshot that shows the split limit, which restricts the number of values after splitting." lightbox="./media/analyze-metrics/segment-dimension-limit.png":::

+1. Choose the sort order on segments: **Descending** (default) or **Ascending**.

+ :::image type="content" source="./media/analyze-metrics/segment-dimension-sort.png" alt-text="Screenshot that shows the sort order on split values." lightbox="./media/analyze-metrics/segment-dimension-sort.png":::

+1. Segment by multiple segments by selecting multiple dimensions from the **Values** dropdown list. The legend shows a comma-separated list of dimension values for each segment.

+ :::image type="content" source="./media/analyze-metrics/segment-dimension-multiple.png" alt-text="Screenshot that shows multiple segments selected, and the corresponding chart." lightbox="./media/analyze-metrics/segment-dimension-multiple.png":::

+1. Click outside the segment picker to complete the action and update the chart.

+### Split metrics for multiple resources

+When you plot a metric for multiple resources, you can choose **Apply splitting** to split by resource ID or resource group. The split allows you to compare a single metric across multiple resources or resource groups. The following chart shows the percentage CPU across nine virtual machines. When you split by resource ID, you see how percentage CPU differs by virtual machine.

+For more examples that use filtering and splitting, see [Metric chart examples](../essentials/metric-chart-samples.md).

+## Lock the y-axis range

+Locking the range of the value (y) axis becomes important in charts that show small fluctuations of large values. Consider how a drop in the volume of successful requests from 99.99 percent to 99.5 percent might represent a significant reduction in the quality of service. Noticing a small fluctuation in a numeric value would be difficult or even impossible if you're using the default chart settings. In this case, you could lock the lowest boundary of the chart to 99 percent to make a small drop more apparent.

+Another example is a fluctuation in the available memory. In this scenario, the value technically never reaches 0. Fixing the range to a higher value might make drops in available memory easier to spot.

+1. To control the y-axis range, browse to the advanced chart settings by selecting **More options** (...) > **Chart settings**.

+ :::image source="./media/analyze-metrics/select-chart-settings.png" alt-text="Screenshot that shows the menu option for chart settings." lightbox="./media/analyze-metrics/select-chart-settings.png":::

+1. Modify the values in the **Y-axis range** section, or select **Auto** to revert to the default values.

+ :::image type="content" source="./media/analyze-metrics/chart-settings.png" alt-text="Screenshot that shows the Y-axis range section." lightbox="./media/analyze-metrics/chart-settings.png":::

+If you lock the boundaries of the y-axis for a chart that tracks count, sum, minimum, or maximum aggregations over a period of time, specify a fixed time granularity. Don't rely on the automatic defaults.

+You choose a fixed time granularity because chart values change when the time granularity is automatically modified after a user resizes a browser window or changes screen resolution. The resulting change in time granularity affects the appearance of the chart, invalidating the selection of the y-axis range.

+## Set up alert rules

+You can use your visualization criteria to create a metric-based alert rule. The new alert rule includes your chart's target resource, metric, splitting, and filter dimensions. You can modify these settings by using the **Create an alert rule** pane.

+1. To create an alert rule, select **New alert rule** in the upper-right corner of the chart.

+ :::image source="./media/analyze-metrics/new-alert.png" alt-text="Screenshot that shows the button for creating a new alert rule." lightbox="./media/analyze-metrics/new-alert.png":::

+1. Select the **Condition** tab. The **Signal name** entry defaults to the metric from your chart. You can choose a different metric.

+1. Enter a number for **Threshold value**. The threshold value is the value that triggers the alert. The **Preview** chart shows the threshold value as a horizontal line over the metric values. When you're ready, select the **Details** tab.

+ :::image source="./media/analyze-metrics/alert-rule-condition.png" alt-text="Screenshot that shows the Condition tab on the pane for creating an alert rule." lightbox="./media/analyze-metrics/alert-rule-condition.png":::

+1. Enter **Name** and **Description** values for the alert rule.

+1. Select a **Severity** level for the alert rule. Severities include **Critical**, **Error Warning**, **Informational**, and **Verbose**.

+1. Select **Review + create** to review the alert rule.

+ :::image source="./media/analyze-metrics/alert-rule-details.png" alt-text="Screenshot that shows the Details tab on the pane for creating an alert rule." lightbox="./media/analyze-metrics/alert-rule-details.png":::

+1. Select **Create** to create the alert rule.

+For more information, see [Create, view, and manage metric alerts](../alerts/alerts-metric.md).

+## Correlate metrics to logs

+In metrics explorer, the **Drill into Logs** feature helps you diagnose the root cause of anomalies in your metric chart. Drilling into logs allows you to correlate spikes in your metric chart to the following types of logs and queries:

+- **Activity log**: Provides insight into the operations on each Azure resource in the subscription from the outside (the management plane) and updates on Azure Service Health events. Use the activity log to determine the what, who, and when for any write operations (`PUT`, `POST`, or `DELETE`) taken on the resources in your subscription. There's a single activity log for each Azure subscription.

+- **Diagnostic log**: Provides insight into operations that you performed within an Azure resource (the data plane). Examples include getting a secret from a key vault or making a request to a database. The content of resource logs varies by the Azure service and resource type. You must enable logs for the resource.

+- **Recommended log** Provides scenario-based queries that you can use to investigate anomalies in metrics explorer.

+Currently, **Drill into Logs** is available for select resource providers. Resource providers that offer the complete **Drill into Logs** experience include Azure Application Insights, Autoscale, Azure App Service, and Azure Storage.

+1. To diagnose a spike in failed requests, select **Drill into Logs**.

+ :::image source="./media/analyze-metrics/drill-into-log-ai.png" alt-text="Screenshot that shows a spike in failures on an Application Insights metrics pane." lightbox="./media/analyze-metrics/drill-into-log-ai.png":::

+1. In the dropdown list, select **Failures**.

+ :::image source="./media/analyze-metrics/drill-into-logs-dropdown.png" alt-text="Screenshot that shows the dropdown menu for drilling into logs." lightbox="./media/analyze-metrics/drill-into-logs-dropdown.png":::

+1. On the custom failure pane, check for failed operations, top exception types, and failed dependencies.

+ :::image source="./media/analyze-metrics/ai-failure-blade.png" alt-text="Screenshot of the Application Insights failure pane." lightbox="./media/analyze-metrics/ai-failure-blade.png":::

+## Share your charts

+After you configure a chart, you can add it to a dashboard or workbook. By adding a chart to a dashboard or workbook, you can make it accessible to your team. You can also gain insights by viewing it in the context of other monitoring information.

+- To pin a configured chart to a dashboard, in the upper-right corner of the chart, select **Save to dashboard** > **Pin to dashboard**.

+- To save a configured chart to a workbook, in the upper-right corner of the chart, select **Save to dashboard** > **Save to workbook**.

+The Azure Monitor metrics explorer **Share** menu includes several options for sharing your metric chart.

+- Use the **Download to Excel** option to immediately download your chart.

+- Choose the **Copy link** option to add a link to your chart to the clipboard. You receive a notification when the link is copied successfully.

+- In the **Send to Workbook** window, send your chart to a new or existing workbook.

+- In the **Pin to Grafana** window, pin your chart to a new or existing Grafana dashboard.

+## Frequently asked questions

+This section provides answers to common questions.

+### Why are metrics from the guest OS of my Azure virtual machine not showing up in metrics explorer?

+[Platform metrics](./monitor-azure-resource.md#monitoring-data) are collected automatically for Azure resources. You must perform some configuration, though, to collect metrics from the guest OS of a virtual machine. For a Windows VM, install the diagnostic extension and configure the Azure Monitor sink as described in [Install and configure Azure Diagnostics extension for Windows (WAD)](../agents/diagnostics-extension-windows-install.md). For Linux, install the Telegraf agent as described in [Collect custom metrics for a Linux VM with the InfluxData Telegraf agent](./collect-custom-metrics-linux-telegraf.md).

+## Next steps

+- [Troubleshoot metrics explorer](metrics-troubleshoot.md)

+- [Review available metrics for Azure services](./metrics-supported.md)

+- [Explore examples of configured charts](../essentials/metric-chart-samples.md)

+- [Create custom KPI dashboards](../app/tutorial-app-dashboards.md)

+Since *standard metrics* are pre-aggregated during collection, they have better performance at query time. This makes them a better choice for dashboarding and in real-time alerting. The *log-based metrics* have more dimensions, which makes them the superior option for data analysis and ad-hoc diagnostics. Use the [namespace selector](./metrics-custom-overview.md#namespace) to switch between log-based and standard metrics in [metrics explorer](./analyze-metrics.md).

+When you plot the same metric in [metrics explorer](./analyze-metrics.md), there are no defaults - the query is dynamically adjusted based on your chart settings:

+For more information, see [Analyze metrics with Azure Monitor metrics explorer](./analyze-metrics.md).

+See [Apply dimension filters and splitting](analyze-metrics.md?#use-dimension-filters-and-splitting) for details on viewing metric dimensions in metrics explorer.

+Metrics are a series of values stored with a time-stamp. In Azure, most metrics are stored in the Azure Metrics time-series database. When you plot a chart, the values of the selected metrics are retrieved from the database and then separately aggregated based on the chosen time granularity (also known as time grain). You select the size of the time granularity using the [metrics explorer time picker](../essentials/analyze-metrics.md#configure-the-time-range). If you donΓÇÖt make an explicit selection, the time granularity is automatically selected based on the currently selected time range. Once selected, the metric values that were captured during each time granularity interval are aggregated and placed onto the chart - one datapoint per interval.

+- [Analyze metrics with Azure Monitor metrics explorer](../essentials/analyze-metrics.md)

+For more information on viewing metrics in the Azure portal, see [Analyze metrics with Azure Monitor metrics explorer](./analyze-metrics.md).

+> [!NOTE]

+> We strongly recommend identifying a workspace by its unique Workspace ID or Azure Resource ID because they remove ambiguity and are more performant.

+You can set the permissions on a query pack when you view it in the Azure portal. You need the following permissions to use query packs:

+## View query packs

+You can view and manage query packs in the Azure portal from the **Log Analytics query packs** menu. Select a query pack to view and edit its permissions. This article describes how to create a query pack by using the API.

+[](media/query-packs/view-query-pack.png#lightbox)

+Azure Monitor automatically creates a query pack called `DefaultQueryPack` in each subscription in a resource group called `LogAnalyticsDefaultResources` when you save your first query. You can save queries to this query pack or create other query packs depending on your requirements.

+The default query pack is sufficient for most users to save and reuse queries. You might want to create multiple query packs for users in your organization if, for example, you want to load different sets of queries in different Log Analytics sessions and provide different permissions for different collections of queries.

+When you [create a new query pack](#create-a-query-pack), you can add tags that classify queries based on your business needs. For example, you could tag a query pack to relate it to a particular department in your organization or to severity of issues that the included queries are meant to address. By using tags, you can create different sets of queries intended for different sets of users and different situations.

+To add query packs to your Log Analytics workspace:

+1. Open Log Analytics and select **Queries** in the upper-right corner.

+1. In the upper-left corner on the **Queries** dialog, next to **Query packs**, click **0 selected**.

+1. Select the query packs that you want to add to the workspace.

+> [!IMPORTANT]

+> You can add up to five query packs to a Log Analytics workspace.

+Each query in the query pack has the following properties:

+| Property | Description |

+|:|:|

+| `displayName` | Display name listed in Log Analytics for each query. |

+| `description` | Description of the query displayed in Log Analytics for each query. |

+| `body` | Query written in Kusto Query Language. |

+| `related` | Related categories, resource types, and solutions for the query. Used for grouping and filtering in Log Analytics by the user to help locate their query. Each query can have up to 10 of each type. Retrieve allowed values from https://api.loganalytics.io/v1/metadata?select=resourceTypes, solutions, and categories. |

+| `tags` | Other tags used by the user for sorting and filtering in Log Analytics. Each tag will be added to Category, Resource Type, and Solution when you [group and filter queries](queries.md#find-and-filter-queries). |

+You can analyze metrics for *Azure Monitor* with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](./essentials/analyze-metrics.md) for details on using this tool.

+|[Metrics explorer](essentials/analyze-metrics.md)|Use the Azure Monitor metrics explorer user interface in the Azure portal to investigate the health and utilization of your resources. Metrics explorer helps you plot charts, visually correlate trends, and investigate spikes and dips in metric values. Metrics explorer contains features for applying dimensions and filtering, and for customizing charts. These features help you analyze exactly the data you need in a visually intuitive way.|

+| Overview page | Select the **Monitoring** tab to display alerts, [platform metrics](../essentials/data-platform-metrics.md), and other monitoring information for the virtual machine host. You can see the number of active alerts on the tab. In the **Monitoring** tab, you get a quick view of:<br><br>**Alerts:** the alerts fired in the last 24 hours, with some important statistics about those alerts. If you do not have any alerts set up for this VM, there is a link to help you quickly create new alerts for your VM.<br><br>**Key metrics:** the trend over different time periods for important metrics, such as CPU, network, and disk. Because these are host metrics though, counters from the guest operating system such as memory aren't included. Select a graph to work with this data in [metrics explorer](../essentials/analyze-metrics.md) where you can perform different aggregations, and add more counters for analysis. |

+| Metrics | Open [metrics explorer](../essentials/analyze-metrics.md) with no scope selected. This feature is particularly useful when you want to compare trends across multiple machines. Select a subscription or a resource group to quickly add a group of machines to analyze together. |

+By using metrics explorer, you can plot charts, visually correlate trends, and investigate spikes and dips in metrics' values. For details on how to use this tool, see [Analyze metrics with Azure Monitor metrics explorer](../essentials/analyze-metrics.md).

+This article describes special cases that require extra steps when moving a virtual machine to a new resource group or Azure subscription. If your virtual machine uses disk encryption, a Marketplace plan, or Azure Backup, you must use one of the workarounds described in this article. For all other scenarios, move the virtual machine with the standard operations for [Azure portal](../move-resource-group-and-subscription.md#use-the-portal), [Azure CLI](../move-resource-group-and-subscription.md#use-azure-cli), or [Azure PowerShell](../move-resource-group-and-subscription.md#use-azure-powershell). For Azure CLI, use the [az resource move](/cli/azure/resource#az-resource-move) command. For Azure PowerShell, use the [Move-AzResource](/powershell/module/az.resources/move-azresource) command.

+# [Portal](#tab/Portal)

+# [Bicep](#tab/Bicep)

+Use Visual Studio Code or your favorite editor to create a file with the following content and name it main.bicep:

+```bicep

+@description('The name for your SignalR service')

+param primaryName string = 'contoso'

+@description('The region in which to create your SignalR service')

+param primaryLocation string = 'eastus'

+@description('Unit count of your SignalR service')

+param primaryCapacity int = 1

+resource primary 'Microsoft.SignalRService/signalr@2023-08-01-preview' = {

+ name: primaryName

+ location: primaryLocation

+ sku: {

+ capacity: primaryCapacity

+ name: 'Premium_P1'

+ }

+ properties: {

+ }

+}

+@description('The name for your SignalR replica')

+param replicaName string = 'contoso-westus'

+@description('The region in which to create the SignalR replica')

+param replicaLocation string = 'westus'

+@description('Unit count of the SignalR replica')

+param replicaCapacity int = 1

+@description('Whether to enable region endpoint for the replica')

+param regionEndpointEnabled string = 'Enabled'

+resource replica 'Microsoft.SignalRService/signalr/replicas@2023-08-01-preview' = {

+ parent: primary

+ name: replicaName

+ location: replicaLocation

+ sku: {

+ capacity: replicaCapacity

+ name: 'Premium_P1'

+ }

+ properties: {

+ regionEndpointEnabled: regionEndpointEnabled

+ }

+}

+```

+Deploy the Bicep file using Azure CLI

+ ```azurecli

+ az group create --name MyResourceGroup --location eastus

+ az deployment group create --resource-group MyResourceGroup --template-file main.bicep

+ ```

+-

+You can analyze metrics for Azure SignalR with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+You can analyze metrics for Azure Web PubSub with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+ > [!NOTE]

+ > By following these steps, Cloud Shell creates a standard storage account and allocates 5 GB of

+ > storage for the file share. You can also create a storage account manually and specify the

+ > storage account and file share to use. If you use a Premium storage account, Cloud Shell

+ > allocates 100 GB of storage for the file share.

+- **Details**: Administrators might want to disable access to Cloud Shell for their users. Cloud Shell

+ settings to your environment. Even though the Cloud Shell icon still exists in the Azure portal,

+ you can't connect to the service.

+ to domains at `*.console.azure.com` and `*.servicebus.windows.net`.

+- **Details**: Due to the default Windows Firewall settings for WinRM the user might see the following

+purpose computing platform. Excessive automated usage can be considered in breach to the Azure Terms

+ - FireFox might not support clipboard permissions properly.

+description: This article provides step-by-step instructions to deploy Azure Cloud Shell in a private virtual network.

+ms.contributor: jahelmic

+ Title: Deploy Azure Cloud Shell in a virtual network with quickstart templates

+# Deploy Cloud Shell in a virtual network by using quickstart templates

+Before you run quickstart templates to deploy Azure Cloud Shell in a virtual network (VNet), there

+are several prerequisites to complete. You must have the **Owner** role assignment on the

+subscription. To view and assign roles, see [List Owners of a Subscription][10].

+This article walks you through the following steps to configure and deploy Cloud Shell in a virtual

+network:

+1. Register resource providers.

+1. Collect the required information.

+1. Create the virtual networks by using the **Azure Cloud Shell - VNet** Azure Resource Manager

+ template (ARM template).

+1. Create the virtual network storage account by using the **Azure Cloud Shell - VNet storage** ARM

+ template.

+1. Configure and use Cloud Shell in a virtual network.

+## 1. Register resource providers

+Cloud Shell needs access to certain Azure resources. You make that access available through

+resource providers. The following resource providers must be registered in your subscription:

+- **Microsoft.CloudShell**

+- **Microsoft.ContainerInstances**

+- **Microsoft.Relay**

+Depending on when your tenant was created, some of these providers might already be registered.

+To see all resource providers and the registration status for your subscription:

+1. Sign in to the [Azure portal][04].

+1. On the Azure portal menu, search for **Subscriptions**. Select it from the available options.

+1. Select the subscription that you want to view.

+1. On the left menu, under **Settings**, select **Resource providers**.

+1. In the search box, enter `cloudshell` to search for the resource provider.

+1. Select the **Microsoft.CloudShell** resource provider from the provider list.

+1. Select **Register** to change the status from **unregistered** to **registered**.

+1. Repeat the previous steps for the **Microsoft.ContainerInstances** and **Microsoft.Relay**

+ resource providers.

+[![Screenshot of selecting resource providers in the Azure portal.][98]][98a]

+## 2. Collect the required information

+You need to collect several pieces of information before you can deploy Cloud Shell.

+You can use the default Cloud Shell instance to gather the required information and create the

+necessary resources. You should create dedicated resources for the Cloud Shell virtual network

+deployment. All resources must be in the same Azure region and in the same resource group.

+Fill in the following values:

+- **Subscription**: The name of your subscription that contains the resource group for the Cloud

+ Shell virtual network deployment.

+- **Resource Group**: The name of the resource group for the Cloud Shell virtual network deployment.

+- **Region**: The location of the resource group.

+- **Virtual Network**: The name of the Cloud Shell virtual network.

+- **Azure Container Instance OID**: The ID of the Azure container instance for your resource group.

+- **Azure Relay Namespace**: The name that you want to assign to the Azure Relay resource that the

+ template creates.

+### Create a resource group

+You can create the resource group by using the Azure portal, the Azure CLI, or Azure PowerShell. For

+more information, see the following articles:

+- [Manage Azure resource groups by using the Azure portal][02]

+- [Manage Azure resource groups by using Azure CLI][01]

+- [Manage Azure resource groups by using Azure PowerShell][03]

+### Create a virtual network

+You can create the virtual network by using the Azure portal, the Azure CLI, or Azure PowerShell.

+For more information, see the following articles:

+- [Use the Azure portal to create a virtual network][05]

+- [Use Azure PowerShell to create a virtual network][06]

+- [Use Azure CLI to create a virtual network][04]

+> [!NOTE]

+> When you're setting the container subnet address prefix for the Cloud Shell subnet, it's important

+> to consider the number of Cloud Shell sessions that you need to run concurrently. If the number of

+> Cloud Shell sessions exceeds the available IP addresses in the container subnet, users of those

+> sessions can't connect to Cloud Shell. Increase the container subnet range to accommodate your

+> specific needs. For more information, see the "Change subnet settings" section of

+> [Add, change, or delete a virtual network subnet][07].

+### Get the Azure container instance ID

+The Azure container instance ID is a unique value for every tenant. You use this identifier in

+the [quickstart templates][07] to configure a virtual network for Cloud Shell.

+1. Sign in to the [Azure portal][09]. From the home page, select **Microsoft Entra ID**. If the icon

+ isn't displayed, enter `Microsoft Entra ID` in the top search bar.

+1. On the left menu, select **Overview**. Then enter `azure container instance service` in the

+ search bar.

+ [![Screenshot of searching for Azure Container Instance Service.][95]][95a]

+1. In the results, under **Enterprise applications**, select **Azure Container Instance Service**.

+1. On the **Overview** page for **Azure Container Instance Service**, find the **Object ID** value

+ that's listed as a property.

+ You use this ID in the quickstart template for the virtual network.

+ [![Screenshot of Azure Container Instance Service details.][96]][96a]

+## 3. Create the virtual network by using the ARM template

+Use the [Azure Cloud Shell - VNet][08] template to create Cloud Shell resources in a virtual

+network. The template creates three subnets under the virtual network that you created earlier. You

+might choose to change the supplied names of the subnets or use the defaults.

+The virtual network, along with the subnets, requires valid IP address assignments. You need at

+least one IP address for the Relay subnet and enough IP addresses in the container subnet to support

+the number of concurrent sessions that you expect to use.

+The ARM template requires specific information about the resources that you created earlier, along

+with naming information for new resources. This information is filled out along with the prefilled

+information in the form.

+Information that you need for the template includes:

+- **Subscription**: The name of your subscription that contains the resource group for the Cloud

+ Shell virtual network.

+- **Resource Group**: The name of an existing or newly created resource group.

+- **Region**: The location of the resource group.

+- **Virtual Network**: The name of the Cloud Shell virtual network.

+- **Network Security Group**: The name that you want to assign to the network security group (NSG)

+ that the template creates.

+- **Azure Container Instance OID**: The ID of the Azure container instance for your resource group.

+Fill out the form with the following information:

+| Project details | Value |

+| | -- |

+| **Subscription** | Defaults to the current subscription context.<br>The example in this article uses `Contoso (carolb)`. |

+| **Resource group** | Enter the name of the resource group from the prerequisite information.<br>The example in this article uses `rg-cloudshell-eastus`. |

+| Instance details | Value |

+| - | - |

+| **Region** | Prefilled with your default region.<br>The example in this article uses `East US`. |

+| **Existing VNET Name** | Fill in the value from the prerequisite information that you gathered.<br>The example in this article uses `vnet-cloudshell-eastus`. |

+| **Relay Namespace Name** | Create a name that you want to assign to the Relay resource that the template creates.<br>The example in this article uses `arn-cloudshell-eastus`. |

+| **Nsg Name** | Enter the name of the NSG. The deployment creates this NSG and assigns an access rule to it. |

+| **Azure Container Instance OID** | Fill in the value from the prerequisite information that you gathered.<br>The example in this article uses `8fe7fd25-33fe-4f89-ade3-0e705fcf4370`. |

+| **Container Subnet Name** | Defaults to `cloudshellsubnet`. Enter the name of the subnet for your container. |

+| **Container Subnet Address Prefix** | The example in this article uses `10.1.0.0/16`, which provides 65,543 IP addresses for Cloud Shell instances. |

+| **Relay Subnet Name** | Defaults to `relaysubnet`. Enter the name of the subnet that contains your relay. |

+| **Relay Subnet Address Prefix** | The example in this article uses `10.0.2.0/24`. |

+| **Storage Subnet Name** | Defaults to `storagesubnet`. Enter the name of the subnet that contains your storage. |

+| **Storage Subnet Address Prefix** | The example in this article uses `10.0.3.0/24`. |

+| **Private Endpoint Name** | Defaults to `cloudshellRelayEndpoint`. Enter the name of the subnet that contains your container. |

+| **Tag Name** | Defaults to `{"Environment":"cloudshell"}`. Leave unchanged or add more tags. |

+| **Location** | Defaults to `[resourceGroup().location]`. Leave unchanged. |

+After the form is complete, select **Review + Create** and deploy the network ARM template to your

+subscription.

+## 4. Create the virtual network storage by using the ARM template

+Use the [Azure Cloud Shell - VNet storage][09] template to create Cloud Shell resources in a virtual

+network. The template creates the storage account and assigns it to the private virtual network.

+The ARM template requires specific information about the resources that you created earlier, along

+with naming information for new resources.

+Information that you need for the template includes:

+- **Subscription**: The name of the subscription that contains the resource group for the Cloud

+ Shell virtual network.

+- **Resource Group**: The name of an existing or newly created resource group.

+- **Region**: The location of the resource group.

+- **Existing virtual network name**: The name of the virtual network that you created earlier.

+- **Existing Storage Subnet Name**: The name of the storage subnet that you created by using the

+ network quickstart template.

+- **Existing Container Subnet Name**: The name of the container subnet that you created by using the

+ network quickstart template.

+Fill out the form with the following information:

+| Project details | Value |

+| | -- |

+| **Subscription** | Defaults to the current subscription context.<br>The example in this article uses `Contoso (carolb)`. |

+| **Resource group** | Enter the name of the resource group from the prerequisite information.<br>The example in this article uses `rg-cloudshell-eastus`. |

+| Instance details | Value |

+| | |

+| **Region** | Prefilled with your default region.<br>The example in this article uses `East US`. |

+| **Existing VNET Name** | The example in this article uses `vnet-cloudshell-eastus`. |

+| **Existing Storage Subnet Name** | Fill in the name of the resource that the network template creates. |

+| **Existing Container Subnet Name** | Fill in the name of the resource that the network template creates. |

+| **Storage Account Name** | Create a name for the new storage account.<br>The example in this article uses `myvnetstorage1138`. |

+| **File Share Name** | Defaults to `acsshare`. Enter the name of the file share that you want to create. |

+| **Resource Tags** | Defaults to `{"Environment":"cloudshell"}`. Leave unchanged or add more tags. |

+| **Location** | Defaults to `[resourceGroup().location]`. Leave unchanged. |

+After the form is complete, select **Review + Create** and deploy the network ARM template to your

+subscription.

+## 5. Configure Cloud Shell to use a virtual network

+After you deploy your private Cloud Shell instance, each Cloud Shell user must change their

+configuration to use the new private instance.

+If you used the default Cloud Shell instance before you deployed the private instance, you must

+reset your user settings:

+1. Open Cloud Shell.

+1. Select **Cloud Shell settings** from the menu bar (gear icon).

+1. Select **Reset user settings**, and then select **Reset**.

+Resetting the user settings triggers the first-time user experience the next time you start Cloud

+Shell.

+[![Screenshot of the Cloud Shell storage dialog.][97]][97a]

+1. Choose your preferred shell experience (Bash or PowerShell).

+1. Select **Show advanced settings**.

+1. Select the **Show VNET isolation settings** checkbox.

+1. Choose the subscription that contains your private Cloud Shell instance.

+1. Choose the region that contains your private Cloud Shell instance.

+1. For **Resource group**, select the resource group that contains your private Cloud Shell

+ instance.

+ If you select the correct resource group, **Virtual network**, **Network profile**, and **Relay

+ namespace** are automatically populated with the correct values.

+1. For **File share**, enter the name of the file share that you created by using the storage

+ template.

+1. Select **Create storage**.

+## Next steps

+You must complete the Cloud Shell configuration steps for each user who needs to use the new private

+Cloud Shell instance.

+<!-- link references -->

+[01]: /azure/azure-resource-manager/management/manage-resource-groups-cli

+[02]: /azure/azure-resource-manager/management/manage-resource-groups-portal

+[03]: /azure/azure-resource-manager/management/manage-resource-groups-powershell

+[04]: /azure/virtual-network/quick-create-cli

+[05]: /azure/virtual-network/quick-create-portal

+[06]: /azure/virtual-network/quick-create-powershell

+[07]: /azure/virtual-network/virtual-network-manage-subnet?tabs=azure-portal#change-subnet-settings

+[08]: https://aka.ms/cloudshell/docs/vnet/template

+[09]: https://azure.microsoft.com/resources/templates/cloud-shell-vnet-storage/

+[10]: /azure/role-based-access-control/role-assignments-list-portal#list-owners-of-a-subscription

+[95]: media/quickstart-deploy-vnet/container-service-search.png

+[95a]: media/quickstart-deploy-vnet/container-service-search.png#lightbox

+[96]: media/quickstart-deploy-vnet/container-service-details.png

+[96a]: media/quickstart-deploy-vnet/container-service-details.png#lightbox

+[97]: media/quickstart-deploy-vnet/setup-cloud-shell-storage.png

+[97a]: media/quickstart-deploy-vnet/setup-cloud-shell-storage.png#lightbox

+[98]: media/quickstart-deploy-vnet/resource-provider.png

+[98a]: media/quickstart-deploy-vnet/resource-provider.png#lightbox

+description: This article describes a scenario for using Azure Cloud Shell in a private virtual network.

+ms.contributor: jahelmic

+ Title: Use Cloud Shell in an Azure virtual network

+# Use Cloud Shell in an Azure virtual network

+By default, Azure Cloud Shell sessions run in a container in a Microsoft network that's separate

+from your resources. Commands that run inside the container can't access resources in a private

+virtual network. For example, you can't use Secure Shell (SSH) to connect from Cloud Shell to a

+virtual machine that has only a private IP address, or use `kubectl` to connect to a Kubernetes

+cluster that has locked down access.

+To provide access to your private resources, you can deploy Cloud Shell into an Azure virtual

+network that you control. This technique is called _virtual network isolation_.

+## Benefits of virtual network isolation with Cloud Shell

+Deploying Cloud Shell in a private virtual network offers these benefits:

+- The resources that you want to manage don't need to have public IP addresses.

+- You can use command-line tools, SSH, and PowerShell remoting from the Cloud Shell container to

+ manage your resources.

+- The storage account that Cloud Shell uses doesn't have to be publicly accessible.

+## Things to consider before deploying Azure Cloud Shell in a virtual network

+- Starting Cloud Shell in a virtual network is typically slower than a standard Cloud Shell session.

+- Virtual network isolation requires you to use [Azure Relay][01], which is a paid service. In the

+ Cloud Shell scenario, one hybrid connection is used for each administrator while they're using

+ Cloud Shell. The connection is automatically closed when the Cloud Shell session ends.

+## Architecture

+The following diagram shows the resource architecture that you must build to enable this scenario.

+![Illustration of a Cloud Shell isolated virtual network architecture.][03]

+- **Customer client network**: Client users can be located anywhere on the internet to securely

+ access and authenticate to the Azure portal and use Cloud Shell to manage resources contained in

+ the customer's subscription. For stricter security, you can allow users to open Cloud Shell only

+ from the virtual network contained in your subscription.

+- **Microsoft network**: Customers connect to the Azure portal on Microsoft's network to

+ authenticate and open Cloud Shell.

+- **Customer virtual network**: This is the network that contains the subnets to support virtual

+ network isolation. Resources such as virtual machines and services are directly accessible from

+ Cloud Shell without the need to assign a public IP address.

+- **Azure Relay**: [Azure Relay][01] allows two endpoints that aren't directly reachable to

+ communicate. In this case, it's used to allow the administrator's browser to communicate with the

+ container in the private network.

+- **File share**: Cloud Shell requires a storage account that's accessible from the virtual network.

+ The storage account provides the file share used by Cloud Shell users.

+## Related links

+For more information, see the [pricing][02] guide.

+<!-- link references -->

+[01]: ../azure-relay/relay-what-is-it.md

+[02]: https://azure.microsoft.com/pricing/details/service-bus/

+[03]: media/private-vnet/data-diagram.png

+the subscription. To view and assign roles, see [List Owners of a Subscription][01].

+Azure Communication Services currently provides metrics for all Communication Services primitives. You can use [Azure Monitor metrics explorer](../../../azure-monitor\essentials\analyze-metrics.md) to:

+Azure Communication Services currently provides metrics for all Communication Services primitives. You can use [Azure Monitor metrics explorer](../../../azure-monitor\essentials\analyze-metrics.md) to:

+Azure Communication Services currently provides metrics for all Communication Services primitives. [Azure Monitor metrics explorer](../../../azure-monitor\essentials\analyze-metrics.md) can be used to:

+You can analyze metrics for Azure Communications Gateway, along with metrics from other Azure services, by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+ Title: Deploy the Dapr extension for Azure Functions in Azure Container Apps (preview)

+# Deploy the Dapr extension for Azure Functions in Azure Container Apps (preview)

+1. Open the metrics explorer in the Azure portal by selecting **Metrics** from the sidebar menu on your container app's page. To learn more about metrics explorer, see [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md).

+You can analyze metrics for *Azure Container Instances* with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+You can analyze metrics for an Azure container registry with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+> [Get started using the Azure Cosmos DB emulator for development](how-to-develop-emulator.md)

+Diagnostic settings in Azure are used to collect resource logs. Resources emit Azure resource Logs and provide rich, frequent data about the operation of that resource. These logs are captured per request and they're also referred to as "data plane logs." Some examples of the data plane operations include delete, insert, and readFeed. The content of these logs varies by resource type.

+1. Navigate to your Azure Cosmos DB account. Open the **Diagnostic settings** pane under the **Monitoring section** and then select the **Add diagnostic setting** option.

+ > [!IMPORTANT]

+ > You might see a prompt to "enable full-text query \[...\] for more detailed logging" if the **full-text query** feature is not enabled in your account. You can safely ignore this warning if you do not wish to enable this feature. For more information, see [enable full-text query](monitor-resource-logs.md#enable-full-text-query-for-logging-query-text).

+ | **PartitionKeyStatistics** | All APIs | Logs the statistics of logical partition keys by representing the estimated storage size (KB) of the partition keys. This table is useful when troubleshooting storage skews. This PartitionKeyStatistics log is only emitted if the following conditions are true: 1. At least 1% of the documents in the physical partition have same logical partition key. 2. Out of all the keys in the physical partition, the PartitionKeyStatistics log captures the top three keys with largest storage size. </li></ul> If the previous conditions aren't met, the partition key statistics data isn't available. It's okay if the above conditions aren't met for your account, which typically indicates you have no logical partition storage skew. **Note**: The estimated size of the partition keys is calculated using a sampling approach that assumes the documents in the physical partition are roughly the same size. If the document sizes aren't uniform in the physical partition, the estimated partition key size might not be accurate. | `subscriptionId`, `regionName`, `partitionKey`, `sizeKB` |

+Azure Cosmos DB provides a custom experience for working with metrics. You can analyze metrics for Azure Cosmos DB with metrics from other Azure services using Metrics explorer by opening **Metrics** from the **Azure Monitor** menu. For more information about this tool, see [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md).

+ Title: Copy data in Microsoft Fabric Lakehouse Files (Preview)

+description: Learn how to copy data to and from Microsoft Fabric Lakehouse Files (Preview) using Azure Data Factory or Azure Synapse Analytics pipelines.

+# Copy data in Microsoft Fabric Lakehouse Files (Preview) using Azure Data Factory or Azure Synapse Analytics

+This article outlines how to use Copy Activity to copy data from and to Microsoft Fabric Lakehouse Files (Preview). To learn more, read the introductory article for [Azure Data Factory](introduction.md) or [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md).

+> [!IMPORTANT]

+> This connector is currently in preview. You can try it out and give us feedback. If you want to take a dependency on preview connectors in your solution, please contact [Azure support](https://azure.microsoft.com/support/).

+## Supported capabilities

+This Microsoft Fabric Lakehouse Files connector is supported for the following capabilities:

+| Supported capabilities|IR | Managed private endpoint|

+|| --| --|

+|[Copy activity](copy-activity-overview.md) (source/sink)|&#9312; &#9313;|Γ£ô |

+<small>*&#9312; Azure integration runtime &#9313; Self-hosted integration runtime*</small>

+## Get started

+## Create a Microsoft Fabric Lakehouse linked service using UI

+Use the following steps to create a Microsoft Fabric Lakehouse linked service in the Azure portal UI.

+1. Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services, then select New:

+ # [Azure Data Factory](#tab/data-factory)

+ :::image type="content" source="media/doc-common-process/new-linked-service.png" alt-text="Screenshot of creating a new linked service with Azure Data Factory UI.":::

+ # [Azure Synapse](#tab/synapse-analytics)

+ :::image type="content" source="media/doc-common-process/new-linked-service-synapse.png" alt-text="Screenshot of creating a new linked service with Azure Synapse UI.":::

+2. Search for Microsoft Fabric Lakehouse and select the connector.

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/microsoft-fabric-lakehouse-connector.png" alt-text="Screenshot showing select Microsoft Fabric Lakehouse connector.":::

+1. Configure the service details, test the connection, and create the new linked service.

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/configure-microsoft-fabric-lakehouse-linked-service.png" alt-text="Screenshot of configuration for Microsoft Fabric Lakehouse linked service.":::

+## Connector configuration details

+The following sections provide details about properties that are used to define Data Factory entities specific to Microsoft Fabric Lakehouse.

+## Linked service properties

+The Microsoft Fabric Lakehouse connector supports the following authentication types. See the corresponding sections for details:

+- [Service principal authentication](#service-principal-authentication)

+### Service principal authentication

+To use service principal authentication, follow these steps.

+1. Register an application with the Microsoft Identity platform. To learn how, see [Quickstart: Register an application with the Microsoft identity platform](../active-directory/develop/quickstart-register-app.md). Make note of these values, which you use to define the linked service:

+ - Application ID

+ - Application key

+ - Tenant ID

+2. Grant the service principal at least the **Contributor** role in Microsoft Fabric workspace. Follow these steps:

+ 1. Go to your Microsoft Fabric workspace, select **Manage access** on the top bar. Then select **Add people or groups**.

+

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/fabric-workspace-manage-access.png" alt-text="Screenshot shows selecting Fabric workspace Manage access.":::

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/manage-access-pane.png" alt-text=" Screenshot shows Fabric workspace Manage access pane.":::

+

+ 1. In **Add people** pane, enter your service principal name, and select your service principal from the drop-down list.

+

+ 1. Specify the role as **Contributor** or higher (Admin, Member), then select **Add**.

+

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/select-workspace-role.png" alt-text="Screenshot shows adding Fabric workspace role.":::

+ 1. Your service principal is displayed on **Manage access** pane.

+

+These properties are supported for the linked service:

+| Property | Description | Required |

+|: |: |: |

+| type | The type property must be set to **Lakehouse**. |Yes |

+| workspaceId | The Microsoft Fabric workspace ID. | Yes |

+| artifactId | The Microsoft Fabric Lakehouse object ID. | Yes |

+| tenant | Specify the tenant information (domain name or tenant ID) under which your application resides. Retrieve it by hovering the mouse in the upper-right corner of the Azure portal. | Yes |

+| servicePrincipalId | Specify the application's client ID. | Yes |

+| servicePrincipalCredentialType | The credential type to use for service principal authentication. Allowed values are **ServicePrincipalKey** and **ServicePrincipalCert**. | Yes |

+| servicePrincipalCredential | The service principal credential. <br/> When you use **ServicePrincipalKey** as the credential type, specify the application's key. Mark this field as **SecureString** to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). <br/> When you use **ServicePrincipalCert** as the credential, reference a certificate in Azure Key Vault, and ensure the certificate content type is **PKCS #12**.| Yes |

+| azureCloudType | For service principal authentication, specify the type of Azure cloud environment to which your Azure Active Directory application is registered. <br/> Allowed values are **AzurePublic**, **AzureChina**, **AzureUsGovernment**, and **AzureGermany**. By default, the data factory or Synapse pipeline's cloud environment is used. | No |

+| connectVia | The [integration runtime](concepts-integration-runtime.md) to be used to connect to the data store. You can use the Azure integration runtime or a self-hosted integration runtime if your data store is in a private network. If not specified, the default Azure integration runtime is used. |No |

+**Example: using service principal key authentication**

+You can also store service principal key in Azure Key Vault.

+```json

+{

+ "name": "MicrosoftFabricLakehouseLinkedService",

+ "properties": {

+ "type": "Lakehouse",

+ "typeProperties": {

+ "workspaceId": "<Microsoft Fabric workspace ID>",

+ "artifactId": "<Microsoft Fabric Lakehouse object ID>",

+ "tenant": "<tenant info, e.g. microsoft.onmicrosoft.com>",

+ "servicePrincipalId": "<service principal id>",

+ "servicePrincipalCredentialType": "ServicePrincipalKey",

+ "servicePrincipalCredential": {

+ "type": "SecureString",

+ "value": "<service principal key>"

+ }

+ },

+ "connectVia": {

+ "referenceName": "<name of Integration Runtime>",

+ "type": "IntegrationRuntimeReference"

+ }

+ }

+}

+```

+## Dataset properties

+For a full list of sections and properties available for defining datasets, see [Datasets](concepts-datasets-linked-services.md).

+Microsoft Fabric Lakehouse Files supports the following file formats. Refer to each article for format-based settings.

+- [Avro format](format-avro.md)

+- [Binary format](format-binary.md)

+- [Delimited text format](format-delimited-text.md)

+- [JSON format](format-json.md)

+- [ORC format](format-orc.md)

+- [Parquet format](format-parquet.md)

+The following properties are supported for Microsoft Fabric Lakehouse Files under `location` settings in the format-based dataset:

+| Property | Description | Required |

+| - | | -- |

+| type | The type property under `location` in the dataset must be set to **LakehouseLocation**. | Yes |

+| folderPath | The path to a folder under the given Microsoft Fabric Lakehouse. If you want to use a wildcard to filter folders, skip this setting and specify it in activity source settings. | No |

+| fileName | The file name under the given Microsoft Fabric Lakehouse + folderPath. If you want to use a wildcard to filter files, skip this setting and specify it in activity source settings. | No |

+**Example:**

+```json

+{

+ "name": "DelimitedTextDataset",

+ "properties": {

+ "type": "DelimitedText",

+ "linkedServiceName": {

+ "referenceName": "<Microsoft Fabric Lakehouse linked service name>",

+ "type": "LinkedServiceReference"

+ },

+ "typeProperties": {

+ "location": {

+ "type": "LakehouseLocation",

+ "fileName": "<file name>",

+ "folderPath": "<folder name>"

+ },

+ "columnDelimiter": ",",

+ "compressionCodec": "gzip",

+ "escapeChar": "\\",

+ "firstRowAsHeader": true,

+ "quoteChar": "\""

+ },

+ "schema": [ < physical schema, optional, auto retrieved during authoring > ]

+ }

+}

+```

+## Copy activity properties

+For a full list of sections and properties available for defining activities, see [Copy activity configurations](copy-activity-overview.md#configuration) and [Pipelines and activities](concepts-pipelines-activities.md). This section provides a list of properties supported by the Microsoft Fabric Lakehouse Files source and sink.

+### Microsoft Fabric Lakehouse Files as a source type

+Microsoft Fabric Lakehouse Files connector supports the following file formats. Refer to each article for format-based settings.

+- [Avro format](format-avro.md)

+- [Binary format](format-binary.md)

+- [Delimited text format](format-delimited-text.md)

+- [JSON format](format-json.md)

+- [ORC format](format-orc.md)

+- [Parquet format](format-parquet.md)

+You have several options to copy data from Microsoft Fabric Lakehouse Files:

+- Copy from the given path specified in the dataset.

+- Wildcard filter against folder path or file name, see `wildcardFolderPath` and `wildcardFileName`.

+- Copy the files defined in a given text file as file set, see `fileListPath`.

+The following properties are supported for Microsoft Fabric Lakehouse Files under `storeSettings` settings in format-based copy source:

+| Property | Description | Required |

+| | | |

+| type | The type property under `storeSettings` must be set to **LakehouseReadSettings**. | Yes |

+| ***Locate the files to copy:*** | | |

+| OPTION 1: static path<br> | Copy from the given file system or folder/file path specified in the dataset. If you want to copy all files from a file system/folder, additionally specify `wildcardFileName` as `*`. | |

+| OPTION 2: wildcard<br>- wildcardFolderPath | The folder path with wildcard characters under the given file system configured in dataset to filter source folders. <br>Allowed wildcards are: `*` (matches zero or more characters) and `?` (matches zero or single character); use `^` to escape if your actual folder name has wildcard or this escape char inside. <br>See more examples in [Folder and file filter examples](#folder-and-file-filter-examples). | No |

+| OPTION 2: wildcard<br>- wildcardFileName | The file name with wildcard characters under the given file system + folderPath/wildcardFolderPath to filter source files. <br>Allowed wildcards are: `*` (matches zero or more characters) and `?` (matches zero or single character); use `^` to escape if your actual file name has wildcard or this escape char inside. See more examples in [Folder and file filter examples](#folder-and-file-filter-examples). | Yes |

+| OPTION 3: a list of files<br>- fileListPath | Indicates to copy a given file set. Point to a text file that includes a list of files you want to copy, one file per line, which is the relative path to the path configured in the dataset.<br/>When using this option, do not specify file name in dataset. See more examples in [File list examples](#file-list-examples). |No |

+| ***Additional settings:*** | | |

+| recursive | Indicates whether the data is read recursively from the subfolders or only from the specified folder. Note that when recursive is set to true and the sink is a file-based store, an empty folder or subfolder isn't copied or created at the sink. <br>Allowed values are **true** (default) and **false**.<br>This property doesn't apply when you configure `fileListPath`. |No |

+| deleteFilesAfterCompletion | Indicates whether the binary files will be deleted from source store after successfully moving to the destination store. The file deletion is per file, so when copy activity fails, you will see some files have already been copied to the destination and deleted from source, while others are still remaining on source store. <br/>This property is only valid in binary files copy scenario. The default value: false. |No |

+| modifiedDatetimeStart | Files filter based on the attribute: Last Modified. <br>The files will be selected if their last modified time is greater than or equal to `modifiedDatetimeStart` and less than `modifiedDatetimeEnd`. The time is applied to UTC time zone in the format of "2018-12-01T05:00:00Z". <br> The properties can be NULL, which means no file attribute filter will be applied to the dataset. When `modifiedDatetimeStart` has datetime value but `modifiedDatetimeEnd` is NULL, it means the files whose last modified attribute is greater than or equal with the datetime value will be selected. When `modifiedDatetimeEnd` has datetime value but `modifiedDatetimeStart` is NULL, it means the files whose last modified attribute is less than the datetime value will be selected.<br/>This property doesn't apply when you configure `fileListPath`. | No |

+| modifiedDatetimeEnd | Same as above. | No |

+| enablePartitionDiscovery | For files that are partitioned, specify whether to parse the partitions from the file path and add them as additional source columns.<br/>Allowed values are **false** (default) and **true**. | No |

+| partitionRootPath | When partition discovery is enabled, specify the absolute root path in order to read partitioned folders as data columns.<br/><br/>If it is not specified, by default,<br/>- When you use file path in dataset or list of files on source, partition root path is the path configured in dataset.<br/>- When you use wildcard folder filter, partition root path is the sub-path before the first wildcard.<br/><br/>For example, assuming you configure the path in dataset as "root/folder/year=2020/month=08/day=27":<br/>- If you specify partition root path as "root/folder/year=2020", copy activity will generate two more columns `month` and `day` with value "08" and "27" respectively, in addition to the columns inside the files.<br/>- If partition root path is not specified, no extra column will be generated. | No |

+| maxConcurrentConnections | The upper limit of concurrent connections established to the data store during the activity run. Specify a value only when you want to limit concurrent connections.| No |

+**Example:**

+```json

+"activities": [

+ {

+ "name": "CopyFromLakehouseFiles",

+ "type": "Copy",

+ "inputs": [

+ {

+ "referenceName": "<Delimited text input dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "outputs": [

+ {

+ "referenceName": "<output dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "typeProperties": {

+ "source": {

+ "type": "DelimitedTextSource",

+ "storeSettings": {

+ "type": "LakehouseReadSettings",

+ "recursive": true,

+ "enablePartitionDiscovery": false

+ },

+ "formatSettings": {

+ "type": "DelimitedTextReadSettings"

+ }

+ },

+ "sink": {

+ "type": "<sink type>"

+ }

+ }

+ }

+]

+```

+### Microsoft Fabric Lakehouse Files as a sink type

+Microsoft Fabric Lakehouse Files connector supports the following file formats. Refer to each article for format-based settings.

+- [Avro format](format-avro.md)

+- [Binary format](format-binary.md)

+- [Delimited text format](format-delimited-text.md)

+- [JSON format](format-json.md)

+- [ORC format](format-orc.md)

+- [Parquet format](format-parquet.md)

+The following properties are supported for Microsoft Fabric Lakehouse Files under `storeSettings` settings in format-based copy sink:

+| Property | Description | Required |

+| | | -- |

+| type | The type property under `storeSettings` must be set to **LakehouseWriteSettings**. | Yes |

+| copyBehavior | Defines the copy behavior when the source is files from a file-based data store.<br/><br/>Allowed values are:<br/><b>- PreserveHierarchy (default)</b>: Preserves the file hierarchy in the target folder. The relative path of the source file to the source folder is identical to the relative path of the target file to the target folder.<br/><b>- FlattenHierarchy</b>: All files from the source folder are in the first level of the target folder. The target files have autogenerated names. <br/><b>- MergeFiles</b>: Merges all files from the source folder to one file. If the file name is specified, the merged file name is the specified name. Otherwise, it's an autogenerated file name. | No |

+| maxConcurrentConnections | The upper limit of concurrent connections established to the data store during the activity run. Specify a value only when you want to limit concurrent connections.| No |

+| metadata |Set custom metadata when copy to sink. Each object under the `metadata` array represents an extra column. The `name` defines the metadata key name, and the `value` indicates the data value of that key. If [preserve attributes feature](./copy-activity-preserve-metadata.md#preserve-metadata) is used, the specified metadata will union/overwrite with the source file metadata.<br/><br/>Allowed data values are:<br/>- `$$LASTMODIFIED`: a reserved variable indicates to store the source files' last modified time. Apply to file-based source with binary format only.<br/><b>- Expression<b><br/>- <b>Static value<b>| No |

+**Example:**

+```json

+"activities": [

+ {

+ "name": "CopyToLakehouseFiles",

+ "type": "Copy",

+ "inputs": [

+ {

+ "referenceName": "<input dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "outputs": [

+ {

+ "referenceName": "<Parquet output dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "typeProperties": {

+ "source": {

+ "type": "<source type>"

+ },

+ "sink": {

+ "type": "ParquetSink",

+ "storeSettings": {

+ "type": "LakehouseWriteSettings",

+ "copyBehavior": "PreserveHierarchy",

+ "metadata": [

+ {

+ "name": "testKey1",

+ "value": "value1"

+ },

+ {

+ "name": "testKey2",

+ "value": "value2"

+ }

+ ]

+ },

+ "formatSettings": {

+ "type": "ParquetWriteSettings"

+ }

+ }

+ }

+ }

+]

+```

+### Folder and file filter examples

+This section describes the resulting behavior of the folder path and file name with wildcard filters.

+| folderPath | fileName | recursive | Source folder structure and filter result (files in **bold** are retrieved)|

+|: |: |: |: |

+| `Folder*` | (Empty, use default) | false | FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File2.json**<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5.csv<br/>AnotherFolderB<br/>&nbsp;&nbsp;&nbsp;&nbsp;File6.csv |

+| `Folder*` | (Empty, use default) | true | FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File2.json**<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File3.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File4.json**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File5.csv**<br/>AnotherFolderB<br/>&nbsp;&nbsp;&nbsp;&nbsp;File6.csv |

+| `Folder*` | `*.csv` | false | FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5.csv<br/>AnotherFolderB<br/>&nbsp;&nbsp;&nbsp;&nbsp;File6.csv |

+| `Folder*` | `*.csv` | true | FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File3.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File5.csv**<br/>AnotherFolderB<br/>&nbsp;&nbsp;&nbsp;&nbsp;File6.csv |

+### File list examples

+This section describes the resulting behavior of using file list path in copy activity source.

+Assuming you have the following source folder structure and want to copy the files in bold:

+| Sample source structure | Content in FileListToCopy.txt | ADF configuration |

+| | | |

+| filesystem<br/>&nbsp;&nbsp;&nbsp;&nbsp;FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File2.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File3.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File5.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;Metadata<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;FileListToCopy.txt | File1.csv<br>Subfolder1/File3.csv<br>Subfolder1/File5.csv | **In dataset:**<br>- File system: `filesystem`<br>- Folder path: `FolderA`<br><br>**In copy activity source:**<br>- File list path: `filesystem/Metadata/FileListToCopy.txt` <br><br>The file list path points to a text file in the same data store that includes a list of files you want to copy, one file per line with the relative path to the path configured in the dataset. |

+### Some recursive and copyBehavior examples

+This section describes the resulting behavior of the copy operation for different combinations of recursive and copyBehavior values.

+| recursive | copyBehavior | Source folder structure | Resulting target |

+|: |: |: |: |

+| true |preserveHierarchy | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the same structure as the source:<br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 |

+| true |flattenHierarchy | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the following structure: <br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File5 |

+| true |mergeFiles | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the following structure: <br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1 + File2 + File3 + File4 + File5 contents are merged into one file with an autogenerated file name. |

+| false |preserveHierarchy | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the following structure: <br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/><br/>Subfolder1 with File3, File4, and File5 isn't picked up. |

+| false |flattenHierarchy | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the following structure: <br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File2<br/><br/>Subfolder1 with File3, File4, and File5 isn't picked up. |

+| false |mergeFiles | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the following structure: <br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1 + File2 contents are merged into one file with an autogenerated file name. autogenerated name for File1<br/><br/>Subfolder1 with File3, File4, and File5 isn't picked up. |

+## Next steps

+For a list of data stores supported as sources and sinks by the copy activity, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).

+ Title: Copy data in Microsoft Fabric Lakehouse Table (Preview)

+description: Learn how to copy data to and from Microsoft Fabric Lakehouse Table (Preview) using Azure Data Factory or Azure Synapse Analytics pipelines.

+# Copy data in Microsoft Fabric Lakehouse Table (Preview) using Azure Data Factory or Azure Synapse Analytics

+This article outlines how to use Copy Activity to copy data from and to Microsoft Fabric Lakehouse Table (Preview). To learn more, read the introductory article for [Azure Data Factory](introduction.md) or [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md).

+> [!IMPORTANT]

+> This connector is currently in preview. You can try it out and give us feedback. If you want to take a dependency on preview connectors in your solution, please contact [Azure support](https://azure.microsoft.com/support/).

+## Supported capabilities

+This Microsoft Fabric Lakehouse Table connector is supported for the following capabilities:

+| Supported capabilities|IR | Managed private endpoint|

+|| --| --|

+|[Copy activity](copy-activity-overview.md) (source/sink)|&#9312; &#9313;|Γ£ô |

+<small>*&#9312; Azure integration runtime &#9313; Self-hosted integration runtime*</small>

+## Get started

+## Create a Microsoft Fabric Lakehouse linked service using UI

+Use the following steps to create a Microsoft Fabric Lakehouse linked service in the Azure portal UI.

+1. Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services, then select New:

+ # [Azure Data Factory](#tab/data-factory)

+ :::image type="content" source="media/doc-common-process/new-linked-service.png" alt-text="Screenshot of creating a new linked service with Azure Data Factory UI.":::

+ # [Azure Synapse](#tab/synapse-analytics)

+ :::image type="content" source="media/doc-common-process/new-linked-service-synapse.png" alt-text="Screenshot of creating a new linked service with Azure Synapse UI.":::

+2. Search for Microsoft Fabric Lakehouse and select the connector.

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/microsoft-fabric-lakehouse-connector.png" alt-text="Screenshot showing select Microsoft Fabric Lakehouse connector.":::

+1. Configure the service details, test the connection, and create the new linked service.

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/configure-microsoft-fabric-lakehouse-linked-service.png" alt-text="Screenshot of configuration for Microsoft Fabric Lakehouse linked service.":::

+## Connector configuration details

+The following sections provide details about properties that are used to define Data Factory entities specific to Microsoft Fabric Lakehouse.

+## Linked service properties

+The Microsoft Fabric Lakehouse connector supports the following authentication types. See the corresponding sections for details:

+- [Service principal authentication](#service-principal-authentication)

+### Service principal authentication

+To use service principal authentication, follow these steps.

+1. Register an application with the Microsoft Identity platform. To learn how, see [Quickstart: Register an application with the Microsoft identity platform](../active-directory/develop/quickstart-register-app.md). Make note of these values, which you use to define the linked service:

+ - Application ID

+ - Application key

+ - Tenant ID

+2. Grant the service principal at least the **Contributor** role in Microsoft Fabric workspace. Follow these steps:

+ 1. Go to your Microsoft Fabric workspace, select **Manage access** on the top bar. Then select **Add people or groups**.

+

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/fabric-workspace-manage-access.png" alt-text="Screenshot shows selecting Fabric workspace Manage access.":::

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/manage-access-pane.png" alt-text=" Screenshot shows Fabric workspace Manage access pane.":::

+

+ 1. In **Add people** pane, enter your service principal name, and select your service principal from the drop-down list.

+

+ 1. Specify the role as **Contributor** or higher (Admin, Member), then select **Add**.

+

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/select-workspace-role.png" alt-text="Screenshot shows adding Fabric workspace role.":::

+ 1. Your service principal is displayed on **Manage access** pane.

+These properties are supported for the linked service:

+| Property | Description | Required |

+|: |: |: |

+| type | The type property must be set to **Lakehouse**. |Yes |

+| workspaceId | The Microsoft Fabric workspace ID. | Yes |

+| artifactId | The Microsoft Fabric Lakehouse object ID. | Yes |

+| tenant | Specify the tenant information (domain name or tenant ID) under which your application resides. Retrieve it by hovering the mouse in the upper-right corner of the Azure portal. | Yes |

+| servicePrincipalId | Specify the application's client ID. | Yes |

+| servicePrincipalCredentialType | The credential type to use for service principal authentication. Allowed values are **ServicePrincipalKey** and **ServicePrincipalCert**. | Yes |

+| servicePrincipalCredential | The service principal credential. <br/> When you use **ServicePrincipalKey** as the credential type, specify the application's key. Mark this field as **SecureString** to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). <br/> When you use **ServicePrincipalCert** as the credential, reference a certificate in Azure Key Vault, and ensure the certificate content type is **PKCS #12**.| Yes |

+| azureCloudType | For service principal authentication, specify the type of Azure cloud environment to which your Azure Active Directory application is registered. <br/> Allowed values are **AzurePublic**, **AzureChina**, **AzureUsGovernment**, and **AzureGermany**. By default, the data factory or Synapse pipeline's cloud environment is used. | No |

+| connectVia | The [integration runtime](concepts-integration-runtime.md) to be used to connect to the data store. You can use the Azure integration runtime or a self-hosted integration runtime if your data store is in a private network. If not specified, the default Azure integration runtime is used. |No |

+**Example: using service principal key authentication**

+You can also store service principal key in Azure Key Vault.

+```json

+{

+ "name": "MicrosoftFabricLakehouseLinkedService",

+ "properties": {

+ "type": "Lakehouse",

+ "typeProperties": {

+ "workspaceId": "<Microsoft Fabric workspace ID>",

+ "artifactId": "<Microsoft Fabric Lakehouse object ID>",

+ "tenant": "<tenant info, e.g. microsoft.onmicrosoft.com>",

+ "servicePrincipalId": "<service principal id>",

+ "servicePrincipalCredentialType": "ServicePrincipalKey",

+ "servicePrincipalCredential": {

+ "type": "SecureString",

+ "value": "<service principal key>"

+ }

+ },

+ "connectVia": {

+ "referenceName": "<name of Integration Runtime>",

+ "type": "IntegrationRuntimeReference"

+ }

+ }

+}

+```

+## Dataset properties

+For a full list of sections and properties available for defining datasets, see the [Datasets](concepts-datasets-linked-services.md) article.

+The following properties are supported for Microsoft Fabric Lakehouse Table dataset:

+| Property | Description | Required |

+| :-- | :-- | :-- |

+| type | The **type** property of the dataset must be set to **LakehouseTable**. | Yes |

+| schema | Name of the schema. |No for source. Yes for sink |

+| table | Name of the table/view. |No for source. Yes for sink |

+### Dataset properties example

+```json

+{

+    "name": "LakehouseTableDataset",

+    "properties": {

+        "type": "LakehouseTable",

+        "linkedServiceName": {

+            "referenceName": "<Microsoft Fabric Lakehouse linked service name>",

+            "type": "LinkedServiceReference"

+        },

+        "typeProperties": {

+ "table": "<table_name>"

+        },

+        "schema": [< physical schema, optional, retrievable during authoring >]

+    }

+}

+```

+## Copy activity properties

+For a full list of sections and properties available for defining activities, see [Copy activity configurations](copy-activity-overview.md#configuration) and [Pipelines and activities](concepts-pipelines-activities.md). This section provides a list of properties supported by the Microsoft Fabric Lakehouse Table source and sink.

+### Microsoft Fabric Lakehouse Table as a source type

+To copy data from Microsoft Fabric Lakehouse Table, set the **type** property in the Copy Activity source to **LakehouseTableSource**. The following properties are supported in the Copy Activity **source** section:

+| Property | Description | Required |

+| : | :-- | :- |

+| type | The **type** property of the Copy Activity source must be set to **LakehouseTableSource**. | Yes |

+| timestampAsOf | The timestamp to query an older snapshot. | No |

+| versionAsOf | The version to query an older snapshot. | No |

+**Example: Microsoft Fabric Lakehouse Table source**

+```json

+"activities":[

+ {

+ "name": "CopyFromLakehouseTable",

+ "type": "Copy",

+ "inputs": [

+ {

+ "referenceName": "<Microsoft Fabric Lakehouse Table input dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "outputs": [

+ {

+ "referenceName": "<output dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "typeProperties": {

+ "source": {

+ "type": "LakehouseTableSource",

+ "timestampAsOf": "2023-09-23T00:00:00.000Z",

+ "versionAsOf": 2

+ },

+ "sink": {

+ "type": "<sink type>"

+ }

+ }

+ }

+]

+```

+### Microsoft Fabric Lakehouse Table as a sink type

+To copy data from Microsoft Fabric Lakehouse Table, set the **type** property in the Copy Activity source to **LakehouseTableSink**. The following properties are supported in the Copy activity **sink** section:

+| Property | Description | Required |

+| : | :-- | :- |

+| type | The **type** property of the Copy Activity source must be set to **LakehouseTableSink**. | Yes |

+| tableActionOption | The way to write data to the sink table. Allowed values are `Append` and `Overwrite`. | No |

+| partitionOption | Allowed values are `None` and `PartitionByKey`. Create partitions in folder structure based on one or multiple columns when the value is `PartitionByKey`. Each distinct column value (pair) will be a new partition (e.g. year=2000/month=01/file). It supports insert-only mode and requires an empty directory in sink. | No |

+| partitionNameList | The destination columns in schemas mapping. Supported data types are string, integer, boolean and datetime. Format respects type conversion settings under "Mapping" tab. | No |

+**Example: Microsoft Fabric Lakehouse Table sink**

+```json

+"activities":[

+ {

+ "name": "CopyToLakehouseTable",

+ "type": "Copy",

+ "inputs": [

+ {

+ "referenceName": "<input dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "outputs": [

+ {

+ "referenceName": "<Microsoft Fabric Lakehouse Table output dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "typeProperties": {

+ "source": {

+ "type": "<source type>"

+ },

+ "sink": {

+ "type": "LakehouseTableSink",

+ "tableActionOption ": "Append"

+ }

+ }

+ }

+]

+```

+## Next steps

+For a list of data stores supported as sources and sinks by the copy activity, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).

+When you configure diagnostic settings and workspace for your ADF on Azure Monitor, selecting the _AllMetrics_ check box will make SSIS operational metrics available for [interactive analysis using Azure metrics explorer](../azure-monitor/essentials/analyze-metrics.md), [presentation on Azure dashboard](../azure-monitor/app/tutorial-app-dashboards.md), and [near-real time alerts](../azure-monitor/alerts/alerts-metric.md).

+Once the credentials required to access the APIs is obtained, you need to call the fetch weather data API [here](/rest/api/data-manager-for-agri/dataplane-version2022-11-01-preview/weather-data) to fetch weather data.

+1. Once you've installed an instance of Azure Data Manager for Agriculture from Azure portal, navigate to Settings ->Solutions tab on the left hand side in your instance. Ensure you have application admin permission.

+ Title: Update a VM for Azure Stack Edge with custom number of cores, memory, and GPU count.

+description: Learn how to update a VM with custom size for Azure Stack Edge.

+# Customer intent: As an IT admin, I need to understand how to update a VM with custom number of cores, memory, and GPU count.

+# Update custom VM size

+This article describes how to modify a VM size with a custom number of cores, memory, and GPU count, which can be used to create a VM image for Azure Stack Edge.

+## Get existing custom VM sizes

+Use the following steps to get custom VM sizes for Azure Stack Edge.

+1. Run the following command to see available VM sizes on your device, including custom sizes:

+## Update custom VM size

+1. Run the following command to update the **Custom VM size** with the `Cores` or `MemoryMb` values for a VM you deploy to your device.

+ In Azure portal, the VM size dropdown will update in about five minutes with the new VM options you just created.

+ - [Create a VM](azure-stack-edge-gpu-virtual-machine-overview.md#create-a-vm).

+You might want to modify the parameters of a rule that has been recommended. For example, you might want to change the recommended IP ranges.

+When necessary, you can delete a recommended rule for the current session. For example, you might determine that applying a suggested rule could block legitimate traffic.

+ - each image pushed or imported to a container registry is scanned after being pushed or imported to a registry. In most cases, the scan is completed within a few minutes, but sometimes it might take up to an hour.

+Azure Container Registries notifies Defender for Cloud when images are deleted, and removes the vulnerability assessment for deleted images within one hour. In some rare cases, Defender for Cloud might not be notified on the deletion, and deletion of associated vulnerabilities in such cases might take up to three days.

+1. Upload a file to that container. Avoid uploading any file that might contain sensitive data.

+Attack path analysis is a graph-based algorithm that scans the cloud security graph. The scans expose exploitable paths that attackers might use to breach your environment to reach your high-impact assets. Attack path analysis exposes attack paths and suggests recommendations as to how best remediate issues that will break the attack path and prevent successful breach.

+When you take your environment's contextual information into account, attack path analysis identifies issues that might lead to a breach on your environment, and helps you to remediate the highest risk ones first. For example its exposure to the internet, permissions, lateral movement, and more.

+ Due to the structure and capabilities of Azure Cosmos DB queries, many known SQL injection attacks canΓÇÖt work in Azure Cosmos DB. However, there are some variations of SQL injections that can succeed and might result in exfiltrating data from your Azure Cosmos DB accounts. Defender for Azure Cosmos DB detects both successful and failed attempts, and helps you harden your environment to prevent these threats.

+- For RDS instances: cross-account KMS encryption is supported, but additional policies on KMS access might prevent access.

+When you enable Defender for Storage Malware Scanning, it might share metadata, including metadata classified as customer data (e.g. SHA-256 hash), with Microsoft Defender for Endpoint.

+To provide security recommendations and investigate potential security threats, Microsoft personnel might access information collected or analyzed by Azure services, including process creation events, and other artifacts, which might unintentionally include customer data or personal data from your machines.

+Depending on the size and structure of your organization, multiple individuals and teams might use Defender for Cloud to perform different security-related tasks. In the following diagram, you have an example of fictitious personas and their respective roles and security responsibilities:

+- Access to the workspace might be required.

+If at some point you want to disable Data Collection, you can turn it off in the security policy. However, because the Log Analytics agent might be used by other Azure management and monitoring services, the agent won't be uninstalled automatically when you turn off data collection in Defender for Cloud. You can manually uninstall the agent if needed.

+This page shows the details regarding the time that the attack took place, the source hostname, the target VM and also gives recommendation steps. In some circumstances, the source information of the attack might be empty. Read [Missing Source Information in Defender for Cloud alerts](/archive/blogs/azuresecurity/missing-source-information-in-azure-security-center-alerts) for more information about this type of behavior.

+Defender for Cloud provides vulnerability assessments for every image pushed or pulled in a registry. Some images might reuse tags from an image that was already scanned. For example, you might reassign the tag ΓÇ£LatestΓÇ¥ every time you add an image to a digest. In such cases, the ΓÇÿoldΓÇÖ image does still exist in the registry and might still be pulled by its digest. If the image has security findings and is pulled, it'll expose security vulnerabilities.

+ - Each image pushed/imported to a container registry is scanned shortly after being pushed to a registry. In most cases, the scan is completed within a few minutes, but sometimes it might take up to an hour.

+| Release state: | Preview<br>The [Azure Preview Supplemental Terms](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) include other legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability. |

+Defender for DevOps allows you to manage your connected environments and provides your security teams with a high level overview of discovered issues that might exist within them through the [Defender for DevOps console](https://portal.azure.com/#view/Microsoft_Azure_Security/SecurityMenuBlade/~/DevOpsSecurity).

+ - If theyΓÇÖre unfamiliar, delete them as they might have been created by a threat actor

+Microsoft Defender for Azure SQL helps you discover and mitigate potential [database vulnerabilities](sql-azure-vulnerability-assessment-overview.md) and alerts you to [anomalous activities](#advanced-threat-protection) that might be an indication of a threat to your databases.

+You can specify the region where your SQL Vulnerability Assessment data will be stored by choosing the Log Analytics workspace location. Microsoft might replicate to other regions for data resiliency, but Microsoft does not replicate data outside the geography.

+The new pricing plan charges based on the number of storage accounts you protect, which simplifies cost calculations and allows for easy scaling as your needs change. You can enable it at the subscription or resource level and can also exclude specific storage accounts from protected subscriptions, providing more granular control over your security coverage. Extra charges might apply to storage accounts with high-volume transactions that exceed a high monthly threshold.

+If the classic Defender for Storage plan is enabled with per-storage account pricing, you can switch to the new plan at either the subscription or resource level. The new Defender for Storage plan has the same pricing plan with the exception of malware scanning which might incur extra charges and is billed per GB scanned.

+| Category | New Defender for Storage plan | Classic (per-transaction plan) | Classic (per-storage account plan) |

+| Pricing structure | Cost is based on the number of storage accounts you protect\*. Add-on costs for GB scanned for malware, if enabled| Cost is based on the number of transactions processed | Cost is based on the number of storage accounts you protect* |

+| Enablement options | Subscription and resource level | Subscription and resource level | Subscription only |

+| Exclusion of storage accounts from protected subscriptions | Yes | Yes | No |

+| Activity monitoring (security alerts) | Yes | Yes | Yes |

+| Malware scanning in uploaded Blobs | Yes (add-on) | No (only hash-reputation analysis) | No (only hash-reputation analysis) |

+| Sensitive data threat detection | Yes (add-on) | No | No |

+| Detection of leaked/compromised SAS tokens (entities without identities) | Yes | No | No |

+\* extra charges might apply to storage accounts with high-volume transactions.

+You might want only certain users, such as a security admin or a SOC analyst, to have permission to access this dedicated container or storage account.

+ - Using [Microsoft Entra ID to control access to blob storage](../storage/blobs/authorize-access-azure-active-directory.md) is considered a best practice. To control access to the dedicated quarantine storage container, you can use [container-level role assignments using Microsoft Entra role-based access control (RBAC)](../storage/blobs/authorize-access-azure-active-directory.md). Users with storage account-level permissions might still be able to access the ΓÇ£quarantineΓÇ¥ container. You can either edit their permissions to be container-level or choose a different approach and move the malicious file to a dedicated storage account.

+- **Detection of entities without identities**: Defender for Storage detects suspicious activities generated by entities without identities that access your data using misconfigured and overly permissive Shared Access Signatures (SAS tokens) that might have leaked or compromised so that you can improve the security hygiene and reduce the risk of unauthorized access. This capability is an expansion of the Activity Monitoring security alerts suite.

+Defender for Storage continuously analyzes data and control plane logs from protected storage accounts when enabled. There's no need to turn on resource logs for security benefits. Use Microsoft Threat Intelligence to identify suspicious signatures such as malicious IP addresses, Tor exit nodes, and potentially dangerous apps. It also builds data models and uses statistical and machine-learning methods to spot baseline activity anomalies, which might indicate malicious behavior. You receive security alerts for suspicious activities, but Defender for Storage ensures you won't get too many similar alerts. Activity monitoring won't affect performance, ingestion capacity, or access to your data.

+The new Microsoft Defender for Storage plan has predictable pricing based on the number of storage accounts you protect. With the option to enable at the subscription or resource level and exclude specific storage accounts from protected subscriptions, you have increased flexibility to manage your security coverage. The pricing plan simplifies the cost calculation process, allowing you to scale easily as your needs change. Other charges might apply to storage accounts with high-volume transactions.

+In addition to security threats, configuration errors might inadvertently expose sensitive resources. Some common misconfiguration issues include:

+description: Prevent passwords and other secrets that might be stored in your code from being accessed by outside individuals by using Defender for Cloud's secret scanning for Defender for DevOps.

+When the scanner runs, it might detect credentials that are false positives. Inline-suppression tools can be used to suppress false positives.

+- Placeholder strings. For example, placeholder strings might be used to initialize a variable, which is then populated using a secret store such as AKV.

+- Self-signed certificates that are used locally and not used as a root. For example, they might be used when running localhost to allow HTTPS.

+You might want to suppress fake secrets in unit tests or mock paths, or inaccurate results. We don't recommend using suppression to suppress test credentials. Test credentials can still pose a security risk and should be securely stored.

+- Learn how to [configure pull request annotations](enable-pull-request-annotations.md) in Defender for Cloud to remediate secrets in code before they're shipped to production.

+**Episode description**: In this episode of Defender for Cloud in the field, Maya Herskovic joins Yuri Diogenes to talk about Microsoft Defender for Containers. Maya explains what's new in Microsoft Defender for Containers, the new capabilities that are available, the new pricing model, and the multicloud coverage. Maya also demonstrates the overall experience of Microsoft Defender for Containers from the recommendations to the alerts that you might receive.

+- Subscribe to [Microsoft Security on YouTube](https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqa0ZoTml2Qm9kZ2pjRzNMUXFqVUwyNl80YVNtd3xBQ3Jtc0trVm9QM2Z0NlpOeC1KSUE2UEd1cVJ5aHQ0MTN6WjJEYmNlOG9rWC1KZ1ZqaTNmcHdOOHMtWXRLSGhUTVBhQlhhYzlUc2xmTHZtaUpkd1c4LUQzLWt1YmRTbkVQVE5EcTJIM0Foc042SGdQZU5acVRJbw&q=https%3A%2F%2Faka.ms%2FSubscribeMicrosoftSecurity)

+- Follow us on social media:

+- Join our [Tech Community](https://aka.ms/SecurityTechCommunity)

+- For more about [Microsoft Security](https://msft.it/6002T9HQY)

+> [Security posture management improvements](episode-four.md)

+> File Integrity Monitoring might create the following account on monitored SQL Servers: `NT Service\HealthService` \

+ - Total number of changes that occurred in the last week (you might see a dash "-ΓÇ£ if FIM isn't enabled on the workspace)

+The FIM registry hive defaults provide a convenient way to monitor recursive changes within common security areas. For example, an adversary might configure a script to execute in LOCAL_SYSTEM context by configuring an execution at startup or shutdown. To monitor changes of this type, enable the built-in check.

+Security teams are responsible for improving the security posture of their organizations but they might not have the resources or authority to actually implement security recommendations. [Assigning owners with due dates](#manually-assigning-owners-and-due-dates-for-recommendation-remediation) and [defining governance rules](#building-an-automated-process-for-improving-security-with-governance-rules) creates accountability and transparency so you can drive the process of improving the security posture in your organization.

+ > An attack path might have more than one path that is at risk. The path count will tell you how many paths need to be remediated. If the attack path has more than one path, you will need to select each path within that attack path to remediate all risks.

+Attack path analysis is a graph-based algorithm that scans the cloud security graph. The scans expose exploitable paths that attackers might use to breach your environment to reach your high-impact assets. Attack path analysis exposes attack paths and suggests recommendations as to how best remediate issues that will break the attack path and prevent successful breach.

+- **Location:** Data collected by Defender for Endpoint is stored in the geo-location of the tenant as identified during provisioning. Customer data - in pseudonymized form - might also be stored in the central storage and processing systems in the United States. After you've configured the location, you can't change it. If you have your own license for Microsoft Defender for Endpoint and need to move your data to another location, [contact Microsoft support](https://portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/overview) to reset the tenant.

+> When you enable automatic deployment, Defender for Endpoint for Linux installation will abort on machines with pre-existing running services using [fanotify](/microsoft-365/security/defender-endpoint/microsoft-defender-endpoint-linux#system-requirements) and other services that can also cause Defender for Endpoint to malfunction or might be affected by Defender for Endpoint, such as security services.

+ - When the Monitoring Agent is installed as an extension, the extension configuration allows reporting to only a single workspace. Defender for Cloud doesn't override existing connections to user workspaces. Defender for Cloud will store security data from the VM in the workspace already connected, if the "Security" or "SecurityCenterFree" solution has been installed on it. Defender for Cloud might upgrade the extension version to the latest version in this process.

+You've now successfully enabled direct onboarding on your tenant. After you enable it for the first time, it might take up to 24 hours to see your non-Azure servers in your designated subscription.

+- **Simultaneous onboarding limited support**: Defender for Cloud makes a best effort to correlate servers onboarded using multiple billing methods. However, in certain server deployment use cases, there might be limitations where Defender for Cloud is unable to correlate your machines. This might result in overcharges on certain devices if direct onboarding is also enabled on your tenant.

+* **Not reported** (gray) - the solution hasn't reported anything yet and no health data is available. A solution's status might be unreported if it was connected recently and is still deploying.

+ > When running the CloudFormation StackSets when onboarding an AWS management account, you might encounter the following error message:

+> - Defender for Containers when deployed on GCP, might incur external costs such as [logging costs](https://cloud.google.com/stackdriver/pricing), [pub/sub costs](https://cloud.google.com/pubsub/pricing) and [egress costs](https://cloud.google.com/vpc/network-pricing#:~:text=Platform%20SKUs%20apply.-%2cInternet%20egress%20rates%2c-Premium%20Tier%20pricing).

+You can use the information in the regulatory compliance dashboard to investigate any issues that might be affecting your compliance posture.

+The regulatory compliance has both automated and manual assessments that might need to be remediated. Using the information in the regulatory compliance dashboard, improve your compliance posture by resolving recommendations directly within the dashboard.

+The regulatory compliance has automated and manual assessments that might need to be remediated. Manual assessments are assessments that require input from the customer to remediate them.

+| October 30 | [Changing adaptive application controlΓÇÖs security alert's severity](#changing-adaptive-application-controls-security-alerts-severity)

+## Changing adaptive application controls security alert's severity

+Announcement date: October 30, 2023

+As part of security alert quality improvement process of Defender for Servers, and as part of the [adaptive application controls](adaptive-application-controls.md) feature, the severity of the following security alert is changing to ΓÇ£InformationalΓÇ¥:

+| Alert [Alert Type] | Alert Description |

+|--|--|

+| Adaptive application control policy violation was audited.[VM_AdaptiveApplicationControlWindowsViolationAudited, VM_AdaptiveApplicationControlWindowsViolationAudited] | The below users ran applications that are violating the application control policy of your organization on this machine. It can possibly expose the machine to malware or application vulnerabilities.|

+To keep viewing this alert in the ΓÇ£Security alertsΓÇ¥ blade in the Microsoft Defender for Cloud portal, change the default view filter **Severity** to include **informational** alerts in the grid.

+ :::image type="content" source="media/release-notes/add-informational-severity.png" alt-text="Screenshot that shows you where to add the informational severity for alerts." lightbox="media/release-notes/add-informational-severity.png":::

+Attack path analysis is a graph-based algorithm that scans your [cloud security graph](concept-attack-path.md#what-is-cloud-security-graph). These scans expose exploitable paths that attackers might use to breach your environment to reach your high-impact assets. Attack path analysis exposes attack paths and suggests recommendations as to how best remediate issues that break the attack path and prevent successful breach.

+Attack path analysis takes into account the contextual information of your environment to identify issues that might compromise it. This analysis helps prioritize the riskiest issues for faster remediation.

+> You can find this dashboard, as well as other tools for working programmatically with secure score, in the dedicated area of the Microsoft Defender for Cloud community on GitHub: <https://github.com/Azure/Azure-Security-Center/tree/master/Secure%20Score>

+- **Secure Score Summary** - provides summarized data regarding your score progress. Use the ΓÇ£Secure score over time per subscriptionΓÇ¥ chart to view changes in the score. If you notice a dramatic change in your score, check the ΓÇ£detected changes that might affect your secure scoreΓÇ¥ table for possible changes that could have caused the change. This table presents deleted resources, newly deployed resources, or resources that their security status changed for one of the recommendations.

+> Each control is calculated every eight hours per subscription or cloud connector. Recommendations within a control are updated more frequently than the control, and so there might be discrepancies between the resources count on the recommendations versus the one found on the control.

+When a finding matches the criteria you've defined in your disable rules, it won't appear in the list of findings. Typical scenarios might include:

+ You might have to tweak `Update-AzSqlServerVulnerabilityAssessmentSetting` according to [Store Vulnerability Assessment scan results in a storage account accessible behind firewalls and VNets](/azure/azure-sql/database/sql-database-vulnerability-assessment-storage).

+Typical scenarios might include:

+When you navigate to Defender for Cloud, you might see a banner that alerts you to the fact that your view is limited. If you see this banner, select it to send a request to the global administrator for your organization. In the request, you can include the role you'd like to be assigned and the global administrator will make a decision about which role to grant.

+The guest agent is the parent process of everything the [Microsoft Antimalware](../security/fundamentals/antimalware.md) extension does. When the guest agent process fails, the Microsoft Antimalware protection that runs as a child process of the guest agent might also fail.

+- Some third-party administration software might disable the guest agent, or block access to certain file locations. If third-party administration software is installed on your VM, make sure that the antimalware agent is on the exclusion list.

+1. When you've reviewed the information on this page, you might have enough to proceed with a response. If you need further details:

+It might be necessary to configure additional parameters for some recommendations.

+Malware scanning might fail to scan a blob. When this happens, the scan result indicates what the error was.

+| SAM259204: "Scan failed - the requested blob wasn't found." | The blob wasn't found. This might be due to deletion, relocation, or renaming after uploading. | N/A | No |

+| SAM259207: "Scan timed out - the requested scan exceeded the `ScanTimeout` minutes time limitation." | The scan timed out before completion. This error might also occur if a preceding step, such as downloading the blob for scanning, takes too long. | This is a transient error and subsequent upload of blobs that failed to be scanned with this error should succeed. | No |

+| [General availability of Containers Vulnerability Assessment powered by Microsoft Defender Vulnerability Management (MDVM) in Defender for Containers and Defender for Container Registries](#general-availability-of-containers-vulnerability-assessment-powered-by-microsoft-defender-vulnerability-management-mdvm-in-defender-for-containers-and-defender-for-container-registries) | October 30, 2023 | November 15, 2023 |

+## General availability of Containers Vulnerability Assessment powered by Microsoft Defender Vulnerability Management (MDVM) in Defender for Containers and Defender for Container Registries

+**Announcement date: October 30, 2023**

+**Estimated date for change: November 15, 2023**

+Vulnerability assessment (VA) for Linux container images in Azure container registries powered by Microsoft Defender Vulnerability Management (MDVM) will soon be released for General Availability (GA) in Defender for Containers and Defender for Container Registries.

+As part of this change, the following recommendations will also be released for GA and renamed:

+|Current recommendation name|New recommendation name|Description|Assessment key|

+|--|--|--|--|

+|Container registry images should have vulnerability findings resolved (powered by Microsoft Defender Vulnerability Management)|Azure registry container images should have vulnerabilities resolved (powered by Microsoft Defender Vulnerability Management)|Container image vulnerability assessments scans your registry for commonly known vulnerabilities (CVEs) and provides a detailed vulnerability report for each image. Resolving vulnerabilities can greatly improve your security posture, ensuring images are safe to use prior to deployment. |c0b7cfc6-3172-465a-b378-53c7ff2cc0d5|

+|Running container images should have vulnerability findings resolved (powered by Microsoft Defender Vulnerability Management)|Azure running container images should have vulnerabilities resolved (powered by Microsoft Defender Vulnerability Management|Container image vulnerability assessment scans your registry for commonly known vulnerabilities (CVEs) and provides a detailed vulnerability report for each image. This recommendation provides visibility to vulnerable images currently running in your Kubernetes clusters. Remediating vulnerabilities in container images that are currently running is key to improving your security posture, significantly reducing the attack surface for your containerized workloads.|c609cf0f-71ab-41e9-a3c6-9a1f7fe1b8d5|

+Once the recommendations are released for GA, they will be included in the secure score calculation, and will also incur charges as per [plan pricing](https://azure.microsoft.com/pricing/details/defender-for-cloud/?v=17.23h#pricing).

+> [!NOTE]

+> Images scanned both by our container VA offering powered by Qualys and Container VA offering powered by MDVM, will only be billed once.

+**Announcement date: October 26, 2023**

+| SOC 2 Type 2 | AWS Foundational Security Best Practices | NIST 800-53 |

+> It might take a few hours for a newly added standard to appear in the compliance dashboard.

+ 1. Push the updated image to trigger a scan and delete the old image. It might take up to 24 hours for the previous image to be removed from the results, and for the new image to be included in the results.

+ 1. Push the updated image to trigger a scan and delete the old image. It might take up to 24 hours for the previous image to be removed from the results, and for the new image to be included in the results.

+ > It might take some time after onboarding for recommendations to appear. In fact, depending on on your server activity you might not receive *any* alerts. To generate test alerts to test your alerts are working correctly, follow the instructions in [the alert validation procedure](alert-validation.md).

+|Required roles and permissions:|**Security admin role** or **Owner** on the resource group<br>Must also have write permissions for the target resource<br><br>To work with Azure Logic Apps workflows, you must also have the following Logic Apps roles/permissions:<br> - [Logic App Operator](../role-based-access-control/built-in-roles.md#logic-app-operator) permissions are required or Logic App read/trigger access (this role can't create or edit logic apps; only *run* existing ones)<br> - [Logic App Contributor](../role-based-access-control/built-in-roles.md#logic-app-contributor) permissions are required for logic app creation and modification<br>If you want to use Logic Apps connectors, you might need other credentials to sign in to their respective services (for example, your Outlook/Teams/Slack instances)|

+You might be charged for storing data in Log Analytics. For more information, see the [pricing page](https://azure.microsoft.com/pricing/details/defender-for-cloud/).

+You can analyze metrics for Azure Event Hubs, along with metrics from other Azure services, by selecting **Metrics** from the **Azure Monitor** section on the home page for your Event Hubs namespace. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool. For a list of the platform metrics collected, see [Monitoring Azure Event Hubs data reference metrics](monitor-event-hubs-reference.md#metrics).

+You can analyze metrics for *Azure ExpressRoute* with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+# required metadata

+ Title: Security Copilot (preview) and Defender EASM

+description: You can use Security Copilot to get information about your EASM data.

+ms.localizationpriority: high

+# Microsoft Security Copilot (preview) and Defender EASM

+> [!IMPORTANT]

+> The information in this article applies to the Microsoft Security Copilot Early Access Program, which is an invite-only paid preview program. Some information in this article relates to prereleased product, which may be substantially modified before it's commercially released. Microsoft makes no warranties, express or implied, with respect to the information provided in this article.

+Security Copilot is a cloud-based AI platform that provides a natural language copilot experience. It can help support security professionals in different scenarios, like incident response, threat hunting, and intelligence gathering. For more information about what it can do, go to [What is Microsoft Security Copilot?](/security-copilot/microsoft-security-copilot).

+**Security Copilot integrates with Defender EASM**.

+Security Copilot can surface insights from Defender EASM about an organization's attack surface. You can use the system features built into Security Copilot, and use prompts to get more information. This information can help you understand your security posture and mitigate vulnerabilities.

+This article introduces you to Security Copilot and includes sample prompts that can help Defender EASM users.

+## Know before you begin

+- Ensure that you reference the company name in your first prompt. Unless otherwise specified, all future prompts will provide data about the initially specified company.

+- Be clear and specific with your prompts. You might get better results if you include specific asset names or metadata values (e.g. CVE IDs) in your prompts.

+ It might also help to add **Defender EASM** to your prompt, like:

+ - **According to Defender EASM, what are my expired domains?**

+ - **Tell me about Defender EASM high priority attack surface insights.**

+- Experiment with different prompts and variations to see what works best for your use case. Chat AI models vary, so iterate and refine your prompts based on the results you receive.

+- Security Copilot saves your prompt sessions. To see the previous sessions, in Security Copilot, go to the menu > **My investigations**:

+ 

+ For a walkthrough on Security Copilot, including the pin and share feature, go to [Navigating Microsoft Security Copilot](/security-copilot/navigating-security-copilot).

+For more information on writing Security Copilot prompts, go to [Microsoft Security Copilot prompting tips](/security-copilot/prompting-tips).

+## Open Security Copilot

+1. Go to [Microsoft Security Copilot](https://go.microsoft.com/fwlink/?linkid=2247989) and sign in with your credentials.

+2. By default, Defender EASM should be enabled. To confirm, select **plugins** (bottom left corner):

+ 

+ In **My plugins**, confirm Defender EASM is on. Close **Plugins**.

+ > [!NOTE]

+ > Some roles can enable or disable plugins, like Defender EASM. For more information, go to [Manage plugins in Microsoft Security Copilot](/security-copilot/manage-plugins).

+3. Enter your prompt.

+## Built-in system features

+In Security Copilot, there are built in system features. These features can get data from the different plugins that are enabled.

+To view the list of built-in system capabilities for Defender EASM, use the following steps:

+1. In the prompt, enter **/**.

+2. Select **See all system capabilities**.

+3. In the Defender EASM section, you can:

+ - Get attack surface summary.

+ - Get attack surface insights.

+ - Get assets affected by CVEs by priority or CVE ID.

+ - Get assets by CVSS score.

+ - Get expired domains.

+ - Get expired SSL certificates.

+ - Get SHA1 certificates.

+## Sample prompts for Defender EASM?

+There are many prompts you can use to get information about your Defender EASM data. This section lists some ideas and examples.

+### General information about your attack surface

+Get **general information** about your Defender EASM data, like an attack surface summary or insights about your inventory.

+**Sample prompts**:

+- Get the external attack surface for my organization.

+- What are the high priority attack surface insights for my organization?

+### CVE vulnerability data

+Get details on **CVEs that are applicable to your inventory**.

+**Sample prompts**:

+- Is my external attack surface impacted by CVE-2023-21709?

+- Get assets affected by high priority CVSS's in my attack surface.

+- How many assets have critical CVSS's for my organization?

+### Domain and SSL certificate posture

+Get information about **domain and SSL certificate posture**, like expired domains and usage of SHA1 certificates.

+**Sample prompts**:

+- How many domains are expired in my organization's attack surface?

+- How many SSL certificates are expired for my organization?

+- How many assets are using SSL SHA1 for my organization?

+- Get list of expired SSL certificates.

+## Provide feedback

+Your feedback on the Defender EASM integration with Security Copilot helps with development. To provide feedback, in Security Copilot, use the feedback buttons at the bottom of each completed prompt. Your options are "Looks Right," "Needs Improvement" and "Inappropriate."

+Your options:

+- **Confirm**: The results match expectations.

+- **Off-target**: The results don't match expectations.

+- **Report**: The results are harmful in some way.

+Whenever possible, and when the result is **Off-target**, write a few words explaining what can be done to improve the outcome. If you entered Defender EASM-specific prompts and the results aren't EASM related, then include that information.

+## Data processing and privacy

+When you interact with the Security Copilot to get Defender EASM data, Security Copilot pulls that data from Defender EASM. The prompts, the data that's retrieved, and the output shown in the prompt results is processed and stored within the Security Copilot service.

+For more information about data privacy in Security Copilot, go to [Privacy and data security in Microsoft Security Copilot](/security-copilot/privacy-data-security).

+## Related articles

+- [What is Microsoft Security Copilot?](/security-copilot/microsoft-security-copilot)

+- [Privacy and data security in Microsoft Security Copilot](/security-copilot/privacy-data-security)

+> Change ssl_keystore_file_path depends on the java cert location. Apache Flink cluster on HDInsight on AKS, the path is `/usr/lib/jvm/msopenjdk-11-jre/lib/security`

+This example demonstrates on how to use Apache Flink 1.16.0 on HDInsight on AKS along with your existing MongoDB as Sink and Source with Flink DataStream API MongoDB connector.

+Apache Flink clusters in HDInsight on AKS include the following components, they are available on the clusters by default.

+This example illustrates various snippets of connecting to hive, using Apache Flink on HDInsight on AKS, you're required to use `/opt/hive-conf` as hive configuration directory to connect with Hive metastore

+ |Name |Optional. Enter a name such as HDInsight on AKS Private Preview to easily identify all resources associated with your resources|

+1. The **Deployment is in process** page is displayed which the cluster is created. It takes 5-10 minutes to create the cluster. Once the cluster is created, **Your deployment is complete** message is displayed. If you navigate away from the page, you can check your Notifications for the status.

+All product and service names used in these pages are for identification purposes only and do not imply endorsement.

+ Title: Export DICOM files by using the export API of the DICOM service

+description: This how-to guide explains how to export DICOM files to an Azure Blob Storage account.

+# Export DICOM files

+The DICOM&reg; service provides the ability to easily export DICOM data in a file format. The service simplifies the process of using medical imaging in external workflows, such as AI and machine learning. You can use the export API to export DICOM studies, series, and instances in bulk to an [Azure Blob Storage account](../../storage/blobs/storage-blobs-introduction.md). DICOM data that's exported to a storage account is exported as a `.dcm` file in a folder structure that organizes instances by `StudyInstanceID` and `SeriesInstanceID`.

+There are three steps to exporting data from the DICOM service:

+- Enable a system-assigned managed identity for the DICOM service.

+- Configure a new or existing storage account and give permission to the system-assigned managed identity.

+- Use the export API to create a new export job to export the data.

+## Enable managed identity for the DICOM service

+The first step to export data from the DICOM service is to enable a system-assigned managed identity. This managed identity is used to authenticate the DICOM service and give permission to the storage account used as the destination for export. For more information about managed identities in Azure, see [About managed identities for Azure resources](../../active-directory/managed-identities-azure-resources/overview.md).

+1. In the Azure portal, browse to the DICOM service that you want to export from and select **Identity**.

+ :::image type="content" source="media/dicom-export-identity.png" alt-text="Screenshot that shows selection of Identity view." lightbox="media/dicom-export-identity.png":::

+1. Set the **Status** option to **On**, and then select **Save**.

+ :::image type="content" source="media/dicom-export-enable-system-identity.png" alt-text="Screenshot that shows the system-assigned identity toggle." lightbox="media/dicom-export-enable-system-identity.png":::

+1. Select **Yes** in the confirmation dialog that appears.

+ :::image type="content" source="media/dicom-export-confirm-enable.png" alt-text="Screenshot that shows the dialog confirming enabling system identity." lightbox="media/dicom-export-confirm-enable.png":::

+It takes a few minutes to create the system-assigned managed identity. After the system identity is enabled, an **Object (principal) ID** appears.

+## Assign storage account permissions

+The system-assigned managed identity needs **Storage Blob Data Contributor** permission to write data to the destination storage account.

+1. Under **Permissions**, select **Azure role assignments**.

+ :::image type="content" source="media/dicom-export-azure-role-assignments.png" alt-text="Screenshot that shows the Azure role assignments button on the Identity view." lightbox="media/dicom-export-azure-role-assignments.png":::

+1. Select **Add role assignment**. On the **Add role assignment** pane, make the following selections:

+ * Under **Scope**, select **Storage**.

+ * Under **Resource**, select the destination storage account for the export operation.

+ * Under **Role**, select **Storage Blob Data Contributor**.

+ :::image type="content" source="media/dicom-export-add-role-assignment.png" alt-text="Screenshot that shows the Add role assignment pane." lightbox="media/dicom-export-add-role-assignment.png":::

+1. Select **Save** to add the permission to the system-assigned managed identity.

+## Use the export API

+The export API exposes one `POST` endpoint for exporting data.

+```

+POST <dicom-service-url>/<version>/export

+```

+Given a *source*, the set of data to be exported, and a *destination*, the location to which data will be exported, the endpoint returns a reference to a new, long-running export operation. The duration of this operation depends on the volume of data to be exported. For more information about monitoring progress of export operations, see the [Operation status](#operation-status) section.

+Any errors encountered while you attempt to export are recorded in an error log. For more information, see the [Errors](#errors) section.

+#### Request

+The request body consists of the export source and destination.

+```json

+{

+ "source": {

+ "type": "identifiers",

+ "settings": {

+ "values": [

+ "..."

+ ]

+ }

+ },

+ "destination": {

+ "type": "azureblob",

+ "settings": {

+ "setting": "<value>"

+ }

+ }

+}

+```

+#### Source settings

+The only setting is the list of identifiers to export.

+| Property | Required | Default | Description |

+| :- | :- | : | :- |

+| `Values` | Yes | | A list of one or more DICOM studies, series, and/or SOP instance identifiers in the format of `"<StudyInstanceUID>[/<SeriesInstanceUID>[/<SOPInstanceUID>]]"` |

+#### Destination settings

+The connection to the Blob Storage account is specified with `BlobContainerUri`.

+| Property | Required | Default | Description |

+| :- | :- | : | :- |

+| `BlobContainerUri` | No | `""` | The complete URI for the blob container |

+| `UseManagedIdentity` | Yes | `false` | A required flag that indicates whether managed identity should be used to authenticate to the blob container |

+#### Example

+The following example requests the export of the following DICOM resources to the blob container named `export` in the storage account named `dicomexport`:

+- All instances within the study whose `StudyInstanceUID` is `1.2.3`

+- All instances within the series whose `StudyInstanceUID` is `12.3` and `SeriesInstanceUID` is `4.5.678`

+- The instance whose `StudyInstanceUID` is `123.456`, `SeriesInstanceUID` is `7.8`, and `SOPInstanceUID` is `9.1011.12`

+```http

+POST /export HTTP/1.1

+Accept: */*

+Content-Type: application/json

+{

+ "sources": {

+ "type": "identifiers",

+ "settings": {

+ "values": [

+ "1.2.3",

+ "12.3/4.5.678",

+ "123.456/7.8/9.1011.12"

+ ]

+ }

+ },

+ "destination": {

+ "type": "azureblob",

+ "settings": {

+ "blobContainerUri": "https://dicomexport.blob.core.windows.net/export",

+ "UseManagedIdentity": true

+ }

+ }

+}

+```

+#### Response

+The export API returns a `202` status code when an export operation is started successfully. The body of the response contains a reference to the operation, while the value of the `Location` header is the URL for the export operation's status (the same as `href` in the body).

+Inside the destination container, use the path format `<operation id>/results/<study>/<series>/<sop instance>.dcm` to find the DCM files.

+```http

+HTTP/1.1 202 Accepted

+Content-Type: application/json

+{

+ "id": "df1ff476b83a4a3eaf11b1eac2e5ac56",

+ "href": "https://example-dicom.dicom.azurehealthcareapis.com/v1/operations/df1ff476b83a4a3eaf11b1eac2e5ac56"

+}

+```

+#### Operation status

+Poll the preceding `href` URL for the current status of the export operation until completion. After the job has reached a terminal state, the API returns a 200 status code instead of 202. The value of its status property is updated accordingly.

+```http

+HTTP/1.1 200 OK

+Content-Type: application/json

+{

+ "operationId": "df1ff476b83a4a3eaf11b1eac2e5ac56",

+ "type": "export",

+ "createdTime": "2022-09-08T16:40:36.2627618Z",

+ "lastUpdatedTime": "2022-09-08T16:41:01.2776644Z",

+ "status": "completed",

+ "results": {

+ "errorHref": "https://dicomexport.blob.core.windows.net/export/4853cda8c05c44e497d2bc071f8e92c4/errors.log",

+ "exported": 1000,

+ "skipped": 3

+ }

+}

+```

+## Errors

+If there are any user errors when you export a DICOM file, the file is skipped and its corresponding error is logged. This error log is also exported alongside the DICOM files and the caller can review it. You can find the error log at `<export blob container uri>/<operation ID>/errors.log`.

+#### Format

+Each line of the error log is a JSON object with the following properties. A given error identifier might appear multiple times in the log as each update to the log is processed *at least once*.

+| Property | Description |

+| | -- |

+| `Timestamp` | The date and time when the error occurred |

+| `Identifier` | The identifier for the DICOM study, series, or SOP instance in the format of `"<study instance UID>[/<series instance UID>[/<SOP instance UID>]]"` |

+| `Error` | The detailed error message |

+ > To learn more about advanced metrics display and sharing options, see [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md).

+ > To learn more about advanced metrics display and sharing options, see [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md).

+You can analyze metrics for DPS with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+RuntimeLogLevel = "debug"

+UpstreamProtocol = "AmqpWs"

+storageFolder = "/iotedge/storage"

+You can analyze metrics for Azure IoT Hub with metrics from other Azure services using metrics explorer. For more information on this tool, see [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md).

+You can analyze metrics for Key Vault with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+You can analyze metrics for Load Balancer with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+## Install the 'AzureBasicLoadBalancerUpgrade' module

+- **PowerShell**: A supported version of PowerShell version 7 or higher is recommended for use with the AzureBasicLoadBalancerUpgrade module on all platforms including Windows, Linux, and macOS. However, PowerShell 5.1 on Windows is supported.

+### Module Installation

+## Pre- and Post-migration Steps

+### Pre-migration steps

+- [Validate](#example-validate-a-scenario) that your scenario is supported

+- Plan for [application downtime](#how-long-does-the-upgrade-take) during migration

+- Develop inbound and outbound connectivity tests for your traffic

+- Plan for instance-level Public IP changes on Virtual Machine Scale Set instances (see note above)

+- [Recommended] Create Network Security Groups or add security rules to an existing Network Security Group for your backend pool members, allowing the traffic through the Load Balancer and any other traffic which will need to be explicitly allowed on public Standard SKU resources

+- [Recommended] Prepare your [outbound connectivity](../virtual-network/ip-services/default-outbound-access.md), taking one of the following approaches:

+ - Add a NAT Gateway to your backend member's subnets

+ - Add Public IP addresses to each backend Virtual Machine or [Virtual Machine Scale Set instance](../virtual-machine-scale-sets/virtual-machine-scale-sets-networking.md#public-ipv4-per-virtual-machine)

+ - Plan to create [Outbound Rules](./outbound-rules.md) for Public Load Balancers with multiple backend pools post-migration

+### Post-migration steps

+- [Validate that your migration was successful](#example-validate-completed-migration)

+- Test inbound application connectivity through the Load Balancer

+- Test outbound connectivity from backend pool members to the Internet

+- For Public Load Balancers with multiple backend pools, create [Outbound Rules](./outbound-rules.md) for each backend pool

+### Example: validate a scenario

+Validate that a Basic Load Balancer is supported for upgrade

+```powershell

+PS C:\> Start-AzBasicLoadBalancerUpgrade -ResourceGroupName <loadBalancerRGName> -BasicLoadBalancerName <basicLBName> -validateScenarioOnly

+```

+### Example: upgrade by name

+Upgrade a Basic Load Balancer to a Standard Load Balancer with the same name, providing the Basic Load Balancer name and resource group name

+PS C:\> Start-AzBasicLoadBalancerUpgrade -ResourceGroupName <loadBalancerRGName> -BasicLoadBalancerName <basicLBName>

+### Example: upgrade, change name, and show logs

+Upgrade a Basic Load Balancer to a Standard Load Balancer with the specified name, displaying logged output on screen

+PS C:\> Start-AzBasicLoadBalancerUpgrade -ResourceGroupName <loadBalancerRGName> -BasicLoadBalancerName <basicLBName> -StandardLoadBalancerName <newStandardLBName> -FollowLog

+### Example: upgrade with alternate backup path

+Upgrade a Basic Load Balancer to a Standard Load Balancer with the specified name and store the Basic Load Balancer backup file at the specified path

+PS C:\> Start-AzBasicLoadBalancerUpgrade -ResourceGroupName <loadBalancerRGName> -BasicLoadBalancerName <basicLBName> -StandardLoadBalancerName <newStandardLBName> -RecoveryBackupPath C:\BasicLBRecovery

+### Example: validate completed migration

+Validate a completed migration by passing the Basic Load Balancer state file backup and the Standard Load Balancer name

+### Example: migrate multiple, related Load Balancers

+Migrate multiple Load Balancers with shared backend members at the same time, usually when an application has an internal and an external Load Balancer

+ 'standardLoadBalancerName' = 'myStandardInternalLB01' # specifying the standard load balancer name is optional

+ 'standardLoadBalancerName' = 'myStandardExternalLB02'

+# pass the array of load balancer configurations to the -MultiLBConfig parameter

+### Example: retry failed virtual machine scale set migration

+Retry a failed upgrade for a virtual machine scale set's load balancer (due to error or script termination) by providing the Basic Load Balancer and Virtual Machine Scale Set backup state file

+### Example: retry failed virtual machine migration

+Retry a failed upgrade for a VM load balancer (due to error or script termination) by providing the Basic Load Balancer backup state file

+In a scenario where your backend pool members are also members of backend pools on another Load Balancer, such as when you have internal and external Load Balancers for the same application, the Basic Load Balancers need to be migrated at the same time. Trying to migrate the Load Balancers one at a time would attempt to mix Basic and Standard SKU resources, which is not allowed. The migration script supports this by passing multiple Basic Load Balancers into the same [script execution using the `-MultiLBConfig` parameter](#example-migrate-multiple-related-load-balancers).

+| Triggers per workflow | - Consumption (designer): 1 trigger <br>- Consumption (JSON): 10 triggers <br><br>- Standard: 1 trigger | - Consumption: Multiple triggers are possible only when you work on the JSON workflow definition, whether in code view or an Azure Resource Manager (ARM) template, not the designer. <br><br>- Standard: Only one trigger is possible, whether in the designer, code view, or an Azure Resource Manager (ARM) template. |

+ * A Standard logic app workflow can have only one trigger and doesn't support multiple triggers.

+You can analyze metrics for Azure Machine Learning, along with metrics from other Azure services, by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+# What's new in Azure Managed Grafana

+## October 2023

+* Azure Managed Grafana has a new [Essential pricing plan](overview.md#service-tiers) available in preview. This plan provides core Grafana functionalities at a reduced price and is designed to be used in non-production environments.

+## September 2023

+* [Microsoft Entra groups](how-to-sync-teams-with-azure-ad-groups.md) is available in preview in Azure Managed Grafana.

+* Plugin management is available in preview. This feature lets you manage installed Grafana plugins directly within an Azure Managed Grafana workspace.

+* Azure Monitor workspaces integration is available in preview. This feature allows you to link your Grafana dashboard to Azure Monitor workspaces. This integration simplifies the process of connecting AKS clusters to an Azure Managed Grafana workspace and collecting metrics.

+## May 2023

+* Connecting Azure Managed Grafana instances to data sources using managed private endpoints is available in preview. For more information about managed private endpoints, go to [Connect to a data source privately](how-to-connect-to-data-source-privately.md).

+* SMTP support in Azure Managed Grafana is now generally available. For more information, go to [Configure SMTP settings](how-to-smtp-settings.md).

+* Reporting is now supported in Azure Managed Grafana as a preview. For more information, go to [Use reporting and image rendering](how-to-use-reporting-and-image-rendering.md).

+* Configuring SMTP settings for Azure Managed Grafana is now supported. For more information, go to [SMTP settings](how-to-smtp-settings.md).

+* Grafana Enterprise is supported. For more information, go to [Subscribe to Grafana Enterprise](how-to-grafana-enterprise.md).

+* Service accounts are supported. For more information, go to [How to use service accounts](how-to-service-accounts.md).

+The following table lists the main features supported in each tier:

+> If you have deployed an appliance using a template (OVA for servers on a VMware environment and VHD for a Hyper-V environment), you can use the same appliance and register it with an Azure Migrate project with private endpoint connectivity. You will need to run the Azure Migrate installer script and select the private endpoint connectivity option mentioned in the instructions below.

+## Enabling DNS Resolution to Private Endpoints

+1. The DNS records required for the private endpoints can be downloaded from the Azure Migrate project. Instructions on how to download the DNS entries is [here](./troubleshoot-network-connectivity.md#verify-dns-resolution)

+2. Add these DNS records to your DNS server on-premises using our [Private Endpoint Connectivity documentation](../private-link/private-endpoint-dns.md) or add these DNS records to the local host file in the Azure Migrate appliance.

+- [Migrate servers to Azure using Private Link](migrate-servers-to-azure-using-private-link.md).

+Data-out replication allows you to synchronize data out of an Azure Database for MySQL flexible server to another MySQL server using MySQL native replication. The MySQL server (replica) can be on-premises, in virtual machines, or a database service hosted by other cloud providers. While [Data-in replication](concepts-data-in-replication.md) helps to move data into an Azure Database for MySQL flexible server (replica), Data-out replication would allow you to transfer data out of an Azure Database for MySQL flexible server (Primary). With Data-out replication, the binary log (binlog) is made community consumable allowing the an Azure Database for MySQL flexible server to act as a Primary server for the external replicas. To learn more about binlog replication, see the [MySQL binlog replication overview](https://dev.mysql.com/doc/refman/5.7/en/binlog-replication-configuration-overview.html).

+## Reschedule Maintenance (Public Preview)

+1. In the Maintenance window, you'll notice a new button labeled Reschedule.

+2. Upon clicking **Reschedule**, a "Maintenance Reschedule" window appears where you can select a new date and time for the scheduled maintenance activity.

+3. After selecting your preferred date and time, select **Reschedule** to confirm your choice.

+4. You also have an option for on-demand maintenance by clicking **Reschedule to Now**. A confirmation dialog appears to verify that you understand the potential effect, including possible server downtime.

+Rescheduling maintenance also triggers email notifications to keep you informed.

+The availability of the rescheduling window isn't fixed, it often depends on the size of the overall maintenance window for the region in which your server resides. This means the reschedule options can vary based on regional operations and workload.

+> [!NOTE]

+> Maintenance Reschedule is only available for General Purpose and Business Critical service tiers.

+### Considerations and limitations

+Be aware of the following when using this feature:

+- **Demand Constraints:** Your rescheduled maintenance might be canceled due to a high number of maintenance activities occurring simultaneously in the same region.

+- **Lock-in Period:** Rescheduling is unavailable 15 minutes prior to the initially scheduled maintenance time to maintain the reliability of the service.

+| [Create a server and enable public access connectivity](scripts/sample-cli-create-connect-public-access.md) | Creates an Azure Database for MySQL - Flexible Server, configures a server-level firewall rule (public access connectivity method) and connects to the server. |

+Create an Azure Database for MySQL with the az mysql server create command. Remember that the name of your MySQL Server must be unique across Azure, so replace the placeholder value in brackets with your own unique value:

+> [!IMPORTANT]

+> As of July 1, 2021, you can no longer add new tests in an existing workspace or enable a new workspace in Network Performance Monitor (NPM). You're also no longer able to add new connection monitors in Connection Monitor (Classic). You can continue to use the tests and connection monitors that you've created prior to July 1, 2021.

+>

+> To minimize service disruption to your current workloads, [migrate your tests from Network Performance Monitor](/azure/network-watcher/migrate-to-connection-monitor-from-network-performance-monitor), or [migrate from Connection Monitor (Classic)](/azure/network-watcher/migrate-to-connection-monitor-from-connection-monitor-classic) to the new Connection Monitor in Azure Network Watcher before February 29, 2024.

+See **[Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md)** for details on using this tool.

+az networkfabric l3domain create --resource-group "ResourceGroupName" --resource-name "l3untrust" --location "eastus" --nf-id "/subscriptions/xxxxxx-xxxxxx-xxxx-xxxx-xxxxxx/resourceGroups/NFResourceGroupName/providers/Microsoft.ManagedNetworkFabric/networkFabrics/NFName"

+az networkfabric l3domain create --resource-group "ResourceGroupName" --resource-name "l3trust" --location "eastus" --nf-id "/subscriptions/xxxxxx-xxxxxx-xxxx-xxxx-xxxxxx/resourceGroups/NFResourceGroupName/providers/Microsoft.ManagedNetworkFabric/networkFabrics/NFName"

+az networkfabric l3domain create --resource-group "ResourceGroupName" --resource-name "l3mgmt" --location "eastus" --nf-id "/subscriptions/xxxxxx-xxxxxx-xxxx-xxxx-xxxxxx/resourceGroups/NFResourceGroupName/providers/Microsoft.ManagedNetworkFabric/networkFabrics/NFName"

+az networkfabric l3domain update-admin-state --resource-group "ResourceGroupName" --resource-name "example-l3domain" --state Enable/Disable

+ az networkfabric l3domain delete --resource-group "ResourceGroupName" --resource-name "example-l3domain"

+az networkfabric internalnetwork create --resource-group "ResourceGroupName" --l3-isolation-domain-name l3untrust --resource-name untrustnetwork --location "eastus" --vlan-id 502 --fabric-asn 65048 --peer-asn 65047--connected-i-pv4-subnets prefix="10.151.3.11/24" --mtu 1500

+az networkfabric internalnetwork create --resource-group "ResourceGroupName" --l3-isolation-domain-name l3trust --resource-name trustnetwork --location "eastus" --vlan-id 503 --fabric-asn 65048 --peer-asn 65047--connected-i-pv4-subnets prefix="10.151.1.11/24" --mtu 1500

+az networkfabric internalnetwork create --resource-group "ResourceGroupName" --l3-isolation-domain-name l3mgmt --resource-name mgmtnetwork --location "eastus" --vlan-id 504 --fabric-asn 65048 --peer-asn 65047--connected-i-pv4-subnets prefix="10.151.2.11/24" --mtu 1500

+az networkfabric l2domain update-administrative-state --resource-group "ResourceGroupName" --resource-name "l2HAnetwork" --state Enable

+az networkfabric l3domain update-admin-state --resource-group "ResourceGroupName" --resource-name "l3untrust" --state Enable

+az networkfabric l3domain update-admin-state --resource-group "ResourceGroupName" --resource-name "l3trust" --state Enable

+az networkfabric l3domain update-admin-state --resource-group "ResourceGroupName" --resource-name "l3mgmt" --state Enable

+| 1.25.4 | 1 | Calico v3.24.0<br>metrics-server v0.6.3<br>Multus v3.8.0<br>CoreDNS v1.8.4<br>etcd v3.5.6-5<br>sriov-dp v3.5.1 | Mariner 2.0 (2023-05-04) | No breaking changes | |

+| 1.25.4 | 2 | Calico v3.24.0<br>metrics-server v0.6.3<br>Multus v3.8.0<br>CoreDNS v1.8.4<br>etcd v3.5.6-5<br>sriov-dp v3.5.1 | Mariner 2.0 (2023-06-18) | No breaking changes | |

+| 1.25.4 | 3 | Calico v3.24.0<br>metrics-server v0.6.3<br>Multus v3.8.0<br>CoreDNS v1.8.4<br>etcd v3.5.6-5<br>sriov-dp v3.5.1 | Mariner 2.0 (2023-06-18) | No breaking changes | |

+| 1.25.4 | 4 | Calico v3.24.0<br>metrics-server v0.6.3<br>Multus v3.8.0<br>CoreDNS v1.8.4<br>etcd v3.5.6-5<br>sriov-dp v3.5.1 | Mariner 2.0 (2023-09-21) | No breaking changes | |

+| 1.25.6 | 1 | Calico v3.24.0<br>metrics-server v0.6.3<br>Multus v3.8.0<br>CoreDNS v1.8.6<br>etcd v3.5.6-5<br>sriov-dp v3.5.1 | Mariner 2.0 (2023-09-21) | No breaking changes | |

+| 1.26.3 | 1 | Calico v3.24.0<br>metrics-server v0.6.3<br>Multus v3.8.0<br>CoreDNS v1.8.6<br>etcd v3.5.6-5<br>sriov-dp v3.5.1 | Mariner 2.0 (2023-09-21) | No breaking changes | |

+| 1.27.1 | 1 | Calico v3.24.0<br>metrics-server v0.6.3<br>Multus v3.8.0<br>CoreDNS v1.9.3<br>etcd v3.5.6-5<br>sriov-dp v3.5.1 | Mariner 2.0 (2023-09-21) | Cgroupv2 | Steps to disable cgroupv2 can be found [here](./howto-disable-cgroupsv2.md) |

+This article provides an overview of the autovacuum feature for [Azure Database for PostgreSQL - Flexible Server](overview.md) and the feature troubleshooting guides that are available to monitor the database bloat, autovacuum blockers and also information around how far the database is from emergency or wraparound situation.

+## What is autovacuum

+Internal data consistency in PostgreSQL is based on the Multi-Version Concurrency Control (MVCC) mechanism, which allows the database engine to maintain multiple versions of a row and provides greater concurrency with minimal blocking between the different processes.

+PostgreSQL databases need appropriate maintenance. For example, when a row is deleted, it isn't removed physically. Instead, the row is marked as "dead". Similarly for updates, the row is marked as "dead" and a new version of the row is inserted. These operations leave behind dead records, called dead tuples, even after all the transactions that might see those versions finish. Unless cleaned up, dead tuples remain, consuming disk space and bloating tables and indexes which result in slow query performance.

+PostgreSQL uses a process called autovacuum to automatically clean-up dead tuples.

+The amount of work autovacuum does depends on two parameters:

+That means in one-second autovacuum can do:

+## Monitor autovacuum

+Use the following queries to monitor autovacuum:

+- **Last_autovacuum**: The date of the last time the table was autovacuumed.

+- **Last_autoanalyze**: The date of the last time the table was automatically analyzed.

+## When does PostgreSQL trigger autovacuum

+An autovacuum action (either *ANALYZE* or *VACUUM*) triggers when the number of dead tuples exceeds a particular number that is dependent on two factors: the total count of rows in a table, plus a fixed threshold. *ANALYZE*, by default, triggers when 10% of the table plus 50 rows changes, while *VACUUM* triggers when 20% of the table plus 50 rows changes. Since the *VACUUM* threshold is twice as high as the *ANALYZE* threshold, *ANALYZE* gets triggered earlier than *VACUUM*.

+The exact equations for each action are:

+- **Autoanalyze** = autovacuum_analyze_scale_factor * tuples + autovacuum_analyze_threshold

+- **Autovacuum** = autovacuum_vacuum_scale_factor * tuples + autovacuum_vacuum_threshold

+For example, analyze triggers after 60 rows change on a table that contains 100 rows, and vacuum triggers when 70 rows change on the table, using the following equations:

+`Autoanalyze = 0.1 * 100 + 50 = 60`

+`Autovacuum = 0.2 * 100 + 50 = 70`

+Use the following query to list the tables in a database and identify the tables that qualify for the autovacuum process:

+ ,CASE

+ ORDER BY av_needed DESC ,n_dead_tup DESC;

+> [!NOTE]

+> The query doesn't take into consideration that autovacuum can be configured on a per-table basis using the "alter table" DDL command.

+Review the possible common problems with the autovacuum process.

+The autovacuum process estimates the cost of every I/O operation, accumulates a total for each operation it performs and pauses once the upper limit of the cost is reached. `autovacuum_vacuum_cost_delay` and `autovacuum_vacuum_cost_limit` are the two server parameters that are used in the process.

+In case the autovacuum isn't keeping up, the following parameters might be changed:

+| Parameter | Description |

+| | |

+| `autovacuum_vacuum_scale_factor` | Default: `0.2`, range: `0.05 - 0.1`. The scale factor is workload-specific and should be set depending on the amount of data in the tables. Before changing the value, investigate the workload and individual table volumes. |

+| `autovacuum_vacuum_cost_limit` | Default: `200`. Cost limit might be increased. CPU and I/O utilization on the database should be monitored before and after making changes. |

+| `autovacuum_vacuum_cost_delay` | **Postgres Versions 9.6,10,11** - Default: `20 ms`. The parameter might be decreased to `2-10 ms`.<br />**Postgres Versions 12 and above** - Default: `2 ms`. |

+> [!NOTE]

+Continuously running autovacuum might affect CPU and IO utilization on the server. The following might be possible reasons:

+If `maintenance_work_mem` is low, it might be increased to up to 2 GB on Flexible Server. A general rule of thumb is to allocate 50 MB to `maintenance_work_mem` for every 1 GB of RAM.

+Autovacuum tries to start a worker on each database every `autovacuum_naptime` seconds.

+For example, if a server has 60 databases and `autovacuum_naptime` is set to 60 seconds, then the autovacuum worker starts every second [autovacuum_naptime/Number of DBs].

+It's a good idea to increase `autovacuum_naptime` if there are more databases in a cluster. At the same time, the autovacuum process can be made more aggressive by increasing the `autovacuum_cost_limit` and decreasing the `autovacuum_cost_delay` parameters and increasing the `autovacuum_max_workers` from the default of 3 to 4 or 5.

+### Out of memory errors

+Overly aggressive `maintenance_work_mem` values could periodically cause out-of-memory errors in the system. It's important to understand available RAM on the server before any change to the `maintenance_work_mem` parameter is made.

+Evaluate the parameters `autovacuum_vacuum_cost_delay`, `autovacuum_vacuum_cost_limit`, `autovacuum_max_workers`. Improperly setting autovacuum parameters might lead to scenarios where autovacuum becomes too disruptive.

+If autovacuum is too disruptive, consider the following:

+- IncreaseΓÇ»`autovacuum_vacuum_cost_delay` and reduce `autovacuum_vacuum_cost_limit` if set higher than the default of 200.

+- Reduce the number of `autovacuum_max_workers` if it's set higher than the default of 3.

+#### Too many autovacuum workers

+Increasing the number of autovacuum workers won't necessarily increase the speed of vacuum. Having a high number of autovacuum workers isn't recommended.

+Increasing the number of autovacuum workers will result in more memory consumption, and depending on the value of `maintenance_work_mem` , could cause performance degradation.

+Database isn't accepting commands to avoid wraparound data loss in database 'xx'

+Stop the postmaster and vacuum that database in single-user mode.

+> [!NOTE]

+The wraparound problem occurs when the database is either not vacuumed or there are too many dead tuples that couldn't be removed by autovacuum. The reasons for this might be:

+#### Heavy workload

+#### Long-running transactions

+Any long-running transactions in the system won't allow dead tuples to be removed while autovacuum is running. They're a blocker to the vacuum process. Removing the long running transactions frees up dead tuples for deletion when autovacuum runs.

+Long-running transactions can be detected using the following query:

+ SELECT pid, age(backend_xid) AS age_in_xids,

+ now () - xact_start AS xact_age,

+ now () - query_start AS query_age,

+ state,

+ query

+ FROM pg_stat_activity

+ WHERE state != 'idle'

+ ORDER BY 2 DESC

+ LIMIT 10;

+#### Prepared statements

+If there are prepared statements that aren't committed, they would prevent dead tuples from being removed.

+The following query helps find noncommitted prepared statements:

+ SELECT gid, prepared, owner, database, transaction

+ FROM pg_prepared_xacts

+ ORDER BY age(transaction) DESC;

+Use COMMIT PREPARED or ROLLBACK PREPARED to commit or roll back these statements.

+#### Unused replication slots

+Unused replication slots prevent autovacuum from claiming dead tuples. The following query helps identify unused replication slots:

+ SELECT slot_name, slot_type, database, xmin

+ FROM pg_replication_slots

+ ORDER BY age(xmin) DESC;

+UseΓÇ»`pg_drop_replication_slot()` to delete unused replication slots.

+When the database runs into transaction ID wraparound protection, check for any blockers as mentioned previously, and remove those manually for autovacuum to continue and complete. You can also increase the speed of autovacuum by setting `autovacuum_cost_delay` to 0 and increasing the `autovacuum_cost_limit` to a value greater than 200. However, changes to these parameters won't be applied to existing autovacuum workers. Either restart the database or kill existing workers manually to apply parameter changes.

+### Table-specific requirements

+Autovacuum parameters might be set for individual tables. It's especially important for small and big tables. For example, for a small table that contains only 100 rows, autovacuum triggers VACUUM operation when 70 rows change (as calculated previously). If this table is frequently updated, you might see hundreds of autovacuum operations a day. This prevents autovacuum from maintaining other tables on which the percentage of changes aren't as big. Alternatively, a table containing a billion rows needs to change 200 million rows to trigger autovacuum operations. Setting autovacuum parameters appropriately prevents such scenarios.

+To set autovacuum setting per table, change the server parameters as the following examples:

+ ALTER TABLE <table name> SET (autovacuum_vacuum_scale_factor =xx);

+ ALTER TABLE <table name> SET (autovacuum_vacuum_threshold = xx);

+ ALTER TABLE <table name> SET (autovacuum_vacuum_cost_delay = xx);

+ ALTER TABLE <table name> SET (autovacuum_vacuum_cost_limit = xx);

+### Insert-only workloads

+In versions of PostgreSQL prior to 13, autovacuum won't run on tables with an insert-only workload, because if there are no updates or deletes, there are no dead tuples and no free space that needs to be reclaimed. However, autoanalyze will run for insert-only workloads since there's new data. The disadvantages of this are:

+- Hint bits won't be set.

+#### Solutions

+##### Postgres versions prior to 13

+Using the **pg_cron** extension, a cron job can be set up to schedule a periodic vacuum analyze on the table. The frequency of the cron job depends on the workload.

+##### Postgres 13 and higher versions

+Autovacuum will run on tables with an insert-only workload. Two new server parameters `autovacuum_vacuum_insert_threshold` and  `autovacuum_vacuum_insert_scale_factor` help control when autovacuum can be triggered on insert-only tables.

+## Troubleshooting guides

+Using the feature troubleshooting guides which is available on the Azure Database for PostgreSQL - Flexible Server portal it is possible to monitor bloat at database or individual schema level along with identifying potential blockers to autovacuum process. Two troubleshooting guides are available first one is autovacuum monitoring that can be used to monitor bloat at database or individual schema level. The second troubleshooting guide is autovacuum blockers and wraparound which helps to identify potential autovacuum blockers along with information on how far the databases on the server are from wraparound or emergency situation. The troubleshooting guides also share recommendations to mitigate potential issues. How to set up the troubleshooting guides to use them please follow [setup troubleshooting guides](how-to-troubleshooting-guides.md).

+## Related content

+- [High CPU Utilization](how-to-high-cpu-utilization.md)

+- [High Memory Utilization](how-to-high-memory-utilization.md)

+- [Server Parameters](howto-configure-server-parameters-using-portal.md)

+# Troubleshoot high CPU utilization in Azure Database for PostgreSQL - Flexible Server

+This article shows you how to quickly identify the root cause of high CPU utilization, and possible remedial actions to control CPU utilization when using [Azure Database for PostgreSQL - Flexible Server](overview.md).

+In this article, you'll learn:

+- About troubleshooting guides to identify and get recommendations to mitigate root causes.

+- About tools to identify high CPU utilization such as Azure Metrics, Query Store, and pg_stat_statements.

+- How to identify root causes, such as long running queries and total connections.

+- How to resolve high CPU utilization by using Explain Analyze, Connection Pooling, and Vacuuming tables.

+## Troubleshooting guides

+Using the feature troubleshooting guides which is available on the Azure Database for PostgreSQL - Flexible Server portal the probable root cause and recommendations to the mitigate high CPU scenario can be found. How to setup the troubleshooting guides to use them please follow [setup troubleshooting guides](how-to-troubleshooting-guides.md).

+## Tools to identify high CPU utilization

+Consider these tools to identify high CPU utilization.

+### Azure Metrics

+#### Mean or average execution time

+For Postgres versions 13 and above, use the following statement to view the top five SQL statements by mean or average execution time:

+SELECT userid::regrole, dbid, query, mean_exec_time

+FROM pg_stat_statements

+ORDER BY mean_exec_time

+DESC LIMIT 5;

+For Postgres versions 9.6, 10, 11, and 12, use the following statement to view the top five SQL statements by mean or average execution time:

+SELECT userid::regrole, dbid, query

+FROM pg_stat_statements

+ORDER BY mean_time

+DESC LIMIT 5;

+Execute the following statements to view the top five SQL statements by total execution time.

+For Postgres versions 13 and above, use the following statement to view the top five SQL statements by total execution time:

+SELECT userid::regrole, dbid, query

+FROM pg_stat_statements

+ORDER BY total_exec_time

+DESC LIMIT 5;

+For Postgres versions 9.6, 10, 11, and 12, use the following statement to view the top five SQL statements by total execution time:

+SELECT userid::regrole, dbid, query,

+FROM pg_stat_statements

+ORDER BY total_time

+DESC LIMIT 5;

+## Identify root causes

+If CPU consumption levels are high in general, the following could be possible root causes:

+### Long-running transactions

+The following query helps identify connections running for the longest time:

+SELECT pid, usename, datname, query, now() - xact_start as duration

+FROM pg_stat_activity

+WHERE pid <> pg_backend_pid() and state IN ('idle in transaction', 'active')

+ORDER BY duration DESC;

+### Total number of connections and number connections by state

+The following query gives information about the number of connections by state:

+SELECT state, count(*)

+FROM pg_stat_activity

+WHERE pid <> pg_backend_pid()

+GROUP BY 1 ORDER BY 1;

+## Resolve high CPU utilization

+Use Explain Analyze, PG Bouncer, connection pooling and terminate long running transactions to resolve high CPU utilization.

+### Use Explain Analyze

+Once you know the query that's running for a long time, use **EXPLAIN** to further investigate the query and tune it.

+For more information about the **EXPLAIN** command, review [Explain Plan](https://www.postgresql.org/docs/current/sql-explain.html).

+### PGBouncer and connection pooling

+For more details about PgBouncer, review:

+### Terminate long running transactions

+To terminate a session's PID, you'll need to detect the PID using the following query:

+SELECT pid, usename, datname, query, now() - xact_start as duration

+FROM pg_stat_activity

+WHERE pid <> pg_backend_pid() and state IN ('idle in transaction', 'active')

+ORDER BY duration DESC;

+You can also filter by other properties like `usename` (username), `datname` (database name) etc.

+### Monitor vacuum and table stats

+Keeping table statistics up to date helps improve query performance. Monitor whether regular autovacuuming is being carried out.

+The following query helps to identify the tables that need vacuuming:

+select schemaname,relname,n_dead_tup,n_live_tup,last_vacuum,last_analyze,last_autovacuum,last_autoanalyze

+from pg_stat_all_tables where n_live_tup > 0;

+## Related content

+- [Autovacuum Tuning](how-to-high-cpu-utilization.md)

+- [High Memory Utilization](how-to-high-memory-utilization.md)

+- [Identify Slow Queries](how-to-identify-slow-queries.md)

+description: This article is a troubleshooting guide for high IOPS utilization in Azure Database for PostgreSQL - Flexible Server

+ - template-how-to

+This article shows you how to quickly identify the root cause of high IOPS (input/output operations per second) utilization and provides remedial actions to control IOPS utilization when you're using [Azure Database for PostgreSQL - Flexible Server](overview.md).

+- About troubleshooting guides to identify and get recommendations to mitigate root causes.

+## Troubleshooting guides

+Using the feature troubleshooting guides which is available on the Azure Database for PostgreSQL - Flexible Server portal the probable root cause and recommendations to the mitigate high IOPS utilization scenario can be found. How to setup the troubleshooting guides to use them please follow [setup troubleshooting guides](how-to-troubleshooting-guides.md).

+select * from query_store.qs_view qv where is_system_query is FALSE

+FROM pg_stat_statements

+ORDER BY blk_read_time + blk_write_time desc

+LIMIT 5;

+> [!NOTE]

+> When using query store or pg_stat_statements for columns blk_read_time and blk_write_time to be populated, you need to enable server parameter `track_io_timing`. For more information about `track_io_timing`, review [Server parameters](https://www.postgresql.org/docs/current/runtime-config-statistics.html).

+## Identify root causes

+If I/O consumption levels are high in general, the following could be the root causes:

+### Long-running transactions

+The following query helps identify connections that are running for the longest time:

+SELECT pid, usename, datname, query, now() - xact_start as duration

+FROM pg_stat_activity

+WHERE pid <> pg_backend_pid() and state IN ('idle in transaction', 'active')

+ORDER BY duration DESC;

+You could also investigate by using an approach where periodic snapshots of `pg_stat_bgwriter` with a time stamp are saved. By using the saved snapshots, you can calculate the average checkpoint interval, number of checkpoints requested, and number of checkpoints timed.

+SELECT schemaname, relname, n_dead_tup, n_live_tup, autovacuum_count, last_vacuum, last_autovacuum, last_autoanalyze, autovacuum_count, autoanalyze_count FROM pg_stat_all_tables WHERE n_live_tup > 0;

+The query is used to check how frequently the tables in the database are being vacuumed.

+- `last_autovacuum`: The date and time when the last autovacuum ran on the table.

+- `autovacuum_count`: The number of times the table was vacuumed.

+- `autoanalyze_count`: The number of times the table was analyzed.

+### The `EXPLAIN ANALYZE` command

+After you've identified the query that's consuming high I/O, use `EXPLAIN ANALYZE` to further investigate the query and tune it. For more information about the `EXPLAIN ANALYZE` command, review the [EXPLAIN plan](https://www.postgresql.org/docs/current/sql-explain.html).

+### Terminate long-running transactions

+To terminate a session's process ID (PID), you need to detect the PID by using the following query:

+SELECT pid, usename, datname, query, now() - xact_start as duration

+FROM pg_stat_activity

+WHERE pid <> pg_backend_pid() and state IN ('idle in transaction', 'active')

+ORDER BY duration DESC;

+You can also filter by other properties, such as `usename` (username) or `datname` (database name).

+- `max_wal_size`: Peak business hours are a good time to arrive at a `max_wal_size` value. To arrive at a value, do the following:

+ ```sql

+ ```

+- `checkpoint_completion_target`: A good practice would be to set the value to 0.9. As an example, a value of 0.9 for a `checkpoint_timeout` of 5 minutes indicates that the target to complete a checkpoint is 270 seconds (0.9\*300 seconds). A value of 0.9 provides a fairly consistent I/O load. An aggressive value of `checkpoint_completion_target` might result in an increased I/O load on the server.

+- `checkpoint_timeout`: You can increase the `checkpoint_timeout` value from the default value that's set on the server. As you're increasing the value, take into consideration that increasing it would also increase the time for crash recovery.

+### Increase storage

+## Related content

+- [Troubleshoot and tune autovacuum](how-to-autovacuum-tuning.md)

+- [Compute and storage options](concepts-compute-storage.md)

+This article introduces common scenarios and root causes that might lead to high memory utilization in [Azure Database for PostgreSQL - Flexible Server](overview.md).

+In this article, you learn:

+- About troubleshooting guides to identify and get recommendations to mitigate root causes.

+## Troubleshooting guides

+Using the feature troubleshooting guides which is available on the Azure Database for PostgreSQL - Flexible Server portal the probable root cause and recommendations to the mitigate high memory scenario can be found. How to setup the troubleshooting guides to use them please follow [setup troubleshooting guides](how-to-troubleshooting-guides.md).

+## Tools to identify high memory utilization

+Consider the following tools to identify high memory utilization.

+Use Azure Metrics to monitor the percentage of memory in use for the definite date and time frame.

+Query Store automatically captures the history of queries and their runtime statistics, and it retains them for your review.

+Query Store can correlate wait event information with query run time statistics. Use Query Store to identify queries that have high memory consumption during the period of interest.

+Consider the following reasons and remedial actions for resolving high memory utilization.

+#### Work_Mem

+The `work_mem` parameter specifies the amount of memory to be used by internal sort operations and hash tables before writing to temporary disk files. It isn't on a per-query basis rather, it's set based on the number of sort and hash operations.

+If the workload has many short-running queries with simple joins and minimal sort operations, it's advised to keep lower `work_mem`. If there are a few active queries with complex joins and sorts, then it's advised to set a higher value for work_mem.

+It's tough to get the value of `work_mem` right. If you notice high memory utilization or out-of-memory issues, consider decreasing `work_mem`.

+The default value of `work_mem` = 4 MB. You can set the `work_mem` value on multiple levels including at the server level via the parameters page in the Azure portal.

+A good strategy is to monitor memory consumption during peak times.

+Similarly, if the memory use looks high, reduce `work_mem`.

+#### Maintenance_Work_Mem

+`maintenance_work_mem` is for maintenance tasks like vacuuming, adding indexes or foreign keys. The usage of memory in this scenario is per session.

+For example, consider a scenario where there are three autovacuum workers running.

+#### Shared buffers

+A reasonable setting for shared buffers is 25% of RAM. Setting a value of greater than 40% of RAM isn't recommended for most common workloads.

+### Max connections

+All new and idle connections on a Postgres database consume up to 2 MB of memory. One way to monitor connections is by using the following query:

+### Explain Analyze

+## Related content

+- [Autovacuum Tuning](how-to-autovacuum-tuning.md)

+- [High CPU Utilization](how-to-high-cpu-utilization.md)

+- [Server Parameters](howto-configure-server-parameters-using-portal.md)

+ Title: Identify Slow Running Query for Azure Database for PostgreSQL - Flexible Server

+description: Troubleshooting guide for identifying slow running queries in Azure Database for PostgreSQL - Flexible Server

+# Troubleshoot and identify slow-running queries in Azure Database for PostgreSQL - Flexible Server

+This article shows you how to troubleshoot and identify slow-running queries using [Azure Database for PostgreSQL - Flexible Server](overview.md).

+In a high CPU utilization scenario, in this article, you learn how to:

+- Identify slow-running queries.

+- Identify a slow-running procedure along with it. Identify a slow query among a list of queries that belong to the same slow-running stored procedure.

+## High CPU scenario - Identify slow query

+### Prerequisites

+One must enable troubleshooting guides and auto_explain extension on the Azure Database for PostgreSQL ΓÇô Flexible Server. To enable troubleshooting guides, follow the steps mentioned [here](how-to-troubleshooting-guides.md).

+To enable auto_explain extension, follow the steps below:

+1. Add auto_explain extension to the shared preload libraries as shown below from the server parameters page on the Flexible Server portal

+

+ :::image type="content" source="./media/how-to-identify-slow-queries/shared-preload-library.png" alt-text="Screenshot of server parameters page with shared preload libraries parameter." lightbox="./media/how-to-identify-slow-queries/shared-preload-library.png":::

+> [!NOTE]

+> Making this change will require a server restart.

+2. After the auto_explain extension is added to shared preload libraries and the server has restarted, change the below highlighted auto_explain server parameters to `ON` from the server parameters page on the Flexible Server portal and leave the remaining ones

+ with default values as shown below.

+ :::image type="content" source="./media/how-to-identify-slow-queries/auto-explain-parameters.png" alt-text="Screenshot of server parameters page with auto_explain parameters." lightbox="./media/how-to-identify-slow-queries/auto-explain-parameters.png":::

+> [!NOTE]

+> Updating `auto_explain.log_min_duration` parameter to 0 will start logging all queries being executed on the server. This may impact performance of the database. Proper due diligence must be made to come to a value which is considered slow on the server. Example if 30 seconds is considered threshold and all queries being run below 30 seconds is acceptable for application then it is advised to update the parameter to 30000 milliseconds. This would then log any query which is executed more than 30 seconds on the server.

+### Scenario - Identify slow-running query

+With troubleshooting guides and auto_explain extension in place, we explain the scenario with the help of an example.

+We have a scenario where CPU utilization has spiked to 90% and would like to know the root cause of the spike. To debug the scenario, follow the steps mentioned below.

+1. As soon as you're alerted by a CPU scenario, go to the troubleshooting guides available under the Help tab on the Flexible server portal overview page.

+ :::image type="content" source="./media/how-to-identify-slow-queries/troubleshooting-guides-blade.png" alt-text="Screenshot of troubleshooting guides menu." lightbox="./media/how-to-identify-slow-queries/troubleshooting-guides-blade.png":::

+2. Select the High CPU Usage tab from the page opened. The high CPU Utilization troubleshooting guide opens.

+ :::image type="content" source="./media/how-to-identify-slow-queries/high-cpu-troubleshooting-guide.png" alt-text="Screenshot of troubleshooting guides menu - tabs. " lightbox="./media/how-to-identify-slow-queries/high-cpu-troubleshooting-guide.png":::

+3. Select the time range of the reported CPU spike using the time range dropdown list.

+ :::image type="content" source="./media/how-to-identify-slow-queries/high-cpu-timerange.png" alt-text="Screenshot of troubleshooting guides menu - CPU tab." lightbox="./media/how-to-identify-slow-queries/high-cpu-timerange.png":::

+4. Select Top CPU Consuming Queries tab.

+ The tab shares details of all the queries that ran in the interval where 90% CPU utilization was seen. From the snapshot, it looks like the query with the slowest average execution time during the time interval was ~2.6 minutes, and the query ran 22 times during the interval. This is most likely the cause of CPU spikes.

+ :::image type="content" source="./media/how-to-identify-slow-queries/high-cpu-query.png" alt-text="Screenshot of troubleshooting guides menu - Top CPU consuming queries tab." lightbox="./media/how-to-identify-slow-queries/high-cpu-query.png":::

+5. Connect to azure_sys database and execute the query to retrieve actual query text using the script below

+```sql

+ psql -h ServerName.postgres.database.azure.com -U AdminUsername -d azure_sys

+ SELECT query_sql_text

+ FROM query_store.query_texts_view

+ WHERE query_text_id = <add query id identified>;

+```

+6. In the example considered, the query that was found slow was the following:

+```sql

+SELECT c_id, SUM(c_balance) AS total_balance

+FROM customer

+GROUP BY c_w_id,c_id

+order by c_w_id;

+```

+7. To understand what exact explain plan was generated, use Postgres logs. Auto explain extension would have logged an entry in the logs every time the query execution was completed during the interval. Select Logs section from the `Monitoring` tab from the Flexible server portal overview page.

+ :::image type="content" source="./media/how-to-identify-slow-queries/log-analytics-tab.png" alt-text="Screenshot of troubleshooting guides menu - Logs." lightbox="./media/how-to-identify-slow-queries/log-analytics-tab.png":::

+

+8. Select the time range where 90% CPU Utilization was found.

+

+ :::image type="content" source="./media/how-to-identify-slow-queries/log-analytics-timerange.png" alt-text="Screenshot of troubleshooting guides menu - Logs Timerange." lightbox="./media/how-to-identify-slow-queries/log-analytics-timerange.png":::

+9. Execute the below query to retrieve the explain analyze output of the query identified.

+```sql

+AzureDiagnostics

+| where Category contains 'PostgreSQLLogs'

+| where Message contains "<add snippet of SQL text identified or add table name involved in the query>"

+| project TimeGenerated, Message

+```

+The message column will store the execution plan as shown below:

+```sql

+2023-10-10 19:56:46 UTC-6525a8e7.2e3d-LOG: duration: 150692.864 ms plan:

+Query Text: SELECT c_id, SUM(c_balance) AS total_balance

+FROM customer

+GROUP BY c_w_id,c_id

+order by c_w_id;

+GroupAggregate (cost=1906820.83..2131820.83 rows=10000000 width=40) (actual time=70930.833..129231.950 rows=10000000 loops=1)

+Output: c_id, sum(c_balance), c_w_id

+Group Key: customer.c_w_id, customer.c_id

+Buffers: shared hit=44639 read=355362, temp read=77521 written=77701

+-> Sort (cost=1906820.83..1931820.83 rows=10000000 width=13) (actual time=70930.821..81898.430 rows=10000000 loops=1)

+Output: c_id, c_w_id, c_balance

+Sort Key: customer.c_w_id, customer.c_id

+Sort Method: external merge Disk: 225104kB

+Buffers: shared hit=44639 read=355362, temp read=77521 written=77701

+-> Seq Scan on public.customer (cost=0.00..500001.00 rows=10000000 width=13) (actual time=0.021..22997.697 rows=10000000 loops=1)

+Output: c_id, c_w_id, c_balance

+```

+The query ran for ~2.5 minutes, as shown in troubleshooting guides, and is confirmed by the `duration` value of 150692.864 ms from the execution plan output fetched. Use the explain analyze output to troubleshoot further and tune the query.

+> [!NOTE]

+> Note that the query ran 22 times during the interval, and the logs shown above are one such entry captured during the interval.

+## High CPU scenario - Identify slow-running procedure and slow queries associated with the procedure

+In the second scenario, a stored procedure execution time is found to be slow, and the goal is to identify and tune the slow-running query that is part of the stored procedure.

+### Prerequisites

+One must enable troubleshooting guides and auto_explain extension on the Azure Database for PostgreSQL ΓÇô Flexible Server as a prerequisite. To enable troubleshooting guides, follow the steps mentioned [here](how-to-troubleshooting-guides.md).

+To enable auto_explain extension, follow the steps below:

+1. Add auto_explain extension to the shared preload libraries as shown below from the server parameters page on the Flexible Server portal

+ :::image type="content" source="./media/how-to-identify-slow-queries/shared-preload-library.png" alt-text="Screenshot of server parameters page with shared preload libraries parameter - Procedure." lightbox="./media/how-to-identify-slow-queries/shared-preload-library.png":::

+> [!NOTE]

+> Making this change will require a server restart.

+2. After the auto_explain extension is added to shared preload libraries and the server has restarted, change the below highlighted auto_explain server parameters to `ON` from the server parameters page on the Flexible Server portal and leave the remaining ones

+ with default values as shown below.

+ :::image type="content" source="./media/how-to-identify-slow-queries/auto-explain-procedure-parameters.png" alt-text="Screenshot of server parameters blade with auto_explain parameters - Procedure." lightbox="./media/how-to-identify-slow-queries/auto-explain-procedure-parameters.png":::

+> [!NOTE]

+>- Updating `auto_explain.log_min_duration` parameter to 0 will start logging all queries being executed on the server. This may impact performance of the database. Proper due diligence must be made to come to a value which is considered slow on the server. Example if 30 seconds is considered threshold and all queries being run below 30 seconds is acceptable for application then it is advised to update the parameter to 30000 milliseconds. This would then log any query which is executed more than 30 seconds on the server.

+>- The parameter `auto_explain.log_nested_statements` causes nested statements (statements executed inside a function or procedure) to be considered for logging. When it is off, only top-level query plans are logged.ΓÇ»

+### Scenario - Identify slow-running query in a stored procedure

+With troubleshooting guides and auto_explain extension in place, we explain the scenario with the help of an example.

+We have a scenario where CPU utilization has spiked to 90% and would like to know the root cause of the spike. To debug the scenario, follow the steps mentioned below.

+1. As soon as you're alerted by a CPU scenario, go to the troubleshooting guides available under the Help tab on the Flexible server portal overview page.

+ :::image type="content" source="./media/how-to-identify-slow-queries/troubleshooting-guides-blade.png" alt-text="Screenshot of troubleshooting guides menu." lightbox="./media/how-to-identify-slow-queries/troubleshooting-guides-blade.png":::

+2. Select the High CPU Usage tab from the page opened. The high CPU Utilization troubleshooting guide opens.

+ :::image type="content" source="./media/how-to-identify-slow-queries/high-cpu-troubleshooting-guide.png" alt-text="Screenshot of troubleshooting guides tabs." lightbox="./media/how-to-identify-slow-queries/high-cpu-troubleshooting-guide.png":::

+3. Select the time range of the reported CPU spike using the time range dropdown list.

+ :::image type="content" source="./media/how-to-identify-slow-queries/high-cpu-procedure-timerange.png" alt-text="Screenshot of troubleshooting guides - CPU tab." lightbox="./media/how-to-identify-slow-queries/high-cpu-procedure-timerange.png":::

+4. Select the Top CPU Consuming Queries tab.

+ The tab shares details of all the queries that ran in the interval where 90% CPU utilization was seen. From the snapshot, it looks like the query with the slowest average execution time during the time interval was ~6.3 minutes, and the query ran 35 times during the interval. This is most likely the cause of CPU spikes.

+ :::image type="content" source="./media/how-to-identify-slow-queries/high-cpu-procedure.png" alt-text="Screenshot of troubleshooting guides - CPU tab - queries." lightbox="./media/how-to-identify-slow-queries/high-cpu-procedure.png":::

+ It's important to note from the snapshot below that the query type as highlighted below is `Utility``. Generally, a utility can be a stored procedure or function running during the interval.

+5. Connect to azure_sys database and execute the query to retrieve actual query text using the below script.

+```sql

+ psql -h ServerName.postgres.database.azure.com -U AdminUsername -d azure_sys

+ SELECT query_sql_text

+ FROM query_store.query_texts_view

+ WHERE query_text_id = <add query id identified>;

+```

+6. In the example considered, the query that was found slow was a stored procedure as mentioned below:

+```sql

+ call autoexplain_test ();

+```

+7. To understand what exact explanations are generated for the queries that are part of the stored procedure, use Postgres logs. Auto explain extension would have logged an entry in the logs every time the query execution was completed during the interval. Select the Logs section from the `Monitoring` tab from the Flexible server portal overview page.

+ :::image type="content" source="./media/how-to-identify-slow-queries/log-analytics-tab.png" alt-text="Screenshot of troubleshooting guides menu - Logs." lightbox="./media/how-to-identify-slow-queries/log-analytics-tab.png":::

+8. Select the time range where 90% CPU Utilization was found.

+ :::image type="content" source="./media/how-to-identify-slow-queries/log-analytics-timerange.png" alt-text="Screenshot of troubleshooting guides menu - Logs Time range." lightbox="./media/how-to-identify-slow-queries/log-analytics-timerange.png":::

+9. Execute the query below to retrieve the explained analyze output of the identified query.

+```sql

+AzureDiagnostics

+| where Category contains 'PostgreSQLLogs'

+| where Message contains "<add a snippet of SQL text identified or add table name involved in the queries related to stored procedure>"

+| project TimeGenerated, Message

+```

+The procedure has multiple queries, which are highlighted below. The explain analyze output of every query used in the stored procedure is logged in to analyze further and troubleshoot. The execution time of the queries logged can be used to identify the slowest queries that are part of the stored procedure.

+```sql

+2023-10-11 17:52:45 UTC-6526d7f0.7f67-LOG: duration: 38459.176 ms plan:

+Query Text: insert into customer_balance SELECT c_id, SUM(c_balance) AS total_balance FROM customer GROUP BY c_w_id,c_id order by c_w_id

+Insert on public.customer_balance (cost=1906820.83..2231820.83 rows=0 width=0) (actual time=38459.173..38459.174 rows=0 loops=1)Buffers: shared hit=10108203 read=454055 dirtied=54058, temp read=77521 written=77701 WAL: records=10000000 fpi=1 bytes=640002197

+ -> Subquery Scan on "*SELECT*" (cost=1906820.83..2231820.83 rows=10000000 width=36) (actual time=20415.738..29514.742 rows=10000000 loops=1)

+ Output: "*SELECT*".c_id, "*SELECT*".total_balance Buffers: shared hit=1 read=400000, temp read=77521 written=77701

+ -> GroupAggregate (cost=1906820.83..2131820.83 rows=10000000 width=40) (actual time=20415.737..28574.266 rows=10000000 loops=1)

+ Output: customer.c_id, sum(customer.c_balance), customer.c_w_id Group Key: customer.c_w_id, customer.c_id Buffers: shared hit=1 read=400000, temp read=77521 written=77701

+ -> Sort (cost=1906820.83..1931820.83 rows=10000000 width=13) (actual time=20415.723..22023.515 rows=10000000 loops=1)

+ Output: customer.c_id, customer.c_w_id, customer.c_balance Sort Key: customer.c_w_id, customer.c_id Sort Method: external merge Disk: 225104kB Buffers: shared hit=1 read=400000, temp read=77521 written=77701

+ -> Seq Scan on public.customer (cost=0.00..500001.00 rows=10000000 width=13) (actual time=0.310..15061.471 rows=10000000 loops=1) Output: customer.c_id, customer.c_w_id, customer.c_balance Buffers: shared hit=1 read=400000

+2023-10-11 17:52:07 UTC-6526d7f0.7f67-LOG: duration: 61939.529 ms plan:

+Query Text: delete from customer_balance

+Delete on public.customer_balance (cost=0.00..799173.51 rows=0 width=0) (actual time=61939.525..61939.526 rows=0 loops=1) Buffers: shared hit=50027707 read=620942 dirtied=295462 written=71785 WAL: records=50026565 fpi=34 bytes=2711252160

+ -> Seq Scan on public.customer_balance (cost=0.00..799173.51 rows=15052451 width=6) (actual time=3185.519..35234.061 rows=50000000 loops=1)

+ Output: ctid Buffers: shared hit=27707 read=620942 dirtied=26565 written=71785 WAL: records=26565 fpi=34 bytes=11252160

+2023-10-11 17:51:05 UTC-6526d7f0.7f67-LOG: duration: 10387.322 ms plan:

+Query Text: select max(c_id) from customer

+Finalize Aggregate (cost=180185.84..180185.85 rows=1 width=4) (actual time=10387.220..10387.319 rows=1 loops=1) Output: max(c_id) Buffers: shared hit=37148 read=1204 written=69

+ -> Gather (cost=180185.63..180185.84 rows=2 width=4) (actual time=10387.214..10387.314 rows=1 loops=1)

+ Output: (PARTIAL max(c_id)) Workers Planned: 2 Workers Launched: 0 Buffers: shared hit=37148 read=1204 written=69

+ -> Partial Aggregate (cost=179185.63..179185.64 rows=1 width=4) (actual time=10387.012..10387.013 rows=1 loops=1) Output: PARTIAL max(c_id) Buffers: shared hit=37148 read=1204 written=69

+ -> Parallel Index Only Scan using customer_i1 on public.customer (cost=0.43..168768.96 rows=4166667 width=4) (actual time=0.446..7676.356 rows=10000000 loops=1)

+ Output: c_w_id, c_d_id, c_id Heap Fetches: 24 Buffers: shared hit=37148 read=1204 written=69

+```

+> [!NOTE]

+> please note for demonstration purpose explain analyze output of only few queries used in the procedure are shared. The idea is one can gather explain analyze output of all queries from the logs, and then identify the slowest of those and try to tune them.

+## Related content

+- [High CPU Utilization](how-to-high-cpu-utilization.md)

+- [Autovacuum Tuning](how-to-autovacuum-tuning.md)

+ Title: Optimize Azure Database for PostgreSQL Flexible Server by using pg_repack

+description: Perform full vacuum using pg_Repack extension in Azure Database for PostgreSQL - Flexible Server

+# Optimize Azure Database for PostgreSQL Flexible Server by using pg_repack

+In this article, you learn how to use pg_repack to remove bloat and improve your Azure Database performance for PostgreSQL Flexible Server. Bloat is the unnecessary data accumulating in tables and indexes due to frequent updates and deletes. Bloat can cause the database size to grow larger than expected and affect query performance. Using pg_repack, you can reclaim the wasted space and reorganize the data more efficiently.

+## What is pg_repack?

+pg_repack is a PostgreSQL extension that removes bloat from tables and indexes and reorganizes them more efficiently. pg_repack works by creating a new copy of the target table or index, applying any changes that occurred during the process, and then swapping the old and new versions atomically. pg_repack doesn't require any downtime or exclusive locks on the target table or index except for a brief period at the beginning and end of the operation. You can use pg_repack to optimize any table or index in your PostgreSQL database, except for the default PostgreSQL database.

+### How to use pg_repack?

+To use pg_repack, you need to install the extension in your PostgreSQL database and then run the pg_repack command, specifying the table name or index you want to optimize. The extension acquires locks on the table or index to prevent other operations from being performed while the optimization is in progress. It will then remove the bloat and reorganize the data more efficiently.

+### How full table repack works

+To perform a full table repack, pg_repack will follow these steps:

+1. Create a log table to record changes made to the original table.

+2. Add a trigger to the original table, logging INSERTs, UPDATEs, and DELETEs into the log table.

+3. Create a new table containing all the rows in the old table.

+4. Build indexes on the new table.

+5. Apply all changes recorded in the log table to the new table.

+6. Swap the tables, including indexes and toast tables, using the system catalogs.

+7. Drop the original table.

+During these steps, pg_repack will only hold an ACCESS EXCLUSIVE lock for a short period during the initial setup (steps 1 and 2) and the final swap-and-drop phase (steps 6 and 7). For the rest of the time, pg_repack will only need to hold an ACCESS SHARE lock on the original table, allowing INSERTs, UPDATEs, and DELETEs to proceed as usual.

+### Limitations

+pg_repack has some limitations that you should be aware of before using it:

+- The pg_repack extension can't be used to repack the default database named `postgres`. This is due to pg_repack not having the necessary permissions to operate against extensions installed by default on this database. The extension can be created in PostgreSQL, but it can't run.

+- The target table must have either a PRIMARY KEY or a UNIQUE index on a NOT NULL column for the operation to be successful.

+- While pg_repack is running, you won't be able to perform any DDL commands on the target table(s) except for VACUUM or ANALYZE. To ensure these restrictions are enforced, pg_repack will hold an ACCESS SHARE lock on the target table during a

+ full table repack.

+## Setup

+### Prerequisites

+To enable the pg_repack extension, follow the steps below:

+1. Add pg_repack extension under Azure extensions as shown below from the server parameters blade on Flexible server portal

+ :::image type="content" source="./media/how-to-perform-fullvacuum-pg-repack/portal.png" alt-text="Screenshot of server parameters blade with Azure extensions parameter." lightbox="./media/how-to-perform-fullvacuum-pg-repack/portal.png":::

+> [!NOTE]

+> Making this change will not require a server restart.

+### Install the packages for Ubuntu virtual machine

+Using the extension requires a client with psql and pg_repack installed. All examples in this document use an Ubuntu VM with PostgreSQL 11 to 15.

+Run the following packages on the Ubuntu machine to install the pg_repack client 1.4.7

+```psql

+sudo sh -c 'echo "deb https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' && \

+wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - && \

+sudo apt-get update && \

+sudo apt install -y postgresql-server-dev-14 && \

+sudo apt install -y unzip make gcc && \

+sudo apt-get install -y libssl-dev liblz4-dev zlib1g-dev libreadline-dev && \

+wget 'https://api.pgxn.org/dist/pg_repack/1.4.7/pg_repack-1.4.7.zip' && \

+unzip pg_repack-1.4.7.zip && \

+cd pg_repack-1.4.7 && \

+sudo make && \

+sudo cp bin/pg_repack /usr/local/bin && \

+pg_repack -V

+```

+## Use pg_repack

+Example of how to run pg_repack on a table named info in a public schema within the Flexible Server with endpoint pgserver.postgres.database.azure.com, username azureuser, and database foo using the following command.

+1. Connect to the Flexible Server instance. This article uses psql for simplicity.

+ ```psql

+ psql "host=xxxxxxxxx.postgres.database.azure.com port=5432 dbname=foo user=xxxxxxxxxxxxx password=[my_password] sslmode=require"

+ ```

+2. Create the pg_repack extension in the databases intended to be repacked.

+ ```psql

+ foo=> create extension pg_repack;

+ CREATE EXTENSION

+ ```dotnetcli

+3. Find the pg_repack version installed on the server.

+ ```psql

+ foo=> \dx

+ List of installed extensions

+ Name | Version | Schema | Description

+

+ --+++--

+

+ pg_repack | 1.4.7 | public | Reorganize tables in PostgreSQL databases with minimal locks

+

+ (one row)

+

+ This version should match with the pg_repack in the virtual machine. Check this by running the following.

+

+ azureuser@azureuser:~$ pg_repack --version

+ pg_repack 1.4.7

+ ```

+4. Run pg_repack client against a table *info* within database *foo*.

+ ```psql

+ pg_repack --host=xxxxxxxxxxxx.postgres.database.azure.com --username=xxxxxxxxxx --dbname=foo --table=info --jobs=2 --no-kill-backend --no-superuser-check

+ ```

+### pg_repack options

+Useful pg_repack options for production workloads:

+- -k, --no-superuser-check

+ Skip the superuser checks in the client. This setting is helpful for using pg_repack on platforms that support running it as non-superusers, like Azure Database for PostgreSQL Flexible Servers.

+- -j, --jobs

+ Create the specified number of extra connections to PostgreSQL and use these extra connections to parallelize the rebuild of indexes on each table. Parallel index builds are only supported for full-table repacks.

+- --index or --only indexes options

+ If your PostgreSQL server has extra cores and disk I/O available, this can be a useful way to speed up pg_repack.

+- -D, --no-kill-backend

+ Skip to repack table if the lock can't be taken for duration specified --wait-timeout default 60 sec, instead of canceling conflicting queries. The default is false.

+- -E LEVEL, --elevel=LEVEL

+ Choose the output message level from DEBUG, INFO, NOTICE, WARNING, ERROR, LOG, FATAL, and PANIC. The default is INFO.

+To understand all the options, refer to [pg_repack options](https://reorg.github.io/pg_repack/)

+## Related content

+> [!div class="nextstepaction"]

+> [Autovacuum Tuning](how-to-high-cpu-utilization.md)

+## Remove a resource on portal

+1. On the **Azure Resource Mover** > **Across regions** pane, select all the resources you want to remove from the collection, and select **Remove**.

+ :::image type="content" source="./media/remove-move-resources/across-region.png" alt-text="Screenshot of the **Across regions** pane." lightbox="./media/remove-move-resources/across-region.png" :::

+ :::image type="content" source="./media/remove-move-resources/portal-select-resources.png" alt-text="Screenshot of the Button to select to remove." :::

+2. In **Remove resources**, select **Remove**.

+ :::image type="content" source="./media/remove-move-resources/remove-portal.png" alt-text="Screenshot of the Button to select to remove resources from a move collection." :::

+## Remove a move collection or a resource group on portal

+You can remove a move collection/resource group in the portal. Removing a move collection/resource group deletes all the resources in the collection.

+To remove a move collection/resource group, follow these steps:

+1. Follow [these instructions](#remove-a-resource-on-portal) to remove resources from the collection. If you're removing a resource group, make sure it doesn't contain any resources.

+## Remove a resource using PowerShell

+ :::image type="content" source="./media/remove-move-resources/remove-multiple-validate-dependencies.png" alt-text="Screenshot of output text after removing multiple resources from a move collection." :::

+ :::image type="content" source="./media/remove-move-resources/remove-multiple-get-dependencies.png" alt-text="Screenshot of output text after retrieving dependent resources that need to be removed." :::

+ :::image type="content" source="./media/remove-move-resources/remove-multiple-all.png" alt-text="Screenshot of output text after removing all resources from a move collection." :::

+## Remove a collection using PowerShell

+1. Follow [these instructions](#remove-a-resource-using-powershell) to remove resources in the collection using PowerShell.

+2. Then remove a collection as follows:

+ :::image type="content" source="./media/remove-move-resources/remove-collection.png" alt-text="Screenshot of output text after removing a move collection." :::

+> [!NOTE]

+> For removing resources in bulk where the dependency tree is not identified, use [Invoke-AzResourceMoverBulkRemove (Az.ResourceMover)](/powershell/module/az.resourcemover/invoke-azresourcemoverbulkremove).

+| **Subscription permissions** | Check you have *Owner* access on the subscription containing the resources that you want to move.<br/><br/> The first time you add a resource for a specific source and destination pair in an Azure subscription, a [system-assigned managed identity](../active-directory/managed-identities-azure-resources/overview.md#managed-identity-types) (formerly known as Managed Service Identify (MSI)) that's trusted by the subscription is necessary. To create the identity, and to assign it the required role (Contributor or User Access administrator in the source subscription), the account you use to add resources needs *Owner* permissions on the subscription. [Learn more](../role-based-access-control/rbac-and-directory-admin-roles.md#azure-roles) about Azure roles. |

+ :::image type="content" source="./media/tutorial-move-region-powershell/output-retrieve-resource.png" alt-text="Screenshot of the output text after retrieving the resource ID." :::

+ :::image type="content" source="./media/tutorial-move-region-powershell/output-add-resource.png" alt-text="Screenshot of the output text after adding the resource." :::

+ :::image type="content" source="./media/tutorial-move-region-powershell/dependency-output.png" alt-text="Screenshot of the output text after validating dependencies." :::

+ :::image type="content" source="./media/tutorial-move-region-powershell/dependencies-list.png" alt-text="Screenshot of the output text after retrieving a list of all dependencies." :::

+ :::image type="content" source="./media/tutorial-move-region-powershell/dependencies-list-direct.png" alt-text="Screenshot of the output text after retrieving a list of first-level dependencies." :::

+ :::image type="content" source="./media/tutorial-move-region-powershell/validate-vm-before-move.png" alt-text="Screenshot of the output text after validating the VM before preparing it." :::

+ :::image type="content" source="./media/tutorial-move-region-powershell/get-resources-before-prepare.png" alt-text="Screenshot of the output text after retrieving dependent VM resources." :::

+ :::image type="content" source="./media/tutorial-move-region-powershell/initiate-prepare-all.png" alt-text="Screenshot of the output text after initiating prepare of all resources." :::

+ :::image type="content" source="./media/tutorial-move-region-powershell/verify-initiate-move-pending.png" alt-text="Screenshot of the output text after checking initiate state." :::

+ :::image type="content" source="./media/tutorial-move-region-powershell/initiate-resources-move.png" alt-text="Screenshot of the output text after initiating the move of resources." :::

+ :::image type="content" source="./media/tutorial-move-region-powershell/commit-move.png" alt-text="Screenshot of the output text after committing the move." :::

+Azure Center for SAP solutions supports the following SAP software versions: S/4HANA 1909 SPS 03, S/4HANA 2020 SPS 03, S/4HANA 2021 ISS 00 and S/4HANA 2022 ISS 00.

+- You can use `latest` if you want to use the latest image and not a specific older version. If the *latest* image version is newly released in marketplace and has an unforeseen issue, the deployment might fail. If you are using Portal for deployment, we recommend choosing a different image *sku train* (e.g. 12-SP4 instead of 15-SP3) till the issues are resolved. However, if deploying via API/CLI, you can provide any other *image version* which is available. To view and select the available image versions from a publisher, use below commands

+- Azure Center for SAP solutions now supports deployment of SAP system VMs with custom OS images along with the Azure Marketplace images. For deployment using custom OS images, follow the steps [here](deploy-s4hana.md#using-a-custom-os-image).

+1. Under **Operating systems**, select the source of the image.

+1. If you're using Azure Marketplace OS images, use these settings:

+ 1. For **Application OS image**, select the OS image for the application server.

+

+ 1. If you're using [custom OS images](deploy-s4hana.md#using-a-custom-os-image), use these settings:

+

+ 1. For **Application OS image**, select the image version from the Azure Compute Gallery.

+ 1. For **Database OS image**, select the image version from the Azure Compute Gallery.

+ 1. If you choose to **Create a new SAP transport Directory**, this will create and mount a new transport fileshare on the SID. By Default, this option will create an NFS on AFS storage account and a transport fileshare in the resource group where SAP system will be deployed. However, you can choose to create this storage account in a different resource group by providing the resource group name in **Transport Resource Group**. You can also provide a custom name for the storage account to be created under **Storage account name** section. Leaving the **Storage account name** will create the storage account with service default name **""SIDname""nfs""random characters""** in the chosen transport resource group. Creating a new transport directory will create a ZRS based replication for zonal deployments and LRS based replication for non-zonal deployments. If your region doesn't support ZRS replication deploying a zonal VIS will lead to a failure. In such cases, you can deploy a transport fileshare outside Azure Center for SAP Solutions with ZRS replication and then create a zonal VIS where you select **Use an existing SAP transport Directory** to mount the pre-created fileshare.

+ 1. For **Managed identity name**, enter a name for a new identity you want to create or select an existing identity from the drop down menu. If you are selecting an existing identity, it should have **Contributor** role access on the Subscription or on Resource Groups related to this SAP system you are trying to deploy. That is, it requires Contributor access to the SAP application Resource Group, Virtual Network Resource Group and Resource Group which has the existing SSHKEY. If you wish to later install the SAP system using Azure Center for SAP Solutions, we also recommend giving the **Storage Blob Data Reader and Reader** and **Data Access roles** on the Storage Account which has the SAP software media.

+ 1. Click on **Reset** to reset the visualization to its default state. That is, revert any changes you might have made to the position of resources or containers.

+## Using a Custom OS Image

+You can use custom images for deployment in Azure Center for SAP Solutions from the [Azure Compute Gallery](../../virtual-machines/capture-image-portal.md#capture-a-vm-in-the-portal)

+### Custom image prerequisites

+- Make sure that you've met the [general SAP deployment prerequisites](#prerequisites), [downloaded the SAP media](../../sap/center-sap-solutions/get-sap-installation-media.md#prerequisites), and [installed the SAP software](../../sap/center-sap-solutions/install-software.md#install-sap-software).

+- Before you use an image from Azure Marketplace for customization, check the [list of supported OS image](#deployment-types) versions in Azure Center for SAP Solutions. BYOI is supported on the OS version supported by Azure Center for SAP Solutions. Make sure that Azure Center for SAP Solutions has support for the image, or else the deployment will fail with the following error:

+ *The resource ID provided consists of an OS image which is not supported in ACSS. Please ensure that the OS image version is supported in ACSS for a successful installation.*

+- Refer to SAP installation documentation to ensure the operating system prerequisites are met for the deployment to be successful.

+- Check that the user-assigned managed identity has the **Reader role** on the gallery of the custom OS image. Otherwise, the deployment will fail.

+- [Create and upload a VM to a gallery in Azure Compute Gallery](../../virtual-machines/capture-image-portal.md#capture-a-vm-in-the-portal)

+- Before beginning the deployment, make sure the image is available in Azure Compute Gallery.

+- Verify that the image is in same subscription as the deployment.

+- Check that the image VM is of the **Standard** security type.

+### Deploying using Custom Operating System Image

+- Select the **Use a custom image** option during deployment. Choose which image to use for the application and database OS.

+- Azure Center for SAP Solutions validates the base operating system version of the custom OS Image is available in the supportability matrix in Azure Center for SAP Solutions. If the versions are unsupported, the deployment fails. To fix this problem, delete the VIS and infrastructure resources from the resource group, then deploy again with a supported image.

+- Make sure the image version that you're using is [compatible with the SAP software version](#deployment-types).

+

+You can analyze metrics for Azure Service Bus, along with metrics from other Azure services, by selecting **Metrics** from the **Azure Monitor** section on the home page for your Service Bus namespace. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool. For a list of the platform metrics collected, see [Monitoring Azure Service Bus data reference metrics](monitor-service-bus-reference.md#metrics).

+ Title: Azure Service Bus - messaging exceptions | Microsoft Docs

+description: This article provides a list of Azure Service Bus messaging exceptions and suggested actions to taken when the exception occurs.

+# Service Bus messaging exceptions (.NET)

+The Service Bus .NET client library surfaces exceptions when a service operation or a client encounters an error. When possible, standard .NET exception types are used to convey error information. For scenarios specific to Service Bus, a [ServiceBusException](/dotnet/api/azure.messaging.servicebus.servicebusexception) is thrown.

+The Service Bus clients automatically retry exceptions that are considered transient, following the configured [retry options](/dotnet/api/azure.messaging.servicebus.servicebusretryoptions). When an exception is surfaced to the application, either all retries were applied unsuccessfully, or the exception was considered nontransient. More information on configuring retry options can be found in the [Customizing the retry options](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Azure.Messaging.ServiceBus/samples/Sample13_AdvancedConfiguration.md#customizing-the-retry-options) sample.

+## ServiceBusException

+The exception includes some contextual information to help you understand the context of the error and its relative severity.

+- `EntityPath` : Identifies the Service Bus entity from which the exception occurred, if available.

+- `IsTransient` : Indicates whether or not the exception is considered recoverable. In the case where it was deemed transient, the appropriate retry policy has already been applied and all retries were unsuccessful.

+- `Message` : Provides a description of the error that occurred and relevant context.

+- `StackTrace` : Represents the immediate frames of the call stack, highlighting the location in the code where the error occurred.

+- `InnerException` : When an exception was the result of a service operation, it's often a `Microsoft.Azure.Amqp.AmqpException` instance describing the error, following the [OASIS Advanced Message Queuing Protocol (AMQP) 1.0 spec](https://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-types-v1.0-os.html).

+- `Reason` : Provides a set of well-known reasons for the failure that help to categorize and clarify the root cause. These values are intended to allow for applying exception filtering and other logic where inspecting the text of an exception message wouldn't be ideal. Some key failure reasons are:

+ - `ServiceTimeout`: Indicates that the Service Bus service didn't respond to an operation request within the expected amount of time. It might be due to a transient network issue or service problem. The Service Bus service might or might not have successfully completed the request; the status isn't known. In the context of the next available session, this exception indicates that there were no unlocked sessions available in the entity. These errors are transient errors that are automatically retried.

+ - `QuotaExceeded`: Typically indicates that there are too many active receive operations for a single entity. In order to avoid this error, reduce the number of potential concurrent receives. You can use batch receives to attempt to receive multiple messages per receive request. For more information, see [Service Bus quotas](service-bus-quotas.md).

+ - `MessageSizeExceeded`: Indicates that the max message size has been exceeded. The message size includes the body of the message, and any associated metadata. The best approach for resolving this error is to reduce the number of messages being sent in a batch or the size of the body included in the message. Because size limits are subject to change, see [Service Bus quotas](service-bus-quotas.md) for specifics.

+ - `MessageLockLost`: Indicates that the lock on the message is lost. Callers should attempt to receive and process the message again. This exception only applies to non-session entities. This error occurs if processing takes longer than the lock duration and the message lock isn't renewed. This error can also occur when the link is detached due to a transient network issue or when the link is idle for 10 minutes.

+

+ The Service Bus service uses the AMQP protocol, which is stateful. Due to the nature of the protocol, if the link that connects the client and the service is detached after a message is received, but before the message is settled, the message isn't able to be settled on reconnecting the link. Links can be detached due to a short-term transient network failure, a network outage, or due to the service enforced 10-minute idle timeout. The reconnection of the link happens automatically as a part of any operation that requires the link, that is, settling or receiving messages. Because of this behavior, you might encounter `ServiceBusException` with `Reason` of `MessageLockLost` or `SessionLockLost` even if the lock expiration time hasn't yet passed.

+ - `SessionLockLost`: Indicates that the lock on the session has expired. Callers should attempt to accept the session again. This exception applies only to session-enabled entities. This error occurs if processing takes longer than the lock duration and the session lock isn't renewed. This error can also occur when the link is detached due to a transient network issue or when the link is idle for 10 minutes. The Service Bus service uses the AMQP protocol, which is stateful. Due to the nature of the protocol, if the link that connects the client and the service is detached after a message is received, but before the message is settled, the message isn't able to be settled on reconnecting the link. Links can be detached due to a short-term transient network failure, a network outage, or due to the service enforced 10-minute idle timeout. The reconnection of the link happens automatically as a part of any operation that requires the link, that is, settling or receiving messages. Because of this behavior, you might encounter `ServiceBusException` with `Reason` of `MessageLockLost` or `SessionLockLost` even if the lock expiration time hasn't yet passed.

+ - `MessageNotFound`: This error occurs when attempting to receive a deferred message by sequence number for a message that either doesn't exist in the entity, or is currently locked.

+ - `SessionCannotBeLocked`: Indicates that the requested session can't be locked because the lock is already held elsewhere. Once the lock expires, the session can be accepted.

+ - `GeneralError`: Indicates that the Service Bus service encountered an error while processing the request. This error is often caused by service upgrades and restarts. These errors are transient errors that are automatically retried.

+ - `ServiceCommunicationProblem`: Indicates that there was an error communicating with the service. The issue might stem from a transient network problem, or a service problem. These errors are transient errors that will be automatically retried.

+ - `ServiceBusy`: Indicates that a request was throttled by the service. The details describing what can cause a request to be throttled and how to avoid being throttled can be found [here](service-bus-throttling.md). Throttled requests are retried, but the client library automatically applies a 10 second back off before attempting any more requests using the same `ServiceBusClient` (or any subtypes created from that client). It can cause issues if your entity's lock duration is less than 10 seconds, as message or session locks are likely to be lost for any unsettled messages or locked sessions. Because throttled requests are generally retried successfully, the exceptions generated would be logged as warnings rather than errors - the specific warning-level event source event is 43 (RunOperation encountered an exception and retry occurs.).

+ - `MessagingEntityAlreadyExists`: Indicates that An entity with the same name exists under the same namespace.

+ - `MessagingEntityDisabled`: The Messaging Entity is disabled. Enable the entity again using Portal.

+ - `MessagingEntityNotFound`: Service Bus service can't find a Service Bus resource.

+## Handle ServiceBusException - example

+Here's an example of how to handle a `ServiceBusException` and filter by the `Reason`.

+```csharp

+try

+{

+ // Receive messages using the receiver client

+}

+catch (ServiceBusException ex) when

+ (ex.Reason == ServiceBusFailureReason.ServiceTimeout)

+{

+ // Take action based on a service timeout

+}

+```

+### Other common exceptions

+- `ArgumentException`: Client throws this exception deriving from `ArgumentException` when a parameter provided when interacting with the client is invalid. Information about the specific parameter and the nature of the problem can be found in the `Message`.

+- `InvalidOperationException`: Occurs when attempting to perform an operation that isn't valid for its current configuration. This exception typically occurs when a client wasn't configured to support the operation. Often, it can be mitigated by adjusting the options passed to the client.

+- `NotSupportedException`: Occurs when a requested operation is valid for the client, but not supported by its current state. Information about the scenario can be found in the `Message`.

+- `AggregateException`: Occurs when an operation might encounter multiple exceptions and is surfacing them as a single failure. This exception is most commonly encountered when starting or stopping the Service Bus processor or Service Bus session processor.

+## Reason: QuotaExceeded

+[ServiceBusException](/dotnet/api/azure.messaging.servicebus.servicebusexception) with reason set to `QuotaExceeded` indicates that a quota for a specific entity has been exceeded.

+> [!NOTE]

+> For Service Bus quotas, see [Quotas](service-bus-quotas.md).

+### Queues and topics

+For queues and topics, it's often the size of the queue. The error message property contains further details, as in the following example:

+```output

+Message: The maximum entity size has been reached or exceeded for Topic: 'xxx-xxx-xxx'.

+ Size of entity in bytes:1073742326, Max entity size in bytes:

+1073741824..TrackingId:xxxxxxxxxxxxxxxxxxxxxxxxxx, TimeStamp:3/15/2013 7:50:18 AM

+```

+The message states that the topic exceeded its size limit, in this case 1 GB (the default size limit).

+### Namespaces

+For namespaces, QuotaExceeded exception can indicate that an application has exceeded the maximum number of connections to a namespace. For example:

+```output

+<tracking-id-guid>_G12 >

+System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]:

+ConnectionsQuotaExceeded for namespace xxx.

+```

+### Common causes

+There are two common causes for this error: the dead-letter queue, and nonfunctioning message receivers.

+- **[Dead-letter queue](service-bus-dead-letter-queues.md)**

+ A reader is failing to complete messages and the messages are returned to the queue/topic when the lock expires. It can happen if the reader encounters an exception that prevents it from completing the message. After a message has been read 10 times, it moves to the dead-letter queue by default. This behavior is controlled by the [MaxDeliveryCount](/dotnet/api/azure.messaging.servicebus.administration.queueproperties.maxdeliverycount) property and has a default value of 10. As messages pile up in the dead letter queue, they take up space.

+ To resolve the issue, read and complete the messages from the dead-letter queue, as you would from any other queue.

+- **Receiver stopped**. A receiver has stopped receiving messages from a queue or subscription. The way to identify the issue is to look at the [active message count](/dotnet/api/azure.messaging.servicebus.administration.queueruntimeproperties.activemessagecount). If the active message count is high or growing, then the messages aren't being read as fast as they're being written.

+## Reason: MessageLockLost

+### Cause

+[ServiceBusException](/dotnet/api/azure.messaging.servicebus.servicebusexception) with reason set to `MessageLockLost` indicates that a message is received using the [PeekLock](message-transfers-locks-settlement.md#peeklock) Receive mode and the lock held by the client expires on the service side.

+The lock on a message might expire due to various reasons:

+ * The lock timer has expired before it was renewed by the client application.

+ * The client application acquired the lock, saved it to a persistent store and then restarted. Once it restarted, the client application looked at the inflight messages and tried to complete the messages.

+You might also receive this exception in the following scenarios:

+* Service Update

+* OS update

+* Changing properties on the entity (queue, topic, subscription) while holding the lock.

+### Resolution

+When a client application receives **MessageLockLostException**, it can no longer process the message. The client application might optionally consider logging the exception for analysis, but the client *must* dispose off the message.

+Since the lock on the message has expired, it would go back on the Queue (or Subscription) and can be processed by the next client application that calls receive.

+If the **MaxDeliveryCount** has exceeded, then the message might be moved to the **DeadLetterQueue**.

+## Reason: SessionLockLost

+### Cause

+[ServiceBusException](/dotnet/api/azure.messaging.servicebus.servicebusexception) with reason set to `MessageLockLost` is thrown when a session is accepted and the lock held by the client expires on the service side.

+The lock on a session might expire due to various reasons:

+ * The lock timer has expired before it was renewed by the client application.

+ * The client application acquired the lock, saved it to a persistent store and then restarted. Once it restarted, the client application looked at the inflight sessions and tried to process the messages in those sessions.

+You might also receive this exception in the following scenarios:

+* Service Update

+* OS update

+* Changing properties on the entity (queue, topic, subscription) while holding the lock.

+### Resolution

+When a client application receives **SessionLockLostException**, it can no longer process the messages on the session. The client application might consider logging the exception for analysis, but the client *must* dispose off the message.

+Since the lock on the session has expired, it would go back on the Queue (or Subscription) and can be locked by the next client application that accepts the session. Since the session lock is held by a single client application at any given time, the in-order processing is guaranteed.

+## TimeoutException

+A [TimeoutException](/dotnet/api/system.timeoutexception) indicates that a user-initiated operation is taking longer than the operation timeout.

+You should check the value of the [ServicePointManager.DefaultConnectionLimit](/dotnet/api/system.net.servicepointmanager.defaultconnectionlimit) property, as hitting this limit can also cause a [TimeoutException](/dotnet/api/system.timeoutexception).

+Timeouts are expected to happen during or in-between maintenance operations such as Service Bus service updates (or) OS updates on resources that run the service. During OS updates, entities are moved around and nodes are updated or rebooted, which can cause timeouts. For service level agreement (SLA) details for the Azure Service Bus service, see [SLA for Service Bus](https://azure.microsoft.com/support/legal/sla/service-bus/).

+## SocketException

+### Cause

+A **SocketException** is thrown in the following cases:

+ * When a connection attempt fails because the host didn't properly respond after a specified time (TCP error code 10060).

+ * An established connection failed because connected host has failed to respond.

+ * There was an error processing the message or the timeout is exceeded by the remote host.

+ * Underlying network resource issue.

+### Resolution

+The **SocketException** errors indicate that the VM hosting the applications is unable to convert the name `<mynamespace>.servicebus.windows.net` to the corresponding IP address.

+Check to see if the following command succeeds in mapping to an IP address.

+```powershell

+PS C:\> nslookup <mynamespace>.servicebus.windows.net

+```

+Which should provide an output like:

+```bash

+Name: <cloudappinstance>.cloudapp.net

+Address: XX.XX.XXX.240

+Aliases: <mynamespace>.servicebus.windows.net

+```

+If the above name **does not resolve** to an IP and the namespace alias, check with the network administrator to investigate further. Name resolution is done through a DNS server typically a resource in the customer network. If the DNS resolution is done by Azure DNS, contact Azure support.

+If name resolution **works as expected**, check if connections to Azure Service Bus is allowed [here](service-bus-troubleshooting-guide.md#connectivity-certificate-or-timeout-issues).

+## Next steps

+For the complete Service Bus .NET API reference, see the [Azure .NET API reference](/dotnet/api/overview/azure/service-bus).

+For troubleshooting tips, see the [Troubleshooting guide](service-bus-troubleshooting-guide.md).

+ Title: Azure Service Bus - messaging exceptions (deprecated) | Microsoft Docs

+description: This article provides a list of Azure Service Bus messaging exceptions from the deprecated packages and suggested actions to taken when the exception occurs.

+# Service Bus messaging exceptions (deprecated)

+This article lists the .NET exceptions generated by .NET Framework APIs.

+## Exception categories

+The messaging APIs generate exceptions that can fall into the following categories, along with the associated action you can take to try to fix them. The meaning and causes of an exception can vary depending on the type of messaging entity:

+1. User coding error ([System.ArgumentException](/dotnet/api/system.argumentexception), [System.InvalidOperationException](/dotnet/api/system.invalidoperationexception), [System.OperationCanceledException](/dotnet/api/system.operationcanceledexception), [System.Runtime.Serialization.SerializationException](/dotnet/api/system.runtime.serialization.serializationexception)). General action: try to fix the code before proceeding.

+2. Setup/configuration error ([Microsoft.ServiceBus.Messaging.MessagingEntityNotFoundException](/dotnet/api/microsoft.azure.servicebus.messagingentitynotfoundexception), [System.UnauthorizedAccessException](/dotnet/api/system.unauthorizedaccessexception). General action: review your configuration and change if necessary.

+3. Transient exceptions ([Microsoft.ServiceBus.Messaging.MessagingException](/dotnet/api/microsoft.servicebus.messaging.messagingexception), [Microsoft.ServiceBus.Messaging.ServerBusyException](/dotnet/api/microsoft.azure.servicebus.serverbusyexception), [Microsoft.ServiceBus.Messaging.MessagingCommunicationException](/dotnet/api/microsoft.servicebus.messaging.messagingcommunicationexception)). General action: retry the operation or notify users. The `RetryPolicy` class in the client SDK can be configured to handle retries automatically. For more information, see [Retry guidance](/azure/architecture/best-practices/retry-service-specific#service-bus).

+4. Other exceptions ([System.Transactions.TransactionException](/dotnet/api/system.transactions.transactionexception), [System.TimeoutException](/dotnet/api/system.timeoutexception), [Microsoft.ServiceBus.Messaging.MessageLockLostException](/dotnet/api/microsoft.azure.servicebus.messagelocklostexception), [Microsoft.ServiceBus.Messaging.SessionLockLostException](/dotnet/api/microsoft.azure.servicebus.sessionlocklostexception)). General action: specific to the exception type; refer to the table in the following section:

+> [!IMPORTANT]

+> - Azure Service Bus doesn't retry an operation in case of an exception when the operation is in a transaction scope.

+> - For retry guidance specific to Azure Service Bus, see [Retry guidance for Service Bus](/azure/architecture/best-practices/retry-service-specific#service-bus).

+## Exception types

+The following table lists messaging exception types, and their causes, and notes suggested action you can take.

+| **Exception Type** | **Description/Cause/Examples** | **Suggested Action** | **Note on automatic/immediate retry** |

+| | | | |

+| [TimeoutException](/dotnet/api/system.timeoutexception) |The server didn't respond to the requested operation within the specified time, which is controlled by [OperationTimeout](/dotnet/api/microsoft.servicebus.messaging.messagingfactorysettings). The server might have completed the requested operation. It can happen because of network or other infrastructure delays. |Check the system state for consistency and retry if necessary. See [Timeout exceptions](#timeoutexception). |Retry might help in some cases; add retry logic to code. |

+| [InvalidOperationException](/dotnet/api/system.invalidoperationexception) |The requested user operation isn't allowed within the server or service. See the exception message for details. For example, [Complete()](/dotnet/api/microsoft.azure.servicebus.queueclient.completeasync) generates this exception if the message was received in [ReceiveAndDelete](/dotnet/api/microsoft.azure.servicebus.receivemode) mode. |Check the code and the documentation. Make sure the requested operation is valid. |Retry doesn't help. |

+| [OperationCanceledException](/dotnet/api/system.operationcanceledexception) |An attempt is made to invoke an operation on an object that has already been closed, aborted, or disposed. In rare cases, the ambient transaction is already disposed. |Check the code and make sure it doesn't invoke operations on a disposed object. |Retry doesn't help. |

+| [UnauthorizedAccessException](/dotnet/api/system.unauthorizedaccessexception) |The [TokenProvider](/dotnet/api/microsoft.servicebus.tokenprovider) object couldn't acquire a token, the token is invalid, or the token doesn't contain the claims required to do the operation. |Make sure the token provider is created with the correct values. Check the configuration of the Access Control Service. |Retry might help in some cases; add retry logic to code. |

+| [ArgumentException](/dotnet/api/system.argumentexception)<br /> [ArgumentNullException](/dotnet/api/system.argumentnullexception)<br />[ArgumentOutOfRangeException](/dotnet/api/system.argumentoutofrangeexception) |One or more arguments supplied to the method are invalid.<br /> The URI supplied to [NamespaceManager](/dotnet/api/microsoft.servicebus.namespacemanager) or [Create](/dotnet/api/microsoft.servicebus.messaging.messagingfactory) contains path segments.<br /> The URI scheme supplied to [NamespaceManager](/dotnet/api/microsoft.servicebus.namespacemanager) or [Create](/dotnet/api/microsoft.servicebus.messaging.messagingfactory) is invalid. <br />The property value is larger than 32 KB. |Check the calling code and make sure the arguments are correct. |Retry doesn't help. |

+| [MessagingEntityNotFoundException](/dotnet/api/microsoft.azure.servicebus.messagingentitynotfoundexception) |Entity associated with the operation doesn't exist or it has been deleted. |Make sure the entity exists. |Retry doesn't help. |

+| [MessageNotFoundException](/dotnet/api/microsoft.servicebus.messaging.messagenotfoundexception) |Attempt to receive a message with a particular sequence number. This message isn't found. |Make sure the message hasn't been received already. Check the deadletter queue to see if the message has been deadlettered. |Retry doesn't help. |

+| [MessagingCommunicationException](/dotnet/api/microsoft.servicebus.messaging.messagingcommunicationexception) |Client isn't able to establish a connection to Service Bus. |Make sure the supplied host name is correct and the host is reachable. <p>If your code runs in an environment with a firewall/proxy, ensure that the traffic to the Service Bus domain/IP address and ports isn't blocked.</p>|Retry might help if there are intermittent connectivity issues. |

+| [ServerBusyException](/dotnet/api/microsoft.azure.servicebus.serverbusyexception) |Service isn't able to process the request at this time. |Client can wait for a period of time, then retry the operation. |Client might retry after certain interval. If a retry results in a different exception, check retry behavior of that exception. |

+| [MessagingException](/dotnet/api/microsoft.servicebus.messaging.messagingexception) |Generic messaging exception that might be thrown in the following cases:<p>An attempt is made to create a [QueueClient](/dotnet/api/microsoft.azure.servicebus.queueclient) using a name or path that belongs to a different entity type (for example, a topic).</p><p>An attempt is made to send a message larger than 256 KB. </p>The server or service encountered an error during processing of the request. See the exception message for details. It's usually a transient exception.</p><p>The request was terminated because the entity is being throttled. Error code: 50001, 50002, 50008. </p> | Check the code and ensure that only serializable objects are used for the message body (or use a custom serializer). <p>Check the documentation for the supported value types of the properties and only use supported types.</p><p> Check the [IsTransient](/dotnet/api/microsoft.servicebus.messaging.messagingexception) property. If it's **true**, you can retry the operation. </p>| If the exception is due to throttling, wait for a few seconds and retry the operation again. Retry behavior is undefined and might not help in other scenarios.|

+| [MessagingEntityAlreadyExistsException](/dotnet/api/microsoft.servicebus.messaging.messagingentityalreadyexistsexception) |Attempt to create an entity with a name that is already used by another entity in that service namespace. |Delete the existing entity or choose a different name for the entity to be created. |Retry doesn't help. |

+| [QuotaExceededException](/dotnet/api/microsoft.azure.servicebus.quotaexceededexception) |The messaging entity has reached its maximum allowable size, or the maximum number of connections to a namespace has been exceeded. |Create space in the entity by receiving messages from the entity or its subqueues. See [QuotaExceededException](#quotaexceededexception). |Retry might help if messages have been removed in the meantime. |

+| [RuleActionException](/dotnet/api/microsoft.servicebus.messaging.ruleactionexception) |Service Bus returns this exception if you attempt to create an invalid rule action. Service Bus attaches this exception to a deadlettered message if an error occurs while processing the rule action for that message. |Check the rule action for correctness. |Retry doesn't help. |

+| [FilterException](/dotnet/api/microsoft.servicebus.messaging.filterexception) |Service Bus returns this exception if you attempt to create an invalid filter. Service Bus attaches this exception to a deadlettered message if an error occurred while processing the filter for that message. |Check the filter for correctness. |Retry doesn't help. |

+| [SessionCannotBeLockedException](/dotnet/api/microsoft.servicebus.messaging.sessioncannotbelockedexception) |Attempt to accept a session with a specific session ID, but the session is currently locked by another client. |Make sure the session is unlocked by other clients. |Retry might help if the session has been released in the interim. |

+| [TransactionSizeExceededException](/dotnet/api/microsoft.servicebus.messaging.transactionsizeexceededexception) |Too many operations are part of the transaction. |Reduce the number of operations that are part of this transaction. |Retry doesn't help. |

+| [MessagingEntityDisabledException](/dotnet/api/microsoft.azure.servicebus.messagingentitydisabledexception) |Request for a runtime operation on a disabled entity. |Activate the entity. |Retry might help if the entity has been activated in the interim. |

+| [NoMatchingSubscriptionException](/dotnet/api/microsoft.servicebus.messaging.nomatchingsubscriptionexception) |Service Bus returns this exception if you send a message to a topic that has prefiltering enabled and none of the filters match. |Make sure at least one filter matches. |Retry doesn't help. |

+| [MessageSizeExceededException](/dotnet/api/microsoft.servicebus.messaging.messagesizeexceededexception) |A message payload exceeds the 256-KB limit. The 256-KB limit is the total message size, which can include system properties and any .NET overhead. |Reduce the size of the message payload, then retry the operation. |Retry doesn't help. |

+| [TransactionException](/dotnet/api/system.transactions.transactionexception) |The ambient transaction (`Transaction.Current`) is invalid. It might have been completed or aborted. Inner exception might provide additional information. | |Retry doesn't help. |

+| [TransactionInDoubtException](/dotnet/api/system.transactions.transactionindoubtexception) |An operation is attempted on a transaction that is in doubt, or an attempt is made to commit the transaction and the transaction becomes in doubt. |Your application must handle this exception (as a special case), as the transaction might have already been committed. |- |

+## QuotaExceededException

+[QuotaExceededException](/dotnet/api/microsoft.azure.servicebus.quotaexceededexception) indicates that a quota for a specific entity has been exceeded.

+> [!NOTE]

+> For Service Bus quotas, see [Quotas](service-bus-quotas.md).

+### Queues and topics

+For queues and topics, it's often the size of the queue. The error message property contains further details, as in the following example:

+```output

+Microsoft.ServiceBus.Messaging.QuotaExceededException

+Message: The maximum entity size has been reached or exceeded for Topic: 'xxx-xxx-xxx'.

+ Size of entity in bytes:1073742326, Max entity size in bytes:

+1073741824..TrackingId:xxxxxxxxxxxxxxxxxxxxxxxxxx, TimeStamp:3/15/2013 7:50:18 AM

+```

+The message states that the topic exceeded its size limit, in this case 1 GB (the default size limit).

+### Namespaces

+For namespaces, [QuotaExceededException](/dotnet/api/microsoft.azure.servicebus.quotaexceededexception) can indicate that an application has exceeded the maximum number of connections to a namespace. For example:

+```output

+Microsoft.ServiceBus.Messaging.QuotaExceededException: ConnectionsQuotaExceeded for namespace xxx.

+<tracking-id-guid>_G12 >

+System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]:

+ConnectionsQuotaExceeded for namespace xxx.

+```

+### Common causes

+There are two common causes for this error: the dead-letter queue, and nonfunctioning message receivers.

+1. **[Dead-letter queue](service-bus-dead-letter-queues.md)**

+ A reader is failing to complete messages and the messages are returned to the queue/topic when the lock expires. It can happen if the reader encounters an exception that prevents it from calling [BrokeredMessage.Complete](/dotnet/api/microsoft.servicebus.messaging.brokeredmessage.complete). After a message has been read 10 times, it moves to the dead-letter queue by default. This behavior is controlled by the [QueueDescription.MaxDeliveryCount](/dotnet/api/microsoft.servicebus.messaging.queuedescription.maxdeliverycount) property and has a default value of 10. As messages pile up in the dead letter queue, they take up space.

+ To resolve the issue, read and complete the messages from the dead-letter queue, as you would from any other queue. You can use the [FormatDeadLetterPath](/dotnet/api/microsoft.azure.servicebus.entitynamehelper.formatdeadletterpath) method to help format the dead-letter queue path.

+2. **Receiver stopped**. A receiver has stopped receiving messages from a queue or subscription. The way to identify this is to look at the [QueueDescription.MessageCountDetails](/dotnet/api/microsoft.servicebus.messaging.messagecountdetails) property, which shows the full breakdown of the messages. If the [ActiveMessageCount](/dotnet/api/microsoft.servicebus.messaging.messagecountdetails.activemessagecount) property is high or growing, then the messages aren't being read as fast as they're being written.

+## TimeoutException

+A [TimeoutException](/dotnet/api/system.timeoutexception) indicates that a user-initiated operation is taking longer than the operation timeout.

+You should check the value of the [ServicePointManager.DefaultConnectionLimit](/dotnet/api/system.net.servicepointmanager.defaultconnectionlimit) property, as hitting this limit can also cause a [TimeoutException](/dotnet/api/system.timeoutexception).

+Timeouts are expected to happen during or in-between maintenance operations such as Service Bus service updates (or) OS updates on resources that run the service. During OS updates, entities are moved around and nodes are updated or rebooted, which can cause timeouts. For service level agreement (SLA) details for the Azure Service Bus service, see [SLA for Service Bus](https://azure.microsoft.com/support/legal/sla/service-bus/).

+### Queues and topics

+For queues and topics, the timeout is specified either in the [MessagingFactorySettings.OperationTimeout](/dotnet/api/microsoft.servicebus.messaging.messagingfactorysettings) property, as part of the connection string, or through [ServiceBusConnectionStringBuilder](/dotnet/api/microsoft.azure.servicebus.servicebusconnectionstringbuilder). The error message itself might vary, but it always contains the timeout value specified for the current operation.

+## MessageLockLostException

+### Cause

+The **MessageLockLostException** is thrown when a message is received using the [PeekLock](message-transfers-locks-settlement.md#peeklock) Receive mode and the lock held by the client expires on the service side.

+The lock on a message might expire due to various reasons:

+ * The lock timer has expired before it was renewed by the client application.

+ * The client application acquired the lock, saved it to a persistent store and then restarted. Once it restarted, the client application looked at the inflight messages and tried to complete these.

+You might also receive this exception in the following scenarios:

+* Service Update

+* OS update

+* Changing properties on the entity (queue, topic, subscription) while holding the lock.

+### Resolution

+When a client application receives **MessageLockLostException**, it can no longer process the message. The client application might optionally consider logging the exception for analysis, but the client *must* dispose off the message.

+Since the lock on the message has expired, it would go back on the Queue (or Subscription) and can be processed by the next client application that calls receive.

+If the MaxDeliveryCount has exceeded, then the message might be moved to the **DeadLetterQueue**.

+## SessionLockLostException

+### Cause

+The **SessionLockLostException** is thrown when a session is accepted and the lock held by the client expires on the service side.

+The lock on a session might expire due to various reasons:

+ * The lock timer has expired before it was renewed by the client application.

+ * The client application acquired the lock, saved it to a persistent store and then restarted. Once it restarted, the client application looked at the inflight sessions and tried to process the messages in those sessions.

+You might also receive this exception in the following scenarios:

+* Service Update

+* OS update

+* Changing properties on the entity (queue, topic, subscription) while holding the lock.

+### Resolution

+When a client application receives **SessionLockLostException**, it can no longer process the messages on the session. The client application might consider logging the exception for analysis, but the client *must* dispose off the message.

+Since the lock on the session has expired, it would go back on the Queue (or Subscription) and can be locked by the next client application that accepts the session. Since the session lock is held by a single client application at any given time, the in-order processing is guaranteed.

+## SocketException

+### Cause

+A **SocketException** is thrown in the following cases:

+ * When a connection attempt fails because the host didn't properly respond after a specified time (TCP error code 10060).

+ * An established connection failed because connected host has failed to respond.

+ * There was an error processing the message or the timeout is exceeded by the remote host.

+ * Underlying network resource issue.

+### Resolution

+The **SocketException** errors indicate that the VM hosting the applications is unable to convert the name `<mynamespace>.servicebus.windows.net` to the corresponding IP address.

+Check to see if the following command succeeds in mapping to an IP address.

+```powershell

+PS C:\> nslookup <mynamespace>.servicebus.windows.net

+```

+Which should provide an output like:

+```bash

+Name: <cloudappinstance>.cloudapp.net

+Address: XX.XX.XXX.240

+Aliases: <mynamespace>.servicebus.windows.net

+```

+If the above name **does not resolve** to an IP and the namespace alias, check with the network administrator to investigate further. Name resolution is done through a DNS server typically a resource in the customer network. If the DNS resolution is done by Azure DNS, contact Azure support.

+If name resolution **works as expected**, check if connections to Azure Service Bus is allowed [here](service-bus-troubleshooting-guide.md#connectivity-certificate-or-timeout-issues).

+## MessagingException

+### Cause

+**MessagingException** is a generic exception that might be thrown for various reasons. Some of the reasons are:

+ * An attempt is made to create a **QueueClient** on a **Topic** or a **Subscription**.

+ * The size of the message sent is greater than the limit for the given tier. Read more about the Service Bus [quotas and limits](service-bus-quotas.md).

+ * Specific data plane request (send, receive, complete, abandon) was terminated due to throttling.

+ * Transient issues caused due to service upgrades and restarts.

+> [!NOTE]

+> The above list of exceptions is not exhaustive.

+### Resolution

+The resolution steps depend on what caused the **MessagingException** to be thrown.

+ * For **transient issues** (where ***isTransient*** is set to ***true***) or for **throttling issues**, retrying the operation might resolve it. The default retry policy on the SDK can be used for this.

+ * For other issues, the details in the exception indicate the issue and resolution steps can be deduced from the same.

+## StorageQuotaExceededException

+### Cause

+The **StorageQuotaExceededException** is generated when the total size of entities in a premium namespace exceeds the limit of 1 TB per [messaging unit](service-bus-premium-messaging.md).

+### Resolution

+- Increase the number of messaging units assigned to the premium namespace

+- If you're already using maximum allowed messaging units for a namespace, create a separate namespace.

+## Next steps

+For the complete Service Bus .NET API reference, see the [Azure .NET API reference](/dotnet/api/overview/azure/service-bus).

+For troubleshooting tips, see the [Troubleshooting guide](service-bus-troubleshooting-guide.md).

+ Title: Create connections with IaC tools

+description: Learn how to translate your infrastructure to an IaC template

+# How to translate your infrastructure to an IaC template

+Service Connector helps users connect their compute services to target backing services in just a few clicks or commands. When moving from a getting-started to a production stage, users also need to make the transition from using manual configurations to using Infrastructure as Code (IaC) templates in their CI/CD pipelines. In this guide, we show how to translate your connected Azure services to IaC templates.

+## Prerequisites

+- This guide assumes that you're aware of the [Service Connector IaC limitations](./known-limitations.md).

+## Solution overview

+Translating the infrastructure to IaC templates usually involves two major parts: the logics to provision source and target services, and the logics to build connections. To implement the logics to provision source and target services, there are two options:

+* Authoring the template from scratch.

+* Exporting the template from Azure and polish it.

+To implement the logics to build connections, there are also two options:

+* Using Service Connector in the template.

+* Using template logics to configure source and target services directly.

+Combinations of these different options can produce different solutions. Due to [IaC limitations](./known-limitations.md) in Service Connector, we recommend that you implement the following solutions in the order presented below. To apply these solutions, you must understand the IaC tools and the template authoring grammar.

+| Solution | Provision source and target | Build connection | Applicable scenario | Pros | Cons |

+| :: | :-: | :-: | :-: | - | - |

+| 1 | Authoring from scratch | Use Service Connector | Has liveness check on the cloud resources before allowing live traffics | - Template is simple and readable<br />- Service Connector brings extra values | - Cost to check cloud resources liveness |

+| 2 | Authoring from scratch | Configure source and target services directly in template | No liveness check on the cloud resources | - Template is simple and readable | - Service Connector features aren't available |

+| 3 | Export and polish | Use Service Connector | Has liveness check on the cloud resources before allowing live traffics | - Resources are exactly the same as in the cloud<br />- Service Connector brings extra values | - Cost to check cloud resources liveness<br />- Supports only ARM templates<br />- Efforts required to understand and polish the template |

+| 4 | Export and polish | Configure source and target services directly in template | No liveness check on the cloud resources | - Resources are exactly same as on the cloud | - Support only ARM template<br />- Efforts to understand and polish the template<br />- Service Connector features aren't available |

+## Authoring templates

+The following sections show how to create a web app and a storage account and connect them with a system-assigned identity using Bicep. It shows how to do this both using Service Connector and using template logics.

+### Provision source and target services

+**Authoring from scratch**

+Authoring the template from scratch is the preferred and recommended way to provision source and target services, as it's easy to get started and makes the template simple and readable. Following is an example, using a minimal set of parameters to create a webapp and a storage account.

+```bicep

+// This template creates a webapp and a storage account.

+// In order to make it more readable, we use only the mininal set of parameters to create the resources.

+param location string = resourceGroup().location

+// App Service plan parameters

+param planName string = 'plan_${uniqueString(resourceGroup().id)}'

+param kind string = 'linux'

+param reserved bool = true

+param sku string = 'B1'

+// Webapp parameters

+param webAppName string = 'webapp-${uniqueString(resourceGroup().id)}'

+param linuxFxVersion string = 'PYTHON|3.8'

+param identityType string = 'SystemAssigned'

+param appSettings array = []

+// Storage account parameters

+param storageAccountName string = 'account${uniqueString(resourceGroup().id)}'

+// Create an app service plan

+resource appServicePlan 'Microsoft.Web/serverfarms@2022-09-01' = {

+ name: planName

+ location: location

+ kind: kind

+ sku: {

+ name: sku

+ }

+ properties: {

+ reserved: reserved

+ }

+}

+// Create a web app

+resource appService 'Microsoft.Web/sites@2022-09-01' = {

+ name: webAppName

+ location: location

+ properties: {

+ serverFarmId: appServicePlan.id

+ siteConfig: {

+ linuxFxVersion: linuxFxVersion

+ appSettings: appSettings

+ }

+ }

+ identity: {

+ type: identityType

+ }

+}

+// Create a storage account

+resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {

+ name: storageAccountName

+ location: location

+ sku: {

+ name: 'Standard_LRS'

+ }

+ kind: 'StorageV2'

+}

+```

+**Export and polish**

+If the resources you're provisioning are exactly the same ones as the ones you have in the cloud, exporting the template from Azure might be another option. The two premises of this approach are: the resources exist in Azure and you're using ARM templates for your IaC. The `Export template` button is usually at the bottom of the sidebar on Azure portal. The exported ARM template reflects the resource's current states, including the settings configured by Service Connector. You usually need to know about the resource properties to polish the exported template.

+### Build connection logics

+**Using Service Connector**

+Creating connections between the source and target service using Service Connector is the preferred and recommended way if the [Service Connector ](./known-limitations.md)[IaC limitation](./known-limitations.md) doesn't matter for your scenario. Service Connector makes the template simpler and also provides additional elements, such as the connection health validation, which you won't have if you're building connections through template logics directly.

+```bicep

+// The template builds a connection between a webapp and a storage account

+// with a system-assigned identity using Service Connector

+param webAppName string = 'webapp-${uniqueString(resourceGroup().id)}'

+param storageAccountName string = 'account${uniqueString(resourceGroup().id)}'

+param connectorName string = 'connector_${uniqueString(resourceGroup().id)}'

+// Get an existing webapp

+resource webApp 'Microsoft.Web/sites@2022-09-01' existing = {

+ name: webAppName

+}

+// Get an existig storage

+resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' existing = {

+ name: storageAccountName

+}

+// Create a Service Connector resource for the webapp

+// to connect to a storage account using system identity

+resource serviceConnector 'Microsoft.ServiceLinker/linkers@2022-05-01' = {

+ name: connectorName

+ scope: webApp

+ properties: {

+ clientType: 'python'

+ targetService: {

+ type: 'AzureResource'

+ id: storageAccount.id

+ }

+ authInfo: {

+ authType: 'systemAssignedIdentity'

+ }

+ }

+}

+```

+For the formats of properties and values needed when creating a Service Connector resource, check [how to provide correct parameters](./how-to-provide-correct-parameters.md). You can also preview and download an ARM template for reference when creating a Service Connector resource in the Azure portal.

+**Using template logics**

+For the scenarios where the Service Connector [IaC limitation](./known-limitations.md) matters, consider building connections using the template logics directly. The following template is an example showing how to connect a storage account to a web app using a system-assigned identity.

+```bicep

+// The template builds a connection between a webapp and a storage account

+// with a system-assigned identity without using Service Connector

+param webAppName string = 'webapp-${uniqueString(resourceGroup().id)}'

+param storageAccountName string = 'account${uniqueString(resourceGroup().id)}'

+param storageBlobDataContributorRole string = 'ba92f5b4-2d11-453d-a403-e96b0029c9fe'

+// Get an existing webapp

+resource webApp 'Microsoft.Web/sites@2022-09-01' existing = {

+ name: webAppName

+}

+// Get an existing storage account

+resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' existing = {

+ name: storageAccountName

+}

+// Operation: Enable system-assigned identity on the source service

+// No action needed as this is enabled when creating the webapp

+// Operation: Configure the target service's endpoint on the source service's app settings

+resource appSettings 'Microsoft.Web/sites/config@2022-09-01' = {

+ name: 'appsettings'

+ parent: webApp

+ properties: {

+ AZURE_STORAGEBLOB_RESOURCEENDPOINT: storageAccount.properties.primaryEndpoints.blob

+ }

+}

+// Operation: Configure firewall on the target service to allow the source service's outbound IPs

+// No action needed as storage account allows all IPs by default

+// Operation: Create role assignment for the source service's identity on the target service

+resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {

+ scope: storageAccount

+ name: guid(resourceGroup().id, storageBlobDataContributorRole)

+ properties: {

+ roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', storageBlobDataContributorRole)

+ principalId: webApp.identity.principalId

+ }

+}

+```

+When building connections using template logics directly, it's crucial to understand what Service Connector does for each kind of authentication type, as the template logics are equivalent to the Service Connector backend operations. The following table shows the operation details that you need translate to template logics for each kind of authentication type.

+| Auth type | Service Connector operations |

+| -- | - |

+| Secret / Connection string | - Configure the target service's connection string on the source service's app settings<br />- Configure firewall on the target service to allow the source service's outbound IPs |

+| System-assigned managed identity | - Configure the target service's endpoint on the source service's app settings<br />- Configure firewall on the target service to allow the source service's outbound IPs<br />- Enable system assigned identity on the source service<br />- Create role assignment for the source service's identity on the target service |

+| User-assigned managed identity | - Configure the target service's endpoint on the source service's app settings<br />- Configure firewall on the target service to allow the source service's outbound IPs<br />- Bind user assigned identity to the source service<br />- Create role assignment for the user assigned identity on the target service |

+| Service principal | - Configure the target service's endpoint on the source service's app settings<br />- Configure the service principal's appId and secret on the source service's app settings<br />- Configure firewall on the target service to allow the source service's outbound IPs<br />- Create role assignment for the service principal on the target service |

+If you run into any issues when using Service Connector, [file an issue with us](https://github.com/Azure/ServiceConnector/issues/new).

+We suggest the following solutions:

+- Reference [how to build connections with IaC tools](how-to-build-connections-with-iac-tools.md) to build your infrastructure or translate your existing infrastructure to IaC templates.

+- If your CI/CD pipelines contain templates of source compute or backing services, suggested flow is: reapplying the templates, adding sanity check or smoke tests to make sure the application is up and running, then allowing live traffic to the application. The flow adds a verification step before allowing live traffic.

+- The order in which automation operations are performed matters greatly. Ensure your connection endpoints are there before the connection itself is created. Ideally, create the backing service, then the compute service, and then the connection between the two. This way, Service Connector can configure both the compute service and the backing service appropriately.

+* [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md)

+To track anonymous requests to a storage account, use Azure Metrics Explorer in the Azure portal. For more information about Metrics Explorer, see [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md).

+You can analyze metrics for Azure Storage with metrics from other Azure services by using Metrics Explorer. Open Metrics Explorer by choosing **Metrics** from the **Azure Monitor** menu. For details on using this tool, see [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md).

+| [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md) | A tour of Metrics Explorer.

+| Public key |ssh-rsa ²<br>rsa-sha2-256<br>rsa-sha2-512<br>ecdsa-sha2-nistp256<br>ecdsa-sha2-nistp384<br>ecdsa-sha2-nistp521|

+Versioning is not supported for blobs that are uploaded by using [Data Lake Storage Gen2](/rest/api/storageservices/data-lake-storage-gen2) APIs.

+To track how requests to a storage account are being authorized, use Azure Metrics Explorer in the Azure portal. For more information about Metrics Explorer, see [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md).

+For details on how to use metrics, see [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md).

+You can analyze metrics for Azure Storage with metrics from other Azure services by using Metrics Explorer. Open Metrics Explorer by choosing **Metrics** from the **Azure Monitor** menu. For details on using this tool, see [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md).

+ File Explorer calls an RPC API directly to the server (Azure Files) to translate the SID to a UPN. Azure Files doesn't support this API, so in File Explorer, the SID of a file/directory owner is displayed instead of the UPN for files and directories hosted on Azure Files. However, you can use the following PowerShell command to view all items in a directory and their owner, including UPN:

+Users that leverage Active Directory Domain Services (AD DS) as their on-premises domain controller can natively access an Azure file share. So can users of Microsoft Entra Domain Services. Each uses their current identity to get access based on share permissions and on file and folder ACLs. This behavior is similar to a user connecting to an on-premises file share.

+There are several file-copy tools available from Microsoft and others. To select the right tool for your migration scenario, consider these fundamental questions:

+ By mirroring a source to a target (as with **robocopy /MIR**), you can run the tool again on that same source and target. This second run is much faster because it needs to transport only source changes that happened after the previous run. Rerunning a copy tool this way can reduce downtime significantly.

+| | Supported. | Full fidelity.* |

+| to load files onto the device)| Supported. </br>(Data Box Disks doesn't support large file shares) | Data Box and Data Box Heavy fully support metadata. </br>Data Box Disks does not preserve file metadata. |

+*\* Full fidelity: meets or exceeds Azure file share capabilities.*

+#### RoboCopy

+Included in Windows, RoboCopy is one of the tools most applicable to file migrations. The main [RoboCopy documentation](/windows-server/administration/windows-commands/robocopy) is a helpful resource for this tool's many options.

+#### Azure Storage Migration Program

+Understanding your data is the first step in selecting the appropriate Azure storage service and migration strategy. Azure Storage Migration Program provides different tools that can analyze your data and storage infrastructure to provide valuable insights. These tools can help you understand the size and type of data, file and folder count, and access patterns. They provide a consolidated view of your data and enable the creation of various customized reports.

+This information can help:

+- Identify duplicate and redundant data sets

+- Identify colder data that can be moved to less expensive storage

+To learn more, see [Comparison Matrix for Azure Storage Migration Program participants](../solution-integration/validated-partners/data-management/azure-file-migration-program-solutions.md).

+1. Review the list of available migration guides to find the guide that matches your source and deployment of Azure file shares.

+- [Azure file share overview](storage-files-introduction.md)

+- [Planning for an Azure File Sync deployment](../file-sync/file-sync-planning.md)

+- [Azure File Sync: Cloud tiering](../file-sync/file-sync-cloud-tiering-overview.md)

+You can analyze metrics for Azure Storage with metrics from other Azure services by using Azure Metrics Explorer. Open Metrics Explorer by choosing **Metrics** from the **Azure Monitor** menu. For details on using this tool, see [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md).

+| [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md) | A tour of Metrics Explorer.

+You can analyze metrics for Azure Storage with metrics from other Azure services by using Metrics Explorer. Open Metrics Explorer by choosing **Metrics** from the **Azure Monitor** menu. For details on using this tool, see [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md).

+You can analyze metrics for **Azure Stream Analytics** with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+ ORDER BY PassengerCount;

+> This feature is only available to Azure Synapse workspaces associated with [Azure Synapse Analytics Managed Virtual Network](synapse-workspace-managed-vnet.md). However, you can still open your Synapse workspaces to the public network regardless of its association with managed VNet.

+>

+> When the public network access is disabled, access to GIT mode in Synapse Studio and commit changes won't be blocked as long as the user has enough permission to access the integrated Git repo or the corresponding Git branch. However, the publish button won't work because the access to Live mode is blocked by the firewall settings.

+For details on how to configure the Azure Metrics Explorer see the [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json) article. See the [Resource utilization](sql-data-warehouse-concept-resource-utilization-query-activity.md#resource-utilization) section in Azure Synapse Analytics Monitoring documentation for details on how to monitor system resource consumption.

+You can analyze metrics for Azure Time Series Insights, along with metrics from other Azure services, by opening Metrics from the Azure Monitor menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+- Learn how to [create a chart in Azure Monitor](../azure-monitor/essentials/analyze-metrics.md#create-a-metric-chart)

+The following control plane operations on Managed Disks may involve movement of the Disk from one Storage location to another. This is orchestrated via background copy of data that can take several hours to complete, typically less than 24 hours depending on the amount of data in the disks. During that time your application can experience higher than usual read latency as some reads can get redirected to the original location and can take longer to complete. There is no impact on write latency during this period. For Premium SSD v2 and Ultra disks, if the disk has a 4k sector size, it would experience higher read latency. If the disk has a 512e sector size, it would experience both higher read and write latency.

+You can analyze metrics for *Public IP Addresses* with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+> For the following examples, the process applies mainly to Virtual Machines & VM Scale Sets resources (`Microsoft.Compute/virtualMachines` & `Microsoft.Compute/virtualMachineScaleSets`). It's possible to use port 25 for outbound communication on [Azure App Service](https://azure.microsoft.com/services/app-service) and [Azure Functions](https://azure.microsoft.com/services/functions) through the [virtual network integration feature](/azure/app-service/overview-vnet-integration#application-routing) or when using [App Service Environment v3](../app-service/environment/networking.md#network-routing). However, the the following subscription limitations described still apply. Sending email on Port 25 is unsupported for all other Azure Platform-as-a-Service (PaaS) resources.

+For VMs that are deployed in standard Enterprise Agreement subscriptions, the outbound SMTP connections on TCP port 25 aren't blocked. However, there's no guarantee that external domains accept the incoming emails from the VMs. For emails rejected or filtered by the external domains, contact the email service providers of the external domains to resolve the problems. These problems aren't covered by Azure support.

+It's possible to have this block removed. To request to have the block removed, go to the **Cannot send email (SMTP-Port 25)** section of the **Diagnose and Solve** section in the Azure Virtual Network resource in the Azure portal and run the diagnostic. This process exempts the qualified enterprise dev/test subscriptions automatically.

+The Azure platform blocks outbound SMTP connections on TCP port 25 for deployed VMs. This block is to ensure better security for Microsoft partners and customers, protect MicrosoftΓÇÖs Azure platform, and conform to industry standards.

+If you're using a nonenterprise subscription type, we encourage you to use an authenticated SMTP relay service, as outlined earlier in this article.

+If you change your subscription type from Enterprise Agreement to another type of subscription, changes to your deployments might result in outbound SMTP being blocked.

+If you're using an Enterprise Agreement subscription and still need help, [contact support](https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade) to get your problem resolved quickly. Use this issue type: **Technical** > **Virtual Network** > **Cannot send email (SMTP/Port 25)**.

+* See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for additional details on **Azure Monitor Metrics**.

+* See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for more details about **Azure Monitor Metrics**.

+* The virtual network connection that has the NVA BGP connection endpoint must always be associated and propagating to defaultRouteTable. Custom route tables aren't supported at this time.

+* Routes from NVA in a virtual network that are more specific than the virtual network address space, when advertised to the virtual hub through BGP aren't propagated further to on-premises.

+* Traffic destined for addresses in the virtual network directly connected to the virtual hub can't be configured to route through the NVA using BGP peering between the hub and NVA. This is because the virtual hub automatically learns about system routes associated with addresses in the spoke virtual network when the spoke virtual network connection is created. These automatically learned system routes are preferred over routes learned by the hub through BGP.

+* BGP peering between an NVA in a spoke VNet and a secured virtual hub (hub with an integrated security solution) is supported if Routing Intent **is** configured on the hub. BGP peering feature isn't supported for secured virtual hubs where routing intent is **not** configured.

+* When configuring BGP peering with the hub, you'll see two IP addresses. Peering with both these addresses is required. Not peering with both addresses can cause routing issues. The same routes must be advertised to both of these addresses. Advertising different routes will cause routing issues.

+The following steps are required when BGP peering isn't used on the virtual hub:

+The following table shows few entries from Hub 1's effective routes in the defaultRouteTable. Notice that the route for VNET5 (subnet 10.2.1.0/24) and this confirms VNET1 and VNET5 will be able to communicate with each other.

+The following steps are required when BGP peering isn't used on the virtual hub:

+### Is there a way to change the ASN for a VPN gateway?

+No. Virtual WAN does not support ASN changes for VPN gateways.

+You can analyze metrics for VPN Gateway with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.