+ Title: Planning red teaming for large language models (LLMs) and their applications

+description: Learn about how red teaming and adversarial testing are an essential practice in the responsible development of systems and features using large language models (LLMs)

+# Planning red teaming for large language models (LLMs) and their applications

+This guide offers some potential strategies for planning how to set up and manage red teaming for responsible AI (RAI) risks throughout the large language model (LLM) product life cycle.

+## What is red teaming?

+## Why is RAI red teaming an important practice?

+Red teaming is a best practice in the responsible development of systems and features using LLMs. While not a replacement for systematic measurement and mitigation work, red teamers help to uncover and identify harms and, in turn, enable measurement strategies to validate the effectiveness of mitigations.

+While Microsoft has conducted red teaming exercises and implemented safety systems (including [content filters](./content-filter.md) and other [mitigation strategies](./prompt-engineering.md)) for its Azure OpenAI Service models (see this [Overview of responsible AI practices](/legal/cognitive-services/openai/overview)), the context of each LLM application will be unique and you also should conduct red teaming to:

+- Test the LLM base model and determine whether there are gaps in the existing safety systems, given the context of your application.

+- Provide feedback on failures in order to make improvements.

+- Note that red teaming is not a replacement for systematic measurement. A best practice is to complete an initial round of manual red teaming before conducting systematic measurements and implementing mitigations. As highlighted above, the goal of RAI red teaming is to identify harms, understand the risk surface, and develop the list of harms that can inform what needs to be measured and mitigated.

+Here is how you can get started and plan your process of red teaming LLMs. Advance planning is critical to a productive red teaming exercise.

+## Before testing

+### Plan: Who will do the testing

+**Assemble a diverse group of red teamers**

+Determine the ideal composition of red teamers in terms of peopleΓÇÖs experience, demographics, and expertise across disciplines (for example, experts in AI, social sciences, security) for your productΓÇÖs domain. For example, if youΓÇÖre designing a chatbot to help health care providers, medical experts can help identify risks in that domain.

+**Recruit red teamers with both benign and adversarial mindsets**

+**Assign red teamers to harms and/or product features**

+- Assign RAI red teamers with specific expertise to probe for specific types of harms (for example, security subject matter experts can probe for jailbreaks, meta prompt extraction, and content related to cyberattacks).

+- For multiple rounds of testing, decide whether to switch red teamer assignments in each round to get diverse perspectives on each harm and maintain creativity. If switching assignments, allow time for red teamers to get up to speed on the instructions for their newly assigned harm.

+- In later stages, when the application and its UI are developed, you might want to assign red teamers to specific parts of the application (i.e., features) to ensure coverage of the entire application.

+- Consider how much time and effort each red teamer should dedicate (for example, those testing for benign scenarios might need less time than those testing for adversarial scenarios).

+It can be helpful to provide red teamers with:

+ - Clear instructions that could include:

+ - An introduction describing the purpose and goal of the given round of red teaming; the product and features that will be tested and how to access them; what kinds of issues to test for; red teamersΓÇÖ focus areas, if the testing is more targeted; how much time and effort each red teamer should spend on testing; how to record results; and who to contact with questions.

+ - A file or location for recording their examples and findings, including information such as:

+ - The date an example was surfaced; a unique identifier for the input/output pair if available, for reproducibility purposes; the input prompt; a description or screenshot of the output.

+### Plan: What to test

+Because an application is developed using a base model, you may need to test at several different layers:

+- The LLM base model with its safety system in place to identify any gaps that may need to be addressed in the context of your application system. (Testing is usually done through an API endpoint.)

+- Your application. (Testing is best done through a UI.)

+- Both the LLM base model and your application, before and after mitigations are in place.

+The following recommendations help you choose what to test at various points during red teaming:

+- You can begin by testing the base model to understand the risk surface, identify harms, and guide the development of RAI mitigations for your product.

+- Test versions of your product iteratively with and without RAI mitigations in place to assess the effectiveness of RAI mitigations. (Note, manual red teaming might not be sufficient assessmentΓÇöuse systematic measurements as well, but only after completing an initial round of manual red teaming.)

+- Conduct testing of application(s) on the production UI as much as possible because this most closely resembles real-world usage.

+When reporting results, make clear which endpoints were used for testing. When testing was done in an endpoint other than product, consider testing again on the production endpoint or UI in future rounds.

+### Plan: How to test

+**Conduct open-ended testing to uncover a wide range of harms.**

+The benefit of RAI red teamers exploring and documenting any problematic content (rather than asking them to find examples of specific harms) enables them to creatively explore a wide range of issues, uncovering blind spots in your understanding of the risk surface.

+**Create a list of harms from the open-ended testing.**

+- Consider creating a list of harms, with definitions and examples of the harms.

+- Provide this list as a guideline to red teamers in later rounds of testing.

+**Conduct guided red teaming and iterate: Continue probing for harms in the list; identify new harms that surface.**

+Use a list of harms if available and continue testing for known harms and the effectiveness of their mitigations. In the process, you will likely identify new harms. Integrate these into the list and be open to shifting measurement and mitigation priorities to address the newly identified harms.

+Plan which harms to prioritize for iterative testing. Several factors can inform your prioritization, including, but not limited to, the severity of the harms and the context in which they are more likely to surface.

+### Plan: How to record data

+**Decide what data you need to collect and what data is optional.**

+- Decide what data the red teamers will need to record (for example, the input they used; the output of the system; a unique ID, if available, to reproduce the example in the future; and other notes.)

+- Be strategic with what data you are collecting to avoid overwhelming red teamers, while not missing out on critical information.

+**Create a structure for data collection**

+A shared Excel spreadsheet is often the simplest method for collecting red teaming data. A benefit of this shared file is that red teamers can review each otherΓÇÖs examples to gain creative ideas for their own testing and avoid duplication of data.

+## During testing

+**Plan to be on active standby while red teaming is ongoing**

+- Be prepared to assist red teamers with instructions and access issues.

+- Monitor progress on the spreadsheet and send timely reminders to red teamers.

+## After each round of testing

+**Report data**

+Share a short report on a regular interval with key stakeholders that:

+1. Lists the top identified issues.

+2. Provides a link to the raw data.

+3. Previews the testing plan for the upcoming rounds.

+4. Acknowledges red teamers.

+5. Provides any other relevant information.

+**Differentiate between identification and measurement**

+In the report, be sure to clarify that the role of RAI red teaming is to expose and raise understanding of risk surface and is not a replacement for systematic measurement and rigorous mitigation work. It is important that people do not interpret specific examples as a metric for the pervasiveness of that harm.

+Additionally, if the report contains problematic content and examples, consider including a content warning.

+The guidance in this document is not intended to be, and should not be construed as providing, legal advice. The jurisdiction in which you're operating may have various regulatory or legal requirements that apply to your AI system. Be aware that not all of these recommendations are appropriate for every scenario and, conversely, these recommendations may be insufficient for some scenarios.

+|**Retrieved documents** | Specifies the number of top-scoring documents from your data index used to generate responses. You might want to increase the value when you have short documents or want to provide more context. The default value is 5. This is the `topNDocuments` parameter in the API. |

+# [OpenAI Python 0.28.1](#tab/python)

+# [OpenAI Python 1.x](#tab/python-new)

+```python

+import os

+from openai import AzureOpenAI

+client = AzureOpenAI(

+ api_key = os.getenv("AZURE_OPENAI_KEY"),

+ api_version = "2023-05-15",

+ azure_endpoint =os.getenv("AZURE_OPENAI_ENDPOINT")

+)

+response = client.embeddings.create(

+ input = "Your text string goes here",

+ model= "text-embedding-ada-002"

+)

+print(response.model_dump_json(indent=2))

+```

+| `topNDocuments` | number | Optional | 5 | Specifies the number of top-scoring documents from your data index used to generate responses. You might want to increase the value when you have short documents or want to provide more context. This is the *retrieved documents* parameter in Azure OpenAI studio. |

+ <version>1.33.0</version>

+ implementation 'com.microsoft.cognitiveservices.speech:client-sdk-embedded:1.33.0@aar'

+For embedded speech, you need to download the speech recognition models for [speech to text](speech-to-text.md) and voices for [text to speech](text-to-speech.md). Instructions are provided upon successful completion of the [limited access review](https://aka.ms/csgate-embedded-speech) process.

+All text to speech locales [here](language-support.md?tabs=tts) (except fa-IR, Persian (Iran)) are available out of box with either 1 selected female and/or 1 selected male voices. We welcome your input to help us gauge demand for more languages and voices.

+With hybrid speech configuration for [text to speech](text-to-speech.md) (voices), embedded and cloud synthesis are run in parallel and the final result is selected based on response speed. The best result is evaluated again on each new synthesis request.

+For embedded voices, it is essential to note that certain SSML tags may not be currently supported due to differences in the model structure. For detailed information regarding the unsupported SSML tags, refer to the following table.

+ Title: Deploy an AI model on Azure Kubernetes Service (AKS) with the AI toolchain operator (Preview)

+description: Learn how to enable the AI toolchain operator add-on on Azure Kubernetes Service (AKS) to simplify OSS AI model management and deployment.

+# Deploy an AI model on Azure Kubernetes Service (AKS) with the AI toolchain operator (Preview)

+The AI toolchain operator (KAITO) is a managed add-on for AKS that simplifies the experience of running OSS AI models on your AKS clusters. The AI toolchain operator automatically provisions the necessary GPU nodes and sets up the associated inference server as an endpoint server to your AI models. Using this add-on reduces your onboarding time and enables you to focus on AI model usage and development rather than infrastructure setup.

+This article shows you how to enable the AI toolchain operator add-on and deploy an AI model on AKS.

+## Before you begin

+* This article assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for AKS](./concepts-clusters-workloads.md).

+* If you aren't familiar with Microsoft Entra Workload Identity, see the [Workload Identity overview](../active-directory/workload-identities/workload-identities-overview.md).

+* For ***all hosted model inference files*** and recommended infrastructure setup, see the [KAITO GitHub repository](https://github.com/Azure/kaito).

+## Prerequisites

+* If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.

+ * If you have multiple Azure subscriptions, make sure you select the correct subscription in which the resources will be created and charged using the [`az account set`][az-account-set] command.

+ > [!NOTE]

+ > The subscription you use must have GPU VM quota.

+* Azure CLI version 2.47.0 or later installed and configured. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI](/cli/azure/install-azure-cli).

+* Helm v3 installed. For more information, see [Installing Helm](https://helm.sh/docs/intro/install/).

+* The Kubernetes command-line client, kubectl, installed and configured. For more information, see [Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/).

+## Enable the Azure CLI preview extension

+* Enable the Azure CLI preview extension using the [`az extension add`][az-extension-add] command.

+ ```azurecli-interactive

+ az extension add --name aks-preview

+ ```

+### Export environment variables

+* To simplify the configuration steps in this article, you can define environment variables using the following commands. Make sure to replace the placeholder values with your own.

+ ```azurecli-interactive

+ export AZURE_SUBSCRIPTION_ID="mySubscriptionID"

+ export AZURE_RESOURCE_GROUP="myResourceGroup"

+ export CLUSTER_NAME="myClusterName"

+ ```

+## Enable the AI toolchain operator add-on on an AKS cluster

+1. Create an Azure resource group using the [`az group create`][az-group-create] command.

+ ```azurecli-interactive

+ az group create --name AZURE_RESOURCE_GROUP --location eastus

+ ```

+2. Create an AKS cluster with the AI toolchain operator add-on enabled using the [`az aks create`][az-aks-create] command with the `--enable-ai-toolchain-operator`, `--enable-workload-identity`, and `--enable-oidc-issuer` flags.

+ ```azurecli-interactive

+ az aks create --resource-group AZURE_RESOURCE_GROUP --name CLUSTER_NAME --generate-ssh-keys --enable-managed-identity --enable-workload-identity --enable-oidc-issuer --enable-ai-toolchain-operator

+ ```

+ > [!NOTE]

+ > AKS creates a managed identity once you enable the AI toolchain operator add-on. The managed identity is used to access the AI toolchain operator workspace CRD. The AI toolchain operator workspace CRD is used to create and manage AI toolchain operator workspaces.

+ >

+ > AI toolchain operator enablement requires the enablement of workload identity and OIDC issuer.

+## Connect to your cluster

+1. Configure `kubectl` to connect to your cluster using the [`az aks get-credentials`][az-aks-get-credentials] command.

+ ```azurecli-interactive

+ az aks get-credentials --resource-group AZURE_RESOURCE_GROUP --name CLUSTER_NAME

+ ```

+2. Verify the connection to your cluster using the `kubectl get` command.

+ ```azurecli-interactive

+ kubectl get nodes

+ ```

+3. Export environment variables for the principal ID identity and client ID identity using the following commands:

+ ```azurecli-interactive

+ export MC_RESOURCE_GROUP=$(az aks show --resource-group AZURE_RESOURCE_GROUP --name CLUSTER_NAME --query nodeResourceGroup -o tsv)

+ export PRINCIPAL_ID=$(az identity show --name "ai-toolchain-operator-{CLUSTER_NAME}" --resource-group "{MC_RESOURCE_GROUP} --query 'principalId' -o tsv)

+ export CLIENT_ID=$(az identity show --name gpuIdentity --resource-group "${AZURE_RESOURCE_GROUP}" --subscription "${AZURE_SUBSCRIPTION_ID}" --query 'clientId' -o tsv)

+ ```

+## Create a role assignment for the principal ID identity

+1. Create a new role assignment for the service principal using the [`az role assignment create`][az-role-assignment-create] command.

+ ```azurecli-interactive

+ az role assignment create --role "Contributor" --assignee "${PRINCIPAL_ID}" --scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"/providers/Microsoft.ContainerService/managedClusters/${CLUSTER_NAME}"

+ ```

+2. Get the AKS OIDC Issuer URL and export it as an environment variable using the following command:

+ ```azurecli-interactive

+ export AKS_OIDC_ISSUER=$(az aks show --resource-group "${AZURE_RESOURCE_GROUP}" --name "${CLUSTER_NAME}" --subscription "${AZURE_SUBSCRIPTION_ID}" --query "oidcIssuerProfile.issuerUrl" -o tsv)

+ ```

+## Establish a federated identity credential

+* Create the federated identity credential between the managed identity, AKS OIDC issuer, and subject using the [`az identity federated-credential create`][az-identity-federated-credential-create] command.

+ ```azurecli-interactive

+ az identity federated-credential create --name "${FEDERATED_IDENTITY}" --identity-name "ai-toolchain-operator-{CLUSTER_NAME}" --resource-group "${AZURE_RESOURCE_GROUP} --issuer "${AKS_OIDC_ISSUER}" --subject system:serviceaccount:"kube-system":"gpu-provisioner" --audience api://AzureADTokenExchange --subscription "${AZURE_SUBSCRIPTION_ID}"

+ ```

+## Deploy a default hosted AI model

+1. Deploy the Falcon 7B model YAML file from the GitHub repository using the `kubectl apply` command.

+ ```azurecli-interactive

+ kubectl apply -f https://raw.githubusercontent.com/Azure/kaito/main/examples/kaito_workspace_falcon_7b.yaml

+ ```

+2. Track the live resource changes in your workspace using the `kubectl get` command.

+ ```azurecli-interactive

+ kubectl get workspace workspace-falcon-7b -w

+ ```

+3. Check your service and get the service IP address using the `kubectl get svc` command.

+ ```azurecli-interactive

+ export SERVICE_IP=$(kubectl get svc workspace-falcon-7b -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

+ ```

+4. Run the Falcon 7B model with a sample input of your choice using the following `curl` command:

+ ```azurecli-interactive

+ curl -X POST "http://SERVICE_IP:80/chat" -H "accept: application/json" -H "Content-Type: application/json" -d '{"prompt":"YOUR_PROMPT_HERE"}'

+ ```

+## Clean up resources

+If you no longer need these resources, you can delete them to avoid incurring extra Azure charges.

+* Delete the resource group and its associated resources using the [`az group delete`][az-group-delete] command.

+ ```azurecli-interactive

+ az group delete --name AZURE_RESOURCE_GROUP --yes --no-wait

+ ```

+## Next steps

+For more inference model options, see the [KAITO GitHub repository](https://github.com/Azure/kaito).

+<!-- LINKS -->

+[az-group-create]: /cli/azure/group#az_group_create

+[az-group-delete]: /cli/azure/group#az_group_delete

+[az-aks-create]: /cli/azure/aks#az_aks_create

+[az-aks-get-credentials]: /cli/azure/aks#az_aks_get_credentials

+[az-role-assignment-create]: /cli/azure/role/assignment#az_role_assignment_create

+[az-identity-federated-credential-create]: /cli/azure/identity/federated-credential#az_identity_federated_credential_create

+[az-account-set]: /cli/azure/account#az_account_set

+[az-extension-add]: /cli/azure/extension#az_extension_add

+ Title: Performance and scaling best practices for large workloads in Azure Kubernetes Service (AKS)

+description: Learn the best practices for performance and scaling for large workloads in Azure Kubernetes Service (AKS).

+# Best practices for performance and scaling for large workloads in Azure Kubernetes Service (AKS)

+> [!NOTE]

+> This article focuses on general best practices for **large workloads**. For best practices specific to **small to medium workloads**, see [Performance and scaling best practices for small to medium workloads in Azure Kubernetes Service (AKS)](./best-practices-performance-scale.md).

+As you deploy and maintain clusters in AKS, you can use the following best practices to help you optimize performance and scaling.

+Keep in mind that *large* is a relative term. Kubernetes has a multi-dimensional scale envelope, and the scale envelope for your workload depends on the resources you use. For example, a cluster with 100 nodes and thousands of pods or CRDs might be considered large. A 1,000 node cluster with 1,000 pods and various other resources might be considered small from the control plane perspective. The best signal for scale of a Kubernetes control plane is API server HTTP request success rate and latency, as that's a proxy for the amount of load on the control plane.

+In this article, you learn about:

+> [!div class="checklist"]

+>

+> * AKS and Kubernetes control plane scalability.

+> * Kubernetes Client best practices, including backoff, watches, and pagination.

+> * Azure API and platform throttling limits.

+> * Feature limitations.

+> * Networking and node pool scaling best practices.

+## AKS and Kubernetes control plane scalability

+In AKS, a *cluster* consists of a set of nodes (physical or virtual machines (VMs)) that run Kubernetes agents and are managed by the Kubernetes control plane hosted by AKS. While AKS optimizes the Kubernetes control plane and its components for scalability and performance, it's still bound by the upstream project limits.

+Kubernetes has a multi-dimensional scale envelope with each resource type representing a dimension. Not all resources are alike. For example, *watches* are commonly set on secrets, which result in list calls to the kube-apiserver that add cost and a disproportionately higher load on the control plane compared to resources without watches.

+The control plane manages all the resource scaling in the cluster, so the more you scale the cluster within a given dimension, the less you can scale within other dimensions. For example, running hundreds of thousands of pods in an AKS cluster impacts how much pod churn rate (pod mutations per second) the control plane can support.

+The size of the envelope is proportional to the size of the Kubernetes control plane. AKS supports two control plane tiers as part of the Base SKU: the Free tier and the Standard tier. For more information, see [Free and Standard pricing tiers for AKS cluster management][free-standard-tier].

+> [!IMPORTANT]

+> We highly recommend using the Standard tier for production or at-scale workloads. AKS automatically scales up the Kubernetes control plane to support the following scale limits:

+>

+> * Up to 5,000 nodes per AKS cluster

+> * 200,000 pods per AKS cluster (with Azure CNI Overlay)

+In most cases, crossing the scale limit threshold results in degraded performance, but doesn't cause the cluster to immediately fail over. To manage load on the Kubernetes control plane, consider scaling in batches of up to 10-20% of the current scale. For example, for a 5,000 node cluster, scale in increments of 500-1,000 nodes. While AKS does autoscale your control plane, it doesn't happen instantaneously.

+You can leverage API Priority and Fairness (APF) to throttle specific clients and request types to protect the control plane during high churn and load.

+## Kubernetes clients

+Kubernetes clients are the applications clients, such as operators or monitoring agents, deployed in the Kubernetes cluster that need to communicate with the kube-api server to perform read or mutate operations. It's important to optimize the behavior of these clients to minimize the load they add to the kube-api server and Kubernetes control plane.

+AKS doesn't expose control plane and API server metrics via Prometheus or through platform metrics. However, you can analyze API server traffic and client behavior through Kube Audit logs. For more information, see [Troubleshoot the Kubernetes control plane](/troubleshoot/azure/azure-kubernetes/troubleshoot-apiserver-etcd).

+LIST requests can be expensive. When working with lists that might have more than a few thousand small objects or more than a few hundred large objects, you should consider the following guidelines:

+* **Consider the number of objects (CRs) you expect to eventually exist** when defining a new resource type (CRD).

+* **The load on etcd and API server primarily relies on the number of objects that exist, not the number of objects that are returned**. Even if you use a field selector to filter the list and retrieve only a small number of results, these guidelines still apply. The only exception is retrieval of a single object by `metadata.name`.

+* **Avoid repeated LIST calls if possible** if your code needs to maintain an updated list of objects in memory. Instead, consider using the Informer classes provided in most Kubernetes libraries. Informers automatically combine LIST and WATCH functionalities to efficiently maintain an in-memory collection.

+* **Consider whether you need strong consistency** if Informers don't meet your needs. Do you need to see the most recent data, up to the exact moment in time you issued the query? If not, set `ResourceVersion=0`. This causes the API server cache to serve your request instead of etcd.

+* **If you can't use Informers or the API server cache, read large lists in chunks**.

+* **Avoid listing more often than needed**. If you can't use Informers, consider how often your application lists the resources. After you read the last object in a large list, don't immediately re-query the same list. You should wait awhile instead.

+* **Consider the number of running instances of your client application**. There's a big difference between having a single controller listing objects vs. having pods on each node doing the same thing. If you plan to have multiple instances of your client application periodically listing large numbers of objects, your solution won't scale to large clusters.

+## Azure API and Platform throttling

+The load on a cloud application can vary over time based on factors such as the number of active users or the types of actions that users perform. If the processing requirements of the system exceed the capacity of the available resources, the system can become overloaded and suffer from poor performance and failures.

+To handle varying load sizes in a cloud application, you can allow the application to use resources up to a specified limit and then throttle them when the limit is reached. On Azure, throttling happens at two levels. Azure Resource Manager (ARM) throttles requests for the subscription and tenant. If the request is under the throttling limits for the subscription and tenant, ARM routes the request to the resource provider. The resource provider then applies throttling limits tailored to its operations. For more information, see [ARM throttling requests](../azure-resource-manager/management/request-limits-and-throttling.md).

+### Manage throttling in AKS

+Azure API limits are usually defined at a subscription-region combination level. For example, all clients within a subscription in a given region share API limits for a given Azure API, such as Virtual Machine Scale Sets PUT APIs. Every AKS cluster has several AKS-owned clients, such as cloud provider or cluster autoscaler, or customer-owned clients, such as Datadog or self-hosted Prometheus, that call Azure APIs. When running multiple AKS clusters in a subscription within a given region, all the AKS-owned and customer-owned clients within the clusters share a common set of API limits. Therefore, the number of clusters you can deploy in a subscription region is a function of the number of clients deployed, their call patterns, and the overall scale and elasticity of the clusters.

+Keeping the above considerations in mind, customers are typically able to deploy between 20-40 small to medium scale clusters per subscription-region. You can maximize your subscription scale using the following best practices:

+Always upgrade your Kubernetes clusters to the latest version. Newer versions contain many improvements that address performance and throttling issues. If you're using an upgraded version of Kubernetes and still see throttling due to the actual load or the number of clients in the subscription, you can try the following options:

+* **Analyze errors using AKS Diagnose and Solve Problems**: You can use [AKS Diagnose and Solve Problems](./aks-diagnostics.md) to analyze errors, identity the root cause, and get resolution recommendations.

+ * **Increase the Cluster Autoscaler scan interval**: If the diagnostic reports show that [Cluster Autoscaler throttling has been detected](/troubleshoot/azure/azure-kubernetes/429-too-many-requests-errors#analyze-and-identify-errors-by-using-aks-diagnose-and-solve-problems), you can [increase the scan interval](./cluster-autoscaler.md#change-the-cluster-autoscaler-settings) to reduce the number of calls to Virtual Machine Scale Sets from the Cluster Autoscaler.

+ * **Reconfigure third-party applications to make fewer calls**: If you filter by *user agents* in the ***View request rate and throttle details*** diagnostic and see that [a third-party application, such as a monitoring application, makes a large number of GET requests](/troubleshoot/azure/azure-kubernetes/429-too-many-requests-errors#analyze-and-identify-errors-by-using-aks-diagnose-and-solve-problems), you can change the settings of these applications to reduce the frequency of the GET calls. Make sure the application clients use exponential backoff when calling Azure APIs.

+* **Split your clusters into different subscriptions or regions**: If you have a large number of clusters and node pools that use Virtual Machine Scale Sets, you can split them into different subscriptions or regions within the same subscription. Most Azure API limits are shared at the subscription-region level, so you can move or scale your clusters to different subscriptions or regions to get unblocked on Azure API throttling. This option is especially helpful if you expect your clusters to have high activity. There are no generic guidelines for these limits. If you want specific guidance, you can create a support ticket.

+## Feature limitations

+As you scale your AKS clusters to larger scale points, keep the following feature limitations in mind:

+* AKS supports up to a 1,000 node scale in an AKS cluster by default. While AKS doesn't prevent you from scaling further, doing so might result in degraded performance. If you want to scale beyond 1,000 nodes, you can request a limit increase. For more information, see [Best practices for creating and running AKS clusters at scale][run-aks-at-scale].

+* [Azure Network Policy Manager (Azure npm)][azure-npm] only supports up to 250 nodes.

+* You can't use the Stop and Start feature with clusters that have more than 100 nodes. For more information, see [Stop and start an AKS cluster](./start-stop-cluster.md).

+## Networking

+As you scale your AKS clusters to larger scale points, keep the following networking best practices in mind:

+* Use Managed NAT for cluster egress with at least two public IPs on the NAT gateway. For more information, see [Create a managed NAT gateway for your AKS cluster][managed-nat-gateway].

+* Use Azure CNI Overlay to scale up to 200,000 pods and 5,000 nodes per cluster. For more information, see [Configure Azure CNI Overlay networking in AKS][azure-cni-overlay].

+* If your application needs direct pod-to-pod communication across clusters, use Azure CNI with dynamic IP allocation and scale up to 50,000 application pods per cluster with one routable IP per pod. For more information, see [Configure Azure CNI networking for dynamic IP allocation in AKS][azure-cni-dynamic-ip].

+* When using internal Kubernetes services behind an internal load balancer, we recommend creating an internal load balancer or service below a 750 node scale for optimal scaling performance and load balancer elasticity.

+* Azure npm only supports up to 250 nodes. If you want to enforce network policies for larger clusters, consider using [Azure CNI powered by Cilium](./azure-cni-powered-by-cilium.md), which combines the robust control plane of Azure CNI with the Cilium data plane to provide high performance networking and security.

+## Node pool scaling

+As you scale your AKS clusters to larger scale points, keep the following node pool scaling best practices in mind:

+* For system node pools, use the *Standard_D16ds_v5* SKU or an equivalent core/memory VM SKU with ephemeral OS disks to provide sufficient compute resources for kube-system pods.

+* Since AKS has a limit of 1,000 nodes per node pool, we recommend creating at least five user node pools to scale up to 5,000 nodes.

+* When running at-scale AKS clusters, use the cluster autoscaler whenever possible to ensure dynamic scaling of node pools based on the demand for compute resources. For more information, see [Automatically scale an AKS cluster to meet application demands][cluster-autoscaler].

+* If you're scaling beyond 1,000 nodes and are *not* using the cluster autoscaler, we recommend scaling in batches of 500-700 nodes at a time. The scaling operations should have a two-minute to five-minute wait time between scale up operations to prevent Azure API throttling. For more information, see [API management: Caching and throttling policies][throttling-policies].

+> [!NOTE]

+> You can't use [Azure Network Policy Manager (Azure NPM)][azure-npm] with clusters that have more than 500 nodes.

+<!-- LINKS - Internal >

+[run-aks-at-scale]: ./operator-best-practices-run-at-scale.md

+[managed-nat-gateway]: ./nat-gateway.md

+[azure-cni-dynamic-ip]: ./configure-azure-cni-dynamic-ip-allocation.md

+[azure-cni-overlay]: ./azure-cni-overlay.md

+[free-standard-tier]: ./free-standard-pricing-tiers.md

+[cluster-autoscaler]: cluster-autoscaler.md

+[azure-npm]: ../virtual-network/kubernetes-network-policies.md

+<!-- LINKS - External -->

+[throttling-policies]: https://azure.microsoft.com/blog/api-management-advanced-caching-and-throttling-policies/

+ Title: Performance and scaling best practices for small to medium workloads in Azure Kubernetes Service (AKS)

+description: Learn the best practices for performance and scaling for small to medium workloads in Azure Kubernetes Service (AKS).

+# Best practices for performance and scaling for small to medium workloads in Azure Kubernetes Service (AKS)

+> [!NOTE]

+> This article focuses on general best practices for **small to medium workloads**. For best practices specific to **large workloads**, see [Performance and scaling best practices for large workloads in Azure Kubernetes Service (AKS)](./best-practices-performance-scale-large.md).

+As you deploy and maintain clusters in AKS, you can use the following best practices to help you optimize performance and scaling.

+In this article, you learn about:

+> [!div class="checklist"]

+>

+> * Tradeoffs and recommendations for autoscaling your workloads.

+> * Managing node scaling and efficiency based on your workload demands.

+> * Networking considerations for ingress and egress traffic.

+> * Monitoring and troubleshooting control plane and node performance.

+> * Capacity planning, surge scenarios, and cluster upgrades.

+> * Storage and networking considerations for data plane performance.

+## Application autoscaling vs. infrastructure autoscaling

+### Application autoscaling

+Application autoscaling is useful when dealing with cost optimization or infrastructure limitations. A well-configured autoscaler maintains high availability for your application while also minimizing costs. You only pay for the resources required to maintain availability, regardless of the demand.

+For example, if an existing node has space but not enough IPs in the subnet, it might be able to skip the creation of a new node and instead immediately start running the application on a new pod.

+#### Horizontal Pod autoscaling

+Implementing [horizontal pod autoscaling](./concepts-scale.md#horizontal-pod-autoscaler) is useful for applications with a steady and predictable resource demand. The Horizontal Pod Autoscaler (HPA) dynamically scales the number of pod replicas, which effectively distributes the load across multiple pods and nodes. This scaling mechanism is typically most beneficial for applications that can be decomposed into smaller, independent components capable of running in parallel.

+The HPA provides resource utilization metrics by default. You can also integrate custom metrics or leverage tools like the [Kubernetes Event-Driven Autoscaler (KEDA) (Preview)](./keda-about.md). These extensions allow the HPA to make scaling decisions based on multiple perspectives and criteria, providing a more holistic view of your application's performance. This is especially helpful for applications with varying complex scaling requirements.

+> [!NOTE]

+> If maintaining high availability for your application is a top priority, we recommend leaving a slightly higher buffer for the minimum pod number for your HPA to account for scaling time.

+#### Vertical Pod autoscaling

+Implementing [vertical pod autoscaling](./vertical-pod-autoscaler.md) is useful for applications with fluctuating and unpredictable resource demands. The Vertical Pod Autoscaler (VPA) allows you to fine-tune resource requests, including CPU and memory, for individual pods, enabling precise control over resource allocation. This granularity minimizes resource waste and enhances the overall efficiency of cluster utilization. The VPA also streamlines application management by automating resource allocation, freeing up resources for critical tasks.

+> [!WARNING]

+> You shouldn't use the VPA in conjunction with the HPA on the same CPU or memory metrics. This combination can lead to conflicts, as both autoscalers attempt to respond to changes in demand using the same metrics. However, you can use the VPA for CPU or memory in conjunction with the HPA for custom metrics to prevent overlap and ensure that each autoscaler focuses on distinct aspects of workload scaling.

+> [!NOTE]

+> The VPA works based on historical data. We recommend waiting at least *24 hours* after deploying the VPA before applying any changes to give it time to collect recommendation data.

+### Infrastructure autoscaling

+#### Cluster autoscaling

+Implementing cluster autoscaling is useful if your existing nodes lack sufficient capacity, as it helps with scaling up and provisioning new nodes.

+When considering cluster autoscaling, the decision of when to remove a node involves a tradeoff between optimizing resource utilization and ensuring resource availability. Eliminating underutilized nodes enhances cluster utilization but might result in new workloads having to wait for resources to be provisioned before they can be deployed. It's important to find a balance between these two factors that aligns with your cluster and workload requirements and [configure the cluster autoscaler profile settings accordingly](./cluster-autoscaler.md#change-the-cluster-autoscaler-settings).

+The Cluster Autoscaler profile settings apply universally to all autoscaler-enabled node pools in your cluster. This means that any scaling actions occurring in one autoscaler-enabled node pool might impact the autoscaling behavior in another node pool. It's important to apply consistent and synchronized profile settings across all relevant node pools to ensure that the autoscaler behaves as expected.

+##### Overprovisioning

+Overprovisioning is a strategy that helps mitigate the risk of application pressure by ensuring there's an excess of readily available resources. This approach is especially useful for applications that experience highly variable loads and cluster scaling patterns that show frequent scale ups and scale downs.

+To determine the optimal amount of overprovisioning, you can use the following formula:

+```txt

+1-buffer/1+traffic

+```

+For example, let's say you want to avoid hitting 100% CPU utilization in your cluster. You might opt for a 30% buffer to maintain a safety margin. If you anticipate an average traffic growth rate of 40%, you might consider overprovisioning by 50%, as calculated by the formula:

+```txt

+1-30%/1+40%=50%

+```

+An effective overprovisioning method involves the use of *pause pods*. Pause pods are low-priority deployments that can be easily replaced by high-priority deployments. You create low priority pods that serve the sole purpose of reserving buffer space. When a high-priority pod requires space, the pause pods are removed and rescheduled on another node or a new node to accommodate the high priority pod.

+The following YAML shows an example pause pod manifest:

+```yml

+apiVersion: scheduling.k8s.io/v1

+kind: PriorityClass

+metadata:

+ name: overprovisioning

+value: -1

+globalDefault: false

+description: "Priority class used by overprovisioning."

+

+apiVersion: apps/v1

+kind: Deployment

+metadata:

+ name: overprovisioning

+ namespace: kube-system

+spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ run: overprovisioning

+ template:

+ metadata:

+ labels:

+ run: overprovisioning

+ spec:

+ priorityClassName: overprovisioning

+ containers:

+ - name: reserve-resources

+ image: your-custome-pause-image

+ resources:

+ requests:

+ cpu: 1

+ memory: 4Gi

+```

+## Node scaling and efficiency

+> **Best practice guidance**:

+>

+> Carefully monitor resource utilization and scheduling policies to ensure nodes are being used efficiently.

+Node scaling allows you to dynamically adjust the number of nodes in your cluster based on workload demands. It's important to understand that adding more nodes to a cluster isn't always the best solution for improving performance. To ensure optimal performance, you should carefully monitor resource utilization and scheduling policies to ensure nodes are being used efficiently.

+### Node images

+> **Best practice guidance**:

+>

+> Use the latest node image version to ensure that you have the latest security patches and bug fixes.

+Using the latest node image version provides the best performance experience. AKS ships performance improvements within the weekly image releases. The latest daemonset images are cached on the latest VHD image, which provide lower latency benefits for node provisioning and bootstrapping. Falling behind on updates might have a negative impact on performance, so it's important to avoid large gaps between versions.

+#### Azure Linux

+The [Azure Linux Container Host on AKS](../azure-linux/intro-azure-linux.md) uses a native AKS image and provides a single place for Linux development. Every package is built from source and validated, ensuring your services run on proven components.

+Azure Linux is lightweight, only including the necessary set of packages to run container workloads. It provides a reduced attack surface and eliminates patching and maintenance of unnecessary packages. At its base layer, it has a Microsoft-hardened kernel tuned for Azure. This image is ideal for performance-sensitive workloads and platform engineers or operators that manage fleets of AKS clusters.

+#### Ubuntu 2204

+The [Ubuntu 2204 image](https://github.com/Azure/AKS/blob/master/CHANGELOG.md) is the default node image for AKS. It's a lightweight and efficient operating system optimized for running containerized workloads. This means that it can help reduce resource usage and improve overall performance. The image includes the latest security patches and updates, which help ensure that your workloads are protected from vulnerabilities.

+The Ubuntu 2204 image is fully supported by Microsoft, Canonical, and the Ubuntu community and can help you achieve better performance and security for your containerized workloads.

+### Virtual machines (VMs)

+> **Best practice guidance**:

+>

+> When selecting a VM, ensure the size and performance of the OS disk and VM SKU don't have a large discrepancy. A discrepancy in size or performance can cause performance issues and resource contention.

+Application performance is closely tied to the VM SKUs you use in your workloads. Larger and more powerful VMs, generally provide better performance. For *mission critical or product workloads*, we recommend using VMs with at least an 8-core CPU. VMs with newer hardware generations, like v4 and v5, can also help improve performance. Keep in mind that create and scale latency might vary depending on the VM SKUs you use.

+### Use dedicated system node pools

+For scaling performance and reliability, we recommend using a dedicated system node pool. With this configuration, the dedicated system node pool reserves space for critical system resources such as system OS daemons. Your application workload can then run in a user node pool to increase the availability of allocatable resources for your application. This configuration also helps mitigate the risk of resource competition between the system and application.

+### Create operations

+Review the extensions and add-ons you have enabled during create provisioning. Extensions and add-ons can add latency to overall duration of create operations. If you don't need an extension or add-on, we recommend removing it to improve create latency.

+You can also use availability zones to provide a higher level of availability to protect against potential hardware failures or planned maintenance events. AKS clusters distribute resources across logical sections of underlying Azure infrastructure. Availability zones physically separate nodes from other nodes to help ensure that a single failure doesn't impact the availability of your application. Availability zones are only available in certain regions. For more information, see [Availability zones in Azure](../reliability/availability-zones-overview.md).

+## Kubernetes API server

+### LIST and WATCH operations

+Kubernetes uses the LIST and WATCH operations to interact with the Kubernetes API server and monitor information about cluster resources. These operations are fundamental to how Kubernetes performs resource management.

+**The LIST operation retrieves a list of resources that fit within certain criteria**, such as all pods in a specific namespace or all services in the cluster. This operation is useful when you want to get an overview of your cluster resources or you need to operator on multiple resources at once.

+The LIST operation can retrieve large amounts of data, especially in large clusters with multiple resources. Be mindful of the fact that making unbounded or frequent LIST calls puts a significant load on the API server and can close down response times.

+**The WATCH operation performs real-time resource monitoring**. When you set up a WATCH on a resource, the API server sends you updates whenever there are changes to that resource. This is important for controllers, like the ReplicaSet controller, which rely on WATCH to maintain the desired state of resources.

+Be mindful of the fact that watching too many mutable resources or making too many concurrent WATCH requests can overwhelm the API server and cause excessive resource consumption.

+To avoid potential issues and ensure the stability of the Kubernetes control plane, you can use the following strategies:

+**Resource quotas**

+Implement resource quotas to limit the number of resources that can be listed or watched by a particular user or namespace to prevent excessive calls.

+**API Priority and Fairness**

+Kubernetes introduced the concept of API Priority and Fairness (APF) to prioritize and manage API requests. You can use APF in Kubernetes to protect the cluster's API server and reduce the number of `HTTP 429 Too Many Requests` responses seen by client applications.

+| Custom resource | Key features |

+| -- | |

+| PriorityLevelConfigurations | ΓÇó Define different priority levels for API requests.<br/> ΓÇó Specifies a unique name and assigns an integer value representing the priority level. Higher priority levels have lower integer values, indicating they're more critical.<br/> ΓÇó Can use multiple to categorize requests into different priority levels based on their importance.<br/> ΓÇó Allow you to specify whether requests at a particular priority level should be subject to rate limits. |

+| FlowSchemas | ΓÇó Define how API requests should be routed to different priority levels based on request attributes.<br/> ΓÇó Specify rules that match requests based on criteria like API groups, versions, and resources.<br/> ΓÇó When a request matches a given rule, the request is directed to the priority level specified in the associated PriorityLevelConfiguration.<br/> ΓÇó Can use to set the order of evaluation when multiple FlowSchemas match a request to ensure that certain rules take precedence. |

+Configuring API with PriorityLevelConfigurations and FlowSchemas enables the prioritization of critical API requests over less important requests. This ensures that essential operations don't starve or experience delays because of lower priority requests.

+**Optimize labeling and selectors**

+When using LIST operations, optimize label selectors to narrow down the scope of the resources you want to query to reduce the amount of data returned and the load on the API server.

+In Kubernetes CREATE and UPDATE operations refer to actions that manage and modify cluster resources.

+### CREATE and UPDATE operations

+**The CREATE operation creates new resources in the Kubernetes cluster**, such as pods, services, deployments, configmaps, and secrets. During a CREATE operation, a client, such as `kubectl` or a controller, sends a request to the Kubernetes API server to create the new resource. The API server validates the request, ensures compliance with any admission controller policies, and then creates the resource in the cluster's desired state.

+**The UPDATE operation modifies existing resources in the Kubernetes cluster**, including changes to resources specifications, like number of replicas, container images, environment variables, or labels. During an UPDATE operation, a client sends a request to the API server to update an existing resource. The API server validates the request, applies the changes to the resource definition, and updates the cluster resource.

+CREATE and UPDATE operations can impact the performance of the Kubernetes API server under the following conditions:

+* **High concurrency**: When multiple users or applications make concurrent CREATE or UPDATE requests, it can lead to a surge in API requests arriving at the server at the same time. This can stress the API server's processing capacity and cause performance issues.

+* **Complex resource definitions**: Resource definitions that are overly complex or involve multiple nested objects can increase the time it takes for the API server to validate and process CREATE and UPDATE requests, which can lead to performance degradation.

+* **Resource validation and admission control**: Kubernetes enforces various admission control policies and validation checks on incoming CREATE and UPDATE requests. Large resource definitions, like ones with extensive annotations or configurations, might require more processing time.

+* **Custom controllers**: Custom controllers that watch for changes in resources, like Deployments or StatefulSet controllers, can generate a significant number of updates when scaling or rolling out changes. These updates can strain the API server's resources.

+For more information, see [Troubleshoot API server and etcd problems in AKS](/troubleshoot/azure/azure-kubernetes/troubleshoot-apiserver-etcd).

+## Data plane performance

+The Kubernetes data plane is responsible for managing network traffic between containers and services. Issues with the data plane can lead to slow response times, degraded performance, and application downtime. It's important to carefully monitor and optimize data plane configurations, such as network latency, resource allocation, container density, and network policies, to ensure your containerized applications run smoothly and efficiently.

+### Storage types

+AKS recommends and defaults to using ephemeral OS disks. Ephemeral OS disks are created on local VM storage and aren't saved to remote Azure storage like managed OS disks. They have faster reimaging and boot times, enabling faster cluster operations, and they provide lower read/write latency on the OS disk of AKS agent nodes. Ephemeral OS disks work well for stateless workloads, where applications are tolerant of individual VM failures but not of VM deployment time or individual VM reimaging instances. Only certain VM SKUs support ephemeral OS disks, so you need to ensure that your desired SKU generation and size is compatible. For more information, see [Ephemeral OS disks in Azure Kubernetes Service (AKS)](./cluster-configuration.md#use-ephemeral-os-on-new-clusters).

+If your workload is unable to use ephemeral OS disks, AKS defaults to using Premium SSD OS disks. If Premium SSD OS disks aren't compatible with your workload, AKS defaults to Standard SSD disks. Currently, the only other available OS disk type is Standard HDD. For more information, see [Storage options in Azure Kubernetes Service (AKS)](./concepts-storage.md).

+The following table provides a breakdown of suggested use cases for OS disks supported in AKS:

+| OS disk type | Key features | Suggested use cases |

+|--|--||

+| Ephemeral OS disks | ΓÇó Faster reimaging and boot times.<br/> ΓÇó Lower read/write latency on OS disk of AKS agent nodes.<br/> ΓÇó High performance and availability. | ΓÇó Demanding enterprise workloads, such as SQL Server, Oracle, Dynamics, Exchange Server, MySQL, Cassandra, MongoDB, SAP Business Suite, etc.<br/> ΓÇó Stateless production workloads that require high availability and low latency. |

+| Premium SSD OS disks | ΓÇó Consistent performance and low latency.<br/> ΓÇó High availability. | ΓÇó Demanding enterprise workloads, such as SQL Server, Oracle, Dynamics, Exchange Server, MySQL, Cassandra, MongoDB, SAP Business Suite, etc.<br/> ΓÇó Input/output (IO) intensive enterprise workloads. |

+| Standard SSD OS disks | ΓÇó Consistent performance.<br/> ΓÇó Better availability and latency compared to Standard HDD disks. | ΓÇó Web servers.<br/> ΓÇó Low input/output operations per second (IOPS) application servers.<br/> ΓÇó Lightly used enterprise applications.<br/> ΓÇó Dev/test workloads. |

+| Standard HDD disks | ΓÇó Low cost.<br/> ΓÇó Exhibits variability in performance and latency. | ΓÇó Backup storage.<br/> ΓÇó Mass storage with infrequent access. |

+#### IOPS and throughput

+Input/output operations per second (IOPS) refers to the number of read and write operations that a disk can perform in a second. Throughout refers to the amount of data that can be transferred in a given time period.

+OS disks are responsible for storing the operating system and its associated files, and the VMs are responsible for running the applications. When selecting a VM, ensure the size and performance of the OS disk and VM SKU don't have a large discrepancy. A discrepancy in size or performance can cause performance issues and resource contention. For example, if the OS disk is significantly smaller than the VMs, it can limit the amount of space available for application data and cause the system to run out of disk space. If the OS disk has lower performance than the VMs, it can become a bottleneck and limit the overall performance of the system. Make sure the size and performance are balanced to ensure optimal performance in Kubernetes.

+You can use the following steps to monitor IOPS and bandwidth meters on OS disks in the Azure portal:

+1. Navigate to the [Azure portal](https://portal.azure.com/).

+2. Search for **Virtual machine scale sets** and select your virtual machine scale set.

+3. Under **Monitoring**, select **Metrics**.

+Ephemeral OS disks can provide dynamic IOPS and throughput for your application, whereas managed disks have capped IOPS and throughput. For more information, see [Ephemeral OS disks for Azure VMs](../virtual-machines/ephemeral-os-disks.md).

+[Azure Premium SSD v2](../virtual-machines/disks-types.md#premium-ssd-v2) is designed for IO-intense enterprise workloads that require sub-millisecond disk latencies and high IOPS and throughput at a low cost. It's suited for a broad range of workloads, such as SQL server, Oracle, MariaDB, SAP, Cassandra, MongoDB, big data/analytics, gaming, and more. This disk type is the highest performing option currently available for persistent volumes.

+### Pod scheduling

+The memory and CPU resources allocated to a VM have a direct impact on the performance of the pods running on the VM. When a pod is created, it's assigned a certain amount of memory and CPU resources, which are used to run the application. If the VM doesn't have enough memory or CPU resources available, it can cause the pods to slow down or even crash. If the VM has too much memory or CPU resources available, it can cause the pods to run inefficiently, wasting resources and increasing costs. We recommend monitoring the total pod requests across your workloads against the total allocatable resources for best scheduling predictability and performance. You can also set the maximum pods per node based on your capacity planning using `--max-pods`.

+> This feature enables more control over creating and managing multiple node pools and requires separate commands for *create/update/delete* (CRUD) operations. Previously, cluster operations through [`az aks create`][az-aks-create] or [`az aks update`][az-aks-update] used the managedCluster API and were the only options to change your control plane and a single node pool. This feature exposes a separate operation set for agent pools through the agentPool API and requires use of the [`az aks nodepool`][az-aks-nodepool] command set to execute operations on an individual node pool.

+* The AKS cluster must use the Standard SKU load balancer to use multiple node pools. This feature isn't supported with Basic SKU load balancers.

+* If you expand your VNET after creating the cluster, you must update your cluster before adding a subnet outside the original CIDR block. While AKS errors-out on the agent pool add, the `aks-preview` Azure CLI extension (version 0.5.66 and higher) now supports running [`az aks update`][az-aks-update] command with only the required `-g <resourceGroup> -n <clusterName>` arguments. This command performs an update operation without making any changes, which can recover a cluster stuck in a failed state.

+* In clusters with Kubernetes version less than 1.23.3, kube-proxy SNATs traffic from new subnets, which can cause Azure Network Policy to drop the packets.

+> * When you create or update a node pool to run Windows Server containers, the default value for `--node-vm-size` is *Standard_D2s_v3*, which was minimum recommended size for Windows Server 2019 node pools prior to Kubernetes version 1.20. The minimum recommended size for Windows Server 2019 node pools using `containerd` is *Standard_D4s_v3*. When setting the `--node-vm-size` parameter, check the list of [restricted VM sizes][restricted-vm-sizes].

+> * We recommended using [taints or labels][aks-taints] with your Windows Server 2019 node pools running `containerd` and tolerations or node selectors with your deployments to guarantee your workloads are scheduled correctly.

+[az-aks-update]: /cli/azure/aks#az_aks_update

+[az-aks-nodepool]: /cli/azure/aks/nodepool

+

+No, you cannot restore your cluster after deleting it. When you delete your cluster, the node resource group and all its resources are also deleted. An example of the second resource group is *MC_myResourceGroup_myAKSCluster_eastus*.

+If you want to keep any of your resources, move them to another resource group before deleting your cluster. If you want to protect against accidental deletes, you can lock the AKS managed resource group hosting your cluster resources using [Node resource group lockdown][node-resource-group-lockdown].

+- [Local-gadget](https://inspektor-gadget.io/docs/): Uses in-kernel eBPF helper programs to monitor events related to syscalls from userspace programs in a pod.

+[node-resource-group-lockdown]: cluster-configuration.md#create-an-aks-cluster-with-node-resource-group-lockdown

+Graphical processing units (GPUs) are often used for compute-intensive workloads, such as graphics and visualization workloads. AKS supports GPU-enabled Linux node pools to run compute-intensive Kubernetes workloads. For more information on available GPU-enabled VMs, see [GPU-optimized VM sizes in Azure][gpu-skus]. For AKS node pools, we recommend a minimum size of *Standard_NC6s_v3*. The NVv4 series (based on AMD GPUs) aren't supported with AKS.

+ --node-vm-size Standard_NC6s_v3 \

+ * `--node-vm-size`: Sets the VM size for the node in the node pool to *Standard_NC6s_v3*.

+ --node-vm-size Standard_NC6s_v3 \

+ * `--node-vm-size`: Sets the VM size for the node in the node pool to *Standard_NC6s_v3*.

+In the following example, we create a GPU-based node pool that uses the *Standard_NC6s_v3* VM size. These VMs are powered by the NVIDIA Tesla K80 card. For information, see [Available sizes for Linux virtual machines in Azure][vm-sizes].

+ --node-vm-size Standard_NC6s_v3 \

+ "vmSize": "Standard_NC6s_v3",

+This article describes how to update the SSH key (preview) on your AKS clusters or node pools.

+## Update SSH public key (preview) on an existing AKS cluster

+> [Use PaaS services for stateful workloads in AKS][aks-tutorial-paas]

+> [!IMPORTANT]

+> This tutorial uses *myResourceGroup* as a placeholder for the resource group name. If you want to use a different name, replace *myResourceGroup* with your own resource group name.

+2. Create an ACR instance using the [`az acr create`][az-acr-create] command and provide your own unique registry name. The registry name must be unique within Azure and contain 5-50 alphanumeric characters. The rest of this tutorial uses an environment variable, `$ACRNAME`, as a placeholder for the container registry name. You can set this environment variable to your unique ACR name to use in future commands. The *Basic* SKU is a cost-optimized entry point for development purposes that provides a balance of storage and throughput.

+ az acr create --resource-group myResourceGroup --name $ACRNAME --sku Basic

+2. Create an ACR instance using the [`New-AzContainerRegistry`][new-azcontainerregistry] cmdlet and provide your own unique registry name. The registry name must be unique within Azure and contain 5-50 alphanumeric characters. The rest of this tutorial uses an environment variable, `$ACRNAME`, as a placeholder for the container registry name. You can set this environment variable to your unique ACR name to use in future commands. The *Basic* SKU is a cost-optimized entry point for development purposes that provides a balance of storage and throughput.

+ New-AzContainerRegistry -ResourceGroupName myResourceGroup -Name $ACRNAME -Location eastus -Sku Basic

+ az acr build --registry $ACRNAME --image aks-store-demo/product-service:latest ./src/product-service/

+ az acr build --registry $ACRNAME --image aks-store-demo/order-service:latest ./src/order-service/

+ az acr build --registry $ACRNAME --image aks-store-demo/store-front:latest ./src/store-front/

+ az acr repository list --name $ACRNAME --output table

+ Get-AzContainerRegistryRepository -RegistryName $ACRNAME

+[az-aks-update]: /cli/azure/aks#az_aks_update

+| node.kubernetes.io/instance-type | \<VM size> | Standard_NC6s_v3 | Virtual |

+## Limitations

+Virtual nodes functionality is heavily dependent on ACI's feature set. In addition to the [quotas and limits for Azure Container Instances](../container-instances/container-instances-quotas.md), the following are scenarios not supported with virtual nodes or are deployment considerations:

+* To schedule Windows Server containers to ACI, you need to manually install the open source [Virtual Kubelet ACI](https://github.com/virtual-kubelet/azure-aci) provider.

+* Using API server authorized ip ranges for AKS.

+> You can create private and public listeners with the same port number. However, be aware of any Network Security Group (NSG) associated with the application gateway subnet. Depending on your NSG's configuration, you may need an allow-inbound rule with **Destination IP addresses** as your application gateway's Public and Private frontend IPs. When using the same port, your application gateway changes the "Destination" of the inbound flow to the frontend IPs of your gateway.

+Upon configuring **active public and private listeners** (with Rules) **with the same port number**, your application gateway changes the "Destination" of all inbound flows to the frontend IPs of your gateway. This is true even for the listeners not sharing any port. You must thus include your gateway's frontend Public and Private IP addresses in the Destination of the inbound rule when using the same port configuration.

+

+ - **Subnet name** (backend server subnet): In the second row of the **Subnets** grid, enter *myBackendSubnet* in the **Subnet name** column.

+ - **Address range** (backend server subnet): In the second row of the **Subnets** Grid, enter an address range that doesn't overlap with the address range of *myAGSubnet*. For example, if the address range of *myAGSubnet* is 10.0.0.0/24, enter *10.0.1.0/24* for the address range of *myBackendSubnet*.

+1. Sign in to the [Azure portal](https://portal.azure.com) and select your virtual machine

+- Ensure to have the Windows PowerShell console installed. We recommend that you use PowerShell version 7.2 or higher. Follow the steps to [Install PowerShell on Windows](/powershell/scripting/install/installing-powershell-on-windows).

+ | | |

+`OutputDCRTemplateFolderPath`| Yes | Folder path where DCR templates are created. |

+1. Any VM with > 100 file/registry settings for migration via portal isn't supported now.

+1. Any VM with > 100 file/registry settings for migration via portal isn't supported now.

+- [Dev/Test (Visual Studio)](/azure/devtest/offer/overview-what-is-devtest-offer-visual-studio)

+- Disaster Recovery ([Entitled benefit DR instances from Software Assurance](https://www.microsoft.com/en-us/licensing/licensing-programs/software-assurance-by-benefits) or subscription only)

+ :::image type="content" source="media/quick-enable-hybrid-vm/add-single-server.png" alt-text="Screenshot of Azure portal's add server page." lightbox="media/quick-enable-hybrid-vm/add-single-server.png":::

+1. On the **Basics** page, provide the following:

+ definition (if you have enabled the Azure Connected Machine agent on a Windows-based machine). For a Linux-based machine, find the corresponding _\[Preview]: Log Analytics extension should be installed on your Linux Azure Arc machines_ policy definition. Click on that policy and click **Add**.

+1. Launch the Azure Arc service in the Azure portal by clicking **All services**, then searching for and selecting **Machines - Azure Arc**.

+1. On the **Azure Arc - Machines** page, select the connected machine you created in the [quickstart](quick-enable-hybrid-vm.md) article.

+When a new version of a VM extension is published, it becomes available for installation and manual upgrade on Arc-enabled servers. For servers that already have the extension installed and automatic extension upgrade enabled, it might take 5 - 8 weeks for every server with that extension to get the automatic upgrade. Upgrades are issued in batches across Azure regions and subscriptions, so you might see the extension get upgraded on some of your servers before others. If you need to upgrade an extension immediately, follow the guidance to manually upgrade extensions using the [Azure portal](manage-vm-extensions-portal.md#upgrade-extensions), [Azure PowerShell](manage-vm-extensions-powershell.md#upgrade-extension) or [Azure CLI](manage-vm-extensions-cli.md#upgrade-extensions).

+1. Go to the [Azure portal](https://portal.azure.com) navigate to **Machines - Azure Arc**.

+1. Select the applicable server.

+1. In the left pane, select the **Extensions** tab to see a list of all extensions installed on the server.

+If multiple extension upgrades are available for a machine, the upgrades might be batched together, but each extension upgrade is applied individually on a machine. A failure on one extension doesn't impact the other extension(s) to be upgraded. For example, if two extensions are scheduled for an upgrade, and the first extension upgrade fails, the second extension will still be upgraded.

+2. In the portal, browse to **Machines - Azure Arc** and select your machine from the list.

+3. Choose **Extensions**, then select **Add**.

+4. Choose the extension you want from the list of available extensions and follow the instructions in the wizard. In this example, we will deploy the Log Analytics VM extension.

+5. After confirming the required information provided, select **Review + Create**. A summary of the deployment is displayed and you can review the status of the deployment.

+2. In the portal, browse to **Machines - Azure Arc** and select your machine from the list.

+2. In the portal, browse to **Machines - Azure Arc** and select your hybrid machine from the list.

+2. In the portal, browse to **Machines - Azure Arc** and select your hybrid machine from the list.

+1. On the **Machines - Azure Arc** page, select **Add/Create** at the upper left, then select **Add a machine** from the drop-down menu.

+1. On the **Add servers with Azure Arc** page, select the **Add multiple servers** tile, and then select **Generate script**.

+1. On the **Basics** page, provide the following:

+ 1. Select the **Subscription** and **Resource group** for the machines.

+ 1. In the **Region** drop-down list, select the Azure region to store the servers' metadata.

+ 1. Select **Next**.

+ 1. In the **Authentication** section, under the **Service principal** drop-down list, select **Arc-for-servers**. Then select, **Next**.

+1. Select **Next**.

+1. Navigate to the **Machines - Azure Arc** page, select **Add/Create**, and then select **Add a machine** from the drop-down menu.

+1. On the **Add servers with Azure Arc** page, select **Add servers** from the **Add managed servers from Update Management** tile.

+1. On the **Resource details** page, configure the following:

+ 1. Select the **Subscription** and **Resource group** where you want the server to be managed within Azure.

+ 1. Select **Next**.

+1. On the **Servers** page, select **Add Servers**, then select the **Subscription** and **Automation account** from the drop-down list that has the Update Management feature enabled and includes the machines you want to onboard to Azure Arc-enabled servers.

+ Once you confirm your selection, select **Next**.

+With Windows Server 2012 and Windows Server 2012 R2 having reached end of support on October 10, 2023, Azure Arc-enabled servers lets you enroll your existing Windows Server 2012/2012 R2 machines in [Extended Security Updates (ESUs)](/windows-server/get-started/extended-security-updates-overview). Affording both cost flexibility and an enhanced delivery experience, Azure Arc better positions you to migrate to Azure.

+ Enrollment in ESUs does not impact Azure Update Manager. After enrollment in ESUs through Azure Arc, the server becomes eligible for ESU patches. These patches can be delivered through Azure Update Manager or any other patching solution. You'll still need to configure updates from Microsoft Updates or Windows Server Update Services.

+ >Activation of ESU is planned for the third quarter of 2023. Using Azure services such as Azure Update Manager and Azure Policy to support managing ESU-eligible Windows Server 2012/2012 R2 machines are also planned for the third quarter.

+We recommend you deploy your machines to Azure Arc in preparation for when the related Azure services deliver supported functionality to manage ESU. Once these machines are onboarded to Azure Arc-enabled servers, you'll have visibility into their ESU coverage and enroll through the Azure portal or using Azure Policy. Billing for this service starts from October 2023 (i.e., after Windows Server 2012 end of support).

+The Azure Connected Machine agent hasn't been tested on operating systems hardened by the Center for Information Security (CIS) Benchmark.

+[Azure Private Link](../../private-link/private-link-overview.md) allows you to securely link Azure PaaS services to your virtual network using private endpoints. For many services, you just set up an endpoint per resource. This means you can connect your on-premises or multicloud servers with Azure Arc and send all traffic over an Azure [ExpressRoute](../../expressroute/expressroute-introduction.md) or site-to-site [VPN connection](../../vpn-gateway/vpn-gateway-about-vpngateways.md) instead of using public networks.

+- Configure the firewall on your local network to allow outbound TCP 443 (HTTPS) access to Microsoft Entra ID and Azure using the downloadable service tag files. The [JSON file](https://www.microsoft.com/en-us/download/details.aspx?id=56519) contains all the public IP address ranges used by Microsoft Entra ID and Azure and is updated monthly to reflect any changes. Azure AD's service tag is `AzureActiveDirectory` and Azure's service tag is `AzureResourceManager`. Consult with your network administrator and network firewall vendor to learn how to configure your firewall rules.

+ b. Choose **Yes** for **Integrate with private DNS zone**, and let it automatically create a new Private DNS Zone. The actual DNS zones might be different from what is shown in the screenshot below.

+If you're only planning to use Private Links to support a few machines or servers, you might not want to update your entire network's DNS configuration. In this case, you can add the private endpoint hostnames and IP addresses to your operating systems **Hosts** file. Depending on the OS configuration, the Hosts file can be the primary or alternative method for resolving hostname to IP address.

+1. Save the file with your changes. You might need to save to another directory first, then copy the file to the original path.

+1. Navigate to **Machines - Azure Arc**.

+1. On the **Machines - Azure Arc** page, select **Add/Create** at the upper left, and then select **Add a machine** from the drop-down menu.

+1. On the **Basics** page, provide the following:

+ 1. Select the **Subscription** and **Resource group** for the machine.

+ 1. Under **Connectivity method**, select **Private endpoint** and select the Azure Arc Private Link Scope created in Part 1 from the drop-down list.

+After downloading the script, you have to run it on your machine or server using a privileged (administrator or root) account. Depending on your network configuration, you might need to download the agent from a computer with internet access and transfer it to your machine or server, and then modify the script with the path to the agent.

+> Network traffic from the Azure Connected Machine agent to Microsoft Entra ID and Azure Resource Manager will continue to use public endpoints. If your server needs to communicate through a proxy server to reach these endpoints, [configure the agent with the proxy server URL](manage-agent.md#update-or-remove-proxy-settings) before connecting it to Azure. You might also need to [configure a proxy bypass](manage-agent.md#proxy-bypass-for-private-endpoints) for the Azure Arc services if your private endpoint is not accessible from your proxy server.

+It might take up to 15 minutes for the Private Link Scope to accept connections from the recently associated server(s).

+2. Find and delete the old Arc resource bridge resource under the [Resource Bridges tab from the Azure Arc center](https://ms.portal.azure.com/#view/Microsoft_Azure_HybridCompute/AzureArcCenterBlade/~/resourceBridges).

+6. In the same machine, run the following scripts, as applicable:

+ - [Download the script](https://download.microsoft.com/download/6/b/4/6b4a5009-fed8-46c2-b22b-b24a4d0a06e3/arcvmm-appliance-dr.ps1) if you are running the script from a Windows machine

+ - [Download the script](https://download.microsoft.com/download/0/5/c/05c2bcb8-87f8-4ead-9757-a87a0759071c/arcvmm-appliance-dr.sh) if you are running the script from a Linux machine

+7. Once the script is run successfully, the old Resource Bridge will be recovered and the connection is re-established to the existing Azure-enabled SCVMM resources.

+| **SCVMM** | You need an SCVMM management server running version 2019 or later.<br/><br/> A private cloud with minimum free capacity of 32 GB of RAM, 4 vCPUs with 100 GB of free disk space. <br/><br/> A VM network with internet access, directly or through proxy. Appliance VM will be deployed using this VM network.<br/><br/> Only Static IP allocation is supported and VMM Static IP Pool is required. Follow [these steps](https://learn.microsoft.com/system-center/vmm/network-pool?view=sc-vmm-2022) to create a VMM Static IP Pool and ensure that the Static IP Pool has at least four IP addresses. Dynamic IP allocation using DHCP is not supported. |

+In this article, you learn how to perform various administrative operations related to Azure Arc-enabled VMware vSphere:

+- Upgrading the Azure Arc resource bridge

+ Title: What is Azure Arc-enabled VMware vSphere?

+# What is Azure Arc-enabled VMware vSphere?

+Azure Arc-enabled VMware vSphere is an [Azure Arc](../overview.md) service that helps you simplify management of hybrid IT estate distributed across VMware vSphere and Azure. It does so by extending the Azure control plane to VMware vSphere infrastructure and enabling the use of Azure security, governance, and management capabilities consistently across VMware vSphere and Azure.

+Arc-enabled VMware vSphere allows you to:

+Arc-enabled VMware vSphere provides these capabilities by integrating with your VMware vCenter Server. To connect your VMware vCenter Server to Azure Arc, you need to deploy the [Azure Arc resource bridge](../resource-bridge/overview.md) in your vSphere environment. Azure Arc resource bridge is a virtual appliance that hosts the components that communicate with your vCenter Server and Azure.

+Azure Arc-enabled VMware vSphere currently works with vCenter Server versions 7 and 8.

+> Azure Arc-enabled VMware vSphere supports vCenters with a maximum of 9500 VMs. If your vCenter has more than 9500 VMs, we don't recommend you to use Arc-enabled VMware vSphere with it at this point.

+You can use Azure Arc-enabled VMware vSphere in these supported regions:

+- East US2

+- West US2

+- West US3

+- South Central US

+- Canada Central

+- North Europe

+- Sweden Central

+- Southeast Asia

+- Australia East

+In this article, you learn how to perform various operations on the Azure Arc-enabled VMware vSphere VMs such as:

+First, the script deploys a virtual appliance called [Azure Arc resource bridge](../resource-bridge/overview.md) in your vCenter environment. Then, it installs a VMware cluster extension to provide a continuous connection between vCenter Server and Azure Arc.

+This account is used for the ongoing operation of Azure Arc-enabled VMware vSphere and the deployment of the Azure Arc resource bridge VM.

+| **Control Plane IP address** | Azure Arc resource bridge runs a Kubernetes cluster, and its control plane always requires a static IP address. Provide an IP address that meets the following requirements: <br> - The IP address must have internet access. <br> - The IP address must be within the subnet defined by IP address prefix. <br> - If you're using static IP address option for resource bridge VM IP address, the control plane IP address must be outside of the IP address range provided for the VM (Start range IP - End range IP). <br> - If there's a DHCP service on the network, the IP address must be outside of DHCP range.|

+In this article, you learn how to recover the Azure Arc resource bridge connection into a working state in disaster scenarios such as accidental deletion. In such cases, the connection between on-premises infrastructure and Azure is lost and any operations performed through Arc fail.

+## Recovering the Arc resource bridge if there is VM deletion

+6. Once the script successfully finishes, the resource bridge should be recovered, and the previously disconnected Arc-enabled resources are manageable in Azure again.

+[Troubleshoot Azure Arc resource bridge issues](../resource-bridge/troubleshoot-resource-bridge.md)

+1. Go to the [**VMware vCenters** list in Arc center](https://portal.azure.com/#view/Microsoft_Azure_HybridCompute/AzureArcCenterBlade/~/vCenter).

+# Support matrix for Azure Arc-enabled VMware vSphere

+This article documents the prerequisites and support requirements for using [Azure Arc-enabled VMware vSphere](overview.md) to manage your VMware vSphere VMs through Azure Arc.

+To use Arc-enabled VMware vSphere, you must deploy an Azure Arc resource bridge in your VMware vSphere environment. The resource bridge provides an ongoing connection between your VMware vCenter Server and Azure. Once you've connected your VMware vCenter Server to Azure, components on the resource bridge discover your vCenter inventory. You can enable them in Azure and start performing virtual hardware and guest OS operations on them using Azure Arc.

+Azure Arc-enabled VMware vSphere works with vCenter Server versions 7 and 8.

+> Azure Arc-enabled VMware vSphere currently supports vCenters with a maximum of 9500 VMs. If your vCenter has more than 9500 VMs, it's not recommended to use Arc-enabled VMware vSphere with it at this point.

+This account is used for the ongoing operation of Azure Arc-enabled VMware vSphere and the deployment of the Azure Arc resource bridge VM.

+ Title: Switch to the new version of VMware vSphere

+description: Learn to switch to the new version of VMware vSphere and use its capabilities

+# Customer intent: As a VI admin, I want to switch to the new version of Arc-enabled VMware vSphere and leverage the associated capabilities.

+# Switch to the new version of VMware vSphere

+On August 21, 2023, we rolled out major changes to **Azure Arc-enabled VMware vSphere**. By switching to the new version, you can use all the Azure management services that are available for Arc-enabled Servers.

+> [!NOTE]

+> If you're new to Arc-enabled VMware vSphere, you'll be able to leverage the new capabilities by default. To get started with the new version, see [Quickstart: Connect VMware vCenter Server to Azure Arc by using the helper script](quick-start-connect-vcenter-to-arc-using-script.md).

+## Switch to the new version (Existing customer)

+If you've onboarded to **Azure Arc-enabled VMware** before August 21, 2023, for VMs that are Azure-enabled, follow these steps to switch to the new version:

+>[!Note]

+>If you had enabled guest management on any of the VMs, remove [VM extensions](/azure/azure-arc/vmware-vsphere/remove-vcenter-from-arc-vmware#step-1-remove-vm-extensions) and [disconnect agents](/azure/azure-arc/vmware-vsphere/remove-vcenter-from-arc-vmware#step-2-disconnect-the-agent-from-azure-arc).

+1. From your browser, go to the vCenters blade on [Azure Arc Center](https://ms.portal.azure.com/#view/Microsoft_Azure_HybridCompute/AzureArcCenterBlade/~/overview) and select the vCenter resource.

+2. Select all the virtual machines that are Azure enabled with the older version.

+3. Select **Remove from Azure**.

+ :::image type="VM Inventory view" source="media/switch-to-new-version-vmware/vm-inventory-view-inline.png" alt-text="Screenshot of VM Inventory view." lightbox="media/switch-to-new-version-vmware/vm-inventory-view-expanded.png":::

+4. After successful removal from Azure, enable the same resources again in Azure.

+5. Once the resources are re-enabled, the VMs are auto switched to the new version. The VM resources will now be represented as **Machine - Azure Arc (VMware)**.

+ :::image type=" New VM browse view" source="media/switch-to-new-version-vmware/new-vm-browse-view-inline.png" alt-text="Screenshot of New VM browse view." lightbox="media/switch-to-new-version-vmware/new-vm-browse-view-expanded.png":::

+

+## Next steps

+[Create a virtual machine on VMware vCenter using Azure Arc](/azure/azure-arc/vmware-vsphere/quick-start-create-a-vm).

+>[!IMPORTANT]

+>Microsoft periodically updates the underlying VM used in cache instances. This can change the performance characteristics from cache to cache and from region to region. The example benchmarking values on this page reflect older generation cache hardware in a single region. You may see better or different results in practice.

+>

+Durable Functions is supported in the new [Python v2 programming model](../functions-reference-python.md?pivots=python-mode-decorators). To use the v2 model, you must install the Durable Functions SDK, which is the PyPI package `azure-functions-durable`, version `1.2.2` or a later version. You must also check `host.json` to make sure your app is referencing [Extension Bundles](../functions-bindings-register.md#extension-bundles) version 4.x to use the v2 model with Durable Functions.

+You can provide feedback and suggestions in the [Durable Functions SDK for Python repo](https://github.com/Azure/azure-functions-durable-python/issues).

+ :::image type="content" source="./media/tutorial-search-location/basic-map.png" lightbox="./media/tutorial-search-location/basic-map.png" alt-text="A screenshot showing the most basic map that you can make by calling `atlas.Map` using your Azure Maps account key.":::

+ :::image type="content" source="./media/tutorial-search-location/pins-map.png" lightbox="./media/tutorial-search-location/pins-map.png" alt-text="A screenshot showing the map resulting from the search, which is a map showing Seattle with round-blue pins at locations of gas stations.":::

+ :::image type="content" source="./media/tutorial-search-location/popup-map.png" lightbox="./media/tutorial-search-location/popup-map.png" alt-text="A screenshot of a map with information popups that appear when you hover over a search pin.":::

+The Azure Maps Web SDK provides a powerful canvas for rendering large spatial data sets in many different ways. In some cases, there are multiple ways to render data the same way, but depending on the size of the data set and the desired functionality, one method might perform better than others. This article highlights best practices and tips and tricks to maximize performance and create a smooth user experience.

+* **GeoJSON source**: The `DataSource` class manages raw location data in GeoJSON format locally. Good for small to medium data sets (upwards of hundreds of thousands of features).

+* **Vector tile source**: The `VectorTileSource` class loads data formatted as vector tiles for the current map view, based on the maps tiling system. Ideal for large to massive data sets (millions or billions of features).

+Azure maps provide several different layers for rendering data on a map. There are many optimizations you can take advantage of to tailor these layers to your scenario the increase performances and the overall user experience.

+That said, if you only have a few points to render on the map, the simplicity of HTML markers might be preferred. Additionally, HTML markers can easily be made draggable if needed.

+When working with large sets of data points you might find that when rendered at certain zoom levels, many of the points overlap and are only partial visible, if at all. Clustering is process of grouping points that are close together and representing them as a single clustered point. As the user zooms in the map, clusters break apart into their individual points. This can significantly reduce the amount of data that needs to be rendered, make the map feel less cluttered, and improve performance. The `DataSource` class has options for clustering data locally. Additionally, many tools that generate vector tiles also have clustering options.

+The heat map layer can render tens of thousands of data points easily. For larger data sets, consider enabling clustering on the data source and using a small cluster radius and use the clusters `point_count` property as a weight for the height map. When the cluster radius is only a few pixels in size, there's little visual difference in the rendered heat map. Using a larger cluster radius improves performance more but might reduce the resolution of the rendered heat map.

+The above code functions fine if all features in the data source have a `myColor` property, and the value of that property is a color. This might not be an issue if you have complete control of the data in the data source and know for certain all features have a valid color in a `myColor` property. That said, to make this code safe from errors, a `case` expression can be used with the `has` expression to check that the feature has the `myColor` property. If it does, the `to-color` type expression can then be used to try to convert the value of that property to a color. If the color is invalid, a fallback color can be used. The following code demonstrates how to do this and sets the fallback color to green.

+* Check the console for errors of the browser's developer tools. Some errors might cause the map not to render. Debug your application.

+If the symbol is only out of place when the map is rotated, check the `rotationAlignment` option. By default, symbols rotate with the maps viewport, appearing upright to the user. However, depending on your scenario, it might be desirable to lock the symbol to the map's orientation by setting the `rotationAlignment` option to `map`.

+If the symbol is only out of place when the map is pitched/tilted, check the `pitchAlignment` option. By default, symbols stay upright in the maps viewport when the map is pitched or tilted. However, depending on your scenario, it might be desirable to lock the symbol to the map's pitch by setting the `pitchAlignment` option to `map`.

+* Try removing data-driven expressions from your rendering layer. It's possible that one of them might have an error in it that is causing the issue.

+| AlmaLinux 9 | Γ£ô³ | | |

+| Oracle Linux 9 | Γ£ô | | |

+| Rocky Linux 9 | Γ£ô | | |

+- Azure Monitor supports monitoring multiple resources of the same type with one metric alert rule for resources that exist in the same Azure region. For a list of Azure services that are currently supported for this feature, see [Supported resources for metric alerts in Azure Monitor](alerts-metric-near-real-time.md).

+|Log alert|You can use log alerts to perform advanced logic operations on your data. If the data you want to monitor is available in logs, or requires advanced logic, you can use the robust features of Kusto Query Language (KQL) for data manipulation by using log alerts.|Each log alert rule is billed based on the interval at which the log query is evaluated. More frequent query evaluation results in a higher cost. For log alerts configured for at-scale monitoring using splitting by dimensions, the cost also depends on the number of time series created by the dimensions resulting from your query. |

+For troubleshooting information, including "no data" scenarios and customizing logs, see [Troubleshoot Application Insights monitoring of Node.js apps and services](/troubleshoot/azure/azure-monitor/app-insights/troubleshoot-app-insights-nodejs).

+> [!NOTE]

+> - If you've adopted our OpenTelemetry Distro and are looking for configuration options, see [Enable Sampling](opentelemetry-configuration.md#enable-sampling).

+| Network managers | [AVNMConnectivityConfigurationChange](/azure/azure-monitor/reference/tables/AVNMConnectivityConfigurationChange) |

+> [!NOTE]

+> The table name is case sensitive.

+## October 2023

+|Subservice | Article | Description |

+||||

+General|[Best practices for monitoring Kubernetes with Azure Monitor](best-practices-containers.md)|New article.|

+General|[Estimate Azure Monitor costs](cost-estimate.md)|New article describing use of Azure Monitor pricing calculator.|

+General|[Azure Monitor billing meter names](cost-meters.md)|Billing meters moved into dedicated reference article.|

+General|[Azure Monitor cost and usage](cost-usage.md)|Rewritten.|

+Agents|[Collect logs from a text or JSON file with Azure Monitor Agent](agents/data-collection-text-log.md)|Added the ability to collect logs from a JSON file with Azure Monitor Agent.|

+Alerts|[Create or edit an alert rule](alerts/alerts-create-new-alert-rule.md)|Custom properties for Azure Monitor alerts are now located in the Details tab when creating or editing an alert rule. |

+Alerts|[Create or edit an alert rule](alerts/alerts-create-new-alert-rule.md)|Added note clarifying the limitations of setting the frequency of alert rules to one minute. |

+Application-Insights|[IP addresses used by Azure Monitor](app/ip-addresses.md)|A logic model diagram is available to assist with troubleshooting scenarios.|

+Application-Insights|[Application Insights Overview dashboard](app/overview-dashboard.md)|All of the Application Insights experiences are now defined in a manner that mirrors the Azure portal experience. We've included a logic model diagram to visually convey how Application Insights works at a high level.|

+Application-Insights|[Enable Azure Monitor OpenTelemetry for .NET, Node.js, Python, and Java applications](app/opentelemetry-enable.md)|Our OpenTelemetry Distro released for .NET, Java, Python, and Node.js. This is a replacement for classic Application Insights SDKs.|

+Essentials|[Collect IIS logs with Azure Monitor Agent](agents/data-collection-iis.md)|Added guidance on setting up data collection endpoints based on deployment.|

+Logs|[Restore logs in Azure Monitor](logs/restore.md)|Updated information about the cost of restoring logs. |

+Logs|[Log Analytics workspace data export in Azure Monitor](logs/logs-data-export.md)|Billing for Data Export was enabled in early October 2023.|

+Logs|[Analyze usage in a Log Analytics workspace](logs/analyze-usage.md)|Added support for querying data volume from events directly, and by computer.|

+* [Configuring Azure NetApp Files Application Volume Group (AVG) for zonal SAP HANA deployment](https://techcommunity.microsoft.com/t5/running-sap-applications-on-the/configuring-azure-netapp-files-anf-application-volume-group-avg/ba-p/3943801)

+* Germany North

+This article connects resource provider namespaces to Azure services. If you don't know the resource provider, see [Find resource provider](#find-resource-provider).

+## AI and machine learning resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.AutonomousSystems | [Autonomous Systems](https://www.microsoft.com/ai/autonomous-systems) |

+| Microsoft.BotService | [Azure Bot Service](/azure/bot-service/) |

+| Microsoft.CognitiveServices | [Cognitive Services](../../ai-services/index.yml) |

+| Microsoft.EnterpriseKnowledgeGraph | Enterprise Knowledge Graph |

+| Microsoft.MachineLearning | [Machine Learning Studio](../../machine-learning/classic/index.yml) |

+| Microsoft.MachineLearningServices | [Azure Machine Learning](../../machine-learning/index.yml) |

+| Microsoft.Search | [Azure Cognitive Search](../../search/index.yml) |

+## Analytics resource providers

+| Microsoft.Databricks | [Azure Databricks](/azure/azure-databricks/) |

+| Microsoft.DataCatalog | [Data Catalog](../../data-catalog/index.yml) |

+| Microsoft.DataFactory | [Data Factory](../../data-factory/index.yml) |

+| Microsoft.DataLakeAnalytics | [Data Lake Analytics](../../data-lake-analytics/index.yml) |

+| Microsoft.DataLakeStore | [Azure Data Lake Storage Gen2](../../storage/blobs/data-lake-storage-introduction.md) |

+| Microsoft.DataShare | [Azure Data Share](../../data-share/index.yml) |

+| Microsoft.HDInsight | [HDInsight](../../hdinsight/index.yml) |

+| Microsoft.Kusto | [Azure Data Explorer](/azure/data-explorer/) |

+| Microsoft.PowerBI | [Power BI](/power-bi/power-bi-overview) |

+| Microsoft.PowerBIDedicated | [Power BI Embedded](/azure/power-bi-embedded/) |

+| Microsoft.ProjectBabylon | [Azure Data Catalog](../../data-catalog/overview.md) |

+| Microsoft.Purview | [Microsoft Purview](/purview/purview) |

+| Microsoft.StreamAnalytics | [Azure Stream Analytics](../../stream-analytics/index.yml) |

+| Microsoft.Synapse | [Azure Synapse Analytics](/azure/sql-data-warehouse/) |

+## Blockchain resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.Blockchain | [Azure Blockchain Service](../../blockchain/workbench/index.yml) |

+| Microsoft.BlockchainTokens | [Azure Blockchain Tokens](https://azure.microsoft.com/services/blockchain-tokens/) |

+## Compute resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.DesktopVirtualization | [Azure Virtual Desktop](../../virtual-desktop/index.yml) |

+| Microsoft.DevTestLab | [Azure Lab Services](../../lab-services/index.yml) |

+| Microsoft.HanaOnAzure | [SAP HANA on Azure Large Instances](../../virtual-machines/workloads/sap/hana-overview-architecture.md) |

+| Microsoft.LabServices | [Azure Lab Services](../../lab-services/index.yml) |

+| Microsoft.Maintenance | [Azure Maintenance](../../virtual-machines/maintenance-configurations.md) |

+| Microsoft.Microservices4Spring | [Azure Spring Apps](../../spring-apps/overview.md) |

+| Microsoft.Quantum | [Azure Quantum](https://azure.microsoft.com/services/quantum/) |

+| Microsoft.SerialConsole - [registered by default](#registration) | [Azure Serial Console for Windows](/troubleshoot/azure/virtual-machines/serial-console-windows) |

+| Microsoft.ServiceFabric | [Service Fabric](../../service-fabric/index.yml) |

+| Microsoft.VirtualMachineImages | [Azure Image Builder](../../virtual-machines/image-builder-overview.md) |

+| Microsoft.VMware | [Azure VMware Solution](../../azure-vmware/index.yml) |

+| Microsoft.VMwareCloudSimple | [Azure VMware Solution by CloudSimple](../../vmware-cloudsimple/index.md) |

+## Container resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.App | [Azure Container Apps](../../container-apps/index.yml) |

+| Microsoft.RedHatOpenShift | [Azure Red Hat OpenShift](../../virtual-machines/linux/openshift-get-started.md) |

+## Core resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.Addons | core |

+| Microsoft.AzureStack | core |

+| Microsoft.Capacity | core |

+| Microsoft.Commerce - [registered by default](#registration) | core |

+| Microsoft.Marketplace | core |

+| Microsoft.MarketplaceApps | core |

+| Microsoft.MarketplaceOrdering - [registered by default](#registration) | core |

+| Microsoft.SaaS | core |

+| Microsoft.Services | core |

+| Microsoft.Subscription | core |

+| microsoft.support - [registered by default](#registration) | core |

+## Database resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.AzureData | SQL Server registry |

+| Microsoft.Cache | [Azure Cache for Redis](../../azure-cache-for-redis/index.yml) |

+| Microsoft.Sql | [Azure SQL Database](/azure/azure-sql/database/index)<br /> [Azure SQL Managed Instance](/azure/azure-sql/managed-instance/index) <br />[Azure Synapse Analytics](/azure/sql-data-warehouse/) |

+| Microsoft.SqlVirtualMachine | [SQL Server on Azure Virtual Machines](/azure/azure-sql/virtual-machines/windows/sql-server-on-azure-vm-iaas-what-is-overview) |

+## Developer tools resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.AppConfiguration | [Azure App Configuration](../../azure-app-configuration/index.yml) |

+| Microsoft.DevSpaces | [Azure Dev Spaces](/previous-versions/azure/dev-spaces/) |

+| Microsoft.MixedReality | [Azure Spatial Anchors](../../spatial-anchors/index.yml) |

+| Microsoft.Notebooks | [Azure Notebooks](https://notebooks.azure.com/help/introduction) |

+## DevOps resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| microsoft.visualstudio | [Azure DevOps](/azure/devops/) |

+| Microsoft.VSOnline | [Azure DevOps](/azure/devops/) |

+## Hybrid resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.AzureArcData | Azure Arc-enabled data services |

+| Microsoft.AzureStackHCI | [Azure Stack HCI](/azure-stack/hci/overview) |

+| Microsoft.HybridCompute | [Azure Arc-enabled servers](../../azure-arc/servers/index.yml) |

+| Microsoft.Kubernetes | [Azure Arc-enabled Kubernetes](../../azure-arc/kubernetes/index.yml) |

+| Microsoft.KubernetesConfiguration | [Azure Arc-enabled Kubernetes](../../azure-arc/kubernetes/index.yml) |

+## Identity resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.AAD | [Microsoft Entra Domain Services](../../active-directory-domain-services/index.yml) |

+| Microsoft.ADHybridHealthService - [registered by default](#registration) | [Microsoft Entra ID](../../active-directory/index.yml) |

+| Microsoft.AzureActiveDirectory | [Microsoft Entra ID B2C](../../active-directory-b2c/index.yml) |

+| Microsoft.ManagedIdentity | [Managed identities for Azure resources](../../active-directory/managed-identities-azure-resources/index.yml) |

+| Microsoft.Token | Token |

+## Integration resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.ApiManagement | [API Management](../../api-management/index.yml) |

+| Microsoft.Communication | [Azure Communication Services](../../communication-services/overview.md) |

+| Microsoft.Logic | [Logic Apps](../../logic-apps/index.yml) |

+| Microsoft.NotificationHubs | [Notification Hubs](../../notification-hubs/index.yml) |

+| Microsoft.PowerPlatform | [Power Platform](/power-platform/) |

+| Microsoft.Relay | [Azure Relay](../../azure-relay/relay-what-is-it.md) |

+| Microsoft.ServiceBus | [Service Bus](/azure/service-bus/) |

+## IoT resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.Devices | [Azure IoT Hub](../../iot-hub/index.yml)<br />[Azure IoT Hub Device Provisioning Service](../../iot-dps/index.yml) |

+| Microsoft.DeviceUpdate | [Device Update for IoT Hub](../../iot-hub-device-update/index.yml) |

+| Microsoft.DigitalTwins | [Azure Digital Twins](../../digital-twins/overview.md) |

+| Microsoft.TimeSeriesInsights | [Azure Time Series Insights](../../time-series-insights/index.yml) |

+| Microsoft.WindowsIoT | [Windows 10 IoT Core Services](/windows-hardware/manufacture/iot/iotcoreservicesoverview) |

+## Management resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.Advisor | [Azure Advisor](../../advisor/index.yml) |

+| Microsoft.Authorization - [registered by default](#registration) | [Azure Resource Manager](../index.yml) |

+| Microsoft.Automation | [Automation](../../automation/index.yml) |

+| Microsoft.Billing - [registered by default](#registration) | [Cost Management and Billing](/azure/billing/) |

+| Microsoft.Blueprint | [Azure Blueprints](../../governance/blueprints/index.yml) |

+| Microsoft.ClassicSubscription - [registered by default](#registration) | Classic deployment model |

+| Microsoft.Consumption - [registered by default](#registration) | [Cost Management](/azure/cost-management/) |

+| Microsoft.CostManagement - [registered by default](#registration) | [Cost Management](/azure/cost-management/) |

+| Microsoft.CostManagementExports | [Cost Management](/azure/cost-management/) |

+| Microsoft.CustomProviders | [Azure Custom Providers](../custom-providers/overview.md) |

+| Microsoft.DynamicsLcs | [Lifecycle Services](https://lcs.dynamics.com/Logon/Index) |

+| Microsoft.Features - [registered by default](#registration) | [Azure Resource Manager](../index.yml) |

+| Microsoft.GuestConfiguration | [Azure Policy](../../governance/policy/index.yml) |

+| Microsoft.PolicyInsights | [Azure Policy](../../governance/policy/index.yml) |

+| Microsoft.Portal - [registered by default](#registration) | [Azure portal](../../azure-portal/index.yml) |

+| Microsoft.RecoveryServices | [Azure Site Recovery](../../site-recovery/index.yml) |

+| Microsoft.ResourceGraph - [registered by default](#registration) | [Azure Resource Graph](../../governance/resource-graph/index.yml) |

+| Microsoft.ResourceHealth | [Azure Service Health](../../service-health/index.yml) |

+| Microsoft.Resources - [registered by default](#registration) | [Azure Resource Manager](../index.yml) |

+| Microsoft.Scheduler | [Scheduler](../../scheduler/index.yml) |

+| Microsoft.SoftwarePlan | License |

+| Microsoft.Solutions | [Azure Managed Applications](../managed-applications/index.yml) |

+## Media resource providers

+| Resource provider namespace | Azure service |

+| | - |

+## Migration resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.ClassicInfrastructureMigrate | Classic deployment model migration |

+| Microsoft.DataBox | [Azure Data Box](../../databox/index.yml) |

+| Microsoft.DataBoxEdge | [Azure Stack Edge](../../databox-online/azure-stack-edge-overview.md) |

+| Microsoft.DataMigration | [Azure Database Migration Service](../../dms/index.yml) |

+| Microsoft.Migrate | [Azure Migrate](../../migrate/migrate-services-overview.md) |

+## Monitoring resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.AlertsManagement | [Azure Monitor](../../azure-monitor/index.yml) |

+| Microsoft.ChangeAnalysis | [Azure Monitor](../../azure-monitor/index.yml) |

+| Microsoft.Insights | [Azure Monitor](../../azure-monitor/index.yml) |

+| Microsoft.Intune | [Azure Monitor](../../azure-monitor/index.yml) |

+| Microsoft.WorkloadMonitor | [Azure Monitor](../../azure-monitor/index.yml) |

+## Network resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.Cdn | [Content Delivery Network](../../cdn/index.yml) |

+| Microsoft.ClassicNetwork | Classic deployment model virtual network |

+| Microsoft.ManagedNetwork | Virtual networks managed by PaaS services |

+| Microsoft.Network | [Application Gateway](../../application-gateway/index.yml)<br />[Azure Bastion](../../bastion/index.yml)<br />[Azure DDoS Protection](../../ddos-protection/ddos-protection-overview.md)<br />[Azure DNS](../../dns/index.yml)<br />[Azure ExpressRoute](../../expressroute/index.yml)<br />[Azure Firewall](../../firewall/index.yml)<br />[Azure Front Door Service](../../frontdoor/index.yml)<br />[Azure Private Link](../../private-link/index.yml)<br />[Azure Route Server](../../route-server/index.yml)<br />[Load Balancer](../../load-balancer/index.yml)<br />[Network Watcher](../../network-watcher/index.yml)<br />[Traffic Manager](../../traffic-manager/index.yml)<br />[Virtual Network](../../virtual-network/index.yml)<br />[Virtual Network NAT](../../virtual-network/nat-gateway/nat-overview.md)<br />[Virtual WAN](../../virtual-wan/index.yml)<br />[VPN Gateway](../../vpn-gateway/index.yml)<br /> |

+## Security resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.Attestation | [Azure Attestation Service](../../attestation/overview.md) |

+| Microsoft.CustomerLockbox | [Customer Lockbox for Microsoft Azure](../../security/fundamentals/customer-lockbox-overview.md) |

+| Microsoft.DataProtection | Data Protection |

+| Microsoft.HardwareSecurityModules | [Azure Dedicated HSM](../../dedicated-hsm/index.yml) |

+| Microsoft.KeyVault | [Key Vault](../../key-vault/index.yml) |

+| Microsoft.WindowsDefenderATP | [Microsoft Defender Advanced Threat Protection](../../security-center/security-center-wdatp.md) |

+| Microsoft.WindowsESU | Extended Security Updates |

+## Storage resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.ClassicStorage | Classic deployment model storage |

+| Microsoft.ElasticSan | [Elastic SAN Preview](../../storage/elastic-san/index.yml) |

+| Microsoft.HybridData | [StorSimple](../../storsimple/index.yml) |

+| Microsoft.ImportExport | [Azure Import/Export](../../import-export/storage-import-export-service.md) |

+| Microsoft.NetApp | [Azure NetApp Files](../../azure-netapp-files/index.yml) |

+| Microsoft.ObjectStore | Object Store |

+## Web resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.BingMaps | [Bing Maps](/BingMaps/#pivot=main&panel=BingMapsAPI) |

+| Microsoft.CertificateRegistration | [App Service Certificates](../../app-service/configure-ssl-app-service-certificate.md) |

+| Microsoft.DomainRegistration | [App Service](../../app-service/index.yml) |

+| Microsoft.Maps | [Azure Maps](../../azure-maps/index.yml) |

+| Microsoft.SignalRService | [Azure SignalR Service](../../azure-signalr/index.yml) |

+## 5G & Space resource providers

+| Resource provider namespace | Azure service |

+| | - |

+| Microsoft.HybridNetwork | [Network Function Manager](../../network-function-manager/index.yml) |

+| Microsoft.MobileNetwork | [Azure Private 5G Core](../../private-5g-core/index.yml) |

+| Microsoft.Orbital | [Azure Orbital Ground Station](../../orbital/overview.md) |

+Resource providers marked with **- registered by default** in the previous section are automatically registered for your subscription. For other resource providers, you need to [register them](resource-providers-and-types.md). However, many resource providers are registered automatically when you perform specific actions. For example, when you create resources through the portal or by deploying an [Azure Resource Manager template](../templates/overview.md), Azure Resource Manager automatically registers any required unregistered resource providers.

+| The AV64 SKU currently supports RAID-1 FTT1, RAID-5 FTT1, and RAID-1 FTT2 vSAN storage policies. For more information, see [AV64 supported RAID configuration](introduction.md#av64-supported-raid-configuration) |Nov 2023 |N/A|N/A|

+ Title: Deploy Arc-enabled Azure VMware Solution

+# Deploy Arc-enabled Azure VMware Solution

+In this article, learn how to deploy Arc for Azure VMware Solution. Once you set up the components needed for this public preview, you're ready to execute operations in Azure VMware Solution vCenter Server from the Azure portal. Arc-enabled Azure VMware Solution allows you to do the actions:

+- Identify your VMware vSphere resources (VMs, templates, networks, datastores, clusters/hosts/resource pools) and register them with Arc at scale.

+- Perform different virtual machine (VM) operations directly from Azure like; create, resize, delete, and power cycle operations (start/stop/restart) on VMware VMs consistently with Azure.

+- Permit developers and application teams to use VM operations on-demand with [Role-based access control (RBAC)](https://learn.microsoft.com/azure/role-based-access-control/overview).

+- Install the Arc-connected machine agent to [govern, protect, configure, and monitor](https://learn.microsoft.com/azure/azure-arc/servers/overview#supported-cloud-operations) them.

+- Browse your VMware vSphere resources (vms, templates, networks, and storage) in Azure

+## How Arc-enabled VMware vSphere differs from Arc-enabled servers

+You have the flexibility to start with either option, Arc-enabled servers or Arc-enabled VMware vSphere. With both options, you receive the same consistent experience. Regardless of the initial option chosen, you can incorporate the other one later without disruption. The following information helps you understand the difference between both options:

+**Arc-enabled servers**

+Azure Arc-enabled servers interact on the guest operating system level. They do that with no awareness of the underlying infrastructure or the virtualization platform they're running on. Since Arc-enabled servers support bare-metal machines, there might not be a host hypervisor in some cases.

+**Arc-enabled VMware vSphere**

+Arc-enabled VMware vSphere is a superset of Arc-enabled servers that extends management capabilities beyond the quest operating system to the VM itself that provides lifecycle management and CRUD (Create, Read, Update, Delete) operations on a VMware vSphere VM. These lifecycle management capabilities are exposed in the Azure portal with a look and feel just like a regular Azure VM. Azure Arc-enabled VMware vSphere provides guest operating system management that uses the same components as Azure Arc-enabled servers.

+## Deploy Arc

+There should be an isolated NSX-T Data Center network segment for deploying the Arc for Azure VMware Solution Open Virtualization Appliance (OVA). If an isolated NSX-T Data Center network segment doesn't exist, one is created.

+### Prerequisites

+> [!IMPORTANT]

+> You can't create the resources in a separate resource group. Ensure you use the same resource group from where the Azure VMware Solution private cloud was created to create your resources.

+You need the following items to ensure you're set up to begin the onboarding process to deploy Arc for Azure VMware Solution.

+- Validate the regional support before you start the onboarding process. Arc for Azure VMware Solution is supported in all regions where Arc for VMware vSphere on-premises is supported. For details, see [Azure Arc-enabled VMware vSphere](https://learn.microsoft.com/azure/azure-arc/vmware-vsphere/overview#supported-regions).

+- A jump box virtual machine (VM) or a [management VM](https://learn.microsoft.com/azure/azure-arc/resource-bridge/system-requirements#management-machine-requirements) with internet access that has a direct line of site to the vCenter.

+ - From the jump box VM, verify you have access to [vCenter Server and NSX-T manager portals](https://learn.microsoft.com/azure/azure-vmware/tutorial-access-private-cloud#connect-to-the-vcenter-server-of-your-private-cloud).

+- A resource group in the subscription where you have an owner or contributor role.

+- An unused, isolated [NSX Data Center network segment](https://learn.microsoft.com/azure/azure-vmware/tutorial-nsx-t-network-segment) that is a static network segment with static IP assignment of size /28 CIDR for deploying the Arc for Azure VMware Solution OVA. If an isolated NSX-T Data Center network segment doesn't exist, one gets created.

+- Verify your Azure subscription is enabled and has connectivity to Azure end points.

+- The firewall and proxy URLs must be allowlisted in order to enable communication from the management machine, Appliance VM, and Control Plane IP to the required Arc resource bridge URLs. See the [Azure eArc resource bridge (Preview) network requirements](https://learn.microsoft.com/azure/azure-arc/resource-bridge/network-requirements).

+- Verify your vCenter Server version is 6.7 or higher.

+- A resource pool or a cluster with a minimum capacity of 16 GB of RAM and four vCPUs.

+- A datastore with a minimum of 100 GB of free disk space is available through the resource pool or cluster.

+- On the vCenter Server, allow inbound connections on TCP port 443. This action ensures that the Arc resource bridge and VMware vSphere cluster extension can communicate with the vCenter Server.

+> [!NOTE]

+> - Private endpoint is currently not supported.

+> - DHCP support isn't available to customers at this time, only static IP addresses are currently supported.

+## Registration to Arc for Azure VMware Solution feature set

+For feature registration, users need to sign into their **Subscription**, navigate to the **Preview features** tab, and search for 'Azure Arc for Azure VMware Solution'. Once registered, no other permissions are required for users to access Arc.

+Use the following steps to guide you through the process to onboard Azure Arc for Azure VMware Solution.

+2. Open the 'config_avs.json' file and populate all the variables.

+ - `networkForApplianceVM` is the name for the segment for Arc appliance VM. One gets created if it doesn't already exist.

+ - `applianceControlPlaneIpAddress` is the IP address for the Kubernetes API server that should be part of the segment IP CIDR provided. It shouldn't be part of the K8s node pool IP range.

+ - `k8sNodeIPPoolStart`, `k8sNodeIPPoolEnd`, `gatewayIPAddress` ,`applianceControlPlaneIpAddress` are optional. You can choose to skip all the optional fields or provide values for all. If you choose not to provide the optional fields, then you must use /28 address space for `networkCIDRForApplianceVM`

+3. Run the installation scripts. You can optionionally setup this preview from a Windows or Linux-based jump box/VM.

+4. More Azure resources are created in your resource group.

+> After the successful installation of Azure Arc resource bridge, it's recommended to retain a copy of the resource bridge config.yaml files and the kubeconfig file safe and secure them in a place that facilitates easy retrieval. These files could be needed later to run commands to perform management operations on the resource bridge. You can find the 3 .yaml files (config files) and the kubeconfig file in the same folder where you ran the script.

+When the script is run successfully, check the status to see if Azure Arc is now configured. To verify if your private cloud is Arc-enabled, do the following actions:

+- Choose **Azure Arc**.

+- Azure Arc state shows as **Configured**.

+Recover from failed deployments

+If the Azure Arc resource bridge deployment fails, consult the [Azure Arc resource bridge troubleshooting](https://learn.microsoft.com/azure/azure-arc/resource-bridge/troubleshoot-resource-bridge) guide. While there can be many reasons why the Azure Arc resource bridge deployment fails, one of them is KVA timeout error. Learn more about the [KVA timeout error](https://learn.microsoft.com/azure/azure-arc/resource-bridge/troubleshoot-resource-bridge#kva-timeout-error) and how to troubleshoot.

+## Discover and project your VMware vSphere infrastructure resources to Azure

+When Arc appliance is successfully deployed on your private cloud, you can do the following actions.

+- View the status from within the private cloud left navigation under **Operations > Azure Arc**.

+- View the VMware vSphere infrastructure resources from the private cloud left navigation under **Private cloud** then select **Azure Arc vCenter resources**.

+- Discover your VMware vSphere infrastructure resources and project them to Azure by navigating, **Private cloud > Arc vCenter resources > Virtual Machines**.

+- Similar to VMs, customers can enable networks, templates, resource pools, and data-stores in Azure.

+## Enable resource pools, clusters, hosts, datastores, networks, and VM templates in Azure

+Once you connected your Azure VMware Solution private cloud to Azure, you can browse your vCenter inventory from the Azure portal. This section shows you how to enable resource pools, networks, and other non-VM resources in Azure.

+> [!NOTE]

+> Enabling Azure Arc on a VMware vSphere resource is a read-only operation on vCenter. It doesn't make changes to your resource in vCenter.

+1. On your Azure VMware Solution private cloud, in the left navigation, locate **vCenter Inventory**.

+2. Select the resource(s) you want to enable, then select **Enable in Azure**.

+3. Select your Azure **Subscription** and **Resource Group**, then select **Enable**.

+ The enable action starts a deployment and creates a resource in Azure, creating representations for your VMware vSphere resources. It allows you to manage who can access those resources through Role-based access control (RBAC) granularly.

+4. Repeat the previous steps for one or more network, resource pool, and VM template resources.

+## Enable guest management and extension installation

+Before you install an extension, you need to enable guest management on the VMware VM.

+### Prerequisite

+Before you can install an extension, ensure your target machine meets the following conditions:

+- Is running a [supported operating system](https://learn.microsoft.com/azure/azure-arc/servers/prerequisites#supported-operating-systems).

+- Is able to connect through the firewall to communicate over the internet and these [URLs](https://learn.microsoft.com/azure/azure-arc/servers/network-requirements?tabs=azure-cloud#urls) aren't blocked.

+- Has VMware tools installed and running.

+- Is powered on and the resource bridge has network connectivity to the host running the VM.

+### Enable guest management

+You need to enable guest management on the VMware VM before you can install an extension. Use the following steps to enable guest management.

+1. From the left navigation, locate **vCenter Server Inventory** and choose **Virtual Machines** to view the list of VMs.

+1. Select the VM you want to install the guest management agent on.

+1. Select **Enable guest management** and provide the administrator username and password to enable guest management then select **Apply**.

+1. Verify **Enable guest management** is now checked.

+### Install the LogAnalytics extension

+1. Locate **Extensions** from the left navigation and select **Add**.

+ 1. Based on the extension, you need to provide details. For example, `workspace Id` and `key` for LogAnalytics extension.

+## Supported extensions and management services

+Perform VM operations on VMware VMs through Azure using [supported extensions and management services](https://learn.microsoft.com/azure/azure-arc/vmware-vsphere/perform-vm-ops-through-azure#supported-extensions-and-management-services)

+## Azure VMware Solution private cloud extension with AV64 node size

+The AV64 is a new Azure VMware Solution host SKU, which is available to expand (not to create) the Azure VMware Solution private cloud built with the existing AV36, AV36P, or AV52 SKU. Use the [Microsoft documentation](https://azure.microsoft.com/explore/global-infrastructure/products-by-region/?products=azure-vmware) to check for availability of the AV64 SKU in the region.

+### Prerequisite for AV64 usage

+See the following prerequisites for AV64 cluster deployment.

+- An Azure VMware solution private cloud is created using AV36, AV36P, or AV52 in AV64 supported [region/AZ](https://azure.microsoft.com/explore/global-infrastructure/products-by-region/?products=azure-vmware).

+- You need one /23 or three (contiguous or noncontiguous) /25 address blocks for AV64 cluster management.

+### Supportability for customer scenarios

+**Customer with existing Azure VMware Solution private cloud**:

+When a customer has a deployed Azure VMware Solution private cloud, they can scale the private cloud by adding a separate AV64 vCenter node cluster to that private cloud. In this scenario, customers should use the following steps:

+1. Get an AV64 [quota approval from Microsoft](/azure/azure-vmware/request-host-quota-azure-vmware-solution) with the minimum of three nodes. Add other details on the Azure VMware Solution private cloud that you plan to extend using AV64.

+2. Use an existing Azure VMware Solution add-cluster workflow with AV64 hosts to expand.

+**Customer plans to create a new Azure VMware Solution private cloud**: When a customer wants a new Azure VMware Solution private cloud that can use AV64 SKU but only for expansion. In this case, the customer meets the prerequisite of having an Azure VMware Solution private cloud built with AV36, AV36P, or AV52 SKU. The customer needs to buy a minimum of three nodes of AV36, AV36P, or AV52 SKU before expanding using AV64. For this scenario, use the following steps:

+1. Get AV36, AV36P, AV52 and AV64 [quota approval from Microsoft](/azure/azure-vmware/request-host-quota-azure-vmware-solution) with a minimum of three nodes each.

+2. Create an Azure VMware Solution private cloud using AV36, AV36P, or AV52 SKU.

+3. Use an existing Azure VMware Solution add-cluster workflow with AV64 hosts to expand.

+**Azure VMware Solution stretched cluster private cloud**: The AV64 SKU isn't supported with Azure VMware Solution stretched cluster private cloud. This means that an AV64-based expansion isn't possible for an Azure VMware Solution stretched cluster private cloud.

+### AV64 Cluster vSAN fault domain (FD) design and recommendations

+The traditional Azure VMware Solution host clusters don't have explicit vSAN FD configuration. The reasoning is the host allocation logic ensures, within clusters, that no two hosts reside in the same physical fault domain within an Azure region. This feature inherently brings resilience and high availability for storage, which the vSAN FD configuration is supposed to bring. More information on vSAN FD can be found in the [VMware documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vsan.doc/GUID-8491C4B0-6F94-4023-8C7A-FD7B40D0368D.html).

+The Azure VMware Solution AV64 host clusters have an explicit vSAN fault domain (FD) configuration. Azure VMware Solution control plane configures five vSAN fault domains for AV64 clusters, and hosts are balanced evenly across these five FDs, as users scale up the hosts in a cluster from three nodes to 16 nodes.

+### Cluster size recommendation

+The Azure VMware Solution minimum vSphere node cluster size supported is three. The vSAN data redundancy is handled by ensuring the minimum cluster size of three hosts are in different vSAN FDs. In a vSAN cluster with three hosts, each in a different FD, Should an FD fail (for example, the top of rack switch fails), the vSAN data would be protected. Operations such as object creation (new VM, VMDK, and others) would fail. The same is true of any maintenance activities where an ESXi host is placed into maintenance mode and/or rebooted. To avoid scenarios such as these, it's recommended to deploy vSAN clusters with a minimum of four ESXi hosts.

+### AV64 host removal workflow and best practices

+Because of the AV64 cluster vSAN fault domain (FD) configuration and need for hosts balanced across all FDs, the host removal from AV64 cluster differs from traditional Azure VMware Solution host clusters with other SKUs.

+Currently, a user can select one or more hosts to be removed from the cluster using portal or API. One condition is that a cluster should have a minimum of three hosts. However, an AV64 cluster behaves differently in certain scenarios when AV64 uses vSAN FDs. Any host removal request is checked against potential vSAN FD imbalance. If a host removal request creates an imbalance, the request is rejected with the http 409-Conflict response. The http 409-Conflict response status code indicates a request conflict with the current state of the target resource (hosts).

+The following three scenarios show examples of instances that would normally error out and demonstrate different methods that can be used to remove hosts without creating a vSAN fault domain (FD) imbalance.

+- When removing a host creates a vSAN FD imbalance with a difference of hosts between most and least populated FD to be more than one.

+ In the following example users, need to remove one of the hosts from FD 1 before removing hosts from other FDs.

+ :::image type="content" source="media/introduction/remove-host-scenario-1.png" alt-text="Diagram showing how users need to remove one of the hosts from FD 1 before removing hosts from other FDs." border="false":::

+- When multiple host removal requests are made at the same time and certain host removals create an imbalance. In this scenario, the Azure VMware Solution control plane removes only hosts, which don't create imbalance.

+ In the following example users can't take both of the hosts from the same FDs unless they're reducing the cluster size to four or lower.

+ :::image type="content" source="media/introduction/remove-host-scenario-2.png" alt-text="Diagram showing how users can't take both of the hosts from the same FDs unless they're reducing the cluster size to four or lower." border="false":::

+- When a selected host removal causes less than three active vSAN FDs. This scenario isn't expected to occur given that all AV64 regions have five FDs and, while adding hosts, the Azure VMware Solution control plane takes care of adding hosts from all five FDs evenly.

+ In the following example users can remove one of the hosts from FD 1, but not from FD 2 or 3.

+ :::image type="content" source="media/introduction/remove-host-scenario-3.png" alt-text="Diagram showing how users can remove one of the hosts from FD 1, but not from FD 2 or 3." border="false":::

+**How to identify the host that can be removed without causing a vSAN FD imbalance**: A user can go to the vSphere user interface to get the current state of vSAN FDs and hosts associated with each of them. This helps to identify hosts (based on the previous examples) that can be removed without affecting the vSAN FD balance and avoid any errors in the removal operation.

+### AV64 supported RAID configuration

+This table provides the list of RAID configuration supported and host requirements in AV64 cluster. The RAID6/FTT2 and RAID1/FTT3 policies will be supported in future on AV64 SKU. Microsoft allows customers to use the RAID-5 FTT1 vSAN storage policy for AV64 clusters with six or more nodes to meet the service level agreement.

+|RAID configuration |Failures to tolerate (FTT) | Minimum hosts required |

+|-|--||

+|RAID-1 (Mirroring) Default setting.| 1 | 3 |

+|RAID-5 (Erasure Coding) | 1 | 4 |

+|RAID-1 (Mirroring) | 2 | 5 |

+

+| Microsoft - Azure VMware Solution | Physical infrastructure<ul><li>Azure regions</li><li>Azure availability zones</li><li>Express Route/Global Reach</ul></li>Compute/Network/Storage<ul><li>Rack and power Bare Metal hosts</li><li>Rack and power network equipment</ul></li>Software defined Data Center (SDDC) deploy/lifecycle<ul><li>VMware ESXi deploy, patch, and upgrade</li><li>VMware vCenter Servers deploy, patch, and upgrade</li><li>VMware NSX-T Data Centers deploy, patch, and upgrade</li><li>VMware vSAN deploy, patch, and upgrade</ul></li>SDDC Networking - VMware NSX-T Data Center provider config<ul><li>Microsoft Edge node/cluster, VMware NSX-T Data Center host preparation</li><li>Provider Tier-0 and Tenant Tier-1 Gateway</li><li>Connectivity from Tier-0 (using BGP) to Azure Network via Express Route</ul></li>SDDC Compute - VMware vCenter Server provider config<ul><li>Create default cluster</li><li>Configure virtual networking for vMotion, Management, vSAN, and others</ul></li>SDDC backup/restore<ul><li>Back up and restore VMware vCenter Server</li><li>Back up and restore VMware NSX-T Data Center NSX-T Manager</ul></li>SDDC health monitoring and corrective actions, for example: replace failed hosts</br><br>(optional) VMware HCX deploys with fully configured compute profile on cloud side as add-on</br><br>(optional) SRM deploys, upgrade, and scale up/down</br><br>Support - SDDC platforms and VMware HCX |

+| Customer | Request Azure VMware Solution host quote with Microsoft<br>Plan and create a request for SDDCs on Azure portal with:<ul><li>Host count</li><li>Management network range</li><li>Other information</ul></li>Configure SDDC network and security (VMware NSX-T Data Center)<ul><li>Network segments to host applications</li><li>More Tier -1 routers</li><li>Firewall</li><li>VMware NSX-T Data Center LB</li><li>IPsec VPN</li><li>NAT</li><li>Public IP addresses</li><li>Distributed firewall/gateway firewall</li><li>Network extension using VMware HCX or VMware NSX-T Data Center</li><li>AD/LDAP config for RBAC</ul></li>Configure SDDC - VMware vCenter Server<ul><li>AD/LDAP config for RBAC</li><li>Deploy and lifecycle management of Virtual Machines (VMs) and application<ul><li>Install operating systems</li><li>Patch operating systems</li><li>Install antivirus software</li><li>Install backup software</li><li>Install configuration management software</li><li>Install application components</li><li>VM networking using VMware NSX-T Data Center segments</ul></li><li>Migrate Virtual Machines (VMs)<ul><li>VMware HCX configuration</li><li>Live vMotion</li><li>Cold migration</li><li>Content library sync</ul></li></ul></li>Configure SDDC - vSAN<ul><li>Define and maintain vSAN VM policies</li><li>Add hosts to maintain adequate 'slack space'</ul></li>Configure VMware HCX<ul><li>Download and deploy HCA connector OVA in on-premises</li><li>Pairing on-premises VMware HCX connector</li><li>Configure the network profile, compute profile, and service mesh</li><li>Configure VMware HCX network extension/MON</li><li>Upgrade/updates</ul></li>Network configuration to connect to on-premises, virtual network, or internet</br><br>Add or delete hosts requests to cluster from Portal</br><br>Deploy/lifecycle management of partner (third party) solutions |

+ Title: Manage Arc-enabled Azure VMware private cloud

+description: Learn how to manage your Arc-enabled Azure VMware private cloud.

+# Manage Arc-enabled Azure VMware private cloud

+In this article, learn how to update the Arc appliance credentials, upgrade the Arc resource bridge, and collect logs from the Arc resource bridge.

+## Update Arc appliance credential

+When **cloud admin** credentials are updated, use the following steps to update the credentials in the appliance store.

+1. Sign into the jumpbox VM from where the [onboard process](https://learn.microsoft.com/azure/azure-vmware/arc-enabled-azure-vmware-solution?tabs=windows#onboard-process-to-deploy-azure-arc) was performed. Change the directory to **onboarding directory**.

+1. Run the following command:

+ For Windows-based jumpbox VM.

+

+ `./.temp/.env/Scripts/activate`

+ For Linux-based jumpbox VM

+ `./.temp/.env/bin/activate

+1. Run the following command:

+ `az arcappliance update-infracredentials vmware --kubeconfig <kubeconfig file>`

+1. Run the following command:

+`az connectedvmware vcenter connect --debug --resource-group {resource-group} --name {vcenter-name-in-azure} --location {vcenter-location-in-azure} --custom-location {custom-location-name} --fqdn {vcenter-ip} --port {vcenter-port} --username cloudadmin@vsphere.local --password {vcenter-password}`

+

+> [!NOTE]

+> Customers need to ensure kubeconfig and SSH keys remain available as they will be required for log collection, appliance Upgrade, and credential rotation. These parameters will be required at the time of upgrade, log collection, and credential update scenarios.

+**Parameters**

+Required parameters

+`-kubeconfig # kubeconfig of Appliance resource`

+**Examples**

+The following command invokes the set credential for the specified appliance resource.

+` az arcappliance setcredential <provider> --kubeconfig <kubeconfig>`

+## Upgrade the Arc resource bridge

+Azure Arc-enabled Azure VMware Private Cloud requires the Arc resource bridge to connect your VMware vSphere environment with Azure. Periodically, new images of Arc resource bridge are released to include security and feature updates.

+> [!NOTE]

+> To upgrade the Arc resource bridge VM to the latest version, you'll need to perform the onboarding again with the **same resource IDs**. This will cause some downtime as operations that are performed through Arc during this time might fail.

+Use the following steps to perform a manual upgrade for Arc appliance virtual machine (VM).

+1. Sign into vCenter Server.

+1. Locate the Arc appliance VM, which should be in the resource pool that was configured during onboarding.

+1. Power off the VM.

+1. Delete the VM.

+1. Delete the download template corresponding to the VM.

+1. Delete the resource bridge **Azure Resource Manager** resource.

+1. Get the previous script `Config_avs` file and add the following configuration item:

+ `"register":false`

+1. Download the latest version of the Azure VMware Solution [onboarding script](https://learn.microsoft.com/azure/azure-vmware/deploy-arc-for-azure-vmware-solution?tabs=windows#onboard-process-to-deploy-azure-arc).

+1. Run the new onboarding script with the previous `config_avs.json` from the jump box VM, without changing other config items.

+## Collect logs from the Arc resource bridge

+Perform ongoing administration for Arc-enabled VMware vSphere by [collecting logs from the Arc resource bridge](https://learn.microsoft.com/azure/azure-arc/vmware-vsphere/administer-arc-vmware#collecting-logs-from-the-arc-resource-bridge).

+ Title: Remove Arc-enabled Azure VMware Solution vSphere resources from Azure

+description: Learn how to remove Arc-enabled Azure VMware Solution vSphere resources from Azure.

+# Remove Arc-enabled Azure VMware Solution vSphere resources from Azure

+In this article, learn how to cleanly remove your VMware vCenter environment from Azure Arc-enabled VMware vSphere. For VMware vSphere environments that you no longer want to manage with Azure Arc-enabled VMware vSphere, use the information in this article to perform the following actions:

+- Remove guest management from VMware virtual machines (VMs).

+- Remove VMware vSphere resource from Azure Arc.

+- Remove Arc resource bridge related items in your vCenter.

+## Remove guest management from VMware VMs

+To prevent continued billing of Azure management services, after you remove the vSphere environment from Azure Arc, you must first remove guest management from all Arc-enabled Azure VMware Solution VMs where it was enabled.

+When you enable guest management on Arc-enabled Azure VMware Solution VMs, the Arc connected machine agent is installed on them. Once guest management is enabled, you can install VM extensions on them and use Azure management services like the Log Analytics on them.

+To completely remove guest management, use the following steps to remove any VM extensions from the virtual machine, disconnect the agent, and uninstall the software from your virtual machine. It's important to complete each of the three steps to fully remove all related software components from your virtual machines.

+### Remove VM extensions

+Use the following steps to uninstall extensions from the portal.

+> [!NOTE]

+> **Steps 2-5** must be performed for all the VMs that have VM extensions installed.

+1. Sign in to your Azure VMware Solution private cloud.

+1. Select **Virtual machines** in **Private cloud**, found in the left navigation under ΓÇ£vCenter Server Inventory Page".

+1. Search and select the virtual machine where you have **Guest management** enabled.

+1. Select **Extensions**.

+1. Select the extensions and select **Uninstall**.

+### Disable guest management from Azure Arc

+To avoid problems onboarding the same VM to **Guest management**, we recommend you do the following steps to cleanly disable guest management capabilities.

+> [!NOTE]

+> **Steps 2-3** must be performed for **all VMs** that have **Guest management** enabled.

+1. Sign into the virtual machine using administrator or root credentials and run the following command in the shell.

+ 1. `azcmagent disconnect --force-local-only`.

+1. Uninstall the `ConnectedMachine agent` from the machine.

+1. Set the **identity** on the VM resource to **none**.

+## Uninstall agents from Virtual Machines (VMs)

+### Windows VM uninstall

+To uninstall the Windows agent from the machine, use the following steps:

+1. Sign in to the computer with an account that has administrator permissions.

+2. In **Control Panel**, select **Programs and Features**.

+3. In **Programs and Features**, select **Azure Connected machine Agent**, select **Uninstall**, then select **Yes**.

+4. Delete the `C:\Program Files\AzureConnectedMachineAgent` folder.

+### Linux VM uninstall

+To uninstall the Linux agent, the command to use depends on the Linux operating system. You must have `root` access permissions or your account must have elevated rights using sudo.

+- For Ubuntu, run the following command:

+ ```bash

+ sudo apt purge azcmagent

+ ```

+- For RHEL, CentOS, Oracle Linux run the following command:

+ ```bash

+ sudo yum remove azcmagent

+ ```

+- For SLES, run the following command:

+ ```bash

+ sudo zypper remove azcmagent

+ ```

+## Remove VMware vSphere resources from Azure

+When you activate Arc-enabled Azure VMware Solution resources in Azure, a representation is created for them in Azure. Before you can delete the vCenter Server resource in Azure, you need to delete all of the Azure resource representations you created for your vSphere resources. To delete the Azure resource representations you created, do the following steps:

+1. Go to the Azure portal.

+1. Choose **Virtual machines** from Arc-enabled VMware vSphere resources in the private cloud.

+1. Select all the VMs that have an Azure Enabled value as **Yes**.

+1. Select **Remove from Azure**. This step starts deployment and removes these resources from Azure. The resources remain in your vCenter Server.

+ 1. Repeat steps 2, 3 and 4 for **Resourcespools/clusters/hosts**, **Templates**, **Networks**, and **Datastores**.

+1. When the deletion completes, select **Overview**.

+ 1. Note the Custom location and the Azure Arc Resource bridge resources in the Essentials section.

+1. Select **Remove from Azure** to remove the vCenter Server resource from Azure.

+1. Go to vCenter Server resource in Azure and delete it.

+1. Go to the Custom location resource and select **Delete**.

+1. Go to the Azure Arc Resource bridge resources and select **Delete**.

+At this point, all of your Arc-enabled VMware vSphere resources are removed from Azure.

+## Remove Arc resource bridge related items in your vCenter

+During onboarding, to create a connection between your VMware vCenter and Azure, an Azure Arc resource bridge is deployed into your VMware vSphere environment. As the last step, you must delete the resource bridge VM as well the VM template created during the onboarding.

+As a last step, run the following command:

+[`az rest --method delete`](https://management.azure.com/subscriptions/f%7BsubId%7D/resourcegroups/%7Brg)

+Once that step is done, Arc no longer works on the Azure VMware Solution private cloud. When you delete Arc resources from vCenter Server, it doesn't affect the Azure VMware Solution private cloud for the customer.

+- **Add PSTN participants. (Currently in [public preview](https://azure.microsoft.com/support/legal/preview-supplemental-terms/))** Invite public switched telephone network (PSTN) participants to a call using a number purchased through your subscription or via Azure direct routing to your Session Border Controller (SBC).

+| **Add PSTN participants** **| | |

+| - Call participants using phone calls | ✔️** | ❌ | ❌ |

+\* Only available on the web calling SDK. Not available on iOS and Android calling SDKs

+** Currently in [public preview](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+> [!IMPORTANT]

+> If you plan to enable open/click tracking for your email links, ensure that you are formatting the email content in HTML correctly. Specifically, make sure your tracking content is properly encapsulated within the payload, as demonstrated below:

+```html

+ <a href="https://www.contoso.com">Contoso Inc.,</a>.

+```

+The following documents might be interesting to you:

+| `pstnDialOutEnabled`* | Enable or disable dialing out to a PSTN number in a room.|

+*pstnDialOutEnabled is currently in [public preview](https://azure.microsoft.com/support/legal/preview-supplemental-terms/)

+> For a walkthrough of setting up a customer tenant and numbers for your testing, see [Configure a test customer for Microsoft Teams Direct Routing with Azure Communications Gateway](configure-test-customer-teams-direct-routing.md) and [Configure test numbers for Microsoft Teams Direct Routing with Azure Communications Gateway](configure-test-numbers-teams-direct-routing.md). When you onboard a real customer, you'll need to follow a similar process, but you'll typically need to ask your customer to carry out the steps that need access to their tenant.

+Azure Communications Gateway sits at the edge of your fixed networks. It connects these networks to Zoom servers, allowing you to support the Zoom Phone Cloud Peering program. The following diagram shows where Azure Communications Gateway sits in your network.

+You provide a trunk towards Zoom (via Azure Communications Gateway) for your customers. Calls flow from Zoom clients through the Zoom servers and Azure Communications Gateway into your network. [!INCLUDE [communications-gateway-multitenant](includes/communications-gateway-multitenant.md)].

+You must provision Azure Communications Gateway with the details of the numbers that you upload to Zoom. This provisioning allows Azure Communications Gateway to route calls correctly. For more information, see [Identifying Zoom calls](#identifying-zoom-calls).

+Azure Communications Gateway doesn't support Premises Peering (where each customer has an eSBC) for Zoom Phone.

+Azure Communications Gateway can use Opus, G.722 and G.711 towards Zoom servers, with a packetization time of 20 ms. You must select the codecs that you want to support when you deploy Azure Communications Gateway.

+If your network can't support a packetization time of 20 ms, you must contact your onboarding team or raise a support request to discuss your requirements for transrating (changing packetization time).

+## Identifying Zoom calls

+You must provision Azure Communications Gateway with all the numbers that you upload to Zoom and indicate that these numbers are enabled for Zoom service. This provisioning allows Azure Communications Gateway to route calls to and from Zoom. It requires [Azure Communications Gateway's Provisioning API](integrate-with-provisioning-api.md).

+> [!IMPORTANT]

+> If numbers that you upload to Zoom aren't configured on Azure Communications Gateway, calls involving those numbers fail.

+>

+> [Configure test numbers for Zoom Phone Cloud Peering with Azure Communications Gateway](configure-test-numbers-zoom.md) explains how to set up test numbers for integration testing. You will need to follow a similar process for real customer numbers.

+Optionally, you can indicate to your network that calls are from Zoom by:

+- Using the Provisioning API to add a header to calls associated with Zoom numbers.

+- Configuring Zoom to add a header with custom contents to SIP INVITEs (as part of uploading numbers to Zoom). For more information on this header, see Zoom's _Zoom Phone Provider Exchange Solution Reference Guide_.

+1. Open a new Visual Studio Code window.

+1. Select <kbd>F1</kbd> to open the command palette.

+1. Enter **Git: Clone** and press enter.

+1. Enter the following URL to clone the sample project:

+ https://github.com/Azure-Samples/containerapps-albumapi-javascript.git

+1. Select a folder to clone the project into.

+1. Select **Open** to open the project in Visual Studio Code.

+1. Select <kbd>F1</kbd> to open the command palette.

+## Create and deploy to Azure Container Apps

+1. Select <kbd>F1</kbd> to open the command palette and run the **Azure Container Apps: Deploy Project from Workspace** command.

+1. Enter the following values as prompted by the extension.

+ | Select subscription | Select the Azure subscription you want to use. |

+ | Select a container apps environment | Select **Create new container apps environment**. You're only asked this question if you have existing Container Apps environments. |

+ | Enter a name for the new container app resource(s) | Enter **my-container-app**. |

+ | Select a location | Select an Azure region close to you. |

+ | Would you like to save your deployment configuration? | Select **Save**. |

+ The Azure activity log panel opens and displays the deployment progress. This process might take a few minutes to complete.

+1. Once this process finishes, Visual Studio Code displays a notification. Select **Browse** to open the deployed app in a browser.

+ In the browser's location bar, append the `/albums` path at the end of the app URL to view data from a sample API request.

+1. Select the **my-container-app** resource group from the *Overview* section.

+1. Enter the resource group name **my-container-app** in the *Are you sure you want to delete "my-container-apps"* confirmation dialog.

+ The process to delete the resource group might take a few minutes to complete.

+* You may use the portal [Cloud Shell](/azure/cloud-shell/get-started?tabs=powershell) to run container copy commands. Alternately, you may run the commands locally; make sure you have [Azure CLI](/cli/azure/install-azure-cli) downloaded and installed on your machine.

+1. Install the certificate according to the process typically used for your operating system. For example, in Linux you would copy the certificate to the `/usr/local/share/ca-certificates/` path.

+> for production workloads. Certain features might not be supported or might have constrained

+An Azure Cosmos DB for PostgreSQL cluster is created with one built-in native PostgreSQL role named 'citus'. You can add more native PostgreSQL roles after cluster provisioning is completed.

+1. In **Microsoft Entra authentication (preview)** section, select **Add Microsoft Entra admins**.

+Use the following procedures to authenticate with Microsoft Entra ID as an Azure Cosmos DB for PostgreSQL user. You can follow along in [Azure Cloud Shell](./../../cloud-shell/get-started.md), on an Azure virtual machine, or on your local machine.

+> Make sure PGPASSWORD variable is set to the Microsoft Entra access token for your

+> subscription for Microsoft Entra authentication. If you need to do Postgres role authentication

+> from the same session you can set PGPASSWORD to the Postgres role password

+> or clear the PGPASSWORD variable value to enter the password interactively.

+To grant the same permissions to Microsoft Entra role `user@tenant.onmicrosoft.com` use the following command:

+ - You can run the script in the Bash environment in [Azure Cloud Shell](../../../../cloud-shell/get-started.md). When Cloud Shell opens, make sure to select **Bash** in the environment field at the upper left of the shell window. Cloud Shell has the latest version of Azure CLI.

+ - You can run the script in the Bash environment in [Azure Cloud Shell](../../../../cloud-shell/get-started.md). When Cloud Shell opens, make sure to select **Bash** in the environment field at the upper left of the shell window. Cloud Shell has the latest version of Azure CLI.

+ - You can run the script in the Bash environment in [Azure Cloud Shell](../../../../cloud-shell/get-started.md). When Cloud Shell opens, make sure to select **Bash** in the environment field at the upper left of the shell window. Cloud Shell has the latest version of Azure CLI.

+ - You can run the script in the Bash environment in [Azure Cloud Shell](../../../../cloud-shell/get-started.md). When Cloud Shell opens, make sure **Bash** appears in the environment field at the upper left of the shell window. Cloud Shell always has the latest version of Azure CLI.

+ - You can run the script in the Bash environment in [Azure Cloud Shell](../../../../cloud-shell/get-started.md). When Cloud Shell opens, make sure **Bash** appears in the environment field at the upper left of the shell window. Cloud Shell always has the latest version of Azure CLI.

+The script in this article demonstrates performing resource lock operations for a API for Table table.

+ - You can run the script in the Bash environment in [Azure Cloud Shell](../../../../cloud-shell/get-started.md). When Cloud Shell opens, make sure **Bash** appears in the environment field at the upper left of the shell window. Cloud Shell always has the latest version of Azure CLI.

+ - You can run the script in the Bash environment in [Azure Cloud Shell](../../../../cloud-shell/get-started.md). When Cloud Shell opens, make sure **Bash** appears in the environment field at the upper left of the shell window. Cloud Shell always has the latest version of Azure CLI.

+> On November 15, 2023, the Azure Enterprise portal is retiring for EA enrollments in the Commercial cloud and is becoming read-only for EA enrollments in the Azure Government cloud.

+> Customers and Partners should use Cost Management + Billing in the Azure portal to manage their enrollments. For more information about enrollment management in the Azure portal, see [Get started with EA billing in the Azure portal](ea-direct-portal-get-started.md).

+> Since August 14, 2023, EA customers is not be able to manage their Azure Government EA enrollments from the [Azure portal](https://portal.azure.com). Instead, they can manage it from the Azure Government portal at [https://portal.azure.us](https://portal.azure.us). The functionality mentioned in this article is same as the Azure Government portal.

+To view detailed usage for specific accounts, download the usage detail report. Usage files can be large. If you prefer, you can use the exports feature to get the same data exported to an Azure Storage account. For more information, see [Export usage details to a storage account](../costs/tutorial-export-acm-data.md).

+An EA administrator can download the invoice from the [Azure portal](https://portal.azure.com) or send it in email. Invoices are sent to whoever is set up to receive invoices for the enrollment. If someone other than an EA administrator needs an email copy of the invoice, an EA administrator can send them a copy.

+Your invoice displays Azure usage charges with costs associated to them first, followed by any Marketplace charges. If you have a credit balance, it gets applied to Azure usage. Your invoice shows Azure usage and Marketplace usage without any cost last.

+### Advanced report download

+You can use the Download Advanced Report to get reports that cover specific date ranges for the selected accounts. The output file is in CSV format to accommodate large record sets.

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Search for **Cost Management + Billing** and select it.

+1. Select **Billing scopes** from the navigation menu and then select the billing account that you want to work with.4. In the left navigation menu, select Billing profiles and select the billing profile that you want to work with.

+1. In the navigation menu, select **Usage + Charges**.

+1. At the top of the Usage + charges page, select **Download Advanced report**.

+1. Select a date range and the accounts to include in the report.

+1. Select **Download**.

+1. You can also download files from the **Report History**. It shows the latest reports that you downloaded.

+If your currency uses a period (**.**) for the thousandth place separator and a comma (**,**) for the decimal place separator, it gets displayed incorrectly. For example: *1.000,00*. The import results might vary depending on your regional language setting.

+1. Under **Delimiters**, select the box for **Comma**. Clear **Tab** if selected.

+> [!NOTE]

+> On November 15, 2023, the Azure Enterprise portal is retiring for EA enrollments in the Commercial cloud and is becoming read-only for EA enrollments in the Azure Government cloud.

+> Customers and Partners should use Cost Management + Billing in the Azure portal to manage their enrollments. For more information about enrollment management in the Azure portal, see [Get started with EA billing in the Azure portal](ea-direct-portal-get-started.md).

+ Title: Copy and transform data in Microsoft Fabric Lakehouse (Preview)

+description: Learn how to copy and transform data to and from Microsoft Fabric Lakehouse (Preview) using Azure Data Factory or Azure Synapse Analytics pipelines.

+# Copy and transform data in Microsoft Fabric Lakehouse (Preview) using Azure Data Factory or Azure Synapse Analytics

+Microsoft Fabric Lakehouse is a data architecture platform for storing, managing, and analyzing structured and unstructured data in a single location. In order to achieve seamless data access across all compute engines in Microsoft Fabric, go to [Lakehouse and Delta Tables](/fabric/data-engineering/lakehouse-and-delta-tables) to learn more.

+This article outlines how to use Copy activity to copy data from and to Microsoft Fabric Lakehouse (Preview) and use Data Flow to transform data in Microsoft Fabric Lakehouse (Preview). To learn more, read the introductory article for [Azure Data Factory](introduction.md) or [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md).

+> [!IMPORTANT]

+> This connector is currently in preview. You can try it out and give us feedback. If you want to take a dependency on preview connectors in your solution, please contact [Azure support](https://azure.microsoft.com/support/).

+## Supported capabilities

+This Microsoft Fabric Lakehouse connector is supported for the following capabilities:

+| Supported capabilities|IR | Managed private endpoint|

+|| --| --|

+|[Copy activity](copy-activity-overview.md) (source/sink)|&#9312; &#9313;|Γ£ô |

+|[Mapping data flow](concepts-data-flow-overview.md) (source/sink)|&#9312; |Γ£ô |

+<small>*&#9312; Azure integration runtime &#9313; Self-hosted integration runtime*</small>

+## Get started

+## Create a Microsoft Fabric Lakehouse linked service using UI

+Use the following steps to create a Microsoft Fabric Lakehouse linked service in the Azure portal UI.

+1. Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services, then select New:

+ # [Azure Data Factory](#tab/data-factory)

+ :::image type="content" source="media/doc-common-process/new-linked-service.png" alt-text="Screenshot of creating a new linked service with Azure Data Factory UI.":::

+ # [Azure Synapse](#tab/synapse-analytics)

+ :::image type="content" source="media/doc-common-process/new-linked-service-synapse.png" alt-text="Screenshot of creating a new linked service with Azure Synapse UI.":::

+1. Search for Microsoft Fabric Lakehouse and select the connector.

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/microsoft-fabric-lakehouse-connector.png" alt-text="Screenshot showing select Microsoft Fabric Lakehouse connector.":::

+1. Configure the service details, test the connection, and create the new linked service.

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/configure-microsoft-fabric-lakehouse-linked-service.png" alt-text="Screenshot of configuration for Microsoft Fabric Lakehouse linked service.":::

+## Connector configuration details

+The following sections provide details about properties that are used to define Data Factory entities specific to Microsoft Fabric Lakehouse.

+## Linked service properties

+The Microsoft Fabric Lakehouse connector supports the following authentication types. See the corresponding sections for details:

+- [Service principal authentication](#service-principal-authentication)

+### Service principal authentication

+To use service principal authentication, follow these steps.

+1. Register an application with the Microsoft Identity platform. To learn how, see [Quickstart: Register an application with the Microsoft identity platform](../active-directory/develop/quickstart-register-app.md). Make note of these values, which you use to define the linked service:

+ - Application ID

+ - Application key

+ - Tenant ID

+2. Grant the service principal at least the **Contributor** role in Microsoft Fabric workspace. Follow these steps:

+ 1. Go to your Microsoft Fabric workspace, select **Manage access** on the top bar. Then select **Add people or groups**.

+

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/fabric-workspace-manage-access.png" alt-text="Screenshot shows selecting Fabric workspace Manage access.":::

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/manage-access-pane.png" alt-text=" Screenshot shows Fabric workspace Manage access pane.":::

+

+ 1. In **Add people** pane, enter your service principal name, and select your service principal from the drop-down list.

+

+ 1. Specify the role as **Contributor** or higher (Admin, Member), then select **Add**.

+

+ :::image type="content" source="media/connector-microsoft-fabric-lakehouse/select-workspace-role.png" alt-text="Screenshot shows adding Fabric workspace role.":::

+ 1. Your service principal is displayed on **Manage access** pane.

+

+These properties are supported for the linked service:

+| Property | Description | Required |

+|: |: |: |

+| type | The type property must be set to **Lakehouse**. |Yes |

+| workspaceId | The Microsoft Fabric workspace ID. | Yes |

+| artifactId | The Microsoft Fabric Lakehouse object ID. | Yes |

+| tenant | Specify the tenant information (domain name or tenant ID) under which your application resides. Retrieve it by hovering the mouse in the upper-right corner of the Azure portal. | Yes |

+| servicePrincipalId | Specify the application's client ID. | Yes |

+| servicePrincipalCredentialType | The credential type to use for service principal authentication. Allowed values are **ServicePrincipalKey** and **ServicePrincipalCert**. | Yes |

+| servicePrincipalCredential | The service principal credential. <br/> When you use **ServicePrincipalKey** as the credential type, specify the application's key. Mark this field as **SecureString** to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). <br/> When you use **ServicePrincipalCert** as the credential, reference a certificate in Azure Key Vault, and ensure the certificate content type is **PKCS #12**.| Yes |

+| connectVia | The [integration runtime](concepts-integration-runtime.md) to be used to connect to the data store. You can use the Azure integration runtime or a self-hosted integration runtime if your data store is in a private network. If not specified, the default Azure integration runtime is used. |No |

+**Example: using service principal key authentication**

+You can also store service principal key in Azure Key Vault.

+```json

+{

+ "name": "MicrosoftFabricLakehouseLinkedService",

+ "properties": {

+ "type": "Lakehouse",

+ "typeProperties": {

+ "workspaceId": "<Microsoft Fabric workspace ID>",

+ "artifactId": "<Microsoft Fabric Lakehouse object ID>",

+ "tenant": "<tenant info, e.g. microsoft.onmicrosoft.com>",

+ "servicePrincipalId": "<service principal id>",

+ "servicePrincipalCredentialType": "ServicePrincipalKey",

+ "servicePrincipalCredential": {

+ "type": "SecureString",

+ "value": "<service principal key>"

+ }

+ },

+ "connectVia": {

+ "referenceName": "<name of Integration Runtime>",

+ "type": "IntegrationRuntimeReference"

+ }

+ }

+}

+```

+## Dataset properties

+Microsoft Fabric Lakehouse connector supports two types of datasets, which are Microsoft Fabric Lakehouse Files dataset

+and Microsoft Fabric Lakehouse Table dataset. See the corresponding sections for details.

+- [Microsoft Fabric Lakehouse Files dataset](#microsoft-fabric-lakehouse-files-dataset)

+- [Microsoft Fabric Lakehouse Table dataset](#microsoft-fabric-lakehouse-table-dataset)

+For a full list of sections and properties available for defining datasets, see [Datasets](concepts-datasets-linked-services.md).

+### Microsoft Fabric Lakehouse Files dataset

+Microsoft Fabric Lakehouse connector supports the following file formats. Refer to each article for format-based settings.

+- [Avro format](format-avro.md)

+- [Binary format](format-binary.md)

+- [Delimited text format](format-delimited-text.md)

+- [JSON format](format-json.md)

+- [ORC format](format-orc.md)

+- [Parquet format](format-parquet.md)

+The following properties are supported under `location` settings in the format-based Microsoft Fabric Lakehouse Files dataset:

+| Property | Description | Required |

+| - | | -- |

+| type | The type property under `location` in the dataset must be set to **LakehouseLocation**. | Yes |

+| folderPath | The path to a folder. If you want to use a wildcard to filter folders, skip this setting and specify it in activity source settings. | No |

+| fileName | The file name under the given folderPath. If you want to use a wildcard to filter files, skip this setting and specify it in activity source settings. | No |

+**Example:**

+```json

+{

+ "name": "DelimitedTextDataset",

+ "properties": {

+ "type": "DelimitedText",

+ "linkedServiceName": {

+ "referenceName": "<Microsoft Fabric Lakehouse linked service name>",

+ "type": "LinkedServiceReference"

+ },

+ "typeProperties": {

+ "location": {

+ "type": "LakehouseLocation",

+ "fileName": "<file name>",

+ "folderPath": "<folder name>"

+ },

+ "columnDelimiter": ",",

+ "compressionCodec": "gzip",

+ "escapeChar": "\\",

+ "firstRowAsHeader": true,

+ "quoteChar": "\""

+ },

+ "schema": [ < physical schema, optional, auto retrieved during authoring > ]

+ }

+}

+```

+### Microsoft Fabric Lakehouse Table dataset

+The following properties are supported for Microsoft Fabric Lakehouse Table dataset:

+| Property | Description | Required |

+| :-- | :-- | :-- |

+| type | The **type** property of the dataset must be set to **LakehouseTable**. | Yes |

+| table | The name of your table. | Yes |

+**Example:**

+```json

+{

+    "name": "LakehouseTableDataset",

+    "properties": {

+        "type": "LakehouseTable",

+        "linkedServiceName": {

+            "referenceName": "<Microsoft Fabric Lakehouse linked service name>",

+            "type": "LinkedServiceReference"

+        },

+        "typeProperties": {

+ "table": "<table_name>"

+        },

+        "schema": [< physical schema, optional, retrievable during authoring >]

+    }

+}

+```

+## Copy activity properties

+The copy activity properties for Microsoft Fabric Lakehouse Files dataset and Microsoft Fabric Lakehouse Table dataset are different. See the corresponding sections for details.

+- [Microsoft Fabric Lakehouse Files in Copy activity](#microsoft-fabric-lakehouse-files-in-copy-activity)

+- [Microsoft Fabric Lakehouse Table in Copy activity](#microsoft-fabric-lakehouse-table-in-copy-activity)

+For a full list of sections and properties available for defining activities, see [Copy activity configurations](copy-activity-overview.md#configuration) and [Pipelines and activities](concepts-pipelines-activities.md).

+### Microsoft Fabric Lakehouse Files in Copy activity

+To use Microsoft Fabric Lakehouse Files dataset type as a source or sink in Copy activity, go to the following sections for the detailed configurations.

+#### Microsoft Fabric Lakehouse Files as a source type

+Microsoft Fabric Lakehouse connector supports the following file formats. Refer to each article for format-based settings.

+- [Avro format](format-avro.md)

+- [Binary format](format-binary.md)

+- [Delimited text format](format-delimited-text.md)

+- [JSON format](format-json.md)

+- [ORC format](format-orc.md)

+- [Parquet format](format-parquet.md)

+You have several options to copy data from Microsoft Fabric Lakehouse using the Microsoft Fabric Lakehouse Files dataset:

+- Copy from the given path specified in the dataset.

+- Wildcard filter against folder path or file name, see `wildcardFolderPath` and `wildcardFileName`.

+- Copy the files defined in a given text file as file set, see `fileListPath`.

+The following properties are under `storeSettings` settings in format-based copy source when using Microsoft Fabric Lakehouse Files dataset:

+| Property | Description | Required |

+| | | |

+| type | The type property under `storeSettings` must be set to **LakehouseReadSettings**. | Yes |

+| ***Locate the files to copy:*** | | |

+| OPTION 1: static path<br> | Copy from the folder/file path specified in the dataset. If you want to copy all files from a folder, additionally specify `wildcardFileName` as `*`. | |

+| OPTION 2: wildcard<br>- wildcardFolderPath | The folder path with wildcard characters to filter source folders. <br>Allowed wildcards are: `*` (matches zero or more characters) and `?` (matches zero or single character); use `^` to escape if your actual folder name has wildcard or this escape char inside. <br>See more examples in [Folder and file filter examples](#folder-and-file-filter-examples). | No |

+| OPTION 2: wildcard<br>- wildcardFileName | The file name with wildcard characters under the given folderPath/wildcardFolderPath to filter source files. <br>Allowed wildcards are: `*` (matches zero or more characters) and `?` (matches zero or single character); use `^` to escape if your actual file name has wildcard or this escape char inside. See more examples in [Folder and file filter examples](#folder-and-file-filter-examples). | Yes |

+| OPTION 3: a list of files<br>- fileListPath | Indicates to copy a given file set. Point to a text file that includes a list of files you want to copy, one file per line, which is the relative path to the path configured in the dataset.<br/>When using this option, don't specify file name in dataset. See more examples in [File list examples](#file-list-examples). |No |

+| ***Additional settings:*** | | |

+| recursive | Indicates whether the data is read recursively from the subfolders or only from the specified folder. Note that when recursive is set to true and the sink is a file-based store, an empty folder or subfolder isn't copied or created at the sink. <br>Allowed values are **true** (default) and **false**.<br>This property doesn't apply when you configure `fileListPath`. |No |

+| deleteFilesAfterCompletion | Indicates whether the binary files will be deleted from source store after successfully moving to the destination store. The file deletion is per file, so when copy activity fails, you'll see some files have already been copied to the destination and deleted from source, while others are still remaining on source store. <br/>This property is only valid in binary files copy scenario. The default value: false. |No |

+| modifiedDatetimeStart | Files filter based on the attribute: Last Modified. <br>The files will be selected if their last modified time is greater than or equal to `modifiedDatetimeStart` and less than `modifiedDatetimeEnd`. The time is applied to UTC time zone in the format of "2018-12-01T05:00:00Z". <br> The properties can be NULL, which means no file attribute filter will be applied to the dataset. When `modifiedDatetimeStart` has datetime value but `modifiedDatetimeEnd` is NULL, it means the files whose last modified attribute is greater than or equal with the datetime value will be selected. When `modifiedDatetimeEnd` has datetime value but `modifiedDatetimeStart` is NULL, it means the files whose last modified attribute is less than the datetime value will be selected.<br/>This property doesn't apply when you configure `fileListPath`. | No |

+| modifiedDatetimeEnd | Same as above. | No |

+| enablePartitionDiscovery | For files that are partitioned, specify whether to parse the partitions from the file path and add them as additional source columns.<br/>Allowed values are **false** (default) and **true**. | No |

+| partitionRootPath | When partition discovery is enabled, specify the absolute root path in order to read partitioned folders as data columns.<br/><br/>If it isn't specified, by default,<br/>- When you use file path in dataset or list of files on source, partition root path is the path configured in dataset.<br/>- When you use wildcard folder filter, partition root path is the sub-path before the first wildcard.<br/><br/>For example, assuming you configure the path in dataset as "root/folder/year=2020/month=08/day=27":<br/>- If you specify partition root path as "root/folder/year=2020", copy activity will generate two more columns `month` and `day` with value "08" and "27" respectively, in addition to the columns inside the files.<br/>- If partition root path isn't specified, no extra column will be generated. | No |

+| maxConcurrentConnections | The upper limit of concurrent connections established to the data store during the activity run. Specify a value only when you want to limit concurrent connections.| No |

+**Example:**

+```json

+"activities": [

+ {

+ "name": "CopyFromLakehouseFiles",

+ "type": "Copy",

+ "inputs": [

+ {

+ "referenceName": "<Delimited text input dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "outputs": [

+ {

+ "referenceName": "<output dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "typeProperties": {

+ "source": {

+ "type": "DelimitedTextSource",

+ "storeSettings": {

+ "type": "LakehouseReadSettings",

+ "recursive": true,

+ "enablePartitionDiscovery": false

+ },

+ "formatSettings": {

+ "type": "DelimitedTextReadSettings"

+ }

+ },

+ "sink": {

+ "type": "<sink type>"

+ }

+ }

+ }

+]

+```

+#### Microsoft Fabric Lakehouse Files as a sink type

+Microsoft Fabric Lakehouse connector supports the following file formats. Refer to each article for format-based settings.

+- [Avro format](format-avro.md)

+- [Binary format](format-binary.md)

+- [Delimited text format](format-delimited-text.md)

+- [JSON format](format-json.md)

+- [ORC format](format-orc.md)

+- [Parquet format](format-parquet.md)

+The following properties are under `storeSettings` settings in format-based copy sink when using Microsoft Fabric Lakehouse Files dataset:

+| Property | Description | Required |

+| | | -- |

+| type | The type property under `storeSettings` must be set to **LakehouseWriteSettings**. | Yes |

+| copyBehavior | Defines the copy behavior when the source is files from a file-based data store.<br/><br/>Allowed values are:<br/><b>- PreserveHierarchy (default)</b>: Preserves the file hierarchy in the target folder. The relative path of the source file to the source folder is identical to the relative path of the target file to the target folder.<br/><b>- FlattenHierarchy</b>: All files from the source folder are in the first level of the target folder. The target files have autogenerated names. <br/><b>- MergeFiles</b>: Merges all files from the source folder to one file. If the file name is specified, the merged file name is the specified name. Otherwise, it's an autogenerated file name. | No |

+| blockSizeInMB | Specify the block size in MB used to write data to Microsoft Fabric Lakehouse. Learn more [about Block Blobs](/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-block-blobs). <br/>Allowed value is **between 4 MB and 100 MB**. <br/>By default, ADF automatically determines the block size based on your source store type and data. For non-binary copy into Microsoft Fabric Lakehouse, the default block size is 100 MB so as to fit in at most approximately 4.75-TB data. It might be not optimal when your data isn't large, especially when you use Self-hosted Integration Runtime with poor network resulting in operation timeout or performance issue. You can explicitly specify a block size, while ensure blockSizeInMB*50000 is big enough to store the data, otherwise copy activity run will fail. | No |

+| maxConcurrentConnections | The upper limit of concurrent connections established to the data store during the activity run. Specify a value only when you want to limit concurrent connections.| No |

+| metadata |Set custom metadata when copy to sink. Each object under the `metadata` array represents an extra column. The `name` defines the metadata key name, and the `value` indicates the data value of that key. If [preserve attributes feature](./copy-activity-preserve-metadata.md#preserve-metadata) is used, the specified metadata will union/overwrite with the source file metadata.<br/><br/>Allowed data values are:<br/>- `$$LASTMODIFIED`: a reserved variable indicates to store the source files' last modified time. Apply to file-based source with binary format only.<br/><b>- Expression<b><br/>- <b>Static value<b>| No |

+**Example:**

+```json

+"activities": [

+ {

+ "name": "CopyToLakehouseFiles",

+ "type": "Copy",

+ "inputs": [

+ {

+ "referenceName": "<input dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "outputs": [

+ {

+ "referenceName": "<Parquet output dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "typeProperties": {

+ "source": {

+ "type": "<source type>"

+ },

+ "sink": {

+ "type": "ParquetSink",

+ "storeSettings": {

+ "type": "LakehouseWriteSettings",

+ "copyBehavior": "PreserveHierarchy",

+ "metadata": [

+ {

+ "name": "testKey1",

+ "value": "value1"

+ },

+ {

+ "name": "testKey2",

+ "value": "value2"

+ }

+ ]

+ },

+ "formatSettings": {

+ "type": "ParquetWriteSettings"

+ }

+ }

+ }

+ }

+]

+```

+#### Folder and file filter examples

+This section describes the resulting behavior of the folder path and file name with wildcard filters.

+| folderPath | fileName | recursive | Source folder structure and filter result (files in **bold** are retrieved)|

+|: |: |: |: |

+| `Folder*` | (Empty, use default) | false | FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File2.json**<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5.csv<br/>AnotherFolderB<br/>&nbsp;&nbsp;&nbsp;&nbsp;File6.csv |

+| `Folder*` | (Empty, use default) | true | FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File2.json**<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File3.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File4.json**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File5.csv**<br/>AnotherFolderB<br/>&nbsp;&nbsp;&nbsp;&nbsp;File6.csv |

+| `Folder*` | `*.csv` | false | FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3.csv<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5.csv<br/>AnotherFolderB<br/>&nbsp;&nbsp;&nbsp;&nbsp;File6.csv |

+| `Folder*` | `*.csv` | true | FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File3.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File5.csv**<br/>AnotherFolderB<br/>&nbsp;&nbsp;&nbsp;&nbsp;File6.csv |

+#### File list examples

+This section describes the resulting behavior of using file list path in copy activity source.

+Assuming you have the following source folder structure and want to copy the files in bold:

+| Sample source structure | Content in FileListToCopy.txt | ADF configuration |

+| | | |

+| filesystem<br/>&nbsp;&nbsp;&nbsp;&nbsp;FolderA<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File1.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File2.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File3.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4.json<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**File5.csv**<br/>&nbsp;&nbsp;&nbsp;&nbsp;Metadata<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;FileListToCopy.txt | File1.csv<br>Subfolder1/File3.csv<br>Subfolder1/File5.csv | **In dataset:**<br>- File system: `filesystem`<br>- Folder path: `FolderA`<br><br>**In copy activity source:**<br>- File list path: `filesystem/Metadata/FileListToCopy.txt` <br><br>The file list path points to a text file in the same data store that includes a list of files you want to copy, one file per line with the relative path to the path configured in the dataset. |

+#### Some recursive and copyBehavior examples

+This section describes the resulting behavior of the copy operation for different combinations of recursive and copyBehavior values.

+| recursive | copyBehavior | Source folder structure | Resulting target |

+|: |: |: |: |

+| true |preserveHierarchy | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the same structure as the source:<br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 |

+| true |flattenHierarchy | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the following structure: <br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File5 |

+| true |mergeFiles | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the following structure: <br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1 + File2 + File3 + File4 + File5 contents are merged into one file with an autogenerated file name. |

+| false |preserveHierarchy | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the following structure: <br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/><br/>Subfolder1 with File3, File4, and File5 isn't picked up. |

+| false |flattenHierarchy | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the following structure: <br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;autogenerated name for File2<br/><br/>Subfolder1 with File3, File4, and File5 isn't picked up. |

+| false |mergeFiles | Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File2<br/>&nbsp;&nbsp;&nbsp;&nbsp;Subfolder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File3<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File4<br/>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File5 | The target Folder1 is created with the following structure: <br/><br/>Folder1<br/>&nbsp;&nbsp;&nbsp;&nbsp;File1 + File2 contents are merged into one file with an autogenerated file name. autogenerated name for File1<br/><br/>Subfolder1 with File3, File4, and File5 isn't picked up. |

+### Microsoft Fabric Lakehouse Table in Copy activity

+To use Microsoft Fabric Lakehouse Table dataset as a source or sink dataset in Copy activity, go to the following sections for the detailed configurations.

+#### Microsoft Fabric Lakehouse Table as a source type

+To copy data from Microsoft Fabric Lakehouse using Microsoft Fabric Lakehouse Table dataset, set the **type** property in the Copy activity source to **LakehouseTableSource**. The following properties are supported in the Copy activity **source** section:

+| Property | Description | Required |

+| : | :-- | :- |

+| type | The **type** property of the Copy Activity source must be set to **LakehouseTableSource**. | Yes |

+| timestampAsOf | The timestamp to query an older snapshot. | No |

+| versionAsOf | The version to query an older snapshot. | No |

+**Example:**

+```json

+"activities":[

+ {

+ "name": "CopyFromLakehouseTable",

+ "type": "Copy",

+ "inputs": [

+ {

+ "referenceName": "<Microsoft Fabric Lakehouse Table input dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "outputs": [

+ {

+ "referenceName": "<output dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "typeProperties": {

+ "source": {

+ "type": "LakehouseTableSource",

+ "timestampAsOf": "2023-09-23T00:00:00.000Z",

+ "versionAsOf": 2

+ },

+ "sink": {

+ "type": "<sink type>"

+ }

+ }

+ }

+]

+```

+#### Microsoft Fabric Lakehouse Table as a sink type

+To copy data to Microsoft Fabric Lakehouse using Microsoft Fabric Lakehouse Table dataset, set the **type** property in the Copy Activity sink to **LakehouseTableSink**. The following properties are supported in the Copy activity **sink** section:

+| Property | Description | Required |

+| : | :-- | :- |

+| type | The **type** property of the Copy Activity source must be set to **LakehouseTableSink**. | Yes |

+**Example:**

+```json

+"activities":[

+ {

+ "name": "CopyToLakehouseTable",

+ "type": "Copy",

+ "inputs": [

+ {

+ "referenceName": "<input dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "outputs": [

+ {

+ "referenceName": "<Microsoft Fabric Lakehouse Table output dataset name>",

+ "type": "DatasetReference"

+ }

+ ],

+ "typeProperties": {

+ "source": {

+ "type": "<source type>"

+ },

+ "sink": {

+ "type": "LakehouseTableSink",

+ "tableActionOption ": "Append"

+ }

+ }

+ }

+]

+```

+## Mapping data flow properties

+When transforming data in mapping data flow, you can read and write to files or tables in Microsoft Fabric Lakehouse. See the corresponding sections for details.

+- [Microsoft Fabric Lakehouse Files in mapping data flow](#microsoft-fabric-lakehouse-files-in-mapping-data-flow)

+- [Microsoft Fabric Lakehouse Table in mapping data flow](#microsoft-fabric-lakehouse-table-in-mapping-data-flow)

+For more information, see the [source transformation](data-flow-source.md) and [sink transformation](data-flow-sink.md) in mapping data flows.

+### Microsoft Fabric Lakehouse Files in mapping data flow

+To use Microsoft Fabric Lakehouse Files dataset as a source or sink dataset in mapping data flow, go to the following sections for the detailed configurations.

+#### Microsoft Fabric Lakehouse Files as a source type

+Microsoft Fabric Lakehouse connector supports the following file formats. Refer to each article for format-based settings.

+- [Avro format](format-avro.md)

+- [Delimited text format](format-delimited-text.md)

+- [JSON format](format-json.md)

+- [ORC format](format-orc.md)

+- [Parquet format](format-parquet.md)

+#### Microsoft Fabric Lakehouse Files as a sink type

+Microsoft Fabric Lakehouse connector supports the following file formats. Refer to each article for format-based settings.

+- [Avro format](format-avro.md)

+- [Delimited text format](format-delimited-text.md)

+- [JSON format](format-json.md)

+- [ORC format](format-orc.md)

+- [Parquet format](format-parquet.md)

+### Microsoft Fabric Lakehouse Table in mapping data flow

+To use Microsoft Fabric Lakehouse Table dataset as a source or sink dataset in mapping data flow, go to the following sections for the detailed configurations.

+#### Microsoft Fabric Lakehouse Table as a source type

+There are no configurable properties under source options.

+#### Microsoft Fabric Lakehouse Table as a sink type

+The following properties are supported in the Mapping Data Flows **sink** section:

+| Name | Description | Required | Allowed values | Data flow script property |

+| - | -- | -- | -- | - |

+| Update method | When you select "Allow insert" alone or when you write to a new delta table, the target receives all incoming rows regardless of the Row policies set. If your data contains rows of other Row policies, they need to be excluded using a preceding Filter transform. <br><br> When all Update methods are selected a Merge is performed, where rows are inserted/deleted/upserted/updated as per the Row Policies set using a preceding Alter Row transform. | yes | `true` or `false` | insertable <br> deletable <br> upsertable <br> updateable |

+| Optimized Write | Achieve higher throughput for write operation via optimizing internal shuffle in Spark executors. As a result, you might notice fewer partitions and files that are of a larger size | no | `true` or `false` | optimizedWrite: true |

+| Auto Compact | After any write operation has completed, Spark will automatically execute the ```OPTIMIZE``` command to re-organize the data, resulting in more partitions if necessary, for better reading performance in the future | no | `true` or `false` | autoCompact: true |

+| Merge Schema | Merge schema option allows schema evolution, i.e. any columns that are present in the current incoming stream but not in the target Delta table is automatically added to its schema. This option is supported across all update methods. | no | `true` or `false` | mergeSchema: true |

+**Example: Microsoft Fabric Lakehouse Table sink**

+```

+sink(allowSchemaDrift: true,

+ΓÇ» ΓÇ» validateSchema: false,

+ΓÇ» ΓÇ» input(

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» CustomerID as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» NameStyle as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» Title as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» FirstName as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» MiddleName as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» LastName as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» Suffix as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» CompanyName as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» SalesPerson as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» EmailAddress as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» Phone as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» PasswordHash as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» PasswordSalt as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» rowguid as string,

+ΓÇ» ΓÇ» ΓÇ» ΓÇ» ModifiedDate as string

+ΓÇ» ΓÇ» ),

+ΓÇ» ΓÇ» deletable:false,

+ΓÇ» ΓÇ» insertable:true,

+ΓÇ» ΓÇ» updateable:false,

+ΓÇ» ΓÇ» upsertable:false,

+ΓÇ» ΓÇ» optimizedWrite: true,

+ΓÇ» ΓÇ» mergeSchema: true,

+ΓÇ» ΓÇ» autoCompact: true,

+ΓÇ» ΓÇ» skipDuplicateMapInputs: true,

+ΓÇ» ΓÇ» skipDuplicateMapOutputs: true) ~> CustomerTable

+```

+## Next steps

+For a list of data stores supported as sources and sinks by the copy activity, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).

+You can only simulate attacks using our approved testing partners:

+- Two virtual machines

+- For this tutorial you'll need to deploy a Load Balancer, a public IP address, Bastion, and two virtual machines. For more information, see [Deploy Load Balancer with DDoS Protection](../load-balancer/tutorial-protect-load-balancer-ddos.md). You can skip the NAT Gateway step in the Deploy Load Balancer with DDoS Protection tutorial.

+In this tutorial, we'll configure DDoS Protection metrics and alerts to monitor for attacks and traffic patterns.

+> For BreakingPoint Cloud, you must first [create a BreakingPoint Cloud account](https://www.ixiacom.com/products/breakingpoint-cloud).

+| Alert (alert type) | Description | MITRE tactics<br>([Learn more](#intentions)) | Severity |

+| | | :-: | - |

+| **A logon from a malicious IP has been detected. [seen multiple times]** | A successful remote authentication for the account [account] and process [process] occurred, however the logon IP address (x.x.x.x) has previously been reported as malicious or highly unusual. A successful attack has probably occurred. Files with the .scr extensions are screen saver files and are normally reside and execute from the Windows system directory. | - | High |

+| **Addition of Guest account to Local Administrators group** | Analysis of host data has detected the addition of the built-in Guest account to the Local Administrators group on %{Compromised Host}, which is strongly associated with attacker activity. | - | Medium |

+| **An event log was cleared** | Machine logs indicate a suspicious event log clearing operation by user: '%{user name}' in Machine: '%{CompromisedEntity}'. The %{log channel} log was cleared. | - | Informational |

+| **Antimalware Action Failed** | Microsoft Antimalware has encountered an error when taking an action on malware or other potentially unwanted software. | - | Medium |

+| **Antimalware Action Taken** | Microsoft Antimalware for Azure has taken an action to protect this machine from malware or other potentially unwanted software. | - | Medium |

+| **Antimalware broad files exclusion in your virtual machine**<br>(VM_AmBroadFilesExclusion) | Files exclusion from antimalware extension with broad exclusion rule was detected in your virtual machine by analyzing the Azure Resource Manager operations in your subscription. Such exclusion practically disabling the Antimalware protection.<br>Attackers might exclude files from the antimalware scan on your virtual machine to prevent detection while running arbitrary code or infecting the machine with malware. | - | Medium |

+| **Antimalware disabled and code execution in your virtual machine**<br>(VM_AmDisablementAndCodeExecution) | Antimalware disabled at the same time as code execution on your virtual machine. This was detected by analyzing Azure Resource Manager operations in your subscription.<br>Attackers disable antimalware scanners to prevent detection while running unauthorized tools or infecting the machine with malware. | - | High |

+| **Antimalware disabled in your virtual machine**<br>(VM_AmDisablement) | Antimalware disabled in your virtual machine. This was detected by analyzing Azure Resource Manager operations in your subscription.<br>Attackers might disable the antimalware on your virtual machine to prevent detection. | Defense Evasion | Medium |

+| **Antimalware file exclusion and code execution in your virtual machine**<br>(VM_AmFileExclusionAndCodeExecution) | File excluded from your antimalware scanner at the same time as code was executed via a custom script extension on your virtual machine. This was detected by analyzing Azure Resource Manager operations in your subscription.<br>Attackers might exclude files from the antimalware scan on your virtual machine to prevent detection while running unauthorized tools or infecting the machine with malware. | Defense Evasion, Execution | High |

+| **Antimalware file exclusion and code execution in your virtual machine**<br>(VM_AmTempFileExclusionAndCodeExecution) | Temporary file exclusion from antimalware extension in parallel to execution of code via custom script extension was detected in your virtual machine by analyzing the Azure Resource Manager operations in your subscription.<br>Attackers might exclude files from the antimalware scan on your virtual machine to prevent detection while running arbitrary code or infecting the machine with malware. | Defense Evasion, Execution | High |

+| **Antimalware file exclusion in your virtual machine**<br>(VM_AmTempFileExclusion) | File excluded from your antimalware scanner on your virtual machine. This was detected by analyzing Azure Resource Manager operations in your subscription.<br>Attackers might exclude files from the antimalware scan on your virtual machine to prevent detection while running unauthorized tools or infecting the machine with malware. | Defense Evasion | Medium |

+| **Antimalware real-time protection was disabled in your virtual machine**<br>(VM_AmRealtimeProtectionDisabled) | Real-time protection disablement of the antimalware extension was detected in your virtual machine by analyzing the Azure Resource Manager operations in your subscription.<br>Attackers might disable real-time protection from the antimalware scan on your virtual machine to avoid detection while running arbitrary code or infecting the machine with malware. | Defense Evasion | Medium |

+| **Antimalware real-time protection was disabled temporarily in your virtual machine**<br>(VM_AmTempRealtimeProtectionDisablement) | Real-time protection temporary disablement of the antimalware extension was detected in your virtual machine by analyzing the Azure Resource Manager operations in your subscription.<br>Attackers might disable real-time protection from the antimalware scan on your virtual machine to avoid detection while running arbitrary code or infecting the machine with malware. | Defense Evasion | Medium |

+| **Antimalware real-time protection was disabled temporarily while code was executed in your virtual machine**<br>(VM_AmRealtimeProtectionDisablementAndCodeExec) | Real-time protection temporary disablement of the antimalware extension in parallel to code execution via custom script extension was detected in your virtual machine by analyzing the Azure Resource Manager operations in your subscription.<br>Attackers might disable real-time protection from the antimalware scan on your virtual machine to avoid detection while running arbitrary code or infecting the machine with malware. | - | High |

+| **Antimalware scans blocked for files potentially related to malware campaigns on your virtual machine (Preview)**<br>(VM_AmMalwareCampaignRelatedExclusion) | An exclusion rule was detected in your virtual machine to prevent your antimalware extension scanning certain files that are suspected of being related to a malware campaign. The rule was detected by analyzing the Azure Resource Manager operations in your subscription. Attackers might exclude files from antimalware scans to prevent detection while running arbitrary code or infecting the machine with malware. | Defense Evasion | Medium |

+| **Antimalware temporarily disabled in your virtual machine**<br>(VM_AmTemporarilyDisablement) | Antimalware temporarily disabled in your virtual machine. This was detected by analyzing Azure Resource Manager operations in your subscription.<br>Attackers might disable the antimalware on your virtual machine to prevent detection. | - | Medium |

+| **Antimalware unusual file exclusion in your virtual machine**<br>(VM_UnusualAmFileExclusion) | Unusual file exclusion from antimalware extension was detected in your virtual machine by analyzing the Azure Resource Manager operations in your subscription.<br>Attackers might exclude files from the antimalware scan on your virtual machine to prevent detection while running arbitrary code or infecting the machine with malware. | Defense Evasion | Medium |

+| **Communication with suspicious domain identified by threat intelligence**<br>(AzureDNS_ThreatIntelSuspectDomain) | Communication with suspicious domain was detected by analyzing DNS transactions from your resource and comparing against known malicious domains identified by threat intelligence feeds. Communication to malicious domains is frequently performed by attackers and could imply that your resource is compromised. | Initial Access, Persistence, Execution, Command And Control, Exploitation | Medium |

+| **Detected actions indicative of disabling and deleting IIS log files** | Analysis of host data detected actions that show IIS log files being disabled and/or deleted. | - | Medium |

+| **Detected anomalous mix of upper and lower case characters in command-line** | Analysis of host data on %{Compromised Host} detected a command line with anomalous mix of upper and lower case characters. This kind of pattern, while possibly benign, is also typical of attackers trying to hide from case-sensitive or hash-based rule matching when performing administrative tasks on a compromised host. | - | Medium |

+| **Detected change to a registry key that can be abused to bypass UAC** | Analysis of host data on %{Compromised Host} detected that a registry key that can be abused to bypass UAC (User Account Control) was changed. This kind of configuration, while possibly benign, is also typical of attacker activity when trying to move from unprivileged (standard user) to privileged (for example administrator) access on a compromised host. | - | Medium |

+| **Detected decoding of an executable using built-in certutil.exe tool** | Analysis of host data on %{Compromised Host} detected that certutil.exe, a built-in administrator utility, was being used to decode an executable instead of its mainstream purpose that relates to manipulating certificates and certificate data. Attackers are known to abuse functionality of legitimate administrator tools to perform malicious actions, for example using a tool such as certutil.exe to decode a malicious executable that will then be subsequently executed. | - | High |

+| **Detected enabling of the WDigest UseLogonCredential registry key** | Analysis of host data detected a change in the registry key HKLM\SYSTEM\ CurrentControlSet\Control\SecurityProviders\WDigest\ "UseLogonCredential". Specifically this key has been updated to allow logon credentials to be stored in clear text in LSA memory. Once enabled, an attacker can dump clear text passwords from LSA memory with credential harvesting tools such as Mimikatz. | - | Medium |

+| **Detected encoded executable in command line data** | Analysis of host data on %{Compromised Host} detected a base-64 encoded executable. This has previously been associated with attackers attempting to construct executables on-the-fly through a sequence of commands, and attempting to evade intrusion detection systems by ensuring that no individual command would trigger an alert. This could be legitimate activity, or an indication of a compromised host. | - | High |

+| **Detected obfuscated command line** | Attackers use increasingly complex obfuscation techniques to evade detections that run against the underlying data. Analysis of host data on %{Compromised Host} detected suspicious indicators of obfuscation on the commandline. | - | Informational |

+| **Detected possible execution of keygen executable** | Analysis of host data on %{Compromised Host} detected execution of a process whose name is indicative of a keygen tool; such tools are typically used to defeat software licensing mechanisms but their download is often bundled with other malicious software. Activity group GOLD has been known to make use of such keygens to covertly gain back door access to hosts that they compromise. | - | Medium |

+| **Detected possible execution of malware dropper** | Analysis of host data on %{Compromised Host} detected a filename that has previously been associated with one of activity group GOLD's methods of installing malware on a victim host. | - | High |

+| **Detected possible local reconnaissance activity** | Analysis of host data on %{Compromised Host} detected a combination of systeminfo commands that has previously been associated with one of activity group GOLD's methods of performing reconnaissance activity. While 'systeminfo.exe' is a legitimate Windows tool, executing it twice in succession in the way that has occurred here is rare. | - | |

+| **Detected potentially suspicious use of Telegram tool** | Analysis of host data shows installation of Telegram, a free cloud-based instant messaging service that exists both for mobile and desktop system. Attackers are known to abuse this service to transfer malicious binaries to any other computer, phone, or tablet. | - | Medium |

+| **Detected suppression of legal notice displayed to users at logon** | Analysis of host data on %{Compromised Host} detected changes to the registry key that controls whether a legal notice is displayed to users when they log on. Microsoft security analysis has determined that this is a common activity undertaken by attackers after having compromised a host. | - | Low |

+| **Detected suspicious combination of HTA and PowerShell** | mshta.exe (Microsoft HTML Application Host) which is a signed Microsoft binary is being used by the attackers to launch malicious PowerShell commands. Attackers often resort to having an HTA file with inline VBScript. When a victim browses to the HTA file and chooses to run it, the PowerShell commands and scripts that it contains are executed. Analysis of host data on %{Compromised Host} detected mshta.exe launching PowerShell commands. | - | Medium |

+| **Detected suspicious commandline arguments** | Analysis of host data on %{Compromised Host} detected suspicious commandline arguments that have been used in conjunction with a reverse shell used by activity group HYDROGEN. | - | High |

+| **Detected suspicious commandline used to start all executables in a directory** | Analysis of host data has detected a suspicious process running on %{Compromised Host}. The commandline indicates an attempt to start all executables (*.exe) that may reside in a directory. This could be an indication of a compromised host. | - | Medium |

+| **Detected suspicious credentials in commandline** | Analysis of host data on %{Compromised Host} detected a suspicious password being used to execute a file by activity group BORON. This activity group has been known to use this password to execute Pirpi malware on a victim host. | - | High |

+| **Detected suspicious document credentials** | Analysis of host data on %{Compromised Host} detected a suspicious, common precomputed password hash used by malware being used to execute a file. Activity group HYDROGEN has been known to use this password to execute malware on a victim host. | - | High |

+| **Detected suspicious execution of VBScript.Encode command** | Analysis of host data on %{Compromised Host} detected the execution of VBScript.Encode command. This encodes the scripts into unreadable text, making it more difficult for users to examine the code. Microsoft threat research shows that attackers often use encoded VBscript files as part of their attack to evade detection systems. This could be legitimate activity, or an indication of a compromised host. | - | Medium |

+| **Detected suspicious execution via rundll32.exe** | Analysis of host data on %{Compromised Host} detected rundll32.exe being used to execute a process with an uncommon name, consistent with the process naming scheme previously seen used by activity group GOLD when installing their first stage implant on a compromised host. | - | High |

+| **Detected suspicious file cleanup commands** | Analysis of host data on %{Compromised Host} detected a combination of systeminfo commands that has previously been associated with one of activity group GOLD's methods of performing post-compromise self-cleanup activity. While 'systeminfo.exe' is a legitimate Windows tool, executing it twice in succession, followed by a delete command in the way that has occurred here is rare. | - | High |

+| **Detected suspicious file creation** | Analysis of host data on %{Compromised Host} detected creation or execution of a process that has previously indicated post-compromise action taken on a victim host by activity group BARIUM. This activity group has been known to use this technique to download more malware to a compromised host after an attachment in a phishing doc has been opened. | - | High |

+| **Detected suspicious named pipe communications** | Analysis of host data on %{Compromised Host} detected data being written to a local named pipe from a Windows console command. Named pipes are known to be a channel used by attackers to task and communicate with a malicious implant. This could be legitimate activity, or an indication of a compromised host. | - | High |

+| **Detected suspicious network activity** | Analysis of network traffic from %{Compromised Host} detected suspicious network activity. Such traffic, while possibly benign, is typically used by an attacker to communicate with malicious servers for downloading of tools, command-and-control and exfiltration of data. Typical related attacker activity includes copying remote administration tools to a compromised host and exfiltrating user data from it. | - | Low |

+| **Detected suspicious new firewall rule** | Analysis of host data detected a new firewall rule has been added via netsh.exe to allow traffic from an executable in a suspicious location. | - | Medium |

+| **Detected suspicious use of Cacls to lower the security state of the system** | Attackers use myriad ways like brute force, spear phishing etc. to achieve initial compromise and get a foothold on the network. Once initial compromise is achieved they often take steps to lower the security settings of a system. CaclsΓÇöshort for change access control list is Microsoft Windows native command-line utility often used for modifying the security permission on folders and files. A lot of time the binary is used by the attackers to lower the security settings of a system. This is done by giving Everyone full access to some of the system binaries like ftp.exe, net.exe, wscript.exe etc. Analysis of host data on %{Compromised Host} detected suspicious use of Cacls to lower the security of a system. | - | Medium |

+| **Detected suspicious use of FTP -s Switch** | Analysis of process creation data from the %{Compromised Host} detected the use of the FTP "-s:filename" switch. This switch is used to specify an FTP script file for the client to run. Malware or malicious processes are known to use this FTP switch (-s:filename) to point to a script file, which is configured to connect to a remote FTP server and download more malicious binaries. | - | Medium |

+| **Detected suspicious use of Pcalua.exe to launch executable code** | Analysis of host data on %{Compromised Host} detected the use of pcalua.exe to launch executable code. Pcalua.exe is component of the Microsoft Windows "Program Compatibility Assistant", which detects compatibility issues during the installation or execution of a program. Attackers are known to abuse functionality of legitimate Windows system tools to perform malicious actions, for example using pcalua.exe with the -a switch to launch malicious executables either locally or from remote shares. | - | Medium |

+| **Detected the disabling of critical services** | The analysis of host data on %{Compromised Host} detected execution of "net.exe stop" command being used to stop critical services like SharedAccess or the Windows Security app. The stopping of either of these services can be indication of a malicious behavior. | - | Medium |

+| **Digital currency mining related behavior detected** | Analysis of host data on %{Compromised Host} detected the execution of a process or command normally associated with digital currency mining. | - | High |

+| **Dynamic PS script construction** | Analysis of host data on %{Compromised Host} detected a PowerShell script being constructed dynamically. Attackers sometimes use this approach of progressively building up a script in order to evade IDS systems. This could be legitimate activity, or an indication that one of your machines has been compromised. | - | Medium |

+| **Executable found running from a suspicious location** | Analysis of host data detected an executable file on %{Compromised Host} that is running from a location in common with known suspicious files. This executable could either be legitimate activity, or an indication of a compromised host. | - | High |

+| **Fileless attack behavior detected**<br>(VM_FilelessAttackBehavior.Windows) | The memory of the process specified contains behaviors commonly used by fileless attacks. Specific behaviors include:<br>1) Shellcode, which is a small piece of code typically used as the payload in the exploitation of a software vulnerability.<br>2) Active network connections. See NetworkConnections below for details.<br>3) Function calls to security sensitive operating system interfaces. See Capabilities below for referenced OS capabilities.<br>4) Contains a thread that was started in a dynamically allocated code segment. This is a common pattern for process injection attacks. | Defense Evasion | Low |

+| **Fileless attack technique detected**<br>(VM_FilelessAttackTechnique.Windows) | The memory of the process specified below contains evidence of a fileless attack technique. Fileless attacks are used by attackers to execute code while evading detection by security software. Specific behaviors include:<br>1) Shellcode, which is a small piece of code typically used as the payload in the exploitation of a software vulnerability.<br>2) Executable image injected into the process, such as in a code injection attack.<br>3) Active network connections. See NetworkConnections below for details.<br>4) Function calls to security sensitive operating system interfaces. See Capabilities below for referenced OS capabilities.<br>5) Process hollowing, which is a technique used by malware in which a legitimate process is loaded on the system to act as a container for hostile code.<br>6) Contains a thread that was started in a dynamically allocated code segment. This is a common pattern for process injection attacks. | Defense Evasion, Execution | High |

+| **Fileless attack toolkit detected**<br>(VM_FilelessAttackToolkit.Windows) | The memory of the process specified contains a fileless attack toolkit: [toolkit name]. Fileless attack toolkits use techniques that minimize or eliminate traces of malware on disk, and greatly reduce the chances of detection by disk-based malware scanning solutions. Specific behaviors include:<br>1) Well-known toolkits and crypto mining software.<br>2) Shellcode, which is a small piece of code typically used as the payload in the exploitation of a software vulnerability.<br>3) Injected malicious executable in process memory. | Defense Evasion, Execution | Medium |

+| **High risk software detected** | Analysis of host data from %{Compromised Host} detected the usage of software that has been associated with the installation of malware in the past. A common technique utilized in the distribution of malicious software is to package it within otherwise benign tools such as the one seen in this alert. When you use these tools, the malware can be silently installed in the background. | - | Medium |

+| **Local Administrators group members were enumerated** | Machine logs indicate a successful enumeration on group %{Enumerated Group Domain Name}\%{Enumerated Group Name}. Specifically, %{Enumerating User Domain Name}\%{Enumerating User Name} remotely enumerated the members of the %{Enumerated Group Domain Name}\%{Enumerated Group Name} group. This activity could either be legitimate activity, or an indication that a machine in your organization has been compromised and used to reconnaissance %{vmname}. | - | Informational |

+| **Malicious firewall rule created by ZINC server implant [seen multiple times]** | A firewall rule was created using techniques that match a known actor, ZINC. The rule was possibly used to open a port on %{Compromised Host} to allow for Command & Control communications. This behavior was seen [x] times today on the following machines: [Machine names] | - | High |

+| **Malicious SQL activity** | Machine logs indicate that '%{process name}' was executed by account: %{user name}. This activity is considered malicious. | - | High |

+| **Multiple Domain Accounts Queried** | Analysis of host data has determined that an unusual number of distinct domain accounts are being queried within a short time period from %{Compromised Host}. This kind of activity could be legitimate, but can also be an indication of compromise. | - | Medium |

+| **Possible credential dumping detected [seen multiple times]** | Analysis of host data has detected use of native windows tool (for example, sqldumper.exe) being used in a way that allows to extract credentials from memory. Attackers often use these techniques to extract credentials that they then further use for lateral movement and privilege escalation. This behavior was seen [x] times today on the following machines: [Machine names] | - | Medium |

+| **Potential attempt to bypass AppLocker detected** | Analysis of host data on %{Compromised Host} detected a potential attempt to bypass AppLocker restrictions. AppLocker can be configured to implement a policy that limits what executables are allowed to run on a Windows system. The command-line pattern similar to that identified in this alert has been previously associated with attacker attempts to circumvent AppLocker policy by using trusted executables (allowed by AppLocker policy) to execute untrusted code. This could be legitimate activity, or an indication of a compromised host. | - | High |

+| **PsExec execution detected**<br>(VM_RunByPsExec) | Analysis of host data indicates that the process %{Process Name} was executed by PsExec utility. PsExec can be used for running processes remotely. This technique might be used for malicious purposes. | Lateral Movement, Execution | Informational |

+| **Rare SVCHOST service group executed**<br>(VM_SvcHostRunInRareServiceGroup) | The system process SVCHOST was observed running a rare service group. Malware often uses SVCHOST to masquerade its malicious activity. | Defense Evasion, Execution | Informational |

+| **Sticky keys attack detected** | Analysis of host data indicates that an attacker may be subverting an accessibility binary (for example sticky keys, onscreen keyboard, narrator) in order to provide backdoor access to the host %{Compromised Host}. | - | Medium |

+| **Successful brute force attack**<br>(VM_LoginBruteForceSuccess) | Several sign in attempts were detected from the same source. Some successfully authenticated to the host.<br>This resembles a burst attack, in which an attacker performs numerous authentication attempts to find valid account credentials. | Exploitation | Medium/High |

+| **Suspect integrity level indicative of RDP hijacking** | Analysis of host data has detected the tscon.exe running with SYSTEM privileges - this can be indicative of an attacker abusing this binary in order to switch context to any other logged on user on this host; it's a known attacker technique to compromise more user accounts and move laterally across a network. | - | Medium |

+| **Suspect service installation** | Analysis of host data has detected the installation of tscon.exe as a service: this binary being started as a service potentially allows an attacker to trivially switch to any other logged on user on this host by hijacking RDP connections; it's a known attacker technique to compromise more user accounts and move laterally across a network. | - | Medium |

+| **Suspected Kerberos Golden Ticket attack parameters observed** | Analysis of host data detected commandline parameters consistent with a Kerberos Golden Ticket attack. | - | Medium |

+| **Suspicious Account Creation Detected** | Analysis of host data on %{Compromised Host} detected creation or use of a local account %{Suspicious account name} : this account name closely resembles a standard Windows account or group name '%{Similar To Account Name}'. This is potentially a rogue account created by an attacker, so named in order to avoid being noticed by a human administrator. | - | Medium |

+| **Suspicious Activity Detected**<br>(VM_SuspiciousActivity) | Analysis of host data has detected a sequence of one or more processes running on %{machine name} that have historically been associated with malicious activity. While individual commands may appear benign the alert is scored based on an aggregation of these commands. This could either be legitimate activity, or an indication of a compromised host. | Execution | Medium |

+| **Suspicious authentication activity**<br>(VM_LoginBruteForceValidUserFailed) | Although none of them succeeded, some of them used accounts were recognized by the host. This resembles a dictionary attack, in which an attacker performs numerous authentication attempts using a dictionary of predefined account names and passwords in order to find valid credentials to access the host. This indicates that some of your host account names might exist in a well-known account name dictionary. | Probing | Medium |

+| **Suspicious code segment detected** | Indicates that a code segment has been allocated by using non-standard methods, such as reflective injection and process hollowing. The alert provides more characteristics of the code segment that have been processed to provide context for the capabilities and behaviors of the reported code segment. | - | Medium |

+| **Suspicious double extension file executed** | Analysis of host data indicates an execution of a process with a suspicious double extension. This extension may trick users into thinking files are safe to be opened and might indicate the presence of malware on the system. | - | High |

+| **Suspicious download using Certutil detected [seen multiple times]** | Analysis of host data on %{Compromised Host} detected the use of certutil.exe, a built-in administrator utility, for the download of a binary instead of its mainstream purpose that relates to manipulating certificates and certificate data. Attackers are known to abuse functionality of legitimate administrator tools to perform malicious actions, for example using certutil.exe to download and decode a malicious executable that will then be subsequently executed. This behavior was seen [x] times today on the following machines: [Machine names] | - | Medium |

+| **Suspicious download using Certutil detected** | Analysis of host data on %{Compromised Host} detected the use of certutil.exe, a built-in administrator utility, for the download of a binary instead of its mainstream purpose that relates to manipulating certificates and certificate data. Attackers are known to abuse functionality of legitimate administrator tools to perform malicious actions, for example using certutil.exe to download and decode a malicious executable that will then be subsequently executed. | - | Medium |

+| **Suspicious PowerShell Activity Detected** | Analysis of host data detected a PowerShell script running on %{Compromised Host} that has features in common with known suspicious scripts. This script could either be legitimate activity, or an indication of a compromised host. | - | High |

+| **Suspicious PowerShell cmdlets executed** | Analysis of host data indicates execution of known malicious PowerShell PowerSploit cmdlets. | - | Medium |

+| **Suspicious process executed [seen multiple times]** | Machine logs indicate that the suspicious process: '%{Suspicious Process}' was running on the machine, often associated with attacker attempts to access credentials. This behavior was seen [x] times today on the following machines: [Machine names] | - | High |

+| **Suspicious process executed** | Machine logs indicate that the suspicious process: '%{Suspicious Process}' was running on the machine, often associated with attacker attempts to access credentials. | - | High |

+| **Suspicious process name detected [seen multiple times]** | Analysis of host data on %{Compromised Host} detected a process whose name is suspicious, for example corresponding to a known attacker tool or named in a way that is suggestive of attacker tools that try to hide in plain sight. This process could be legitimate activity, or an indication that one of your machines has been compromised. This behavior was seen [x] times today on the following machines: [Machine names] | - | Medium |

+| **Suspicious process name detected** | Analysis of host data on %{Compromised Host} detected a process whose name is suspicious, for example corresponding to a known attacker tool or named in a way that is suggestive of attacker tools that try to hide in plain sight. This process could be legitimate activity, or an indication that one of your machines has been compromised. | - | Medium |

+| **Suspicious process termination burst**<br>(VM_TaskkillBurst) | Analysis of host data indicates a suspicious process termination burst in %{Machine Name}. Specifically, %{NumberOfCommands} processes were killed between %{Begin} and %{Ending}. | Defense Evasion | Low |

+| **Suspicious SQL activity** | Machine logs indicate that '%{process name}' was executed by account: %{user name}. This activity is uncommon with this account. | - | Medium |

+| **Suspicious SVCHOST process executed** | The system process SVCHOST was observed running in an abnormal context. Malware often uses SVCHOST to masquerade its malicious activity. | - | High |

+| **Suspicious system process executed**<br>(VM_SystemProcessInAbnormalContext) | The system process %{process name} was observed running in an abnormal context. Malware often uses this process name to masquerade its malicious activity. | Defense Evasion, Execution | High |

+| **Suspicious Volume Shadow Copy Activity** | Analysis of host data has detected a shadow copy deletion activity on the resource. Volume Shadow Copy (VSC) is an important artifact that stores data snapshots. Some malware and specifically Ransomware, targets VSC to sabotage backup strategies. | - | High |

+| **Suspicious WindowPosition registry value detected** | Analysis of host data on %{Compromised Host} detected an attempted WindowPosition registry configuration change that could be indicative of hiding application windows in nonvisible sections of the desktop. This could be legitimate activity, or an indication of a compromised machine: this type of activity has been previously associated with known adware (or unwanted software) such as Win32/OneSystemCare and Win32/SystemHealer and malware such as Win32/Creprote. When the WindowPosition value is set to 201329664, (Hex: 0x0c00 0c00, corresponding to X-axis=0c00 and the Y-axis=0c00) this places the console app's window in a non-visible section of the user's screen in an area that is hidden from view below the visible start menu/taskbar. Known suspect Hex value includes, but not limited to c000c000 | - | Low |

+| **Suspiciously named process detected** | Analysis of host data on %{Compromised Host} detected a process whose name is very similar to but different from a very commonly run process (%{Similar To Process Name}). While this process could be benign attackers are known to sometimes hide in plain sight by naming their malicious tools to resemble legitimate process names. | - | Medium |

+| **Unusual config reset in your virtual machine**<br>(VM_VMAccessUnusualConfigReset) | An unusual config reset was detected in your virtual machine by analyzing the Azure Resource Manager operations in your subscription.<br>While this action may be legitimate, attackers can try utilizing VM Access extension to reset the configuration in your virtual machine and compromise it. | Credential Access | Medium |

+| **Unusual process execution detected** | Analysis of host data on %{Compromised Host} detected the execution of a process by %{User Name} that was unusual. Accounts such as %{User Name} tend to perform a limited set of operations, this execution was determined to be out of character and may be suspicious. | - | High |

+| **Unusual user password reset in your virtual machine**<br>(VM_VMAccessUnusualPasswordReset) | An unusual user password reset was detected in your virtual machine by analyzing the Azure Resource Manager operations in your subscription.<br>While this action may be legitimate, attackers can try utilizing the VM Access extension to reset the credentials of a local user in your virtual machine and compromise it. | Credential Access | Medium |

+| **Unusual user SSH key reset in your virtual machine**<br>(VM_VMAccessUnusualSSHReset) | An unusual user SSH key reset was detected in your virtual machine by analyzing the Azure Resource Manager operations in your subscription.<br>While this action may be legitimate, attackers can try utilizing VM Access extension to reset SSH key of a user account in your virtual machine and compromise it. | Credential Access | Medium |

+| **VBScript HTTP object allocation detected** | Creation of a VBScript file using Command Prompt has been detected. The following script contains HTTP object allocation command. This action can be used to download malicious files. | | |

+| **Suspicious installation of GPU extension in your virtual machine (Preview)** <br> (VM_GPUDriverExtensionUnusualExecution) | Suspicious installation of a GPU extension was detected in your virtual machine by analyzing the Azure Resource Manager operations in your subscription. Attackers may use the GPU driver extension to install GPU drivers on your virtual machine via the Azure Resource Manager to perform cryptojacking. | Impact | Low |

+| Alert (alert type) | Description | MITRE tactics<br>([Learn more](#intentions)) | Severity |

+| | | :-: | - |

+| **Exposed Postgres service with trust authentication configuration in Kubernetes detected (Preview)**<br>(K8S_ExposedPostgresTrustAuth) | Kubernetes cluster configuration analysis detected exposure of a Postgres service by a load balancer. The service is configured with trust authentication method, which doesn't require credentials. | InitialAccess | Medium |

+| **Exposed Postgres service with risky configuration in Kubernetes detected (Preview)**<br>(K8S_ExposedPostgresBroadIPRange) | Kubernetes cluster configuration analysis detected exposure of a Postgres service by a load balancer with a risky configuration. Exposing the service to a wide range of IP addresses poses a security risk. | InitialAccess | Medium |

+| **Attempt to create a new Linux namespace from a container detected**<br>(K8S.NODE_NamespaceCreation) ^{[1](#footnote1)} | Analysis of processes running within a container in Kubernetes cluster detected an attempt to create a new Linux namespace. While this behavior might be legitimate, it might indicate that an attacker tries to escape from the container to the node. Some CVE-2022-0185 exploitations use this technique. | PrivilegeEscalation | Medium |

+| **A history file has been cleared**<br>(K8S.NODE_HistoryFileCleared) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected that the command history log file has been cleared. Attackers may do this to cover their tracks. The operation was performed by the specified user account. | DefenseEvasion | Medium |

+| **Abnormal activity of managed identity associated with Kubernetes (Preview)**<br>(K8S_AbnormalMiActivity) | Analysis of Azure Resource Manager operations detected an abnormal behavior of a managed identity used by an AKS addon. The detected activity isn\'t consistent with the behavior of the associated addon. While this activity can be legitimate, such behavior might indicate that the identity was gained by an attacker, possibly from a compromised container in the Kubernetes cluster. | Lateral Movement | Medium |

+| **Abnormal Kubernetes service account operation detected**<br>(K8S_ServiceAccountRareOperation) | Kubernetes audit log analysis detected abnormal behavior by a service account in your Kubernetes cluster. The service account was used for an operation, which isn't common for this service account. While this activity can be legitimate, such behavior might indicate that the service account is being used for malicious purposes. | Lateral Movement, Credential Access | Medium |

+| **An uncommon connection attempt detected**<br>(K8S.NODE_SuspectConnection) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected an uncommon connection attempt utilizing a socks protocol. This is very rare in normal operations, but a known technique for attackers attempting to bypass network-layer detections. | Execution, Exfiltration, Exploitation | Medium |

+| **Anomalous pod deployment (Preview)**<br>(K8S_AnomalousPodDeployment) ^{[3](#footnote3)} | Kubernetes audit log analysis detected pod deployment which is anomalous based on previous pod deployment activity. This activity is considered an anomaly when taking into account how the different features seen in the deployment operation are in relations to one another. The features monitored include the container image registry used, the account performing the deployment, day of the week, how often this account performs pod deployments, user agent used in the operation, whether this is a namespace to which pod deployments often occur, and other features. Top contributing reasons for raising this alert as anomalous activity are detailed under the alert's extended properties. | Execution | Medium |

+| **Anomalous secret access (Preview)**<br>(K8S_AnomalousSecretAccess) ^{[2](#footnote2)} | Kubernetes audit log analysis detected secret access request which is anomalous based on previous secret access activity. This activity is considered an anomaly when taking into account how the different features seen in the secret access operation are in relations to one another. The features monitored by this analytics include the user name used, the name of the secret, the name of the namespace, user agent used in the operation, or other features. Top contributing reasons for raising this alert as anomalous activity are detailed under the alert extended properties. | CredentialAccess | Medium |

+| **Attempt to stop apt-daily-upgrade.timer service detected**<br>(K8S.NODE_TimerServiceDisabled) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected an attempt to stop apt-daily-upgrade.timer service. Attackers have been observed stopping this service to download malicious files and grant execution privileges for their attacks. This activity can also happen if the service is updated through normal administrative actions. | DefenseEvasion | Informational |

+| **Behavior similar to common Linux bots detected (Preview)**<br>(K8S.NODE_CommonBot) | Analysis of processes running within a container or directly on a Kubernetes node, has detected the execution of a process normally associated with common Linux botnets. | Execution, Collection, Command And Control | Medium |

+| **Command within a container running with high privileges**<br>(K8S.NODE_PrivilegedExecutionInContainer) ^{[1](#footnote1)} | Machine logs indicate that a privileged command was run in a Docker container. A privileged command has extended privileges on the host machine. | PrivilegeEscalation | Low |

+| **Container running in privileged mode**<br>(K8S.NODE_PrivilegedContainerArtifacts) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected the execution of a Docker command that is running a privileged container. The privileged container has full access to the hosting pod or host resource. If compromised, an attacker may use the privileged container to gain access to the hosting pod or host. | PrivilegeEscalation, Execution | Low |

+| **Container with a sensitive volume mount detected**<br>(K8S_SensitiveMount) | Kubernetes audit log analysis detected a new container with a sensitive volume mount. The volume that was detected is a hostPath type which mounts a sensitive file or folder from the node to the container. If the container gets compromised, the attacker can use this mount for gaining access to the node. | Privilege Escalation | Medium |

+| **CoreDNS modification in Kubernetes detected**<br>(K8S_CoreDnsModification) ^{[2](#footnote2)} ^{[3](#footnote3)} | Kubernetes audit log analysis detected a modification of the CoreDNS configuration. The configuration of CoreDNS can be modified by overriding its configmap. While this activity can be legitimate, if attackers have permissions to modify the configmap, they can change the behavior of the cluster's DNS server and poison it. | Lateral Movement | Low |

+| **Creation of admission webhook configuration detected**<br>(K8S_AdmissionController) ^{[3](#footnote3)} | Kubernetes audit log analysis detected a new admission webhook configuration. Kubernetes has two built-in generic admission controllers: MutatingAdmissionWebhook and ValidatingAdmissionWebhook. The behavior of these admission controllers is determined by an admission webhook that the user deploys to the cluster. The usage of such admission controllers can be legitimate, however attackers can use such webhooks for modifying the requests (in case of MutatingAdmissionWebhook) or inspecting the requests and gain sensitive information (in case of ValidatingAdmissionWebhook). | Credential Access, Persistence | Low |

+| **Detected file download from a known malicious source**<br>(K8S.NODE_SuspectDownload) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a download of a file from a source frequently used to distribute malware. | PrivilegeEscalation, Execution, Exfiltration, Command And Control | Medium |

+| **Detected suspicious file download**<br>(K8S.NODE_SuspectDownloadArtifacts) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a suspicious download of a remote file. | Persistence | Low |

+| **Detected suspicious use of the nohup command**<br>(K8S.NODE_SuspectNohup) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a suspicious use of the nohup command. Attackers have been seen using the command nohup to run hidden files from a temporary directory to allow their executables to run in the background. It's rare to see this command run on hidden files located in a temporary directory. | Persistence, DefenseEvasion | Medium |

+| **Detected suspicious use of the useradd command**<br>(K8S.NODE_SuspectUserAddition) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a suspicious use of the useradd command. | Persistence | Medium |

+| **Digital currency mining container detected**<br>(K8S_MaliciousContainerImage) ^{[3](#footnote3)} | Kubernetes audit log analysis detected a container that has an image associated with a digital currency mining tool. | Execution | High |

+| **Digital currency mining related behavior detected**<br>(K8S.NODE_DigitalCurrencyMining) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected an execution of a process or command normally associated with digital currency mining. | Execution | High |

+| **Docker build operation detected on a Kubernetes node**<br>(K8S.NODE_ImageBuildOnNode) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a build operation of a container image on a Kubernetes node. While this behavior might be legitimate, attackers might build their malicious images locally to avoid detection. | DefenseEvasion | Low |

+| **Excessive role permissions assigned in Kubernetes cluster (Preview)**<br>(K8S_ServiceAcountPermissionAnomaly) ^{[3](#footnote3)} | Analysis of the Kubernetes audit logs detected an excessive permissions role assignment to your cluster. The listed permissions for the assigned roles are uncommon to the specific service account. This detection considers previous role assignments to the same service account across clusters monitored by Azure, volume per permission, and the impact of the specific permission. The anomaly detection model used for this alert takes into account how this permission is used across all clusters monitored by Microsoft Defender for Cloud. | Privilege Escalation | Low |

+| **Executable found running from a suspicious location (Preview)**<br>(K8S.NODE_SuspectExecutablePath) | Analysis of processes running within a container or directly on a Kubernetes node, has detected an executable file that is running from a location associated with known suspicious files. This executable could either be legitimate activity, or an indication of a compromised system. | Execution | Medium |

+| **Exposed Kubeflow dashboard detected**<br>(K8S_ExposedKubeflow) | The Kubernetes audit log analysis detected exposure of the Istio Ingress by a load balancer in a cluster that runs Kubeflow. This action might expose the Kubeflow dashboard to the internet. If the dashboard is exposed to the internet, attackers can access it and run malicious containers or code on the cluster. Find more details in the following article: <https://aka.ms/exposedkubeflow-blog> | Initial Access | Medium |

+| **Exposed Kubernetes dashboard detected**<br>(K8S_ExposedDashboard) | Kubernetes audit log analysis detected exposure of the Kubernetes Dashboard by a LoadBalancer service. Exposed dashboard allows an unauthenticated access to the cluster management and poses a security threat. | Initial Access | High |

+| **Exposed Kubernetes service detected**<br>(K8S_ExposedService) | The Kubernetes audit log analysis detected exposure of a service by a load balancer. This service is related to a sensitive application that allows high impact operations in the cluster such as running processes on the node or creating new containers. In some cases, this service doesn't require authentication. If the service doesn't require authentication, exposing it to the internet poses a security risk. | Initial Access | Medium |

+| **Exposed Redis service in AKS detected**<br>(K8S_ExposedRedis) | The Kubernetes audit log analysis detected exposure of a Redis service by a load balancer. If the service doesn't require authentication, exposing it to the internet poses a security risk. | Initial Access | Low |

+| **Indicators associated with DDOS toolkit detected**<br>(K8S.NODE_KnownLinuxDDoSToolkit) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected file names that are part of a toolkit associated with malware capable of launching DDoS attacks, opening ports and services, and taking full control over the infected system. This could also possibly be legitimate activity. | Persistence, LateralMovement, Execution, Exploitation | Medium |

+| **K8S API requests from proxy IP address detected**<br>(K8S_TI_Proxy) ^{[3](#footnote3)} | Kubernetes audit log analysis detected API requests to your cluster from an IP address that is associated with proxy services, such as TOR. While this behavior can be legitimate, it's often seen in malicious activities, when attackers try to hide their source IP. | Execution | Low |

+| **Kubernetes events deleted**<br>(K8S_DeleteEvents) ^{[2](#footnote2)} ^{[3](#footnote3)} | Defender for Cloud detected that some Kubernetes events have been deleted. Kubernetes events are objects in Kubernetes that contain information about changes in the cluster. Attackers might delete those events for hiding their operations in the cluster. | Defense Evasion | Low |

+| **Kubernetes penetration testing tool detected**<br>(K8S_PenTestToolsKubeHunter) | Kubernetes audit log analysis detected usage of Kubernetes penetration testing tool in the AKS cluster. While this behavior can be legitimate, attackers might use such public tools for malicious purposes. | Execution | Low |

+| **Manipulation of host firewall detected**<br>(K8S.NODE_FirewallDisabled) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a possible manipulation of the on-host firewall. Attackers will often disable this to exfiltrate data. | DefenseEvasion, Exfiltration | Medium |

+| **Microsoft Defender for Cloud test alert (not a threat).**<br>(K8S.NODE_EICAR) ^{[1](#footnote1)} | This is a test alert generated by Microsoft Defender for Cloud. No further action is needed. | Execution | High |

+| **New container in the kube-system namespace detected**<br>(K8S_KubeSystemContainer) ^{[3](#footnote3)} | Kubernetes audit log analysis detected a new container in the kube-system namespace that isn't among the containers that normally run in this namespace. The kube-system namespaces shouldn't contain user resources. Attackers can use this namespace for hiding malicious components. | Persistence | Low |

+| **New high privileges role detected**<br>(K8S_HighPrivilegesRole) ^{[3](#footnote3)} | Kubernetes audit log analysis detected a new role with high privileges. A binding to a role with high privileges gives the user\group high privileges in the cluster. Unnecessary privileges might cause privilege escalation in the cluster. | Persistence | Low |

+| **Possible attack tool detected**<br>(K8S.NODE_KnownLinuxAttackTool) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a suspicious tool invocation. This tool is often associated with malicious users attacking others. | Execution, Collection, Command And Control, Probing | Medium |

+| **Possible backdoor detected**<br>(K8S.NODE_LinuxBackdoorArtifact) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a suspicious file being downloaded and run. This activity has previously been associated with installation of a backdoor. | Persistence, DefenseEvasion, Execution, Exploitation | Medium |

+| **Possible command line exploitation attempt**<br>(K8S.NODE_ExploitAttempt) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a possible exploitation attempt against a known vulnerability. | Exploitation | Medium |

+| **Possible credential access tool detected**<br>(K8S.NODE_KnownLinuxCredentialAccessTool) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a possible known credential access tool was running on the container, as identified by the specified process and commandline history item. This tool is often associated with attacker attempts to access credentials. | CredentialAccess | Medium |

+| **Possible Cryptocoinminer download detected**<br>(K8S.NODE_CryptoCoinMinerDownload) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected download of a file normally associated with digital currency mining. | DefenseEvasion, Command And Control, Exploitation | Medium |

+| **Possible data exfiltration detected**<br>(K8S.NODE_DataEgressArtifacts) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a possible data egress condition. Attackers will often egress data from machines they have compromised. | Collection, Exfiltration | Medium |

+| **Possible Log Tampering Activity Detected**<br>(K8S.NODE_SystemLogRemoval) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a possible removal of files that tracks user's activity during the course of its operation. Attackers often try to evade detection and leave no trace of malicious activities by deleting such log files. | DefenseEvasion | Medium |

+| **Possible password change using crypt-method detected**<br>(K8S.NODE_SuspectPasswordChange) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a password change using the crypt method. Attackers can make this change to continue access and gain persistence after compromise. | CredentialAccess | Medium |

+| **Potential port forwarding to external IP address**<br>(K8S.NODE_SuspectPortForwarding) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected an initiation of port forwarding to an external IP address. | Exfiltration, Command And Control | Medium |

+| **Potential reverse shell detected**<br>(K8S.NODE_ReverseShell) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a potential reverse shell. These are used to get a compromised machine to call back into a machine an attacker owns. | Exfiltration, Exploitation | Medium |

+| **Privileged container detected**<br>(K8S_PrivilegedContainer) | Kubernetes audit log analysis detected a new privileged container. A privileged container has access to the node's resources and breaks the isolation between containers. If compromised, an attacker can use the privileged container to gain access to the node. | Privilege Escalation | Low |

+| **Process associated with digital currency mining detected**<br>(K8S.NODE_CryptoCoinMinerArtifacts) ^{[1](#footnote1)} | Analysis of processes running within a container detected the execution of a process normally associated with digital currency mining. | Execution, Exploitation | Medium |

+| **Process seen accessing the SSH authorized keys file in an unusual way**<br>(K8S.NODE_SshKeyAccess) ^{[1](#footnote1)} | An SSH authorized_keys file was accessed in a method similar to known malware campaigns. This access could signify that an actor is attempting to gain persistent access to a machine. | Unknown | Low |

+| **Role binding to the cluster-admin role detected**<br>(K8S_ClusterAdminBinding) | Kubernetes audit log analysis detected a new binding to the cluster-admin role which gives administrator privileges. Unnecessary administrator privileges might cause privilege escalation in the cluster. | Persistence | Low |

+| **Security-related process termination detected**<br>(K8S.NODE_SuspectProcessTermination) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected an attempt to terminate processes related to security monitoring on the container. Attackers will often try to terminate such processes using predefined scripts post-compromise. | Persistence | Low |

+| **SSH server is running inside a container**<br>(K8S.NODE_ContainerSSH) ^{[1](#footnote1)} | Analysis of processes running within a container detected an SSH server running inside the container. | Execution | Medium |

+| **Suspicious file timestamp modification**<br>(K8S.NODE_TimestampTampering) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a suspicious timestamp modification. Attackers will often copy timestamps from existing legitimate files to new tools to avoid detection of these newly dropped files. | Persistence, DefenseEvasion | Low |

+| **Suspicious request to Kubernetes API**<br>(K8S.NODE_KubernetesAPI) ^{[1](#footnote1)} | Analysis of processes running within a container indicates that a suspicious request was made to the Kubernetes API. The request was sent from a container in the cluster. Although this behavior can be intentional, it might indicate that a compromised container is running in the cluster. | LateralMovement | Medium |

+| **Suspicious request to the Kubernetes Dashboard**<br>(K8S.NODE_KubernetesDashboard) ^{[1](#footnote1)} | Analysis of processes running within a container indicates that a suspicious request was made to the Kubernetes Dashboard. The request was sent from a container in the cluster. Although this behavior can be intentional, it might indicate that a compromised container is running in the cluster. | LateralMovement | Medium |

+| **Potential crypto coin miner started**<br>(K8S.NODE_CryptoCoinMinerExecution) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a process being started in a way normally associated with digital currency mining. | Execution | Medium |

+| **Suspicious password access**<br>(K8S.NODE_SuspectPasswordFileAccess) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected suspicious attempt to access encrypted user passwords. | Persistence | Informational |

+| **Suspicious use of DNS over HTTPS**<br>(K8S.NODE_SuspiciousDNSOverHttps) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected the use of a DNS call over HTTPS in an uncommon fashion. This technique is used by attackers to hide calls out to suspect or malicious sites. | DefenseEvasion, Exfiltration | Medium |

+| **A possible connection to malicious location has been detected.**<br>(K8S.NODE_ThreatIntelCommandLineSuspectDomain) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a connection to a location that has been reported to be malicious or unusual. This is an indicator that a compromise may have occurred. | InitialAccess | Medium |

+| **Possible malicious web shell detected.**<br>(K8S.NODE_Webshell) ^{[1](#footnote1)} | Analysis of processes running within a container detected a possible web shell. Attackers will often upload a web shell to a compute resource they have compromised to gain persistence or for further exploitation. | Persistence, Exploitation | Medium |

+| **Burst of multiple reconnaissance commands could indicate initial activity after compromise**<br>(K8S.NODE_ReconnaissanceArtifactsBurst) ^{[1](#footnote1)} | Analysis of host/device data detected execution of multiple reconnaissance commands related to gathering system or host details performed by attackers after initial compromise. | Discovery, Collection | Low |

+| **Suspicious Download Then Run Activity**<br>(K8S.NODE_DownloadAndRunCombo) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a file being downloaded then run in the same command. While this isn't always malicious, this is a very common technique attackers use to get malicious files onto victim machines. | Execution, CommandAndControl, Exploitation | Medium |

+| **Digital currency mining activity**<br>(K8S.NODE_CurrencyMining) ^{[1](#footnote1)} | Analysis of DNS transactions detected digital currency mining activity. Such activity, while possibly legitimate user behavior, is frequently performed by attackers following compromise of resources. Typical related attacker activity is likely to include the download and execution of common mining tools. | Exfiltration | Low |

+| **Access to kubelet kubeconfig file detected**<br>(K8S.NODE_KubeConfigAccess) ^{[1](#footnote1)} | Analysis of processes running on a Kubernetes cluster node detected access to kubeconfig file on the host. The kubeconfig file, normally used by the Kubelet process, contains credentials to the Kubernetes cluster API server. Access to this file is often associated with attackers attempting to access those credentials, or with security scanning tools which check if the file is accessible. | CredentialAccess | Medium |

+| **Access to cloud metadata service detected**<br>(K8S.NODE_ImdsCall) ^{[1](#footnote1)} | Analysis of processes running within a container detected access to the cloud metadata service for acquiring identity token. The container doesn't normally perform such operation. While this behavior might be legitimate, attackers might use this technique to access cloud resources after gaining initial access to a running container. | CredentialAccess | Medium |

+| **MITRE Caldera agent detected**<br>(K8S.NODE_MitreCalderaTools) ^{[1](#footnote1)} | Analysis of processes running within a container or directly on a Kubernetes node, has detected a suspicious process. This is often associated with the MITRE 54ndc47 agent which could be used maliciously to attack other machines. | Persistence, PrivilegeEscalation, DefenseEvasion, CredentialAccess, Discovery, LateralMovement, Execution, Collection, Exfiltration, Command And Control, Probing, Exploitation | Medium |

+| Required roles and permissions: | No other roles needed aside from what is required for the security explorer. <br><br> To access the dashboard with more than 1000 subscriptions, you must have tenant-level permissions, which include one of the following roles: **Global Reader**, **Global Administrator**, **Security Administrator**, or **Security Reader**. |

+In order to view the dashboard, you must enable Defender CSPM and also enable the sensitive data discovery extension button underneath. In addition, to receive the alerts for data sensitivity, you must also enable the Defender for Storage plan for storage related alerts or Defender for Databases for database related alerts.

+- To view the dashboard, you must have either one of the following scenarios:

+**Sensitive resources status over time** - displays how data security evolves over time with a graph that shows the number of sensitive resources affected by alerts, attack paths, and recommendations within a defined period (last 30, 14, or 7 days).

+## Remediate recommendations using Workflow Automation

+You can remediate recommendations generated by Defender for APIs using workflow automations.

+1. In an eligible recommendation, select one or more unhealthy resources.

+2. Select **Trigger logic app**.

+3. Confirm the **Selected subscription**.

+4. Select a relevant logic app from the list.

+5. Select **Trigger**.

+You can browse the [Microsoft Defender for Cloud GitHub](https://github.com/Azure/Microsoft-Defender-for-Cloud/tree/main/Workflow%20automation/Defender%20for%20API) repository for available workflow automation.

+|Pricing:|**Microsoft Defender for Storage** pricing applies to commercial clouds. Learn more about [pricing and availability per region.](https://azure.microsoft.com/pricing/details/defender-for-cloud/)<br>|

+|<br><br> Supported storage types:|[Blob Storage](https://azure.microsoft.com/products/storage/blobs/)ΓÇ»(Standard/Premium StorageV2, including Data Lake Gen2): Activity monitoring, Malware Scanning, Sensitive Data Discovery<br>Azure Files (over REST API and SMB): Activity monitoring |

+Ensure that your SSM Agent has the managed policy [AmazonSSMManagedInstanceCore](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSSMManagedInstanceCore.html), which enables core functionality for the AWS Systems Manager service.

+**You must have the SSM Agent for auto provisioning Arc agent on EC2 machines. If the SSM doesn't exist, or is removed from the EC2, the Arc provisioning wonΓÇÖt be able to procced.**

+> [!NOTE]

+> As part of the cloud formation template that is run during the onboarding process, an automation process is created and triggered every 30 days, over all the EC2s that existed during the initial run of the cloud formation. The goal of this scheduled scan is to ensure that all the relevant EC2s have an IAM profile with the required IAM policy that allows Defender for Cloud to access, manage, and provide the relevant security features (including the Arc agent provisioning). The scan does not apply to EC2s that were created after the run of the cloud formation.

+To ensure a smooth transition, we've taken measures to maintain the consistency of the Product/Service name, SKU, and Meter IDs. Impacted customers will receive an informational Azure Service Notification to communicate the changes.

+Organizations that retrieve cost data by calling our APIs, will need to update the values in their calls to accomodate the change. For example, in this filter function, the values will return no information:

+```json

+"filter": {

+ "dimensions": {

+ "name": "MeterCategory",

+ "operator": "In",

+ "values": [

+ "Advanced Threat Protection",

+ "Advanced Data Security",

+ "Azure Defender",

+ "Security Center"

+ ]

+ }

+ }

+```

+| SOC TSP **(deprecated)** | PCI DSS v3.2.1 | PCI DSS v3.2.1 |

+| SWIFT CSP CSCF v2022 |||

+If you're using [Enterprise IoT security](eiot-defender-for-endpoint.md) in Microsoft 365 Defender, alerts for Enterprise IoT devices detected by Microsoft Defender for Endpoint are available in Microsoft 365 Defender only.

+Alerts triggered by [Enterprise IoT sensors](eiot-sensor.md) are shown in the Azure portal only.

+Users working in hybrid environments might be managing OT alerts in [Defender for IoT](https://portal.azure.com/#view/Microsoft_Azure_IoT_Defender/IoTDefenderDashboard/~/Getting_started) on the Azure portal, the OT sensor, and an on-premises management console.

+In addition to the default behavior, you might want to help your SOC and OT management teams triage and remediate alerts faster. Sign into an OT sensor or an on-premises management console as an **Admin** user to use the following options:

+|**Mute** | - OT network sensors <br><br>- On-premises management console <br><br>*Unmuting* an alert is available only on the OT sensor. | Mute an alert when you want to close it and not see again for the same traffic, but without adding the alert allowed traffic. <br><br>For example, when the Operational engine triggers an alert indicating that the PLC Mode was changed on a device. The new mode might indicate that the PLC isn't secure, but after investigation, it's determined that the new mode is acceptable. <br><br>Muting an alert closes it, but doesn't add an item to the sensor event timeline. Detected traffic is included in data mining reports, but not when calculating data for other sensor reports. <br><br>Muting an alert is available for selected alerts only, mostly those triggered by the *Anomaly*, *Protocol Violation*, or *Operational* engines. |

+ Title: Microsoft Defender for IoT billing

+description: Learn how you're billed for the Microsoft Defender for IoT service.

+#CustomerIntent: As a Defender for IoT customer, I want to understand how I'm billed for Defender for IoT services so that I can best plan my deployment.

+# Defender for IoT billing

+**OT monitoring** is billed using site-based licenses, where each license applies to an individual site, based on the site size. A site is a physical location, such as a facility, campus, office building, hospital, rig, and so on. Each site can contain any number of network sensors, all which monitor devices detected in connected networks.

+**Enterprise IoT** monitoring supports 5 devices per Microsoft 365 E5 (ME5) or E5 Security license, or is available as standalone, per-device licenses for Microsoft Defender for Endpoint P2 customers.

+To evaluate Defender for IoT, start a free trial as follows:

+- **For OT networks**, use a trial license for 60 days. Deploy one or more Defender for IoT sensors on your network to monitor traffic, analyze data, generate alerts, learn about network risks and vulnerabilities, and more. An OT trial supports a **Large** site license for 60 days. For more information, see [Start a Microsoft Defender for IoT trial](getting-started.md).

+- **For Enterprise IoT networks**, use a trial, standalone license for 90 days as an add-on to Microsoft Defender for Endpoint. Trial licenses support 100 devices. For more information, see [Securing IoT devices in the enterprise](concept-enterprise.md) and [Enable Enterprise IoT security with Defender for Endpoint](eiot-defender-for-endpoint.md).

+We recommend that you have a sense of how many devices you want to monitor so that you know how many OT sites you need to license, or if you need any standalone licenses for enterprise IoT security.

+- **Enterprise IoT monitoring**: Five devices are supported for each ME5/E5 Security user license. If you have more devices to monitor, and are a Defender for Endpoint P2 customer, purchase extra, standalone licenses for each device you want to monitor.

+ Title: Securing IoT devices | Microsoft Defender for IoT

+description: Learn how integrating Microsoft Defender for Endpoint and Microsoft Defender for IoT's security content enhances your IoT network security.

+#CustomerIntent: As a Defender for IoT customer, I want to understand how I can secure my enterprise IoT devices with Microsoft Defender for IoT so that I can protect my organization from IoT threats.

+The number of IoT devices continues to grow exponentially across enterprise networks, such as the printers, Voice over Internet Protocol (VoIP) devices, smart TVs, and conferencing systems scattered around many office buildings.

+[Microsoft Defender for IoT](./index.yml) seamlessly integrates with [Microsoft 365 Defender](/microsoft-365/security/defender) and [Microsoft Defender for Endpoint](/microsoft-365/security/defender-endpoint/) to provide both IoT device discovery and security value for IoT devices, including purpose-built alerts, recommendations, and vulnerability data.

+## Enterprise IoT security in Microsoft 365 Defender

+Enterprise IoT security in Microsoft 365 Defender provides IoT-specific security value, including alerts, risk and exposure levels, vulnerabilities, and recommendations in Microsoft 365 Defender.

+- If you're a Microsoft 365 E5 (ME5)/ E5 Security and Defender for Endpoint P2 customer, [toggle on support](eiot-defender-for-endpoint.md) for **Enterprise IoT Security** in the Microsoft 365 Defender portal.

+- If you don't have ME5/E5 Security licenses, but you're a Microsoft Defender for Endpoint customer, start with a [free trial](billing.md#free-trial) or purchase standalone, per-device licenses to gain the same IoT-specific security value.

+The following image shows the architecture and extra features added with **Enterprise IoT security** in Microsoft 365 Defender:

+- [Get started with enterprise IoT monitoring in Microsoft 365 Defender](eiot-defender-for-endpoint.md)

+- [Defender for IoT subscription billing](billing.md)

+- [Device discovery overview](/microsoft-365/security/defender-endpoint/device-discovery)

+## Frequently asked questions

+This section provides a list of frequently asked questions about securing Enterprise IoT networks with Microsoft Defender for IoT.

+### What is the difference between OT and Enterprise IoT?

+- **Operational Technology (OT)**: OT network sensors use agentless, patented technology to discover, learn, and continuously monitor network devices for a deep visibility into Operational Technology (OT) / Industrial Control System (ICS) risks. Sensors carry out data collection, analysis, and alerting on-site, making them ideal for locations with low bandwidth or high latency.

+- **Enterprise IoT**: Enterprise IoT provides visibility and security for IoT devices in the corporate environment.

+ Enterprise IoT network protection extends agentless features beyond operational environments, providing coverage for all IoT devices in your environment. For example, an enterprise IoT environment might include printers, cameras, and purpose-built, proprietary, devices.

+### What extra security value can Enterprise IoT provide Microsoft Defender for Endpoint customers?

+Enterprise IoT is designed to help customers secure unmanaged devices throughout the organization and extend IT security to also cover IoT devices.

+While Microsoft 365 P2 customers already have visibility for discovered IoT devices in the **Device inventory** page in Defender for Endpoint, they can use enterprise IoT security to gain security value with extra alerts, recommendations and vulnerabilities for their discovered IoT devices.

+### How can I start using Enterprise IoT?

+Microsoft E5 (ME5) and E5 Security customers already have devices supported for enterprise IoT security. If you only have a Defender for Endpoint P2 license, you can purchase standalone, per-device licenses for enterprise IoT monitoring, or use a trial.

+For more information, see:

+- [Get started with enterprise IoT monitoring in Microsoft 365 Defender](eiot-defender-for-endpoint.md)

+- [Manage enterprise IoT monitoring support with Microsoft Defender for IoT](manage-subscriptions-enterprise.md)

+### What permissions do I need to use Enterprise IoT security with Defender for IoT?

+For information on required permissions, see [Prerequisites](eiot-defender-for-endpoint.md#prerequisites).

+### Which devices are billable?

+For more information, see [Devices monitored by Defender for IoT](architecture.md#devices-monitored-by-defender-for-iot).

+### How should I estimate the number of devices I want to monitor?

+For more information, see [Calculate monitored devices for Enterprise IoT monitoring](manage-subscriptions-enterprise.md#calculate-monitored-devices-for-enterprise-iot-monitoring).

+### How can I cancel Enterprise IoT?

+For more information, see [Turn off enterprise IoT security](manage-subscriptions-enterprise.md#turn-off-enterprise-iot-security).

+### What happens when the trial ends?

+If you haven't added a standalone license by the time your trial ends, your trial is automatically canceled, and you lose access to Enterprise IoT security features.

+For more information, see [Defender for IoT subscription billing](billing.md).

+### How can I resolve billing issues associated with my Defender for IoT plan?

+For any billing or technical issues, open a support ticket for Microsoft 365 Defender.

+Defender for IoT device inventory is available in the following locations:

+|**Azure portal** | OT devices detected from all cloud-connected OT sensors. | - If you also use [Microsoft Sentinel](iot-solution.md), incidents in Microsoft Sentinel are linked to related devices in Defender for IoT. <br><br>- Use Defender for IoT [workbooks](workbooks.md) for visibility into all cloud-connected device inventory, including related alerts and vulnerabilities. <br><br>- If you have a [legacy Enterprise IoT plan](whats-new.md#enterprise-iot-protection-now-included-in-microsoft-365-e5-and-e5-security-licenses) on your Azure subscription, the Azure portal also includes devices detected by Microsoft Defender for Endpoint agents. If you have an [Enterprise IoT sensor](eiot-sensor.md), the Azure portal also includes devices detected by the Enterprise IoT sensor. |

+| **Microsoft 365 Defender** | Enterprise IoT devices detected by Microsoft Defender for Endpoint agents | Correlate devices across Microsoft 365 Defender in purpose-built alerts, vulnerabilities, and recommendations. |

+|

+- [Defender for Endpoint device discovery](/microsoft-365/security/defender-endpoint/device-discovery)

+For example, any devices with the same IP and MAC address detected in the same zone are consolidated and identified as a single device in the device inventory. If you have separate devices from recurring IP addresses that are detected by multiple sensors, you want each of these devices to be identified separately. In such cases, [onboard your OT sensors](onboard-sensors.md) to different zones so that each device is identified as a separate and unique device, even if they have the same IP address. Devices that have the same MAC addresses, but different IP addresses aren't merged, and continue to be listed as unique devices.

+|**Authorization** * |Editable. Determines whether or not the device is marked as *authorized*. This value might need to change as the device security changes. |

+ Title: Get started with enterprise IoT monitoring in Microsoft 365 Defender | Microsoft Defender for IoT

+description: Learn how to get added value for enterprise IoT devices in Microsoft 365 Defender.

+#CustomerIntent: As a Microsoft 365 administrator, I want to understand how to turn on support for enterprise IoT monitoring in Microsoft 365 Defender and where I can find the added security value so that I can keep my EIoT devices safe.

+# Get started with enterprise IoT monitoring in Microsoft 365 Defender

+This article describes how [Microsoft Defender for Endpoint](/microsoft-365/security/defender-endpoint/) customers can monitor enterprise IoT devices in their environment, using added security value in Microsoft 365 Defender.

+While IoT device inventory is already available for Defender for Endpoint P2 customers, turning on enterprise IoT security adds alerts, recommendations, and vulnerability data, purpose-built for IoT devices in your enterprise network.

+IoT devices include printers, cameras, VOIP phones, smart TVs, and more. Turning on enterprise IoT security means, for example, that you can use a recommendation in Microsoft 365 Defender to open a single IT ticket for patching vulnerable applications across both servers and printers.

+- Access to the Microsoft 365 Defender portal as a [Security administrator](../../active-directory/roles/permissions-reference.md#security-administrator)

+- One of the following licenses:

+ - A Microsoft 365 E5 (ME5) or E5 Security license

+ - Microsoft Defender for Endpoint P2, with an extra, standalone **Microsoft Defender for IoT - EIoT Device License - add-on** license, available for purchase or trial from the Microsoft 365 admin center.

+ > [!TIP]

+ > If you have a standalone license, you don't need to toggle on **Enterprise IoT Security** and can skip directly to [View added security value in Microsoft 365 Defender](#view-added-security-value-in-microsoft-365-defender).

+ >

+ For more information, see [Enterprise IoT security in Microsoft 365 Defender](concept-enterprise.md#enterprise-iot-security-in-microsoft-365-defender).

+## Turn on enterprise IoT monitoring

+This procedure describes how to turn on enterprise IoT monitoring in Microsoft 365 Defender, and is relevant only for ME5/E5 Security customers.

+Skip this procedure if you have one of the following types of licensing plans:

+- Customers with legacy Enterprise IoT pricing plan and an ME5/E5 Security license.

+- Customers with standalone, per-device licenses added on to Microsoft Defender for Endpoint P2. In such cases, the Enterprise IoT security setting is turned on as read-only.

+**To turn on enterprise IoT monitoring**:

+1. In [Microsoft 365 Defender](https://security.microsoft.com/), select **Settings** \> **Device discovery** \> **Enterprise IoT**.

+1. Toggle the Enterprise IoT security option to **On**. For example:

+ :::image type="content" source="media/enterprise-iot/eiot-toggle-on.png" alt-text="Screenshot of Enterprise IoT toggled on in Microsoft 365 Defender.":::

+This procedure describes how to view related alerts, recommendations, and vulnerabilities for a specific device in Microsoft 365 Defender, when the **Enterprise IoT security** option is turned on.

+1. In [Microsoft 365 Defender](https://security.microsoft.com/), select **Assets** \> **Devices** to open the **Device inventory** page.

+1. On the device details page, explore the following tabs to view data added by the enterprise IoT security for your device:

+Microsoft 365 Defender customers with an Enterprise IoT network sensor can see all discovered devices in the **Device inventory** in either Microsoft 365 Defender or Defender for IoT. You'll also get extra security value from more alerts, vulnerabilities, and recommendations in Microsoft 365 Defender for the newly discovered devices.

+If you're a Defender for IoT customer working solely in the Azure portal, an Enterprise IoT network sensor provides extra device visibility to Enterprise IoT devices, such as Voice over Internet Protocol (VoIP) devices, printers, and cameras, which might not be covered by your OT network sensors.

+- To view Defender for IoT data in Microsoft 365 Defender, including devices, alerts, recommendations, and vulnerabilities, you must have **Enterprise IoT security** turned on in [Microsoft 365 Defender](eiot-defender-for-endpoint.md).

+ If you only want to view data in the Azure portal, you don't need Microsoft 365 Defender. You can also turn on **Enterprise IoT security** in Microsoft 365 Defender after registering your network sensor to bring [extra device visibility and security value](concept-enterprise.md#enterprise-iot-security-in-microsoft-365-defender) to your organization.

+- Identify the devices and subnets you want to monitor so that you understand where to place an Enterprise IoT sensor in your network. You might want to deploy multiple Enterprise IoT sensors.

+ Identify the interfaces that you want to monitor, which are usually the interfaces with no IP address listed. Interfaces with incoming traffic show an increasing number of RX packets.

+This section describes how to register an Enterprise IoT sensor in Defender for IoT. When you're done registering your sensor, you continue on with installing the Enterprise IoT monitoring software on your sensor machine.

+1. Copy the command to a safe location, where you're able to copy it to your physical appliance or VM in order to [install sensor software](#install-enterprise-iot-sensor-software).

+ :::image type="content" source="media/tutorial-get-started-eiot/proxy.png" alt-text="Screenshot of the Set up a proxy server screen.":::

+If you're a Defender for Endpoint customer with a [legacy Enterprise IoT plan](whats-new.md#enterprise-iot-protection-now-included-in-microsoft-365-e5-and-e5-security-licenses), you're able to view all detected devices in the **Device inventory** pages, in both Defender for IoT and Microsoft 365 Defender. Detected devices include both devices detected by Defender for Endpoint and devices detected by the Enterprise IoT sensor.

+If you want to cancel enterprise IoT security with Microsoft 365 Defender, do so from the Microsoft 365 Defender portal. For more information, see [Turn off enterprise IoT security](manage-subscriptions-enterprise.md#turn-off-enterprise-iot-security).

+- If you have **Enterprise IoT security** [turned on in Microsoft 365 Defender](eiot-defender-for-endpoint.md), alerts for Enterprise IoT devices detected by Microsoft Defender for Endpoint are available in Defender for Endpoint only.

+- **To have alerts in Defender for IoT**, you must have an [OT](onboard-sensors.md) onboarded, and network data streaming into Defender for IoT.

+ | **Last activity** | The last time the alert was changed, including manual updates for severity or status, or automated changes for device updates or device/alert deduplication |

+For example, while the total number of alerts appears above the grid, you might want more specific information about alert count breakdown, such as the number of alerts with a specific severity, protocol, or site.

+You might want to export a selection of alerts to a CSV file for offline sharing and reporting.

+If you're looking to manage support for enterprise IoT security, see [Manage enterprise IoT monitoring support with Microsoft Defender for IoT](manage-subscriptions-enterprise.md).

+This article is relevant for commercial Defender for IoT customers. If you're a government customer, contact your Microsoft sales representative for more information.

+- A [Security admin](../../role-based-access-control/built-in-roles.md#security-admin), [Contributor](../../role-based-access-control/built-in-roles.md#contributor), or [Owner](../../role-based-access-control/built-in-roles.md#owner) user role for the Azure subscription that you're using for the integration

+1. In the **Plan settings** pane, select the Azure subscription where you want to add a plan. You can only add a single subscription, and you need a [Security admin](../../role-based-access-control/built-in-roles.md#security-admin), [Contributor](../../role-based-access-control/built-in-roles.md#contributor), or [Owner](../../role-based-access-control/built-in-roles.md#owner) role for the selected subscription.

+ When you're finished, select **Save**. If you've selected to download the on-premises management console activation file, the file is downloaded and you're prompted to save it locally. You use it later, when [activating your on-premises management console](ot-deploy/activate-deploy-management.md#activate-the-on-premises-management-console).

+## Cancel a Defender for IoT plan for OT networks

+You might need to cancel a Defender for IoT plan from your Azure subscription, for example, if you need to work with a different subscription, or if you no longer need the service.

+**To cancel an OT network plan**:

+If the number of actual devices detected by Defender for IoT exceeds the number of committed devices currently listed on your subscription, you might see a warning message in the Azure portal and on your OT sensor that you have exceeded the maximum number of devices for your subscription.

+This warning indicates you need to update the number of committed devices on the relevant subscription to the actual number of devices being monitored. Select the link in the warning message to take you to the **Plans and pricing** page, with the **Edit plan** pane already open.

+If you have multiple legacy subscriptions and are migrating to a Microsoft 365 plan, you'll first need to consolidate your sensors to a single subscription. To do this, you need to register the sensors under the new subscription and remove them from the original subscription.

+ - For sensors monitoring overlapping network segments, create the activation file under the same zone. Identical devices that are detected in more than one sensor in a zone are merged into one device.

+1. If relevant, cancel the Defender for IoT plan from the previous subscription. For more information, see [Cancel a Defender for IoT plan for OT networks](#cancel-a-defender-for-iot-plan-for-ot-networks).

+Changes to your plan will take effect one hour after confirming the change. This change appears on your next monthly statement, and you're charged based on the length of time each plan was in effect.

+ Title: Manage EIoT monitoring support | Microsoft Defender for IoT

+description: Learn how to manage your EIoT monitoring support with Microsoft Defender for IoT.

+#CustomerIntent: As a Defender for IoT customer, I want to understand how to manage my EIoT monitoring support with Microsoft Defender for IoT so that I can best plan my deployment.

+# Manage enterprise IoT monitoring support with Microsoft Defender for IoT

+Enterprise IoT security monitoring with Defender for IoT is supported by a Microsoft 365 E5 (ME5) or E5 Security license, or extra standalone, per-device licenses purchased as add-ons to Microsoft Defender for Endpoint.

+This article describes how to:

+- Calculate the devices detected in your environment so that you can understand if you need extra, standalone licenses.

+- Cancel support for enterprise IoT monitoring with Microsoft Defender for IoT

+- One of the following sets of licenses:

+ - A Microsoft 365 E5 (ME5) or E5 Security license and a Microsoft Defender for Endpoint P2 license

+ - A Microsoft Defender for Endpoint P2 license alone

+ For more information, see [Enterprise IoT security in Microsoft 365 Defender](concept-enterprise.md#enterprise-iot-security-in-microsoft-365-defender).

+- Access to the Microsoft 365 Defender portal as a [Global administrator](../../active-directory/roles/permissions-reference.md#global-administrator)

+## Obtain a standalone, Enterprise IoT trial license

+This procedure describes how to start using a trial, standalone license for enterprise IoT monitoring, for customers who have a Microsoft Defender for Endpoint P2 license only.

+Customers with ME5/E5 Security plans have support for enterprise IoT monitoring available on by default, and don't need to start a trial. For more information, see [Get started with enterprise IoT monitoring in Microsoft 365 Defender](eiot-defender-for-endpoint.md).

+Start your enterprise IoT trial using the [Microsoft Defender for IoT - EIoT Device License - add-on wizard](https://signup.microsoft.com/get-started/signup?products=b2f91841-252f-4765-94c3-75802d7c0ddb&ali=1&bac=1) or via the Microsoft 365 admin center.

+**To start an Enterprise IoT trial**:

+1. Go to the [Microsoft 365 admin center](https://portal.office.com/AdminPortal/Home#/catalog) > **Marketplace**.

+1. Search for the **Microsoft Defender for IoT - EIoT Device License - add-on** and filter the results by **Other services**. For example:

+ :::image type="content" source="media/enterprise-iot/eiot-standalone.png" alt-text="Screenshot of the Marketplace search results for the EIoT Device License.":::

+ > [!IMPORTANT]

+ > The prices shown in this image are for example purposes only and are not intended to reflect actual prices.

+ >

+1. Under **Microsoft Defender for IoT - EIoT Device License - add-on**, select **Details**.

+1. On the **Microsoft Defender for IoT - EIoT Device License - add-on** page, select **Start free trial**. On the **Check out** page, select **Try now**.

+> [!TIP]

+> Make sure to [assign your licenses to specific users]/microsoft-365/admin/manage/assign-licenses-to-users to start using them.

+>

+For more information, see [Free trial](billing.md#free-trial).

+## Calculate monitored devices for Enterprise IoT monitoring

+Use the following procedure to calculate how many devices you need to monitor if:

+- You're an ME5/E5 Security customer and thinks you need to monitor more devices than the devices allocated per ME5/E5 Security license

+- You're a Defender for Endpoint P2 customer who's purchasing standalone enterprise IoT licenses

+1. In [Microsoft 365 Defender](https://security.microsoft.com/), select **Assets** \> **Devices** to open the **Device inventory** page.

+1. Round up your total to a multiple of 100 and compare it against the number of licenses you have.

+- You have 320 ME5 licenses, which cover **1600** devices

+You need **79** standalone devices to cover the gap.

+## Purchase standalone licenses

+Purchase standalone, per-device licenses if you're an ME5/E5 Security customer who needs more than the five devices allocated per license, or if you're a Defender for Endpoint customer who wants to add enterprise IoT security to your organization.

+**To purchase standalone licenses**:

+1. Go to the [Microsoft 365 admin center](https://portal.office.com/AdminPortal/Home#/catalog) **Billing > Purchase services**. If you don't have this option, select **Marketplace** instead.

+1. Search for the **Microsoft Defender for IoT - EIoT Device License - add-on** and filter the results by **Other services**. For example:

+ :::image type="content" source="media/enterprise-iot/eiot-standalone.png" alt-text="Screenshot of the Marketplace search results for the EIoT Device License.":::

+ > [!IMPORTANT]

+ > The prices shown in this image are for example purposes only and are not intended to reflect actual prices.

+ >

+1. On the **Microsoft Defender for IoT - EIoT Device License - add-on** page, enter your selected license quantity, select a billing frequency, and then select **Buy**.

+For more information, see the [Microsoft 365 admin center help](/microsoft-365/admin/).

+## Turn off enterprise IoT security

+This procedure describes how to turn off enterprise IoT monitoring in Microsoft 365 Defender, and is supported only for customers who don't have any standalone, per-device licenses added on to Microsoft 365 Defender.

+Turn off the **Enterprise IoT security** option if you're no longer using the service.

+**To turn off enterprise IoT monitoring**:

+1. In [Microsoft 365 Defender](https://security.microsoft.com/), select **Settings** \> **Device discovery** \> **Enterprise IoT**.

+1. Toggle the option to **Off**.

+You stop getting security value in Microsoft 365 Defender, including purpose-built alerts, vulnerabilities, and recommendations.

+### Cancel a legacy Enterprise IoT plan

+If you have a legacy Enterprise IoT plan, are *not* an ME5/E5 Security customer, and no longer to use the service, cancel your plan as follows:

+1. In [Microsoft 365 Defender](https://security.microsoft.com/) portal, select **Settings** \> **Device discovery** \> **Enterprise IoT**.

+1. Select **Cancel plan**. This page is available only for legacy Enterprise IoT plan customers.

+The cancellation takes effect one hour after confirming the change. This change appears on your next monthly statement, and you're charged based on the length of time the plan was in effect.

+- [Securing IoT devices in the enterprise](concept-enterprise.md)

+The Internet of Things (IoT) supports billions of connected devices that use both operational technology (OT) and IoT networks. IoT/OT devices and networks are often built using specialized protocols, and might prioritize operational challenges over security.

+Microsoft Defender for IoT is a unified security solution built specifically to identify IoT and OT devices, vulnerabilities, and threats. Use Defender for IoT to secure your entire IoT/OT environment, including existing devices that might not have built-in security agents.

+If your IoT and OT devices don't have embedded security agents, they might remain unpatched, misconfigured, and invisible to IT and security teams. Unmonitored devices can be soft targets for threat actors looking to pivot deeper into corporate networks.

+ - Detect advanced threats that you might have missed by static indicators of compromise (IOCs), such as zero-day malware, fileless malware, and living-off-the-land tactics.

+ You might have hybrid network requirements where you can deliver some data to the cloud and other data must remain on-premises.

+IoT and industrial control system (ICS) devices can be secured using both embedded protocols and proprietary, custom, or nonstandard protocols. If you have devices that run on protocols that aren't supported by Defender for IoT out-of-the-box, use the Horizon Open Development Environment (ODE) SDK to develop dissector plug-ins to decode network traffic for your protocols.

+Extend Defender for IoT's agentless security features beyond OT environments to enterprise IoT devices by using enterprise IoT security with Microsoft Defender for Endpoint, and view related alerts, vulnerabilities, and recommendations for IoT devices in Microsoft 365 Defender.

+Microsoft Defender for IoT uses [Azure Role-Based Access Control (RBAC)](../../role-based-access-control/index.yml) to provide access to Defender for IoT monitoring services and data on the Azure portal.

+| **Onboard [OT](onboard-sensors.md) or [Enterprise IoT sensors](eiot-sensor.md)** <br>Apply per subscription only | - | Γ£ö | Γ£ö | Γ£ö |

+| **[View values on the Plans and pricing page](how-to-manage-subscriptions.md)** <br>Apply per subscription only| Γ£ö | Γ£ö | Γ£ö | Γ£ö |

+| **[Modify values on the Plans and pricing page](how-to-manage-subscriptions.md)** <br>Apply per subscription only| - | Γ£ö | Γ£ö | Γ£ö |

+| **[View values on the Sites and sensors page](how-to-manage-sensors-on-the-cloud.md)** <br>Apply per subscription only | Γ£ö | Γ£ö | Γ£ö | Γ£ö|

+| **[Modify values on the Sites and sensors page](how-to-manage-sensors-on-the-cloud.md#sensor-management-options-from-the-azure-portal)** , including remote OT sensor updates<br>Apply per subscription only | - | Γ£ö | Γ£ö | Γ£ö|

+| **Enterprise IoT networks** | [Enterprise IoT protection now included in Microsoft 365 E5 and E5 Security licenses](#enterprise-iot-protection-now-included-in-microsoft-365-e5-and-e5-security-licenses) |

+### Enterprise IoT protection now included in Microsoft 365 E5 and E5 Security licenses

+Enterprise IoT (EIoT) security with Defender for IoT discovers unmanaged IoT devices and also provides added security value, including continuous monitoring, vulnerability assessments and tailored recommendations specifically designed for Enterprise IoT devices. Seamlessly integrated with Microsoft 365 Defender, Microsoft Defender Vulnerability Management, and Microsoft Defender for Endpoint on the Microsoft 365 Defender portal, it ensures a holistic approach to safeguarding an organization's network.

+De