+1. [Microsoft Authenticator notifications](concept-authentication-authenticator-app.md)

+| Software |<li>Windows Server 2022¹<li>Windows Server 2019¹</li><li>Windows Server 2016</li><li>Windows Server 2012 R2</li><li>Windows Server 2012</li><li>Windows Server 2008/R2 (with [ESU](/lifecycle/faq/extended-security-updates) only)</li><li>Windows 10</li><li>Windows 8.1, all editions</li><li>Windows 8, all editions</li><li>Windows 7, all editions (with [ESU](/lifecycle/faq/extended-security-updates) only)</li><li>Microsoft .NET 4.0 Framework</li><li>IIS 7.0 or greater if installing the user portal or web service SDK</li> |

+¹If Azure MFA Server fails to activate on an Azure VM that runs Windows Server 2019 or later, try using an earlier version of Windows Server.

+Each piece is separated by a period (`.`) and separately Base 64 encoded.

+| `acrs` | JSON array of strings | Indicates the Auth Context IDs of the operations that the bearer is eligible to perform. Auth Context IDs can be used to trigger a demand for step-up authentication from within your application and services. Often used along with the `xms_cc` claim. |

+| `xms_cc` | JSON array of strings | Indicates whether the client application that acquired the token is capable of handling claims challenges. It's often used along with claim `acrs`. This claim is commonly used in Conditional Access and Continuous Access Evaluation scenarios. The resource server or service application that the token is issued for controls the presence of this claim in a token. A value of `cp1` in the access token is the authoritative way to identify that a client application is capable of handling a claims challenge. For more information, see [Claims challenges, claims requests and client capabilities](claims-challenge.md?tabs=dotnet). |

+If none of the previously described scenarios apply, there's no need to validate the token. Public clients like native, desktop or single-page applications don't benefit from validating ID tokens because the application communicates directly with the IDP where SSL protection ensures the ID tokens are valid. They shouldn't validate the access tokens, as they are for the web API to validate, not the client.

+The following examples suppose that your application is validating a v2.0 access token (and therefore reference the v2.0 versions of the OIDC metadata documents and keys). Just remove the "/v2.0" in the URL if you validate v1.0 tokens.

+[OpenID Connect Core](https://openid.net/specs/openid-connect-core-1_0.html#IDTokenValidation) says "The Issuer Identifier \[...\] MUST exactly match the value of the `iss` (issuer) Claim." For applications that use a tenant-specific metadata endpoint, such as `https://login.microsoftonline.com/{example-tenant-id}/v2.0/.well-known/openid-configuration` or `https://login.microsoftonline.com/contoso.onmicrosoft.com/v2.0/.well-known/openid-configuration`.

+For these applications, Azure AD exposes tenant-independent versions of the OIDC document at `https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration` and `https://login.microsoftonline.com/organizations/v2.0/.well-known/openid-configuration` respectively. These endpoints return an issuer value, which is a template parametrized by the `tenantid`: `https://login.microsoftonline.com/{tenantid}/v2.0`. Applications may use these tenant-independent endpoints to validate tokens from every tenant with the following stipulations:

+ - Validate the signing key issuer

+ - Validate that the `tid` claim is a GUID and the `iss` claim is of the form `https://login.microsoftonline.com/{tid}/v2.0` where `{tid}` is the exact `tid` claim. This validation ties the tenant back to the issuer and back to the scope of the signing key creating a chain of trust.

+The `{example-tenant-id}` value can be replaced by a GUID, a domain name, or **common**, **organizations** and **consumers**

+tenant-independent "common" key endpoint `https://login.microsoftonline.com/common/discovery/v2.0/keys` returns a document like:

+| `acrs` | Auth Context IDs | JWT | Azure AD | Indicates the Auth Context IDs of the operations that the bearer is eligible to perform. Auth Context IDs can be used to trigger a demand for step-up authentication from within your application and services. Often used along with the `xms_cc` claim. |

+| `email` | The reported email address for this user | JWT, SAML | MSA, Azure AD | This value is included by default if the user is a guest in the tenant. For managed users (the users inside the tenant), it must be requested through this optional claim or, on v2.0 only, with the OpenID scope. This value isn't guaranteed to be correct, and is mutable over time - never use it for authorization or to save data for a user. For more information, see [Validate the user has permission to access this data](access-tokens.md). If you're using the email claim for authorization, we recommend [performing a migration to move to a more secure claim](./migrate-off-email-claim-authorization.md). If you require an addressable email address in your app, request this data from the user directly, using this claim as a suggestion or prefill in your UX. |

+| `fwd` | IP address | JWT | | Adds the original address of the requesting client (when inside a VNET). |

+| `login_hint` | Login hint | JWT | MSA, Azure AD | An opaque, reliable login hint claim that's base 64 encoded. Don't modify this value. This claim is the best value to use for the `login_hint` OAuth parameter in all flows to get SSO. It can be passed between applications to help them silently SSO as well - application A can sign in a user, read the `login_hint` claim, and then send the claim and the current tenant context to application B in the query string or fragment when the user selects on a link that takes them to application B. To avoid race conditions and reliability issues, the `login_hint` claim *doesn't* include the current tenant for the user, and defaults to the user's home tenant when used. In a guest scenario where the user is from another tenant, a tenant identifier must be provided in the sign-in request. and pass the same to apps you partner with. This claim is intended for use with your SDK's existing `login_hint` functionality, however that it exposed. |

+| `xms_cc` | Client Capabilities | JWT | Azure AD | Indicates whether the client application that acquired the token is capable of handling claims challenges. It's often used along with claim `acrs`. This claim is commonly used in Conditional Access and Continuous Access Evaluation scenarios. The resource server or service application that the token is issued for controls the presence of this claim in a token. A value of `cp1` in the access token is the authoritative way to identify that a client application is capable of handling a claims challenge. For more information, see [Claims challenges, claims requests and client capabilities](claims-challenge.md?tabs=dotnet). |

+| `xms_edov` | Boolean value indicating whether the user's email domain owner has been verified. | JWT | | An email is considered to be domain verified if it belongs to the tenant where the user account resides and the tenant admin has done verification of the domain. Also, the email must be from a Microsoft account (MSA), a Google account, or used for authentication using the one-time passcode (OTP) flow. It should also be noted the Facebook and SAML/WS-Fed accounts **do not** have verified domains. |

+Get-MgUser -Filter "UserType eq 'Member'"

+The default specific parameters for the **Real-time employee change** template are as follows:

+Get-AzureADMSRoleAssignment -Filter "principalId eq '<object id of group>'"

+> | [Viva Pulse Administrator](#viva-pulse-administrator) | Can manage all settings for Microsoft Viva Pulse app. | 87761b17-1ed2-4af3-9acd-92a150038160 |

+## Viva Pulse Administrator

+Assign the Viva Pulse Administrator role to users who need to do the following tasks:

+- Read and configure all settings of Viva Pulse

+- Read basic properties on all resources in the Microsoft 365 admin center

+- Read and configure Azure Service Health

+- Create and manage Azure support tickets

+- Read messages in Message Center in the Microsoft 365 admin center, excluding security messages

+- Read usage reports in the Microsoft 365 admin center

+For more information, see [Assign a Viva Pulse admin in the Microsoft 365 admin center](/viva/pulse/setup-admin-access/assign-a-viva-pulse-admin-in-m365-admin-center).

+> [!div class="mx-tableFixed"]

+> | Actions | Description |

+> | | |

+> | microsoft.office365.messageCenter/messages/read | Read messages in Message Center in the Microsoft 365 admin center, excluding security messages |

+> | microsoft.office365.serviceHealth/allEntities/allTasks | Read and configure Service Health in the Microsoft 365 admin center |

+> | microsoft.office365.supportTickets/allEntities/allTasks | Create and manage Microsoft 365 service requests |

+> | microsoft.office365.usageReports/allEntities/allProperties/read | Read Office 365 usage reports |

+> | microsoft.office365.webPortal/allEntities/standard/read | Read basic properties on all resources in the Microsoft 365 admin center |

+> | microsoft.viva.pulse/allEntities/allProperties/allTasks | Manage all aspects of Microsoft Viva Pulse |

+# Document Intelligence container tags

+<!-- markdownlint-disable MD051 -->

+## Microsoft container registry (MCR)

+Document Intelligence container images can be found within the [**Microsoft Artifact Registry** (also know as Microsoft Container Registry(MCR))](https://mcr.microsoft.com/catalog?search=document%20intelligence), the primary registry for all Microsoft published container images.

+The following containers support DocumentIntelligence v3.0 models and features:

+| Container name |image |

+|||

+|[**Document Intelligence Studio**](https://mcr.microsoft.com/product/azure-cognitive-services/form-recognizer/studio/tags)| `mcr.microsoft.com/azure-cognitive-services/form-recognizer/studio:latest`|

+| [**Business Card 3.0**](https://mcr.microsoft.com/product/azure-cognitive-services/form-recognizer/businesscard-3.0/tags) | `mcr.microsoft.com/azure-cognitive-services/form-recognizer/businesscard-3.0:latest` |

+| [**Custom Template 3.0**](https://mcr.microsoft.com/product/azure-cognitive-services/form-recognizer/custom-template-3.0/tags) | `mcr.microsoft.com/azure-cognitive-services/form-recognizer/custom-template-3.0:latest` |

+| [**Document 3.0**](https://mcr.microsoft.com/product/azure-cognitive-services/form-recognizer/document-3.0/tags)| `mcr.microsoft.com/azure-cognitive-services/form-recognizer/document-3.0:latest`|

+| [**ID Document 3.0**](https://mcr.microsoft.com/product/azure-cognitive-services/form-recognizer/id-document-3.0/tags) | `mcr.microsoft.com/azure-cognitive-services/form-recognizer/id-document-3.0:latest` |

+| [**Invoice 3.0**](https://mcr.microsoft.com/product/azure-cognitive-services/form-recognizer/invoice-3.0/tags) |`mcr.microsoft.com/azure-cognitive-services/form-recognizer/invoice-3.0:latest`|

+| [**Layout 3.0**](https://mcr.microsoft.com/product/azure-cognitive-services/form-recognizer/layout/tags) |`mcr.microsoft.com/azure-cognitive-services/form-recognizer/layout-3.0:latest`|

+| [**Read 3.0**](https://mcr.microsoft.com/product/azure-cognitive-services/form-recognizer/read-3.0/tags) |`mcr.microsoft.com/azure-cognitive-services/form-recognizer/read-3.0:latest`|

+| [**Receipt 3.0**](https://mcr.microsoft.com/product/azure-cognitive-services/form-recognizer/receipt-3.0/tags) |`mcr.microsoft.com/azure-cognitive-services/form-recognizer/receipt-3.0:latest`|

+> [!IMPORTANT]

+>

+> Document Intelligence v3.0 containers are now generally available. If you are getting started with containers, consider using the v3 containers.

+The following containers:

+> The Document Intelligence v1.0 container is retired.

+² Version `0314` of gpt-4 and gpt-4-32k will be retired no earlier than July 5, 2024. See [model updates](#model-updates) for model upgrade behavior.<br>

+³ We are rolling out availability of new regions to customers gradually to ensure a smooth experience. In East US and France Central, customers with existing deployments of GPT-4 can create additional deployments of GPT-4 version 0613. For customers new to GPT-4 on Azure OpenAI, please use one of the other available regions.

+### Deploying the model

+After you connect Azure OpenAI to your data, you can deploy it using the **Deploy to** button in Azure OpenAI studio.

+#### Using Power Virtual Agents

+You can deploy your model to [Power Virtual Agents](/power-virtual-agents/fundamentals-what-is-power-virtual-agents) directly from Azure OpenAI studio, enabling you to bring conversational experiences to various Microsoft Teams, Websites, Power Platform solutions, Dynamics 365, and other [Azure Bot Service channels](/power-virtual-agents/publication-connect-bot-to-azure-bot-service-channels). Power Virtual Agents acts as a conversational and generative AI platform, making the process of creating, publishing and deploying a bot to any number of channels simple and accessible.

+While Power Virtual Agents has features that leverage Azure OpenAI such as [generative answers](/power-virtual-agents/nlu-boost-conversations), deploying a model grounded on your data lets you create a chatbot that will respond using your data, and connect it to the Power Platform. For more information, see [Use a connection to Azure OpenAI on your data](/power-virtual-agents/nlu-generative-answers-azure-openai).

+> [!VIDEO https://www.microsoft.com/videoplayer/embed/RW18YwQ]

+#### Using the web app

+You can also use the available standalone web app to interact with your model using a graphical user interface, which you can deploy using either Azure OpenAI studio or a [manual deployment](https://github.com/microsoft/sample-app-aoai-chatGPT).

+#### Important considerations

+- Publishing creates an Azure App Service in your subscription. It may incur costs depending on the

+[pricing plan](https://azure.microsoft.com/pricing/details/app-service/windows/) you select. When you're done with your app, you can delete it from the Azure portal.

+- You can [customize](../concepts/use-your-data.md#using-the-web-app) the frontend and backend logic of the web app.

+- By default, the app will only be accessible to you. To add authentication (for example, restrict access to the app to members of your Azure tenant):

+ 1. Go to the [Azure portal](https://portal.azure.com/#home) and search for the app name you specified during publishing. Select the web app, and go to the **Authentication** tab on the left navigation menu. Then select **Add an identity provider**.

+

+ :::image type="content" source="../media/quickstarts/web-app-authentication.png" alt-text="Screenshot of the authentication page in the Azure portal." lightbox="../media/quickstarts/web-app-authentication.png":::

+ 1. Select Microsoft as the identity provider. The default settings on this page will restrict the app to your tenant only, so you don't need to change anything else here. Then select **Add**

+

+ Now users will be asked to sign in with their Azure Active Directory account to be able to access your app. You can follow a similar process to add another identity provider if you prefer. The app doesn't use the user's login information in any other way other than verifying they are a member of your tenant.

+## August 2023

+- You can now deploy Azure OpenAI on your data to [Power Virtual Agents](/azure/ai-services/openai/concepts/use-your-data#deploying-the-model).

+ Title: Azure Kubernetes Service (AKS) ingress with the application routing add-on (preview)

+# Azure Kubernetes Service (AKS) ingress with the application routing add-on (preview)

+> If using the [Virtual Nodes add-on](virtual-nodes-cli.md#enable-the-virtual-nodes-addon), DaemonSets will not create pods on the virtual node.

+ Title: Create an unmanaged ingress controller

+# Create an unmanaged ingress controller

+ Title: Azure Kubernetes Service (AKS) external or internal ingresses for Istio service mesh add-on (preview)

+# Azure Kubernetes Service (AKS) external or internal ingresses for Istio service mesh add-on deployment (preview)

+ Title: Create virtual nodes in Azure Kubernetes Service (AKS) using Azure CLI

+description: Learn how to use Azure CLI to create an Azure Kubernetes Services (AKS) cluster that uses virtual nodes to run pods.

+# Create and configure an Azure Kubernetes Services (AKS) cluster to use virtual nodes using Azure CLI

+Virtual nodes enable network communication between pods that run in Azure Container Instances (ACI) and AKS clusters. To provide this communication, you create a virtual network subnet and assign delegated permissions. Virtual nodes only work with AKS clusters created using *advanced* networking (Azure CNI). By default, AKS clusters are created with *basic* networking (kubenet). This article shows you how to create a virtual network and subnets, then deploy an AKS cluster that uses advanced networking.

+This article shows you how to use the Azure CLI to create and configure virtual network resources and an AKS cluster enabled with virtual nodes.

+* You need the ACI service provider registered with your subscription. You can check the status of the ACI provider registration using the [`az provider list`][az-provider-list] command.

+ ```azurecli-interactive

+ az provider list --query "[?contains(namespace,'Microsoft.ContainerInstance')]" -o table

+ ```

+ The *Microsoft.ContainerInstance* provider should report as *Registered*, as shown in the following example output:

+ ```output

+ Namespace RegistrationState RegistrationPolicy

+ - --

+ Microsoft.ContainerInstance Registered RegistrationRequired

+ ```

+ If the provider shows as *NotRegistered*, register the provider using the [`az provider register`][az-provider-register].

+ ```azurecli-interactive

+ az provider register --namespace Microsoft.ContainerInstance

+ ```

+* If using Azure CLI, this article requires Azure CLI version 2.0.49 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI]( /cli/azure/install-azure-cli). You can also use [Azure Cloud Shell](#launch-azure-cloud-shell).

+### Launch Azure Cloud Shell

+The Azure Cloud Shell is a free interactive shell you can use to run the steps in this article. It has common Azure tools preinstalled and configured.

+To open the Cloud Shell, select **Try it** from the upper right corner of a code block. You can also launch Cloud Shell in a separate browser tab by going to [https://shell.azure.com/bash](https://shell.azure.com/bash). Select **Copy** to copy the blocks of code, paste it into the Cloud Shell, and press enter to run it.

+An Azure resource group is a logical group in which Azure resources are deployed and managed.

+* Create a resource group using the [`az group create`][az-group-create] command.

+ ```azurecli-interactive

+ az group create --name myResourceGroup --location eastus

+ ```

+> Virtual node requires a custom virtual network and associated subnet. It can't be associated with the same virtual network as the AKS cluster.

+1. Create a virtual network using the [`az network vnet create`][az-network-vnet-create] command. The following example creates a virtual network named *myVnet* with an address prefix of *10.0.0.0/8* and a subnet named *myAKSSubnet*. The address prefix of this subnet defaults to *10.240.0.0/16*.

+ ```azurecli-interactive

+ az network vnet create \

+ --resource-group myResourceGroup \

+ --name myVnet \

+ --address-prefixes 10.0.0.0/8 \

+ --subnet-name myAKSSubnet \

+ --subnet-prefix 10.240.0.0/16

+ ```

+2. Create an extra subnet for the virtual nodes using the [`az network vnet subnet create`][az-network-vnet-subnet-create] command. The following example creates a subnet named *myVirtualNodeSubnet* with an address prefix of *10.241.0.0/16*.

+ ```azurecli-interactive

+ az network vnet subnet create \

+ --resource-group myResourceGroup \

+ --vnet-name myVnet \

+ --name myVirtualNodeSubnet \

+ --address-prefixes 10.241.0.0/16

+ ```

+1. Get the subnet ID using the [`az network vnet subnet show`][az-network-vnet-subnet-show] command.

+ ```azurecli-interactive

+ az network vnet subnet show --resource-group myResourceGroup --vnet-name myVnet --name myAKSSubnet --query id -o tsv

+ ```

+2. Create an AKS cluster using the [`az aks create`][az-aks-create] command and replace `<subnetId>` with the ID obtained in the previous step. The following example creates a cluster named *myAKSCluster* with five nodes.

+ ```azurecli-interactive

+ az aks create \

+ --resource-group myResourceGroup \

+ --name myAKSCluster \

+ --node-count 5 \

+ --network-plugin azure \

+ --vnet-subnet-id <subnetId>

+ ```

+ After several minutes, the command completes and returns JSON-formatted information about the cluster.

+For more information on managed identities, see [Use managed identities](use-managed-identity.md).

+## Enable the virtual nodes addon

+* Enable virtual nodes using the [`az aks enable-addons`][az-aks-enable-addons] command. The following example uses the subnet named *myVirtualNodeSubnet* created in a previous step.

+ ```azurecli-interactive

+ az aks enable-addons \

+ --resource-group myResourceGroup \

+ --name myAKSCluster \

+ --addons virtual-node \

+ --subnet-name myVirtualNodeSubnet

+ ```

+1. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az-aks-get-credentials] command. This step downloads credentials and configures the Kubernetes CLI to use them.

+ ```azurecli-interactive

+ az aks get-credentials --resource-group myResourceGroup --name myAKSCluster

+ ```

+2. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command, which returns a list of the cluster nodes.

+ ```console

+ kubectl get nodes

+ ```

+ The following example output shows the single VM node created and the virtual node for Linux, *virtual-node-aci-linux*:

+ ```output

+ NAME STATUS ROLES AGE VERSION

+ virtual-node-aci-linux Ready agent 28m v1.11.2

+ aks-agentpool-14693408-0 Ready agent 32m v1.11.2

+ ```

+1. Create a file named `virtual-node.yaml` and copy in the following YAML. The YAML schedules the container on the node by defining a [nodeSelector][node-selector] and [toleration][toleration].

+ ```yaml

+ apiVersion: apps/v1

+ kind: Deployment

+ name: aci-helloworld

+ replicas: 1

+ selector:

+ matchLabels:

+ app: aci-helloworld

+ template:

+ metadata:

+ labels:

+ app: aci-helloworld

+ spec:

+ containers:

+ - name: aci-helloworld

+ image: mcr.microsoft.com/azuredocs/aci-helloworld

+ ports:

+ - containerPort: 80

+ nodeSelector:

+ kubernetes.io/role: agent

+ beta.kubernetes.io/os: linux

+ type: virtual-kubelet

+ tolerations:

+ - key: virtual-kubelet.io/provider

+ operator: Exists

+ - key: azure.com/aci

+ effect: NoSchedule

+ ```

+2. Run the application using the [`kubectl apply`][kubectl-apply] command.

+ ```console

+ kubectl apply -f virtual-node.yaml

+ ```

+3. Get a list of pods and the scheduled node using the [`kubectl get pods`][kubectl-get] command with the `-o wide` argument.

+ ```console

+ kubectl get pods -o wide

+ ```

+ The pod is scheduled on the virtual node *virtual-node-aci-linux*, as shown in the following example output:

+ ```output

+ NAME READY STATUS RESTARTS AGE IP NODE

+ aci-helloworld-9b55975f-bnmfl 1/1 Running 0 4m 10.241.0.4 virtual-node-aci-linux

+ ```

+ The pod is assigned an internal IP address from the Azure virtual network subnet delegated for use with virtual nodes.

+> If you use images stored in Azure Container Registry, [configure and use a Kubernetes secret][acr-aks-secrets]. A current limitation of virtual nodes is you can't use integrated Azure AD service principal authentication. If you don't use a secret, pods scheduled on virtual nodes fail to start and report the error `HTTP response status code 400 error code "InaccessibleImage"`.

+1. Test the pod running on the virtual node by browsing to the demo application with a web client. As the pod is assigned an internal IP address, you can quickly test this connectivity from another pod on the AKS cluster.

+2. Create a test pod and attach a terminal session to it using the following `kubectl run -it` command.

+ ```console

+ kubectl run -it --rm testvk --image=mcr.microsoft.com/dotnet/runtime-deps:6.0

+ ```

+3. Install `curl` in the pod using `apt-get`.

+ ```console

+ apt-get update && apt-get install -y curl

+ ```

+4. Access the address of your pod using `curl`, such as *http://10.241.0.4*. Provide your own internal IP address shown in the previous `kubectl get pods` command.

+ ```console

+ curl -L http://10.241.0.4

+ ```

+ The demo application is displayed, as shown in the following condensed example output:

+ ```output

+ <html>

+ <head>

+ <title>Welcome to Azure Container Instances!</title>

+ </head>

+ [...]

+ ```

+5. Close the terminal session to your test pod with `exit`. When your session is ends, the pod is deleted.

+1. Delete the `aci-helloworld` pod running on the virtual node using the `kubectl delete` command.

+ ```console

+ kubectl delete -f virtual-node.yaml

+ ```

+2. Disable the virtual nodes using the [`az aks disable-addons`][az aks disable-addons] command.

+ ```azurecli-interactive

+ az aks disable-addons --resource-group myResourceGroup --name myAKSCluster --addons virtual-node

+ ```

+3. Remove the virtual network resources and resource group using the following commands.

+ ```azurecli-interactive

+ # Change the name of your resource group, cluster and network resources as needed

+ RES_GROUP=myResourceGroup

+ AKS_CLUSTER=myAKScluster

+ AKS_VNET=myVnet

+ AKS_SUBNET=myVirtualNodeSubnet

+ # Get AKS node resource group

+ NODE_RES_GROUP=$(az aks show --resource-group $RES_GROUP --name $AKS_CLUSTER --query nodeResourceGroup --output tsv)

+ # Get network profile ID

+ NETWORK_PROFILE_ID=$(az network profile list --resource-group $NODE_RES_GROUP --query "[0].id" --output tsv)

+ # Delete the network profile

+ az network profile delete --id $NETWORK_PROFILE_ID -y

+ # Grab the service association link ID

+ SAL_ID=$(az network vnet subnet show --resource-group $RES_GROUP --vnet-name $AKS_VNET --name $AKS_SUBNET --query id --output tsv)/providers/Microsoft.ContainerInstance/serviceAssociationLinks/default

+ # Delete the service association link for the subnet

+ az resource delete --ids $SAL_ID --api-version 2021-10-01

+ # Delete the subnet delegation to Azure Container Instances

+ az network vnet subnet update --resource-group $RES_GROUP --vnet-name $AKS_VNET --name $AKS_SUBNET --remove delegations

+ ```

+In this article, you scheduled a pod on the virtual node and assigned a private internal IP address. You could instead create a service deployment and route traffic to your pod through a load balancer or ingress controller. For more information, see [Create a basic ingress controller in AKS][aks-basic-ingress].

+* [Use the Kubernetes horizontal pod autoscaler][aks-hpa]

+* [Use the Kubernetes cluster autoscaler][aks-cluster-autoscaler]

+* [Check out the Autoscale sample for Virtual Nodes][virtual-node-autoscale]

+* [Read more about the Virtual Kubelet open source library][virtual-kubelet-repo]

+ Title: Windows AKS Partner Solutions

+description: Find partner-tested solutions that enable you to build, test, deploy, manage and monitor your Windows-based apps on Windows containers on AKS.

+# Windows AKS Partners Solutions

+Microsoft has collaborated with partners to ensure your build, test, deployment, configuration, and monitoring of your applications perform optimally with Windows containers on AKS.

+Our 3rd party partners featured below have published introduction guides to start using their solutions with your applications running on Windows containers on AKS.

+| Solutions | Partners |

+|--|--|

+| DevOps | [GitLab](#gitlab) <br> [CircleCI](#circleci) |

+| Networking | [NGINX](#f5-nginx) <br> [Calico](#calico) |

+| Observability | [Datadog](#datadog) <br> [New Relic](#new-relic) |

+| Security | [Prisma](#prisma) |

+| Storage | [NetApp](#netapp) |

+| Config Management | [Chef](#chef) |

+## DevOps

+DevOps streamlines the delivery process, improves collaboration across teams, and enhances software quality, ensuring swift, reliable, and continuous deployment of your Windows-based applications.

+### GitLab

+The GitLab DevSecOps Platform supports the Microsoft development ecosystem with performance, accessibility testing, SAST, DAST and Fuzzing security scanning, dependency scanning, SBOM, license management and more.

+As an extensible platform, GitLab also allows you to plug in your own tooling for any stage. GitLab's integration with Azure Kubernetes Services (AKS) enables full DevSecOps workflows for Windows and Linux Container workloads using either Push CD or GitOps Pull CD with flux manifests. Using Cloud Native Buildpaks, GitLab Auto DevOps can build, test and autodeploy OSS .NET projects.

+To learn more, please our see our [joint blog](https://techcommunity.microsoft.com/t5/containers/using-gitlab-to-build-and-deploy-windows-containers-on-azure/ba-p/3889929).

+### CircleCI

+CircleCIΓÇÖs integration with Azure Kubernetes Services (AKS) allows you to automate, build, validate, and ship containerized Windows applications, ensuring faster and more reliable software deployment. You can easily integrate your pipeline with AKS using CircleCI orbs, which are prepacked snippets of YAML configuration.

+

+Follow this [tutorial](https://techcommunity.microsoft.com/t5/containers/continuous-deployment-of-windows-containers-with-circleci-and/ba-p/3841220) to learn how to set up a CI/CD pipeline to build a Dockerized ASP.NET application and deploy it to an AKS cluster.

+## Networking

+Ensure efficient traffic management, enhanced security, and optimal network performance with these solutions to achieve smooth application connectivity and communication.

+### F5 NGINX

+NGINX Ingress Controller deployed in AKS, on-premises, and in the cloud implements unified Kubernetes-native API gateways, load balancers, and Ingress controllers to reduce complexity, increase uptime, and provide in-depth insights into app health and performance for containerized Windows workloads.

+Running at the edge of a Kubernetes cluster, NGINX Ingress Controller ensures holistic app security with user and service identities, authorization, access control, encrypted communications, and additional NGINX App Protect modules for Layer 7 WAF and DoS app protection.

+Learn how to manage connectivity to your Windows applications running on Windows nodes in a mixed-node AKS cluster with NGINX Ingress controller in this [blog](https://techcommunity.microsoft.com/t5/containers/improving-customer-experiences-with-f5-nginx-and-windows-on/ba-p/3820344).

+### Calico

+Tigera provides an active security platform with full-stack observability for containerized workloads and Microsoft AKS as a fully managed SaaS (Calico Cloud) or a self-managed service (Calico Enterprise). The platform prevents, detects, troubleshoots, and automatically mitigates exposure risks of security breaches for workloads in Microsoft AKS.

+Its open-source offering, Calico Open Source, is the most widely adopted container networking and security solution. It specifies security and observability as code to ensure consistent enforcement of security policies, which enables DevOps, platform, and security teams to protect workloads, detect threats, achieve continuous compliance, and troubleshoot service issues in real-time.

+To learn more, [click here](https://techcommunity.microsoft.com/t5/containers/securing-windows-workloads-on-azure-kubernetes-service-with/ba-p/3815429).

+## Observability

+Observability provides deep insights into your systems, enabling rapid issue detection and resolution to enhance your applicationΓÇÖs reliability and performance.

+### Datadog

+Datadog is the essential monitoring and security platform for cloud applications. We bring together end-to-end traces, metrics, and logs to make your applications, infrastructure, and third-party services entirely observable. Partner with Datadog for Windows on AKS environments to streamline monitoring, proactively resolve issues, and optimize application performance and availability.

+Get started by following the recommendations in our [joint blog](https://techcommunity.microsoft.com/t5/containers/gain-full-observability-into-windows-containers-on-azure/ba-p/3853603).

+### New Relic

+New Relic's Azure Kubernetes integration is a powerful solution that seamlessly connects New Relic's monitoring and observability capabilities with Azure Kubernetes Service (AKS). By deploying the New Relic Kubernetes integration, users gain deep insights into their AKS clusters' performance, health, and resource utilization. This integration allows users to efficiently manage and troubleshoot containerized applications, optimize resource allocation, and proactively identify and resolve issues in their AKS environments. With New Relic's comprehensive monitoring and analysis tools, businesses can ensure the smooth operation and optimal performance of their Kubernetes workloads on Azure.

+Check this [blog](https://techcommunity.microsoft.com/t5/containers/persistent-storage-for-windows-containers-on-azure-kubernetes/ba-p/3836781) for detailed information.

+## Security

+Ensure the integrity and confidentiality of applications, thereby fostering trust and compliance across your infrastructure.

+### Prisma

+Prisma Cloud is a comprehensive Cloud-Native Application Protection Platform (CNAPP) tailor-made to help secure Windows containers on Azure Kubernetes Service (AKS). Gain continuous, real-time visibility and control over Windows container environments including vulnerability and compliance management, identities and permissions, and AI-assisted runtime defense. Integrated container scanning across the pipeline and in Azure Container Registry ensure security throughout the entire application lifecycle.

+See [our guidance](https://techcommunity.microsoft.com/t5/containers/unlocking-new-possibilities-with-prisma-cloud-and-windows/ba-p/3866485) for more details.

+## Storage

+Storage enables standardized and seamless storage interactions, ensuring high application performance and data consistency.

+### NetApp

+Astra Control provides application data management for stateful workloads on Azure Kubernetes Service (AKS). Discover your apps and define protection policies that automatically back up workloads offsite. Protect, clone, and move applications across Kubernetes environments with ease.

+Follow the steps provided in [this blog](https://techcommunity.microsoft.com/t5/containers/persistent-storage-for-windows-containers-on-azure-kubernetes/ba-p/3836781) post to dynamically provision SMB volumes for Windows AKS workloads.

+## Config management

+Automate and standardize the system settings across your environments to enhance efficiency, reduce errors, and ensuring system stability and compliance.

+### Chef

+Chef provides visibility and threat detection from build to runtime that monitors, audits, and remediates the security of your Azure cloud services and Kubernetes and Windows container assets. Chef provides comprehensive visibility and continuous compliance into your cloud security posture and helps limit the risk of misconfigurations in cloud-native environments by providing best practices based on CIS, STIG, SOC2, PCI-DSS and other benchmarks. This is part of a broader compliance offering that supports on-premises or hybrid cloud environments including applications deployed on the edge.

+To learn more about ChefΓÇÖs capabilities, check out the comprehensive ΓÇÿhow-toΓÇÖ blog post here: [Securing Your Windows Environments Running on Azure Kubernetes Service with Chef](https://techcommunity.microsoft.com/t5/containers/securing-your-windows-environments-running-on-azure-kubernetes/ba-p/3821830).

+ $connStrings[$item.Name] = @{value=$item.ConnectionString; type=$item.Type.ToString()}

+**x-request-id** is a unique guid generated by Application Gateway for Containers for each client request and presented in the forwarded request to the backend target. The guid consists of 32 alphanumeric characters, separated by dashes (for example: d23387ab-e629-458a-9c93-6108d374bc75). This guid can be used to correlate a request received by Application Gateway for Containers and initiated to a backend target as defined in access logs.

+ Title: Back up controller database

+description: Explains how to back up the controller database for Azure Arc-enabled data services

+# Back up and recover controller database

+When you deploy Azure Arc data services, the Azure Arc Data Controller is one of the most critical components that is deployed. The functions of the data controller include:

+- Provision, de-provision and update resources

+- Orchestrate most of the activities for Azure Arc-enabled SQL Managed Instance such as upgrades, scale out etc.

+- Capture the billing and usage information of each Arc SQL managed instance.

+In order to perform above functions, the Data controller needs to store an inventory of all the current Arc SQL managed instances, billing, usage and the current state of all these SQL managed instances. All this data is stored in a database called `controller` within the SQL Server instance that is deployed into the `controldb-0` pod.

+## Back up data controller database

+As part of built-in capabilities, the Data controller database `controller` is automatically backed up every 5 minutes once backups are enabled. To enable backups:

+- Create a `backups-controldb` `PersistentVolumeClaim` with a storage class that supports `ReadWriteMany` access:

+```yaml

+apiVersion: v1

+kind: PersistentVolumeClaim

+metadata:

+ name: backups-controldb

+ namespace: <namespace>

+spec:

+ accessModes:

+ - ReadWriteMany

+ resources:

+ requests:

+ storage: 15Gi

+ storageClassName: <storage-class>

+- Edit the `DataController` custom resource spec to include a `backups` storage definition:

+```yaml

+storage:

+ backups:

+ accessMode: ReadWriteMany

+ className: <storage-class>

+ size: 15Gi

+ data:

+ accessMode: ReadWriteOnce

+ className: managed-premium

+ size: 15Gi

+ logs:

+ accessMode: ReadWriteOnce

+ className: managed-premium

+ size: 10Gi

+The `.bak` files for the `controller` database are stored on the `backups` volume of the `controldb` pod at `/var/opt/backups/mssql`.

+## Recover controller database

+There are two types of recovery possible:

+1. `controller` is corrupted and you just need to restore the database

+1. the entire storage that contains the `controller` data and log files is corrupted/gone and you need to recover

+### Corrupted controller database scenario

+In this scenario, all the pods are up and running, you are able to connect to the `controldb` SQL Server, and there may be a corruption with the `controller` database. You just need to restore the database from a backup.

+Follow these steps to restore the controller database from a backup, if the SQL Server is still up and running on the `controldb` pod, and you are able to connect to it:

+1. Verify connectivity to SQL Server pod hosting the `controller` database.

+ - First, retrieve the credentials for the secret. `controller-system-secret` is the secret that holds the credentials for the `system` user account that can be used to connect to the SQL instance.

+ Run the following command to retrieve the secret contents:

+

+ ```console

+ kubectl get secret controller-system-secret --namespace [namespace] -o yaml

+ ```

+ For example:

+ ```console

+ kubectl get secret controller-system-secret --namespace arcdataservices -o yaml

+ ```

+ - Decode the base64 encoded credentials. The contents of the yaml file of the secret `controller-system-secret` contain a `password` and `username`. You can use any base64 decoder tool to decode the contents of the `password`.

+ - Verify connectivity: With the decoded credentials, run a command such as `SELECT @@SERVERNAME` to verify connectivity to the SQL Server.

+ ```powershell

+ kubectl exec controldb-0 -n <namespace> -c mssql-server -- /opt/mssql-tools/bin/sqlcmd -S localhost -U system -P "<password>" -Q "SELECT @@SERVERNAME"

+ ```

+

+ ```powershell

+ kubectl exec controldb-0 -n contosons -c mssql-server -- /opt/mssql-tools/bin/sqlcmd -S localhost -U system -P "<password>" -Q "SELECT @@SERVERNAME"

+ ```

+1. Scale the controller ReplicaSet down to 0 replicas as follows:

+ ```console

+ kubectl scale --replicas=0 rs/control -n <namespace>`

+ ```

+ For example:

+ ```console

+ kubectl scale --replicas=0 rs/control -n arcdataservices

+ ```

+1. Connect to the `controldb` SQL Server as `system` as described in step 1.

+1. Delete the corrupted controller database using T-SQL:

+ ```sql

+ DROP DATABASE controller

+ ```

+1. Restore the database from backup - after the corrupted `controllerdb` is dropped. For example:

+ ```sql

+ RESTORE DATABASE test FROM DISK = '/var/opt/backups/mssql/<controller backup file>.bak'

+ WITH MOVE 'controller' to '/var/opt/mssql/datf

+ ,MOVE 'controller' to '/var/opt/mssql/data/controller_log.ldf'

+ ,RECOVERY;

+ GO

+ ```

+

+1. Scale the controller ReplicaSet back up to 1 replica.

+ ```console

+ kubectl scale --replicas=1 rs/control -n <namespace>

+ ```

+ For example:

+ ```console

+ kubectl scale --replicas=1 rs/control -n arcdataservices

+ ```

+### Corrupted storage scenario

+In this scenario, the storage hosting the Data controller data and log files, has corruption and a new storage was provisioned and you need to restore the controller database.

+Follow these steps to restore the controller database from a backup with new storage for the `controldb` StatefulSet:

+1. Ensure that you have a backup of the last known good state of the `controller` database

+2. Scale the controller ReplicaSet down to 0 replicas as follows:

+ ```console

+ kubectl scale --replicas=0 rs/control -n <namespace>

+ ```

+ For example:

+ ```console

+ kubectl scale --replicas=0 rs/control -n arcdataservices

+ ``

+3. Scale the `controldb` StatefulSet down to 0 replicas, as follows:

+ ```console

+ kubectl scale --replicas=0 sts/controldb -n <namespace>

+ ```

+ For example:

+ ```console

+ kubectl scale --replicas=0 sts/controldb -n arcdataservices`

+ ```

+4. Create a kubernetes secret named `controller-sa-secret` with the following YAML:

+ ```yml

+ apiVersion: v1

+ kind: Secret

+ metadata:

+ name: controller-sa-secret

+ namespace: <namespace>

+ type: Opaque

+ data:

+ password: <base64 encoded password>

+ ```

+5. Edit the `controldb` StatefulSet to include a `controller-sa-secret` volume and corresponding volume mount (`/var/run/secrets/mounts/credentials/mssql-sa-password`) in the `mssql-server` container, by using `kubectl edit sts controldb -n <namespace>` command.

+6. Create new data (`data-controldb`) and logs (`logs-controldb`) persistent volume claims for the `controldb` pod as follows:

+ ```yml

+ apiVersion: v1

+ kind: PersistentVolumeClaim

+ metadata:

+ name: data-controldb

+ namespace: <namespace>

+ spec:

+ accessModes:

+ - ReadWriteOnce

+ resources:

+ requests:

+ storage: 15Gi

+ storageClassName: <storage class>

+

+

+ apiVersion: v1

+ kind: PersistentVolumeClaim

+ metadata:

+ name: logs-controldb

+ namespace: <namespace>

+ spec:

+ accessModes:

+ - ReadWriteOnce

+ resources:

+ requests:

+ storage: 10Gi

+ storageClassName: <storage class>

+ ```

+7. Scale the `controldb` StatefulSet back to 1 replica using:

+ ```console

+ kubectl scale --replicas=1 sts/controldb -n <namespace>

+ ```

+8. Connect to the `controldb` SQL server as `sa` using the password in the `controller-sa-secret` secret created earlier.

+9. Create a `system` login with sysadmin role using the password in the `controller-system-secret` kubernetes secret as follows:

+ ```sql

+ CREATE LOGIN [system] WITH PASSWORD = '<password-from-secret>'

+ ALTER SERVER ROLE sysadmin ADD MEMBER [system]

+ ```

+10. Restore the backup using the `RESTORE` command as follows:

+ ```sql

+ RESTORE DATABASE [controller] FROM DISK = N'/var/opt/backups/mssql/<controller backup file>.bak' WITH FILE = 1

+ ```

+11. Create a `controldb-rw-user` login using the password in the `controller-db-rw-secret` secret `CREATE LOGIN [controldb-rw-user] WITH PASSWORD = '<password-from-secret>'` and associate it with the existing `controldb-rw-user` user in the controller DB `ALTER USER [controldb-rw-user] WITH LOGIN = [controldb-rw-user]`.

+12. Disable the `sa` login using TSQL - `ALTER LOGIN [sa] DISABLE`.

+13. Edit the `controldb` StatefulSet to remove the `controller-sa-secret` volume and corresponding volume mount.

+14. Delete the `controller-sa-secret` secret.

+16. Scale the controller ReplicaSet back up to 1 replica using the `kubectl scale` command.

+[Azure Data Studio dashboards](azure-data-studio-dashboards.md)

+ Title: Configure failover group - CLI

+description: Describes how to configure disaster recovery with a failover group for Azure Arc-enabled SQL Managed Instance with the CLI

+# Configure failover group - CLI

+This article explains how to configure disaster recovery for Azure Arc-enabled SQL Managed Instance with the CLI. Before you proceed, review the information and prerequisites in [Azure Arc-enabled SQL Managed Instance - disaster recovery](managed-instance-disaster-recovery.md).

+## Configure Azure failover group - direct mode

+Follow the steps below if the Azure Arc data services are deployed in `directly` connected mode.

+Once the prerequisites are met, run the below command to set up Azure failover group between the two Azure Arc-enabled SQL managed instances:

+```azurecli

+az sql instance-failover-group-arc create --name <name of failover group> --mi <primary SQL MI> --partner-mi <Partner MI> --resource-group <name of RG> --partner-resource-group <name of partner MI RG>

+```

+Example:

+```azurecli

+az sql instance-failover-group-arc create --name sql-fog --mi sql1 --partner-mi sql2 --resource-group rg-name --partner-resource-group rg-name

+```

+The above command:

+- Creates the required custom resources on both primary and secondary sites

+- Copies the mirroring certificates and configures the failover group between the instances

+## Configure Azure failover group - indirect mode

+Follow the steps below if Azure Arc data services are deployed in `indirectly` connected mode.

+1. Provision the managed instance in the primary site.

+ ```azurecli

+ az sql mi-arc create --name <primaryinstance> --tier bc --replicas 3 --k8s-namespace <namespace> --use-k8s

+ ```

+2. Switch context to the secondary cluster by running ```kubectl config use-context <secondarycluster>``` and provision the managed instance in the secondary site that will be the disaster recovery instance. At this point, the system databases are not part of the contained availability group.

+ > [!NOTE]

+ > It is important to specify `--license-type DisasterRecovery` **during** the Azure Arc-enabled SQL MI creation. This will allow the DR instance to be seeded from the primary instance in the primary data center. Updating this property post deployment will not have the same effect.

+ ```azurecli

+ az sql mi-arc create --name <secondaryinstance> --tier bc --replicas 3 --license-type DisasterRecovery --k8s-namespace <namespace> --use-k8s

+ ```

+3. Mirroring certificates - The binary data inside the Mirroring Certificate property of the Azure Arc-enabled SQL MI is needed for the Instance Failover Group CR (Custom Resource) creation.

+ This can be achieved in a few ways:

+ (a) If using `az` CLI, generate the mirroring certificate file first, and then point to that file while configuring the Instance Failover Group so the binary data is read from the file and copied over into the CR. The cert files are not needed after failover group creation.

+ (b) If using `kubectl`, directly copy and paste the binary data from the Azure Arc-enabled SQL MI CR into the yaml file that will be used to create the Instance Failover Group.

+ Using (a) above:

+ Create the mirroring certificate file for primary instance:

+ ```azurecli

+ az sql mi-arc get-mirroring-cert --name <primaryinstance> --cert-file </path/name>.pemΓÇï --k8s-namespace <namespace> --use-k8s

+ ```

+ Example:

+ ```azurecli

+ az sql mi-arc get-mirroring-cert --name sqlprimary --cert-file $HOME/sqlcerts/sqlprimary.pemΓÇï --k8s-namespace my-namespace --use-k8s

+ ```

+ Connect to the secondary cluster and create the mirroring certificate file for secondary instance:

+ ```azurecli

+ az sql mi-arc get-mirroring-cert --name <secondaryinstance> --cert-file </path/name>.pem --k8s-namespace <namespace> --use-k8s

+ ```

+ Example:

+ ```azurecli

+ az sql mi-arc get-mirroring-cert --name sqlsecondary --cert-file $HOME/sqlcerts/sqlsecondary.pem --k8s-namespace my-namespace --use-k8s

+ ```

+ Once the mirroring certificate files are created, copy the certificate from the secondary instance to a shared/local path on the primary instance cluster and vice-versa.

+4. Create the failover group resource on both sites.

+ > [!NOTE]

+ > Ensure the SQL instances have different names for both primary and secondary sites, and the `shared-name` value should be identical on both sites.

+

+ ```azurecli

+ az sql instance-failover-group-arc create --shared-name <name of failover group> --name <name for primary failover group resource> --mi <local SQL managed instance name> --role primary --partner-mi <partner SQL managed instance name> --partner-mirroring-url tcp://<secondary IP> --partner-mirroring-cert-file <secondary.pem> --k8s-namespace <namespace> --use-k8s

+ ```

+ Example:

+ ```azurecli

+ az sql instance-failover-group-arc create --shared-name myfog --name primarycr --mi sqlinstance1 --role primary --partner-mi sqlinstance2 --partner-mirroring-url tcp://10.20.5.20:970 --partner-mirroring-cert-file $HOME/sqlcerts/sqlinstance2.pem --k8s-namespace my-namespace --use-k8s

+ ```

+ On the secondary instance, run the following command to set up the failover group custom resource. The `--partner-mirroring-cert-file` in this case should point to a path that has the mirroring certificate file generated from the primary instance as described in 3(a) above.

+ ```azurecli

+ az sql instance-failover-group-arc create --shared-name <name of failover group> --name <name for secondary failover group resource> --mi <local SQL managed instance name> --role secondary --partner-mi <partner SQL managed instance name> --partner-mirroring-url tcp://<primary IP> --partner-mirroring-cert-file <primary.pem> --k8s-namespace <namespace> --use-k8s

+ ```

+ Example:

+ ```azurecli

+ az sql instance-failover-group-arc create --shared-name myfog --name secondarycr --mi sqlinstance2 --role secondary --partner-mi sqlinstance1 --partner-mirroring-url tcp://10.10.5.20:970 --partner-mirroring-cert-file $HOME/sqlcerts/sqlinstance1.pem --k8s-namespace my-namespace --use-k8s

+ ```

+## Retrieve Azure failover group health state

+Information about the failover group such as primary role, secondary role, and the current health status can be viewed on the custom resource on either primary or secondary site.

+Run the below command on primary and/or the secondary site to list the failover groups custom resource:

+```azurecli

+kubectl get fog -n <namespace>

+```

+Describe the custom resource to retrieve the failover group status, as follows:

+```azurecli

+kubectl describe fog <failover group cr name> -n <namespace>

+```

+## Failover group operations

+Once the failover group is set up between the managed instances, different failover operations can be performed depending on the circumstances.

+Possible failover scenarios are:

+- The Azure Arc-enabled SQL managed instances at both sites are in healthy state and a failover needs to be performed:

+ + perform a manual failover from primary to secondary without data loss by setting `role=secondary` on the primary SQL MI.

+

+- Primary site is unhealthy/unreachable and a failover needs to be performed:

+

+ + the primary Azure Arc-enabled SQL managed instance is down/unhealthy/unreachable

+ + the secondary Azure Arc-enabled SQL managed instance needs to be force-promoted to primary with potential data loss

+ + when the original primary Azure Arc-enabled SQL managed instance comes back online, it will report as `Primary` role and unhealthy state and needs to be forced into a `secondary` role so it can join the failover group and data can be synchronized.

+

+## Manual failover (without data loss)

+Use `az sql instance-failover-group-arc update ...` command group to initiate a failover from primary to secondary. Any pending transactions on the geo-primary instance are replicated over to the geo-secondary instance before the failover.

+### Directly connected mode

+Run the following command to initiate a manual failover, in `direct` connected mode using ARM APIs:

+```azurecli

+az sql instance-failover-group-arc update --name <shared name of failover group> --mi <primary Azure Arc-enabled SQL MI> --role secondary --resource-group <resource group>

+```

+Example:

+```azurecli

+az sql instance-failover-group-arc update --name myfog --mi sqlmi1 --role secondary --resource-group myresourcegroup

+```

+### Indirectly connected mode

+Run the following command to initiate a manual failover, in `indirect` connected mode using kubernetes APIs:

+```azurecli

+az sql instance-failover-group-arc update --name <name of failover group resource> --role secondary --k8s-namespace <namespace> --use-k8s

+```

+Example:

+```azurecli

+az sql instance-failover-group-arc update --name myfog --role secondary --k8s-namespace my-namespace --use-k8s

+```

+## Forced failover with data loss

+In the circumstance when the geo-primary instance becomes unavailable, the following commands can be run on the geo-secondary DR instance to promote to primary with a forced failover incurring potential data loss.

+On the geo-secondary DR instance, run the following command to promote it to primary role, with data loss.

+> [!NOTE]

+> If the `--partner-sync-mode` was configured as `sync`, it needs to be reset to `async` when the secondary is promoted to primary.

+### Directly connected mode

+```azurecli

+az sql instance-failover-group-arc update --name <shared name of failover group> --mi <secondary Azure Arc-enabled SQL MI> --role force-primary-allow-data-loss --resource-group <resource group> --partner-sync-mode async

+```

+Example:

+```azurecli

+az sql instance-failover-group-arc update --name myfog --mi sqlmi2 --role force-primary-allow-data-loss --resource-group myresourcegroup --partner-sync-mode async

+```

+### Indirectly connected mode

+```azurecli

+az sql instance-failover-group-arc update --k8s-namespace my-namespace --name secondarycr --use-k8s --role force-primary-allow-data-loss --partner-sync-mode async

+```

+When the geo-primary Azure Arc-enabled SQL MI instance becomes available, run the below command to bring it into the failover group and synchronize the data:

+### Directly connected mode

+```azurecli

+az sql instance-failover-group-arc update --name <shared name of failover group> --mi <old primary Azure Arc-enabled SQL MI> --role force-secondary --resource-group <resource group>

+```

+### Indirectly connected mode

+```azurecli

+az sql instance-failover-group-arc update --k8s-namespace my-namespace --name secondarycr --use-k8s --role force-secondary

+```

+Optionally, the `--partner-sync-mode` can be configured back to `sync` mode if desired.

+## Post failover operations

+Once you perform a failover from primary site to secondary site, either with or without data loss, you may need to do the following:

+- Update the connection string for your applications to connect to the newly promoted primary Arc SQL managed instance

+- If you plan to continue running the production workload off of the secondary site, update the `--license-type` to either `BasePrice` or `LicenseIncluded` to initiate billing for the vCores consumed.

+## Next steps

+- [Overview: Azure Arc-enabled SQL Managed Instance business continuity](managed-instance-business-continuity-overview.md)

+- [Configure failover group - portal](managed-instance-disaster-recovery-portal.md)

+ Title: Disaster recovery - Azure Arc-enabled SQL Managed Instance - portal

+description: Describes how to configure disaster recovery for Azure Arc-enabled SQL Managed Instance in the portal

+# Configure failover group - portal

+This article explains how to configure disaster recovery for Azure Arc-enabled SQL Managed Instance with Azure portal. Before you proceed, review the information and prerequisites in [Azure Arc-enabled SQL Managed Instance - disaster recovery](managed-instance-disaster-recovery.md).

+To configure disaster recovery through Azure portal, the Azure Arc-enabled data service requires direct connectivity to Azure.

+## Configure Azure failover group

+1. In the portal, go to your primary availability group.

+1. Under **Data Management**, select **Failover Groups**.

+ Azure portal presents **Create instance failover group**.

+ :::image type="content" source="media/managed-instance-disaster-recovery-portal/create-failover-group.png" alt-text="Screenshot of the Azure portal create instance failover group control.":::

+1. Provide the information to define the failover group.

+ * **Primary mirroring URL**: The mirroring endpoint for the failover group instance.

+ * **Resource group**: The resource group for the failover group instance.

+ * **Secondary managed instance**: The Azure SQL Managed Instance at the DR location.

+ * **Synchronization mode**: Select either *Sync* for synchronous mode, or *Async* for asynchronous mode.

+ * **Instance failover group name**: The name of the failover group.

+

+1. Select **Create**.

+Azure portal begins to provision the instance failover group.

+## View failover group

+After the failover group is provisioned, you can view it in Azure portal.

+## Fail over

+In the disaster recovery configuration, only one of the instances in the failover group is primary. You can fail over from the portal to migrate the primary role to the other instance in your failover group. To fail over:

+1. In portal, locate your managed instance.

+1. Under **Data Management** select **Failover Groups**.

+1. Select **Failover**.

+Monitor failover progress in Azure portal.

+## Set synchronization mode

+To set the synchronization mode:

+1. From **Failover Groups**, select **Edit configuration**.

+ Azure portal shows an **Edit Configuration** control.

+ :::image type="content" source="media/managed-instance-disaster-recovery-portal/edit-synchronization.png" alt-text="Screenshot of the Edit Configuration control.":::

+1. Under **Edit configuration**, select your desired mode, and select **Apply**.

+## Delete failover group

+1. From Failover Groups**, select **Delete Failover Group**.

+ Azure portal asks you to confirm your choice to delete the failover group.

+1. Select **Delete failover group** to proceed. Otherwise select **Cancel**, to not delete the group.

+## Next steps

+- [Overview: Azure Arc-enabled SQL Managed Instance business continuity](managed-instance-business-continuity-overview.md)

+- [Configure failover group - CLI](managed-instance-disaster-recovery-cli.md)

+To configure disaster recovery in Azure Arc-enabled SQL Managed Instance, set up Azure failover groups. This article explains failover groups.

+You can configure failover groups in with the CLI or in the portal. For prerequisites and instructions see the respective content below:

+- [Configure failover group - portal](managed-instance-disaster-recovery-portal.md)

+- [Configure failover group - CLI](managed-instance-disaster-recovery-cli.md)

+- [Overview: Azure Arc-enabled SQL Managed Instance business continuity](managed-instance-business-continuity-overview.md)

+## August 8, 2023

+### Image tag

+`v1.22.0_2023-08-08`

+For complete release version information, review [Version log](version-log.md#august-8-2023).

+### Release notes

+- Support for configuring and managing Azure Failover groups between two Azure Arc-enabled SQL managed instances using Azure portal. For details, review [Configure failover group - portal](managed-instance-disaster-recovery-portal.md).

+- Upgraded OpenSearch and OpenSearch Dashboards from 2.7.0 to 2.8.0

+- Improvements and examples to [Back up and recover controller database](backup-controller-database.md).

+## August 8, 2023

+Aug 2023 preview release is now available.

+|Component|Value|

+|--|--|

+|Container images tag |`v1.22.0_2023-08-08`|

+|**CRD names and version:**| |

+|`activedirectoryconnectors.arcdata.microsoft.com`| v1beta1, v1beta2, v1, v2|

+|`datacontrollers.arcdata.microsoft.com`| v1beta1, v1 through v5|

+|`exporttasks.tasks.arcdata.microsoft.com`| v1beta1, v1, v2|

+|`failovergroups.sql.arcdata.microsoft.com`| v1beta1, v1beta2, v1, v2|

+|`kafkas.arcdata.microsoft.com`| v1beta1 through v1beta4|

+|`monitors.arcdata.microsoft.com`| v1beta1, v1, v3|

+|`postgresqls.arcdata.microsoft.com`| v1beta1 through v1beta6|

+|`postgresqlrestoretasks.tasks.postgresql.arcdata.microsoft.com`| v1beta1|

+|`sqlmanagedinstances.sql.arcdata.microsoft.com`| v1beta1, v1 through v13|

+|`sqlmanagedinstancemonitoringprofiles.arcdata.microsoft.com`| v1beta1, v1beta2|

+|`sqlmanagedinstancereprovisionreplicatasks.tasks.sql.arcdata.microsoft.com`| v1beta1|

+|`sqlmanagedinstancerestoretasks.tasks.sql.arcdata.microsoft.com`| v1beta1, v1|

+|`telemetrycollectors.arcdata.microsoft.com`| v1beta1 through v1beta5|

+|`telemetryrouters.arcdata.microsoft.com`| v1beta1 through v1beta5|

+|Azure Resource Manager (ARM) API version|2023-01-15-preview|

+|`arcdata` Azure CLI extension version|1.5.4 ([Download](https://aka.ms/az-cli-arcdata-ext))|

+|Arc-enabled Kubernetes helm chart extension version|1.22.0|

+|Azure Arc Extension for Azure Data Studio<br/>`arc`<br/>`azcli`|<br/>1.8.0 ([Download](https://aka.ms/ads-arcdata-ext))</br>1.8.0 ([Download](https://aka.ms/ads-azcli-ext))|

+|SQL Database version | 957 |

+#### Register Azure clients

+Dependency injection can be used to interact with other Azure services. You can inject clients from the [Azure SDK for .NET](/dotnet/azure/sdk/azure-sdk-for-dotnet) using the [Microsoft.Extensions.Azure](https://www.nuget.org/packages/Microsoft.Extensions.Azure) package. After installing the package, [register the clients](/dotnet/azure/sdk/dependency-injection#register-clients) by calling `AddAzureClients()` on the service collection in `Program.cs`. The following example configures a [named client](/dotnet/azure/sdk/dependency-injection#configure-multiple-service-clients-with-different-names) for Azure Blobs:

+```csharp

+using Microsoft.Extensions.Azure;

+using Microsoft.Extensions.Hosting;

+var host = new HostBuilder()

+ .ConfigureFunctionsWorkerDefaults()

+ .ConfigureServices((hostContext, services) =>

+ {

+ services.AddAzureClients(clientBuilder =>

+ {

+ clientBuilder.AddBlobServiceClient(hostContext.Configuration.GetSection("MyStorageConnection"))

+ .WithName("copierOutputBlob");

+ });

+ })

+ .Build();

+host.Run();

+```

+The following example shows how we can use this registration and [SDK types](#sdk-types) to copy blob contents as a stream from one container to another using an injected client:

+```csharp

+using Microsoft.Extensions.Azure;

+using Microsoft.Extensions.Logging;

+namespace MyFunctionApp

+{

+ public class BlobCopier

+ {

+ private readonly ILogger<BlobCopier> _logger;

+ private readonly BlobContainerClient _copyContainerClient;

+ public BlobCopier(ILogger<BlobCopier> logger, IAzureClientFactory<BlobServiceClient> blobClientFactory)

+ {

+ _logger = logger;

+ _copyContainerClient = blobClientFactory.CreateClient("copierOutputBlob").GetBlobContainerClient("samples-workitems-copy");

+ _copyContainerClient.CreateIfNotExists();

+ }

+ [Function("BlobCopier")]

+ public async Task Run([BlobTrigger("samples-workitems/{name}", Connection = "MyStorageConnection")] Stream myBlob, string name)

+ {

+ await _copyContainerClient.UploadBlobAsync(name, myBlob);

+ _logger.LogInformation($"Blob {name} copied!");

+ }

+ }

+}

+```

+The [ILogger&lt;T&gt;] in this example was also obtained through dependency injection. It is registered automatically. To learn more about configuration options for logging, see [Logging](#logging).

+> [!TIP]

+> The example used a literal string for the name of the client in both `Program.cs` and the function. Consider instead using a shared constant string defined on the function class. For example, you could add `public const string CopyStorageClientName = nameof(_copyContainerClient);` and then reference `BlobCopier.CopyStorageClientName` in both locations. You could similarly define the configuration section name with the function rather than in `Program.cs`.

+¹ For output scenarios in which you would use an SDK type, you should create and work with SDK clients directly instead of using an output binding. See [Register Azure clients](#register-azure-clients) for an example of how to do this with dependency injection.

+[ILoggerFactory]: /dotnet/api/microsoft.extensions.logging.iloggerfactory

+This setting is required for Consumption plan apps on Windows and for Elastic Premium plan apps on both Windows and Linux. It's not required for Dedicated plan apps, which aren't dynamically scaled by Functions.

+Azure Files is set up by default for Elastic Premium and non-Linux Consumption plans to serve as a shared file system in high-scale scenarios. The file system is used by the platform for some features such as log streaming, but it primarily ensures consistency of the deployed function payload. When an app is [deployed using an external package URL](./run-functions-from-deployment-package.md), the app content is served from a separate read-only file system. This means that you can create your function app without Azure Files. If you create your function app with Azure Files, a writeable file system is still provided. However, this file system may not be available for all function app instances.

+Because Functions use Azure Files during parts of the dynamic scale-out process, scaling could be limited when running without Azure Files on Consumption and Elastic Premium plans.

+|Feature classes| A common blueprint for features. For example, a _unit_ is a feature class, and an _office_ is a feature. |

+- Produces a _Facility_ feature.

+The DWG file for each level must contain a layer to define that level's perimeter. This layer is referred to as the _exterior_ layer. For example, if a facility contains two levels, then it needs to have two DWG files, with an exterior layer for each file.

+- A _manifest.json_ file that describes the DWG files in the drawing package.

+- All data of a single level must be contained in a single DWG file. Any external references (_xrefs_) must be bound to the parent drawing.

+The file paths in the buildingLevels object of the manifest file must be relative to the root of the drawing package. The DWG file name must exactly match the name of the facility level. For example, a DWG file for the "Basement" level is _Basement.dwg_. A DWG file for level 2 is named as _level_2.dwg_. Filenames can't contain spaces, you can use an underscore to replace any spaces.

+| `version` | string | TRUE | Manifest schema version. Currently version "2.0" |

+|`featureClassProperties`| Array of [featureClassProperty] objects | FALSE | Specifies text layers in the DWG file associated to the feature as a property. For example, a label that falls inside the bounds of a space, such as a room number.|

+<a name="aerial-imagery"></a> **Aerial imagery**: See [Satellite imagery].

+<a name="ambiguous"></a> **Ambiguous**: A state of uncertainty in data classification that exists when an object may appropriately be assigned two or more values for a given attribute. For example, when geocoding "CA", two ambiguous results are returned: "Canada" and "California". "CA" is a country/region and a state code, for "Canada" and "California", respectively.

+<a name="api-key"></a> **API key**: See [Shared key authentication].

+<a name="autocomplete"></a> **Autocomplete**: A feature in an application that predicts the rest of a word a user is typing.

+<a name="azure-active-directory"></a> **Azure Active Directory (Azure AD)**: Azure AD is Microsoft's cloud-based identity and access management service. Azure Maps Azure AD integration is currently available in preview for all Azure Maps APIs. Azure AD supports Azure role-based access control (Azure RBAC) to allow fine-grained access to Azure Maps resources. To learn more about Azure Maps Azure AD integration, see [Azure Maps and Azure AD] and [Manage authentication in Azure Maps].

+<a name="azure-maps-key"></a> **Azure Maps key**: See [Shared key authentication].

+<a name="bearing"></a> **Bearing**: The horizontal direction of a point in relation to another point. This is expressed as an angle relative to north, from 0-degrees to 360 degrees in a clockwise direction.

+<a name="bounds"></a> **Bounds**: See [Bounding box].

+<a name="bounding-box"></a> **Bounding box**: A set of coordinates used to represent a rectangular area on the map.

+<a name="cadastre"></a> **Cadastre**: A record of registered land and properties. See also [Parcel].

+<a name="consumption-model"></a> **Consumption model**: Information that defines the rate at which a vehicle consumes fuel or electricity. Also see the [consumption model documentation].

+<a name="distance-matrix"></a> **Distance matrix**: A matrix that contains travel time and distance information between a set of origins and destinations.

+<a name="envelope"></a> **Envelope**: See [Bounding box].

+<a name="extent"></a> **Extent**: See [Bounding box].

+<a name="federated-authentication"></a> **Federated authentication**: An authentication method that allows a single logon/authentication mechanism to be used across multiple web and mobile apps.

+<a name="geojson"></a> **GeoJSON**: Is a common JSON-based file format used for storing geographical vector data such as points, lines, and polygons. For more information Azure Maps use of an extended version of GeoJSON, see [Extended geojson].

+<a name="heading"></a> **Heading**: The direction something is pointing or facing. See also [Bearing].

+<a name="isochrone"></a> **Isochrone**: An isochrone defines the area in which someone can travel within a specified time for a mode of transportation in any direction from a given location. See also [Reachable Range].

+<a name="isodistance"></a> **Isodistance**: Given a location, an isochrone defines the area in which someone can travel within a specified distance for a mode of transportation in any direction. See also [Reachable Range].

+<a name="kml"></a> **KML**: Also known as Keyhole Markup Language, is a common XML file format for storing geographic vector data such as points, lines, and polygons.

+<a name="map-tile"></a> **Map Tile**: A rectangular image that represents a partition of a map canvas. For more information, see [Zoom levels and tile grid].

+<a name="mercator-projection"></a> **Mercator projection**: A cylindrical map projection that became the standard map projection for nautical purposes because of its ability to represent lines of constant course, known as rhumb lines, as straight segments that conserve the angles with the meridians. All flat map projections distort the shapes or sizes of the map when compared to the true layout of the Earth's surface. The Mercator projection exaggerates areas far from the equator, such that smaller areas appear larger on the map as you approach the poles.

+<a name="post-code"></a> **Post code**: See [Postal code].

+<a name="primary-key"></a> **Primary key**: The first of two subscription keys provided for Azure Maps shared key authentication. See [Shared key authentication].

+<a name="quadkey"></a> **`Quadkey`**: A base-4 address index for a tile within a `quadtree` tiling system. For more information, see [Zoom levels and tile grid].

+<a name="quadtree"></a> **`Quadtree`**: A data structure in which each node has exactly four children. The tiling system used in Azure Maps uses a 'quadtree' structure such that as a user zooms in one level, each map tile breaks up into four subtiles. For more information, see [Zoom levels and tile grid].

+<a name="reachable-range"></a> **Reachable range**: A reachable range defines the area in which someone can travel within a specified time or distance, for a mode of transportation to travel, in any direction from a location. See also [Isochrone] and [Isodistance].

+<a name="reproject"></a> **Reproject**: See [Transformation].

+<a name="requests-per-second-rps"></a> **Requests Per Second (RPS)**: See [Queries Per Second (QPS)].

+<a name="secondary-key"></a> **Secondary key**: The second of two subscriptions keys provided for Azure Maps shared key authentication. See [Shared key authentication].

+<a name="shapefile-shp"></a> **Shapefile (SHP)**: Or _ESRI Shapefile_, is a vector data storage format for storing the location, shape, and attributes of geographic features. A shapefile is stored in a set of related files.

+<a name="shared-key-authentication"></a> **Shared key authentication**: Shared Key authentication relies on passing Azure Maps account generated keys with each request to Azure Maps. These keys are often referred to as subscription keys. It's recommended that keys are regularly regenerated for security. Two keys are provided so that you can maintain connections using one key while regenerating the other. When you regenerate your keys, you must update any applications that access this account to use the new keys. To learn more about Azure Maps authentication, see [Azure Maps and Azure AD] and [Manage authentication in Azure Maps].

+<a name="spherical-mercator-projection"></a> **Spherical Mercator projection**: See [Web Mercator].

+<a name="spatial-reference"></a> **Spatial reference**: A coordinate-based local, regional, or global system used to precisely locate geographical entities. It defines the coordinate system used to relate map coordinates to locations in the real world. Spatial references ensure spatial data from different layers, or sources, can be integrated for accurate viewing or analysis. Azure Maps uses the [EPSG:3857] coordinate reference system and WGS 84 for input geometry data.

+<a name="sql-spatial"></a> **SQL spatial**: Refers to the spatial functionality built into SQL Azure and SQL Server 2008 and above. This spatial functionality is also available as a .NET library that can be used independently of SQL Server. For more information, see [Spatial Data (SQL Server)].

+<a name="subscription-key"></a> **Subscription key**: See [Shared key authentication].

+<a name="transformation"></a> **Transformation**: The process of converting data between different geographic coordinate systems. You may, for example, have some data that was captured in the United Kingdom and based on the OSGB 1936 geographic coordinate system. Azure Maps uses the [EPSG:3857] coordinate reference system variant of WGS84. As such to display the data correctly, it needs to have its coordinates transformed from one system to another.

+<a name="vector-tile"></a> **Vector tile**: An open data specification for storing geospatial vector data using the same tile system as the map control. See also [Tile layer].

+<a name="voronoi-diagram"></a> **Voronoi diagram**: A partition of space into areas, or cells, that surrounds a set of geometric objects, usually point features. These cells, or polygons, must satisfy the criteria for Delaunay triangles. All locations within an area are closer to the object it surrounds than to any other object in the set. Voronoi diagrams are often used to delineate areas of influence around geographic features.

+<a name="waypoint-optimization"></a> **Waypoint optimization**: The process of reordering a set of waypoints to minimize the travel time or distance required to pass through all provided waypoints. Depending on the complexity of the optimization, this optimization is often referred to as the [Traveling Salesmen Problem] or [Vehicle Routing Problem].

+<a name="wgs84"></a> **WGS84**: A set of constants used to relate spatial coordinates to locations on the surface of the map. The WGS84 datum is the standard one used by most online mapping providers and GPS devices. Azure Maps uses the [EPSG:3857] coordinate reference system variant of WGS84.

+<a name="z-coordinate"></a> **Z-coordinate**: See [Altitude].

+<a name="zip-code"></a> **Zip code**: See [Postal code].

+<a name="Zoom level"></a> **Zoom level**: Specifies the level of detail and how much of the map is visible. When zoomed all the way to level 0, the full world map is often visible. But, the map shows limited details such as country/region names, borders, and ocean names. When zoomed in closer to level 17, the map displays an area of a few city blocks with detailed road information. In Azure Maps, the highest zoom level is 22. For more information, see [Zoom levels and tile grid].

+[Satellite imagery]: #satellite-imagery

+[Shared key authentication]: #shared-key-authentication

+[Azure Maps and Azure AD]: azure-maps-authentication.md

+[Manage authentication in Azure Maps]: how-to-manage-authentication.md

+[Bounding box]: #bounding-box

+[Parcel]: #parcel

+[consumption model documentation]: consumption-model.md

+[Extended geojson]: extend-geojson.md

+[Bearing]: #bearing

+[Reachable Range]: #reachable-range

+[Zoom levels and tile grid]: zoom-levels-and-tile-grid.md

+[Postal code]: #postal-code

+[Isochrone]: #isochrone

+[Isodistance]: #isodistance

+[Transformation]: #transformation

+[Queries Per Second (QPS)]: #queries-per-second-qps

+[EPSG:3857]: https://epsg.io/3857

+[Spatial Data (SQL Server)]: /sql/relational-databases/spatial/spatial-data-sql-server

+[Tile layer]: #tile-layer

+[Traveling Salesmen Problem]: #traveling-salesmen-problem-tsp

+[Vehicle Routing Problem]: #vehicle-routing-problem-vrp

+[Altitude]: #altitude

+[Web Mercator]: #web-mercator

+Be sure to complete the steps in the [Quickstart: Create an Android app] document. Code blocks in this article can be inserted into the maps `onReady` event handler.

+> [Create a data source]

+> [Use data-driven style expressions]

+> [Add a line layer]

+> [Add a polygon extrusion layer]

+[Add a line layer]: android-map-add-line-layer.md

+[Add a polygon extrusion layer]: map-extruded-polygon-android.md

+[Create a data source]: create-data-source-android-sdk.md

+[Quickstart: Create an Android app]: quick-android-map.md

+[Use data-driven style expressions]: data-driven-style-expressions-android-sdk.md

+Be sure to complete the steps in the [Quickstart: Create an Android app] document. Code blocks in this article can be inserted into the maps `onReady` event handler.

+For more information, see the [Create a data source] document on creating and adding data to the map.

+> [Create a data source]

+> [Cluster point data]

+> [Add a bubble layer]

+> [Use data-driven style expressions]

+> [Display feature information]

+[Add a bubble layer]: map-add-bubble-layer-android.md

+[Cluster point data]: clustering-point-data-android-sdk.md

+[Create a data source]: create-data-source-android-sdk.md

+[Display feature information]: display-feature-information-android.md

+[Quickstart: Create an Android app]: quick-android-map.md

+[Use data-driven style expressions]: data-driven-style-expressions-android-sdk.md

+This article shows you how to render a tile layer on a map using the Azure Maps Android SDK. Tile layers allow you to superimpose images on top of Azure Maps base map tiles. More information on Azure Maps tiling system can be found in the [Zoom levels and tile grid] documentation.

+* Bounding Box - Bounding box coordinates can be used to specify an image in the format `{west},{south},{east},{north}`, which is commonly used by [web-mapping Services (WMS)].

+To complete the process in this article, you need to install [Azure Maps Android SDK] to load a map.

+This sample shows how to create a tile layer that points to a set of tiles. This sample uses the "x, y, zoom" tiling system. The source of this tile layer is the [OpenSeaMap project], which contains crowd sourced nautical charts. Often when viewing tile layers it's desirable to be able to clearly see the labels of cities on the map. This behavior can be achieved by inserting the tile layer below the map label layers.

+The following screenshot shows the above code overlaying a web-mapping service of geological data from the [U.S. Geological Survey (USGS)] on top of a map, below the labels.

+> [Image layer]

+[Azure Maps Android SDK]: how-to-use-android-map-control-library.md

+[Image layer]: map-add-image-layer-android.md

+[OpenSeaMap project]: https://openseamap.org/index.php

+[U.S. Geological Survey (USGS)]: https://mrdata.usgs.gov

+[web-mapping Services (WMS)]: https://www.opengeospatial.org/standards/wms

+[Zoom levels and tile grid]: zoom-levels-and-tile-grid.md

+Your classic resource data persists and is subject to the retention settings on your classic Application Insights resource. All new data ingested post migration is subject to the [retention settings](../logs/data-retention-archive.md) of the associated Log Analytics workspace, which also supports [different retention settings by data type](../logs/data-retention-archive.md#configure-retention-and-archive-at-the-table-level).

+ > - If you currently store Application Insights data for longer than the default 90 days and want to retain this longer retention period after migration, adjust your [workspace retention settings](../logs/data-retention-archive.md?tabs=portal-1%2cportal-2#configure-retention-and-archive-at-the-table-level).

+Starting with version 8.0 or higher of [Azure PowerShell](/powershell/azure/what-is-azure-powershell), you can use the `Update-AzApplicationInsights` PowerShell command to migrate a classic Application Insights resource to workspace based.

+To use this cmdlet, you need to specify the name and resource group of the Application Insights resource that you want to update. Use the `IngestionMode` and `WorkspaceResoruceId` parameters to migrate your classic instance to workspace-based. For more information on the parameters and syntax of this cmdlet, see [Update-AzApplicationInsights](/powershell/module/az.applicationinsights/update-azapplicationinsights).

+### Is there any impact on Live Metrics or other monitoring experiences?

+Raw data points (that is, items that you can query in Analytics and inspect in Search) are kept for up to 730 days. You can [select a retention duration](../logs/data-retention-archive.md#configure-retention-and-archive-at-the-table-level) of 30, 60, 90, 120, 180, 270, 365, 550, or 730 days. If you need to keep data longer than 730 days, you can use [diagnostic settings](../essentials/diagnostic-settings.md#diagnostic-settings-in-azure-monitor).

+ Title: Add, modify, and filter Azure Monitor OpenTelemetry for .NET, Java, Node.js, and Python applications

+description: This article provides guidance on how to add, modify, and filter OpenTelemetry for applications using Azure Monitor.

+# Add, modify, and filter OpenTelemetry

+This article provides guidance on how to add, modify, and filter OpenTelemetry for applications using [Azure Monitor Application Insights](app-insights-overview.md#application-insights-overview).

+## Filter telemetry

+### [ASP.NET Core](#tab/aspnetcore)

+### [.NET](#tab/net)

+### [Java](#tab/java)

+### [Node.js](#tab/nodejs)

+### [Python](#tab/python)

+## Get the trace ID or span ID

+### [ASP.NET Core](#tab/aspnetcore)

+### [.NET](#tab/net)

+### [Java](#tab/java)

+### [Node.js](#tab/nodejs)

+### [Python](#tab/python)

+ `InstrumentationKey` is a *required* field.

+ Setting the endpoint suffix tells the SDK which Azure cloud to connect to. The SDK assembles the rest of the endpoint for individual services.

+In this example, the connection string specifies the endpoint suffix and the SDK constructs service endpoints:

+In this example, the connection string specifies explicit overrides for every service. The SDK uses the exact endpoints provided without modification:

+| Change to Workspace-based Application Insights | Ensure that your Application Insights resources are [Workspace-based](app/create-workspace-resource.md) so that they can leverage new cost savings tools such as [Basic Logs](logs/basic-logs-configure.md), [Commitment Tiers](logs/cost-logs.md#commitment-tiers), [Retention by data type and Data Archive](logs/data-retention-archive.md#configure-retention-and-archive-at-the-table-level). |

+The following examples requires the configuration described in [Send Prometheus metrics to Log Analytics workspace with Container insights](container-insights-prometheus-logs.md).

+

+

+ Title: Collect Prometheus metrics with Container insights

+description: Describes different methods for configuring the Container insights agent to scrape Prometheus metrics from your Kubernetes cluster.

+# Send Prometheus metrics to Log Analytics workspace with Container insights

+This article describes how to send Prometheus metrics from your Kubernetes cluster monitored by Container insights to a Log Analytics workspace. Before you perform this configuration, you should first ensure that you're [scraping Prometheus metrics from your cluster using Azure Monitor managed service for Prometheus](), which is the recommended method for monitoring your clusters. Use the configuration described in this article only if you also want to send this same data to a Log Analytics workspace where you can analyze it using [log queries](../logs/log-query-overview.md) and [log alerts](../alerts/alerts-log-query.md).

+This configuration requires configuring the *monitoring addon* for the Azure Monitor agent, which is the same one used by Container insights to send data to a Log Analytics workspace. It requires exposing the Prometheus metrics endpoint through your exporters or pods and then configuring the monitoring addon for the Azure Monitor agent used by Container insights as shown the following diagram.

+## Prometheus scraping settings (for metrics stored as logs)

+Active scraping of metrics from Prometheus is performed from one of two perspectives below and metrics are sent to configured log analytics workspace :

+- **Cluster-wide**: Defined in the ConfigMap section *[Prometheus data_collection_settings.cluster]*.

+- **Node-wide**: Defined in the ConfigMap section *[Prometheus_data_collection_settings.node]*.

+| Endpoint | Scope | Example |

+|-|-||

+| Pod annotation | Cluster-wide | `prometheus.io/scrape: "true"` <br>`prometheus.io/path: "/mymetrics"` <br>`prometheus.io/port: "8000"` <br>`prometheus.io/scheme: "http"` |

+| Kubernetes service | Cluster-wide | `http://my-service-dns.my-namespace:9100/metrics` <br>`http://metrics-server.kube-system.svc.cluster.local/metrics`ΓÇï |

+| URL/endpoint | Per-node and/or cluster-wide | `http://myurl:9101/metrics` |

+When a URL is specified, Container insights only scrapes the endpoint. When Kubernetes service is specified, the service name is resolved with the cluster DNS server to get the IP address. Then the resolved service is scraped.

+|Scope | Key | Data type | Value | Description |

+||--|--|-|-|

+| Cluster-wide | | | | Specify any one of the following three methods to scrape endpoints for metrics. |

+| | `urls` | String | Comma-separated array | HTTP endpoint (either IP address or valid URL path specified). For example: `urls=[$NODE_IP/metrics]`. ($NODE_IP is a specific Container insights parameter and can be used instead of a node IP address. Must be all uppercase.) |

+| | `kubernetes_services` | String | Comma-separated array | An array of Kubernetes services to scrape metrics from kube-state-metrics. Fully qualified domain names must be used here. For example,`kubernetes_services = ["http://metrics-server.kube-system.svc.cluster.local/metrics",http://my-service-dns.my-namespace.svc.cluster.local:9100/metrics]`|

+| | `monitor_kubernetes_pods` | Boolean | true or false | When set to `true` in the cluster-wide settings, the Container insights agent scrapes Kubernetes pods across the entire cluster for the following Prometheus annotations:<br> `prometheus.io/scrape:`<br> `prometheus.io/scheme:`<br> `prometheus.io/path:`<br> `prometheus.io/port:` |

+| | `prometheus.io/scrape` | Boolean | true or false | Enables scraping of the pod, and `monitor_kubernetes_pods` must be set to `true`. |

+| | `prometheus.io/scheme` | String | http | Defaults to scraping over HTTP. |

+| | `prometheus.io/path` | String | Comma-separated array | The HTTP resource path from which to fetch metrics. If the metrics path isn't `/metrics`, define it with this annotation. |

+| | `prometheus.io/port` | String | 9102 | Specify a port to scrape from. If the port isn't set, it defaults to 9102. |

+| | `monitor_kubernetes_pods_namespaces` | String | Comma-separated array | An allowlist of namespaces to scrape metrics from Kubernetes pods.<br> For example, `monitor_kubernetes_pods_namespaces = ["default1", "default2", "default3"]` |

+| Node-wide | `urls` | String | Comma-separated array | HTTP endpoint (either IP address or valid URL path specified). For example: `urls=[$NODE_IP/metrics]`. ($NODE_IP is a specific Container insights parameter and can be used instead of a node IP address. Must be all uppercase.) |

+| Node-wide or cluster-wide | `interval` | String | 60s | The collection interval default is one minute (60 seconds). You can modify the collection for either the *[prometheus_data_collection_settings.node]* and/or *[prometheus_data_collection_settings.cluster]* to time units such as s, m, and h. |

+| Node-wide or cluster-wide | `fieldpass`<br> `fielddrop`| String | Comma-separated array | You can specify certain metrics to be collected or not from the endpoint by setting the allow (`fieldpass`) and disallow (`fielddrop`) listing. You must set the allowlist first. |

+## Configure ConfigMaps to specify Prometheus scrape configuration (for metrics stored as logs)

+Perform the following steps to configure your ConfigMap configuration file for your cluster. ConfigMaps is a global list and there can be only one ConfigMap applied to the agent. You can't have another ConfigMaps overruling the collections.

+1. [Download](https://aka.ms/container-azm-ms-agentconfig) the template ConfigMap YAML file and save it as c*ontainer-azm-ms-agentconfig.yaml*. If you've already deployed a ConfigMap to your cluster and you want to update it with a newer configuration, you can edit the ConfigMap file you've previously used.

+1. Edit the ConfigMap YAML file with your customizations to scrape Prometheus metrics.

+ ### [Cluster-wide](#tab/cluster-wide)

+ To collect Kubernetes services cluster-wide, configure the ConfigMap file by using the following example:

+ ```

+ prometheus-data-collection-settings: |- ΓÇï

+ # Custom Prometheus metrics data collection settings

+ [prometheus_data_collection_settings.cluster] ΓÇï

+ interval = "1m" ## Valid time units are s, m, h.

+ fieldpass = ["metric_to_pass1", "metric_to_pass12"] ## specify metrics to pass through ΓÇï

+ fielddrop = ["metric_to_drop"] ## specify metrics to drop from collecting

+ kubernetes_services = ["http://my-service-dns.my-namespace:9102/metrics"]

+ ```

+ ### [Specific URL](#tab/url)

+ To configure scraping of Prometheus metrics from a specific URL across the cluster, configure the ConfigMap file by using the following example:

+ ```

+ prometheus-data-collection-settings: |- ΓÇï

+ # Custom Prometheus metrics data collection settings

+ [prometheus_data_collection_settings.cluster] ΓÇï

+ interval = "1m" ## Valid time units are s, m, h.

+ fieldpass = ["metric_to_pass1", "metric_to_pass12"] ## specify metrics to pass through ΓÇï

+ fielddrop = ["metric_to_drop"] ## specify metrics to drop from collecting

+ urls = ["http://myurl:9101/metrics"] ## An array of urls to scrape metrics from

+ ```

+ ### [DaemonSet](#tab/deamonset)

+ To configure scraping of Prometheus metrics from an agent's DaemonSet for every individual node in the cluster, configure the following example in the ConfigMap:

+ ```

+ prometheus-data-collection-settings: |- ΓÇï

+ # Custom Prometheus metrics data collection settings ΓÇï

+ [prometheus_data_collection_settings.node] ΓÇï

+ interval = "1m" ## Valid time units are s, m, h.

+ urls = ["http://$NODE_IP:9103/metrics"] ΓÇï

+ fieldpass = ["metric_to_pass1", "metric_to_pass2"] ΓÇï

+ fielddrop = ["metric_to_drop"] ΓÇï

+ ```

+ `$NODE_IP` is a specific Container insights parameter and can be used instead of a node IP address. It must be all uppercase.

+ ### [Pod annotation](#tab/pod)

+ To configure scraping of Prometheus metrics by specifying a pod annotation:

+ 1. In the ConfigMap, specify the following configuration:

+ ```

+ prometheus-data-collection-settings: |- ΓÇï

+ # Custom Prometheus metrics data collection settings

+ [prometheus_data_collection_settings.cluster] ΓÇï

+ interval = "1m" ## Valid time units are s, m, h

+ monitor_kubernetes_pods = true

+ ```

+ 2. Specify the following configuration for pod annotations:

+ ```

+ - prometheus.io/scrape:"true" #Enable scraping for this pod ΓÇï

+ - prometheus.io/scheme:"http" #If the metrics endpoint is secured then you will need to set this to `https`, if not default ΓÇÿhttpΓÇÖΓÇï

+ - prometheus.io/path:"/mymetrics" #If the metrics path is not /metrics, define it with this annotation. ΓÇï

+ - prometheus.io/port:"8000" #If port is not 9102 use this annotationΓÇï

+ ```

+

+ If you want to restrict monitoring to specific namespaces for pods that have annotations, for example, only include pods dedicated for production workloads, set the `monitor_kubernetes_pod` to `true` in ConfigMap. Then add the namespace filter `monitor_kubernetes_pods_namespaces` to specify the namespaces to scrape from. An example is `monitor_kubernetes_pods_namespaces = ["default1", "default2", "default3"]`.

+2. Run the following kubectl command: `kubectl apply -f <configmap_yaml_file.yaml>`.

+

+ Example: `kubectl apply -f container-azm-ms-agentconfig.yaml`.

+The configuration change can take a few minutes to finish before taking effect. All ama-logs pods in the cluster will restart. When the restarts are finished, a message appears that's similar to the following and includes the result `configmap "container-azm-ms-agentconfig" created`.

+## Verify configuration

+To verify the configuration was successfully applied to a cluster, use the following command to review the logs from an agent pod: `kubectl logs ama-logs-fdf58 -n=kube-system`.

+If there are configuration errors from the Azure Monitor Agent pods, the output shows errors similar to the following example:

+```

+***************Start Config Processing********************

+config::unsupported/missing config schema version - 'v21' , using defaults

+```

+Errors related to applying configuration changes are also available for review. The following options are available to perform additional troubleshooting of configuration changes and scraping of Prometheus metrics:

+- From an agent pod logs using the same `kubectl logs` command.

+- From Live Data. Live Data logs show errors similar to the following example:

+ ```

+ 2019-07-08T18:55:00Z E! [inputs.prometheus]: Error in plugin: error making HTTP request to http://invalidurl:1010/metrics: Get http://invalidurl:1010/metrics: dial tcp: lookup invalidurl on 10.0.0.10:53: no such host

+ ```

+- From the **KubeMonAgentEvents** table in your Log Analytics workspace. Data is sent every hour with *Warning* severity for scrape errors and *Error* severity for configuration errors. If there are no errors, the entry in the table has data with severity *Info*, which reports no errors. The **Tags** property contains more information about the pod and container ID on which the error occurred and also the first occurrence, last occurrence, and count in the last hour.

+- For Azure Red Hat OpenShift v3.x and v4.x, check the Azure Monitor Agent logs by searching the **ContainerLog** table to verify if log collection of openshift-azure-logging is enabled.

+Errors prevent Azure Monitor Agent from parsing the file, causing it to restart and use the default configuration. After you correct the errors in ConfigMap on clusters other than Azure Red Hat OpenShift v3.x, save the YAML file and apply the updated ConfigMaps by running the command `kubectl apply -f <configmap_yaml_file.yaml`.

+For Azure Red Hat OpenShift v3.x, edit and save the updated ConfigMaps by running the command `oc edit configmaps container-azm-ms-agentconfig -n openshift-azure-logging`.

+## Query Prometheus metrics data

+To view Prometheus metrics scraped by Azure Monitor and any configuration/scraping errors reported by the agent, review [Query Prometheus metrics data](container-insights-log-query.md#prometheus-metrics).

+## View Prometheus metrics in Grafana

+Container insights supports viewing metrics stored in your Log Analytics workspace in Grafana dashboards. We've provided a template that you can download from Grafana's [dashboard repository](https://grafana.com/grafana/dashboards?dataSource=grafana-azure-monitor-datasource&category=docker). Use the template to get started and reference it to help you learn how to query other data from your monitored clusters to visualize in custom Grafana dashboards.

+## Next steps

+- [See the default configuration for Prometheus metrics](../essentials/prometheus-metrics-scrape-default.md).

+- [Customize Prometheus metric scraping for the cluster](../essentials/prometheus-metrics-scrape-configuration.md).

+ Title: Integrate KEDA with your Azure Kubernetes Service cluster

+description: How to integrate KEDA with your Azure Kubernetes Service cluster.

+

+# Integrate KEDA with your Azure Kubernetes Service cluster

+KEDA is a Kubernetes-based Event Driven Autoscaler. KEDA lets you drive the scaling of any container in Kubernetes based on the load to be processed, by querying metrics from systems such as Prometheus. Integrate KEDA with your Azure Kubernetes Service (AKS) cluster to scale your workloads based on Prometheus metrics from your Azure Monitor workspace.

+To integrate KEDA into your Azure Kubernetes Service, you have to deploy and configure a workload identity or pod identity on your cluster. The identity allows KEDA to authenticate with Azure and retrieve metrics for scaling from your Monitor workspace.

+This article walks you through the steps to integrate KEDA into your AKS cluster using a workload identity.

+> [!NOTE]

+> We recommend using Azure Active Directory workload identity. This authentication method replaces pod-managed identity (preview), which integrates with the Kubernetes native capabilities to federate with any external identity providers on behalf of the application.

+>

+> The open source Azure AD pod-managed identity (preview) in Azure Kubernetes Service has been deprecated as of 10/24/2022, and the project will be archived in Sept. 2023. For more information, see the deprecation notice. The AKS Managed add-on begins deprecation in Sept. 2023.

+>

+> Azure Managed Prometheus support starts from KEDA v2.10. If you have an older version of KEDA installed, you must upgrade in order to work with Azure Managed Prometheus.

+## Prerequisites

+## Set up a workload identity

+1. Start by setting up some environment variables. Change the values to suit your AKS cluster.

+

+ ```bash

+ export RESOURCE_GROUP="rg-keda-integration"

+ export LOCATION="eastus"

+ export SUBSCRIPTION="$(az account show --query id --output tsv)"

+ export USER_ASSIGNED_IDENTITY_NAME="keda-int-identity"

+ export FEDERATED_IDENTITY_CREDENTIAL_NAME="kedaFedIdentity"

+ export SERVICE_ACCOUNT_NAMESPACE="keda"

+ export SERVICE_ACCOUNT_NAME="keda-operator"

+ export AKS_CLUSTER_NAME="aks-cluster-name"

+ ```

+ + `SERVICE_ACCOUNT_NAME` - KEDA must use the service account that was used to create federated credentials. This can be any user defined name.

+ + `AKS_CLUSTER_NAME`- The name of the AKS cluster where you want to deploy KEDA.

+ + `SERVICE_ACCOUNT_NAMESPACE` Both KEDA and service account must be in same namespace.

+ + `USER_ASSIGNED_IDENTITY_NAME` is the name of the Azure Active directory identity that's created for KEDA.

+ + `FEDERATED_IDENTITY_CREDENTIAL_NAME` is the name of the credential that's created for KEDA to use to authenticate with Azure.

+1. If your AKS cluster hasn't been created with workload-identity or oidc-issuer enabled, you'll need to enable it. If you aren't sure, you can run the following command to check if it's enabled.

+ ```azurecli

+ az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query oidcIssuerProfile

+ az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query securityProfile.workloadIdentity

+ ```

+

+ To enable workload identity and oidc-issuer, run the following command.

+

+ ```azurecli

+ az aks update -g $RESOURCE_GROUP -n $AKS_CLUSTER_NAME --enable-managed-identity --enable-oidc-issuer

+ ```

+

+1. Store the OIDC issuer url in an environment variable to be used later.

+

+ ```bash

+ export AKS_OIDC_ISSUER="$(az aks show -n $AKS_CLUSTER_NAME -g $RESOURCE_GROUP --query "oidcIssuerProfile.issuerUrl" -otsv)"

+ ```

+

+1. Create a user assigned identity for KEDA. This identity is used by KEDA to authenticate with Azure Monitor.

+

+ ```azurecli

+ az identity create --name $USER_ASSIGNED_IDENTITY_NAME --resource-group $RESOURCE_GROUP --location $LOCATION --subscription $SUBSCRIPTION

+ ```

+

+ The output will be similar to the following:

+

+ ```json

+ {

+ "clientId": "abcd1234-abcd-abcd-abcd-9876543210ab",

+ "id": "/subscriptions/abcdef01-2345-6789-0abc-def012345678/resourcegroups/rg-keda-integration/providers/Microsoft. ManagedIdentity/userAssignedIdentities/keda-int-identity",

+ "location": "eastus",

+ "name": "keda-int-identity",

+ "principalId": "12345678-abcd-abcd-abcd-1234567890ab",

+ "resourceGroup": "rg-keda-integration",

+ "systemData": null,

+ "tags": {},

+ "tenantId": "1234abcd-9876-9876-9876-abcdef012345",

+ "type": "Microsoft.ManagedIdentity/userAssignedIdentities"

+ }

+ ```

+1. Store the `clientId` and `tenantId` in environment variables to use later.

+ ```bash

+ export USER_ASSIGNED_CLIENT_ID="$(az identity show --resource-group $RESOURCE_GROUP --name $USER_ASSIGNED_IDENTITY_NAME --query 'clientId' -otsv)"

+ export TENANT_ID="$(az identity show --resource-group $RESOURCE_GROUP --name $USER_ASSIGNED_IDENTITY_NAME --query 'tenantId' -otsv)"

+ ```

+

+1. Assign the *Monitoring Data Reader* role to the identity for your Azure Monitor workspace. This role allows the identity to read metrics from your workspace. Replace the *Azure Monitor Workspace resource group* and *Azure Monitor Workspace name* with the resource group and name of the Azure Monitor workspace which is configured to collect metrics from the AKS cluster.

+

+ ```azurecli

+ az role assignment create \

+ --assignee $USER_ASSIGNED_CLIENT_ID \

+ --role "Monitoring Data Reader" \

+ --scope /subscriptions/$SUBSCRIPTION/resourceGroups/<Azure Monitor Workspace resource group>/providers/microsoft.monitor/accounts/<Azure monitor workspace name>

+ ```

+

+

+1. Create the KEDA namespace, then create Kubernetes service account. This service account is used by KEDA to authenticate with Azure.

+

+ ```azurecli

+

+ az aks get-credentials -n $AKS_CLUSTER_NAME -g $RESOURCE_GROUP

+

+ kubectl create namespace keda

+

+ cat <<EOF | kubectl apply -f -

+ apiVersion: v1

+ kind: ServiceAccount

+ metadata:

+ annotations:

+ azure.workload.identity/client-id: $USER_ASSIGNED_CLIENT_ID

+ name: $SERVICE_ACCOUNT_NAME

+ namespace: $SERVICE_ACCOUNT_NAMESPACE

+ EOF

+ ```

+1. Check your service account by running

+ ```bash

+ kubectl describe serviceaccount $SERVICE_ACCOUNT_NAME -n keda

+ ```

+1. Establish a federated credential between the service account and the user assigned identity. The federated credential allows the service account to use the user assigned identity to authenticate with Azure.

+ ```azurecli

+ az identity federated-credential create --name $FEDERATED_IDENTITY_CREDENTIAL_NAME --identity-name $USER_ASSIGNED_IDENTITY_NAME --resource-group $RESOURCE_GROUP --issuer $AKS_OIDC_ISSUER --subject system:serviceaccount:$SERVICE_ACCOUNT_NAMESPACE:$SERVICE_ACCOUNT_NAME --audience api://AzureADTokenExchange

+ ```

+ > [!Note]

+ > It takes a few seconds for the federated identity credential to be propagated after being initially added. If a token request is made immediately after adding the federated identity credential, it might lead to failure for a couple of minutes as the cache is populated in the directory with old data. To avoid this issue, you can add a slight delay after adding the federated identity credential.

+## Deploy KEDA

+KEDA can be deployed using YAML manifests, Helm charts, or Operator Hub. This article uses Helm charts. For more information on deploying KEDA, see [Deploying KEDA](https://keda.sh/docs/2.10/deploy/)

+Add helm repository:

+```bash

+helm repo add kedacore https://kedacore.github.io/charts

+helm repo update

+```

+Deploy KEDA using the following command:

+```bash

+helm install keda kedacore/keda --namespace keda \

+--set podIdentity.azureWorkload.enabled=true \

+--set podIdentity.azureWorkload.clientId=$USER_ASSIGNED_CLIENT_ID \

+--set podIdentity.azureWorkload.tenantId=$TENANT_ID

+```

+

+Check your deployment by running the following command.

+```bash

+kubectl get pods -n keda

+```

+The output will be similar to the following:

+```bash

+NAME READY STATUS RESTARTS AGE

+keda-admission-webhooks-ffcb8f688-kqlxp 1/1 Running 0 4m

+keda-operator-5d9f7d975-mgv7r 1/1 Running 1 (4m ago) 4m

+keda-operator-metrics-apiserver-7dc6f59678-745nz 1/1 Running 0 4m

+```

+## Scalers

+Scalers define how and when KEDA should scale a deployment. KEDA supports a variety of scalers. For more information on scalers, see [Scalers](https://keda.sh/docs/2.10/scalers/prometheus/). Azure Managed Prometheus utilizes already existing Prometheus scaler to retrieve Prometheus metrics from Azure Monitor Workspace. The following yaml file is an example to use Azure Managed Prometheus.

+```yml

+apiVersion: keda.sh/v1alpha1

+kind: TriggerAuthentication

+metadata:

+ name: azure-managed-prometheus-trigger-auth

+spec:

+ podIdentity:

+ provider: azure-workload | azure # use "azure" for pod identity and "azure-workload" for workload identity

+ identityId: <identity-id> # Optional. Default: Identity linked with the label set when installing KEDA.

+apiVersion: keda.sh/v1alpha1

+kind: ScaledObject

+metadata:

+ name: azure-managed-prometheus-scaler

+spec:

+ scaleTargetRef:

+ name: deployment-name-to-be-scaled

+ minReplicaCount: 1

+ maxReplicaCount: 20

+ triggers:

+ - type: prometheus

+ metadata:

+ serverAddress: https://test-azure-monitor-workspace-name-1234.eastus.prometheus.monitor.azure.com

+ metricName: http_requests_total

+ query: sum(rate(http_requests_total{deployment="my-deployment"}[2m])) # Note: query must return a vector/scalar single element response

+ threshold: '100.50'

+ activationThreshold: '5.5'

+ authenticationRef:

+ name: azure-managed-prometheus-trigger-auth

+```

+## Troubleshooting

+The following section provides troubleshooting tips for common issues.

+### Federated credentials

+Federated credentials can take up to 10 minutes to propagate. If you're having issues with KEDA authenticating with Azure, try the following steps.

+The following log excerpt shows an error with the federated credentials.

+```

+kubectl logs -n keda keda-operator-5d9f7d975-mgv7r

+{

+ \"error\": \"unauthorized_client\",\n \"error_description\": \"AADSTS70021: No matching federated identity record found for presented assertion.

+Assertion Issuer: 'https://eastus.oic.prod-aks.azure.com/abcdef01-2345-6789-0abc-def012345678/12345678-abcd-abcd-abcd-1234567890ab/'.

+Assertion Subject: 'system:serviceaccount:keda:keda-operator'.

+Assertion Audience: 'api://AzureADTokenExchange'. https://docs.microsoft.com/azure/active-directory/develop/workload-identity-federation

+Trace ID: 12dd9ea0-3a65-408f-a41f-5d0403a25100\\r\\nCorrelation ID: 8a2dce68-17f1-4f11-bed2-4bcf9577f2af\\r\\nTimestamp: 2023-05-30 11:11:53Z\",

+\"error_codes\": [\n 70021\n ],\n \"timestamp\": \"2023-05-30 11:11:53Z\",

+\"trace_id\": \"12345678-3a65-408f-a41f-5d0403a25100\",

+\"correlation_id\": \"12345678-17f1-4f11-bed2-4bcf9577f2af\",

+\"error_uri\": \"https://login.microsoftonline.com/error?code=70021\"\n}

+\n--\n"}

+```

+Check the values used to create the ServiceAccount and the credentials created with `az identity federated-credential create` and ensure the `subject` value matches the `system:serviceaccount` value.

+### Azure Monitor workspace permissions

+If you're having issues with KEDA authenticating with Azure, check the permissions for the Azure Monitor workspace.

+The following log excerpt shows that the identity doesn't have read permissions for the Azure Monitor workspace.

+```

+kubectl logs -n keda keda-operator-5d9f7d975-mgv7r

+2023-05-30T11:15:45Z ERROR scale_handler error getting metric for scaler

+{"scaledObject.Namespace": "default", "scaledObject.Name": "azure-managed-prometheus-scaler", "scaler": "prometheusScaler",

+"error": "prometheus query api returned error. status: 403 response: {\"status\":\"error\",

+\"errorType\":\"Forbidden\",\"error\":\"User \\u0027abc123ab-1234-1234-abcd-abcdef123456

+\\u0027 does not have access to perform any of the following actions

+\\u0027microsoft.monitor/accounts/data/metrics/read, microsoft.monitor/accounts/data/metrics/read

+\\u0027 on resource \\u0027/subscriptions/abcdef01-2345-6789-0abc-def012345678/resourcegroups/rg-azmon-ws-01/providers/microsoft.monitor/accounts/azmon-ws-01\\u0027. RequestId: 123456c427f348258f3e5aeeefef834a\"}"}

+```

+Ensure the identity has the `Monitoring Data Reader` role on the Azure Monitor workspace.

+ Title: Azure Active Directory authorization proxy

+description: Azure Active Directory authorization proxy

+# Azure Active Directory authorization proxy

+The Azure Active Directory authorization proxy is a reverse proxy, which can be used to authenticate requests using Azure Active Directory. This proxy can be used to authenticate requests to any service that supports Azure Active Directory authentication. Use this proxy to authenticate requests to Azure Monitor managed service for Prometheus.

+## Prerequisites

+> [!NOTE]

+> The remote write example in this article uses Prometheus remote write to write data to Azure Monitor. Onboarding your AKS cluster to Prometheus automatically installs Prometheus on your cluster and sends data to your workspace.

+## Deployment

+The proxy can be deployed with custom templates using release image or as a helm chart. Both deployments contain the same customizable parameters. These parameters are described in the [Parameters](#parameters) table.

+For for more information, see [Azure Active Directory authentication proxy](https://github.com/Azure/aad-auth-proxy) project.

+The following examples show how to deploy the proxy for remote write and for querying data from Azure Monitor.

+## [Remote write example](#tab/remote-write-example)

+> [!NOTE]

+> This example shows how to use the proxy to authenticate requests for remote write to an Azure Monitor managed service for Prometheus. Prometheus remote write has a dedicated side car for remote writing which is the recommended method for implementing remote write.

+Before deploying the proxy, find your managed identity and assign it the `Monitoring Metrics Publisher` role for the Azure Monitor workspace's data collection rule.

+1. Find the `clientId` for the managed identity for your AKS cluster. The managed identity is used to authenticate to the Azure Monitor workspace. The managed identity is created when the AKS cluster is created.

+ ```azurecli

+ # Get the identity client_id

+ az aks show -g <AKS-CLUSTER-RESOURCE-GROUP> -n <AKS-CLUSTER-NAME> --query "identityProfile"

+ ```

+ The output has the following format:

+ ```bash

+ {

+ "kubeletidentity": {

+ "clientId": "abcd1234-1243-abcd-9876-1234abcd5678",

+ "objectId": "12345678-abcd-abcd-abcd-1234567890ab",

+ "resourceId": "/subscriptions/def0123-1243-abcd-9876-1234abcd5678/resourcegroups/MC_rg-proxytest-01_proxytest-01_eastus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/proxytest-01-agentpool"

+ }

+ ```

+1. Find your Azure Monitor workspace's data collection rule (DCR) ID.

+ The rule name is same as the workspace name.

+ The resource group name for your data collection rule follows the format: `MA_<workspace-name>_<REGION>_managed`, for example `MA_amw-proxytest_eastus_managed`. Use the following command to find the data collection rule ID:

+ ```azurecli

+ az monitor data-collection rule show --name <dcr-name> --resource-group <resource-group-name> --query "id"

+ ```

+1. Alternatively you can find your DCR ID and Metrics ingestion endpoint using the Azure portal on the Azure Monitor workspace Overview page.

+ Select the **Data collection rule** on the workspace Overview tab, then select **JSON view** to see the **Resource ID**.

+

+ :::image type="content" source="./media/prometheus-authorization-proxy/workspace-overview.png" lightbox="./media/prometheus-authorization-proxy/workspace-overview.png" alt-text="A screenshot showing the overview page for an Azure Monitor workspace.":::

+1. Assign the `Monitoring Metrics Publisher` role to the managed identity's `clientId` so that it can write to the Azure Monitor workspace data collection rule.

+ ```azurecli

+ az role assignment create /

+ --assignee <clientid> /

+ --role "Monitoring Metrics Publisher" /

+ --scope <workspace-dcr-id>

+ ```

+ For example:

+ ```bash

+ az role assignment create \

+ --assignee abcd1234-1243-abcd-9876-1234abcd5678 \

+ --role "Monitoring Metrics Publisher" \

+ --scope /subscriptions/ef0123-1243-abcd-9876-1234abcd5678/resourceGroups/MA_amw-proxytest_eastus_managed/providers/Microsoft.Insights/dataCollectionRules/amw-proxytest

+ ```

+1. Use the following YAML file to deploy the proxy for remote write. Modify the following parameters:

+ + `TARGET_HOST` - The target host where you want to forward the request to. To send data to an Azure Monitor workspace, use the hostname part of the `Metrics ingestion endpoint` from the workspaces Overview page. For example, `http://amw-proxytest-abcd.eastus-1.metrics.ingest.monitor.azure.com`

+ + `AAD_CLIENT_ID` - The `clientId` of the managed identity used that was assigned the `Monitoring Metrics Publisher` role.

+ + `AUDIENCE` - For ingesting metrics to Azure Monitor Workspace, set `AUDIENCE` to `https://monitor.azure.com/.default` .

+ + Remove `OTEL_GRPC_ENDPOINT` and `OTEL_SERVICE_NAME` if you aren't using OpenTelemetry.

+ For more information about the parameters, see the [Parameters](#parameters) table.

+ proxy-ingestion.yaml

+ ```yml

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ labels:

+ app: azuremonitor-ingestion

+ name: azuremonitor-ingestion

+ namespace: observability

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: azuremonitor-ingestion

+ template:

+ metadata:

+ labels:

+ app: azuremonitor-ingestion

+ name: azuremonitor-ingestion

+ spec:

+ containers:

+ - name: aad-auth-proxy

+ image: mcr.microsoft.com/azuremonitor/auth-proxy/prod/aad-auth-proxy/images/aad-auth-proxy:0.1.0-main-05-24-2023-b911fe1c

+ imagePullPolicy: Always

+ ports:

+ - name: auth-port

+ containerPort: 8081

+ env:

+ - name: AUDIENCE

+ value: https://monitor.azure.com/.default

+ - name: TARGET_HOST

+ value: http://<workspace-endpoint-hostname>

+ - name: LISTENING_PORT

+ value: "8081"

+ - name: IDENTITY_TYPE

+ value: userAssigned

+ - name: AAD_CLIENT_ID

+ value: <clientId>

+ - name: AAD_TOKEN_REFRESH_INTERVAL_IN_PERCENTAGE

+ value: "10"

+ - name: OTEL_GRPC_ENDPOINT

+ value: <YOUR-OTEL-GRPC-ENDPOINT> # "otel-collector.observability.svc.cluster.local:4317"

+ - name: OTEL_SERVICE_NAME

+ value: <YOUE-SERVICE-NAME>

+ livenessProbe:

+ httpGet:

+ path: /health

+ port: auth-port

+ initialDelaySeconds: 5

+ timeoutSeconds: 5

+ readinessProbe:

+ httpGet:

+ path: /ready

+ port: auth-port

+ initialDelaySeconds: 5

+ timeoutSeconds: 5

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: azuremonitor-ingestion

+ namespace: observability

+ spec:

+ ports:

+ - port: 80

+ targetPort: 8081

+ selector:

+ app: azuremonitor-ingestion

+ ```

+ 1. Deploy the proxy using commands:

+ ```bash

+ # create the namespace if it doesn't already exist

+ kubectl create namespace observability

+ kubectl apply -f proxy-ingestion.yaml -n observability

+ ```

+1. Alternatively you can deploy the proxy using helm as follows:

+ ```bash

+ helm install aad-auth-proxy oci://mcr.microsoft.com/azuremonitor/auth-proxy/prod/aad-auth-proxy/helmchart/aad-auth-proxy \

+ --version 0.1.0-main-05-24-2023-b911fe1c \

+ -n observability \

+ --set targetHost=https://proxy-test-abc123.eastus-1.metrics.ingest.monitor.azure.com \

+ --set identityType=userAssigned \

+ --set aadClientId= abcd1234-1243-abcd-9876-1234abcd5678 \

+ --set audience=https://monitor.azure.com/.default

+ ```

+1. Configure remote write url.

+ The URL hostname is made up of the ingestion service name and namespace in the following format `<ingestion service name>.<namespace>.svc.cluster.local`. In this example, the host is `azuremonitor-ingestion.observability.svc.cluster.local`.

+ Configure the URL path using the path from the `Metrics ingestion endpoint` from the Azure Monitor workspace Overview page. For example, `dataCollectionRules/dcr-abc123d987e654f3210abc1def234567/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2021-11-01-preview`.

+ ```yml

+ prometheus:

+ prometheusSpec:

+ externalLabels:

+ cluster: <cluster name to be used in the workspace>

+ ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

+ ##

+ remoteWrite:

+ - url: "http://azuremonitor-ingestion.observability.svc.cluster.local/dataCollectionRules/dcr-abc123d987e654f3210abc1def234567/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2021-11-01-preview"

+ ```

+1. Apply the remote write configuration.

+> [!NOTE]

+> For the latest proxy image version ,see the [release notes](https://github.com/Azure/aad-auth-proxy/blob/main/RELEASENOTES.md)

+### Check that the proxy is ingesting data

+Check that the proxy is successfully ingesting metrics by checking the pod's logs, or by querying the Azure Monitor workspace.

+Check the pod's logs by running the following commands:

+```bash

+# Get the azuremonitor-ingestion pod ID

+ kubectl get pods -A | grep azuremonitor-ingestion

+ #Using the returned pod ID, get the logs

+ kubectl logs --namespace observability <pod ID> --tail=10

+ ```

+ Successfully ingesting metrics produces a log with `StatusCode=200` similar to the following:

+ ```

+ time="2023-05-16T08:47:27Z" level=info msg="Successfully sent request, returning response back." ContentLength=0 Request="https://amw-proxytest-05-t16w.eastus-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-688b6ed1f2244e098a88e32dde18b4f6/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2021-11-01-preview" StatusCode=200

+```

+To query your Azure Monitor workspace, follow the steps below:

+1. From your Azure Monitor workspace, select **Workbooks** .

+1. Select the **Prometheus Explorer** tile.

+ :::image type="content" source="./media/prometheus-authorization-proxy/workspace-workbooks.png" lightbox="./media/prometheus-authorization-proxy/workspace-workbooks.png" alt-text="A screenshot showing the workbooks gallery for an Azure Monitor workspace.":::

+1. On the explorer page, enter *up* into the query box.

+1. Select the **Grid** tab to see the results.

+1. Check the **cluster** column to see if from your cluster are displayed.

+ :::image type="content" source="./media/prometheus-authorization-proxy/prometheus-explorer.png" lightbox="./media/prometheus-authorization-proxy/prometheus-explorer.png" alt-text="A screenshot showing the Prometheus explorer query page.":::

+## [Query metrics example](#tab/query-metrics-example)

+This deployment allows external entities to query an Azure Monitor workspace via the proxy.

+Before deploying the proxy, find your managed identity and assign it the `Monitoring Metrics reader` role for the Azure Monitor workspace.

+1. Find the `clientId` for the managed identity for your AKS cluster. The managed identity is used to authenticate to the Azure Monitor workspace. The managed identity is created when the AKS cluster is created.

+ ```azurecli

+ # Get the identity client_id

+ az aks show -g <AKS-CLUSTER-RESOURCE-GROUP> -n <AKS-CLUSTER-NAME> --query "identityProfile"

+ ```

+

+ The output has the following format:

+ ```bash

+ {

+ "kubeletidentity": {

+ "clientId": "abcd1234-1243-abcd-9876-1234abcd5678",

+ "objectId": "12345678-abcd-abcd-abcd-1234567890ab",

+ "resourceId": "/subscriptions/def0123-1243-abcd-9876-1234abcd5678/resourcegroups/MC_rg-proxytest-01_proxytest-01_eastus/providers/Microsoft.ManagedIdentity/ userAssignedIdentities/proxytest-01-agentpool"

+ }

+ }

+ ```

+1. Assign the `Monitoring Data Reader` role to the identity using the `clientId` from the previous command so that it can read from the Azure Monitor workspace.

+ ```azurecli

+ az role assignment create --assignee <clientid> --role "Monitoring Data Reader" --scope <workspace-id>

+ ```

+1. Use the following YAML file to deploy the proxy for remote query. Modify the following parameters:

+ + `TARGET_HOST` - The host that you want to query data from. Use the `Query endpoint` from the Azure monitor workspace Overview page. For example, `https://proxytest-workspace-abcs.eastus.prometheus.monitor.azure.com`

+ + `AAD_CLIENT_ID` - The `clientId` of the managed identity used that was assigned the `Monitoring Metrics Reader` role.

+ + `AUDIENCE` - For querying metrics from Azure Monitor Workspace, set `AUDIENCE` to `https://prometheus.monitor.azure.com/.default`.

+ + Remove `OTEL_GRPC_ENDPOINT` and `OTEL_SERVICE_NAME` if you aren't using OpenTelemetry.

+ For more information on the parameters, see the [Parameters](#parameters) table.

+ proxy-query.yaml

+ ```yml

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ labels:

+ app: azuremonitor-query

+ name: azuremonitor-query

+ namespace: observability

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: azuremonitor-query

+ template:

+ metadata:

+ labels:

+ app: azuremonitor-query

+ name: azuremonitor-query

+ spec:

+ containers:

+ - name: aad-auth-proxy

+ image: mcr.microsoft.com/azuremonitor/auth-proxy/prod/aad-auth-proxy/images/aad-auth-proxy:aad-auth-proxy-0.1.0-main-04-11-2023-623473b0

+ imagePullPolicy: Always

+ ports:

+ - name: auth-port

+ containerPort: 8082

+ env:

+ - name: AUDIENCE

+ value: https://prometheus.monitor.azure.com/.default

+ - name: TARGET_HOST

+ value: <Query endpoint host>

+ - name: LISTENING_PORT

+ value: "8082"

+ - name: IDENTITY_TYPE

+ value: userAssigned

+ - name: AAD_CLIENT_ID

+ value: <clientId>

+ - name: AAD_TOKEN_REFRESH_INTERVAL_IN_PERCENTAGE

+ value: "10"

+ - name: OTEL_GRPC_ENDPOINT

+ value: "otel-collector.observability.svc.cluster.local:4317"

+ - name: OTEL_SERVICE_NAME

+ value: azuremonitor_query

+ livenessProbe:

+ httpGet:

+ path: /health

+ port: auth-port

+ initialDelaySeconds: 5

+ timeoutSeconds: 5

+ readinessProbe:

+ httpGet:

+ path: /ready

+ port: auth-port

+ initialDelaySeconds: 5

+ timeoutSeconds: 5

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: azuremonitor-query

+ namespace: observability

+ spec:

+ ports:

+ - port: 80

+ targetPort: 8082

+ selector:

+ app: azuremonitor-query

+ ```

+1. Deploy the proxy using command:

+ ```bash

+ # create the namespace if it doesn't already exist

+ kubectl create namespace observability

+ kubectl apply -f proxy-query.yaml -n observability

+ ```

+### Check that you can query using the proxy

+To test that the proxy is working, create a port forward to the proxy pod, then query the proxy.

+```bash

+# Get the pod name for azuremonitor-query pod

+kubectl get pods -n observability

+# Use the pod ID to create the port forward in the background

+kubectl port-forward pod/<pod ID> -n observability 8082:8082 &

+# query the proxy

+ curl http://localhost:8082/api/v1/query?query=up

+```

+A successful query returns a response similar to the following:

+```

+{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","cluster":"proxytest-01","instance":"aks-userpool-20877385-vmss000007","job":"kubelet","kubernetes_io_os":"linux","metrics_path":"/metrics"},"value":[1684177493.19,"1"]},{"metric":{"__name__":"up","cluster":"proxytest-01","instance":"aks-userpool-20877385-vmss000007","job":"cadvisor"},"value":[1684177493.19,"1"]},{"metric":{"__name__":"up","cluster":"proxytest-01","instance":"aks-nodepool1-21858175-vmss000007","job":"node","metrics_path":"/metrics"},"value":[1684177493.19,"1"]}]}}

+```

+## Parameters

+| Image Parameter | Helm chart Parameter name | Description | Supported values | Mandatory |

+| | | | | |

+| `TARGET_HOST` | `targetHost` | Target host where you want to forward the request to. <br>When sending data to an Azure Monitor workspace, use the `Metrics ingestion endpoint` from the workspaces Overview page. <br> When reading data from an Azure Monitor workspace, use the `Data collection rule` from the workspaces Overview page| | Yes |

+| `IDENTITY_TYPE` | `identityType` | Identity type that is used to authenticate requests. This proxy supports three types of identities. | `systemassigned`, `userassigned`, `aadapplication` | Yes |

+| `AAD_CLIENT_ID` | `aadClientId` | Client ID of the identity used. This is used for `userassigned` and `aadapplication` identity types. Use `az aks show -g <AKS-CLUSTER-RESOURCE-GROUP> -n <AKS-CLUSTER-NAME> --query "identityProfile"` to retrieve the Client ID | | Yes for `userassigned` and `aadapplication` |

+| `AAD_TENANT_ID` | `aadTenantId` | Tenant ID of the identity used. Tenant ID is used for `aadapplication` identity types. | | Yes for `aadapplication` |

+| `AAD_CLIENT_CERTIFICATE_PATH` | `aadClientCertificatePath` | The path where proxy can find the certificate for aadapplication. This path should be accessible by proxy and should be a either a pfx or pem certificate containing private key. | | For `aadapplication` identity types only |

+| `AAD_TOKEN_REFRESH_INTERVAL_IN_PERCENTAGE` | `aadTokenRefreshIntervalInMinutes` | Token is refreshed based on the percentage of time until token expiry. Default value is 10% time before expiry. | | No |

+| `AUDIENCE` | `audience` | Audience for the token | | No |

+| `LISTENING_PORT` | `listeningPort` | Proxy listening on this port | | Yes |

+| `OTEL_SERVICE_NAME` | `otelServiceName` | Service name for OTEL traces and metrics. Default value: aad_auth_proxy | | No |

+| `OTEL_GRPC_ENDPOINT` | `otelGrpcEndpoint` | Proxy pushes OTEL telemetry to this endpoint. Default value: http://localhost:4317 | | No |

+## Troubleshooting

+Run the following command to show any errors for the proxy container.

+ ```bash

+ kubectl --namespace <Namespace> describe pod <Prometheus-Pod-Name>`

+ ```

+ The proxy checks for a valid identity to fetch a token during startup. If it fails to retrieve a token, start up fails. Errors are logged and can be viewed by running the following command:

+ ```bash

+ kubectl --namespace <Namespace> logs <Proxy-Pod-Name>

+ ```

+ Example output:

+ ```

+ time="2023-05-15T11:24:06Z" level=info msg="Configuration settings loaded:" AAD_CLIENT_CERTIFICATE_PATH= AAD_CLIENT_ID=abc123de-be75-4141-a1e6-abc123987def AAD_TENANT_ID= AAD_TOKEN_REFRESH_INTERVAL_IN_PERCENTAGE=10 AUDIENCE="https://prometheus.monitor.azure.com" IDENTITY_TYPE=userassigned LISTENING_PORT=8082 OTEL_GRPC_ENDPOINT= OTEL_SERVICE_NAME=aad_auth_proxy TARGET_HOST=proxytest-01-workspace-orkw.eastus.prometheus.monitor.azure.com

+ 2023-05-15T11:24:06.414Z [ERROR] TokenCredential creation failed:Failed to get access token: ManagedIdentityCredential authentication failed

+ GET http://169.254.169.254/metadata/identity/oauth2/token

+ --

+ RESPONSE 400 Bad Request

+ --

+ {

+ "error": "invalid_request",

+ "error_description": "Identity not found"

+ }

+ --

+ ```

+ Title: Disable collecting Prometheus metrics on an Azure Kubernetes Service cluster

+description: Disable the collection of Prometheus metrics from an Azure Kubernetes Service cluster and remove the agent from the cluster nodes.

+# Disable Prometheus metrics collection from an AKS cluster

+Currently, the Azure CLI is the only option to remove the metrics add-on from your AKS cluster, and stop sending Prometheus metrics to Azure Monitor managed service for Prometheus.

+The `az aks update --disable-azure-monitor-metrics` command:

+> [!NOTE]

+> This action doesn't remove any existing data stored in your Azure Monitor workspace.

+```azurecli

+az aks update --disable-azure-monitor-metrics -n <cluster-name> -g <cluster-resource-group>

+```

+## Next steps

+- [See the default configuration for Prometheus metrics](./prometheus-metrics-scrape-default.md)

+- [Customize Prometheus metric scraping for the cluster](./prometheus-metrics-scrape-configuration.md)

+- [Use Azure Monitor managed service for Prometheus as the data source for Grafana](../essentials/prometheus-grafana.md)

+- [Configure self-hosted Grafana to use Azure Monitor managed service for Prometheus](../essentials/prometheus-self-managed-grafana-azure-active-directory.md)

+ Title: Enable Azure Monitor managed service for Prometheus

+description: Enable Azure Monitor managed service for Prometheus and configure data collection from your Azure Kubernetes Service (AKS) cluster.

+# Collect Prometheus metrics from an AKS cluster

+This article describes how to configure your Azure Kubernetes Service (AKS) cluster to send data to Azure Monitor managed service for Prometheus. When you perform this configuration, a containerized version of the [Azure Monitor agent](../agents/agents-overview.md) is installed with a metrics extension. This sends data to the Azure Monitor workspace that you specify.

+> [!NOTE]

+> The process described here doesn't enable [Container insights](../containers/container-insights-overview.md) on the cluster. However, both processes use the Azure Monitor agent. For different methods to enable Container insights on your cluster, see [Enable Container insights](../containers/container-insights-onboard.md)..

+The Azure Monitor metrics agent's architecture utilizes a ReplicaSet and a DaemonSet. The ReplicaSet pod scrapes cluster-wide targets such as `kube-state-metrics` and custom application targets that are specified. The DaemonSet pods scrape targets solely on the node that the respective pod is deployed on, such as `node-exporter`. This is so that the agent can scale as the number of nodes and pods on a cluster increases.

+## Prerequisites

+- You must either have an [Azure Monitor workspace](../essentials/azure-monitor-workspace-overview.md) or [create a new one](../essentials/azure-monitor-workspace-manage.md#create-an-azure-monitor-workspace).

+- The cluster must use [managed identity authentication](../../aks/use-managed-identity.md).

+- The following resource providers must be registered in the subscription of the AKS cluster and the Azure Monitor workspace:

+ - Microsoft.ContainerService

+ - Microsoft.Insights

+ - Microsoft.AlertsManagement

+> [!NOTE]

+> `Contributor` permission is enough for enabling the addon to send data to the Azure Monitor workspace. You will need `Owner` level permission in case you're trying to link your Azure Monitor Workspace to view metrics in Azure Managed Grafana. This is required because the user executing the onboarding step, needs to be able to give the Azure Managed Grafana System Identity `Monitoring Reader` role on the Azure Monitor Workspace to query the metrics.

+## Enable Prometheus metric collection

+Use any of the following methods to install the Azure Monitor agent on your AKS cluster and send Prometheus metrics to your Azure Monitor workspace.

+### [Azure portal](#tab/azure-portal)

+> [!NOTE]

+> Azure Managed Grafana is not available in the Azure US Government cloud currently.

+There are multiple options to enable Prometheus metrics on your cluster from the Azure portal.

+#### New cluster

+When you create a new AKS cluster in the Azure portal, you can enable Prometheus, Container insights, and Grafana from the **Integrations** tab.

+#### From the Azure Monitor workspace

+This option enables Prometheus metrics on a cluster without enabling Container insights.

+1. Open the **Azure Monitor workspaces** menu in the Azure portal and select your workspace.

+1. Select **Monitored clusters** in the **Managed Prometheus** section to display a list of AKS clusters.

+1. Select **Configure** next to the cluster you want to enable.

+ :::image type="content" source="media/prometheus-metrics-enable/azure-monitor-workspace-configure-prometheus.png" lightbox="media/prometheus-metrics-enable/azure-monitor-workspace-configure-prometheus.png" alt-text="Screenshot that shows an Azure Monitor workspace with a Prometheus configuration.":::

+#### From an existing cluster monitored with Container insights

+This option adds Prometheus metrics to a cluster already enabled for Container insights.

+1. Open the **Kubernetes services** menu in the Azure portal and select your AKS cluster.

+2. Click **Insights**.

+3. Click **Monitor settings**.

+ :::image type="content" source="media/prometheus-metrics-enable/aks-cluster-monitor-settings.png" lightbox="media/prometheus-metrics-enable/aks-cluster-monitor-settings.png" alt-text="Screenshot of button for monitor settings for an AKS cluster.":::

+4. Click the checkbox for **Enable Prometheus metrics** and select your Azure Monitor workspace.

+5. To send the collected metrics to Grafana, select a Grafana workspace. See [Create an Azure Managed Grafana instance](../../managed-grafan) for details on creating a Grafana workspace.

+ :::image type="content" source="media/prometheus-metrics-enable/aks-cluster-monitor-settings-details.png" lightbox="media/prometheus-metrics-enable/aks-cluster-monitor-settings-details.png" alt-text="Screenshot of monitor settings for an AKS cluster.":::

+6. Click **Configure** to complete the configuration.

+See [Collect Prometheus metrics from AKS cluster (preview)](../essentials/prometheus-metrics-enable.md) for details on [verifying your deployment](../essentials/prometheus-metrics-enable.md#verify-deployment) and [limitations](../essentials/prometheus-metrics-enable.md#limitations-during-enablementdeployment)

+#### From an existing cluster

+This options enables Prometheus, Grafana, and Container insights on a cluster.

+1. Open the clusters menu in the Azure portal and select **Insights**.

+3. Select **Configure monitoring**.

+4. Container insights is already enabled. Select the checkboxes for **Enable Prometheus metrics** and **Enable Grafana**. If you have existing Azure Monitor workspace and Garafana workspace, then they're selected for you. Click **Advanced settings** to select alternate workspaces or create new ones.

+ :::image type="content" source="media/prometheus-metrics-enable/configure-container-insights.png" lightbox="media/prometheus-metrics-enable/configure-container-insights.png" alt-text="Screenshot that shows that show the dialog box to configure Container insights with Prometheus and Grafana.":::

+5. Click **Configure** to save the configuration.

+### [CLI](#tab/cli)

+> [!NOTE]

+> Azure Managed Grafana is not available in the Azure US Government cloud currently.

+#### Prerequisites

+- The aks-preview extension must be uninstalled by using the command `az extension remove --name aks-preview`. For more information on how to uninstall a CLI extension, see [Use and manage extensions with the Azure CLI](/cli/azure/azure-cli-extensions-overview).

+- Az CLI version of 2.49.0 or higher is required for this feature. Check the aks-preview version by using the `az version` command.

+#### Install the metrics add-on

+Use `az aks create` or `az aks update` with the `-enable-azure-monitor-metrics` option to install the metrics add-on. Depending on the Azure Monitor workspace and Grafana workspace you want to use, choose one of the following options:

+- **Create a new default Azure Monitor workspace.**<br>

+If no Azure Monitor workspace is specified, a default Azure Monitor workspace is created in a resource group with the name `DefaultRG-<cluster_region>` and is named `DefaultAzureMonitorWorkspace-<mapped_region>`.

+ ```azurecli

+ az aks create/update --enable-azure-monitor-metrics -n <cluster-name> -g <cluster-resource-group>

+ ```

+- **Use an existing Azure Monitor workspace.**<br>

+If the existing Azure Monitor workspace is already linked to one or more Grafana workspaces, data is available in that Grafana workspace.

+ ```azurecli

+ az aks create/update --enable-azure-monitor-metrics -n <cluster-name> -g <cluster-resource-group> --azure-monitor-workspace-resource-id <workspace-name-resource-id>

+ ```

+- **Use an existing Azure Monitor workspace and link with an existing Grafana workspace.**<br>

+This option creates a link between the Azure Monitor workspace and the Grafana workspace.

+ ```azurecli

+ az aks create/update --enable-azure-monitor-metrics -n <cluster-name> -g <cluster-resource-group> --azure-monitor-workspace-resource-id <azure-monitor-workspace-name-resource-id> --grafana-resource-id <grafana-workspace-name-resource-id>

+ ```

+The output for each command looks similar to the following example:

+```json

+"azureMonitorProfile": {

+ "metrics": {

+ "enabled": true,

+ "kubeStateMetrics": {

+ "metricAnnotationsAllowList": "",

+ "metricLabelsAllowlist": ""

+ }

+ }

+}

+```

+#### Optional parameters

+You can use the following optional parameters with the previous commands:

+- `--ksm-metric-annotations-allow-list` is a comma-separated list of Kubernetes annotations keys used in the resource's kube_resource_annotations metric(For ex- kube_pod_annotations is the annotations metric for the pods resource). By default, the kube_resource_annotations(ex - kube_pod_annotations) metric contains only name and namespace labels. To include more annotations, provide a list of resource names in their plural form and Kubernetes annotation keys that you want to allow for them (Example: 'pods=[kubernetes.io/team,...],namespaces=[kubernetes.io/team],...)'. A single `*` can be provided per resource instead to allow any annotations, but it has severe performance implications.

+- `--ksm-metric-labels-allow-list` is a comma-separated list of more Kubernetes label keys that is used in the resource's kube_resource_labels metric kube_resource_labels metric(For ex- kube_pod_labels is the labels metric for the pods resource). By default the kube_resource_labels(ex - kube_pod_labels) metric contains only name and namespace labels. To include more labels, provide a list of resource names in their plural form and Kubernetes label keys that you want to allow for them (Example: 'pods=[app],namespaces=[k8s-label-1,k8s-label-n,...],...)'. A single asterisk (`*`) can be provided per resource instead to allow any labels, but it has severe performance implications.

+- `--enable-windows-recording-rules` lets you enable the recording rule groups required for proper functioning of the Windows dashboards.

+**Use annotations and labels.**

+```azurecli

+az aks create/update --enable-azure-monitor-metrics -n <cluster-name> -g <cluster-resource-group> --ksm-metric-labels-allow-list "namespaces=[k8s-label-1,k8s-label-n]" --ksm-metric-annotations-allow-list "pods=[k8s-annotation-1,k8s-annotation-n]"

+```

+The output is similar to the following example:

+```json

+ "azureMonitorProfile": {

+ "metrics": {

+ "enabled": true,

+ "kubeStateMetrics": {

+ "metricAnnotationsAllowList": "pods=[k8s-annotation-1,k8s-annotation-n]",

+ "metricLabelsAllowlist": "namespaces=[k8s-label-1,k8s-label-n]"

+ }

+ }

+ }

+```

+## [Azure Resource Manager](#tab/resource-manager)

+> [!NOTE]

+> Azure Managed Grafana is not available in the Azure US Government cloud currently.

+### Prerequisites

+- If the Azure Managed Grafana instance is in a subscription other than the Azure Monitor workspace subscription, register the Azure Monitor workspace subscription with the `Microsoft.Dashboard` resource provider by following [this documentation](../../azure-resource-manager/management/resource-providers-and-types.md#register-resource-provider).

+- The Azure Monitor workspace and Azure Managed Grafana instance must already be created.

+- The template must be deployed in the same resource group as the Azure Managed Grafana instance.

+- Users with the `User Access Administrator` role in the subscription of the AKS cluster can enable the `Monitoring Reader` role directly by deploying the template.

+### Retrieve required values for Grafana resource

+> [!NOTE]

+> Azure Managed Grafana is not available in the Azure US Government cloud currently.

+On the **Overview** page for the Azure Managed Grafana instance in the Azure portal, select **JSON view**.

+If you're using an existing Azure Managed Grafana instance that's already linked to an Azure Monitor workspace, the list of already existing Grafana integrations is needed. Copy the value of the `azureMonitorWorkspaceIntegrations` field. If it doesn't exist, then the instance hasn't been linked with any Azure Monitor workspace.

+```json

+"properties": {

+ "grafanaIntegrations": {

+ "azureMonitorWorkspaceIntegrations": [

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_1"

+ },

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_2"

+ }

+ ]

+ }

+}

+```

+### Download and edit the template and the parameter file

+1. Download the template at [https://aka.ms/azureprometheus-enable-arm-template](https://aka.ms/azureprometheus-enable-arm-template) and save it as **existingClusterOnboarding.json**.

+1. Download the parameter file at [https://aka.ms/azureprometheus-enable-arm-template-parameterss](https://aka.ms/azureprometheus-enable-arm-template-parameters) and save it as **existingClusterParam.json**.

+1. Edit the values in the parameter file.

+ | Parameter | Value |

+ |:|:|

+ | `azureMonitorWorkspaceResourceId` | Resource ID for the Azure Monitor workspace. Retrieve from the **JSON view** on the **Overview** page for the Azure Monitor workspace. |

+ | `azureMonitorWorkspaceLocation` | Location of the Azure Monitor workspace. Retrieve from the **JSON view** on the **Overview** page for the Azure Monitor workspace. |

+ | `clusterResourceId` | Resource ID for the AKS cluster. Retrieve from the **JSON view** on the **Overview** page for the cluster. |

+ | `clusterLocation` | Location of the AKS cluster. Retrieve from the **JSON view** on the **Overview** page for the cluster. |

+ | `metricLabelsAllowlist` | Comma-separated list of Kubernetes labels keys to be used in the resource's labels metric. |

+ | `metricAnnotationsAllowList` | Comma-separated list of more Kubernetes label keys to be used in the resource's annotations metric. |

+ | `grafanaResourceId` | Resource ID for the managed Grafana instance. Retrieve from the **JSON view** on the **Overview** page for the Grafana instance. |

+ | `grafanaLocation` | Location for the managed Grafana instance. Retrieve from the **JSON view** on the **Overview** page for the Grafana instance. |

+ | `grafanaSku` | SKU for the managed Grafana instance. Retrieve from the **JSON view** on the **Overview** page for the Grafana instance. Use the **sku.name**. |

+1. Open the template file and update the `grafanaIntegrations` property at the end of the file with the values that you retrieved from the Grafana instance. The following example is similar:

+ ```json

+ {

+ "type": "Microsoft.Dashboard/grafana",

+ "apiVersion": "2022-08-01",

+ "name": "[split(parameters('grafanaResourceId'),'/')[8]]",

+ "sku": {

+ "name": "[parameters('grafanaSku')]"

+ },

+ "location": "[parameters('grafanaLocation')]",

+ "properties": {

+ "grafanaIntegrations": {

+ "azureMonitorWorkspaceIntegrations": [

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_1"

+ },

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_2"

+ },

+ {

+ "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId')]"

+ }

+ ]

+ }

+ }

+ ```

+In this JSON, `full_resource_id_1` and `full_resource_id_2` were already in the Azure Managed Grafana resource JSON. They're added here to the Azure Resource Manager template (ARM template). If you have no existing Grafana integrations, don't include these entries for `full_resource_id_1` and `full_resource_id_2`.

+The final `azureMonitorWorkspaceResourceId` entry is already in the template and is used to link to the Azure Monitor workspace resource ID provided in the parameters file.

+## [Bicep](#tab/bicep)

+> [!NOTE]

+> Azure Managed Grafana is not available in the Azure US Government cloud currently.

+### Prerequisites

+- The Azure Monitor workspace and Azure Managed Grafana instance must already be created.

+- The template needs to be deployed in the same resource group as the Azure Managed Grafana instance.

+- Users with the `User Access Administrator` role in the subscription of the AKS cluster can enable the `Monitoring Reader` role directly by deploying the template.

+### Limitation with Bicep deployment

+Currently in Bicep, there's no way to explicitly scope the `Monitoring Reader` role assignment on a string parameter "resource ID" for an Azure Monitor workspace (like in an ARM template). Bicep expects a value of type `resource | tenant`. There also is no REST API [spec](https://github.com/Azure/azure-rest-api-specs) for an Azure Monitor workspace.

+Therefore, the default scoping for the `Monitoring Reader` role is on the resource group. The role is applied on the same Azure Monitor workspace (by inheritance), which is the expected behavior. After you deploy this Bicep template, the Grafana instance is given `Monitoring Reader` permissions for all the Azure Monitor workspaces in that resource group.

+### Retrieve required values for a Grafana resource

+On the **Overview** page for the Azure Managed Grafana instance in the Azure portal, select **JSON view**.

+If you're using an existing Azure Managed Grafana instance that's already linked to an Azure Monitor workspace, the list of already existing Grafana integrations is needed. Copy the value of the `azureMonitorWorkspaceIntegrations` field. If it doesn't exist, then the instance hasn't been linked with any Azure Monitor workspace.

+```json

+"properties": {

+ "grafanaIntegrations": {

+ "azureMonitorWorkspaceIntegrations": [

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_1"

+ },

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_2"

+ }

+ ]

+ }

+}

+```

+### Download and edit templates and the parameter file

+1. Download the [main Bicep template](https://aka.ms/azureprometheus-enable-bicep-template). Save it as **FullAzureMonitorMetricsProfile.bicep**.

+2. Download the [parameter file](https://aka.ms/azureprometheus-enable-bicep-template-parameters) and save it as **FullAzureMonitorMetricsProfileParameters.json** in the same directory as the main Bicep template.

+3. Download the [nested_azuremonitormetrics_dcra_clusterResourceId.bicep](https://aka.ms/nested_azuremonitormetrics_dcra_clusterResourceId) and [nested_azuremonitormetrics_profile_clusterResourceId.bicep](https://aka.ms/nested_azuremonitormetrics_profile_clusterResourceId) files into the same directory as the main Bicep template.

+4. Edit the values in the parameter file.

+5. The main Bicep template creates all the required resources. It uses two modules for creating the Data Collection Rule Associations (DCRA) and Azure Monitor metrics profile resources from the other two Bicep files.

+ | Parameter | Value |

+ |:|:|

+ | `azureMonitorWorkspaceResourceId` | Resource ID for the Azure Monitor workspace. Retrieve from the **JSON view** on the **Overview** page for the Azure Monitor workspace. |

+ | `azureMonitorWorkspaceLocation` | Location of the Azure Monitor workspace. Retrieve from the **JSON view** on the **Overview** page for the Azure Monitor workspace. |

+ | `clusterResourceId` | Resource ID for the AKS cluster. Retrieve from the **JSON view** on the **Overview** page for the cluster. |

+ | `clusterLocation` | Location of the AKS cluster. Retrieve from the **JSON view** on the **Overview** page for the cluster. |

+ | `metricLabelsAllowlist` | Comma-separated list of Kubernetes labels keys used in the resource's labels metric. |

+ | `metricAnnotationsAllowList` | Comma-separated list of more Kubernetes label keys used in the resource's annotations metric. |

+ | `grafanaResourceId` | Resource ID for the managed Grafana instance. Retrieve from the **JSON view** on the **Overview** page for the Grafana instance. |

+ | `grafanaLocation` | Location for the managed Grafana instance. Retrieve from the **JSON view** on the **Overview** page for the Grafana instance. |

+ | `grafanaSku` | SKU for the managed Grafana instance. Retrieve from the **JSON view** on the **Overview** page for the Grafana instance. Use the **sku.name**. |

+1. Open the template file and update the `grafanaIntegrations` property at the end of the file with the values that you retrieved from the Grafana instance. The following example is similar:

+ ```json

+ {

+ "type": "Microsoft.Dashboard/grafana",

+ "apiVersion": "2022-08-01",

+ "name": "[split(parameters('grafanaResourceId'),'/')[8]]",

+ "sku": {

+ "name": "[parameters('grafanaSku')]"

+ },

+ "location": "[parameters('grafanaLocation')]",

+ "properties": {

+ "grafanaIntegrations": {

+ "azureMonitorWorkspaceIntegrations": [

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_1"

+ },

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_2"

+ },

+ {

+ "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId')]"

+ }

+ ]

+ }

+ }

+ ```

+In this JSON, `full_resource_id_1` and `full_resource_id_2` were already in the Azure Managed Grafana resource JSON. They're added here to the ARM template. If you have no existing Grafana integrations, don't include these entries for `full_resource_id_1` and `full_resource_id_2`.

+The final `azureMonitorWorkspaceResourceId` entry is already in the template and is used to link to the Azure Monitor workspace resource ID provided in the parameters file.

+## [Terraform](#tab/terraform)

+### Prerequisites

+- If the Azure Managed Grafana instance is in a subscription other than the Azure Monitor Workspaces subscription, register the Azure Monitor Workspace subscription with the `Microsoft.Dashboard` resource provider by following [this documentation](../../azure-resource-manager/management/resource-providers-and-types.md#register-resource-provider).

+- The Azure Monitor workspace and Azure Managed Grafana workspace must already be created.

+- The template needs to be deployed in the same resource group as the Azure Managed Grafana workspace.

+- Users with the User Access Administrator role in the subscription of the AKS cluster can enable the Monitoring Reader role directly by deploying the template.

+### Retrieve required values for a Grafana resource

+On the **Overview** page for the Azure Managed Grafana instance in the Azure portal, select **JSON view**.

+If you're using an existing Azure Managed Grafana instance that's already linked to an Azure Monitor workspace, you need the list of Grafana integrations. Copy the value of the `azureMonitorWorkspaceIntegrations` field. If it doesn't exist, the instance hasn't been linked with any Azure Monitor workspace. Update the azure_monitor_workspace_integrations block(shown here) in main.tf with the list of grafana integrations.

+```.tf

+ azure_monitor_workspace_integrations {

+ resource_id = var.monitor_workspace_id[var.monitor_workspace_id1, var.monitor_workspace_id2]

+ }

+```

+### Download and edit the templates

+If you're deploying a new AKS cluster using Terraform with managed Prometheus addon enabled, follow these steps:

+1. Download all files under [AddonTerraformTemplate](https://aka.ms/AAkm357).

+2. Edit the variables in variables.tf file with the correct parameter values.

+3. Run `terraform init -upgrade` to initialize the Terraform deployment.

+4. Run `terraform plan -out main.tfplan` to initialize the Terraform deployment.

+5. Run `terraform apply main.tfplan` to apply the execution plan to your cloud infrastructure.

+Note: Pass the variables for `annotations_allowed` and `labels_allowed` keys in main.tf only when those values exist. These are optional blocks.

+> [!NOTE]

+> Edit the main.tf file appropriately before running the terraform template. Add in any existing azure_monitor_workspace_integrations values to the grafana resource before running the template. Else, older values gets deleted and replaced with what is there in the template during deployment. Users with 'User Access Administrator' role in the subscription of the AKS cluster can enable 'Monitoring Reader' role directly by deploying the template. Edit the grafanaSku parameter if you're using a nonstandard SKU and finally run this template in the Grafana Resource's resource group.

+## [Azure Policy](#tab/azurepolicy)

+> [!NOTE]

+> Azure Managed Grafana is not available in the Azure US Government cloud currently.

+### Prerequisites

+- The Azure Monitor workspace and Azure Managed Grafana instance must already be created.

+### Download Azure Policy rules and parameters and deploy

+1. Download the main [Azure Policy rules template](https://aka.ms/AddonPolicyMetricsProfile). Save it as **AddonPolicyMetricsProfile.rules.json**.

+1. Download the [parameter file](https://aka.ms/AddonPolicyMetricsProfile.parameters). Save it as **AddonPolicyMetricsProfile.parameters.json** in the same directory as the rules template.

+1. Create the policy definition using the following command:

+ `az policy definition create --name "(Preview) Prometheus Metrics addon" --display-name "(Preview) Prometheus Metrics addon" --mode Indexed --metadata version=1.0.0 category=Kubernetes --rules AddonPolicyMetricsProfile.rules.json --params AddonPolicyMetricsProfile.parameters.json`

+1. After you create the policy definition, in the Azure portal, select **Policy** > **Definitions**. Select the policy definition you created.

+1. Select **Assign**, go to the **Parameters** tab, and fill in the details. Select **Review + Create**.

+1. After the policy is assigned to the subscription, whenever you create a new cluster without Prometheus enabled, the policy will run and deploy to enable Prometheus monitoring. If you want to apply the policy to an existing AKS cluster, create a **Remediation task** for that AKS cluster resource after you go to the **Policy Assignment**.

+1. Now you should see metrics flowing in the existing Azure Managed Grafana instance, which is linked with the corresponding Azure Monitor workspace.

+Afterwards, if you create a new Managed Grafana instance, you can link it with the corresponding Azure Monitor workspace from the **Linked Grafana Workspaces** tab of the relevant **Azure Monitor Workspace** page. The `Monitoring Reader` role must be assigned to the managed identity of the Managed Grafana instance with the scope as the Azure Monitor workspace, so that Grafana has access to query the metrics. Use the following instructions to do so:

+1. On the **Overview** page for the Azure Managed Grafana instance in the Azure portal, select **JSON view**.

+1. Copy the value of the `principalId` field for the `SystemAssigned` identity.

+ ```json

+ "identity": {

+ "principalId": "00000000-0000-0000-0000-000000000000",

+ "tenantId": "00000000-0000-0000-0000-000000000000",

+ "type": "SystemAssigned"

+ },

+ ```

+1. On the **Access control (IAM)** page for the Azure Managed Grafana instance in the Azure portal, select **Add** > **Add role assignment**.

+1. Select `Monitoring Reader`.

+1. Select **Managed identity** > **Select members**.

+1. Select the **system-assigned managed identity** with the `principalId` from the Grafana resource.

+1. Choose **Select** > **Review+assign**.

+### Deploy the template

+Deploy the template with the parameter file by using any valid method for deploying ARM templates. For examples of different methods, see [Deploy the sample templates](../resource-manager-samples.md#deploy-the-sample-templates).

+### Limitations during enablement/deployment

+- Ensure that you update the `kube-state metrics` annotations and labels list with proper formatting. There's a limitation in the ARM template deployments that require exact values in the `kube-state` metrics pods. If the Kubernetes pod has any issues with malformed parameters and isn't running, the feature might not run as expected.

+- A data collection rule and data collection endpoint are created with the name `MSProm-\<short-cluster-region\>-\<cluster-name\>`. Currently, these names can't be modified.

+- You must get the existing Azure Monitor workspace integrations for a Grafana instance and update the ARM template with it. Otherwise, the ARM deployment gets over-written, which removes existing integrations.

+## Enable Windows metrics collection

+> [!NOTE]

+> There is no CPU/Memory limit in windows-exporter-daemonset.yaml so it may over-provision the Windows nodes

+> For more details see [Resource reservation](https://kubernetes.io/docs/concepts/configuration/windows-resource-management/#resource-reservation)

+>

+> As you deploy workloads, set resource memory and CPU limits on containers. This also subtracts from NodeAllocatable and helps the cluster-wide scheduler in determining which pods to place on which nodes.

+> Scheduling pods without limits may over-provision the Windows nodes and in extreme cases can cause the nodes to become unhealthy.

+As of version 6.4.0-main-02-22-2023-3ee44b9e of the Managed Prometheus addon container (prometheus_collector), Windows metric collection has been enabled for the AKS clusters. Onboarding to the Azure Monitor Metrics add-on enables the Windows DaemonSet pods to start running on your node pools. Both Windows Server 2019 and Windows Server 2022 are supported. Follow these steps to enable the pods to collect metrics from your Windows node pools.

+1. Manually install windows-exporter on AKS nodes to access Windows metrics.

+ Enable the following collectors:

+ * `[defaults]`

+ * `container`

+ * `memory`

+ * `process`

+ * `cpu_info`

+ Deploy the [windows-exporter-daemonset YAML](https://github.com/prometheus-community/windows_exporter/blob/master/kubernetes/windows-exporter-daemonset.yaml) file:

+ ```

+ kubectl apply -f windows-exporter-daemonset.yaml

+ ```

+1. Apply the [ama-metrics-settings-configmap](https://github.com/Azure/prometheus-collector/blob/main/otelcollector/configmaps/ama-metrics-settings-configmap.yaml) to your cluster. Set the `windowsexporter` and `windowskubeproxy` Booleans to `true`. For more information, see [Metrics add-on settings configmap](./prometheus-metrics-scrape-configuration.md#metrics-add-on-settings-configmap).

+1. Enable the recording rules that are required for the out-of-the-box dashboards:

+ * If onboarding using the CLI, include the option `--enable-windows-recording-rules`.

+ * If onboarding using an ARM template, Bicep, or Azure Policy, set `enableWindowsRecordingRules` to `true` in the parameters file.

+ * If the cluster is already onboarded, use [this ARM template](https://github.com/Azure/prometheus-collector/blob/kaveesh/windows_recording_rules/AddonArmTemplate/WindowsRecordingRuleGroupTemplate/WindowsRecordingRules.json) and [this parameter file](https://github.com/Azure/prometheus-collector/blob/kaveesh/windows_recording_rules/AddonArmTemplate/WindowsRecordingRuleGroupTemplate/WindowsRecordingRulesParameters.json) to create the rule groups.

+## Verify deployment

+1. Run the following command to verify that the DaemonSet was deployed properly on the Linux node pools:

+ ```

+ kubectl get ds ama-metrics-node --namespace=kube-system

+ ```

+ The number of pods should be equal to the number of Linux nodes on the cluster. The output should resemble the following example:

+ ```

+ User@aksuser:~$ kubectl get ds ama-metrics-node --namespace=kube-system

+ NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE

+ ama-metrics-node 1 1 1 1 1 <none> 10h

+ ```

+1. Run the following command to verify that the DaemonSet was deployed properly on the Windows node pools:

+ ```

+ kubectl get ds ama-metrics-win-node --namespace=kube-system

+ ```

+ The number of pods should be equal to the number of Windows nodes on the cluster. The output should resemble the following example:

+ ```

+ User@aksuser:~$ kubectl get ds ama-metrics-node --namespace=kube-system

+ NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE

+ ama-metrics-win-node 3 3 3 3 3 <none> 10h

+ ```

+1. Run the following command to verify that the two ReplicaSets were deployed properly:

+ ```

+ kubectl get rs --namespace=kube-system

+ ```

+ The output should resemble the following example:

+ ```

+ User@aksuser:~$kubectl get rs --namespace=kube-system

+ NAME DESIRED CURRENT READY AGE

+ ama-metrics-5c974985b8 1 1 1 11h

+ ama-metrics-ksm-5fcf8dffcd 1 1 1 11h

+ ```

+## Artifacts/Resources provisioned/created as a result of metrics addon enablement for an AKS cluster

+When you enable metrics addon, the following resources are provisioned:

+| Resource Name | Resource Type | Resource Group | Region/Location | Description |

+ |:|:|:|:|:|

+ | `MSPROM-<aksclusterregion>-<clustername>` | **Data Collection Rule** | Same Resource group as AKS cluster resource | Same region as Azure Monitor Workspace | This data collection rule is for prometheus metrics collection by metrics addon, which has the chosen Azure monitor workspace as destination, and also it is associated to the AKS cluster resource |

+ | `MSPROM-<aksclusterregion>-<clustername>` | **Data Collection endpoint** | Same Resource group as AKS cluster resource | Same region as Azure Monitor Workspace | This data collection endpoint is used by the above data collection rule for ingesting Prometheus metrics from the metrics addon|

+

+When you create a new Azure Monitor workspace, the following additional resources are created as part of it

+| Resource Name | Resource Type | Resource Group | Region/Location | Description |

+ |:|:|:|:|:|

+ | `<azuremonitor-workspace-name>` | **System Data Collection Rule** | MA_\<azuremonitor-workspace-name>_\<azuremonitor-workspace-region>_managed | Same region as Azure Monitor Workspace | This is **system** data collection rule that customers can use when they use OSS Prometheus server to Remote Write to Azure Monitor Workspace |

+ | `<azuremonitor-workspace-name>` | **System Data Collection endpoint** | MA_\<azuremonitor-workspace-name>_\<azuremonitor-workspace-region>_managed | Same region as Azure Monitor Workspace | This is **system** data collection endpoint that customers can use when they use OSS Prometheus server to Remote Write to Azure Monitor Workspace |

+

+## HTTP Proxy

+Azure Monitor metrics addon supports HTTP Proxy and uses the same settings as the HTTP Proxy settings for the AKS cluster configured with [these instructions](../../../articles/aks/http-proxy.md).

+## Network firewall requirements

+**Azure public cloud**

+The following table lists the firewall configuration required for Azure monitor Prometheus metrics ingestion for Azure Public cloud. All network traffic from the agent is outbound to Azure Monitor.

+|Agent resource| Purpose | Port |

+|--|||

+| `global.handler.control.monitor.azure.com` | Access control service/ Azure Monitor control plane service | 443 |

+| `*.ingest.monitor.azure.com` | Azure monitor managed service for Prometheus - metrics ingestion endpoint (DCE) | 443 |

+| `*.handler.control.monitor.azure.com` | For querying data collection rules | 443 |

+**Azure US Government cloud**

+The following table lists the firewall configuration required for Azure monitor Prometheus metrics ingestion for Azure US Government cloud. All network traffic from the agent is outbound to Azure Monitor.

+|Agent resource| Purpose | Port |

+|--|||

+| `global.handler.control.monitor.azure.us` | Access control service/ Azure Monitor control plane service | 443 |

+| `*.ingest.monitor.azure.us` | Azure monitor managed service for Prometheus - metrics ingestion endpoint (DCE) | 443 |

+| `*.handler.control.monitor.azure.us` | For querying data collection rules | 443 |

+## Uninstall the metrics add-on

+To uninstall the metrics add-on, see [Disable Prometheus metrics collection on an AKS cluster.](./prometheus-metrics-disable.md)

+## Supported regions

+The list of regions Azure Monitor Metrics and Azure Monitor Workspace is supported in can be found [here](https://aka.ms/ama-metrics-supported-regions) under the Managed Prometheus tag.

+## Next steps

+- [See the default configuration for Prometheus metrics](./prometheus-metrics-scrape-default.md)

+- [Customize Prometheus metric scraping for the cluster](./prometheus-metrics-scrape-configuration.md)

+- [Use Azure Monitor managed service for Prometheus as the data source for Grafana](../essentials/prometheus-grafana.md)

+- [Configure self-hosted Grafana to use Azure Monitor managed service for Prometheus](../essentials/prometheus-self-managed-grafana-azure-active-directory.md)

+ Title: Collect Prometheus metrics from an Arc-enabled Kubernetes cluster (preview)

+description: How to configure your Azure Arc-enabled Kubernetes cluster (preview) to send data to Azure Monitor managed service for Prometheus.

+# Collect Prometheus metrics from an Arc-enabled Kubernetes cluster (preview)

+This article describes how to configure your Azure Arc-enabled Kubernetes cluster (preview) to send data to Azure Monitor managed service for Prometheus. When you configure your Azure Arc-enabled Kubernetes cluster to send data to Azure Monitor managed service for Prometheus, a containerized version of the Azure Monitor agent is installed with a metrics extension. You then specify the Azure Monitor workspace where the data should be sent.

+> [!NOTE]

+> The process described here doesn't enable [Container insights](../containers/container-insights-overview.md) on the cluster even though the Azure Monitor agent installed in this process is the same agent used by Container insights.

+> For different methods to enable Container insights on your cluster, see [Enable Container insights](../containers/container-insights-onboard.md). For details on adding Prometheus collection to a cluster that already has Container insights enabled, see [Collect Prometheus metrics with Container insights](../containers/container-insights-prometheus.md).

+## Supported configurations

+The following configurations are supported:

+The following configurations are not supported:

+## Prerequisites

+ + Microsoft.Kubernetes

+ + Microsoft.Insights

+ + Microsoft.AlertsManagement

+ **Azure public cloud**

+ |Endpoint|Port|

+ ||--|

+ |*.ods.opinsights.azure.com |443 |

+ |*.oms.opinsights.azure.com |443 |

+ |dc.services.visualstudio.com |443 |

+ |*.monitoring.azure.com |443 |

+ |login.microsoftonline.com |443 |

+ |global.handler.control.monitor.azure.com |443 |

+ | \<cluster-region-name\>.handler.control.monitor.azure.com |443 |

+## Create an extension instance

+### [Portal](#tab/portal)

+### Onboard from Azure Monitor workspace

+1. Open the **Azure Monitor workspaces** menu in the Azure portal and select your cluster.

+1. Select **Managed Prometheus** to display a list of AKS and Arc clusters.

+1. Select **Configure** for the cluster you want to enable.

+### Onboard from Container insights

+1. In the Azure portal, select the Azure Arc-enabled Kubernetes cluster that you wish to monitor.

+1. From the resource pane on the left, select **Insights** under the **Monitoring** section.

+1. On the onboarding page, select **Configure monitoring**.

+1. On the **Configure Container insights** page, select the **Enable Prometheus metrics** checkbox.

+1. Select **Configure**.

+### [CLI](#tab/cli)

+### Prerequisites

+### Create an extension with default values

+```azurecli

+az k8s-extension create \

+--name azuremonitor-metrics \

+--cluster-name <cluster-name> \

+--resource-group <resource-group> \

+--cluster-type connectedClusters \

+--extension-type Microsoft.AzureMonitor.Containers.Metrics

+```

+### Create an extension with an existing Azure Monitor workspace

+If the Azure Monitor workspace is already linked to one or more Grafana workspaces, the data is available in Grafana.

+```azurecli

+az k8s-extension create\

+--name azuremonitor-metrics\

+--cluster-name <cluster-name>\

+--resource-group <resource-group>\

+--cluster-type connectedClusters\

+--extension-type Microsoft.AzureMonitor.Containers.Metrics\

+--configuration-settings azure-monitor-workspace-resource-id=<workspace-name-resource-id>

+```

+### Create an extension with an existing Azure Monitor workspace and link with an existing Grafana workspace

+This option creates a link between the Azure Monitor workspace and the Grafana workspace.

+```azurecli

+az k8s-extension create\

+--name azuremonitor-metrics\

+--cluster-name <cluster-name>\

+--resource-group <resource-group>\

+--cluster-type connectedClusters\

+--extension-type Microsoft.AzureMonitor.Containers.Metrics\

+--configuration-settings azure-monitor-workspace-resource-id=<workspace-name-resource-id> \

+grafana-resource-id=<grafana-workspace-name-resource-id>

+```

+### Create an extension with optional parameters

+You can use the following optional parameters with the previous commands:

+`--configurationsettings.AzureMonitorMetrics.KubeStateMetrics.MetricsLabelsAllowlist` is a comma-separated list of Kubernetes label keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional labels, provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them. For example, `=namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...`

+`--configurationSettings.AzureMonitorMetrics.KubeStateMetrics.MetricAnnotationsAllowList` is a comma-separated list of Kubernetes annotations keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional annotations, provide a list of resource names in their plural form and Kubernetes annotation keys you would like to allow for them. For example, `=namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...`.

+> [!NOTE]

+> A single `*`, for example `'=pods=[*]'` can be provided per resource to allow any labels, however, this has severe performance implications.

+```azurecli

+az k8s-extension create\

+--name azuremonitor-metrics\

+--cluster-name <cluster-name>\

+--resource-group <resource-group>\

+--cluster-type connectedClusters\

+--extension-type Microsoft.AzureMonitor.Containers.Metrics\

+--configuration-settings azure-monitor-workspace-resource-id=<workspace-name-resource-id> \

+grafana-resource-id=<grafana-workspace-name-resource-id> \

+AzureMonitorMetrics.KubeStateMetrics.MetricAnnotationsAllowList="pods=[k8s-annotation-1,k8s-annotation-n]" \

+AzureMonitorMetrics.KubeStateMetrics.MetricsLabelsAllowlist "namespaces=[k8s-label-1,k8s-label-n]"

+```

+### [Resource Manager](#tab/resource-manager)

+### Prerequisites

+### Create an extension

+1. Retrieve required values for the Grafana resource

+ > [!NOTE]

+ > Azure Managed Grafana is not currently available in the Azure US Government cloud.

+ On the Overview page for the Azure Managed Grafana instance in the Azure portal, select **JSON view**.

+ If you're using an existing Azure Managed Grafana instance that's already linked to an Azure Monitor workspace, you need the list of already existing Grafana integrations. Copy the value of the `azureMonitorWorkspaceIntegrations` field. If the field doesn't exist, the instance hasn't been linked with any Azure Monitor workspace.

+ ```json

+ "properties": {

+ "grafanaIntegrations": {

+ "azureMonitorWorkspaceIntegrations": [

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_1"

+ },

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_2"

+ }

+ ]

+ }

+ }

+ ```

+1. Download and edit the template and the parameter file

+ 1. Download the template at https://aka.ms/azureprometheus-arc-arm-template and save it as *existingClusterOnboarding.json*.

+ 1. Download the parameter file at https://aka.ms/azureprometheus-arc-arm-template-parameters and save it as *existingClusterParam.json*.

+1. Edit the following fields' values in the parameter file.

+ |Parameter|Value |

+ |||

+ |`azureMonitorWorkspaceResourceId` |Resource ID for the Azure Monitor workspace. Retrieve from the **JSON view** on the Overview page for the Azure Monitor workspace. |

+ |`azureMonitorWorkspaceLocation`|Location of the Azure Monitor workspace. Retrieve from the JSON view on the Overview page for the Azure Monitor workspace. |

+ |`clusterResourceId` |Resource ID for the Arc cluster. Retrieve from the **JSON view** on the Overview page for the cluster. |

+ |`clusterLocation` |Location of the Arc cluster. Retrieve from the **JSON view** on the Overview page for the cluster. |

+ |`metricLabelsAllowlist` |Comma-separated list of Kubernetes labels keys to be used in the resource's labels metric.|

+ |`metricAnnotationsAllowList` |Comma-separated list of more Kubernetes label keys to be used in the resource's labels metric. |

+ |`grafanaResourceId` |Resource ID for the managed Grafana instance. Retrieve from the **JSON view** on the Overview page for the Grafana instance. |

+ |`grafanaLocation` |Location for the managed Grafana instance. Retrieve from the **JSON view** on the Overview page for the Grafana instance. |

+ |`grafanaSku` |SKU for the managed Grafana instance. Retrieve from the **JSON view** on the Overview page for the Grafana instance. Use the `sku.name`. |

+1. Open the template file and update the `grafanaIntegrations` property at the end of the file with the values that you retrieved from the Grafana instance. For example:

+ ```json

+ {

+ "type": "Microsoft.Dashboard/grafana",

+ "apiVersion": "2022-08-01",

+ "name": "[split(parameters('grafanaResourceId'),'/')[8]]",

+ "sku": {

+ "name": "[parameters('grafanaSku')]"

+ },

+ "location": "[parameters('grafanaLocation')]",

+ "properties": {

+ "grafanaIntegrations": {

+ "azureMonitorWorkspaceIntegrations": [

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_1"

+ },

+ {

+ "azureMonitorWorkspaceResourceId": "full_resource_id_2"

+ },

+ {

+ "azureMonitorWorkspaceResourceId": "[parameters ('azureMonitorWorkspaceResourceId')]"

+ }

+ ]

+ }

+ }

+ }

+ ```

+ In the example JSON above, `full_resource_id_1` and `full_resource_id_2` are already in the Azure Managed Grafana resource JSON. They're added here to the Azure Resource Manager template (ARM template). If you don't have any existing Grafana integrations, don't include these entries.

+ The final `azureMonitorWorkspaceResourceId` entry is in the template by default and is used to link to the Azure Monitor workspace resource ID provided in the parameters file.

+### Verify extension installation status

+Once you have successfully created the Azure Monitor extension for your Azure Arc-enabled Kubernetes cluster, you can check the status of the installation using the Azure portal or CLI. Successful installations show the status as `Installed`.

+#### Azure portal

+1. In the Azure portal, select the Azure Arc-enabled Kubernetes cluster with the extension installation.

+1. From the resource pane on the left, select the **Extensions** item under the **Setting**' section.

+1. An extension with the name **azuremonitor-metrics** is listed, with the current status in the **Install status** column.

+#### Azure CLI

+Run the following command to show the latest status of the` Microsoft.AzureMonitor.Containers.Metrics` extension.

+```azurecli

+az k8s-extension show \

+--name azuremonitor-metrics \

+--cluster-name <cluster-name> \

+--resource-group <resource-group> \

+--cluster-type connectedClusters

+```

+### Delete the extension instance

+To delete the extension instance, use the following CLI command:

+```azurecli

+az k8s-extension delete --name azuremonitor-metrics -g <cluster_resource_group> -c<cluster_name> -t connectedClusters

+```

+The command only deletes the extension instance. The Azure Monitor workspace and its data are not deleted.

+## Disconnected clusters

+If your cluster is disconnected from Azure for more than 48 hours, Azure Resource Graph won't have information about your cluster. As a result, your Azure Monitor Workspace may have incorrect information about your cluster state.

+## Troubleshooting

+For issues with the extension, see the [Troubleshooting Guide](./prometheus-metrics-troubleshoot.md).

+## Next Steps

+ Title: Send Prometheus metrics to multiple Azure Monitor workspaces

+description: Describes data collection rules required to send Prometheus metrics from a cluster in Azure Monitor to multiple Azure Monitor workspaces.

+# Send Prometheus metrics to multiple Azure Monitor workspaces

+Routing metrics to more Azure Monitor workspaces can be done through the creation of additional data collection rules. All metrics can be sent to all workspaces or different metrics can be sent to different workspaces.

+## Send same metrics to multiple Azure Monitor workspaces

+You can create multiple Data Collection Rules that point to the same Data Collection Endpoint for metrics to be sent to additional Azure Monitor workspaces from the same Kubernetes cluster. In case you have a very high volume of metrics, a new Data Collection Endpoint can be created as well. Please refer to the service limits [document](../service-limits.md) regarding ingestion limits. Currently, this is only available through onboarding through Resource Manager templates. You can follow the [regular onboarding process](prometheus-metrics-enable.md) and then edit the same Resource Manager templates to add additional DCRs and DCEs (if applicable) for your additional Azure Monitor workspaces. You'll need to edit the template to add an additional parameters for every additional Azure Monitor workspace, add another DCR for every additional Azure Monitor workspace, add another DCE (if applicable), add the Monitor Reader Role for the new Azure Monitor workspace and add an additional Azure Monitor workspace integration for Grafana.

+- Add the following parameters:

+ ```json

+ "parameters": {

+ "azureMonitorWorkspaceResourceId2": {

+ "type": "string"

+ },

+ "azureMonitorWorkspaceLocation2": {

+ "type": "string",

+ "defaultValue": "",

+ "allowedValues": [

+ "eastus2euap",

+ "centraluseuap",

+ "centralus",

+ "eastus",

+ "eastus2",

+ "northeurope",

+ "southcentralus",

+ "southeastasia",

+ "uksouth",

+ "westeurope",

+ "westus",

+ "westus2"

+ ]

+ },

+ ...

+ }

+ ```

+- For high metric volume, add an additional Data Collection Endpoint. You *must* replace `<dceName>`:

+ ```json

+ {

+ "type": "Microsoft.Insights/dataCollectionEndpoints",

+ "apiVersion": "2021-09-01-preview",

+ "name": "[variables('dceName')]",

+ "location": "[parameters('azureMonitorWorkspaceLocation2')]",

+ "kind": "Linux",

+ "properties": {}

+ }

+ ```

+- Add an additional DCR with the same or a different Data Collection Endpoint. You *must* replace `<dcrName>`:

+ ```json

+ {

+ "type": "Microsoft.Insights/dataCollectionRules",

+ "apiVersion": "2021-09-01-preview",

+ "name": "<dcrName>",

+ "location": "[parameters('azureMonitorWorkspaceLocation2')]",

+ "kind": "Linux",

+ "properties": {

+ "dataCollectionEndpointId": "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]",

+ "dataFlows": [

+ {

+ "destinations": ["MonitoringAccount2"],

+ "streams": ["Microsoft-PrometheusMetrics"]

+ }

+ ],

+ "dataSources": {

+ "prometheusForwarder": [

+ {

+ "name": "PrometheusDataSource",

+ "streams": ["Microsoft-PrometheusMetrics"],

+ "labelIncludeFilter": {}

+ }

+ ]

+ },

+ "description": "DCR for Azure Monitor Metrics Profile (Managed Prometheus)",

+ "destinations": {

+ "monitoringAccounts": [

+ {

+ "accountResourceId": "[parameters('azureMonitorWorkspaceResourceId2')]",

+ "name": "MonitoringAccount2"

+ }

+ ]

+ }

+ },

+ "dependsOn": [

+ "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]"

+ ]

+ }

+ ```

+- Add an additional DCRA with the relevant Data Collection Rule. You *must* replace `<dcraName>`:

+ ```json

+ {

+ "type": "Microsoft.Resources/deployments",

+ "name": "<dcraName>",

+ "apiVersion": "2017-05-10",

+ "subscriptionId": "[variables('clusterSubscriptionId')]",

+ "resourceGroup": "[variables('clusterResourceGroup')]",

+ "dependsOn": [

+ "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]",

+ "[resourceId('Microsoft.Insights/dataCollectionRules', variables('dcrName'))]"

+ ],

+ "properties": {

+ "mode": "Incremental",

+ "template": {

+ "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",

+ "contentVersion": "1.0.0.0",

+ "parameters": {},

+ "variables": {},

+ "resources": [

+ {

+ "type": "Microsoft.ContainerService/managedClusters/providers/dataCollectionRuleAssociations",

+ "name": "[concat(variables('clusterName'),'/microsoft.insights/', variables('dcraName'))]",

+ "apiVersion": "2021-09-01-preview",

+ "location": "[parameters('clusterLocation')]",

+ "properties": {

+ "description": "Association of data collection rule. Deleting this association will break the data collection for this AKS Cluster.",

+ "dataCollectionRuleId": "[resourceId('Microsoft.Insights/dataCollectionRules', variables('dcrName'))]"

+ }

+ }

+ ]

+ },

+ "parameters": {}

+ }

+ }

+ ```

+- Add an additional Grafana integration:

+ ```json

+ {

+ "type": "Microsoft.Dashboard/grafana",

+ "apiVersion": "2022-08-01",

+ "name": "[split(parameters('grafanaResourceId'),'/')[8]]",

+ "sku": {

+ "name": "[parameters('grafanaSku')]"

+ },

+ "location": "[parameters('grafanaLocation')]",

+ "properties": {

+ "grafanaIntegrations": {

+ "azureMonitorWorkspaceIntegrations": [

+ // Existing azureMonitorWorkspaceIntegrations values (if any)

+ // {

+ // "azureMonitorWorkspaceResourceId": "<value>"

+ // },

+ // {

+ // "azureMonitorWorkspaceResourceId": "<value>"

+ // },

+ {

+ "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId')]"

+ },

+ {

+ "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId2')]"

+ }

+ ]

+ }

+ }

+ }

+ ```

+ - Assign `Monitoring Data Reader` role to read data from the new Azure Monitor Workspace:

+ ```json

+ {

+ "type": "Microsoft.Authorization/roleAssignments",

+ "apiVersion": "2022-04-01",

+ "name": "[parameters('roleNameGuid')]",

+ "scope": "[parameters('azureMonitorWorkspaceResourceId2')]",

+ "properties": {

+ "roleDefinitionId": "[concat('/subscriptions/', variables('clusterSubscriptionId'), '/providers/Microsoft.Authorization/roleDefinitions/', 'b0d8363b-8ddd-447d-831f-62ca05bff136')]",

+ "principalId": "[reference(resourceId('Microsoft.Dashboard/grafana', split(parameters('grafanaResourceId'),'/')[8]), '2022-08-01', 'Full').identity.principalId]"

+ }

+ }

+ ```

+## Send different metrics to different Azure Monitor workspaces

+If you want to send some metrics to one Azure Monitor workspace and other metrics to a different one, follow the above steps to add additional DCRs. The value of `microsoft_metrics_include_label` under the `labelIncludeFilter` in the DCR is the identifier for the workspace. To then configure which metrics are routed to which workspace, you can add an extra pre-defined label, `microsoft_metrics_account` to the metrics. The value should be the same as the corresponding `microsoft_metrics_include_label` in the DCR for that workspace. To add the label to the metrics, you can utilize `relabel_configs` in your scrape config. To send all metrics from one job to a certain workspace, add the following relabel config:

+```yaml

+relabel_configs:

+- source_labels: [__address__]

+ target_label: microsoft_metrics_account

+ action: replace

+ replacement: "MonitoringAccountLabel1"

+```

+The source label is `__address__` because this label will always exist so this relabel config will always be applied. The target label will always be `microsoft_metrics_account` and its value should be replaced with the corresponding label value for the workspace.

+### Example

+If you want to configure three different jobs to send the metrics to three different workspaces, then include the following in each data collection rule:

+```json

+"labelIncludeFilter": {

+ "microsoft_metrics_include_label": "MonitoringAccountLabel1"

+}

+```

+```json

+"labelIncludeFilter": {

+ "microsoft_metrics_include_label": "MonitoringAccountLabel2"

+}

+```

+```json

+"labelIncludeFilter": {

+ "microsoft_metrics_include_label": "MonitoringAccountLabel3"

+}

+```

+Then in your scrape config, include the same label value for each:

+```yaml

+scrape_configs:

+- job_name: prometheus_ref_app_1

+ kubernetes_sd_configs:

+ - role: pod

+ relabel_configs:

+ - source_labels: [__meta_kubernetes_pod_label_app]

+ action: keep

+ regex: "prometheus-reference-app-1"

+ - source_labels: [__address__]

+ target_label: microsoft_metrics_account

+ action: replace

+ replacement: "MonitoringAccountLabel1"

+- job_name: prometheus_ref_app_2

+ kubernetes_sd_configs:

+ - role: pod

+ relabel_configs:

+ - source_labels: [__meta_kubernetes_pod_label_app]

+ action: keep

+ regex: "prometheus-reference-app-2"

+ - source_labels: [__address__]

+ target_label: microsoft_metrics_account

+ action: replace

+ replacement: "MonitoringAccountLabel2"

+- job_name: prometheus_ref_app_3

+ kubernetes_sd_configs:

+ - role: pod

+ relabel_configs:

+ - source_labels: [__meta_kubernetes_pod_label_app]

+ action: keep

+ regex: "prometheus-reference-app-3"

+ - source_labels: [__address__]

+ target_label: microsoft_metrics_account

+ action: replace

+ replacement: "MonitoringAccountLabel3"

+```

+## Next steps

+- [Learn more about Azure Monitor managed service for Prometheus](../essentials/prometheus-metrics-overview.md).

+- [Collect Prometheus metrics from AKS cluster](prometheus-metrics-enable.md).

+ Title: Minimal Prometheus ingestion profile in Azure Monitor

+description: Describes minimal ingestion profile in Azure Monitor managed service for Prometheus and how you can configure it collect more data.

+# Minimal ingestion profile for Prometheus metrics in Azure Monitor

+Azure monitor metrics addon collects number of Prometheus metrics by default. `Minimal ingestion profile` is a setting that helps reduce ingestion volume of metrics, as only metrics used by default dashboards, default recording rules & default alerts are collected. This article describes how this setting is configured. This article also lists metrics collected by default when `minimal ingestion profile` is enabled. You can modify collection to enable collecting more metrics, as specified below.

+> [!NOTE]

+> For addon based collection, `Minimal ingestion profile` setting is enabled by default.

+Following targets are **enabled/ON** by default - meaning you don't have to provide any scrape job configuration for scraping these targets, as metrics addon will scrape these targets automatically by default

+- `cadvisor` (`job=cadvisor`)

+- `nodeexporter` (`job=node`)

+- `kubelet` (`job=kubelet`)

+- `kube-state-metrics` (`job=kube-state-metrics`)

+Following targets are available to scrape, but scraping isn't enabled (**disabled/OFF**) by default - meaning you don't have to provide any scrape job configuration for scraping these targets but they're disabled/OFF by default and you need to turn ON/enable scraping for these targets using [ama-metrics-settings-configmap](https://aka.ms/azureprometheus-addon-settings-configmap) under `default-scrape-settings-enabled` section

+- `core-dns` (`job=kube-dns`)

+- `kube-proxy` (`job=kube-proxy`)

+- `api-server` (`job=kube-apiserver`)

+> [!NOTE]

+> The default scrape frequency for all default targets and scrapes is `30 seconds`. You can override it per target using the [ama-metrics-settings-configmap](https://aka.ms/azureprometheus-addon-settings-configmap) under `default-targets-scrape-interval-settings` section.

+> You can read more about four different configmaps used by metrics addon [here](prometheus-metrics-scrape-configuration.md)

+## Configuration setting

+The setting `default-targets-metrics-keep-list.minimalIngestionProfile="true"` is enabled by default on the metrics addon. You can specify this setting in [ama-metrics-settings-configmap](https://aka.ms/azureprometheus-addon-settings-configmap) under `default-targets-metrics-keep-list` section.

+## Scenarios

+There are four scenarios where you may want to customize this behavior:

+**Ingest only minimal metrics per default target.**<br>

+This is the default behavior with the setting `default-targets-metrics-keep-list.minimalIngestionProfile="true"`. Only metrics listed below are ingested for each of the default targets.

+**Ingest a few other metrics for one or more default targets in addition to minimal metrics.**<br>

+Keep ` minimalIngestionProfile="true"` and specify the appropriate `keeplistRegexes.*` specific to the target, for example `keeplistRegexes.coreDns="X``Y"`. X,Y is merged with default metric list for the target and then ingested. ``

+**Ingest only a specific set of metrics for a target, and nothing else.**<br>

+Set `minimalIngestionProfile="false"` and specify the appropriate `default-targets-metrics-keep-list.="X``Y"` specific to the target in the `ama-metrics-settings-configmap`.

+**Ingest all metrics scraped for the default target.**<br>

+Set `minimalIngestionProfile="false"` and don't specify any `default-targets-metrics-keep-list.<targetname>` for that target. Changing to `false` can increase metric ingestion volume by a factor per target.

+> [!NOTE]

+> `up` metric is not part of the allow/keep list because it is ingested per scrape, per target, regardless of `keepLists` specified. This metric is not actually scraped but produced as result of scrape operation by the metrics addon. For histograms and summaries, each series has to be included explicitly in the list (`*bucket`, `*sum`, `*count` series).

+### Minimal ingestion for default ON targets

+The following metrics are allow-listed with `minimalingestionprofile=true` for default ON targets. The below metrics are collected by default as these targets are scraped by default.

+**kubelet**<br>

+- `kubelet_volume_stats_used_bytes`

+- `kubelet_node_name`

+- `kubelet_running_pods`

+- `kubelet_running_pod_count`

+- `kubelet_running_containers`

+- `kubelet_running_container_count`

+- `volume_manager_total_volumes`

+- `kubelet_node_config_error`

+- `kubelet_runtime_operations_total`

+- `kubelet_runtime_operations_errors_total`

+- `kubelet_runtime_operations_duration_seconds` `kubelet_runtime_operations_duration_seconds_bucket` `kubelet_runtime_operations_duration_seconds_sum` `kubelet_runtime_operations_duration_seconds_count`

+- `kubelet_pod_start_duration_seconds` `kubelet_pod_start_duration_seconds_bucket` `kubelet_pod_start_duration_seconds_sum` `kubelet_pod_start_duration_seconds_count`

+- `kubelet_pod_worker_duration_seconds` `kubelet_pod_worker_duration_seconds_bucket` `kubelet_pod_worker_duration_seconds_sum` `kubelet_pod_worker_duration_seconds_count`

+- `storage_operation_duration_seconds` `storage_operation_duration_seconds_bucket` `storage_operation_duration_seconds_sum` `storage_operation_duration_seconds_count`

+- `storage_operation_errors_total`

+- `kubelet_cgroup_manager_duration_seconds` `kubelet_cgroup_manager_duration_seconds_bucket` `kubelet_cgroup_manager_duration_seconds_sum` `kubelet_cgroup_manager_duration_seconds_count`

+- `kubelet_pleg_relist_duration_seconds` `kubelet_pleg_relist_duration_seconds_bucket` `kubelet_pleg_relist_duration_sum` `kubelet_pleg_relist_duration_seconds_count`

+- `kubelet_pleg_relist_interval_seconds` `kubelet_pleg_relist_interval_seconds_bucket` `kubelet_pleg_relist_interval_seconds_sum` `kubelet_pleg_relist_interval_seconds_count`

+- `rest_client_requests_total`

+- `rest_client_request_duration_seconds` `rest_client_request_duration_seconds_bucket` `rest_client_request_duration_seconds_sum` `rest_client_request_duration_seconds_count`

+- `process_resident_memory_bytes`

+- `process_cpu_seconds_total`

+- `go_goroutines`

+- `kubelet_volume_stats_capacity_bytes`

+- `kubelet_volume_stats_available_bytes`

+- `kubelet_volume_stats_inodes_used`

+- `kubelet_volume_stats_inodes`

+- `kubernetes_build_info"`

+**cadvisor**<br>

+- `container_spec_cpu_period`

+- `container_spec_cpu_quota`

+- `container_cpu_usage_seconds_total`

+- `container_memory_rss`

+- `container_network_receive_bytes_total`

+- `container_network_transmit_bytes_total`

+- `container_network_receive_packets_total`

+- `container_network_transmit_packets_total`

+- `container_network_receive_packets_dropped_total`

+- `container_network_transmit_packets_dropped_total`

+- `container_fs_reads_total`

+- `container_fs_writes_total`

+- `container_fs_reads_bytes_total`

+- `container_fs_writes_bytes_total`

+- `container_memory_working_set_bytes`

+- `container_memory_cache`

+- `container_memory_swap`

+- `container_cpu_cfs_throttled_periods_total`

+- `container_cpu_cfs_periods_total`

+- `container_memory_usage_bytes`

+- `kubernetes_build_info"`

+**kube-state-metrics**<br>

+- `kube_node_status_capacity`

+- `kube_job_status_succeeded`

+- `kube_job_spec_completions`

+- `kube_daemonset_status_desired_number_scheduled`

+- `kube_daemonset_status_number_ready`

+- `kube_deployment_spec_replicas`

+- `kube_deployment_status_replicas_ready`

+- `kube_pod_container_status_last_terminated_reason`

+- `kube_node_status_condition`

+- `kube_pod_container_status_restarts_total`

+- `kube_pod_container_resource_requests`

+- `kube_pod_status_phase`

+- `kube_pod_container_resource_limits`

+- `kube_node_status_allocatable`

+- `kube_pod_info`

+- `kube_pod_owner`

+- `kube_resourcequota`

+- `kube_statefulset_replicas`

+- `kube_statefulset_status_replicas`

+- `kube_statefulset_status_replicas_ready`

+- `kube_statefulset_status_replicas_current`

+- `kube_statefulset_status_replicas_updated`

+- `kube_namespace_status_phase`

+- `kube_node_info`

+- `kube_statefulset_metadata_generation`

+- `kube_pod_labels`

+- `kube_pod_annotations`

+- `kube_horizontalpodautoscaler_status_current_replicas`

+- `kube_horizontalpodautoscaler_status_desired_replicas`

+- `kube_horizontalpodautoscaler_spec_min_replicas`

+- `kube_horizontalpodautoscaler_spec_max_replicas`

+- `kube_node_status_condition`

+- `kube_node_spec_taint`

+- `kube_pod_container_status_waiting_reason`

+- `kube_job_failed`

+- `kube_job_status_start_time`

+- `kube_deployment_spec_replicas`

+- `kube_deployment_status_replicas_available`

+- `kube_deployment_status_replicas_updated`

+- `kube_job_status_active`

+- `kubernetes_build_info`

+- `kube_pod_container_info`

+- `kube_replicaset_owner`

+- `kube_resource_labels` (ex - kube_pod_labels, kube_deployment_labels)

+- `kube_resource_annotations` (ex - kube_pod_annotations, kube_deployment_annotations)

+**node-exporter (linux)**<br>

+- `node_cpu_seconds_total`

+- `node_memory_MemAvailable_bytes`

+- `node_memory_Buffers_bytes`

+- `node_memory_Cached_bytes`

+- `node_memory_MemFree_bytes`

+- `node_memory_Slab_bytes`

+- `node_memory_MemTotal_bytes`

+- `node_netstat_Tcp_RetransSegs`

+- `node_netstat_Tcp_OutSegs`

+- `node_netstat_TcpExt_TCPSynRetrans`

+- `node_load1``node_load5`

+- `node_load15`

+- `node_disk_read_bytes_total`

+- `node_disk_written_bytes_total`

+- `node_disk_io_time_seconds_total`

+- `node_filesystem_size_bytes`

+- `node_filesystem_avail_bytes`

+- `node_filesystem_readonly`

+- `node_network_receive_bytes_total`

+- `node_network_transmit_bytes_total`

+- `node_vmstat_pgmajfault`

+- `node_network_receive_drop_total`

+- `node_network_transmit_drop_total`

+- `node_disk_io_time_weighted_seconds_total`

+- `node_exporter_build_info`

+- `node_time_seconds`

+- `node_uname_info"`

+### Minimal ingestion for default OFF targets

+The following are metrics that are allow-listed with `minimalingestionprofile=true` for default OFF targets. These metrics are not collected by default as these targets are not scraped by default (due to being OFF by default). You can turn ON scraping for these targets using `default-scrape-settings-enabled.<target-name>=true`' using [ama-metrics-settings-configmap](https://aka.ms/azureprometheus-addon-settings-configmap) under `default-scrape-settings-enabled` section.

+**core-dns**

+- `coredns_build_info`

+- `coredns_panics_total`

+- `coredns_dns_responses_total`

+- `coredns_forward_responses_total`

+- `coredns_dns_request_duration_seconds` `coredns_dns_request_duration_seconds_bucket` `coredns_dns_request_duration_seconds_sum` `coredns_dns_request_duration_seconds_count`

+- `coredns_forward_request_duration_seconds` `coredns_forward_request_duration_seconds_bucket` `coredns_forward_request_duration_seconds_sum` `coredns_forward_request_duration_seconds_count`

+- `coredns_dns_requests_total`

+- `coredns_forward_requests_total`

+- `coredns_cache_hits_total`

+- `coredns_cache_misses_total`

+- `coredns_cache_entries`

+- `coredns_plugin_enabled`

+- `coredns_dns_request_size_bytes` `coredns_dns_request_size_bytes_bucket` `coredns_dns_request_size_bytes_sum` `coredns_dns_request_size_bytes_count`

+- `coredns_dns_response_size_bytes` `coredns_dns_response_size_bytes_bucket` `coredns_dns_response_size_bytes_sum` `coredns_dns_response_size_bytes_count`

+- `coredns_dns_response_size_bytes` `coredns_dns_response_size_bytes_bucket` `coredns_dns_response_size_bytes_sum` `coredns_dns_response_size_bytes_count`

+- `process_resident_memory_bytes`

+- `process_cpu_seconds_total`

+- `go_goroutines`

+- `kubernetes_build_info"`

+**kube-proxy**

+- `kubeproxy_sync_proxy_rules_duration_seconds` `kubeproxy_sync_proxy_rules_duration_seconds_bucket` `kubeproxy_sync_proxy_rules_duration_seconds_sum` `kubeproxy_sync_proxy_rules_duration_seconds_count` `kubeproxy_network_programming_duration_seconds`

+- `kubeproxy_network_programming_duration_seconds` `kubeproxy_network_programming_duration_seconds_bucket` `kubeproxy_network_programming_duration_seconds_sum` `kubeproxy_network_programming_duration_seconds_count` `rest_client_requests_total`

+- `rest_client_request_duration_seconds` `rest_client_request_duration_seconds_bucket` `rest_client_request_duration_seconds_sum` `rest_client_request_duration_seconds_count`

+- `process_resident_memory_bytes`

+- `process_cpu_seconds_total`

+- `go_goroutines`

+- `kubernetes_build_info"`

+**api-server**

+- `apiserver_request_duration_seconds` `apiserver_request_duration_seconds_bucket` `apiserver_request_duration_seconds_sum` `apiserver_request_duration_seconds_count`

+- `apiserver_request_total`

+- `workqueue_adds_total``workqueue_depth`

+- `workqueue_queue_duration_seconds` `workqueue_queue_duration_seconds_bucket` `workqueue_queue_duration_seconds_sum` `workqueue_queue_duration_seconds_count`

+- `process_resident_memory_bytes`

+- `process_cpu_seconds_total`

+- `go_goroutines`

+- `kubernetes_build_info"`

+**windows-exporter (job=windows-exporter)**<br>

+- `windows_system_system_up_time`

+- `windows_cpu_time_total`

+- `windows_memory_available_bytes`

+- `windows_os_visible_memory_bytes`

+- `windows_memory_cache_bytes`

+- `windows_memory_modified_page_list_bytes`

+- `windows_memory_standby_cache_core_bytes`

+- `windows_memory_standby_cache_normal_priority_bytes`

+- `windows_memory_standby_cache_reserve_bytes`

+- `windows_memory_swap_page_operations_total`

+- `windows_logical_disk_read_seconds_total`

+- `windows_logical_disk_write_seconds_total`

+- `windows_logical_disk_size_bytes`

+- `windows_logical_disk_free_bytes`

+- `windows_net_bytes_total`

+- `windows_net_packets_received_discarded_total`

+- `windows_net_packets_outbound_discarded_total`

+- `windows_container_available`

+- `windows_container_cpu_usage_seconds_total`

+- `windows_container_memory_usage_commit_bytes`

+- `windows_container_memory_usage_private_working_set_bytes`

+- `windows_container_network_receive_bytes_total`

+- `windows_container_network_transmit_bytes_total`

+**kube-proxy-windows (job=kube-proxy-windows)**<br>

+- `kubeproxy_sync_proxy_rules_duration_seconds`

+- `kubeproxy_sync_proxy_rules_duration_seconds_bucket`

+- `kubeproxy_sync_proxy_rules_duration_seconds_sum`

+- `kubeproxy_sync_proxy_rules_duration_seconds_count`

+- `rest_client_requests_total`

+- `rest_client_request_duration_seconds`

+- `rest_client_request_duration_seconds_bucket`

+- `rest_client_request_duration_seconds_sum`

+- `rest_client_request_duration_seconds_count`

+- `process_resident_memory_bytes`

+- `process_cpu_seconds_total`

+- `go_goroutines`

+## Next steps

+- [Learn more about customizing Prometheus metric scraping in Container insights](prometheus-metrics-scrape-configuration.md).

+ Title: Customize scraping of Prometheus metrics in Azure Monitor

+description: Customize metrics scraping for a Kubernetes cluster with the metrics add-on in Azure Monitor.

+# Customize scraping of Prometheus metrics in Azure Monitor managed service for Prometheus

+This article provides instructions on customizing metrics scraping for a Kubernetes cluster with the [metrics addon](prometheus-metrics-enable.md) in Azure Monitor.

+## Configmaps

+Four different configmaps can be configured to provide scrape configuration and other settings for the metrics add-on. All config-maps should be applied to `kube-system` namespace for any cluster.

+> [!NOTE]

+> None of the four configmaps exist by default in the cluster when Managed Prometheus is enabled. Depending on what needs to be customized, you need to deploy any or all of these four configmaps with the same name specified, in `kube-system` namespace. AMA-Metrics pods will pick up these configmaps after you deploy them to `kube-system` namespace, and will restart in 2-3 minutes to apply the configuration settings specified in the configmap(s).

+1. [`ama-metrics-settings-configmap`](https://aka.ms/azureprometheus-addon-settings-configmap)

+ This config map has below simple settings that can be configured. You can take the configmap from the above git hub repo, change the settings are required and apply/deploy the configmap to `kube-system` namespace for your cluster

+ * cluster alias (to change the value of `cluster` label in every time-series/metric that's ingested from a cluster)

+ * enable/disable default scrape targets - Turn ON/OFF default scraping based on targets. Scrape configuration for these default targets are already pre-defined/built-in

+ * enable pod annotation based scraping per namespace

+ * metric keep-lists - this setting is used to control which metrics are listed to be allowed from each default target and to change the default behavior

+ * scrape intervals for default/pre-definetargets. `30 secs` is the default scrape frequency and it can be changed per default target using this configmap

+ * debug-mode - turning this ON helps to debug missing metric/ingestion issues - see more on [troubleshooting](prometheus-metrics-troubleshoot.md#debug-mode)

+2. [`ama-metrics-prometheus-config`](https://aka.ms/azureprometheus-addon-rs-configmap)

+ This config map can be used to provide Prometheus scrape config for addon replica. Addon runs a singleton replica, and any cluster level services can be discovered and scraped by providing scrape jobs in this configmap. You can take the sample configmap from the above git hub repo, add scrape jobs that you would need and apply/deploy the config map to `kube-system` namespace for your cluster.

+3. [`ama-metrics-prometheus-config-node`](https://aka.ms/azureprometheus-addon-ds-configmap)

+ This config map can be used to provide Prometheus scrape config for addon DaemonSet that runs on every **Linux** node in the cluster, and any node level targets on each node can be scraped by providing scrape jobs in this configmap. When you use this configmap, you can use `$NODE_IP` variable in your scrape config, which gets substituted by corresponding node's ip address in DaemonSet pod running on each node. This way you get access to scrape anything that runs on that node from the metrics addon DaemonSet. **Please be careful when you use discoveries in scrape config in this node level config map, as every node in the cluster will setup & discover the target(s) and will collect redundant metrics**.

+ You can take the sample configmap from the above git hub repo, add scrape jobs that you would need and apply/deploy the config map to `kube-system` namespace for your cluster

+4. [`ama-metrics-prometheus-config-node-windows`](https://aka.ms/azureprometheus-addon-ds-configmap-windows)

+ This config map can be used to provide Prometheus scrape config for addon DaemonSet that runs on every **Windows** node in the cluster, and node level targets on each node can be scraped by providing scrape jobs in this configmap. When you use this configmap, you can use `$NODE_IP` variable in your scrape config, which will be substituted by corresponding node's ip address in DaemonSet pod running on each node. This way you get access to scrape anything that runs on that node from the metrics addon DaemonSet. **Please be careful when you use discoveries in scrape config in this node level config map, as every node in the cluster will setup & discover the target(s) and will collect redundant metrics**.

+ You can take the sample configmap from the above git hub repo, add scrape jobs that you would need and apply/deploy the config map to `kube-system` namespace for your cluster

+## Metrics add-on settings configmap

+The [ama-metrics-settings-configmap](https://aka.ms/azureprometheus-addon-settings-configmap) can be downloaded, edited, and applied to the cluster to customize the out-of-the-box features of the metrics add-on.

+### Enable and disable default targets

+The following table has a list of all the default targets that the Azure Monitor metrics add-on can scrape by default and whether it's initially enabled. Default targets are scraped every 30 seconds. A replica is deployed to scrape cluster-wide targets such as kube-state-metrics. A DaemonSet is also deployed to scrape node-wide targets such as kubelet.

+| Key | Type | Enabled | Pod | Description |

+|--||-|-|-|

+| kubelet | bool | `true` | Linux DaemonSet | Scrape kubelet in every node in the K8s cluster without any extra scrape config. |

+| cadvisor | bool | `true` | Linux DaemonSet | Scrape cadvisor in every node in the K8s cluster without any extra scrape config.<br>Linux only. |

+| kubestate | bool | `true` | Linux replica | Scrape kube-state-metrics in the K8s cluster (installed as a part of the add-on) without any extra scrape config. |

+| nodeexporter | bool | `true` | Linux DaemonSet | Scrape node metrics without any extra scrape config.<br>Linux only. |

+| coredns | bool | `false` | Linux replica | Scrape coredns service in the K8s cluster without any extra scrape config. |

+| kubeproxy | bool | `false` | Linux DaemonSet | Scrape kube-proxy in every Linux node discovered in the K8s cluster without any extra scrape config.<br>Linux only. |

+| apiserver | bool | `false` | Linux replica | Scrape the Kubernetes API server in the K8s cluster without any extra scrape config. |

+| windowsexporter | bool | `false` | Windows DaemonSet | Scrape windows-exporter in every node in the K8s cluster without any extra scrape config.<br>Windows only. |

+| windowskubeproxy | bool | `false` | Windows DaemonSet | Scrape windows-kube-proxy in every node in the K8s cluster without any extra scrape config.<br>Windows only. |

+| prometheuscollectorhealth | bool | `false` | Linux replica | Scrape information about the prometheus-collector container, such as the amount and size of time series scraped. |

+If you want to turn on the scraping of the default targets that aren't enabled by default, edit the [configmap](https://aka.ms/azureprometheus-addon-settings-configmap) `ama-metrics-settings-configmap` to update the targets listed under `default-scrape-settings-enabled` to `true`. Apply the configmap to your cluster.

+### Customize metrics collected by default targets

+By default, for all the default targets, only minimal metrics used in the default recording rules, alerts, and Grafana dashboards are ingested as described in [minimal-ingestion-profile](prometheus-metrics-scrape-configuration-minimal.md). To collect all metrics from default targets, update the keep-lists in the settings configmap under `default-targets-metrics-keep-list`, and set `minimalingestionprofile` to `false`.

+To allowlist more metrics in addition to default metrics that are listed to be allowed, for any default targets, edit the settings under `default-targets-metrics-keep-list` for the corresponding job you want to change.

+For example, `kubelet` is the metric filtering setting for the default target kubelet. Use the following script to filter *in* metrics collected for the default targets by using regex-based filtering.

+```

+kubelet = "metricX|metricY"

+apiserver = "mymetric.*"

+```

+> [!NOTE]

+> If you use quotation marks or backslashes in the regex, you need to escape them by using a backslash like the examples `"test\'smetric\"s\""` and `testbackslash\\*`.

+To further customize the default jobs to change properties like collection frequency or labels, disable the corresponding default target by setting the configmap value for the target to `false`. Then apply the job by using a custom configmap. For details on custom configuration, see [Customize scraping of Prometheus metrics in Azure Monitor](prometheus-metrics-scrape-configuration.md#configure-custom-prometheus-scrape-jobs).

+### Cluster alias

+The cluster label appended to every time series scraped uses the last part of the full AKS cluster's Azure Resource Manager resource ID. For example, if the resource ID is `/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/rg-name/providers/Microsoft.ContainerService/managedClusters/myclustername`, the cluster label is `myclustername`.

+To override the cluster label in the time series scraped, update the setting `cluster_alias` to any string under `prometheus-collector-settings` in the [configmap](https://aka.ms/azureprometheus-addon-settings-configmap) `ama-metrics-settings-configmap`. You can create this configmap if it doesn't exist in the cluster or you can edit the existing one if its already exists in your cluster.

+The new label also shows up in the cluster parameter dropdown in the Grafana dashboards instead of the default one.

+> [!NOTE]

+> Only alphanumeric characters are allowed. Any other characters are replaced with `_`. This change is to ensure that different components that consume this label adhere to the basic alphanumeric convention.

+### Debug mode

+To view every metric that's being scraped for debugging purposes, the metrics add-on agent can be configured to run in debug mode by updating the setting `enabled` to `true` under the `debug-mode` setting in the [configmap](https://aka.ms/azureprometheus-addon-settings-configmap) `ama-metrics-settings-configmap`. You can either create this configmap or edit an existing one. For more information, see the [Debug mode section in Troubleshoot collection of Prometheus metrics](prometheus-metrics-troubleshoot.md#debug-mode).

+### Scrape interval settings

+To update the scrape interval settings for any target, you can update the duration in the setting `default-targets-scrape-interval-settings` for that target in the [configmap](https://aka.ms/azureprometheus-addon-settings-configmap) `ama-metrics-settings-configmap`. You have to set the scrape intervals in the correct format specified in [this website](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Otherwise, the default value of 30 seconds is applied to the corresponding targets. For example - If you want to update the scrape interval for the `kubelet` job to `60s` then you can update the following section in the YAML:

+```

+default-targets-scrape-interval-settings: |-

+ kubelet = "60s"

+ coredns = "30s"

+ cadvisor = "30s"

+ kubeproxy = "30s"

+ apiserver = "30s"

+ kubestate = "30s"

+ nodeexporter = "30s"

+ windowsexporter = "30s"

+ windowskubeproxy = "30s"

+ kappiebasic = "30s"

+ prometheuscollectorhealth = "30s"

+ podannotations = "30s"

+```

+and apply the YAML using the following command: `kubectl apply -f .\ama-metrics-settings-configmap.yaml`

+## Configure custom Prometheus scrape jobs

+You can configure the metrics add-on to scrape targets other than the default ones by using the same configuration format as the [Prometheus configuration file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file).

+Follow the instructions to [create, validate, and apply the configmap](prometheus-metrics-scrape-validate.md) for your cluster.

+### Advanced setup: Configure custom Prometheus scrape jobs for the DaemonSet

+The `ama-metrics` Replica pod consumes the custom Prometheus config and scrapes the specified targets. For a cluster with a large number of nodes and pods and a large volume of metrics to scrape, some of the applicable custom scrape targets can be off-loaded from the single `ama-metrics` Replica pod to the `ama-metrics` DaemonSet pod.

+The [ama-metrics-prometheus-config-node configmap](https://aka.ms/azureprometheus-addon-ds-configmap), is similar to the replica-set configmap, and can be created to have static scrape configs on each node. The scrape config should only target a single node and shouldn't use service discovery. Otherwise, each node tries to scrape all targets and makes many calls to the Kubernetes API server.

+Example:- The following `node-exporter` config is one of the default targets for the DaemonSet pods. It uses the `$NODE_IP` environment variable, which is already set for every `ama-metrics` add-on container to target a specific port on the node.

+ ```yaml

+ - job_name: nodesample

+ scrape_interval: 30s

+ scheme: http

+ metrics_path: /metrics

+ relabel_configs:

+ - source_labels: [__metrics_path__]

+ regex: (.*)

+ target_label: metrics_path

+ - source_labels: [__address__]

+ replacement: '$NODE_NAME'

+ target_label: instance

+ static_configs:

+ - targets: ['$NODE_IP:9100']

+ ```

+Custom scrape targets can follow the same format by using `static_configs` with targets and using the `$NODE_IP` environment variable and specifying the port to scrape. Each pod of the DaemonSet takes the config, scrapes the metrics, and sends them for that node.

+## Prometheus configuration tips and examples

+Learn some tips from examples in this section.

+### Configuration file for custom scrape config

+The configuration format is the same as the [Prometheus configuration file](https://aka.ms/azureprometheus-promioconfig). Currently, the following sections are supported:

+```yaml

+global:

+ scrape_interval: <duration>

+ scrape_timeout: <duration>

+ external_labels:

+ <labelname1>: <labelvalue>

+ <labelname2>: <labelvalue>

+scrape_configs:

+ - <job-x>

+ - <job-y>

+```

+Any other unsupported sections must be removed from the config before they're applied as a configmap. Otherwise, the custom configuration fails validation and isn't applied.

+See the [Apply config file](prometheus-metrics-scrape-validate.md#deploy-config-file-as-configmap) section to create a configmap from the Prometheus config.

+> [!NOTE]

+> When custom scrape configuration fails to apply because of validation errors, default scrape configuration continues to be used.

+## Scrape configs

+Currently, the supported methods of target discovery for a [scrape config](https://aka.ms/azureprometheus-promioconfig-scrape) are either [`static_configs`](https://aka.ms/azureprometheus-promioconfig-static) or [`kubernetes_sd_configs`](https://aka.ms/azureprometheus-promioconfig-sdk8s) for specifying or discovering targets.

+#### Static config

+A static config has a list of static targets and any extra labels to add to them.

+```yaml

+scrape_configs:

+ - job_name: example

+ - targets: [ '10.10.10.1:9090', '10.10.10.2:9090', '10.10.10.3:9090' ... ]

+ - labels: [ label1: value1, label1: value2, ... ]

+```

+#### Kubernetes Service Discovery config

+Targets discovered using [`kubernetes_sd_configs`](https://aka.ms/azureprometheus-promioconfig-sdk8s) each have different `__meta_*` labels depending on what role is specified. You can use the labels in the `relabel_configs` section to filter targets or replace labels for the targets.

+See the [Prometheus examples](https://aka.ms/azureprometheus-promsampleossconfig) of scrape configs for a Kubernetes cluster.

+### Relabel configs

+The `relabel_configs` section is applied at the time of target discovery and applies to each target for the job. The following examples show ways to use `relabel_configs`.

+#### Add a label

+Add a new label called `example_label` with the value `example_value` to every metric of the job. Use `__address__` as the source label only because that label always exists and adds the label for every target of the job.

+```yaml

+relabel_configs:

+- source_labels: [__address__]

+ target_label: example_label

+ replacement: 'example_value'

+```

+#### Use Kubernetes Service Discovery labels

+If a job is using [`kubernetes_sd_configs`](https://aka.ms/azureprometheus-promioconfig-sdk8s) to discover targets, each role has associated `__meta_*` labels for metrics. The `__*` labels are dropped after discovering the targets. To filter by using them at the metrics level, first keep them using `relabel_configs` by assigning a label name. Then use `metric_relabel_configs` to filter.

+```yaml

+# Use the kubernetes namespace as a label called 'kubernetes_namespace'

+relabel_configs:

+- source_labels: [__meta_kubernetes_namespace]

+ action: replace

+ target_label: kubernetes_namespace

+# Keep only metrics with the kubernetes namespace 'default'

+metric_relabel_configs:

+- source_labels: [kubernetes_namespace]

+ action: keep

+ regex: 'default'

+```

+#### Job and instance relabeling

+You can change the `job` and `instance` label values based on the source label, just like any other label.

+```yaml

+# Replace the job name with the pod label 'k8s app'

+relabel_configs:

+- source_labels: [__meta_kubernetes_pod_label_k8s_app]

+ target_label: job

+# Replace the instance name with the node name. This is helpful to replace a node IP

+# and port with a value that is more readable

+relabel_configs:

+- source_labels: [__meta_kubernetes_node_name]]

+ target_label: instance

+```

+### Metric relabel configs

+Metric relabel configs are applied after scraping and before ingestion. Use the `metric_relabel_configs` section to filter metrics after scraping. The following examples show how to do so.

+#### Drop metrics by name

+```yaml

+# Drop the metric named 'example_metric_name'

+metric_relabel_configs:

+- source_labels: [__name__]

+ action: drop

+ regex: 'example_metric_name'

+```

+#### Keep only certain metrics by name

+```yaml

+# Keep only the metric named 'example_metric_name'

+metric_relabel_configs:

+- source_labels: [__name__]

+ action: keep

+ regex: 'example_metric_name'

+```

+```yaml

+# Keep only metrics that start with 'example_'

+metric_relabel_configs:

+- source_labels: [__name__]

+ action: keep

+ regex: '(example_.*)'

+```

+#### Rename metrics

+Metric renaming isn't supported.

+#### Filter metrics by labels

+```yaml

+# Keep metrics only where example_label = 'example'

+metric_relabel_configs:

+- source_labels: [example_label]

+ action: keep

+ regex: 'example'

+```

+```yaml

+# Keep metrics only if `example_label` equals `value_1` or `value_2`

+metric_relabel_configs:

+- source_labels: [example_label]

+ action: keep

+ regex: '(value_1|value_2)'

+```

+```yaml

+# Keep metrics only if `example_label_1 = value_1` and `example_label_2 = value_2`

+metric_relabel_configs:

+- source_labels: [example_label_1, example_label_2]

+ separator: ';'

+ action: keep

+ regex: 'value_1;value_2'

+```

+```yaml

+# Keep metrics only if `example_label` exists as a label

+metric_relabel_configs:

+- source_labels: [example_label_1]

+ action: keep

+ regex: '.+'

+```

+### Pod annotation-based scraping

+The following scrape config uses the `__meta_*` labels added from the `kubernetes_sd_configs` for the `pod` role to filter for pods with certain annotations.

+To scrape certain pods, specify the port, path, and scheme through annotations for the pod and the following job scrapes only the address specified by the annotation:

+- `prometheus.io/scrape`: Enable scraping for this pod.

+- `prometheus.io/scheme`: If the metrics endpoint is secured, you need to set scheme to `https` and most likely set the TLS config.

+- `prometheus.io/path`: If the metrics path isn't /metrics, define it with this annotation.

+- `prometheus.io/port`: Specify a single port that you want to scrape.

+```yaml

+scrape_configs:

+ - job_name: 'kubernetespods-sample'

+ kubernetes_sd_configs:

+ - role: pod

+ relabel_configs:

+ # Scrape only pods with the annotation: prometheus.io/scrape = true

+ - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]

+ action: keep

+ regex: true

+ # If prometheus.io/path is specified, scrape this path instead of /metrics

+ - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]

+ action: replace

+ target_label: __metrics_path__

+ regex: (.+)

+ # If prometheus.io/port is specified, scrape this port instead of the default

+ - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]

+ action: replace

+ regex: ([^:]+)(?::\d+)?;(\d+)

+ replacement: $1:$2

+ target_label: __address__

+ # If prometheus.io/scheme is specified, scrape with this scheme instead of http

+ - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]

+ action: replace

+ regex: (http|https)

+ target_label: __scheme__

+ # Include the pod namespace as a label for each metric

+ - source_labels: [__meta_kubernetes_namespace]

+ action: replace

+ target_label: kubernetes_namespace

+ # Include the pod name as a label for each metric

+ - source_labels: [__meta_kubernetes_pod_name]

+ action: replace

+ target_label: kubernetes_pod_name

+ # [Optional] Include all pod labels as labels for each metric

+ - action: labelmap

+ regex: __meta_kubernetes_pod_label_(.+)

+```

+See the [Apply config file](prometheus-metrics-scrape-validate.md#deploy-config-file-as-configmap) section to create a configmap from the Prometheus config.

+## Next steps

+[Learn more about collecting Prometheus metrics](../essentials/prometheus-metrics-overview.md)

+ Title: Default Prometheus metrics configuration in Azure Monitor

+description: This article lists the default targets, dashboards, and recording rules for Prometheus metrics in Azure Monitor.

+# Default Prometheus metrics configuration in Azure Monitor

+This article lists the default targets, dashboards, and recording rules when you [configure Prometheus metrics to be scraped from an Azure Kubernetes Service (AKS) cluster](prometheus-metrics-enable.md) for any AKS cluster.

+## Scrape frequency

+ The default scrape frequency for all default targets and scrapes is 30 seconds.

+## Targets scraped by default

+- `cadvisor` (`job=cadvisor`)

+- `nodeexporter` (`job=node`)

+- `kubelet` (`job=kubelet`)

+- `kube-state-metrics` (`job=kube-state-metrics`)

+## Metrics collected from default targets

+The following metrics are collected by default from each default target. All other metrics are dropped through relabeling rules.

+ **cadvisor (job=cadvisor)**<br>

+ - `container_spec_cpu_period`

+ - `container_spec_cpu_quota`

+ - `container_cpu_usage_seconds_total`

+ - `container_memory_rss`

+ - `container_network_receive_bytes_total`

+ - `container_network_transmit_bytes_total`

+ - `container_network_receive_packets_total`

+ - `container_network_transmit_packets_total`

+ - `container_network_receive_packets_dropped_total`

+ - `container_network_transmit_packets_dropped_total`

+ - `container_fs_reads_total`

+ - `container_fs_writes_total`

+ - `container_fs_reads_bytes_total`

+ - `container_fs_writes_bytes_total`

+ - `container_memory_working_set_bytes`

+ - `container_memory_cache`

+ - `container_memory_swap`

+ - `container_cpu_cfs_throttled_periods_total`

+ - `container_cpu_cfs_periods_total`

+ - `container_memory_usage_bytes`

+ - `kubernetes_build_info"`

+ **kubelet (job=kubelet)**<br>

+ - `kubelet_volume_stats_used_bytes`

+ - `kubelet_node_name`

+ - `kubelet_running_pods`

+ - `kubelet_running_pod_count`

+ - `kubelet_running_containers`

+ - `kubelet_running_container_count`

+ - `volume_manager_total_volumes`

+ - `kubelet_node_config_error`

+ - `kubelet_runtime_operations_total`

+ - `kubelet_runtime_operations_errors_total`

+ - `kubelet_runtime_operations_duration_seconds` `kubelet_runtime_operations_duration_seconds_bucket` `kubelet_runtime_operations_duration_seconds_sum` `kubelet_runtime_operations_duration_seconds_count`

+ - `kubelet_pod_start_duration_seconds` `kubelet_pod_start_duration_seconds_bucket` `kubelet_pod_start_duration_seconds_sum` `kubelet_pod_start_duration_seconds_count`

+ - `kubelet_pod_worker_duration_seconds` `kubelet_pod_worker_duration_seconds_bucket` `kubelet_pod_worker_duration_seconds_sum` `kubelet_pod_worker_duration_seconds_count`

+ - `storage_operation_duration_seconds` `storage_operation_duration_seconds_bucket` `storage_operation_duration_seconds_sum` `storage_operation_duration_seconds_count`

+ - `storage_operation_errors_total`

+ - `kubelet_cgroup_manager_duration_seconds` `kubelet_cgroup_manager_duration_seconds_bucket` `kubelet_cgroup_manager_duration_seconds_sum` `kubelet_cgroup_manager_duration_seconds_count`

+ - `kubelet_pleg_relist_duration_seconds` `kubelet_pleg_relist_duration_seconds_bucket` `kubelet_pleg_relist_duration_sum` `kubelet_pleg_relist_duration_seconds_count`

+ - `kubelet_pleg_relist_interval_seconds` `kubelet_pleg_relist_interval_seconds_bucket` `kubelet_pleg_relist_interval_seconds_sum` `kubelet_pleg_relist_interval_seconds_count`

+ - `rest_client_requests_total`

+ - `rest_client_request_duration_seconds` `rest_client_request_duration_seconds_bucket` `rest_client_request_duration_seconds_sum` `rest_client_request_duration_seconds_count`

+ - `process_resident_memory_bytes`

+ - `process_cpu_seconds_total`

+ - `go_goroutines`

+ - `kubelet_volume_stats_capacity_bytes`

+ - `kubelet_volume_stats_available_bytes`

+ - `kubelet_volume_stats_inodes_used`

+ - `kubelet_volume_stats_inodes`

+ - `kubernetes_build_info"`

+ **nodexporter (job=node)**<br>

+ - `node_cpu_seconds_total`

+ - `node_memory_MemAvailable_bytes`

+ - `node_memory_Buffers_bytes`

+ - `node_memory_Cached_bytes`

+ - `node_memory_MemFree_bytes`

+ - `node_memory_Slab_bytes`

+ - `node_memory_MemTotal_bytes`

+ - `node_netstat_Tcp_RetransSegs`

+ - `node_netstat_Tcp_OutSegs`

+ - `node_netstat_TcpExt_TCPSynRetrans`

+ - `node_load1``node_load5`

+ - `node_load15`

+ - `node_disk_read_bytes_total`

+ - `node_disk_written_bytes_total`

+ - `node_disk_io_time_seconds_total`

+ - `node_filesystem_size_bytes`

+ - `node_filesystem_avail_bytes`

+ - `node_filesystem_readonly`

+ - `node_network_receive_bytes_total`

+ - `node_network_transmit_bytes_total`

+ - `node_vmstat_pgmajfault`

+ - `node_network_receive_drop_total`

+ - `node_network_transmit_drop_total`

+ - `node_disk_io_time_weighted_seconds_total`

+ - `node_exporter_build_info`

+ - `node_time_seconds`

+ - `node_uname_info"`

+ **kube-state-metrics (job=kube-state-metrics)**<br>

+ - `kube_job_status_succeeded`

+ - `kube_job_spec_completions`

+ - `kube_daemonset_status_desired_number_scheduled`

+ - `kube_daemonset_status_number_ready`

+ - `kube_deployment_status_replicas_ready`

+ - `kube_pod_container_status_last_terminated_reason`

+ - `kube_pod_container_status_waiting_reason`

+ - `kube_pod_container_status_restarts_total`

+ - `kube_node_status_allocatable`

+ - `kube_pod_owner`

+ - `kube_pod_container_resource_requests`

+ - `kube_pod_status_phase`

+ - `kube_pod_container_resource_limits`

+ - `kube_replicaset_owner`

+ - `kube_resourcequota`

+ - `kube_namespace_status_phase`

+ - `kube_node_status_capacity`

+ - `kube_node_info`

+ - `kube_pod_info`

+ - `kube_deployment_spec_replicas`

+ - `kube_deployment_status_replicas_available`

+ - `kube_deployment_status_replicas_updated`

+ - `kube_statefulset_status_replicas_ready`

+ - `kube_statefulset_status_replicas`

+ - `kube_statefulset_status_replicas_updated`

+ - `kube_job_status_start_time`

+ - `kube_job_status_active`

+ - `kube_job_failed`

+ - `kube_horizontalpodautoscaler_status_desired_replicas`

+ - `kube_horizontalpodautoscaler_status_current_replicas`

+ - `kube_horizontalpodautoscaler_spec_min_replicas`

+ - `kube_horizontalpodautoscaler_spec_max_replicas`

+ - `kubernetes_build_info`

+ - `kube_node_status_condition`

+ - `kube_node_spec_taint`

+ - `kube_pod_container_info`

+ - `kube_resource_labels` (ex - kube_pod_labels, kube_deployment_labels)

+ - `kube_resource_annotations` (ex - kube_pod_annotations, kube_deployment_annotations)

+## Default targets scraped for Windows

+Following Windows targets are configured to scrape, but scraping is not enabled (**disabled/OFF**) by default - meaning you don't have to provide any scrape job configuration for scraping these targets but they are disabled/OFF by default and you need to turn ON/enable scraping for these targets using [ama-metrics-settings-configmap](https://aka.ms/azureprometheus-addon-settings-configmap) under `default-scrape-settings-enabled` section

+Two default jobs can be run for Windows that scrape metrics required for the dashboards specific to Windows.

+- `windows-exporter` (`job=windows-exporter`)

+- `kube-proxy-windows` (`job=kube-proxy-windows`)

+> [!NOTE]

+> This requires applying or updating the `ama-metrics-settings-configmap` configmap and installing `windows-exporter` on all Windows nodes. For more information, see the [enablement document](./prometheus-metrics-enable.md#enable-prometheus-metric-collection).

+## Metrics scraped for Windows

+The following metrics are collected when windows-exporter and kube-proxy-windows are enabled.

+**windows-exporter (job=windows-exporter)**<br>

+ - `windows_system_system_up_time`

+ - `windows_cpu_time_total`

+ - `windows_memory_available_bytes`

+ - `windows_os_visible_memory_bytes`

+ - `windows_memory_cache_bytes`

+ - `windows_memory_modified_page_list_bytes`

+ - `windows_memory_standby_cache_core_bytes`

+ - `windows_memory_standby_cache_normal_priority_bytes`

+ - `windows_memory_standby_cache_reserve_bytes`

+ - `windows_memory_swap_page_operations_total`

+ - `windows_logical_disk_read_seconds_total`

+ - `windows_logical_disk_write_seconds_total`

+ - `windows_logical_disk_size_bytes`

+ - `windows_logical_disk_free_bytes`

+ - `windows_net_bytes_total`

+ - `windows_net_packets_received_discarded_total`

+ - `windows_net_packets_outbound_discarded_total`

+ - `windows_container_available`

+ - `windows_container_cpu_usage_seconds_total`

+ - `windows_container_memory_usage_commit_bytes`

+ - `windows_container_memory_usage_private_working_set_bytes`

+ - `windows_container_network_receive_bytes_total`

+ - `windows_container_network_transmit_bytes_total`

+**kube-proxy-windows (job=kube-proxy-windows)**<br>

+ - `kubeproxy_sync_proxy_rules_duration_seconds`

+ - `kubeproxy_sync_proxy_rules_duration_seconds_bucket`

+ - `kubeproxy_sync_proxy_rules_duration_seconds_sum`

+ - `kubeproxy_sync_proxy_rules_duration_seconds_count`

+ - `rest_client_requests_total`

+ - `rest_client_request_duration_seconds`

+ - `rest_client_request_duration_seconds_bucket`

+ - `rest_client_request_duration_seconds_sum`

+ - `rest_client_request_duration_seconds_count`

+ - `process_resident_memory_bytes`

+ - `process_cpu_seconds_total`

+ - `go_goroutines`

+## Dashboards

+The following default dashboards are automatically provisioned and configured by Azure Monitor managed service for Prometheus when you [link your Azure Monitor workspace to an Azure Managed Grafana instance](../essentials/azure-monitor-workspace-manage.md#link-a-grafana-workspace). Source code for these dashboards can be found in [this GitHub repository](https://aka.ms/azureprometheus-mixins). The below dashboards will be provisioned in the specified Azure Grafana instance under `Managed Prometheus` folder in Grafana. These are the standard open source community dashboards for monitoring Kubernetes clusters with Prometheus and Grafana.

+- `Kubernetes / Compute Resources / Cluster`

+- `Kubernetes / Compute Resources / Namespace (Pods)`

+- `Kubernetes / Compute Resources / Node (Pods)`

+- `Kubernetes / Compute Resources / Pod`

+- `Kubernetes / Compute Resources / Namespace (Workloads)`

+- `Kubernetes / Compute Resources / Workload`

+- `Kubernetes / Kubelet`

+- `Node Exporter / USE Method / Node`

+- `Node Exporter / Nodes`

+- `Kubernetes / Compute Resources / Cluster (Windows)`

+- `Kubernetes / Compute Resources / Namespace (Windows)`

+- `Kubernetes / Compute Resources / Pod (Windows)`

+- `Kubernetes / USE Method / Cluster (Windows)`

+- `Kubernetes / USE Method / Node (Windows)`

+## Recording rules

+The following default recording rules are automatically configured by Azure Monitor managed service for Prometheus when you [link your Azure Monitor workspace to an Azure Managed Grafana instance](../essentials/azure-monitor-workspace-manage.md#link-a-grafana-workspace). Source code for these recording rules can be found in [this GitHub repository](https://aka.ms/azureprometheus-mixins). These are the standard open source recording rules used in the dashboards above.

+- `cluster:node_cpu:ratio_rate5m`

+- `namespace_cpu:kube_pod_container_resource_requests:sum`

+- `namespace_cpu:kube_pod_container_resource_limits:sum`

+- `:node_memory_MemAvailable_bytes:sum`

+- `namespace_memory:kube_pod_container_resource_requests:sum`

+- `namespace_memory:kube_pod_container_resource_limits:sum`

+- `namespace_workload_pod:kube_pod_owner:relabel`

+- `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate`

+- `cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests`

+- `cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits`

+- `cluster:namespace:pod_memory:active:kube_pod_container_resource_requests`

+- `cluster:namespace:pod_memory:active:kube_pod_container_resource_limits`

+- `node_namespace_pod_container:container_memory_working_set_bytes`

+- `node_namespace_pod_container:container_memory_rss`

+- `node_namespace_pod_container:container_memory_cache`

+- `node_namespace_pod_container:container_memory_swap`

+- `instance:node_cpu_utilisation:rate5m`

+- `instance:node_load1_per_cpu:ratio`

+- `instance:node_memory_utilisation:ratio`

+- `instance:node_vmstat_pgmajfault:rate5m`

+- `instance:node_network_receive_bytes_excluding_lo:rate5m`

+- `instance:node_network_transmit_bytes_excluding_lo:rate5m`

+- `instance:node_network_receive_drop_excluding_lo:rate5m`

+- `instance:node_network_transmit_drop_excluding_lo:rate5m`

+- `instance_device:node_disk_io_time_seconds:rate5m`

+- `instance_device:node_disk_io_time_weighted_seconds:rate5m`

+- `instance:node_num_cpu:sum`

+- `node:windows_node:sum`

+- `node:windows_node_num_cpu:sum`

+- `:windows_node_cpu_utilisation:avg5m`

+- `node:windows_node_cpu_utilisation:avg5m`

+- `:windows_node_memory_utilisation:`

+- `:windows_node_memory_MemFreeCached_bytes:sum`

+- `node:windows_node_memory_totalCached_bytes:sum`

+- `:windows_node_memory_MemTotal_bytes:sum`

+- `node:windows_node_memory_bytes_available:sum`

+- `node:windows_node_memory_bytes_total:sum`

+- `node:windows_node_memory_utilisation:ratio`

+- `node:windows_node_memory_utilisation:`

+- `node:windows_node_memory_swap_io_pages:irate`

+- `:windows_node_disk_utilisation:avg_irate`

+- `node:windows_node_disk_utilisation:avg_irate`

+- `node:windows_node_filesystem_usage:`

+- `node:windows_node_filesystem_avail:`

+- `:windows_node_net_utilisation:sum_irate`

+- `node:windows_node_net_utilisation:sum_irate`

+- `:windows_node_net_saturation:sum_irate`

+- `node:windows_node_net_saturation:sum_irate`

+- `windows_pod_container_available`

+- `windows_container_total_runtime`

+- `windows_container_memory_usage`

+- `windows_container_private_working_set_usage`

+- `windows_container_network_received_bytes_total`

+- `windows_container_network_transmitted_bytes_total`

+- `kube_pod_windows_container_resource_memory_request`

+- `kube_pod_windows_container_resource_memory_limit`

+- `kube_pod_windows_container_resource_cpu_cores_request`

+- `kube_pod_windows_container_resource_cpu_cores_limit`

+- `namespace_pod_container:windows_container_cpu_usage_seconds_total:sum_rate`

+## Next steps

+[Customize scraping of Prometheus metrics](prometheus-metrics-scrape-configuration.md)

+ Title: Scrape Prometheus metrics at scale in Azure Monitor

+description: Guidance on performance that can be expected when collection metrics at high scale for Azure Monitor managed service for Prometheus.

+# Scrape Prometheus metrics at scale in Azure Monitor

+This article provides guidance on performance that can be expected when collection metrics at high scale for [Azure Monitor managed service for Prometheus](../essentials/prometheus-metrics-overview.md).

+## CPU and memory

+The CPU and memory usage is correlated with the number of bytes of each sample and the number of samples scraped. These benchmarks are based on the [default targets scraped](prometheus-metrics-scrape-default.md), volume of custom metrics scraped, and number of nodes, pods, and containers. These numbers are meant as a reference since usage can still vary significantly depending on the number of time series and bytes per metric.

+The upper volume limit per pod is currently about 3-3.5 million samples per minute, depending on the number of bytes per sample. This limitation is addressed when sharding is added in future.

+The agent consists of a deployment with one replica and DaemonSet for scraping metrics. The DaemonSet scrapes any node-level targets such as cAdvisor, kubelet, and node exporter. You can also configure it to scrape any custom targets at the node level with static configs. The replicaset scrapes everything else such as kube-state-metrics or custom scrape jobs that utilize service discovery.

+## Comparison between small and large cluster for replica

+| Scrape Targets | Samples Sent / Minute | Node Count | Pod Count | Prometheus-Collector CPU Usage (cores) |Prometheus-Collector Memory Usage (bytes) |

+|:|:|:|:|:|:|

+| default targets | 11,344 | 3 | 40 | 12.9 mc | 148 Mi |

+| default targets | 260,000 | 340 | 13000 | 1.10 c | 1.70 GB |

+| default targets<br>+ custom targets | 3.56 million | 340 | 13000 | 5.13 c | 9.52 GB |

+## Comparison between small and large cluster for DaemonSets

+| Scrape Targets | Samples Sent / Minute Total | Samples Sent / Minute / Pod | Node Count | Pod Count | Prometheus-Collector CPU Usage Total (cores) |Prometheus-Collector Memory Usage Total (bytes) | Prometheus-Collector CPU Usage / Pod (cores) |Prometheus-Collector Memory Usage / Pod (bytes) |

+|:|:|:|:|:|:|:|:|:|

+| default targets | 9,858 | 3,327 | 3 | 40 | 41.9 mc | 581 Mi | 14.7 mc | 189 Mi |

+| default targets | 2.3 million | 14,400 | 340 | 13000 | 805 mc | 305.34 GB | 2.36 mc | 898 Mi |

+For more custom metrics, the single pod behaves the same as the replica pod depending on the volume of custom metrics.

+## Schedule ama-metrics replica pod on a node pool with more resources

+A large volume of metrics per pod requires a large enough node to be able to handle the CPU and memory usage required. If the *ama-metrics* replica pod doesn't get scheduled on a node or nodepool that has enough resources, it might keep getting OOMKilled and go to CrashLoopBackoff. In order to overcome this issue, if you have a node or nodepool on your cluster that has higher resources (in [system node pool](../../aks/use-system-pools.md#system-and-user-node-pools)) and want to get the replica scheduled on that node, you can add the label `azuremonitor/metrics.replica.preferred=true` on the node and the replica pod will get scheduled on this node. Also you can create additional system pool(s), if needed, with larger nodes and can add the same label to their node(s) or nodepool. It's also better to add labels to [nodepool](../../aks/use-labels.md#updating-labels-on-existing-node-pools) rather than nodes so newer nodes in the same pool can also be used for scheduling when this label is applicable to all nodes in the pool.

+ ```

+ kubectl label nodes <node-name> azuremonitor/metrics.replica.preferred="true"

+ ```

+## Next steps

+- [Troubleshoot issues with Prometheus data collection](prometheus-metrics-troubleshoot.md).

+ Title: Create, validate and troubleshoot custom configuration file for Prometheus metrics in Azure Monitor

+description: Describes how to create custom configuration file Prometheus metrics in Azure Monitor and use validation tool before applying to Kubernetes cluster.

+# Create and validate custom configuration file for Prometheus metrics in Azure Monitor

+In addition to the default scrape targets that Azure Monitor Prometheus agent scrapes by default, use the following steps to provide more scrape config to the agent using a configmap. The Azure Monitor Prometheus agent doesn't understand or process operator [CRDs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) for scrape configuration, but instead uses the native Prometheus configuration as defined in [Prometheus configuration](https://aka.ms/azureprometheus-promioconfig-scrape).

+The three configmaps that can be used for custom target scraping are -

+- ama-metrics-prometheus-config - When a configmap with this name is created, scrape jobs defined in it are run from the Azure monitor metrics replica pod running in the cluster.

+- ama-metrics-prometheus-config-node - When a configmap with this name is created, scrape jobs defined in it are run from each **Linux** DaemonSet pod running in the cluster. For more information, see [Advanced Setup](prometheus-metrics-scrape-configuration.md#advanced-setup-configure-custom-prometheus-scrape-jobs-for-the-daemonset).

+- ama-metrics-prometheus-config-node-windows - When a configmap with this name is created, scrape jobs defined in it are run from each **windows** DaemonSet. For more information, see [Advanced Setup](prometheus-metrics-scrape-configuration.md#advanced-setup-configure-custom-prometheus-scrape-jobs-for-the-daemonset).

+## Create Prometheus configuration file

+One easier way to author Prometheus scrape configuration jobs:

+- Step:1 Use a config file (yaml) to author/define scrape jobs

+- Step:2 Validate the scrape config file using a custom tool (as specified in this article) and then convert that configfile to configmap

+- Step:3 Deploy the scrape config file as configmap to your clusters.

+

+Doing this way is easier to author yaml config (which is extremely space sensitive), and not add unintended spaces by directly authoring scrape config inside config map.

+Create a Prometheus scrape configuration file named `prometheus-config`. For more information, see [configuration tips and examples](prometheus-metrics-scrape-configuration.md#prometheus-configuration-tips-and-examples) which gives more details on authoring scrape config for Prometheus. You can also refer to [Prometheus.io](https://aka.ms/azureprometheus-promio) scrape configuration [reference](https://aka.ms/azureprometheus-promioconfig-scrape). Your config file lists the scrape configs under the section `scrape_configs` section and can optionally use the global section for setting the global `scrape_interval`, `scrape_timeout`, and `external_labels`.

+> [!TIP]

+> Changes to global section will impact the default configs and the custom config.

+Here is a sample Prometheus scrape config file:

+```

+global:

+ scrape_interval: 30s

+scrape_configs:

+- job_name: my_static_config

+ scrape_interval: 60s

+ static_configs:

+ - targets: ['my-static-service.svc.cluster.local:1234']

+- job_name: prometheus_example_app

+ scheme: http

+ kubernetes_sd_configs:

+ - role: service

+ relabel_configs:

+ - source_labels: [__meta_kubernetes_service_name]

+ action: keep

+ regex: "prometheus-example-service"

+```

+## Validate the scrape config file

+The agent uses a custom `promconfigvalidator` tool to validate the Prometheus config given to it through the configmap. If the config isn't valid, then the custom configuration given gets rejected by the addon agent. Once you have your Prometheus config file, you can *optionally* use the `promconfigvalidator` tool to validate your config before creating a configmap that the agent consumes.

+The `promconfigvalidator` tool is shipped inside the Azure Monitor metrics addon pod(s). You can use any of the `ama-metrics-node-*` pods in `kube-system` namespace in your cluster to download the tool for validation. Use `kubectl cp` to download the tool and its configuration:

+```

+for podname in $(kubectl get pods -l rsName=ama-metrics -n=kube-system -o json | jq -r '.items[].metadata.name'); do kubectl cp -n=kube-system "${podname}":/opt/promconfigvalidator ./promconfigvalidator; kubectl cp -n=kube-system "${podname}":/opt/microsoft/otelcollector/collector-config-template.yml ./collector-config-template.yml; chmod 500 promconfigvalidator; done

+```

+After copying the executable and the yaml, locate the path of your Prometheus configuration file that you authored. Then replace `<config path>` in the command and run the validator with the command:

+```

+./promconfigvalidator/promconfigvalidator --config "<config path>" --otelTemplate "./promconfigvalidator/collector-config-template.yml"

+```

+Running the validator generates the merged configuration file `merged-otel-config.yaml` if no path is provided with the optional `output` parameter. Don't use this autogenerated merged file as config to the metrics collector agent, as it's only used for tool validation and debugging purposes.

+### Deploy config file as configmap

+Your custom Prometheus configuration file is consumed as a field named `prometheus-config` inside metrics addon configmap(s) `ama-metrics-prometheus-config` (or) `ama-metrics-prometheus-config-node` (or) `ama-metrics-prometheus-config-node-windows` in the `kube-system` namespace. You can create a configmap from the scrape config file you created above, by renaming your Prometheus configuration file to `prometheus-config` (with no file extension) and running one or more of the following commands, depending on which configmap you want to create for your custom scrape job(s) config.

+Ex;- to create configmap to be used by replicsset

+```

+kubectl create configmap ama-metrics-prometheus-config --from-file=prometheus-config -n kube-system

+```

+This creates a configmap named `ama-metrics-prometheus-config` in `kube-system` namespace. The Azure Monitor metrics replica pod restarts in 30-60 secs to apply the new config. To see if there any issues with the config validation, processing, or merging, you can look at the `ama-metrics` replica pods

+Ex;- to create configmap to be used by linux DaemonSet

+```

+kubectl create configmap ama-metrics-prometheus-config-node --from-file=prometheus-config -n kube-system

+```

+This creates a configmap named `ama-metrics-prometheus-config-node` in `kube-system` namespace. Every Azure Monitor metrics Linux DaemonSet pod restarts in 30-60 secs to apply the new config. To see if there any issues with the config validation, processing, or merging, you can look at the `ama-metrics-node` linux deamonset pods

+Ex;- to create configmap to be used by windows DaemonSet

+```

+kubectl create configmap ama-metrics-prometheus-config-node-windows --from-file=prometheus-config -n kube-system

+```

+This creates a configmap named `ama-metrics-prometheus-config-node-windows` in `kube-system` namespace. Every Azure Monitor metrics Windows DaemonSet pod restarts in 30-60 secs to apply the new config. To see if there any issues with the config validation, processing, or merging, you can look at the `ama-metrics-win-node` windows deamonset pods

+*Ensure that the Prometheus config file is named `prometheus-config` before running the following command since the file name is used as the configmap setting name.*

+This creates a configmap named `ama-metrics-prometheus-config` in `kube-system` namespace. The Azure Monitor metrics pod restarts to apply the new config. To see if there any issues with the config validation, processing, or merging, you can look at the `ama-metrics` pods.

+A sample of the `ama-metrics-prometheus-config` configmap is [here](https://github.com/Azure/prometheus-collector/blob/main/otelcollector/configmaps/ama-metrics-prometheus-config-configmap.yaml).

+### Troubleshooting

+If you successfully created the configmap (ama-metrics-prometheus-config or ama-metrics-prometheus-config-node) in the **kube-system** namespace and still don't see the custom targets being scraped, check for errors in the **replica pod** logs for **ama-metrics-prometheus-config** configmap or **DaemonSet pod** logs for **ama-metrics-prometheus-config-node** configmap) using *kubectl logs* and make sure there are no errors in the *Start Merging Default and Custom Prometheus Config* section with prefix *prometheus-config-merger*

+## Next steps

+- [Learn more about collecting Prometheus metrics](../essentials/prometheus-metrics-overview.md).

+ Title: Troubleshoot collection of Prometheus metrics in Azure Monitor

+description: Steps that you can take if you aren't collecting Prometheus metrics as expected.

+# Troubleshoot collection of Prometheus metrics in Azure Monitor

+Follow the steps in this article to determine the cause of Prometheus metrics not being collected as expected in Azure Monitor.

+Replica pod scrapes metrics from `kube-state-metrics` and custom scrape targets in the `ama-metrics-prometheus-config` configmap. DaemonSet pods scrape metrics from the following targets on their respective node: `kubelet`, `cAdvisor`, `node-exporter`, and custom scrape targets in the `ama-metrics-prometheus-config-node` configmap. The pod that you want to view the logs and the Prometheus UI for it depends on which scrape target you're investigating.

+## Metrics Throttling

+In the Azure portal, navigate to your Azure Monitor Workspace. Go to `Metrics` and verify that the metrics `Active Time Series % Utilization` and `Events Per Minute Ingested % Utilization` are below 100%.

+If either of them are more than 100%, ingestion into this workspace is being throttled. In the same workspace, navigate to `New Support Request` to create a request to increase the limits. Select the issue type as `Service and subscription limits (quotas)` and the quota type as `Managed Prometheus`.

+## Intermittent gaps in metric data collection

+During node updates, you may see a 1 to 2 minute gap in metric data for metrics collected from our cluster level collector. This gap is because the node it runs on is being updated as part of a normal update process. It affects cluster-wide targets such as kube-state-metrics and custom application targets that are specified. It occurs when your cluster is updated manually or via autoupdate. This behavior is expected and occurs due to the node it runs on being updated. None of our recommended alert rules are affected by this behavior.

+## Pod status

+Check the pod status with the following command:

+```

+kubectl get pods -n kube-system | grep ama-metrics

+```

+- There should be one `ama-metrics-xxxxxxxxxx-xxxxx` replica pod, one `ama-metrics-ksm-*` pod, and an `ama-metrics-node-*` pod for each node on the cluster.

+- Each pod state should be `Running` and have an equal number of restarts to the number of configmap changes that have been applied:

+If each pod state is `Running` but one or more pods have restarts, run the following command:

+```

+kubectl describe pod <ama-metrics pod name> -n kube-system

+```

+- This command provides the reason for the restarts. Pod restarts are expected if configmap changes have been made. If the reason for the restart is `OOMKilled`, the pod can't keep up with the volume of metrics. See the scale recommendations for the volume of metrics.

+If the pods are running as expected, the next place to check is the container logs.

+## Container logs

+View the container logs with the following command:

+```

+kubectl logs <ama-metrics pod name> -n kube-system -c prometheus-collector

+```

+ At startup, any initial errors are printed in red, while warnings are printed in yellow. (Viewing the colored logs requires at least PowerShell version 7 or a linux distribution.)

+- Verify if there's an issue with getting the authentication token:

+ - The message *No configuration present for the AKS resource* gets logged every 5 minutes.

+ - The pod restarts every 15 minutes to try again with the error: *No configuration present for the AKS resource*.

+ - If so, check that the Data Collection Rule and Data Collection Endpoint exist in your resource group.

+ - Also verify that the Azure Monitor Workspace exists.

+ - Verify that you don't have a private AKS cluster and that it's not linked to an Azure Monitor Private Link Scope for any other service. This scenario is currently not supported.

+- Verify there are no errors with parsing the Prometheus config, merging with any default scrape targets enabled, and validating the full config.

+- If you did include a custom Prometheus config, verify that it's recognized in the logs. If not:

+ - Verify that your configmap has the correct name: `ama-metrics-prometheus-config` in the `kube-system` namespace.

+ - Verify that in the configmap your Prometheus config is under a section called `prometheus-config` under `data` like shown here:

+ ```

+ kind: ConfigMap

+ apiVersion: v1

+ metadata:

+ name: ama-metrics-prometheus-config

+ namespace: kube-system

+ data:

+ prometheus-config: |-

+ scrape_configs:

+ - job_name: <your scrape job here>

+ ```

+- Verify there are no errors from `MetricsExtension` regarding authenticating with the Azure Monitor workspace.

+- Verify there are no errors from the `OpenTelemetry collector` about scraping the targets.

+Run the following command:

+```

+kubectl logs <ama-metrics pod name> -n kube-system -c addon-token-adapter

+```

+- This command shows an error if there's an issue with authenticating with the Azure Monitor workspace. The example below shows logs with no issues:

+ :::image type="content" source="media/prometheus-metrics-troubleshoot/addon-token-adapter.png" alt-text="Screenshot showing addon token log." lightbox="media/prometheus-metrics-troubleshoot/addon-token-adapter.png" :::

+If there are no errors in the logs, the Prometheus interface can be used for debugging to verify the expected configuration and targets being scraped.

+## Prometheus interface

+Every `ama-metrics-*` pod has the Prometheus Agent mode User Interface available on port 9090. Port-forward into either the replica pod or one of the daemon set pods to check the config, service discovery and targets endpoints as described here to verify the custom configs are correct, the intended targets have been discovered for each job, and there are no errors with scraping specific targets.

+Run the command `kubectl port-forward <ama-metrics pod> -n kube-system 9090`.

+- Open a browser to the address `127.0.0.1:9090/config`. This user interface has the full scrape configuration. Verify all jobs are included in the config.

+- Go to `127.0.0.1:9090/service-discovery` to view the targets discovered by the service discovery object specified and what the relabel_configs have filtered the targets to be. For example, when missing metrics from a certain pod, you can find if that pod was discovered and what its URI is. You can then use this URI when looking at the targets to see if there are any scrape errors.

+- Go to `127.0.0.1:9090/targets` to view all jobs, the last time the endpoint for that job was scraped, and any errors

+If there are no issues and the intended targets are being scraped, you can view the exact metrics being scraped by enabling debug mode.

+## Debug mode

+The metrics addon can be configured to run in debug mode by changing the configmap setting `enabled` under `debug-mode` to `true` by following the instructions [here](prometheus-metrics-scrape-configuration.md#debug-mode). This mode can affect performance and should only be enabled for a short time for debugging purposes.

+When enabled, all Prometheus metrics that are scraped are hosted at port 9090. Run the following command:

+```

+kubectl port-forward <ama-metrics pod name> -n kube-system 9090

+```

+Go to `127.0.0.1:9090/metrics` in a browser to see if the metrics were scraped by the OpenTelemetry Collector. This user interface can be accessed for every `ama-metrics-*` pod. If metrics aren't there, there could be an issue with the metric or label name lengths or the number of labels. Also check for exceeding the ingestion quota for Prometheus metrics as specified in this article.

+## Metric names, label names & label values

+Agent based scraping currently has the limitations in the following table:

+| Property | Limit |

+|:|:|

+| Label name length | Less than or equal to 511 characters. When this limit is exceeded for any time-series in a job, the entire scrape job fails, and metrics get dropped from that job before ingestion. You can see up=0 for that job and also target Ux shows the reason for up=0. |

+| Label value length | Less than or equal to 1023 characters. When this limit is exceeded for any time-series in a job, the entire scrape fails, and metrics get dropped from that job before ingestion. You can see up=0 for that job and also target Ux shows the reason for up=0. |

+| Number of labels per time series | Less than or equal to 63. When this limit is exceeded for any time-series in a job, the entire scrape job fails, and metrics get dropped from that job before ingestion. You can see up=0 for that job and also target Ux shows the reason for up=0. |

+| Metric name length | Less than or equal to 511 characters. When this limit is exceeded for any time-series in a job, only that particular series get dropped. MetricextensionConsoleDebugLog has traces for the dropped metric. |

+| Label names with different casing | Two labels within the same metric sample, with different casing is treated as having duplicate labels and are dropped when ingested. For example, the time series `my_metric{ExampleLabel="label_value_0", examplelabel="label_value_1}` is dropped due to duplicate labels since `ExampleLabel` and `examplelabel` are seen as the same label name. |

+## Check ingestion quota on Azure Monitor workspace

+If you see metrics missed, you can first check if the ingestion limits are being exceeded for your Azure Monitor workspace. In the Azure portal, you can check the current usage for any Azure monitor Workspace. You can see current usage metrics under `Metrics` menu for the Azure Monitor workspace. Following utilization metrics are available as standard metrics for each Azure Monitor workspace.

+- Active Time Series - The number of unique time series recently ingested into the workspace over the previous 12 hours

+- Active Time Series Limit - The limit on the number of unique time series that can be actively ingested into the workspace

+- Active Time Series % Utilization - The percentage of current active time series being utilized

+- Events Per Minute Ingested - The number of events (samples) per minute recently received

+- Events Per Minute Ingested Limit - The maximum number of events per minute that can be ingested before getting throttled

+- Events Per Minute Ingested % Utilization - The percentage of current metric ingestion rate limit being util

+Refer to [service quotas and limits](../service-limits.md#prometheus-metrics) for default quotas and also to understand what can be increased based on your usage. You can request quota increase for Azure Monitor workspaces using the `Support Request` menu for the Azure Monitor workspace. Ensure you include the ID, internal ID and Location/Region for the Azure Monitor workspace in the support request, which you can find in the `Properties' menu for the Azure Monitor workspace in the Azure portal.

+## Next steps

+- [Check considerations for collecting metrics at high scale](prometheus-metrics-scrape-scale.md).

+ Title: Remote-write in Azure Monitor Managed Service for Prometheus using Azure Active Directory

+description: Describes how to configure remote-write to send data from self-managed Prometheus running in your Kubernetes cluster running on-premises or in another cloud using Azure Active Directory authentication.

+# Configure remote write for Azure Monitor managed service for Prometheus using Azure Active Directory authentication

+This article describes how to configure [remote-write](prometheus-remote-write.md) to send data from self-managed Prometheus running in your AKS cluster or Azure Arc-enabled Kubernetes cluster using Azure Active Directory authentication.

+## Cluster configurations

+This article applies to the following cluster configurations:

+- Azure Kubernetes service (AKS)

+- Azure Arc-enabled Kubernetes cluster

+- Kubernetes cluster running in another cloud or on-premises

+> [!NOTE]

+> For Azure Kubernetes service (AKS) or Azure Arc-enabled Kubernetes cluster, managed identify authentication is recommended. See [Azure Monitor managed service for Prometheus remote write - managed identity](prometheus-remote-write-managed-identity.md).

+## Prerequisites

+See prerequisites at [Azure Monitor managed service for Prometheus remote write](prometheus-remote-write.md#prerequisites).

+## Create Azure Active Directory application

+Follow the procedure at [Register an application with Azure AD and create a service principal](../../active-directory/develop/howto-create-service-principal-portal.md#register-an-application-with-azure-ad-and-create-a-service-principal) to register an application for Prometheus remote-write and create a service principal.

+## Get the client ID of the Azure Active Directory application.

+1. From the **Azure Active Directory** menu in Azure Portal, select **App registrations**.

+2. Locate your application and note the client ID.

+ :::image type="content" source="media/prometheus-remote-write-active-directory/application-client-id.png" alt-text="Screenshot showing client ID of Azure Active Directory application." lightbox="media/prometheus-remote-write-active-directory/application-client-id.png":::

+## Assign Monitoring Metrics Publisher role on the data collection rule to the application

+The application requires the *Monitoring Metrics Publisher* role on the data collection rule associated with your Azure Monitor workspace.

+1. From the menu of your Azure Monitor Workspace account, click the **Data collection rule** to open the **Overview** page for the data collection rule.

+ :::image type="content" source="media/prometheus-remote-write-managed-identity/azure-monitor-account-data-collection-rule.png" alt-text="Screenshot showing data collection rule used by Azure Monitor workspace." lightbox="media/prometheus-remote-write-managed-identity/azure-monitor-account-data-collection-rule.png":::

+2. Click on **Access control (IAM)** in the **Overview** page for the data collection rule.

+ :::image type="content" source="media/prometheus-remote-write-managed-identity/azure-monitor-account-access-control.png" alt-text="Screenshot showing Access control (IAM) menu item on the data collection rule Overview page." lightbox="media/prometheus-remote-write-managed-identity/azure-monitor-account-access-control.png":::

+3. Click **Add** and then **Add role assignment**.

+ :::image type="content" source="media/prometheus-remote-write-managed-identity/data-collection-rule-add-role-assignment.png" alt-text="Screenshot showing adding a role assignment on Access control pages." lightbox="media/prometheus-remote-write-managed-identity/data-collection-rule-add-role-assignment.png":::

+4. Select **Monitoring Metrics Publisher** role and click **Next**.

+ :::image type="content" source="media/prometheus-remote-write-managed-identity/add-role-assignment.png" alt-text="Screenshot showing list of role assignments." lightbox="media/prometheus-remote-write-managed-identity/add-role-assignment.png":::

+5. Select **User, group, or service principal** and then click **Select members**. Select the application that you created and click **Select**.

+ :::image type="content" source="media/prometheus-remote-write-active-directory/select-application.png" alt-text="Screenshot showing selection of application." lightbox="media/prometheus-remote-write-active-directory/select-application.png":::

+6. Click **Review + assign** to complete the role assignment.

+## Create an Azure key vault and generate certificate

+1. If you don't already have an Azure key vault, then create a new one using the guidance at [Create a vault](../../key-vault/general/quick-create-portal.md#create-a-vault).

+2. Create a certificate using the guidance at [Add a certificate to Key Vault](../../key-vault/certificates/quick-create-portal.md#add-a-certificate-to-key-vault).

+3. Download the newly generated certificate in CER format using the guidance at [Export certificate from Key Vault](../../key-vault/certificates/quick-create-portal.md#export-certificate-from-key-vault).

+## Add certificate to the Azure Active Directory application

+1. From the menu for your Azure Active Directory application, select **Certificates & secrets**.

+2. Click **Upload certificate** and select the certificate that you downloaded.

+ :::image type="content" source="media/prometheus-remote-write-active-directory/upload-certificate.png" alt-text="Screenshot showing upload of certificate for Azure Active Directory application." lightbox="media/prometheus-remote-write-active-directory/upload-certificate.png":::

+> [!WARNING]

+> Certificates have an expiration date, and it's the responsibility of the user to keep these certificates valid.

+## Add CSI driver and storage for cluster

+> [!NOTE]

+> Azure Key Vault CSI driver configuration is just one of the ways to get certificate mounted on the pod. The remote write container only needs a local path to a certificate in the pod for the setting `AZURE_CLIENT_CERTIFICATE_PATH` value in the [Deploy Side car and configure remote write on the Prometheus server](#deploy-side-car-and-configure-remote-write-on-the-prometheus-server) step below.

+This step is only required if you didn't enable Azure Key Vault Provider for Secrets Store CSI Driver when you created your cluster.

+1. Run the following Azure CLI command to enable Azure Key Vault Provider for Secrets Store CSI Driver for your cluster.

+ ```azurecli

+ az aks enable-addons --addons azure-keyvault-secrets-provider --name <aks-cluster-name> --resource-group <resource-group-name>

+ ```

+2. Run the following commands to give the identity access to the key vault.

+ ```azurecli

+ # show client id of the managed identity of the cluster

+ az aks show -g <resource-group> -n <cluster-name> --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv

+ # set policy to access keys in your key vault

+ az keyvault set-policy -n <keyvault-name> --key-permissions get --spn <identity-client-id>

+ # set policy to access secrets in your key vault

+ az keyvault set-policy -n <keyvault-name> --secret-permissions get --spn <identity-client-id>

+

+ # set policy to access certs in your key vault

+ az keyvault set-policy -n <keyvault-name> --certificate-permissions get --spn <identity-client-id>

+ ```

+3. Create a *SecretProviderClass* by saving the following YAML to a file named *secretproviderclass.yml*. Replace the values for `userAssignedIdentityID`, `keyvaultName`, `tenantId` and the objects to retrieve from your key vault. See [Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver](../../aks/csi-secrets-store-identity-access.md) for details on values to use.

+ ```yml

+ # This is a SecretProviderClass example using user-assigned identity to access your key vault

+ apiVersion: secrets-store.csi.x-k8s.io/v1

+ kind: SecretProviderClass

+ metadata:

+ name: azure-kvname-user-msi

+ spec:

+ provider: azure

+ parameters:

+ usePodIdentity: "false"

+ useVMManagedIdentity: "true" # Set to true for using managed identity

+ userAssignedIdentityID: <client-id> # Set the clientID of the user-assigned managed identity to use

+ keyvaultName: <key-vault-name> # Set to the name of your key vault

+ cloudName: "" # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud

+ objects: |

+ array:

+ - |

+ objectName: <name-of-cert>

+ objectType: secret # object types: secret, key, or cert

+ objectFormat: pfx

+ objectEncoding: base64

+ objectVersion: ""

+ tenantId: <tenant-id> # The tenant ID of the key vault

+ ```

+4. Apply the *SecretProviderClass* by running the following command on your cluster.

+ ```

+ kubectl apply -f secretproviderclass.yml

+ ```

+## Deploy Side car and configure remote write on the Prometheus server

+1. Copy the YAML below and save to a file. This YAML assumes you're using 8081 as your listening port. Modify that value if you use a different port.

+ ```yml

+ prometheus:

+ prometheusSpec:

+ externalLabels:

+ cluster: <CLUSTER-NAME>

+

+ ## Azure Managed Prometheus currently exports some default mixins in Grafana.

+ ## These mixins are compatible with data scraped by Azure Monitor agent on your

+ ## Azure Kubernetes Service cluster. These mixins aren't compatible with Prometheus

+ ## metrics scraped by the Kube Prometheus stack.

+ ## To make these mixins compatible, uncomment the remote write relabel configuration below:

+ ## writeRelabelConfigs:

+ ## - sourceLabels: [metrics_path]

+ ## regex: /metrics/cadvisor

+ ## targetLabel: job

+ ## replacement: cadvisor

+ ## action: replace

+ ## - sourceLabels: [job]

+ ## regex: 'node-exporter'

+ ## targetLabel: job

+ ## replacement: node

+ ## action: replace

+ ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

+ remoteWrite:

+ - url: 'http://localhost:8081/api/v1/write'

+

+ # Additional volumes on the output StatefulSet definition.

+ # Required only for AAD based auth

+ volumes:

+ - name: secrets-store-inline

+ csi:

+ driver: secrets-store.csi.k8s.io

+ readOnly: true

+ volumeAttributes:

+ secretProviderClass: azure-kvname-user-msi

+ containers:

+ - name: prom-remotewrite

+ image: <CONTAINER-IMAGE-VERSION>

+ imagePullPolicy: Always

+ # Required only for AAD based auth

+ volumeMounts:

+ - name: secrets-store-inline

+ mountPath: /mnt/secrets-store

+ readOnly: true

+ ports:

+ - name: rw-port

+ containerPort: 8081

+ livenessProbe:

+ httpGet:

+ path: /health

+ port: rw-port

+ initialDelaySeconds: 10

+ timeoutSeconds: 10

+ readinessProbe:

+ httpGet:

+ path: /ready

+ port: rw-port

+ initialDelaySeconds: 10

+ timeoutSeconds: 10

+ env:

+ - name: INGESTION_URL

+ value: '<INGESTION_URL>'

+ - name: LISTENING_PORT

+ value: '8081'

+ - name: IDENTITY_TYPE

+ value: aadApplication

+ - name: AZURE_CLIENT_ID

+ value: '<APP-REGISTRATION-CLIENT-ID>'

+ - name: AZURE_TENANT_ID

+ value: '<TENANT-ID>'

+ - name: AZURE_CLIENT_CERTIFICATE_PATH

+ value: /mnt/secrets-store/<CERT-NAME>

+ - name: CLUSTER

+ value: '<CLUSTER-NAME>'

+ ```

+2. Replace the following values in the YAML.

+ | Value | Description |

+ |:|:|

+ | `<CLUSTER-NAME>` | Name of your AKS cluster |

+ | `<CONTAINER-IMAGE-VERSION>` | `mcr.microsoft.com/azuremonitor/prometheus/promdev/prom-remotewrite:prom-remotewrite-20230505.1`<br>This is the remote write container image version. |

+ | `<INGESTION-URL>` | **Metrics ingestion endpoint** from the **Overview** page for the Azure Monitor workspace |

+ | `<APP-REGISTRATION -CLIENT-ID> ` | Client ID of your application |

+ | `<TENANT-ID> ` | Tenant ID of the Azure Active Directory application |

+ | `<CERT-NAME>` | Name of the certificate |

+ | `<CLUSTER-NAME>` | Name of the cluster Prometheus is running on |

+

+3. Open Azure Cloud Shell and upload the YAML file.

+4. Use helm to apply the YAML file to update your Prometheus configuration with the following CLI commands.

+ ```azurecli

+ # set context to your cluster

+ az aks get-credentials -g <aks-rg-name> -n <aks-cluster-name>

+

+ # use helm to update your remote write config

+ helm upgrade -f <YAML-FILENAME>.yml prometheus prometheus-community/kube-prometheus-stack -namespace <namespace where Prometheus pod resides>

+ ```

+## Verification and troubleshooting

+See [Azure Monitor managed service for Prometheus remote write](prometheus-remote-write.md#verify-remote-write-is-working-correctly).

+## Next steps

+= [Setup Grafana to use Managed Prometheus as a data source](../essentials/prometheus-grafana.md).

+- [Learn more about Azure Monitor managed service for Prometheus](../essentials/prometheus-metrics-overview.md).

+ Title: Configure remote write for Azure Monitor managed service for Prometheus using Microsoft Azure Active Directory pod identity (preview)

+description: Configure remote write for Azure Monitor managed service for Prometheus using Azure AD pod identity (preview)

+# Configure remote write for Azure Monitor managed service for Prometheus using Microsoft Azure Active Directory pod identity (preview)

+> [!NOTE]

+> The remote write sidecar should only be configured via the following steps only if the AKS cluster already has the Azure AD pod enabled. This approach is not recommended as AAD pod identity has been deprecated to be replace by [Azure Workload Identity](/azure/active-directory/workload-identities/workload-identities-overview)

+To configure remote write for Azure Monitor managed service for Prometheus using Azure AD pod identity, follow the steps below.

+1. Create user assigned identity or use an existing user assigned managed identity. For information on creating the managed identity, see [Configure remote write for Azure Monitor managed service for Prometheus using managed identity authentication](./prometheus-remote-write-managed-identity.md#get-the-client-id-of-the-user-assigned-identity).

+1. Assign the `Managed Identity Operator` and `Virtual Machine Contributor` roles to the managed identity created/used in the previous step.

+ ```azurecli

+ az role assignment create --role "Managed Identity Operator" --assignee <managed identity clientID> --scope <NodeResourceGroupResourceId>

+

+ az role assignment create --role "Virtual Machine Contributor" --assignee <managed identity clientID> --scope <Node ResourceGroup Id>

+ ```

+ The node resource group of the AKS cluster contains resources that you will require for other steps in this process. This resource group has the name MC_\<AKS-RESOURCE-GROUP\>_\<AKS-CLUSTER-NAME\>_\<REGION\>. You can locate it from the Resource groups menu in the Azure portal.

+1. Grant user-assigned managed identity `Monitoring Metrics Publisher` roles.

+ ```azurecli

+ az role assignment create --role "Monitoring Metrics Publisher" --assignee <managed identity clientID> --scope <NodeResourceGroupResourceId>

+ ```

+1. Create AzureIdentityBinding

+ The user assigned managed identity requires identity binding in order to be used as a pod identity. Run the following commands:

+

+ Copy the following YAML to the `aadpodidentitybinding.yaml` file.

+ ```yml

+ apiVersion: "aadpodidentity.k8s.io/v1"

+ kind: AzureIdentityBinding

+ metadata:

+ name: demo1-azure-identity-binding

+ spec:

+ AzureIdentity: ΓÇ£<AzureIdentityName>ΓÇ¥

+ Selector: ΓÇ£<AzureIdentityBindingSelector>ΓÇ¥

+ ```

+ Run the following command:

+ ```azurecli

+ kubectl create -f aadpodidentitybinding.yaml

+ ```

+

+1. Add a `aadpodidbinding` label to the Prometheus pod.

+ The `aadpodidbinding` label must be added to the Prometheus pod for the pod identity to take effect. This can be achieved by updating the `deployment.yaml` or injecting labels while deploying the sidecar as mentioned in the next step.

+1. Deploy side car and configure remote write on the Prometheus server.

+ 1. Copy the YAML below and save to a file.

+ ```yml

+ prometheus:

+ prometheusSpec:

+ podMetadata:

+ labels:

+ aadpodidbinding: <AzureIdentityBindingSelector>

+ externalLabels:

+ cluster: <AKS-CLUSTER-NAME>

+ remoteWrite:

+ - url: 'http://localhost:8081/api/v1/write'

+ containers:

+ - name: prom-remotewrite

+ image: <CONTAINER-IMAGE-VERSION>

+ imagePullPolicy: Always

+ ports:

+ - name: rw-port

+ containerPort: 8081

+ livenessProbe:

+ httpGet:

+ path: /health

+ port: rw-port

+ initialDelaySeconds: 10

+ timeoutSeconds: 10

+ readinessProbe:

+ httpGet:

+ path: /ready

+ port: rw-port

+ initialDelaySeconds: 10

+ timeoutSeconds: 10

+ env:

+ - name: INGESTION_URL

+ value: <INGESTION_URL>

+ - name: LISTENING_PORT

+ value: '8081'

+ - name: IDENTITY_TYPE

+ value: userAssigned

+ - name: AZURE_CLIENT_ID

+ value: <MANAGED-IDENTITY-CLIENT-ID>

+ # Optional parameter

+ - name: CLUSTER

+ value: <CLUSTER-NAME>

+ ```

+ b. Use helm to apply the YAML file to update your Prometheus configuration with the following CLI commands.

+

+ ```azurecli

+ # set context to your cluster

+ az aks get-credentials -g <aks-rg-name> -n <aks-cluster-name>

+ # use helm to update your remote write config

+ helm upgrade -f <YAML-FILENAME>.yml prometheus prometheus-community/kube-prometheus-stack --namespace <namespace where Prometheus pod resides>

+ ```

+ Title: Remote-write in Azure Monitor Managed Service for Prometheus using managed identity

+description: Describes how to configure remote-write to send data from self-managed Prometheus running in your AKS cluster or Azure Arc-enabled Kubernetes cluster using managed identity authentication.

+# Configure remote write for Azure Monitor managed service for Prometheus using managed identity authentication

+This article describes how to configure [remote-write](prometheus-remote-write.md) to send data from self-managed Prometheus running in your AKS cluster or Azure Arc-enabled Kubernetes cluster using managed identity authentication. You either use an existing identity created by AKS or [create one of your own](../../active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities.md). Both options are described here.

+## Cluster configurations

+This article applies to the following cluster configurations:

+- Azure Kubernetes service (AKS)

+- Azure Arc-enabled Kubernetes cluster

+> [!NOTE]

+> For a Kubernetes cluster running in another cloud or on-premises, see [Azure Monitor managed service for Prometheus remote write - Azure Active Directory](prometheus-remote-write-active-directory.md).

+## Prerequisites

+See prerequisites at [Azure Monitor managed service for Prometheus remote write](prometheus-remote-write.md#prerequisites).

+## Locate AKS node resource group

+The node resource group of the AKS cluster contains resources that you will require for other steps in this process. This resource group has the name `MC_<AKS-RESOURCE-GROUP>_<AKS-CLUSTER-NAME>_<REGION>`. You can locate it from the **Resource groups** menu in the Azure portal. Start by making sure that you can locate this resource group since other steps below will refer to it.

+## Get the client ID of the user assigned identity

+You will require the client ID of the identity that you're going to use. Note this value for use in later steps in this process.

+Get the **Client ID** from the **Overview** page of your [managed identity](../../active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities.md).

+Instead of creating your own ID, you can use one of the identities created by AKS, which are listed in [Use a managed identity in Azure Kubernetes Service](../../aks/use-managed-identity.md). This article uses the `Kubelet` identity. The name of this identity will be `<AKS-CLUSTER-NAME>-agentpool` and located in the node resource group of the AKS cluster.

+## Assign Monitoring Metrics Publisher role on the data collection rule to the managed identity

+The managed identity requires the *Monitoring Metrics Publisher* role on the data collection rule associated with your Azure Monitor workspace.

+1. From the menu of your Azure Monitor Workspace account, click the **Data collection rule** to open the **Overview** page for the data collection rule.

+ :::image type="content" source="media/prometheus-remote-write-managed-identity/azure-monitor-account-data-collection-rule.png" alt-text="Screenshot showing data collection rule used by Azure Monitor workspace." lightbox="media/prometheus-remote-write-managed-identity/azure-monitor-account-data-collection-rule.png":::

+2. Click on **Access control (IAM)** in the **Overview** page for the data collection rule.

+ :::image type="content" source="media/prometheus-remote-write-managed-identity/azure-monitor-account-access-control.png" alt-text="Screenshot showing Access control (IAM) menu item on the data collection rule Overview page." lightbox="media/prometheus-remote-write-managed-identity/azure-monitor-account-access-control.png":::

+3. Click **Add** and then **Add role assignment**.

+ :::image type="content" source="media/prometheus-remote-write-managed-identity/data-collection-rule-add-role-assignment.png" alt-text="Screenshot showing adding a role assignment on Access control pages." lightbox="media/prometheus-remote-write-managed-identity/data-collection-rule-add-role-assignment.png":::

+4. Select **Monitoring Metrics Publisher** role and click **Next**.

+ :::image type="content" source="media/prometheus-remote-write-managed-identity/add-role-assignment.png" alt-text="Screenshot showing list of role assignments." lightbox="media/prometheus-remote-write-managed-identity/add-role-assignment.png":::

+5. Select **Managed Identity** and then click **Select members**. Choose the subscription the user assigned identity is located in and then select **User-assigned managed identity**. Select the User Assigned Identity that you're going to use and click **Select**.

+ :::image type="content" source="media/prometheus-remote-write-managed-identity/select-managed-identity.png" alt-text="Screenshot showing selection of managed identity." lightbox="media/prometheus-remote-write-managed-identity/select-managed-identity.png":::

+6. Click **Review + assign** to complete the role assignment.

+## Grant AKS cluster access to the identity

+This step isn't required if you're using an AKS identity since it will already have access to the cluster.

+> [!IMPORTANT]

+> You must have owner/user access administrator access on the cluster.

+1. Identify the virtual machine scale sets in the [node resource group](#locate-aks-node-resource-group) for your AKS cluster.

+ :::image type="content" source="media/prometheus-remote-write-managed-identity/resource-group-details-virtual-machine-scale-sets.png" alt-text="Screenshot showing virtual machine scale sets in the node resource group." lightbox="media/prometheus-remote-write-managed-identity/resource-group-details-virtual-machine-scale-sets.png":::

+2. Run the following command in Azure CLI for each virtual machine scale set.

+ ```azurecli

+ az vmss identity assign -g <AKS-NODE-RESOURCE-GROUP> -n <AKS-VMSS-NAME> --identities <USER-ASSIGNED-IDENTITY-RESOURCE-ID>

+ ```

+## Deploy Side car and configure remote write on the Prometheus server

+1. Copy the YAML below and save to a file. This YAML assumes you're using 8081 as your listening port. Modify that value if you use a different port.

+ ```yml

+ prometheus:

+ prometheusSpec:

+ externalLabels:

+ cluster: <AKS-CLUSTER-NAME>

+

+ ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

+ remoteWrite:

+ - url: 'http://localhost:8081/api/v1/write'

+

+ ## Azure Managed Prometheus currently exports some default mixins in Grafana.

+ ## These mixins are compatible with Azure Monitor agent on your Azure Kubernetes Service cluster.

+ ## However, these mixins aren't compatible with Prometheus metrics scraped by the Kube Prometheus stack.

+ ## In order to make these mixins compatible, uncomment remote write relabel configuration below:

+

+ ## writeRelabelConfigs:

+ ## - sourceLabels: [metrics_path]

+ ## regex: /metrics/cadvisor

+ ## targetLabel: job

+ ## replacement: cadvisor

+ ## action: replace

+ ## - sourceLabels: [job]

+ ## regex: 'node-exporter'

+ ## targetLabel: job

+ ## replacement: node

+ ## action: replace

+

+ containers:

+ - name: prom-remotewrite

+ image: <CONTAINER-IMAGE-VERSION>

+ imagePullPolicy: Always

+ ports:

+ - name: rw-port

+ containerPort: 8081

+ livenessProbe:

+ httpGet:

+ path: /health

+ port: rw-port

+ initialDelaySeconds: 10

+ timeoutSeconds: 10

+ readinessProbe:

+ httpGet:

+ path: /ready

+ port: rw-port

+ initialDelaySeconds: 10

+ timeoutSeconds: 10

+ env:

+ - name: INGESTION_URL

+ value: <INGESTION_URL>

+ - name: LISTENING_PORT

+ value: '8081'

+ - name: IDENTITY_TYPE

+ value: userAssigned

+ - name: AZURE_CLIENT_ID

+ value: <MANAGED-IDENTITY-CLIENT-ID>

+ # Optional parameter

+ - name: CLUSTER

+ value: <CLUSTER-NAME>

+ ```

+2. Replace the following values in the YAML.

+ | Value | Description |

+ |:|:|

+ | `<AKS-CLUSTER-NAME>` | Name of your AKS cluster |

+ | `<CONTAINER-IMAGE-VERSION>` | `mcr.microsoft.com/azuremonitor/prometheus/promdev/prom-remotewrite:prom-remotewrite-20230505.1`<br>This is the remote write container image version. |

+ | `<INGESTION-URL>` | **Metrics ingestion endpoint** from the **Overview** page for the Azure Monitor workspace |

+ | `<MANAGED-IDENTITY-CLIENT-ID>` | **Client ID** from the **Overview** page for the managed identity |

+ | `<CLUSTER-NAME>` | Name of the cluster Prometheus is running on |

+

+3. Open Azure Cloud Shell and upload the YAML file.

+4. Use helm to apply the YAML file to update your Prometheus configuration with the following CLI commands.

+ ```azurecli

+ # set context to your cluster

+ az aks get-credentials -g <aks-rg-name> -n <aks-cluster-name>

+

+ # use helm to update your remote write config

+ helm upgrade -f <YAML-FILENAME>.yml prometheus prometheus-community/kube-prometheus-stack --namespace <namespace where Prometheus pod resides>

+ ```

+## Verification and troubleshooting

+See [Azure Monitor managed service for Prometheus remote write](prometheus-remote-write.md#verify-remote-write-is-working-correctly).

+## Next steps

+- [Setup Grafana to use Managed Prometheus as a data source](../essentials/prometheus-grafana.md).

+- [Learn more about Azure Monitor managed service for Prometheus](../essentials/prometheus-metrics-overview.md).

+ Title: Remote-write in Azure Monitor Managed Service for Prometheus

+description: Describes how to configure remote-write to send data from self-managed Prometheus running in your AKS cluster or Azure Arc-enabled Kubernetes cluster

+# Azure Monitor managed service for Prometheus remote write

+Azure Monitor managed service for Prometheus is intended to be a replacement for self managed Prometheus so you don't need to manage a Prometheus server in your Kubernetes clusters. You may also choose to use the managed service to centralize data from self-managed Prometheus clusters for long term data retention and to create a centralized view across your clusters. In this case, you can use [remote_write](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage) to send data from your self-managed Prometheus into the Azure managed service.

+## Architecture

+Azure Monitor provides a reverse proxy container (Azure Monitor [side car container](/azure/architecture/patterns/sidecar)) that provides an abstraction for ingesting Prometheus remote write metrics and helps in authenticating packets. The Azure Monitor side car container currently supports User Assigned Identity and Azure Active Directory (Azure AD) based authentication to ingest Prometheus remote write metrics to Azure Monitor workspace.

+## Prerequisites

+- You must have self-managed Prometheus running on your AKS cluster. For example, see [Using Azure Kubernetes Service with Grafana and Prometheus](https://techcommunity.microsoft.com/t5/apps-on-azure-blog/using-azure-kubernetes-service-with-grafana-and-prometheus/ba-p/3020459).

+- You used [Kube-Prometheus Stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) when you set up Prometheus on your AKS cluster.

+- Data for Azure Monitor managed service for Prometheus is stored in an [Azure Monitor workspace](../essentials/azure-monitor-workspace-overview.md). You must [create a new workspace](../essentials/azure-monitor-workspace-manage.md#create-an-azure-monitor-workspace) if you don't already have one.

+## Configure remote write

+The process for configuring remote write depends on your cluster configuration and the type of authentication that you use.

+- **Managed identity** is recommended for Azure Kubernetes service (AKS) and Azure Arc-enabled Kubernetes cluster. See [Azure Monitor managed service for Prometheus remote write - managed identity](prometheus-remote-write-managed-identity.md)

+- **Azure Active Directory** can be used for Azure Kubernetes service (AKS) and Azure Arc-enabled Kubernetes cluster and is required for Kubernetes cluster running in another cloud or on-premises. See [Azure Monitor managed service for Prometheus remote write - Azure Active Directory](prometheus-remote-write-active-directory.md)

+> [!NOTE]

+> Whether you use Managed Identity or Azure Active Directory to enable permissions for ingesting data, these settings take some time to take effect. When following the steps below to verify that the setup is working please allow up to 10-15 minutes for the authorization settings needed to ingest data to complete.

+## Verify remote write is working correctly

+Use the following methods to verify that Prometheus data is being sent into your Azure Monitor workspace.

+### kubectl commands

+Use the following command to view your container log. Remote write data is flowing if the output has non-zero value for `avgBytesPerRequest` and `avgRequestDuration`.

+```azurecli

+kubectl logs <Prometheus-Pod-Name> <Azure-Monitor-Side-Car-Container-Name>

+# example: kubectl logs prometheus-prometheus-kube-prometheus-prometheus-0 prom-remotewrite

+```

+The output from this command should look similar to the following:

+```

+time="2022-11-02T21:32:59Z" level=info msg="Metric packets published in last 1 minute" avgBytesPerRequest=19713 avgRequestDurationInSec=0.023 failedPublishing=0 successfullyPublished=122

+```

+### PromQL queries

+Use PromQL queries in Grafana and verify that the results return expected data. See [getting Grafana setup with Managed Prometheus](../essentials/prometheus-grafana.md) to configure Grafana

+## Troubleshoot remote write

+### No data is flowing

+If remote data isn't flowing, run the following command which will indicate the errors if any in the remote write container.

+```azurecli

+kubectl --namespace <Namespace> describe pod <Prometheus-Pod-Name>

+```

+### Container keeps restarting

+A container regularly restarting is likely due to misconfiguration of the container. Run the following command to view the configuration values set for the container. Verify the configuration values especially `AZURE_CLIENT_ID` and `IDENTITY_TYPE`.

+```azureccli

+kubectl get pod <Prometheus-Pod-Name> -o json | jq -c '.spec.containers[] | select( .name | contains("<Azure-Monitor-Side-Car-Container-Name>"))'

+```

+The output from this command should look similar to the following:

+```

+{"env":[{"name":"INGESTION_URL","value":"https://my-azure-monitor-workspace.eastus2-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-00000000000000000/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2021-11-01-preview"},{"name":"LISTENING_PORT","value":"8081"},{"name":"IDENTITY_TYPE","value":"userAssigned"},{"name":"AZURE_CLIENT_ID","value":"00000000-0000-0000-0000-00000000000"}],"image":"mcr.microsoft.com/azuremonitor/prometheus/promdev/prom-remotewrite:prom-remotewrite-20221012.2","imagePullPolicy":"Always","name":"prom-remotewrite","ports":[{"containerPort":8081,"name":"rw-port","protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","name":"kube-api-access-vbr9d","readOnly":true}]}

+```

+### Hitting your ingestion quota limit

+With remote write you will typically get started using the remote write endpoint shown on the Azure Monitor workspace overview page. Behind the scenes, this uses a system Data Collection Rule (DCR) and system Data Collection Endpoint (DCE). These resources have an ingestion limit covered in the [Azure Monitor service limits](../service-limits.md#prometheus-metrics) document. You may hit these limits if you setup remote write for several clusters all sending data into the same endpoint in the same Azure Monitor workspace. If this is the case you can [create additional DCRs and DCEs](https://aka.ms/prometheus/remotewrite/dcrartifacts) and use them to spread out the ingestion loads across a few ingestion endpoints.

+The INGESTION-URL uses the following format:

+https\://\<Metrics-Ingestion-URL>/dataCollectionRules/\<DCR-Immutable-ID>/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2021-11-01-preview

+Metrics-Ingestion-URL: can be obtained by viewing DCE JSON body with API version 2021-09-01-preview or newer.

+DCR-Immutable-ID: can be obtained by viewing DCR JSON body or running the following command in the Azure CLI:

+```azureccli

+az monitor data-collection rule show --name "myCollectionRule" --resource-group "myResourceGroup"

+```

+

+## Next steps

+- [Setup Grafana to use Managed Prometheus as a data source](../essentials/prometheus-grafana.md).

+- [Learn more about Azure Monitor managed service for Prometheus](../essentials/prometheus-metrics-overview.md).

+The "Audit" category is a subset of "All", but the Azure portal and REST API consider them separate settings. Selecting "All" does collect all audit logs regardless of if the "Audit" category is also selected.

+The following image shows the logs category groups on the add diagnostics settings page.

+ :::image type="content" source="./media/diagnostic-settings/audit-category-group.png" alt-text="A screenshot showing the logs category groups.":::

+> [!NOTE]

+> Enabling *Audit* for Azure SQL Database does not enable auditing for Azure SQL Database. To enable database auditing, you have to enable it from the auditing blade for Azure Database.

+ Title: Data retention and archive in Azure Monitor Logs

+# Data retention and archive in Azure Monitor Logs

+Azure Monitor Logs retains data in two states:

+* **Interactive retention**: Lets you retain Analytics logs for [interactive queries](../logs/get-started-queries.md) of up to 2 years.

+* **Archive**: Lets you keep older, less used data in your workspace at a reduced cost. You can access data in the archived state by using [search jobs](../logs/search-jobs.md) and [restore](../logs/restore.md). You can currently keep data in archived state for up to 7 years. In the coming months, it will be possible to extend archive to 12 years.

+This article describes how to configure data retention and archiving.

+Each workspace has a default retention setting that's applied to all tables. You can configure a different retention setting on individual tables.

+When you shorten an existing retention setting, Azure Monitor waits 30 days before removing the data, so you can revert the change and avoid data loss in the event of an error in configuration. You can [purge data](#purge-retained-data) immediately when required.

+When you increase the retention setting, the new retention period applies to all data that's already been ingested into the table and hasn't yet been purged or removed.

+If you change the archive settings on a table with existing data, the relevant data in the table is also affected immediately. For example, you might have an existing table with 180 days of interactive retention and no archive period. You decide to change the retention setting to 90 days of interactive retention without changing the total retention period of 180 days. Log Analytics immediately archives any data that's older than 90 days and none of the data is deleted.

+## Permissions required

+| Action | Permissions required |

+|:-|:|

+| Configure data retention and archive policies for a Log Analytics workspace | `Microsoft.OperationalInsights/workspaces/write` and `microsoft.operationalinsights/workspaces/tables/write` permissions to the Log Analytics workspace, as provided by the [Log Analytics Contributor built-in role](./manage-access.md#log-analytics-contributor), for example |

+| Get the retention and archive policy by table for a Log Analytics workspace | `Microsoft.OperationalInsights/workspaces/tables/read` permissions to the Log Analytics workspace, as provided by the [Log Analytics Reader built-in role](./manage-access.md#log-analytics-reader), for example |

+| Purge data from a Log Analytics workspace | `Microsoft.OperationalInsights/workspaces/purge/action` permissions to the Log Analytics workspace, as provided by the [Log Analytics Contributor built-in role](./manage-access.md#log-analytics-contributor), for example |

+| Set data retention for a classic Application Insights resource | `microsoft.insights/components/write` permissions to the classic Application Insights resource, as provided by the [Application Insights Component Contributor built-in role](../../role-based-access-control/built-in-roles.md#application-insights-component-contributor), for example |

+| Purge data from a classic Application Insights resource | `Microsoft.Insights/components/purge/action` permissions to the classic Application Insights resource, as provided by the [Application Insights Component Contributor built-in role](../../role-based-access-control/built-in-roles.md#application-insights-component-contributor), for example |

+## Configure the default workspace retention

+You can set a Log Analytics workspace's default retention in the Azure portal to 30, 31, 60, 90, 120, 180, 270, 365, 550, and 730 days. You can apply a different setting to specific tables by [configuring retention and archive at the table level](#configure-retention-and-archive-at-the-table-level). If you're on the *free* tier, you need to upgrade to the paid tier to change the data retention period.

+To set the default workspace retention:

+## Configure retention and archive at the table level

+By default, all tables in your workspace inherit the workspace's interactive retention setting and have no archive. You can modify the retention and archive settings of individual tables, except for workspaces in the legacy Free Trial pricing tier.

+The Analytics log data plan includes 30 days of interactive retention. You can increase the interactive retention period to up to 730 days at an [additional cost](https://azure.microsoft.com/pricing/details/monitor/). If needed, you can reduce the interactive retention period to as little as four days using the API or CLI. However, since 30 days are included in the ingestion price, lowering the retention period below 30 days doesn't reduce costs. You can set the archive period to a total retention time of up to 2,556 days (seven years).

+> [!NOTE]

+> In the coming months, new settings will enable retaining data for up to 12 years.

+|properties.retentionInDays | integer | The table's data retention in days. This value can be between 4 and 730. <br/>Setting this property to null applies the workspace retention period. For a Basic Logs table, the value is always 8. |

+## Get retention and archive settings by table

+To get the retention setting of a particular table (in this example, `SecurityEvent`), call the **Tables - Get** API:

+To get all table-level retention settings in your workspace, don't set a table name.

+To get the retention setting of a particular table, run the [az monitor log-analytics workspace table show](/cli/azure/monitor/log-analytics/workspace/table#az-monitor-log-analytics-workspace-table-show) command.

+To get the retention setting of a particular table, run the [Get-AzOperationalInsightsTable](/powershell/module/az.operationalinsights/get-azoperationalinsightstable) cmdlet.

+If you set the data retention to 30 days, you can purge older data immediately by using the `immediatePurgeDataOn30Days` parameter in Azure Resource Manager. The purge functionality is useful when you need to remove personal data immediately. The immediate purge functionality isn't available through the Azure portal.

+Workspaces with a 30-day retention might keep data for 31 days if you don't set the `immediatePurgeDataOn30Days` parameter.

+## Tables with unique retention periods

+Tables related to Application Insights resources also keep data for 90 days at no charge. You can adjust the retention of each of these tables individually:

+This configuration may result in cost savings if it helps you reach a [commitment tier](#commitment-tiers), which provides a discount to your ingestion charges. For example, consider an organization that has operational data and security data each ingesting about 50 GB per day. Combining the data in the same workspace would allow a commitment tier at 100 GB per day. That scenario would provide a 15% discount for Azure Monitor and a 50% discount for Microsoft Sentinel.

+You can configure default [data retention and archive settings](data-retention-archive.md) for a workspace or [configure different settings for each table](data-retention-archive.md#configure-retention-and-archive-at-the-table-level). You might require different settings for different sets of data in a particular table. If so, you need to separate that data into different workspaces, each with unique retention settings.

+1. Azure NetApp Files uses an LDAP client configuration to make a connection attempt to the AD DS or Azure AD DS LDAP server that is specified in the [Azure NetApp Files AD configuration](create-active-directory-connections.md).

+1. If the TCP connection over the defined AD DS or Azure AD DS LDAP service port is successful, then the Azure NetApp Files LDAP client attempts to ΓÇ£bindΓÇ¥ (sign in) to the AD DS or Azure AD DS LDAP server (domain controller) by using the defined credentials in the LDAP client configuration.

+1. If the bind is successful, then the Azure NetApp Files LDAP client uses the RFC 2307bis LDAP schema to make an LDAP search query to the AD DS or Azure AD DS LDAP server (domain controller).

+1. If the request is successful, then user and group attributes are [cached for future use](configure-ldap-extended-groups.md#considerations). This operation improves the performance of subsequent LDAP queries associated with the cached user or group attributes. It also reduces the load on the AD DS or Azure AD DS LDAP server.

+* [Understand NFS group memberships and supplemental groups](network-file-system-group-memberships.md)

+* [Understand the use of LDAP with Azure NetApp Files](lightweight-directory-access-protocol.md)

+* To understand Azure NetApp Files dual protocols and related considerations, see the [Dual Protocols section in Understand NAS protocols in Azure NetApp Files](network-attached-storage-protocols.md#dual-protocols).

+* [Considerations for Azure NetApp Files dual-protocol volumes](network-attached-storage-protocols.md#considerations-for-azure-netapp-files-dual-protocol-volumes)

+ Title: Understand dual-protocol security style and permission behaviors in Azure NetApp Files | Microsoft Docs

+description: This article helps you understand dual-protocol security style and permission when you use Azure NetApp Files.

+documentationcenter: ''

+editor: ''

+ms.assetid:

+ na

+# Understand dual-protocol security style and permission behaviors in Azure NetApp Files

+SMB and NFS use different permission models for user and group access. As a result, an Azure NetApp File volume must be configured to honor the desired permission model for protocol access. For NFS-only environments, the decision is simple ΓÇô use UNIX security styles. For SMB-only environments, use NTFS security styles.

+If NFS and SMB on the same datasets (dual-protocol) are required, then the decision should be made based on two questions:

+* What protocol will users manage permissions from the most?

+* What is the desired permission management endpoint? In other words, do users require the ability to manage permissions from NFS clients or Windows clients? Or both?

+Volume security styles can really be considered permission styles, where the desired style of ACL management is the deciding factor.

+> [!NOTE]

+> Security styles are chosen at volume creation. Once the security style has been chosen, it cannot be changed.

+## About Azure NetApp Files volume security styles

+There are two main choices for volume security styles in Azure NetApp Files:

+**UNIX** - The UNIX security style provides UNIX-style permissions, such as basic POSIX mode bits (Owner/Group/Everyone access with standard Read/Write/Execute permissions, such as 0755) and NFSv4.x ACLs. POSIX ACLs aren't supported.

+**NTFS** - The NTFS security style provides identical functionality as [standard Windows NTFS permissions](/windows/security/identity-protection/access-control/access-control), with granular user and groups in ACLs and detailed security/audit permissions.

+In a dual-protocol NAS environment, only one security permission style can be active. You should evaluate considerations for each security style before choosing one.

+| Security style | Considerations |

+| - | - |

+| UNIX | - Windows clients can only set UNIX permission attributes through SMBs that map to UNIX attributes (Read/Write/Execute only; no special permissions). <br> - NFSv4.x ACLs don't have GUI management. Management is done only via CLI using [nfs4_getfacl and nfs4_setfacl commands](https://manpages.debian.org/testing/nfs4-acl-tools/https://docsupdatetracker.net/index.html). <br> - If a file or folder has NFSv4.x ACLs, the Windows security properties tab can't display them. <br> |

+| NTFS | - UNIX clients can't set attributes through NFS via commands such as `chown/chmod`. <br> - NFS clients show only approximated NTFS permissions when using `ls` commands. For instance, if a user has a permission in a Windows NTFS ACL that can't be cleanly translated into a POSIX mode bit (such as traverse directory), it's translated into the closest POSIX mode-bit value (such as `1` for execute). <br> |

+The selection of volume security style determines how the name mapping for a user is performed. This operation is the core piece of how dual-protocol volumes maintain predictable permissions regardless of protocol in use.

+Use the following table as a decision matrix for selecting the proper volume security styles.

+| Security style | Mostly NFS | Mostly SMB | Need for granular security |

+| - | - | - | - |

+| UNIX | X | - | X (using NFSv4.x ACLs) |

+| NTFS | - | X | X |

+## How name mapping works in Azure NetApp Files

+In Azure NetApp Files, only users are authenticated and mapped. Groups aren't mapped. Instead, group memberships are determined by using the user identity.

+When a user attempts to access an Azure NetApp Files volume, that attempt passes along an identity to the service. That identity includes a user name and unique numeric identifier (UID number for NFSv3, name string for NFSv4.1, SID for SMB). Azure NetApp Files uses that identity to authenticate against a configured name service to verify the identity of the user.

+* LDAP search for numeric IDs is used to look up a user name in Active Directory.

+* Name strings use LDAP search to look up a user name and the client and server consult the [configured ID domain for NFSv4.1](azure-netapp-files-configure-nfsv41-domain.md) to ensure the match.

+* Windows users are queried using standard Windows RPC calls to Active Directory.

+* Group memberships are also queried, and everything is added to a credential cache for faster processing on subsequent requests to the volume.

+* Currently, custom local users aren't supported for use with Azure NetApp Files. Only users in Active Directory can be used with dual protocols.

+* Currently, the only local users that can be used with dual-protocol volumes are root and the `nfsnobody` user.

+After a user name is authenticated and validated by Azure NetApp Files, the next step for dual-protocol volume authentication is the mapping of user names for UNIX and Windows interoperability.

+A volume's security style determines how a name mapping takes place in Azure NetApp Files. Windows and UNIX permission semantics are different. If a name mapping can't be performed, then authentication fails, and access to a volume from a client is denied. A common scenario where this situation occurs is when NFSv3 access is attempted to a volume with NTFS security style. The initial access request from NFSv3 comes to Azure NetApp Files as a numeric UID. If a user named `user1` with a numeric ID of `1001` tries to access the NFSv3 mount, the authentication request arrives as numeric ID `1001`. Azure NetApp Files then takes numeric ID `1001` and attempts to resolve `1001` to a user name. This user name is required for mapping to a valid Windows user, because the NTFS permissions on the volume will contain Windows user names instead of a numeric ID. Azure NetApp Files will use the configured name service server (LDAP) to search for the user name. If the user name can't be found, then authentication fails, and access is denied. This operation is by design in order to prevent unwanted access to files and folders.

+## Name mapping based on security style

+The direction in which the name mapping occurs in Azure NetApp Files (Windows to UNIX, or UNIX to Windows) depends not only on the protocol being used but also the security style of a volume. A Windows client always requires a Windows-to-UNIX name mapping to allow access, but it doesn't always need a matching UNIX user name. If no valid UNIX user name exists in the configured name service server, Azure NetApp Files provides a fallback default UNIX user with the numeric UID of `65534` to allow initial authentication for SMB connections. After that, file and folder permissions will control access. Because `65534` generally corresponds with the `nfsnobody` user, access is limited in most cases. Conversely, an NFS client only needs to use a UNIX-to-Windows name mapping if the NTFS security style is in use. There's no default Windows user in Azure NetApp Files. As such, if a valid Windows user that matches the requesting name can't be found, access will be denied.

+The following table breaks down the different name mapping permutations and how they behave depending on protocol in use.

+| Protocol | Security style | Name mapping direction | Permissions applied |

+| - | - | - | - |

+| SMB | UNIX | Windows to UNIX | UNIX <br> (mode-bits or NFSv4.x ACLs) |

+| SMB | NTFS | Windows to UNIX | NTFS ACLs <br> (based on Windows SID accessing share) |

+| NFSv3 | UNIX | None | UNIX <br> (mode-bits or NFSv4.x ACLs*) |

+| NFSv4.x | UNIX | Numeric ID to UNIX user name | UNIX <br> (mode-bits or NFSv4.x ACLs) |

+| NFS3/4.x | NTFS | UNIX to Windows | NTFS ACLs <br> (based on mapped Windows user SID) |

+> [!NOTE]

+> NFSv4.x ACLs can be applied using an NFSv4.x administrative client and honored by NFSv3 clients by [switching between protocols](convert-nfsv3-nfsv41.md).

+Name-mapping rules in Azure NetApp Files can currently be controlled only by using LDAP. There is no option to create explicit name mapping rules within the service.

+## Name services with dual-protocol volumes

+Regardless of what NAS protocol is used, dual-protocol volumes use name-mapping concepts to handle permissions properly. As such, name services play a critical role in maintaining functionality in environments that use both SMB and NFS for access to volumes.

+Name services act as identity sources for users and groups accessing NAS volumes. This operation includes Active Directory, which can act as a source for both Windows and UNIX users and groups using both standard domain services and LDAP functionality.

+Name services aren't a hard requirement but highly recommended for Azure NetApp Files dual-protocol volumes. There's no concept of creation of custom local users and groups within the service. As such, to have proper authentication and accurate user and group owner information across protocols, LDAP is a necessity. If you have only a handful of users and you don't need to populate accurate user and group identity information, then consider using the [Allow local NFS users with LDAP to access a dual-protocol volume functionality](create-volumes-dual-protocol.md#allow-local-nfs-users-with-ldap-to-access-a-dual-protocol-volume). Keep in mind that enabling this functionality disables the [extended group functionality](configure-ldap-extended-groups.md#considerations).

+### When clients, name services, and storage reside in different areas

+In some cases, NAS clients might live in a segmented network with multiple interfaces that have isolated connections to the storage and name services.

+One such example is if your storage resides in Azure NetApp Files, while your NAS clients and domain services all reside on-premises (such as a [hub-spoke architecture in Azure](/azure/architecture/reference-architectures/hybrid-networking/hub-spoke)). In those scenarios, you would need to provide network access to both the NAS clients and the name services.

+The following figure shows an example of that kind of configuration.

+## Next steps

+* [Understand the use of LDAP with Azure NetApp Files](lightweight-directory-access-protocol.md)

+* [Create a dual-protocol volume for Azure NetApp Files](create-volumes-dual-protocol.md)

+* [Azure NetApp Files NFS FAQ](faq-nfs.md)

+* [Azure NetApp Files SMB FAQ](faq-smb.md)

+ Title: Understand the use of LDAP with Azure NetApp Files | Microsoft Learn

+description: This article helps you understand how Azure NetApp Files uses lightweight directory access protocol (LDAP).

+documentationcenter: ''

+editor: ''

+ms.assetid:

+ na

+# Understand the use of LDAP with Azure NetApp Files

+Lightweight directory access protocol (LDAP) is a standard directory access protocol that was developed by an international committee called the Internet Engineering Task Force (IETF). LDAP is intended to provide a general-purpose, network-based directory service that you can use across heterogeneous platforms to locate network objects.

+LDAP models define how to communicate with the LDAP directory store, how to find an object in the directory, how to describe the objects in the store, and the security that is used to access the directory. LDAP allows customization and extension of the objects that are described in the store. Therefore, you can use an LDAP store to store many types of diverse information. Many of the initial LDAP deployments focused on the use of LDAP as a directory store for applications such as email and web applications and to store employee information. Many companies are replacing or have replaced Network Information Service (NIS) with LDAP as a network directory store.

+An LDAP server provides UNIX user and group identities for use with NAS volumes. In Azure NetApp Files, Active Directory is the only currently supported LDAP server that can be used. This support includes both Active Directory Domain Services (AD DS) and Azure Active Directory Domain Services (Azure AD DS).

+LDAP requests can be broken down into two main operations.

+* **LDAP binds** are logins to the LDAP server from an LDAP client. The bind is used to authenticate to the LDAP server with read-only access to perform LDAP lookups. Azure NetApp Files acts as an LDAP client.

+* **LDAP lookups** are used to query the directory for user and group information, such as names, numeric IDs, home directory paths, login shell paths, group memberships and more.

+LDAP can store the following information that is used in dual-protocol NAS access:

+* User names

+* Group names

+* Numeric user IDs (UIDs) and group IDs (GIDs)

+* Home directories

+* Login shell

+* Netgroups, DNS names, and IP addresses

+* Group membership

+Currently, Azure NetApp Files only uses LDAP for user and group information ΓÇô no netgroup or host information.

+LDAP offers various benefits for your UNIX users and groups as an identity source.

+* **LDAP is future-proof.**

+ As more NFS clients add support for NFSv4.x, NFSv4.x ID domains that contain an up-to-date list of users and groups accessible from clients and storage are needed to ensure optimal security and guaranteed access when access is defined. Having an identity-management server that provides one-to-one name mappings for SMB and NFS users alike greatly simplifies life for storage administrators, not just in the present, but for years to come.

+* **LDAP is scalable.**

+ LDAP servers offer the ability to contain millions of user and group objects, and with Microsoft Active Directory, multiple servers can be used to replicate across multiple sites for both performance and resiliency scale.

+* **LDAP is secure.**

+ LDAP offers security in the form of how a storage system can connect to the LDAP server to make requests for user information. LDAP servers offer the following bind levels:

+ * Anonymous (disabled by default in Microsoft Active Directory; not supported in Azure NetApp Files)

+ * Simple password (plain text passwords; not supported in Azure NetApp Files)

+ * [Simple Authentication and Security Layer (SASL)](https://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml#:~:text=The%20Simple%20Authentication%20and%20Security,support%20to%20connection%2Dbased%20protocols.) ΓÇô Encrypted bind methods including TLS, SSL, Kerberos, and so on. Azure NetApp Files supports LDAP over TLS, LDAP signing (using Kerberos), LDAP over SSL.

+* **LDAP is robust.**

+ NIS, NIS+, and local files offer basic information such UID, GID, password, home directories, and so on. However, LDAP offers those attributes and many more. The additional attributes that LDAP uses makes dual-protocol management much more integrated with LDAP versus NIS. Only LDAP is supported as an external name service for identity management with Azure NetApp Files.

+* **Microsoft Active Directory is built on LDAP.**

+ By default, Microsoft Active Directory uses an LDAP back-end for its user and group entries. However, this LDAP database doesn't contain UNIX style attributes. These attributes are added when the LDAP schema is extended through Identity Management for UNIX (Windows 2003R2 and later), Service for UNIX (Windows 2003 and earlier), or third-party LDAP tools such as *Centrify*. Because Microsoft uses LDAP as a back-end, it makes LDAP the perfect solution for environments that choose to leverage dual-protocol volumes in Azure NetApp Files.

+ > [!NOTE]

+ > Azure NetApp Files currently only supports native Microsoft Active Directory for LDAP services.

+## LDAP basics in Azure NetApp Files

+The following section discusses the basics of LDAP as it pertains to Azure NetApp Files.

+* LDAP information is stored in flat files in an LDAP server and is organized by way of an LDAP schema. You should configure LDAP clients in a way that coordinates their requests and lookups with the schema on the LDAP server.

+* LDAP clients initiate queries by way of an LDAP bind, which is essentially a login to the LDAP server using an account that has read access to the LDAP schema. The LDAP bind configuration on the clients is configured to use the security mechanism that is defined by the LDAP server. Sometimes, they are user name and password exchanges in plain text (simple). In other cases, binds are secured through Simple Authentication and Security Layer methods (`sasl`) such as Kerberos or LDAP over TLS. Azure NetApp Files uses the SMB machine account to bind using SASL authentication for the best possible security.

+* User and group information that is stored in LDAP is queried by clients by using standard LDAP search requests as defined in [RFC 2307](https://datatracker.ietf.org/doc/html/rfc2307). In addition, newer mechanisms, such as [RFC 2307bis](https://datatracker.ietf.org/doc/html/draft-howard-rfc2307bis-02), allow more streamlined user and group lookups. Azure NetApp Files uses a form of RFC 2307bis for its schema lookups in Windows Active Directory.

+* LDAP servers can store user and group information and netgroup. However, Azure NetApp Files currently can't use netgroup functionality in LDAP on Windows Active Directory.

+* LDAP in Azure NetApp Files operates on port 389. This port currently cannot be modified to use a custom port, such as port 636 (LDAP over SSL) or port 3268 (Active Directory Global Catalog searches).

+* Encrypted LDAP communications can be achieved using [LDAP over TLS](configure-ldap-over-tls.md#considerations) (which operates over port 389) or LDAP signing, both of which can be configured on the Active Directory connection.

+* Azure NetApp Files supports LDAP queries that take no longer than 3 seconds to complete. If the LDAP server has many objects, that timeout may be exceeded, and authentication requests can fail. In those cases, consider specifying an [LDAP search scope](https://ldap.com/the-ldap-search-operation/) to filter queries for better performance.

+* Azure NetApp Files also supports specifying preferred LDAP servers to help speed up requests. Use this setting if you want to ensure the LDAP server closest to your Azure NetApp Files region is being used.

+* If no preferred LDAP server is set, the Active Directory domain name is queried in DNS for LDAP service records to populate the list of LDAP servers available for your region located within that SRV record. You can manually query LDAP service records in DNS from a client using [`nslookup`](/troubleshoot/windows-server/networking/verify-srv-dns-records-have-been-created) or [`dig`](https://www.cyberciti.biz/faq/linux-unix-dig-command-examples-usage-syntax/) commands.

+ For example:

+ ```

+ C:\>nslookup

+ Default Server: localhost

+ Address: ::1

+

+ > set type=SRV

+ > _ldap._tcp.contoso.com.

+

+ Server: localhost

+ Address: ::1

+

+ _ldap._tcp.contoso.com SRV service location:

+ priority = 0

+ weight = 0

+ port = 389

+ svr hostname = oneway.contoso.com

+ _ldap._tcp.contoso.com SRV service location:

+ priority = 0

+ weight = 100

+ port = 389

+ svr hostname = ONEWAY.Contoso.com

+ _ldap._tcp.contoso.com SRV service location:

+ priority = 0

+ weight = 100

+ port = 389

+ svr hostname = oneway.contoso.com

+ _ldap._tcp.contoso.com SRV service location:

+ priority = 0

+ weight = 100

+ port = 389

+ svr hostname = parisi-2019dc.contoso.com

+ _ldap._tcp.contoso.com SRV service location:

+ priority = 0

+ weight = 100

+ port = 389

+ svr hostname = contoso.com

+ oneway.contoso.com internet address = x.x.x.x

+ ONEWAY.Contoso.com internet address = x.x.x.x

+ oneway.contoso.com internet address = x.x.x.x

+ parisi-2019dc.contoso.com internet address = y.y.y.y

+ contoso.com internet address = x.x.x.x

+ contoso.com internet address = y.y.y.y

+ ```

+* LDAP servers can also be used to perform custom name mapping for users. For more information, see [Custom name mapping using LDAP](#custom-name-mapping-using-ldap).

+## Name mapping types

+Name mapping rules can be broken down into two main types: *symmetric* and *asymmetric*.

+* *Symmetric* name mapping is implicit name mapping between UNIX and Windows users who use the same user name. For example, Windows user `CONTOSO\user1` maps to UNIX user `user1`.

+* *Asymmetric* name mapping is name mapping between UNIX and Windows users who use **different** user names. For example, Windows user `CONTOSO\user1` maps to UNIX user `user2`.

+By default, Azure NetApp Files uses symmetric name mapping rules. If asymmetric name mapping rules are required, consider configuring the LDAP user objects to use them.

+## Custom name mapping using LDAP

+LDAP can be a name mapping resource, if the LDAP schema attributes on the LDAP server have been populated. For example, to map UNIX users to corresponding Windows user names that don't match one-to-one (that is, *asymmetric*), you can specify a different value for `uid` in the user object than what is configured for the Windows user name.

+In the following example, a user has a Windows name of `asymmetric` and needs to map to a UNIX identity of `UNIXuser`. To achieve that in Azure NetApp Files, open an instance of the [Active Directory Users and Computers MMC](/troubleshoot/windows-server/system-management-components/remote-server-administration-tools). Then, find the desired user and open the properties box. (Doing so requires [enabling the Attribute Editor](http://directoryadmin.blogspot.com/2019/02/attribute-editor-tab-missing-in-active.html)). Navigate to the Attribute Editor tab and find the UID field, then populate the UID field with the desired UNIX user name `UNIXuser` and click **Add** and **OK** to confirm.

+After this action is done, files written from Windows SMB shares by the Windows user `asymmetric` will be owned by `UNIXuser` from the NFS side.

+The following example shows Windows SMB owner `asymmetric`:

+The following example shows NFS owner `UNIXuser` (mapped from Windows user `asymmetric` using LDAP):

+```

+root@ubuntu:~# su UNIXuser

+UNIXuser@ubuntu:/root$ cd /mnt

+UNIXuser@ubuntu:/mnt$ ls -la

+total 8

+drwxrwxrwx 2 root root 4096 Jul 3 20:09 .

+drwxr-xr-x 21 root root 4096 Jul 3 20:12 ..

+-rwxrwxrwx 1 UNIXuser group1 19 Jul 3 20:10 asymmetric-user-file.txt

+```

+## LDAP schemas

+LDAP schemas are how LDAP servers organize and collect information. LDAP server schemas generally follow the same standards, but different LDAP server providers might have variations on how schemas are presented.

+When Azure NetApp Files queries LDAP, schemas are used to help speed up name lookups because they enable the use of specific attributes to find information about a user, such as the UID. The schema attributes must exist in the LDAP server for Azure NetApp Files to be able to find the entry. Otherwise, LDAP queries might return no data and authentication requests might fail.

+For example, if a UID number (such as root=0) must be queried by Azure NetApp Files, then the schema attribute RFC 2307 `uidNumber Attribute` is used. If no UID number `0` exists in LDAP in the `uidNumber` field, then the lookup request fails.

+The schema type currently used by Azure NetApp Files is a form of schema based on RFC 2307bis and can't be modified.

+[RFC 2307bis](https://tools.ietf.org/html/draft-howard-rfc2307bis-02) is an extension of RFC 2307 and adds support for `posixGroup`, which enables dynamic lookups for auxiliary groups by using the `uniqueMember` attribute, rather than by using the `memberUid` attribute in the LDAP schema. Instead of using just the name of the user, this attribute contains the full distinguished name (DN) of another object in the LDAP database. Therefore, groups can have other groups as members, which allows nesting of groups. Support for RFC 2307bis also adds support for the object class `groupOfUniqueNames`.

+This RFC extension fits nicely into how Microsoft Active Directory manages users and groups through the usual management tools. This is because when you add a Windows user to a group (and if that group has a valid numeric GID) using the standard Windows management methods, LDAP lookups will pull the necessary supplemental group information from the usual Windows attribute and find the numeric GIDs automatically.

+## Next steps

+* [Configure AD DS LDAP over TLS for Azure NetApp Files](configure-ldap-over-tls.md)

+* [Understand NFS group memberships and supplemental groups](network-file-system-group-memberships.md)

+* [Azure NetApp Files NFS FAQ](faq-nfs.md)

+* [Azure NetApp Files SMB FAQ](faq-smb.md)

+ Title: Understand NAS concepts in Azure NetApp Files | Microsoft Docs

+ Title: Understand NAS protocols in Azure NetApp Files | Microsoft Learn

+description: Learn how SMB, NFS, and dual protocols operate in Azure NetApp Files.

+* Both can use encrypted authentication methods for sharing data.

+NFS is primarily used with Linux/UNIX based clients such as Red Hat, SUSE, Ubuntu, AIX, Solaris, and Apple OS. Azure NetApp Files supports any NFS client that operates in the RFC standards. Windows can also use NFS for access, but it doesn't operate using Request for Comments (RFC) standards.

+* NFSv3 is stateless, meaning that the NFS server doesn't keep track of the states of connections (including locks).

+* Locking is handled outside of the NFS protocol, using Network Lock Manager (NLM). Because locks aren't integrated into the protocol, stale locks can sometimes occur.

+* Because NFSv3 is stateless, performance with NFSv3 can be substantially better in some workloads, particularly in workloads with high-metadata operations such as OPEN, CLOSE, SETATTR, and GETATTR. This is the case because there's less general work that needs to be done to process requests on the server and client.

+* NFSv3 can use NFSv4.x ACLs, but an NFSv4.x management client would be required to configure and manage the ACLs. Azure NetApp Files doesn't support the use of nonstandard POSIX draft ACLs.

+* NFSv3 can use security enhancements such as Kerberos, but Kerberos only affects the NFS portion of the packets; ancillary protocols (such as NLM, portmapper, mount) aren't included in the Kerberos conversation.

+* NFSv3 uses numeric IDs for its user and group authentication. Usernames and group names aren't required for communication or permissions, which can make spoofing a user easier, but configuration and management are simpler.

+*NFSv4.x* refers to all NFS versions or minor versions that are under NFSv4, including NFSv4.0, NFSv4.1, and NFSv4.2. Azure NetApp Files currently supports NFSv4.1 only.

+* Locking is integrated into the NFS protocol and doesn't require ancillary locking protocols to keep track of NFS locks. Instead, locks are granted on a lease basis. They expire after a certain duration if a client or server connection is lost, thus returning the lock back to the system for use with other NFS clients.

+* Since NFSv4.x doesn't use ancillary protocols, Kerberos is applied to the entire NFS conversation when in use.

+* NFSv4.x uses a combination of user/group names and domain strings to verify user and group information. The client and server must agree on the domain strings for proper user and group authentication to occur. If the domain strings don't match, then the NFS user or group gets squashed to the specified user in the /etc/idmapd.conf file on the NFS client (for example, nobody).

+* While NFSv4.x does default to using domain strings, it's possible to configure the client and server to fall back on the classic numeric IDs seen in NFSv3 when AUTH_SYS is in use.

+* NFSv4.x has deep integration with user and group name strings, and the server and clients must agree on these users and groups. As such, consider using a name service server for user authentication such as LDAP on NFS clients and servers.

+SMB is primarily used with Windows clients for NAS functionality. However, it can also be used on Linux-based operating systems such as AppleOS, RedHat, etc. This deployment is accomplished using an application called Samba. Azure NetApp Files has official support for SMB using Windows and macOS. SMB/Samba on Linux operating systems can work with Azure NetApp Files, but there's no official support.

+* Locking in SMB is considered mandatory. When a file is locked, no other client can write to that file until the lock is released.

+* SMBv2.x and later use compound calls to perform operations.

+* When Kerberos is unable to be used for authentication, Windows NT LAN Manager (NTLM) may be used as a fallback. If NTLM is disabled in the Active Directory environment, then authentication requests that can't use Kerberos fail.

+## Dual protocols

+Some organizations have pure Windows or pure UNIX environments (homogenous) in which all data is accessed using only one of the following approaches:

+* SMB and [NTFS](/windows-server/storage/file-server/ntfs-overview) file security

+* NFS and UNIX file security - mode bits or [NFSv4.x access control lists (ACLs)](https://wiki.linux-nfs.org/wiki/index.php/ACLs)

+However, many sites must enable data sets to be accessed from both Windows and UNIX clients (heterogenous). For environments with these requirements, Azure NetApp Files has native dual-protocol NAS support. After the user is authenticated on the network and has both appropriate share or export permissions and the necessary file-level permissions, the user can access the data from UNIX hosts using NFS or from Windows hosts using SMB.

+### Reasons for using dual-protocol volumes

+Using dual-protocol volumes with Azure NetApp Files delivers several distinct advantages. When data sets can be seamlessly and simultaneously accessed by clients using different NAS protocols, the following benefits can be achieved:

+* Reduce the overall storage administrator management tasks.

+* Require only a single copy of data to be stored for NAS access from multiple client types.

+* Protocol agnostic NAS allows storage administrators to control the style of ACL and access control being presented to end users.

+* Centralize identity management operations in a NAS environment.

+### Common considerations with dual-protocol environments

+Dual-protocol NAS access is desirable by many organizations for its flexibility. However, there is a perception of difficulty that creates a set of considerations unique to the concept of sharing across protocols. These considerations include, but are not limited to:

+* Requirement of knowledge across multiple protocols, operating systems and storage systems.

+* Working knowledge of name service servers, such as DNS, LDAP, and so on.

+In addition, external factors can come into play, such as:

+* Dealing with multiple departments and IT groups (such as Windows groups and UNIX groups)

+* Company acquisitions

+* Domain consolidations

+* Reorganizations

+Despite these considerations, dual-protocol NAS setup, configuration, and access can be simple and seamlessly integrated into any environment.

+### How Azure NetApp Files simplifies dual-protocol use

+Azure NetApp Files consolidates the infrastructure required for successful dual-protocol NAS environments into a single management plane, including storage and identity management services.

+Dual-protocol configuration is straightforward, and most of the tasks are shielded by the Azure NetApp Files resource management framework to simplify operations for cloud operators.

+After an Active Directory connection is established with Azure NetApp Files, dual-protocol volumes can use the connection to handle both the Windows and UNIX identity management needed for proper user and group authentication with Azure NetApp Files volumes without extra configuration steps outside of the normal user and group management within the Active Directory or LDAP services.

+By removing the extra storage-centric steps for dual-protocol configurations, Azure NetApp Files streamlines the overall dual-protocol deployment for organizations looking to move to Azure.

+### How Azure NetApp Files dual-protocol volumes work

+At a high level, Azure NetApp Files dual-protocol volumes use a combination of name mapping and permissions styles to deliver consistent data access regardless of the protocol in use. That means whether you're accessing a file from NFS or SMB, you can be assured that users with access to those files can access them, and users without access to those files can't access them.

+When a NAS client requests access to a dual-protocol volume in Azure NetApp Files, the following operations occur to provide a transparent experience to the end user.

+1. A NAS client makes a NAS connection to the Azure NetApp Files dual-protocol volume.

+2. The NAS client passes user identity information to Azure NetApp Files.

+3. Azure NetApp Files checks to make sure the NAS client/user has access to the NAS share.

+4. Azure NetApp Files takes that user and maps it to a valid user found in name services.

+5. Azure NetApp Files compares that user against the file-level permissions in the system.

+6. File permissions control the level of access the user has.

+In the following illustration, `user1` authenticates to Azure NetApp Files to access a dual-protocol volume through either SMB or NFS. Azure NetApp Files finds the userΓÇÖs Windows and UNIX information in Azure Active Directory and then maps the user's Windows and UNIX identities one-to-one. The user is verified as `user1` and gets `user1`'s access credentials.

+In this instance, `user1` gets full control on their own folder (`user1-dir`) and no access to the `HR` folder. This setting is based on the security ACLs specified in the file system, and `user1` will get the expected access regardless of which protocol they're accessing the volumes from.

+### Considerations for Azure NetApp Files dual-protocol volumes

+When you use Azure NetApp Files volumes for access to both SMB and NFS, some considerations apply:

+* You need an Active Directory connection. As such, you need to meet the [Requirements for Active Directory connections](create-active-directory-connections.md#requirements-for-active-directory-connections).

+* Dual-protocol volumes require a reverse lookup zone in DNS with an associated pointer (PTR) record of the AD host machine to prevent dual-protocol volume creation failures.

+* Your NFS client and associated packages (such as `nfs-utils`) should be up to date for the best security, reliability and feature support.

+* Dual-protocol volumes support both Active Directory Domain Services (AD DS) and Azure Active Directory Domain Services (Azure AD DS, or AADDS).

+* Dual-protocol volumes don't support the use of LDAP over TLS with AADDS. See [LDAP over TLS considerations](configure-ldap-over-tls.md#considerations).

+* Supported NFS versions include: NFSv3 and NFSv4.1.

+* NFSv4.1 features such as parallel network file system (pNFS), session trunking, and referrals aren't currently supported with Azure NetApp Files volumes.

+* [Windows extended attributes `set`/`get`](/windows/win32/api/fileapi/ns-fileapi-createfile2_extended_parameters) aren't supported in dual-protocol volumes.

+* See additional [considerations for creating a dual-protocol volume for Azure NetApp Files](create-volumes-dual-protocol.md#considerations).

+<!-- planning to consolidate and move considerations from the Create article to this subsection. -->

+* [Understand dual-protocol security style and permission behaviors in Azure NetApp Files](dual-protocol-permission-behaviors.md)

+* [Understand the use of LDAP with Azure NetApp Files](lightweight-directory-access-protocol.md)

+* [Understand NFS group memberships and supplemental groups](network-file-system-group-memberships.md)

+* [Create an NFS volume for Azure NetApp Files](azure-netapp-files-create-volumes.md)

+* [Create an SMB volume for Azure NetApp Files](azure-netapp-files-create-volumes-smb.md)

+* [Create a dual-protocol volume for Azure NetApp Files](create-volumes-dual-protocol.md)

+* [Azure NetApp Files NFS FAQ](faq-nfs.md)

+* [Azure NetApp Files SMB FAQ](faq-smb.md)

+ Title: Understand NFS group memberships and supplemental groups for Azure NetApp Files | Microsoft Learn

+description: This article helps you understand NFS group memberships and supplemental groups as they apply to Azure NetApp Files.

+documentationcenter: ''

+editor: ''

+ms.assetid:

+ na

+# Understand NFS group memberships and supplemental groups

+You can use LDAP to control group membership and to return supplemental groups for NFS users. This behavior is controlled through schema attributes in the LDAP server.

+## Primary GID

+For Azure NetApp Files to be able to authenticate a user properly, LDAP users must always have a primary GID defined. The userΓÇÖs primary GID is defined by the schema `gidNumber` in the LDAP server.

+## Secondary, supplemental, and auxiliary GIDs

+Secondary, supplemental, and auxiliary groups are groups that a user is a member of outside of their primary GID. In Azure NetApp Files, LDAP is implemented using Microsoft Active Directory and supplemental groups are controlled using standard Windows group membership logic.

+When a user is added to a Windows group, the LDAP schema attribute `Member` is populated on the group with the distinguished name (DN) of the user that is a member of that group. When a userΓÇÖs group membership is queried by Azure NetApp Files, an LDAP search is done for the userΓÇÖs DN on all groupsΓÇÖ `Member` attribute. All groups with a UNIX `gidNumber` and the userΓÇÖs DN are returned in the search and populated as the userΓÇÖs supplemental group memberships.

+The following example shows the output from Active Directory with a userΓÇÖs DN populated in the `Member` field of a group and a subsequent LDAP search done using [`ldp.exe`](/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/cc771022(v=ws.11)).

+The following example shows the Windows group member field:

+The following example shows `LDAPsearch` of all groups where `User1` is a member:

+You can also query group memberships for a user in Azure NetApp Files by selecting **LDAP Group ID List** link under **Support + troubleshooting** on the volume menu.

+## Group limits in NFS

+Remote Procedure Call (RPC) in NFS has a specific limitation for the maximum number of auxiliary GIDs that can be honored in a single NFS request. The maximum for [`AUTH_SYS/AUTH_UNIX` is 16](http://tools.ietf.org/html/rfc5531), and for AUTH_GSS (Kerberos), it's 32. This protocol limitation affects all NFS servers - not just Azure NetApp Files. However, many modern NFS servers and clients include ways to work around these limitations.

+To work around this NFS limitation in Azure NetApp Files, see [Enable Active Directory Domain Services (AD DS) LDAP authentication for NFS volumes](configure-ldap-extended-groups.md).

+## How extending the group limitation works

+The options to extend the group limitation work the same way that the `manage-gids` option for other NFS servers works. Basically, rather than dumping the entire list of auxiliary GIDs that a user belongs to, the option performs a lookup for the GID on the file or folder and returns that value instead.

+The following example shows RPC packet with 16 GIDs.

+Any GID past the limit of 16 is dropped by the protocol. With extended groups in Azure NetApp Files, when a new NFS request comes in, information about the userΓÇÖs group membership is requested.

+## Considerations for extended GIDs with Active Directory LDAP

+By default, in Microsoft Active Directory LDAP servers, the `MaxPageSize` attribute is set to a default of 1,000. That setting means that groups beyond 1,000 would be truncated in LDAP queries. To enable full support with the 1,024 value for extended groups, the `MaxPageSize` attribute must be modified to reflect the 1,024 value. For information about how to change that value, see the Microsoft TechNet article [How to View and Set LDAP Policy in Active Directory by Using Ntdsutil.exe](https://support.microsoft.com/kb/315071) and the TechNet library article [MaxPageSize Is Set Too High](https://technet.microsoft.com/library/aa998536(v=exchg.80).aspx).

+## Next steps

+* [Enable Active Directory Domain Services (AD DS) LDAP authentication for NFS volumes](configure-ldap-extended-groups.md)

+* [Understand file locking and lock types in Azure NetApp Files](understand-file-locks.md)

+* [Understand dual-protocol security style and permission behaviors in Azure NetApp Files](dual-protocol-permission-behaviors.md)

+* [Understand the use of LDAP with Azure NetApp Files](lightweight-directory-access-protocol.md)

+* [Azure NetApp Files NFS FAQ](faq-nfs.md)

+* [Enable Active Directory Domain Services (AD DS) LDAP authentication for NFS volumes](configure-ldap-extended-groups.md)

+ Title: Understand file locking and lock types in Azure NetApp Files | Microsoft Docs

+## Reference resource/module collections

+The ARM template [`references`](../templates/template-functions-resource.md#references) function returns an array of objects representing a resource collection's runtime states. In Bicep, there is no explicit references function. Instead, symbolic collection usage is employed directly, and during code generation, Bicep translates it to an ARM template that utilizes the ARM template references function. For the translation feature that transforms symbolic collections into ARM templates using the references function, it is necessary to have Bicep CLI version 0.20.4 or a more recent version. Additionally, in the [`bicepconfig.json`](./bicep-config.md#enable-experimental-features) file, the `symbolicNameCodegen` setting should be presented and set to `true`.

+The outputs of the two samples in [Integer index](#integer-index) can be written as:

+```bicep

+param location string = resourceGroup().location

+param storageCount int = 2

+resource storageAcct 'Microsoft.Storage/storageAccounts@2022-09-01' = [for i in range(0, storageCount): {

+ name: '${i}storage${uniqueString(resourceGroup().id)}'

+ location: location

+ sku: {

+ name: 'Standard_LRS'

+ }

+ kind: 'Storage'

+}]

+output storageInfo array = map(storageAcct, store => {

+ blobEndpoint: store.properties.primaryEndpoints

+ status: store.properties.statusOfPrimary

+})

+output storageAccountEndpoints array = map(storageAcct, store => store.properties.primaryEndpoints)

+```

+This Bicep file is transpiled into the following ARM JSON template that utilizes the `references` function:

+```json

+{

+ "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",

+ "languageVersion": "1.10-experimental",

+ "contentVersion": "1.0.0.0",

+ "parameters": {

+ "location": {

+ "type": "string",

+ "defaultValue": "[resourceGroup().location]"

+ },

+ "storageCount": {

+ "type": "int",

+ "defaultValue": 2

+ }

+ },

+ "resources": {

+ "storageAcct": {

+ "copy": {

+ "name": "storageAcct",

+ "count": "[length(range(0, parameters('storageCount')))]"

+ },

+ "type": "Microsoft.Storage/storageAccounts",

+ "apiVersion": "2022-09-01",

+ "name": "[format('{0}storage{1}', range(0, parameters('storageCount'))[copyIndex()], uniqueString(resourceGroup().id))]",

+ "location": "[parameters('location')]",

+ "sku": {

+ "name": "Standard_LRS"

+ },

+ "kind": "Storage"

+ }

+ },

+ "outputs": {

+ "storageInfo": {

+ "type": "array",

+ "value": "[map(references('storageAcct', 'full'), lambda('store', createObject('blobEndpoint', lambdaVariables('store').properties.primaryEndpoints, 'status', lambdaVariables('store').properties.statusOfPrimary)))]"

+ },

+ "storageAccountEndpoints": {

+ "type": "array",

+ "value": "[map(references('storageAcct', 'full'), lambda('store', lambdaVariables('store').properties.primaryEndpoints))]"

+ }

+ }

+}

+```

+Note in the preceding ARM JSON template, `languageVersion` must be set to `1.10-experimental`, and the resource element is an object instead of an array.

+ :::image type="content" source="./media/add-template-to-azure-pipelines/new-pipeline.png" alt-text="Screenshot of the Add new pipeline button":::

+ :::image type="content" source="./media/add-template-to-azure-pipelines/select-source.png" alt-text="Screenshot of selecting the code source in Azure DevOps":::

+ :::image type="content" source="./media/add-template-to-azure-pipelines/select-repo.png" alt-text="Screenshot of selecting the repository for the project in Azure DevOps":::

+ :::image type="content" source="./media/add-template-to-azure-pipelines/select-pipeline.png" alt-text="Screenshot of selecting the type of pipeline to create in Azure DevOps":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/create-project.png" alt-text="Screenshot of Create a new project window highlighting Azure Resource Group and Next button.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/name-project.png" alt-text="Screenshot of the project naming window in Visual Studio.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/select-project.png" alt-text="Screenshot of the template selection window with Web app template highlighted.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/show-items.png" alt-text="Screenshot of the Visual Studio Solution Explorer showing the resource group deployment project files.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/show-json-outline.png" alt-text="Screenshot of the JSON Outline window in Visual Studio for the Resource Manager template.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/navigate-json.png" alt-text="Screenshot of the Visual Studio editor with a selected element in the JSON Outline window.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/add-resource.png" alt-text="Screenshot of the JSON Outline window highlighting the Add New Resource option.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/add-storage.png" alt-text="Screenshot of the Add New Resource window with Storage Account selected.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/show-new-items.png" alt-text="Screenshot of the JSON Outline window displaying the added Storage Account resource.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/show-intellisense.png" alt-text="Screenshot of Visual Studio editor showing intellisense suggestions for Resource Manager template.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/deploy.png" alt-text="Screenshot of the deployment project context menu with Deploy and New options highlighted.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/show-deployment.png" alt-text="Screenshot of the Deploy to Resource Group dialog box in Visual Studio.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/show-deployed-resources.png" alt-text="Screenshot of the Azure portal displaying the deployed resources in a resource group.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/add-project.png" alt-text="Screenshot of the Add New Project context menu in Visual Studio.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/add-app.png" alt-text="Screenshot of the New Project window with ASP.NET Core Web Application selected.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/name-web-app.png" alt-text="Screenshot of the project naming window for the ASP.NET Core Web Application.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/select-project-type.png" alt-text="Screenshot of the New ASP.NET Core Web Application window with Web Application selected.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/show-projects.png" alt-text="Screenshot of the Visual Studio Solution Explorer displaying both projects in the solution.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/add-new-reference.png" alt-text="Screenshot of the ExampleAppDeploy context menu highlighting the Add Reference option.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/add-reference.png" alt-text="Screenshot of the Add Reference window in Visual Studio with the web app project selected.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/see-reference.png" alt-text="Screenshot of the Properties window displaying the reference properties for the web app project.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/add-resource-2.png" alt-text="Screenshot of the JSON Outline window with the Add New Resource option highlighted.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/add-web-deploy.png" alt-text="Screenshot of the Add New Resource window with Web Deploy for Web Apps selected.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/redeploy.png" alt-text="Screenshot of the deployment project context menu with Deploy and the previously used resource group highlighted.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/redeploy-web-app.png" alt-text="Screenshot of the Deploy to Resource Group dialog box with Artifact storage account selected.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/browse-site.png" alt-text="Screenshot of the Azure portal displaying the web app resource with the URL highlighted.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/show-deployed-app.png" alt-text="Screenshot of the deployed default ASP.NET app in a web browser.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/view-custom-dashboards.png" alt-text="Screenshot of the Azure portal Dashboard page highlighting an example custom dashboard.":::

+ :::image type="content" source="./media/create-visual-studio-deployment-project/Ops-DemoSiteGroup-dashboard.png" alt-text="Screenshot of the customized operational dashboard in the Azure portal.":::

+ :::image type="content" source="./media/deploy-cloud-shell/open-cloud-shell.png" alt-text="Screenshot of the button to open Cloud Shell.":::

+ :::image type="content" source="./media/deploy-cloud-shell/cloud-shell-bash-powershell.png" alt-text="Screenshot of the option to select Bash or PowerShell in Cloud Shell.":::

+ :::image type="content" source="./media/deploy-cloud-shell/cloud-shell-upload.png" alt-text="Screenshot of the Cloud Shell interface with the Upload file option highlighted.":::

+ :::image type="content" source="./media/deploy-portal/select-resource-groups.png" alt-text="Screenshot of selecting resource groups in Azure portal":::

+ :::image type="content" source="./media/deploy-portal/add-resource-group.png" alt-text="Screenshot of adding a resource group in Azure portal":::

+ :::image type="content" source="./media/deploy-portal/set-group-properties.png" alt-text="Screenshot of setting resource group property values in Azure portal":::

+ :::image type="content" source="./media/deploy-portal/new-resources.png" alt-text="Screenshot of creating a new resource in Azure portal":::

+ :::image type="content" source="./media/deploy-portal/select-resource-type.png" alt-text="Screenshot of selecting a resource type in Azure portal":::

+ :::image type="content" source="./media/deploy-portal/select-existing-group.png" alt-text="Screenshot of creating a Linux virtual machine and deploying it to a resource group in Azure portal":::

+ :::image type="content" source="./media/deploy-portal/view-notification.png" alt-text="Screenshot of viewing deployment notification in Azure portal":::

+ :::image type="content" source="./media/deploy-portal/add-resource.png" alt-text="Screenshot of adding a resource to a resource group in Azure portal":::

+ :::image type="content" source="./media/deploy-portal/search-template.png" alt-text="Screenshot of searching for template deployment in Azure portal":::

+ :::image type="content" source="./media/deploy-portal/see-options.png" alt-text="Screenshot of template creation options in Azure portal":::

+ :::image type="content" source="./media/deploy-portal/show-json.png" alt-text="Screenshot of editing a JSON template in Azure portal":::

+ :::image type="content" source="./media/deployment-script-template-configure-dev/deployment-script-container-instance-connect.png" alt-text="Screenshot of the deployment script connect container instance option in the Azure portal.":::

+ :::image type="content" source="./media/deployment-script-template-configure-dev/deployment-script-container-instance-test.png" alt-text="Screenshot of the deployment script connect container instance test output displayed in the console.":::

+ :::image type="content" source="./media/deployment-script-template-configure-dev/deployment-script-container-instance-connect.png" alt-text="Screenshot of the deployment script connect container instance option in the Azure portal.":::

+ :::image type="content" source="./media/deployment-script-template-configure-dev/deployment-script-container-instance-test-cli.png" alt-text="Screenshot of the deployment script container instance test output displayed in the console.":::

+ :::image type="content" source="./medi.png" alt-text="Screenshot of the Resource Manager template deployment script using Docker command.":::

+ :::image type="content" source="./media/deployment-tutorial-pipeline/azure-resource-manager-devops-pipelines-github-repository.png" alt-text="Screenshot of creating a GitHub repository for Azure Resource Manager Azure DevOps Azure Pipelines.":::

+ :::image type="content" source="./media/deployment-tutorial-pipeline/azure-resource-manager-devops-pipelines-create-devops-project.png" alt-text="Screenshot of creating an Azure DevOps project for Azure Resource Manager Azure DevOps Azure Pipelines.":::

+ :::image type="content" source="./media/deployment-tutorial-pipeline/azure-resource-manager-devops-pipelines-only-select-repositories.png" alt-text="Screenshot of selecting repositories for Azure Resource Manager Azure DevOps Azure Pipelines.":::

+ :::image type="content" source="./media/deployment-tutorial-pipeline/resource-manager-template-pipeline-configure.png" alt-text="Screenshot of the ARM template deployment page with required values entered for Azure DevOps Azure Pipelines.":::

+ :::image type="content" source="./media/deployment-tutorial-pipeline/azure-resource-manager-devops-pipelines-yml.png" alt-text="Screenshot of the Review page with the new pipeline titled Review your pipeline YAML for Azure DevOps Azure Pipelines.":::

+ :::image type="content" source="./media/deployment-tutorial-pipeline/azure-resource-manager-devops-pipelines-status.png" alt-text="Screenshot of Azure Resource Manager Azure DevOps Azure Pipelines YAML file.":::

+ :::image type="content" source="./media/deployment-tutorial-pipeline/azure-resource-manager-devops-pipelines-update-yml.png" alt-text="Screenshot of updating the YAML file for Azure Resource Manager Azure DevOps Azure Pipelines.":::

+ :::image type="content" source="./media/quickstart-create-template-specs/search-template-spec.png" alt-text="Screenshot of search bar with 'template specs' query.":::

+ :::image type="content" source="./media/quickstart-create-template-specs/import-template.png" alt-text="Screenshot of 'Import template' button in Template specs page.":::

+ :::image type="content" source="./media/quickstart-create-template-specs/open-folder.png" alt-text="Screenshot of folder icon to open file explorer.":::

+ :::image type="content" source="./media/quickstart-create-template-specs/select-import.png" alt-text="Screenshot of 'Import' button after selecting a template file.":::

+ :::image type="content" source="./media/quickstart-create-template-specs/select-template-spec.png" alt-text="Screenshot of Template specs list with one item selected.":::

+ :::image type="content" source="./media/quickstart-create-template-specs/deploy-template-spec.png" alt-text="Screenshot of 'Deploy' button in the selected Template spec.":::

+ :::image type="content" source="./media/quickstart-create-template-specs/select-versions.png" alt-text="Screenshot of 'Create new version' button in Template spec details.":::

+ :::image type="content" source="./media/quickstart-create-template-specs/add-version-name.png" alt-text="Screenshot of naming the new version and selecting 'Edit template' button.":::

+ :::image type="content" source="./media/quickstart-create-template-specs/see-versions.png" alt-text="Screenshot of 'Versions' tab in Template spec details.":::

+ :::image type="content" source="./media/quickstart-create-template-specs/deploy-version.png" alt-text="Screenshot of 'Deploy' option in the context menu of a specific version.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/search-custom-template.png" alt-text="Screenshot of searching for custom template in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/select-custom-template.png" alt-text="Screenshot of selecting a Quickstart Template in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/input-fields-template.png" alt-text="Screenshot of input fields for custom template in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/template-validation.png" alt-text="Screenshot of template validation and create button in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/deploy-success.png" alt-text="Screenshot of deployment succeeded notification in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/view-storage-account.png" alt-text="Screenshot of view deployment page with storage account in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/build-own-template.png" alt-text="Screenshot of build your own template option in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/blank-template.png" alt-text="Screenshot of blank ARM template in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/view-second-deployment.png" alt-text="Screenshot of view second deployment page in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/view-resource-group.png" alt-text="Screenshot of resource group with storage account and virtual network in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/export-template.png" alt-text="Screenshot of export template option in Azure portal.":::

+ :::image type="content" source="./media/quickstart-create-templates-use-the-portal/download-template.png" alt-text="Screenshot of download button for exported ARM template in Azure portal.":::

+Returns an object representing a resource's runtime state. To return an array of objects representing a resource collections's runtime states, see [references](#references).

+The `references` function works similarly as [`reference`](#reference). Instead of returning an object presenting a resource's runtime state, the `references` function returns an array of objects representing a resource collection's runtime states. This function requires ARM template language version `1.10-experimental` and with [symbolic name](../bicep/file.md#resources) enabled:

+In Bicep, there is no explicit `references` function. Instead, symbolic collection usage is employed directly, and during code generation, Bicep translates it to an ARM template that utilizes the ARM template `references` function. For more information, see [Reference resource/module collections](../bicep/loops.md#reference-resourcemodule-collections).

+ :::image type="content" source="./media/template-specs-create-portal-forms/deploy-template-spec-config.png" alt-text="Screenshot of Azure portal form view sandbox interface.":::

+ :::image type="content" source="./media/template-specs-create-portal-forms/view-portal-basic.png" alt-text="Screenshot of the generated basic Azure portal form.":::

+ :::image type="content" source="./media/template-specs-create-portal-forms/view-steps.png" alt-text="Screenshot of Azure portal form with multiple steps.":::

+ :::image type="content" source="media/template-tutorial-deploy-sql-extensions-bacpac/resource-manager-tutorial-deploy-sql-extensions-bacpac-firewall.png" alt-text="Screenshot of the template with firewall definition.":::

+ :::image type="content" source="media/template-tutorial-deploy-sql-extensions-bacpac/resource-manager-tutorial-deploy-sql-extensions-bacpac.png" alt-text="Screenshot of the template with SQL Database extension.":::

+ :::image type="content" source="media/template-tutorial-deploy-sql-extensions-bacpac/cloud-shell-select.png" alt-text="Screenshot of Azure Cloud Shell in PowerShell with the option to upload a file.":::

+With a trial account, Azure AI Video Indexer provides up to 2,400 minutes of free indexing when using the [Azure AI Video Indexer](https://www.videoindexer.ai/) website or the Azure AI Video Indexer API (see [developer portal](https://api-portal.videoindexer.ai/)).

+When you choose to see **Insights** of your video on the [Azure AI Video Indexer](https://www.videoindexer.ai/) website, the People's detected clothing could be viewed from the **Observed People** tracking insight. When choosing a thumbnail of a person the detected clothing became available.

+The following JSON response illustrates what Azure AI Video Indexer returns when tracking observed people having detected clothing associated:

+[Track observed people in a video](observed-people-tracking.md)

+ Title: Track observed people in a video

+description: This topic gives an overview of Track observed people in a video concept.

+# Track observed people in a video (preview)

+Azure AI Video Indexer detects observed people in videos and provides information such as the location of the person in the video frame and the exact timestamp (start, end) when a person appears. The API returns the bounding box coordinates (in pixels) for each person instance detected, including detection confidence.

+

+Some scenarios where this feature could be useful:

+* Post-event analysisΓÇödetect and track a personΓÇÖs movement to better analyze an accident or crime post-event (for example, explosion, bank robbery, incident).

+* Improve efficiency when creating raw data for content creators, like video advertising, news, or sport games (for example, find people wearing a red shirt in a video archive).

+* Create a summary out of a long video, like court evidence of a specific personΓÇÖs appearance in a video, using the same detected personΓÇÖs ID.

+* Learn and analyze trends over time, for exampleΓÇöhow customers move across aisles in a shopping mall or how much time they spend in checkout lines.

+For example, if a video contains a person, the detect operation will list the personΓÇÖs appearances together with their coordinates in the video frames. You can use this functionality to determine the personΓÇÖs path in a video. It also lets you determine whether there are multiple instances of the same person in a video.

+The newly added **Observed people tracking** feature is available when indexing your file by choosing the **Advanced option** -> **Advanced video** or **Advanced video + audio** preset (under **Video + audio indexing**). Standard indexing will not include this new advanced model.

+

+When you choose to see **Insights** of your video on the [Video Indexer](https://www.videoindexer.ai/account/login) website, the Observed People Tracking will show up on the page with all detected people thumbnails. You can choose a thumbnail of a person and see where the person appears in the video player.

+The following JSON response illustrates what Video Indexer returns when tracking observed people:

+```json

+ {

+ ...

+ "videos": [

+ {

+ ...

+ "insights": {

+ ...

+ "observedPeople": [{

+ "id": 1,

+ "thumbnailId": "560f2cfb-90d0-4d6d-93cb-72bd1388e19d",

+ "instances": [

+ {

+ "adjustedStart": "0:00:01.5682333",

+ "adjustedEnd": "0:00:02.7027",

+ "start": "0:00:01.5682333",

+ "end": "0:00:02.7027"

+ }

+ ]

+ },

+ {

+ "id": 2,

+ "thumbnailId": "9c97ae13-558c-446b-9989-21ac27439da0",

+ "instances": [

+ {

+ "adjustedStart": "0:00:16.7167",

+ "adjustedEnd": "0:00:18.018",

+ "start": "0:00:16.7167",

+ "end": "0:00:18.018"

+ }

+ ]

+ },]

+ }

+ ...

+ }

+ ]

+}

+```

+## Limitations and assumptions

+For more information, see [Considerations and limitations when choosing a use case](observed-matched-people.md#considerations-and-limitations-when-choosing-a-use-case).

+## Next steps

+Review [overview](video-indexer-overview.md)

+ console.log(`Received message ${e.message.data}`)

+});

+#### Connect to your Web PubSub resource and register a listener for the `ServerMessageReceived` event

+A client uses a ***Client Access URL*** to connect and authenticate with your resource.

+This URL follows a pattern of `wss://<service_name>.webpubsub.azure.com/client/hubs/<hub_name>?access_token=<token>`. A client can have a few ways to obtain the Client Access URL. For this quick start, you can copy and paste one from Azure portal shown in the following diagram. It's best practice to not hard code the Client Access URL in your code. In the production world, we usually set up an app server to return this URL on demand. [Generate Client Access URL](./howto-generate-client-access-url.md) describes the practice in detail.

+

+As shown in the diagram above, the client joins the hub named `myHub1`.

+client.Connected += eventArgs =>

+{

+ Console.WriteLine("Connected");

+ return Task.CompletedTask;

+};

+// This keeps the subscriber active until the user closes the stream by pressing Ctrl+C

+var streaming = Console.ReadLine();

+while (streaming != null)

+{

+ if (!string.IsNullOrEmpty(streaming))

+ {

+ await client.SendToGroupAsync("stream", BinaryData.FromString(streaming + Environment.NewLine), WebPubSubDataType.Text);

+ }

+ streaming = Console.ReadLine();

+}

+await client.StopAsync();

+dotnet run

+$connection_string="<connection-string>"

+No, custom domains aren't supported with Bastion shareable links. Users receive a certificate error upon trying to add specific domains in the CN/SAN of the Bastion host certificate.

+Additionally, the user must have the rights (if required) to connect to the VM. For example, if the user is connecting to a Windows VM via RDP and isn't a member of the local Administrators group, they must be a member of the Remote Desktop Users group.

+No. When you connect to a VM using Azure Bastion, you don't need a public IP on the Azure virtual machine that you're connecting to. The Bastion service opens the RDP/SSH session/connection to your virtual machine over the private IP of your virtual machine, within your virtual network.

+This feature doesn't work with AADJ VM extension-joined machines using Azure AD users. For more information, see [Sign in to a Windows virtual machine in Azure by using Azure AD](../active-directory/devices/howto-vm-sign-in-azure-ad-windows.md#requirements).

+Users can use "Ctrl+Shift+Alt" to effectively switch focus between the VM and the browser.

+### <a name="keyboard-focus"></a>How do I take keyboard or mouse focus back from an instance?

+Click the Windows key twice in a row to take back focus within the Bastion window.

+### My privatelink.azure.com can't resolve to management.privatelink.azure.com

+For more information, see [What is Azure Bastion](bastion-overview.md).

+### Multi-connection tunnel

+### Multi-connection tunnel

+By default, each container app issues its own unique cookie or token for authentication. You can also provide your own signing and encryption keys.

+**Enable accelerated networking to reduce latency and CPU jitter**

+It is recommended that you follow the instructions to enable [Accelerated Networking](../../virtual-network/accelerated-networking-overview.md) in your [Windows (click for instructions)](../../virtual-network/create-vm-accelerated-networking-powershell.md) or [Linux (click for instructions)](../../virtual-network/create-vm-accelerated-networking-cli.md) Azure VM, in order to maximize performance.

+Without accelerated networking, IO that transits between your Azure VM and other Azure resources may be unnecessarily routed through a host and virtual switch situated between the VM and its network card. Having the host and virtual switch inline in the datapath not only increases latency and jitter in the communication channel, it also steals CPU cycles from the VM. With accelerated networking, the VM interfaces directly with the NIC without intermediaries; any network policy details which were being handled by the host and virtual switch are now handled in hardware at the NIC; the host and virtual switch are bypassed. Generally you can expect lower latency and higher throughput, as well as more *consistent* latency and decreased CPU utilization when you enable accelerated networking.

+Limitations: accelerated networking must be supported on the VM OS, and can only be enabled when the VM is stopped and deallocated. The VM cannot be deployed with Azure Resource Manager. [App Service](../../app-service/overview.md) has no accelerated network enabled.

+Please see the [Windows](../../virtual-network/create-vm-accelerated-networking-powershell.md) and [Linux](../../virtual-network/create-vm-accelerated-networking-cli.md) instructions for more details.

+**Enable accelerated networking to reduce latency and CPU jitter**

+It is recommended that you follow the instructions to enable [Accelerated Networking](../../virtual-network/accelerated-networking-overview.md) in your [Windows (click for instructions)](../../virtual-network/create-vm-accelerated-networking-powershell.md) or [Linux (click for instructions)](../../virtual-network/create-vm-accelerated-networking-cli.md) Azure VM, in order to maximize performance (reduce latency and CPU jitter).

+Limitations: accelerated networking must be supported on the VM OS, and can only be enabled when the VM is stopped and deallocated. The VM cannot be deployed with Azure Resource Manager. [App Service](../../app-service/overview.md) has no accelerated network enabled.

+### Optimizing single partition queries with Optimistic Direct Execution

+Azure Cosmos DB NoSQL has an optimization called Optimistic Direct Execution (ODE), which can improve the efficiency of certain NoSQL queries. Specifically, queries that donΓÇÖt require distribution include those that can be executed on a single physical partition or that have responses that don't require [pagination](query/pagination.md). Queries that donΓÇÖt require distribution can confidently skip some processes, such as client-side query plan generation and query rewrite, thereby reducing query latency and RU cost. If you specify the partition key in the request or query itself (or have only one physical partition), and the results of your query donΓÇÖt require pagination, then ODE can improve your queries.

+ODE is now available and enabled by default in the .NET SDK (preview) version 3.35.0-preview and later. When you execute a query and specify a partition key in the request or query itself, or your database has only one physical partition, your query execution can leverage the benefits of ODE. To disable ODE, set EnableOptimisticDirectExecution to false in the QueryRequestOptions.

+Single partition queries that feature GROUP BY, ORDER BY, DISTINCT, and aggregation functions (like sum, mean, min, and max) can significantly benefit from using ODE. However, in scenarios where the query is targeting multiple partitions or still requires pagination, the latency of the query response and RU cost might be higher than without using ODE. Therefore, when using ODE, we recommend to:

+- Specify the partition key in the call or query itself.

+- Ensure that your data size hasnΓÇÖt grown and caused the partition to split.

+- Ensure that your query results donΓÇÖt require pagination to get the full benefit of ODE.

+

+Here are a few examples of simple single partition queries which can benefit from ODE:

+```

+- SELECT * FROM r

+- SELECT * FROM r WHERE r.pk == "value"

+- SELECT * FROM r WHERE r.id > 5

+- SELECT r.id FROM r JOIN id IN r.id

+- SELECT TOP 5 r.id FROM r ORDER BY r.id

+- SELECT * FROM r WHERE r.id > 5 OFFSET 5 LIMIT 3

+```

+There can be cases where single partition queries may still require distribution if the number of data items increases over time and your Azure Cosmos DB database [splits the partition](../partitioning-overview.md#physical-partitions). Examples of queries where this could occur include:

+```

+- SELECT Count(r.id) AS count_a FROM r

+- SELECT DISTINCT r.id FROM r

+- SELECT Max(r.a) as min_a FROM r

+- SELECT Avg(r.a) as min_a FROM r

+- SELECT Sum(r.a) as sum_a FROM r WHERE r.a > 0

+```

+Some complex queries can always require distribution, even if targeting a single partition. Examples of such queries include:

+```

+- SELECT Sum(id) as sum_id FROM r JOIN id IN r.id

+- SELECT DISTINCT r.id FROM r GROUP BY r.id

+- SELECT DISTINCT r.id, Sum(r.id) as sum_a FROM r GROUP BY r.id

+- SELECT Count(1) FROM (SELECT DISTINCT r.id FROM root r)

+- SELECT Avg(1) AS avg FROM root r

+```

+It's important to note that ODE might not always retrieve the query plan and, as a result, is not able to disallow or turn off for unsupported queries. For example, after partition split, such queries are no longer eligible for ODE and, therefore, won't run because client-side query plan evaluation will block those. To ensure compatibility/service continuity, it's critical to ensure that only queries that are fully supported in scenarios without ODE (that is, they execute and produce the correct result in the general multi-partition case) are used with ODE.

+>[!NOTE]

+> Using ODE can potentially cause a new type of continuation token to be generated. Such a token is not recognized by the older SDKs by design and this could result in a Malformed Continuation Token Exception. If you have a scenario where tokens generated from the newer SDKs are used by an older SDK, we recommend a 2 step approach to upgrade:

+>

+>- Upgrade to the new SDK and disable ODE, both together as part of a single deployment. Wait for all nodes to upgrade.

+> - In order to disable ODE, set EnableOptimisticDirectExecution to false in the QueryRequestOptions.

+>- Enable ODE as part of second deployment for all nodes.

+- You can choose to restore any combination of provisioned throughput containers, shared throughput database, or the entire account.

+- The restore action restores all data and its index properties into a new account.

+- The duration of restore will depend on the amount of data that needs to be restored.

+- The newly restored database accountΓÇÖs consistency setting will be same as the source database accountΓÇÖs consistency settings.

+- A subset of containers under a shared throughput database cannot be restored. The entire database can be restored as a whole.

+- Database account keys. The restored account will be generated with new database account keys.

+- Firewall, VNET, Data plane RBAC or private endpoint settings.

+- Regions. The restored account will only be a single region account, which is the write region of the source account.

+- Stored procedures, triggers, UDFs.

+- Role-based access control assignments. These will need to be re-assigned.

+- Documents that were deleted because of expired TTL.

+- Analytical data when synapse link is enabled.

+- Materialized views

+### Enterprise Agreement customers

+If the original reservation purchase was made from an overage, the refund is returned to you as a partial credit note. The refund doesnΓÇÖt affect the original or later invoices.

+* Snowflake

+- [Snowflake](connector-snowflake.md)

+- Connectors - Integration Runtime

+DDoS Protection metrics alerts are an important step in alerting your team through Azure portal, email, SMS message, push, or voice notification when an attack is detected.

+In this tutorial, you learn how to:

+> [!div class="checklist"]

+> * Configure metrics alerts through Azure Monitor.

+ :::image type="content" source="./media/ddos-alerts/ddos-protection-alert-page.png" alt-text="Screenshot of creating Alerts." lightbox="./media/ddos-alerts/ddos-protection-alert-page.png":::

+ :::image type="content" source="./media/ddos-alerts/ddos-protection-alert-scope.png" alt-text="Screenshot of selecting DDoS Protection attack alert scope." lightbox="./media/ddos-alerts/ddos-protection-alert-scope.png":::

+ :::image type="content" source="./media/ddos-alerts/ddos-protection-alert-add-condition.png" alt-text="Screenshot of adding DDoS Protection attack alert condition." lightbox="./media/ddos-alerts/ddos-protection-alert-add-condition.png":::

+ :::image type="content" source="./media/ddos-alerts/ddos-protection-alert-signal.png" alt-text="Screenshot of adding DDoS Protection attack alert signal." lightbox="./media/ddos-alerts/ddos-protection-alert-signal.png":::

+ :::image type="content" source="./media/ddos-alerts/ddos-protection-alert-action-group-notification.png" alt-text="Screenshot of adding DDoS Protection attack alert notification type." lightbox="./media/ddos-alerts/ddos-protection-alert-action-group-notification.png":::

+ :::image type="content" source="./media/ddos-alerts/ddos-protection-alert-notification.png" alt-text="Screenshot of adding DDoS Protection attack alert notification page." lightbox="./media/ddos-alerts/ddos-protection-alert-notification.png":::

+ :::image type="content" source="./media/ddos-alerts/ddos-protection-alert-details.png" alt-text="Screenshot of adding DDoS Protection attack alert details page." lightbox="./media/ddos-alerts/ddos-protection-alert-details.png":::

+ :::image type="content" source="./media/ddos-alerts/ddos-protection-alert-rule.png" alt-text="Screenshot of Alerts page." lightbox="./media/ddos-alerts/ddos-protection-alert-rule.png":::

+ :::image type="content" source="./media/ddos-alerts/ddos-protection-delete-alert-rules.png" alt-text="Screenshot of Alert rules page." lightbox="./media/ddos-alerts/ddos-protection-delete-alert-rules.png":::

+In this tutorial you learned how to configure metric alerts through Azure portal.

+To configure diagnostic logging, continue to the next tutorial.

+> [!div class="nextstepaction"]

+> [Configure diagnostic logging](diagnostic-logging.md)

+> [Test through simulations](test-through-simulations.md)

+> [View alerts in Microsoft Defender for Cloud](ddos-view-alerts-defender-for-cloud.md)

+In this tutorial, you learn how to:

+> [!div class="checklist"]

+> * Configure a Log Analytics workspace for DDoS Protection.

+In this tutorial you learned how to create a Log Analytics workspace for Azure DDoS Protection. To learn how to configure alerts, continue to the next article.

+> [!div class="nextstepaction"]

+> [Configure diagnostic logging alerts](ddos-diagnostic-alert-templates.md)

+DDoS Protection diagnostic logging alerts provide visibility into DDoS attacks and mitigation actions. You can configure alerts for all DDoS protected public IP addresses that you have enabled diagnostic logging on.

+In this tutorial, you learn how to:

+> [!div class="checklist"]

+> * Configure diagnostic logging alerts through Azure Monitor and Logic App.

+ :::image type="content" source="./media/ddos-diagnostic-alert-templates/ddos-deploy-alert.png" alt-text="Screenshot of Azure Monitor alert rule template." lightbox="./media/ddos-diagnostic-alert-templates/ddos-deploy-alert.png":::

+ :::image type="content" source="./media/ddos-diagnostic-alert-templates/ddos-deploy-alert-logic-app.png" alt-text="Screenshot of DDoS Mitigation Alert Enrichment template." lightbox="./media/ddos-diagnostic-alert-templates/ddos-deploy-alert-logic-app.png":::

+ :::image type="content" source="./media/ddos-diagnostic-alert-templates/ddos-protection-alert-rule.png" alt-text="Screenshot of Alerts page." lightbox="./media/ddos-diagnostic-alert-templates/ddos-protection-alert-rule.png":::

+ :::image type="content" source="./media/ddos-diagnostic-alert-templates/ddos-protection-delete-alert-rules.png" alt-text="Screenshot of Alert rules page." lightbox="./media/ddos-diagnostic-alert-templates/ddos-protection-delete-alert-rules.png":::

+In this tutorial you learned how to configure diagnostic alerts through Azure portal.

+To test DDoS Protection through simulations, continue to the next guide.

+> [!div class="nextstepaction"]

+> [Test through simulations](test-through-simulations.md)

+> [View alerts in Microsoft Defender for Cloud](ddos-view-alerts-defender-for-cloud.md)

+## Tiers

+### DDoS Network Protection

+Azure DDoS Network Protection, combined with application design best practices, provides enhanced DDoS mitigation features to defend against DDoS attacks. It's automatically tuned to help protect your specific Azure resources in a virtual network. For more information about enabling DDoS Network Protection, see [Quickstart: Create and configure Azure DDoS Network Protection using the Azure portal](manage-ddos-protection.md).

+### DDoS IP Protection

+DDoS IP Protection is a pay-per-protected IP model. DDoS IP Protection contains the same core engineering features as DDoS Network Protection, but will differ in the following value-added

+For more information about the tiers, see [Tier comparison](ddos-protection-sku-comparison.md).

+Azure DDoS Protection supports two tier types, DDoS IP Protection and DDoS Network Protection. The tier is configured in the Azure portal during the workflow when you configure Azure DDoS Protection.

+- PaaS services (multi-tenant), which includes Azure App Service Environment for Power Apps, Azure API Management in deployment modes other than APIM with virtual network integration (For more information see https://techcommunity.microsoft.com/t5/azure-network-security-blog/azure-ddos-standard-protection-now-supports-apim-in-vnet/ba-p/3641671), and Azure Virtual WAN aren't currently supported.

+- VPN gateway or Virtual network gateway is protected by a fixed DDoS policy. Adaptive tuning isn't supported at this stage.

+- Disabling DDoS protection for a public IP address is currently a preview feature. If you disable DDoS protection for a public IP resource that is linked to a virtual network with an active DDoS protection plan, you'll still be billed for DDoS Network Protection. However, the following functionalities will be suspended: mitigation of DDoS attacks, telemetry, and logging of DDoS mitigation events.

+In this tutorial, you learn how to:

+> [!div class="checklist"]

+> * View Azure DDoS Protection alerts in Microsoft Defender for Cloud.

+ :::image type="content" source="./media/ddos-view-alerts-defender-for-cloud/ddos-protection-security-alerts.png" alt-text="Screenshot of Security alert in Microsoft Defender for Cloud." lightbox="./media/ddos-view-alerts-defender-for-cloud/ddos-protection-security-alerts.png":::

+In this tutorial you learned how to view DDoS protection alerts in Microsoft Defender for Cloud. To learn more about the recommended steps to take when you receive an alert, see these next steps.

+> [!div class="nextstepaction"]

+> [Engage with Azure DDoS Rapid Response](ddos-rapid-response.md)

+> [components of a DDoS Rapid Response Strategy](ddos-response-strategy.md)

+DDoS Protection diagnostic logs provide you with the ability to view DDoS Protection notifications, mitigation reports and mitigation flow logs after a DDoS attack. You can view these logs in your Log Analytics workspace.

+In this tutorial, you learn how to:

+> [!div class="checklist"]

+> * view Azure DDoS Protection diagnostic logs including notifications, mitigation reports and mitigation flow logs.

+In this tutorial you learned how to view DDoS Protection diagnostic logs in a Log Analytics workspace. To learn more about the recommended steps to take when you receive a DDoS attack, see these next steps.

+> [!div class="nextstepaction"]

+> [Engage with Azure DDoS Rapid Response](ddos-rapid-response.md)

+> [components of a DDoS Rapid Response Strategy](ddos-response-strategy.md)

+Configure diagnostic logging for Azure DDoS Protection to gain visibility into DDoS attacks.

+In this tutorial, you learn how to:

+> [!div class="checklist"]

+> * Configure diagnostic logs.

+> * Query logs in log analytics workspace.

+In this tutorial you learned how to configure diagnostic logging for DDoS Protection. To learn how to configure diagnostic logging alerts, continue to the next article.

+> [!div class="nextstepaction"]

+> [Configure diagnostic logging alerts](ddos-diagnostic-alert-templates.md)

+> [Test through simulations](test-through-simulations.md)

+> [View logs in Log Analytics workspace](ddos-view-diagnostic-logs.md)

+- Check out common questions about agentless scanning and [how it affects the subscription/account](faq-cspm.yml#how-does-scanning-affect-the-account-subscription-), [agentless data collection](faq-data-collection-agents.yml#agentless), and [permissions used by agentless](faq-permissions.yml#which-permissions-are-used-by-agentless-scanning-).

+ Title: Security alert correlation | Defender for Cloud in the Field

+description: Security alert correlation

+# Security alert correlation

+**Episode description**: In this episode of Defender for Cloud in the Field, Daniel Davrayev joins Yuri Diogenes to talk about security alert correlation capability in Defender for Cloud. Daniel talks about the importance of have a built-in capability to correlate alerts in Defender for Cloud, how this saves time for SOC analysts to investigate alert and respond to potential threats. Daniel also explains how data correlation works and demonstrate how this correlation appears in Defender for Cloud dashboard as a security incident.

+<br>

+<br>

+<iframe src="https://aka.ms/docs/player?id=6573561d-70a6-4b4c-ad16-9efe747c9a61" width="1080" height="530" allowFullScreen="true" frameBorder="0"></iframe>

+- [00:00](/shows/mdc-in-the-field/security-alert-correlation#time=00m00s) - Intro

+- [02:15](/shows/mdc-in-the-field/security-alert-correlation#time=02m15s) - How Defender for Cloud handles alert prioritization

+- [04:29](/shows/mdc-in-the-field/security-alert-correlation#time=04m29s) - How Defender for Cloud can help with alert correlation

+- [07:05](/shows/mdc-in-the-field/security-alert-correlation#time=07m05s) - How Defender for Cloud creates alerts correlation

+- [09:06](/shows/mdc-in-the-field/security-alert-correlation#time=09m06s) - Does alert correlation works across different Defender for Cloud plans?

+- [11:42](/shows/mdc-in-the-field/security-alert-correlation#time=11m42s) - Demonstration

+## Recommended resources

+- [Learn more](https://techcommunity.microsoft.com/t5/microsoft-defender-for-cloud/correlating-alerts-in-microsoft-defender-for-cloud/ba-p/3839209)

+- Subscribe to [Microsoft Security on YouTube](https://www.youtube.com/playlist?list=PL3ZTgFEc7LysiX4PfHhdJPR7S8mGO14YS)

+- Learn more about [Microsoft Security](https://msft.it/6002T9HQY)

+- Follow us on social media:

+ - [LinkedIn](https://www.linkedin.com/showcase/microsoft-security/)

+ - [Twitter](https://twitter.com/msftsecurity)

+- Join our [Tech Community](https://aka.ms/SecurityTechCommunity)

+## Next steps

+> [!div class="nextstepaction"]

+> [New AWS Connector in Microsoft Defender for Cloud](episode-one.md)

+ :::image type="content" source="media/enable-data-collection/event-collection-workspace.png" alt-text="Screenshot of setting the security event data to store in a workspace." lightbox="media/enable-data-collection/event-collection-workspace.png":::

+1. In the Azure portal, navigate to the Defender for Cloud's **Environment Settings** page.

+1. Select the relevant subscription and then select **Settings & monitoring**.

+1. Turn Log Analytics agent/Azure Monitor Agent **Off**.

+ :::image type="content" source="media/working-with-log-analytics-agent/manual-provision.png" alt-text="Screenshot of turning off the Log Analytics setting." lightbox="media/working-with-log-analytics-agent/manual-provision.png":::

+1. Optionally, create a workspace.

+ 1. Select one or both "Servers" or "SQL servers on machines"(Foundational CSPM is the free default), and then select **Save**.

+

+ :::image type="content" source="media/working-with-log-analytics-agent/apply-plan-to-workspace.png" alt-text="Screenshot that shows where to set the workspace on which you're installing the agent." lightbox="media/working-with-log-analytics-agent/apply-plan-to-workspace.png":::

+> [!TIP]

+> Any alerts generated from different sensors in the same zone within a 10-minute timeframe, with the same type, status, alert protocol, and associated devices, are listed as a single, unified alert.

+>

+> - The 10-minute timeframe is based on the alert's *first detection* time.

+> - The single, unified alert lists all of the sensors that detected the alert.

+> - Alerts are combined based on the *alert* protocol, and not the device protocol.

+>

+- [Plan OT sites and zones](best-practices/plan-corporate-monitoring.md#plan-ot-sites-and-zones)

+ Title: "Custom roles for MySQL to Azure Database for MySQL migrations using Database Migration Service"

+description: Learn to use the custom roles for MySQL to Azure Database for MySQL migrations.

+# Custom roles for MySQL to Azure Database for MySQL migrations using Database Migration Service

+This article explains how to set up a custom role for MySQL to Azure Database for MySQL migrations using DMS.

+The role has no permission to create a new Database Migration Service and no permission to create a database migration project. Meaning the user assigned to the custom role will need to have an already created Database Migration Service and migration project under the assigned resource group. The user will then be able to create and run migration activities under the migration project.

+```json

+{

+ "properties": {

+ "roleName": "DmsCustomRoleDemoforMySQL",

+ "description": "",

+ "assignableScopes": [

+ "/subscriptions/<DMSSubscription>/resourceGroups/<dmsServiceRG>"

+ ],

+ "permissions": [

+ {

+ "actions": [

+ "Microsoft.DataMigration/locations/operationResults/read",

+ "Microsoft.DataMigration/locations/operationStatuses/read",

+ "Microsoft.DataMigration/services/read",

+ "Microsoft.DataMigration/services/stop/action",

+ "Microsoft.DataMigration/services/start/action",

+ "Microsoft.DataMigration/services/checkStatus/*",

+ "Microsoft.DataMigration/services/configureWorker/action",

+ "Microsoft.DataMigration/services/addWorker/action",

+ "Microsoft.DataMigration/services/removeWorker/action",

+ "Microsoft.DataMigration/services/updateAgentConfig/action",

+ "Microsoft.DataMigration/services/slots/read",

+ "Microsoft.DataMigration/services/projects/*",

+ "Microsoft.DataMigration/services/serviceTasks/read",

+ "Microsoft.DataMigration/services/serviceTasks/write",

+ "Microsoft.DataMigration/services/serviceTasks/delete",

+ "Microsoft.DataMigration/services/serviceTasks/cancel/action",

+ "Microsoft.DBforMySQL/flexibleServers/read",

+ "Microsoft.DBforMySQL/flexibleServers/databases/read",

+ "Microsoft.DBforMySQL/servers/read",

+ "Microsoft.DBforMySQL/servers/databases/read",

+ "Microsoft.Resources/subscriptions/resourceGroups/read",

+ "Microsoft.DataMigration/skus/read"

+ ],

+ "notActions": [],

+ "dataActions": [],

+ "notDataActions": []

+ }

+ ]

+ }

+}

+```

+You can use either the Azure portal, AZ PowerShell, Azure CLI or Azure REST API to create the roles.

+For more information, see the articles [Create custom roles using the Azure portal](../role-based-access-control/custom-roles-portal.md) and [Azure custom roles](../role-based-access-control/custom-roles.md).

+## Role assignment

+To assign a role to users, open the Azure portal, perform the following steps:

+1. Navigate to the resource, go to **Access Control**, and then scroll to find the custom roles you created.

+2. Select the appropriate role, select the User, and then save the changes.

+ The user now appears listed on the **Role assignments** tab.

+## Next steps

+* For information about Azure Database for MySQL - Flexible Server, see [Overview - Azure Database for MySQL Flexible Server](./../mysql/flexible-server/overview.md).

+* For information about Azure Database Migration Service, see [What is Azure Database Migration Service?](./dms-overview.md).

+* For information about known issues and limitations when migrating to Azure Database for MySQL - Flexible Server using DMS, see [Known Issues With Migrations To Azure Database for MySQL - Flexible Server](./known-issues-azure-mysql-fs-online.md).

+* For information about known issues and limitations when performing migrations using DMS, see [Common issues - Azure Database Migration Service](./known-issues-troubleshooting-dms.md).

+* For troubleshooting source database connectivity issues while using DMS, see article [Issues connecting source databases](./known-issues-troubleshooting-dms-source-connectivity.md).

+ Title: "Tutorial: Migrate from MySQL to Azure Database for MySQL - Flexible Server online using DMS via the Azure portal"

+description: "Learn to perform an online migration from MySQL to Azure Database for MySQL - Flexible Server by using Azure Database Migration Service."

+# Tutorial: Migrate from MySQL to Azure Database for MySQL - Flexible Server online using DMS via the Azure portal

+> [!NOTE]

+> This article contains references to the term *slave*, a term that Microsoft no longer uses. When the term is removed from the software, we'll remove it from this article.

+You can migrate your on-premises or other cloud services MySQL Server to Azure Database for MySQL ΓÇô Flexible Server by using Azure Database Migration Service (DMS), a fully managed service designed to enable seamless migrations from multiple database sources to Azure data platforms. In this tutorial, weΓÇÖll perform an online migration of a sample database from an on-premises MySQL server to an Azure Database for MySQL - Flexible Server (both running version 5.7) using a DMS migration activity.

+> [!NOTE]

+> DMS online migration is now generally available. DMS supports migration to MySQL versions 5.7 and 8.0 and also supports migration from lower version MySQL servers (v5.6 and above) to higher version servers. In addition, DMS supports cross-region, cross-resource group, and cross-subscription migrations, so you can select a region, resource group, and subscription for the target server that is different than what is specified for your source server.

+In this tutorial, you'll learn how to:

+> [!div class="checklist"]

+>

+> * Implement best practices for creating a flexible server for faster data loads using DMS.

+> * Create and configure a target flexible server.

+> * Create a DMS instance.

+> * Create a MySQL migration project in DMS.

+> * Migrate a MySQL schema using DMS.

+> * Run the migration.

+> * Monitor the migration.

+> * Perform post-migration steps.

+> * Implement best practices for performing a migration.

+## Prerequisites

+To complete this tutorial, you need to:

+* Create or use an existing MySQL instance (the source server).

+* To complete the online migration successfully, ensure that the following prerequisites are in place:

+ * Use the MySQL command line tool of your choice to verify that log_bin is enabled on the source server by running the command: SHOW VARIABLES LIKE 'log_binΓÇÖ. If log_bin isn't enabled, be sure to enable it before starting the migration.

+ * Ensure that the user has ΓÇ£REPLICATION CLIENTΓÇ¥ and ΓÇ£REPLICATION SLAVEΓÇ¥ permissions on the source server for reading and applying the bin log.

+ * If you're targeting an online migration, you will need to configure the binlog expiration on the source server to ensure that binlog files aren't purged before the replica commits the changes. We recommend at least two days to start. The parameter will depend on the version of your MySQL server. For MySQL 5.7 the parameter is expire_logs_days (by default it is set to 0, which is no auto purge). For MySQL 8.0 it is binlog_expire_logs_seconds (by default it is set to 30 days). After a successful cutover, you can reset the value.

+* To complete a schema migration successfully, on the source server, the user performing the migration requires the following privileges:

+ * ΓÇ£READΓÇ¥ privilege on the source database.

+ * ΓÇ£SELECTΓÇ¥ privilege for the ability to select objects from the database

+ * If migrating views, the user must have the ΓÇ£SHOW VIEWΓÇ¥ privilege.

+ * If migrating triggers, the user must have the ΓÇ£TRIGGERΓÇ¥ privilege.

+ * If migrating routines (procedures and/or functions), the user must be named in the definer clause of the routine. Alternatively, based on version, the user must have the following privilege:

+ * For 5.7, have ΓÇ£SELECTΓÇ¥ access to the ΓÇ£mysql.procΓÇ¥ table.

+ * For 8.0, have ΓÇ£SHOW_ROUTINEΓÇ¥ privilege or have the ΓÇ£CREATE ROUTINE,ΓÇ¥ ΓÇ£ALTER ROUTINE,ΓÇ¥ or ΓÇ£EXECUTEΓÇ¥ privilege granted at a scope that includes the routine.

+ * If migrating events, the user must have the ΓÇ£EVENTΓÇ¥ privilege for the database from which the events are to be shown.

+## Limitations

+As you prepare for the migration, be sure to consider the following limitations.

+* When migrating non-table objects, DMS doesn't support renaming databases.

+* When migrating to a target server with bin_log enabled, be sure to enable log_bin_trust_function_creators to allow for creation of routines and triggers.

+* Currently, DMS doesn't support migrating the DEFINER clause for objects. All object types with definers on the source are dropped and after the migration, the default definer for all objects that support a definer clause and that are created during schema migration, will be set to the login used to run the migration.

+* Currently, DMS only supports migrating a schema as part of data movement. If nothing is selected for data movement, the schema migration won't occur. Note that selecting a table for schema migration also selects it for data movement.

+* Online migration support is limited to the ROW binlog format.

+* Online migration only replicates DML changes; replicating DDL changes isn't supported. Don't make any schema changes to the source while replication is in progress, if DMS detects DDL while replicating, it will generate a warning that can be viewed in the Azure portal.

+## Best practices for creating a flexible server for faster data loads using DMS

+DMS supports cross-region, cross-resource group, and cross-subscription migrations, so you're free to select appropriate region, resource group and subscription for your target flexible server. Before you create your target flexible server, consider the following configuration guidance to help ensure faster data loads using DMS.

+* Select the compute size and compute tier for the target flexible server based on the source single serverΓÇÖs pricing tier and VCores based on the detail in the following table.

+ | Single Server Pricing Tier | Single Server VCores | Flexible Server Compute Size | Flexible Server Compute Tier |

+ | - | - |:-:|:-:|

+ | Basic\* | 1 | General Purpose | Standard_D16ds_v4 |

+ | Basic\* | 2 | General Purpose | Standard_D16ds_v4 |

+ | General Purpose\* | 4 | General Purpose | Standard_D16ds_v4 |

+ | General Purpose\* | 8 | General Purpose | Standard_D16ds_v4 |

+ | General Purpose | 16 | General Purpose | Standard_D16ds_v4 |

+ | General Purpose | 32 | General Purpose | Standard_D32ds_v4 |

+ | General Purpose | 64 | General Purpose | Standard_D64ds_v4 |

+ | Memory Optimized | 4 | Business Critical | Standard_E4ds_v4 |

+ | Memory Optimized | 8 | Business Critical | Standard_E8ds_v4 |

+ | Memory Optimized | 16 | Business Critical | Standard_E16ds_v4 |

+ | Memory Optimized | 32 | Business Critical | Standard_E32ds_v4 |

+\* For the migration, select General Purpose 16 vCores compute for the target flexible server for faster migrations. Scale back to the desired compute size for the target server after migration is complete by following the compute size recommendation in the Performing post-migration activities section later in this article.

+* The MySQL version for the target flexible server must be greater than or equal to that of the source single server.

+* Unless you need to deploy the target flexible server in a specific zone, set the value of the Availability Zone parameter to ΓÇÿNo preferenceΓÇÖ.

+* For network connectivity, on the Networking tab, if the source single server has private endpoints or private links configured, select Private Access; otherwise, select Public Access.

+* Copy all firewall rules from the source single server to the target flexible server.

+* Copy all the name/value tags from the single to flex server during creation itself.

+## Create and configure the target flexible server

+With these best practices in mind, create your target flexible server, and then configure it.

+* Create the target flexible server. For guided steps, see the quickstart [Create an Azure Database for MySQL flexible server](./../mysql/flexible-server/quickstart-create-server-portal.md).

+* Configure the new target flexible server as follows:

+ * The user performing the migration requires the following permissions:

+ * Ensure that the user has ΓÇ£REPLICATION_APPLIERΓÇ¥ or ΓÇ£BINLOG_ADMINΓÇ¥ permission on target server for applying the bin log.

+ * Ensure that the user has ΓÇ£REPLICATION SLAVEΓÇ¥ permission on target server.

+ * Ensure that the user has ΓÇ£REPLICATION CLIENTΓÇ¥ and ΓÇ£REPLICATION SLAVEΓÇ¥ permission on source server for reading and applying the bin log.

+ * To create tables on the target, the user must have the ΓÇ£CREATEΓÇ¥ privilege.

+ * If migrating a table with ΓÇ£DATA DIRECTORYΓÇ¥ or ΓÇ£INDEX DIRECTORYΓÇ¥ partition options, the user must have the ΓÇ£FILEΓÇ¥ privilege.

+ * If migrating to a table with a ΓÇ£UNIONΓÇ¥ option, the user must have the ΓÇ£SELECT,ΓÇ¥ ΓÇ£UPDATE,ΓÇ¥ and ΓÇ£DELETEΓÇ¥ privileges for the tables you map to a MERGE table.

+ * If migrating views, you must have the ΓÇ£CREATE VIEWΓÇ¥ privilege.

+ Keep in mind that some privileges may be necessary depending on the contents of the views. Refer to the MySQL docs specific to your version for ΓÇ£CREATE VIEW STATEMENTΓÇ¥ for details.

+ * If migrating events, the user must have the ΓÇ£EVENTΓÇ¥ privilege.

+ * If migrating triggers, the user must have the ΓÇ£TRIGGERΓÇ¥ privilege.

+ * If migrating routines, the user must have the ΓÇ£CREATE ROUTINEΓÇ¥ privilege.

+ * Configure the server parameters on the target flexible server as follows:

+ * Set the TLS version and require_secure_transport server parameter to match the values on the source server.

+ * Set the sql_mode server parameter to match the values on the source server.

+ * Configure server parameters on the target server to match any non-default values used on the source server.

+ * To ensure faster data loads when using DMS, configure the following server parameters as described.

+ * max_allowed_packet ΓÇô set to 1073741824 (i.e., 1 GB) to prevent any connection issues due to large rows.

+ * slow_query_log ΓÇô set to OFF to turn off the slow query log. This will eliminate the overhead caused by slow query logging during data loads.

+ * innodb_buffer_pool_size ΓÇô can only be increased by scaling up compute for Azure Database for MySQL server. Scale up the server to 64 vCore General Purpose SKU from the Pricing tier of the portal during migration to increase the innodb_buffer_pool_size.

+ * innodb_io_capacity & innodb_io_capacity_max - Change to 9000 from the Server parameters in Azure portal to improve the IO utilization to optimize for migration speed.

+ * innodb_write_io_threads - Change to 4 from the Server parameters in Azure portal to improve the speed of migration.

+ * Configure the replicas on the target server to match those on the source server.

+ * Replicate the following server management features from the source single server to the target flexible server:

+ * Role assignments, Roles, Deny Assignments, classic administrators, Access Control (IAM)

+ * Locks (read-only and delete)

+ * Alerts

+ * Tasks

+ * Resource Health Alerts

+## Set up DMS

+With your target flexible server deployed and configured, you next need to set up DMS to migrate your single server to a flexible server.

+### Register the resource provider

+To register the Microsoft.DataMigration resource provider, perform the following steps.

+1. Before creating your first DMS instance, sign in to the Azure portal, and then search for and select **Subscriptions**.

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/1-subscriptions.png" alt-text="Screenshot of a Select subscriptions from Azure Marketplace.":::

+2. Select the subscription that you want to use to create the DMS instance, and then select **Resource providers**.

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/2-resource-provider.png" alt-text="Screenshot of a Select Resource Provider.":::

+3. Search for the term ΓÇ£MigrationΓÇ¥, and then, for **Microsoft.DataMigration**, select **Register**.

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/3-register.png" alt-text="Screenshot of a Register your resource provider.":::

+### Create a Database Migration Service (DMS) instance

+1. In the Azure portal, select **+ Create a resource**, search for the term ΓÇ£Azure Database Migration ServiceΓÇ¥, and then select **Azure Database Migration Service** from the drop-down list.

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/4-dms-portal-marketplace.png" alt-text="Screenshot of a Search Azure Database Migration Service.":::

+2. On the **Azure Database Migration Service** screen, select **Create**.

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/5-dms-portal-marketplace-create.png" alt-text="Screenshot of a Create Azure Database Migration Service instance.":::

+

+3. On the **Select migration scenario and Database Migration Service** page, under **Migration scenario**, select **MySQL** as the source server type, and then select **Azure Database for MySQL** as target server type, and then select **Select**.

+ :::image type="content" source="media/tutorial-azure-mysql-external-to-flex-online/create-database-migration-service.png" alt-text="Screenshot of a Select Migration Scenario.":::

+4. On the **Create Migration Service** page, on the **Basics** tab, under **Project details**, select the appropriate subscription, and then select an existing resource group or create a new one.

+5. Under **Instance details**, specify a name for the service, select a region, and then verify that **Azure** is selected as the service mode.

+6. To the right of **Pricing tier**, select **Configure tier**.

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/7-project-details.png" alt-text="Screenshot of a Select Configure Tier.":::

+7. On the **Configure** page, select the **Premium** pricing tier with 4 vCores for your DMS instance, and then select **Apply**.

+ DMS Premium 4-vCore is free for 6 months (183 days) from the DMS service creation date before incurring any charges. For more information on DMS costs and pricing tiers, see the [pricing page](https://aka.ms/dms-pricing).

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/8-configure-pricing-tier.png" alt-text="Screenshot of a Select Pricing tier.":::

+ Next, we need to specify the VNet that will provide the DMS instance with access to the source single server and the target flexible server.

+8. On the **Create Migration Service** page, select **Next : Networking >>**.

+9. On the **Networking** tab, select an existing VNet from the list or provide the name of new VNet to create, and then select **Review + Create**.

+ For more information, see the article [Create a virtual network using the Azure portal.](./../virtual-network/quick-create-portal.md).

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/8-1-networking.png" alt-text="Screenshot of a Select Networking.":::

+ > [!IMPORTANT]

+ > Your VNet must be configured with access to both the source single server and the target flexible server, so be sure to:

+ >

+ > * Create a server-level firewall rule or [configure VNet service endpoints](./../mysql/single-server/how-to-manage-vnet-using-portal.md) for both the source and target Azure Database for MySQL servers to allow the VNet for Azure Database Migration Service access to the source and target databases.

+ > * Ensure that your VNet Network Security Group (NSG) rules don't block the outbound port 443 of ServiceTag for ServiceBus, Storage, and Azure Monitor. For more information about VNet NSG traffic filtering, see [Filter network traffic with network security groups](./../virtual-network/virtual-network-vnet-plan-design-arm.md).

+ > [!NOTE]

+ > To add tags to the service, advance to the **Tags** tab by selecting **Next : Tags**. Adding tags to the service is optional.

+10. Navigate to the **Review + create** tab, review the configurations, view the terms, and then select **Create**.

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/9-review-create.png" alt-text="Screenshot of a Select Review+Create.":::

+

+ Deployment of your instance of DMS now begins. The message **Deployment is in progress** appears for a few minutes, and then the message changes to **Your deployment is complete**.

+11. Select **Go to resource**.

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/9-1-go-to-resource.png" alt-text="Screenshot of a Select Go to resource.":::

+12. Identify the IP address of the DMS instance from the resource overview page and create a firewall rule for your source single server and target flexible server allow-listing the IP address of the DMS instance.

+### Create a migration project

+To create a migration project, perform the following steps.

+1. In the Azure portal, select **All services**, search for Azure Database Migration Service, and then select **Azure Database Migration Services**.

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/10-dms-search.png" alt-text="Screenshot of a Locate all instances of Azure Database Migration Service.":::

+2. In the search results, select the DMS instance that you created, and then select **+ New Migration Project**.

+ :::image type="content" source="media/tutorial-azure-mysql-single-to-flex-online/11-select-create.png" alt-text="Screenshot of a Select a new migration project.":::

+3. On the **New migration project** page, specify a name for the project, in the **Source server type** selection box, select **MySQL**, in the **Target server type** selection box, select **Azure Database For MySQL - Flexible Server**, in the **Migration activity type** selection box, select **Online data migration**, and then select **Create and run activity**.

+ > [!NOTE]

+ > Selecting **Create project only** as the migration activity type will only create the migration project; you can then run the migration project at a later time.

+ :::image type="content" source="media/tutorial-azure-mysql-external-to-flex-online/create-project-external.png" alt-text="Screenshot of a Create a new migration project.":::

+### Configure the migration project

+To configure your DMS migration project, perform the following steps.

+1. On the **Select source** screen, we have to ensure that DMS is in the VNet which has connectivity to the source server. Here you will input source server name, server port, username, and password to your source server.

+ :::image type="content" source="media/tutorial-azure-mysql-external-to-flex-online/select-source-server.png" alt-text="Screenshot of an Add source details screen.":::

+2. Select **Next : Select target>>**, and then, on the **Select target** screen, locate the server based on the subscription, location, and resource group. The user name is auto populated, then provide the password for the target flexible server.

+

+ :::image type="content" source="media/tutorial-mysql-to-azure-mysql-online/select-target-online.png" alt-text="Screenshot of a Select target.":::

+3. Select **Next : Select databases>>**, and then, on the **Select databases** tab, under **Server migration options**, select **Migrate all applicable databases** or under **Select databases** select the server objects that you want to migrate.

+ > [!NOTE]

+ > There is now a **Migrate all applicable databases** option. When selected, this option will migrate all user created databases and tables. Note that because Azure Database for MySQL - Flexible Server does not support mixed case databases, mixed case databases on the source will not be included for an online migration.

+ :::image type="content" source="media/tutorial-azure-mysql-external-to-flex-online/select-databases.png" alt-text="Screenshot of a Select database.":::

+4. In the **Select databases** section, under **Source Database**, select the database(s) to migrate.

+ The non-table objects in the database(s) you specified will be migrated, while the items you didnΓÇÖt select will be skipped. You can only select the source and target databases whose names match that on the source and target server.

+ If you select a database on the source server that doesnΓÇÖt exist on the target server, it will be created on the target server.

+5. Select **Next : Select tables>>** to navigate to the **Select tables** tab.

+ Before the tab populates, DMS fetches the tables from the selected database(s) on the source and target and then determines whether the table exists and contains data.

+6. Select the tables that you want to migrate.

+ If the selected source table doesn't exist on the target server, the online migration process will ensure that the table schema and data is migrated to the target server.

+ :::image type="content" source="media/tutorial-azure-mysql-external-to-flex-online/select-tables.png" alt-text="Screenshot of a Select Tables.":::

+ DMS validates your inputs, and if the validation passes, you'll be able to start the migration.

+7. After configuring for schema migration, select **Review and start migration**.

+ > [!NOTE]

+ > You only need to navigate to the **Configure migration settings** tab if you are trying to troubleshoot failing migrations.

+8. On the **Summary** tab, in the **Activity name** text box, specify a name for the migration activity, and then review the summary to ensure that the source and target details match what you previously specified.

+ :::image type="content" source="media/tutorial-azure-mysql-external-to-flex-online/summary-page.png" alt-text="Screenshot of a Select Summary.":::

+9. Select **Start migration**.

+ The migration activity window appears, and the Status of the activity is Initializing. The Status changes to Running when the table migrations start.

+ :::image type="content" source="media/tutorial-mysql-to-azure-mysql-online/running-online-migration.png" alt-text="Screenshot of a Running status.":::

+### Monitor the migration

+1. After the **Initial Load** activity is completed, navigate to the **Initial Load** tab to view the completion status and the number of tables completed.

+ :::image type="content" source="media/tutorial-azure-mysql-external-to-flex-online/initial-load.png" alt-text="Screenshot of a completed initial load migration.":::

+ After the **Initial Load** activity is completed, you're navigated to the **Replicate Data Changes** tab automatically. You can monitor the migration progress as the screen is auto-refreshed every 30 seconds.

+2. Select **Refresh** to update the display and view the seconds behind source when needed.

+ :::image type="content" source="media/tutorial-azure-mysql-external-to-flex-online/replicate-changes.png" alt-text="Screenshot of a Monitoring migration.":::

+3. Monitor the **Seconds behind source** and as soon as it nears 0, proceed to start cutover by navigating to the **Start Cutover** menu tab at the top of the migration activity screen.

+4. Follow the steps in the cutover window before you're ready to perform a cutover.

+5. After completing all steps, select **Confirm**, and then select **Apply**.

+ :::image type="content" source="media/tutorial-mysql-to-azure-mysql-online/21-complete-cutover-online.png" alt-text="Screenshot of a Perform cutover.":::

+## Perform post-migration activities

+When the migration has finished, be sure to complete the following post-migration activities.

+* Perform sanity testing of the application against the target database to certify the migration.

+* Update the connection string to point to the new flexible server.

+* Delete the source single server after you have ensured application continuity.

+* If you scaled-up the target flexible server for faster migration, scale it back by selecting the compute size and compute tier for the flexible server based on the source single serverΓÇÖs pricing tier and VCores, based on the detail in the following table.

+ | Single Server Pricing Tier | Single Server VCores | Flexible Server Compute Size | Flexible Server Compute Tier |

+ | - | - |:-:|:-:|

+ | Basic | 1 | Burstable | Standard_B1s |

+ | Basic | 2 | Burstable | Standard_B2s |

+ | General Purpose | 4 | General Purpose | Standard_D4ds_v4 |

+ | General Purpose | 8 | General Purpose | Standard_D8ds_v4 |

+* To clean up the DMS resources, perform the following steps:

+ 1. In the Azure portal, select **All services**, search for Azure Database Migration Service, and then select **Azure Database Migration Services**.

+ 2. Select your migration service instance from the search results, and then select **Delete service**.

+ 3. In the confirmation dialog box, in the **TYPE THE DATABASE MIGRATION SERVICE NAME** textbox, specify the name of the instance, and then select **Delete**.

+## Migration best practices

+When performing a migration, be sure to consider the following best practices.

+* As part of discovery and assessment, take the server SKU, CPU usage, storage, database sizes, and extensions usage as some of the critical data to help with migrations.

+* Perform test migrations before migrating for production:

+ * Test migrations are important for ensuring that you cover all aspects of the database migration, including application testing. The best practice is to begin by running a migration entirely for testing purposes. After a newly started migration enters the Replicate Data Changes phase with minimal lag, only use your Flexible Server target for running test workloads. Use that target for testing the application to ensure expected performance and results. If you're migrating to a higher MySQL version, test for application compatibility.

+ * After testing is completed, you can migrate the production databases. At this point, you need to finalize the day and time of production migration. Ideally, there's low application use at this time. All stakeholders who need to be involved should be available and ready. The production migration requires close monitoring. For an online migration, the replication must be completed before you perform the cutover, to prevent data loss.

+* Redirect all dependent applications to access the new primary database and make the source server read-only. Then, open the applications for production usage.

+* After the application starts running on the target flexible server, monitor the database performance closely to see if performance tuning is required.

+## Next steps

+* For information about Azure Database for MySQL - Flexible Server, see [Overview - Azure Database for MySQL Flexible Server](./../mysql/flexible-server/overview.md).

+* For information about Azure Database Migration Service, see [What is Azure Database Migration Service?](./dms-overview.md).

+* For information about known issues and limitations when migrating to Azure Database for MySQL - Flexible Server using DMS, see [Known Issues With Migrations To Azure Database for MySQL - Flexible Server](./known-issues-azure-mysql-fs-online.md).

+* For information about known issues and limitations when performing migrations using DMS, see [Common issues - Azure Database Migration Service](./known-issues-troubleshooting-dms.md).

+* For troubleshooting source database connectivity issues while using DMS, see article [Issues connecting source databases](./known-issues-troubleshooting-dms-source-connectivity.md).

+For example, following ARM template can be used to create an event hub with capture enabled. Azure Storage or Azure Data Lake Storage Gen 2 can be used as the capture destination and user assigned identity is used as the authentication method. The resource ID of the destination can point to a resource in a different subscription.

+ },

+ "identity": {

+ "type": "UserAssigned",

+ "userAssignedIdentities": {

+ "xxxxxxxx": {}

+ }

+ }

+## Schema validation for Event Hubs SDK based applications

+You can use Azure Schema Registry to perform schema validation when you stream data with your Event Hubs SDK-based applications.

+Azure Schema Registry of Event Hubs provides a centralized repository for managing schemas and you can seamlessly connect your new or existing applications with Schema Registry.

+To learn more, see [Validate schemas with Event Hubs SDK](schema-registry-dotnet-send-receive-quickstart.md).

+## Schema validation for Kafka with Schema Registry

+You can use Azure Schema Registry to perform schema validation when you stream data with your Kafka applications using Event Hubs.

+Azure Schema Registry of Event Hubs provides a centralized repository for managing schemas and you can seamlessly connect your new or existing Kafka applications with Schema Registry.

+To learn more, see [Validate schemas for Apache Kafka applications using Avro](schema-registry-kafka-java-send-receive-quickstart.md).

+> [!NOTE]

+> There's no requirement to enable this feature with a feature flag or Azure PowerShell commands.

+ You can use the following three network rules to configure your firewall. You may need to adapt these rules based on your deployment. The first rule allows access to port 9000 via TCP. The second rule allows access to port 1194 and 123 via UDP. Both these rules will only allow traffic destined to the Azure Region CIDR that we're using, in this case East US.

+```azurecli

+```

+```azurecli

+## General

+* The Spark 3.3.1 (HDInsight 5.1) cluster comes with Hive Warehouse Connector (HWC) 2.1, which works together with the Interactive Query (HDInsight 5.1) cluster.

+> [!IMPORTANT]

+> This release addresses the following CVEs released by [MSRC](https://msrc.microsoft.com/update-guide/vulnerability) on August 8, 2023. The action is to update to the latest image **2307201242**. Customers are advised to plan accordingly.

+|CVE | Severity| CVE Title|

+|-|-|-|

+|[CVE-2023-35393](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-35393)| Important|Azure Apache Hive Spoofing Vulnerability|

+|[CVE-2023-35394](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-35394)| Important|Azure HDInsight Jupyter Notebook Spoofing Vulnerability|

+|[CVE-2023-36877](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36877)| Important|Azure Apache Oozie Spoofing Vulnerability|

+|[CVE-2023-36881](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36881)| Important|Azure Apache Ambari Spoofing Vulnerability|

+|[CVE-2023-38188](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-38188)| Important|Azure Apache Hadoop Spoofing Vulnerability|

+

+* The max length of cluster name will be changed to 45 from 59 characters, to improve the security posture of clusters. Customers need to plan for the updates before 30, September 2023.

+ * To improve the overall security posture of the HDInsight clusters, HDInsight clusters using custom VNETs need to ensure that the user needs to have permission for `Microsoft Network/virtualNetworks/subnets/join/action` to perform create operations. Customers would need to plan accordingly as this change would be a mandatory check to avoid cluster creation failures before 30, September 2023.ΓÇ»

+ * On 31 August 2024, we'll retire Basic and Standard A-series VMs. Before that date, you need to migrate your workloads to Av2-series VMs, which provide more memory per vCPU and faster storage on solid-state drives (SSDs). To avoid service disruptions, [migrate your workloads](https://aka.ms/Av1retirement) from Basic and Standard A-series VMs to Av2-series VMs before 31, August 2024.

+#### DICOM Service

+**API Version 2 is Generally Available (GA)**

+The DICOM service API v2 is now Generally Available (GA) and introduces [several changes and new features](dicom/dicom-service-v2-api-changes.md). Most notable is the change to validation of DICOM attributes during store (STOW) operations - beginning with v2, the request fails only if **required attributes** fail validation. See the [DICOM Conformance Statement v2](dicom/dicom-services-conformance-statement-v2.md) for full details.

+It is important to note that floating IP configured on the Azure cross-region Load Balancer operates independently of floating IP configurations on backend regional load balancers. If floating IP is enabled on the cross-region load balancer, the appropriate loopback interface needs to be added to the backend VMs.

+* UDP traffic on port 3 isn't supported on Cross-Region Load Balancer

+Use [az network nic ip-config inbound-nat-rule add](/cli/azure/network/nic/ip-config/inbound-nat-rule) to add the inbound NAT rule to a VM's NIC

+ --frontend-port 500

+ az network nic ip-config inbound-nat-rule add \

+ --resource-group myResourceGroup \

+ --nic-name MyNic \

+ --ip-config-name MyIpConfig \

+ --inbound-nat-rule MyNatRule \

+ --lb-name myLoadBalancer

+ Title: Move across resource group or subscription

+description: Learn how to move an Azure Load testing resource to another resource group or subscription.

+# Move an Azure Load Testing resource to another resource group or subscription

+This article describes how to move your Azure Load Testing resource to either another Azure subscription or another resource group under the same subscription.

+If you want to move Azure Load Testing to a new region, see [Move an Azure Load Testing resource to another region](./how-to-move-between-regions.md).

+When you move an Azure Load Testing resource across resource groups or subscriptions, the following guidance applies:

+- Moving a resource to a new resource group or subscription is a metadata change that shouldn't affect the data. For example, the test and test runs data is preserved.

+- Moving a resource changes the ID of the resource. The standard format for a resource ID is `/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}`. When a resource is moved to a new resource group or subscription, one or more values in the path are impacted. After the resource has been moved, you'll need to update your tools and scripts to use the new resource IDs.

+- Moving a resource across subscriptions is allowed only for subscriptions in the same tenant.

+- Resource move is not supported for Azure Load Testing resources that are encrypted with a customer-managed key.

+- Moving a resource only moves it to a new resource group or subscription. It doesn't change the location of the resource.

+- Any service principal that is currently scoped to a resource, resource group or subscription might not have access to the resource after the move.

+- Automated resource provisioning using ARM templates or Bicep must be updated to the new resource group and / or subscription.

+- For tests that previously ran from Azure Pipelines, the URL to view detailed results from Azure portal will not work after the resources have been moved.

+- If the resource is moved across subscriptions, the service limits of the target subscription apply to the resource after the move.

+- Moving a resource that has a test that is configured for private endpoint testing to another subscription, results in an error while running the test. When the move is complete, you must update the test with a VNet and subnet from the new subscription.

+## Move across resource groups or subscriptions

+You can move an Azure Load Testing resource to a different resource group or subscription by using the Azure portal.

+1. Go to the [Azure portal](https://portal.azure.com/).

+1. Navigate to the resource that you want to move.

+1. On the subscription or resource group overview page, select **Move**.

+1. If you're moving the resource to another subscription, select the destination **Subscription**.

+1. If you're moving the resource to another resource group, select the destination **Resource group**, or create a new resource group.

+1. Select **Next**.

+1. When the validation is completes, acknowledge the warning regarding moving resources.

+1. Select **OK**.

+## Next steps

+- You can move many different types of resources between resource groups and subscriptions. For more information, see [Move resources to a new resource group or subscription](/azure/azure-resource-manager/management/move-resource-group-and-subscription).

+This example passes in the XPath expression, `'/produce/item/name/text()'`, to find the nodes that match the `<name></name>` node in the `'items'` XML string, and returns an array with those node values:

+`xpath(xml(parameters('items')), '/produce/item/name/text()')`

+Here's the result array populated with values of the nodes that match `<name></name>`:

+`[ Gala, Honeycrisp ]`

+Don't store training data on the notebooks file share. For information on the various options to store data, see [Access data in a job](how-to-read-write-data-v2.md).

+You can use the `/tmp` directory on the compute instance for your temporary data. However, don't write large files of data on the OS disk of the compute instance. OS disk on compute instance has 128-GB capacity. You can also store temporary training data on temporary disk mounted on /mnt. Temporary disk size is based on the VM size chosen and can store larger amounts of data if a higher size VM is chosen. Any software packages you install are saved on the OS disk of compute instance. Note customer managed key encryption is currently not supported for OS disk. The OS disk for compute instance is encrypted with Microsoft-managed keys.

+If you don't see these options, make sure you've enabled the **Connect compute instances to Visual Studio Code for the Web** preview feature, as shown in the [Prerequisites](#prerequisites) section.

+Learn how to monitor data drift and set alerts when drift is high.

+* **Monitor model data** for differences between training and serving datasets. Start by [collecting model data from deployed models](how-to-enable-data-collection.md).

+* **Set up alerts on data drift** for early warnings to potential issues.

+You can view data drift metrics with the Python SDK or in Azure Machine Learning studio. Other metrics and insights are available through the [Azure Application Insights](../../azure-monitor/app/app-insights-overview.md) resource associated with the Azure Machine Learning workspace.

+> The preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+Model accuracy degrades over time, largely because of data drift. For machine learning models, data drift is the change in model input data that leads to model performance degradation. Monitoring data drift helps detect these model performance issues.

+- Upstream process changes, such as a sensor being replaced that changes the units of measurement from inches to centimeters.

+- Change in relation between features, or covariate shift.

+Azure Machine Learning simplifies drift detection by computing a single metric abstracting the complexity of datasets being compared. These datasets may have hundreds of features and tens of thousands of rows. Once drift is detected, you drill down into which features are causing the drift. You then inspect feature level metrics to debug and isolate the root cause for the drift.

+### Dataset monitors

+The data drift algorithm provides an overall measure of change in data and indication of which features are responsible for further investigation. Dataset monitors produce many other metrics by profiling new data in the `timeseries` dataset.

+Custom alerting can be set up on all metrics generated by the monitor through [Azure Application Insights](../../azure-monitor/app/app-insights-overview.md). Dataset monitors can be used to quickly catch data issues and reduce the time to debug the issue by identifying likely causes.

+Monitor a time series dataset for drift from a previous time period. | This scenario is more general, and can be used to monitor datasets involved upstream or downstream of model building. The target dataset must have a timestamp column. The baseline dataset can be any tabular dataset that has features in common with the target dataset.

+| *Dataset* | Drift uses Machine Learning datasets to retrieve training data and compare data for model training. Generating profile of data is used to generate some of the reported metrics such as min, max, distinct values, distinct values count. |

+| *Azure Machine Learning pipeline and compute* | The drift calculation job is hosted in an Azure Machine Learning pipeline. The job is triggered on demand or by schedule to run on a compute configured at drift monitor creation time.

+### Baseline and target datasets

+You monitor [Azure Machine Learning datasets](how-to-create-register-datasets.md) for data drift. When you create a dataset monitor, you reference your:

+The monitor compares the baseline and target datasets.

+```python

+# get datastore object

+# create the Tabular dataset with 'state' and 'date' as virtual columns

+In the **Schema** settings, specify the **timestamp** column from a virtual or real column in the specified dataset. This type indicates that your data has a time component.

+If your data is already partitioned by date or time, as is the case here, you can also specify the **Partition timestamp**. This allows more efficient processing of dates and enables time series APIs that you can apply during training.

+Create a dataset monitor to detect and alert to data drift on a new dataset. Use either the [Python SDK](#sdk-monitor) or [Azure Machine Learning studio](#studio-monitor).

+As described later, a dataset monitor runs at a set frequency (daily, weekly, monthly) intervals. It analyzes new data available in the target dataset since its last run. In some cases, such analysis of the most recent data may not suffice:

+- The new data from the upstream source was delayed due to a broken data pipeline, and this new data wasn't available when the dataset monitor ran.

+- A time series dataset had only historical data, and you want to analyze drift patterns in the dataset over time. For example: compare traffic flowing to a website, in both winter and summer seasons, to identify seasonal patterns.

+- You're new to Dataset Monitors. You want to evaluate how the feature works with your existing data before you set it up to monitor future days. In such scenarios, you can submit an on-demand run, with a specific target dataset set date range, to compare with the baseline dataset.

+The **backfill** function runs a backfill job, for a specified start and end date range. A backfill job fills in expected missing data points in a data set, as a way to ensure data accuracy and completeness.

+See the [Python SDK reference documentation on data drift](/python/api/azureml-datadrift/azureml.datadrift) for full details.

+monitor = DataDriftDetector.create_from_datasets(ws, 'drift-monitor', baseline, target,

+ compute_target='cpu-cluster',

+ frequency='Week',

+ feature_list=None,

+ drift_threshold=.6,

+1. Select the **Data** tab.

+1. Select the **+Create monitor** button, and select **Next** to continue through the wizard.

+* **Select target dataset**. The target dataset is a tabular dataset with timestamp column specified which to analyze for data drift. The target dataset must have features in common with the baseline dataset, and should be a `timeseries` dataset, which new data is appended to. Historical data in the target dataset can be analyzed, or new data can be monitored.

+* **Select baseline dataset.** Select the tabular dataset to be used as the baseline for comparison of the target dataset over time. The baseline dataset must have features in common with the target dataset. Select a time range to use a slice of the target dataset, or specify a separate dataset to use as the baseline.

+* **Monitor settings**. These settings are for the scheduled dataset monitor pipeline to create.

+ | Setting | Description | Tips | Mutable |

+ | Features | List of features that to analyze for data drift over time. | Set to a model's output feature(s) to measure concept drift. Don't include features that naturally drift over time (month, year, index, etc.). You can backfill and existing data drift monitor after adjusting the list of features. | Yes |

+ | Compute target | Azure Machine Learning compute target to run the dataset monitor jobs. | | Yes |

+ | Enable | Enable or disable the schedule on the dataset monitor pipeline | Disable the schedule to analyze historical data with the backfill setting. It can be enabled after the dataset monitor is created. | Yes |

+ | Frequency | The frequency that to use, to schedule the pipeline job and analyze historical data if running a backfill. Options include daily, weekly, or monthly. | Each job compares data in the target dataset according to the frequency: <li>Daily: Compare most recent complete day in target dataset with baseline <li>Weekly: Compare most recent complete week (Monday - Sunday) in target dataset with baseline <li>Monthly: Compare most recent complete month in target dataset with baseline | No |

+ | Latency | Time, in hours, it takes for data to arrive in the dataset. For instance, if it takes three days for data to arrive in the SQL DB the dataset encapsulates, set the latency to 72. | Can't be changed after the creation of the dataset monitor | No |

+ | Email addresses | Email addresses for alerting based on breach of the data drift percentage threshold. | Emails are sent through Azure Monitor. | Yes |

+After completion of the wizard, the resulting dataset monitor will appear in the list. Select it to go to that monitor's details page.

+This section shows you the results of monitoring a dataset, found in the **Datasets** / **Dataset monitors** page in Azure studio. You can update the settings, and analyze existing data for a specific time period on this page.

+| Metric | Description |

+| | -- |

+| Data drift magnitude | A percentage of drift between the baseline and target dataset over time. This percentage ranges from 0 to 100, 0 indicates identical datasets and 100 indicates the Azure Machine Learning data drift model can completely tell the two datasets apart. Noise in the precise percentage measured is expected due to machine learning techniques being used to generate this magnitude. |

+| Top drifting features | Shows the features from the dataset that have drifted the most and are therefore contributing the most to the Drift Magnitude metric. Due to covariate shift, the underlying distribution of a feature doesn't necessarily need to change to have relatively high feature importance. |

+| Threshold | Data Drift magnitude beyond the set threshold triggers alerts. Configure the threshold value in the monitor settings. |

+See how the dataset differs from the target dataset in the specified time period. The closer to 100%, the more the two datasets differ.

+This section contains feature-level insights into the change in the selected feature's distribution, and other statistics, over time.

+The target dataset is also profiled over time. The statistical distance between the baseline distribution of each feature is compared with the target dataset's over time. Conceptually, this resembles the data drift magnitude. However this statistical distance is for an individual feature rather than all features. Min, max, and mean are also available.

+In the Azure Machine Learning studio, select a bar in the graph to see the feature-level details for that date. By default, you see the baseline dataset's distribution and the most recent job's distribution of the same feature.

+Finally, scroll down to view details for each individual feature. Use the dropdowns above the chart to select the feature, and additionally select the metric you want to view.

+ | Metric | Description |

+ | | -- |

+ | Metric | Description |

+ | | -- |

+ | Euclidian distance     |  Computed for categorical columns. Euclidean distance is computed on two vectors, generated from empirical distribution of the same categorical column from two datasets. 0 indicates no difference in the empirical distributions.  The more it deviates from 0, the more this column has drifted. Trends can be observed from a time series plot of this metric and can be helpful in uncovering a drifting feature.  |

+On this chart, select a single date to compare the feature distribution between the target and this date for the displayed feature. For numeric features, this shows two probability distributions. If the feature is numeric, a bar chart is shown.

+Metrics can be queried in the [Azure Application Insights](../../azure-monitor/app/app-insights-overview.md) resource associated with your machine learning workspace. You have access to all features of Application Insights including set up for custom alert rules and action groups to trigger an action such as, an Email/SMS/Push/Voice or Azure Function. Refer to the complete Application Insights documentation for details.

+To get started, navigate to the [Azure portal](https://portal.azure.com) and select your workspace's **Overview** page. The associated Application Insights resource is on the far right:

+* The time range when analyzing historical data is limited to 31 intervals of the monitor's frequency setting.

+* Dataset monitors only work on datasets that contain 50 rows or more.

+* Columns, or features, in the dataset are classified as categorical or numeric based on the conditions in the following table. If the feature doesn't meet these conditions - for instance, a column of type string with >100 unique values - the feature is dropped from our data drift algorithm, but is still profiled.

+ | Feature type | Data type | Condition | Limitations |

+ | Categorical | string | The number of unique values in the feature is less than 100 and less than 5% of the number of rows. | Null is treated as its own category. |

+ | Numerical | int, float | The values in the feature are of a numerical data type, and don't meet the condition for a categorical feature. | Feature dropped if >15% of values are null. |

+* When you have created a data drift monitor but can't see data on the **Dataset monitors** page in Azure Machine Learning studio, try the following.

+ 1. Check if you have selected the right date range at the top of the page.

+ 1. On the **Dataset Monitors** tab, select the experiment link to check job status. This link is on the far right of the table.

+ 1. If the job completed successfully, check the driver logs to see how many metrics have been generated or if there's any warning messages. Find driver logs in the **Output + logs** tab after you select an experiment.

+* If the SDK `backfill()` function doesn't generate the expected output, it may be due to an authentication issue. When you create the compute to pass into this function, don't use `Run.get_context().experiment.workspace.compute_targets`. Instead, use [ServicePrincipalAuthentication](/python/api/azureml-core/azureml.core.authentication.serviceprincipalauthentication) such as the following to create the compute that you pass into that `backfill()` function:

+* From the Model Data Collector, it can take up to 10 minutes for data to arrive in your blob storage account. However, it usually takes less time. In a script or Notebook, wait 10 minutes to ensure that the cells below successfully run.

+ Title: Network Insights topology (preview)

+# Topology (preview)

+Topology provides a visualization of the entire network for understanding network configuration. It provides an interactive interface to view resources and their relationships in Azure across multiple subscriptions, resource groups and locations. You can also drill down to a resource view for resources to view their component level visualization.

+The following are the resource types supported by topology:

+- Azure Bastion hosts

+- Azure Front Door profiles

+- ExpressRoute circuits

+- Network interfaces

+- Network security groups

+- Private endpoints

+- Private Link services

+- Public IP addresses

+- Virtual machines

+- Virtual network gateways

+- Virtual networks

+> **_NOTE:_** Bulk import/export is only available for the 'standard' pricing tier

+> Use of the `EnableTestSend` property is heavily throttled. Use this option only in a development/test environment and with a limited set of registrations. Debug notifications are sent to only 10 devices. There's also a limit on processing debug sends, at 10 per minute. Debug notifications are also excluded the the Azure Notification Hubs SLA.

+When the push is made to a handle that has been expired by the PNS, Azure Notification Hubs automatically cleans the associated installation/registration record based on the response received from the PNS server. To clean expired records from a secondary notification hub, add custom logic that processes feedback from each send. Then, expire installation/registration in the secondary notification hub.

+Each template name maps to a template body and an optional set of tags. Moreover, each platform can have additional template properties. For Windows Store (using WNS), an additional set of headers can be part of the template. In the case of APNs, you can set an expiry property to either a constant or to a template expression. For a complete listing of the installation properties see, [Create or Overwrite an Installation with REST](/rest/api/notificationhubs/create-overwrite-installation) topic.

+| **Telemetry Point** | **Source Device / Point** | **Possible Values** | **Definition** |

+| : | : | : | :- |

+| version | Manually set internally | | Release version of the telemetry |

+| contactID | Contact resource | | Identification number of the contact |

+| contactPlatformIdentifier | Contact resource | | |

+| gpsTime | Coversion of utcTime | | Time in GPS time that the customer telemetry message was generated. |

+| utcTime | Current time | | Time in UTC time that the customer telemetry message was generated. |

+| azimuthDecimalDegrees | ACU: AntennaAzimuth | | Antenna's azimuth in decimal degrees. |

+| elevationDecimalDegrees | ACU: AntennaElevation | | Antenna's elevation in decimal degrees. |

+| contactTleLine1 | ACU: Satellite[0].Model.Value | ΓÇó String: TLE <br> ΓÇó "Empty TLE Line 1" if metric is null | First line of the TLE used for the contact. |

+| contactTLeLine2 | ACU: Satellite[0].Model.Value | ΓÇó String: TLE <br> ΓÇó "Empty TLE Line 2" if metric is null | Second line of the TLE used for the contact. |

+| antennaType | Respective 1P/3P telemetry builders set this value | MICROSOFT, KSAT, VIASAT | Antenna network used for the contact. |

+| direction | Contact profile link | Uplink, Downlink | Direction of the link used for the contact. |

+| polarization | Contact profile link | RHCP, LHCP, DualRhcpLhcp, LinearVertical, LinearHorizontal | Polarization of the link used for the contact. |

+| uplinkEnabled | ACU: SBandCurrent or UHFTotalCurrent | ΓÇó NULL (Invalid CenterFrequencyMhz or Downlink direction) <br> ΓÇó False (Bands other than S and UHF or Amp Current < Threshold) <br> ΓÇó True (S/UHF-band, Uplink, Amp Current > Threshold) | Idicates whether uplink was enabled for the contact. |

+| endpointName | Contact profile link channel | | Name of the endpoint used for the contact. |

+| inputEbN0InDb | Modem: measuredEbN0 | ΓÇó NULL (Modem model other than QRadio or QRx) <br> ΓÇó Double: Input EbN0 | Input energy per bit to noise power spectral density in dB. |

+| inputEsN0InDb | Not used in 1P telemetry | NULL (Not used in 1P telemetry) | Input energy per symbol to noise power spectral density in dB. |

+| inputRfPowerDbm | Digitizer: inputRfPower | ΓÇó NULL (Uplink) <br> ΓÇó 0 (Digitizer driver other than SNNB or SNWB) <br> ΓÇó Double: Input Rf Power | Input RF power in dBm. |

+| modemLockStatus | Modem: carrierLockState | ΓÇó NULL (Modem model other than QRadio or QRx; couldnΓÇÖt parse lock status Enum) <br> ΓÇó Empty string (if metric reading was null) <br> ΓÇó String: Lock status | Confirmation that the modem was locked. |

+| commandsSent | Modem: commandsSent | ΓÇó 0 (if not Uplink and QRadio) <br> ΓÇó Double: # of commands sent | Confirmation that commands were sent during the contact. |

+> [!NOTE]

+> While metrics are stored for 93 days, you can only query (in the Metrics tile) for a maximum of 30 days' worth of data on any single chart. If you see a blank chart or your chart displays only part of metric data, verify that the difference between start and end dates in the time picker doesn't exceed the 30-day interval. After you've selected a 30-day interval, you can pan the chart to view the full retention window.

+| United States |East US |West US |

+| United States |East US 2 |Central US |

+| United States |North Central US |South Central US |

+| United States |West US 2 |West Central US |

+| United States|West US 3 |East US |

+Azure continues to expand globally with Qatar as the first region with no regional pair and achieves high availability by leveraging [availability zones](../reliability/availability-zones-overview.md) and [locally redundant or zone-redundant storage (LRS/ZRS)](../storage/common/storage-redundancy.md#zone-redundant-storage). Regions without a pair will not have [geo-redundant storage (GRS)](../storage/common/storage-redundancy.md#geo-redundant-storage). Such regions follow [data residency](https://azure.microsoft.com/global-infrastructure/data-residency/#overview) guidelines allowing the option to keep data resident within the same region. Customers are responsible for data resiliency based on their Recovery Point Objective or Recovery Time Objective (RTO/RPO) needs and may move, copy, or access their data from any location globally. In the rare event that an entire Azure region is unavailable, customers will need to plan for their Cross Region Disaster Recovery per guidance from [Azure services that support high availability](../reliability/availability-zones-service-support.md#azure-services-with-availability-zone-support) and [Azure Resiliency ΓÇô Business Continuity and Disaster Recovery](https://azure.microsoft.com/mediahandler/files/resourcefiles/resilience-in-azure-whitepaper/resiliency-whitepaper-2022.pdf).

+| [Contributor](built-in-roles.md#contributor) | <ul><li>Create and manage all of types of Azure resources</li><li>Can't grant access to others</li></ul> | Applies to all resource types. |

+description: A skillset defines content extraction, natural language processing, and image analysis steps. A skillset is attached to indexer. It's used to enrich and extract information from source data for use in Azure Cognitive Search.

+| [DotNetVectorDemo](https://github.com/Azure/cognitive-search-vector-pr/blob/main/demo-dotnet/readme.md) | [cognitive-search-vector-pr](https://github.com/Azure/cognitive-search-vector-pr) | Calls Azure OpenAI to generate embeddings and Azure Cognitive Search to create, load, and query an index. |

+| [**azure-search-vector-python-sample.ipynb**](https://github.com/Azure/cognitive-search-vector-pr/blob/main/demo-python/code/azure-search-vector-image-python-sample.ipynb) | Uses the latest beta release of the **azure.search.documents** library in the Azure SDK for Python to generate embeddings, create and load an index, and run several vector queries. For more vector search Python demos, see [cognitive-search-vector-pr/demo-python](https://github.com/Azure/cognitive-search-vector-pr/blob/main/demo-python). |