+ > If you intend to purchase a commitment tier for disconnected container usage, you will need to request separate access and select the **Commitment tier disconnected containers** pricing tier. For more information, see [disconnected containers](./containers/disconnected-containers.md).

+## Purchase a commitment tier pricing plan for disconnected containers

+Commitment plans for disconnected containers have a calendar year commitment period. These are different plans than web and connected container commitment plans. When you purchase a commitment plan, you'll be charged the full price immediately. During the commitment period, you can't change your commitment plan, however you can purchase additional unit(s) at a pro-rated price for the remaining days in the year. You have until midnight (UTC) on the last day of your commitment, to end a commitment plan.

+You can choose a different commitment plan in the **Commitment Tier pricing** settings of your resource.

+## Overage pricing for disconnected containers

+To use a disconnected container beyond the quota initially purchased with your disconnected container commitment plan, you can purchase additional quota by updating your commitment plan at any time.

+To purchase additional quota, go to your resource in Azure portal and adjust the "unit count" of your disconnected container commitment plan using the slider. This will add additional monthly quota and you will be charged a pro-rated price based on the remaining days left in the current billing cycle.

+## Purchase a commitment tier pricing plan for disconnected containers

+Commitment plans for disconnected containers have a calendar year commitment period. These are different plans than web and connected container commitment plans. When you purchase a commitment plan, you'll be charged the full price immediately. During the commitment period, you can't change your commitment plan, however you can purchase additional unit(s) at a pro-rated price for the remaining days in the year. You have until midnight (UTC) on the last day of your commitment, to end a commitment plan.

+You can choose a different commitment plan in the **Commitment Tier pricing** settings of your resource. For more information about commitment tier pricing plans, see [purchase commitment tier pricing](../commitment-tier.md).

+## Overage pricing for disconnected containers

+To use a disconnected container beyond the quota initially purchased with your disconnected container commitment plan, you can purchase additional quota by updating your commitment plan at any time.

+To purchase additional quota, go to your resource in Azure portal and adjust the "unit count" of your disconnected container commitment plan using the slider. This will add additional monthly quota and you will be charged a pro-rated price based on the remaining days left in the current billing cycle.

+* [Change or end a commitment plan](../../../ai-services/containers/disconnected-containers.md#purchase-a-commitment-tier-pricing-plan-for-disconnected-containers)

+[Make sure `kubectl` is installed and pointed to your AKS cluster.][kubectl] If you use the Azure Cloud Shell, `kubectl` is already installed.

+ Title: 'Quickstart: Create an Azure Kubernetes Service (AKS) cluster using the Bicep extensibility Kubernetes provider'

+description: Learn how to quickly create a Kubernetes cluster using the Bicep extensibility Kubernetes provider and deploy an application in Azure Kubernetes Service (AKS).

+# Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using the Bicep extensibility Kubernetes provider (Preview)

+Azure Kubernetes Service (AKS) is a managed Kubernetes service that lets you quickly deploy and manage clusters. In this quickstart, you:

+* Deploy an AKS cluster using the Bicep extensibility Kubernetes provider (preview).

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-cli.md).

+* To set up your environment for Bicep development, see [Install Bicep tools](../../azure-resource-manager/bicep/install.md). After completing the steps, you have [Visual Studio Code](https://code.visualstudio.com/) and the [Bicep extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-bicep). You also have either the latest [Azure CLI](/cli/azure/) version or the latest [Azure PowerShell module](/powershell/azure/new-azureps-module-az).

+* To create an AKS cluster using a Bicep file, you provide an SSH public key. If you need this resource, see the following section. Otherwise, skip to [Review the Bicep file](#review-the-bicep-file).

+* Make sure the identity you use to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity.md).

+* To deploy a Bicep file, you need write access on the resources you deploy and access to all operations on the `Microsoft.Resources/deployments` resource type. For example, to deploy a virtual machine, you need `Microsoft.Compute/virtualMachines/write` and `Microsoft.Resources/deployments/*` permissions. For a list of roles and permissions, see [Azure built-in roles](../../role-based-access-control/built-in-roles.md).

+2. Create an SSH key pair using the [`az sshkey create`][az-sshkey-create] Azure CLI command or the `ssh-keygen` command.

+ ```azurecli-interactive

+ # Create an SSH key pair using Azure CLI

+ az sshkey create --name "mySSHKey" --resource-group "myResourceGroup"

+ # Create an SSH key pair using ssh-keygen

+The Bicep file used to create an AKS cluster is from [Azure Quickstart Templates](https://azure.microsoft.com/resources/templates/aks/). For more AKS samples, see [AKS quickstart templates][aks-quickstart-templates].

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` in the same folder as `main.bicep` and copy in the following manifest:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+2. Open `main.bicep` in Visual Studio Code.

+3. Press <kbd>Ctrl+Shift+P</kbd> to open **Command Palette**.

+4. Search for **bicep**, and then select **Bicep: Import Kubernetes Manifest**.

+ :::image type="content" source="./media/quick-kubernetes-deploy-bicep-extensibility-kubernetes-provider/bicep-extensibility-kubernetes-provider-import-kubernetes-manifest.png" alt-text="Screenshot of Visual Studio Code import Kubernetes Manifest." lightbox="./media/quick-kubernetes-deploy-bicep-extensibility-kubernetes-provider/bicep-extensibility-kubernetes-provider-import-kubernetes-manifest.png":::

+5. Select `aks-store-quickstart.yaml` from the prompt. This process creates an `aks-store-quickstart.bicep` file in the same folder.

+6. Open `main.bicep` and add the following Bicep at the end of the file to reference the newly created `aks-store-quickstart.bicep` module:

+ module kubernetes './aks-store-quickstart.bicep' = {

+7. Save both `main.bicep` and `aks-store-quickstart.bicep`.

+### [Azure CLI](#tab/azure-cli)

+1. Create an Azure resource group using the [`az group create`][az-group-create] command.

+ ```azurecli-interactive

+ ```

+2. Deploy the Bicep file using the [`az deployment group create`][az-deployment-group-create] command.

+ ```azurecli-interactive

+### [Azure PowerShell](#tab/azure-powershell)

+1. Create an Azure resource group using the [`New-AzResourceGroup`][new-azresourcegroup] cmdlet.

+ ```azurepowershell-interactive

+2. Deploy the Bicep file using the [`New-AzResourceGroupDeployment`][new-azresourcegroupdeployment] cmdlet.

+ ```azurepowershell-interactive

+ New-AzResourceGroupDeployment -ResourceGroupName myResourceGroup -TemplateFile ./main.bicep -clusterName=<cluster-name> -dnsPrefix=<dns-prefix> -linuxAdminUsername=<linux-admin-username> -sshRSAPublicKey="<ssh-key>"

+ ```

+Provide the following values in the commands:

+* **Cluster name**: Enter a unique name for the AKS cluster, such as *myAKSCluster*.

+* **DNS prefix**: Enter a unique DNS prefix for your cluster, such as *myakscluster*.

+* **Linux Admin Username**: Enter a username to connect using SSH, such as *azureuser*.

+* **SSH RSA Public Key**: Copy and paste the *public* part of your SSH key pair (by default, the contents of *~/.ssh/id_rsa.pub*).

+It takes a few minutes to create the AKS cluster. Wait for the cluster successfully deploy before you move on to the next step.

+1. Sign in to the [Azure portal](https://portal.azure.com/).

+2. On the Azure portal menu or from the **Home** page, navigate to your AKS cluster.

+3. Under **Kubernetes resources**, select **Services and ingresses**.

+4. Find the **store-front** service and copy the value for **External IP**.

+5. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-bicep-extensibility-kubernetes-provider/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-bicep-extensibility-kubernetes-provider/aks-store-application.png":::

+## Delete the cluster

+If you don't plan on going through the following tutorials, clean up unnecessary resources to avoid Azure charges.

+* Remove the resource group, container service, and all related resources using the [`az group delete`][az-group-delete] command.

+ ```azurecli-interactive

+ az group delete --name myResourceGroup --yes --no-wait

+ ```

+* Remove the resource group, container service, and all related resources using the [`Remove-AzResourceGroup`][remove-azresourcegroup] cmdlet.

+ ```azurepowershell-interactive

+ Remove-AzResourceGroup -Name myResourceGroup

+ ```

+> The AKS cluster was created with a system-assigned managed identity, which is the default identity option used in this quickstart. The platform manages this identity so you don't need to manually remove it.

+To learn more about AKS and walk through a complete code to deployment example, continue to the Kubernetes cluster tutorial.

+> [AKS tutorial][aks-tutorial]

+[az-deployment-group-create]: /cli/azure/group/deployment#az_deployment_group_create

+[new-azresourcegroup]: /powershell/module/az.resources/new-azresourcegroup

+[new-azresourcegroupdeployment]: /powershell/module/az.resources/new-azresourcegroupdeployment

+[az-sshkey-create]: /cli/azure/sshkey#az_sshkey_create

+ Title: 'Quickstart: Create an Azure Kubernetes Service (AKS) cluster using Bicep'

+description: Learn how to quickly create a Kubernetes cluster using a Bicep file and deploy an application in Azure Kubernetes Service (AKS).

+* Deploy an AKS cluster using Bicep.

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-cli.md).

+* [!INCLUDE [About Bicep](../../../includes/resource-manager-quickstart-bicep-introduction.md)]

+* This article requires Azure CLI version 2.0.64 or later. If you're using Azure Cloud Shell, the latest version is already installed.

+* This article requires an existing Azure resource group. If you need to create one, you can use the [`az group create`][az-group-create] command.

+* If you're running PowerShell locally, install the `Az PowerShell` module. If using Azure Cloud Shell, the latest version is already installed.

+* You need the Bicep CLI. For more information, see [Azure PowerShell](../../azure-resource-manager/bicep/install.md#azure-powershell).

+* This article requires an existing Azure resource group. If you need to create one, you can use the [`New-AzAksCluster`][new-az-aks-cluster] cmdlet.

+* To create an AKS cluster using a Bicep file, you provide an SSH public key. If you need this resource, see the following section. Otherwise, skip to [Review the Bicep file](#review-the-bicep-file).

+* Make sure the identity you use to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity.md).

+* To deploy a Bicep file, you need write access on the resources you deploy and access to all operations on the `Microsoft.Resources/deployments` resource type. For example, to deploy a virtual machine, you need `Microsoft.Compute/virtualMachines/write` and `Microsoft.Resources/deployments/*` permissions. For a list of roles and permissions, see [Azure built-in roles](../../role-based-access-control/built-in-roles.md).

+ ```azurecli-interactive

+ ### [Azure CLI](#tab/azure-cli)

+ ```azurecli-interactive

+ ### [Azure PowerShell](#tab/azure-powershell)

+ ```azurepowershell-interactive

+1. Install `kubectl` locally using the [`az aks install-cli`][az-aks-install-cli] command.

+ ```azurecli-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az-aks-get-credentials] command. This command downloads credentials and configures the Kubernetes CLI to use them.

+3. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command. This command returns a list of the cluster nodes.

+ ```azurecli-interactive

+ The following example output shows the single node created in the previous steps. Make sure the node status is *Ready*.

+1. Install `kubectl` locally using the [`Install-AzAksKubectl`][install-azakskubectl] cmdlet.

+ ```azurepowershell-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet. This command downloads credentials and configures the Kubernetes CLI to use them.

+3. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command. This command returns a list of the cluster nodes.

+ The following example output shows the three nodes created in the previous steps. Make sure the node status is *Ready*.

+ ```output

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` and copy in the following manifest:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+2. Deploy the application using the [`kubectl apply`][kubectl-apply] command and specify the name of your YAML manifest.

+ kubectl apply -f aks-store-quickstart.yaml

+ The following example output shows the deployments and

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+1. Check the status of the deployed pods using the [`kubectl get pods`][kubectl-get] command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ ```console

+ kubectl get service store-front --watch

+ ```

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+ ```

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+ ```

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-bicep/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-bicep/aks-store-application.png":::

+## Delete the cluster

+If you don't plan on going through the following tutorials, clean up unnecessary resources to avoid Azure charges.

+* Remove the resource group, container service, and all related resources using the [`az group delete`][az-group-delete] command.

+ ```azurecli-interactive

+ az group delete --name myResourceGroup --yes --no-wait

+ ```

+* Remove the resource group, container service, and all related resources using the [`Remove-AzResourceGroup`][remove-azresourcegroup] cmdlet

+ ```azurepowershell-interactive

+ Remove-AzResourceGroup -Name myResourceGroup

+ ```

+ > [!NOTE]

+ > The AKS cluster was created with a system-assigned managed identity, which is the default identity option used in this quickstart. The platform manages this identity so you don't need to manually remove it.

+[az-sshkey-create]: /cli/azure/sshkey#az_sshkey_create

+description: Learn how to quickly create a Kubernetes cluster, deploy an application, and monitor performance in Azure Kubernetes Service (AKS) using Azure CLI.

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+ ```azurecli-interactive

+ ```azurecli-interactive

+ The following example output resembles successful creation of the resource group:

+ ```output

+* Create an AKS cluster using the [`az aks create`][az-aks-create] command with the `--enable-addons monitoring` and `--enable-msi-auth-for-monitoring` parameters to enable [Azure Monitor Container insights][azure-monitor-containers] with managed identity authentication (preview).

+ az aks create -g myResourceGroup -n myAKSCluster --enable-managed-identity --node-count 1 --enable-addons monitoring --enable-msi-auth-for-monitoring --generate-ssh-keys

+ ```azurecli-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az-aks-get-credentials] command. This command downloads credentials and configures the Kubernetes CLI to use them.

+ The following example output shows the single node created in the previous steps. Make sure the node status is *Ready*.

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` and copy in the following manifest:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+ ```azurecli-interactive

+ kubectl apply -f aks-store-quickstart.yaml

+ The following example output shows the deployments and

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+1. Check the status of the deployed pods using the [`kubectl get pods`][kubectl-get] command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ kubectl get service store-front --watch

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-portal/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-portal/aks-store-application.png":::

+To learn more about AKS and walk through a complete code-to-deployment example, continue to the Kubernetes cluster tutorial.

+[intro-azure-linux]: ../../azure-linux/intro-azure-linux.md

+ Title: 'Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using the Azure portal'

+# Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using Azure portal

+Azure Kubernetes Service (AKS) is a managed Kubernetes service that lets you quickly deploy and manage clusters. In this quickstart, you:

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* If you're unfamiliar with the Azure Cloud Shell, review [Overview of Azure Cloud Shell](../../cloud-shell/overview.md).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-portal.md).

+* The identity you use to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity.md).

+ :::image type="content" source="media/quick-kubernetes-deploy-portal/create-cluster-basics.png" alt-text="Screenshot of Create AKS cluster - provide basic information." lightbox="media/quick-kubernetes-deploy-portal/create-cluster-basics.png":::

+ > :::image type="content" source="media/quick-kubernetes-deploy-portal/cluster-preset-options.png" alt-text="Screenshot of Create AKS cluster - portal preset options." lightbox="media/quick-kubernetes-deploy-portal/cluster-preset-options.png":::

+1. Select **Next: Node pools** when complete.

+ - The default value for **Resource identity** is **System-assigned managed identity**. Managed identities provide an identity for applications to use when connecting to resources that support Microsoft Entra authentication. For more details about managed identities, see [What are managed identities for Azure resources?](../../active-directory/managed-identities-azure-resources/overview.md).

+ :::image type="content" source="media/quick-kubernetes-deploy-portal/aks-cloud-shell.png" alt-text="Screenshot of Open the Azure Cloud Shell in the portal option." lightbox="media/quick-kubernetes-deploy-portal/aks-cloud-shell.png":::

+ >

+1. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az-aks-get-credentials] command. This command downloads credentials and configures the Kubernetes CLI to use them.

+2. Verify the connection to your cluster using `kubectl get` to return a list of the cluster nodes.

+ ```azurecli-interactive

+ kubectl get nodes

+ ```

+ The following example output shows the single node created in the previous steps. Make sure the node status is *Ready*.

+ ```output

+ NAME STATUS ROLES AGE VERSION

+ aks-nodepool1-31718369-0 Ready agent 6m44s v1.15.10

+ ```

+1. Configure `kubectl` to connect to your Kubernetes cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet. This command downloads credentials and configures the Kubernetes CLI to use them.

+2. Verify the connection to your cluster using `kubectl get` to return a list of the cluster nodes.

+ ```azurepowershell-interactive

+ The following example output shows the single node created in the previous steps. Make sure the node status is *Ready*.

+ aks-nodepool1-31718369-0 Ready agent 6m44s v1.15.10

+## Deploy the application

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A Kubernetes manifest file defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. In the Cloud Shell, open an editor and create a file named `aks-store-quickstart.yaml`.

+2. Paste the following manifest into the editor:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+3. Deploy the application using the `kubectl apply` command and specify the name of your YAML manifest:

+ The following example output shows the deployments and

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+1. Check the status of the deployed pods using the [`kubectl get pods`][kubectl-get] command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ ```azurecli-interactive

+ kubectl get service store-front --watch

+ ```

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+ ```

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+ ```

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-portal/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-portal/aks-store-application.png":::

+## Delete the cluster

+If you don't plan on going through the following tutorials, clean up unnecessary resources to avoid Azure charges.

+1. In the Azure portal, navigate to your AKS cluster resource group.

+2. Select **Delete resource group**.

+3. Enter the name of the resource group to delete, and then select **Delete** > **Delete**.

+ > [!NOTE]

+ > The AKS cluster was created with a system-assigned managed identity. This identity is managed by the platform and doesn't require removal.

+In this quickstart, you deployed a Kubernetes cluster and deployed a simple multi-container application to it.

+To learn more about AKS and walk through a complete code-to-deployment example, continue to the Kubernetes cluster tutorial.

+[intro-azure-linux]: ../../azure-linux/intro-azure-linux.md

+ Title: 'Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using Azure PowerShell'

+# Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using Azure PowerShell

+Azure Kubernetes Service (AKS) is a managed Kubernetes service that lets you quickly deploy and manage clusters. In this quickstart, you:

+* Deploy an AKS cluster using Azure PowerShell.

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-powershell.md).

+* If you're running PowerShell locally, install the `Az PowerShell` module and connect to your Azure account using the [`Connect-AzAccount`](/powershell/module/az.accounts/Connect-AzAccount) cmdlet. For more information about installing the Az PowerShell module, see [Install Azure PowerShell][install-azure-powershell].

+* The identity you use to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity.md).

+* If you have multiple Azure subscriptions, select the appropriate subscription ID in which the resources should be billed using the

+[`Set-AzContext`](/powershell/module/az.accounts/set-azcontext) cmdlet.

+> [!NOTE]

+> If you plan to run the commands locally instead of in Azure Cloud Shell, make sure you run the commands with administrative privileges.

+An [Azure resource group][azure-resource-group] is a logical group in which Azure resources are deployed and managed. When you create a resource group, you're prompted to specify a location. This location is the storage location of your resource group metadata and where your resources run in Azure if you don't specify another region during resource creation.

+The following example creates a resource group named *myResourceGroup* in the *eastus* location.

+* Create a resource group using the [`New-AzResourceGroup`][new-azresourcegroup] cmdlet.

+ ```azurepowershell-interactive

+ New-AzResourceGroup -Name myResourceGroup -Location eastus

+ ```

+ The following example output resembles successful creation of the resource group:

+ ```output

+ ResourceGroupName : myResourceGroup

+ Location : eastus

+ ProvisioningState : Succeeded

+ Tags :

+ ResourceId : /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup

+ ```

+The following example creates a cluster named *myAKSCluster* with one node and enables a system-assigned managed identity.

+* Create an AKS cluster using the [`New-AzAksCluster`][new-azakscluster] cmdlet with the `-WorkspaceResourceId` parameter to enable [Azure Monitor container insights][azure-monitor-containers].

+ After a few minutes, the command completes and returns information about the cluster.

+ > [!NOTE]

+ > When you create an AKS cluster, a second resource group is automatically created to store the AKS resources. For more information, see [Why are two resource groups created with AKS?](../faq.md#why-are-two-resource-groups-created-with-aks)

+1. Install `kubectl` locally using the `Install-AzAksCliTool` cmdlet.

+ ```azurepowershell-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet. This command downloads credentials and configures the Kubernetes CLI to use them.

+3. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command. This command returns a list of the cluster nodes.

+ The following example output shows the single node created in the previous steps. Make sure the node status is *Ready*.

+ ```output

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` and copy in the following manifest:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+1. Deploy the application using the [`kubectl apply`][kubectl-apply] command and specify the name of your YAML manifest.

+ ```console

+ kubectl apply -f aks-store-quickstart.yaml

+ The following example output shows the deployments and

+ ```output

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+1. Check the status of the deployed pods using the [`kubectl get pods`][kubectl-get] command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ ```azurecli-interactive

+ kubectl get service store-front --watch

+ ```

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+ ```

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+ ```

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-powershell/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-powershell/aks-store-application.png":::

+If you don't plan on going through the following tutorials, clean up unnecessary resources to avoid Azure charges.

+* Remove the resource group, container service, and all related resources using the [`Remove-AzResourceGroup`][remove-azresourcegroup] cmdlet

+ ```azurepowershell-interactive

+ Remove-AzResourceGroup -Name myResourceGroup

+ ```

+ > [!NOTE]

+ > The AKS cluster was created with system-assigned managed identity (default identity option used in this quickstart), the identity is managed by the platform and doesn't require removal.

+To learn more about AKS and walk through a complete code to deployment example, continue to the Kubernetes cluster tutorial.

+[azure-resource-group]: ../../azure-resource-manager/management/overview.md

+ Title: 'Quickstart: Create an Azure Kubernetes Service (AKS) cluster using an ARM template'

+description: Learn how to quickly create a Kubernetes cluster using an Azure Resource Manager template and deploy an application in Azure Kubernetes Service (AKS).

+Azure Kubernetes Service (AKS) is a managed Kubernetes service that lets you quickly deploy and manage clusters. In this quickstart, you:

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-cli.md).

+* [!INCLUDE [About Azure Resource Manager](../../../includes/resource-manager-quickstart-introduction.md)]

+* If your environment meets the prerequisites and you're familiar with ARM templates, select **Deploy to Azure** to open the template in the Azure portal.

+ [](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fquickstarts%2Fmicrosoft.kubernetes%2Faks%2Fazuredeploy.json)

+* This article requires Azure CLI version 2.0.64 or later. If using Azure Cloud Shell, the latest version is already installed.

+* If you're running PowerShell locally, install the `Az PowerShell` module. If using Azure Cloud Shell, the latest version is already installed.

+* To create an AKS cluster using an ARM template, you provide an SSH public key. If you need this resource, see the following section. Otherwise, skip [Review the template](#review-the-template).

+* Make sure the identity you use to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity.md).

+* To deploy an ARM template, you need write access on the resources you're deploying and access to all operations on the `Microsoft.Resources/deployments` resource type. For example, to deploy a virtual machine, you need `Microsoft.Compute/virtualMachines/write` and `Microsoft.Resources/deployments/*` permissions. For a list of roles and permissions, see [Azure built-in roles](../../role-based-access-control/built-in-roles.md).

+2. Create an SSH key pair using the [`az sshkey create`][az-sshkey-create] Azure CLI command or the `ssh-keygen` command.

+ ```azurecli-interactive

+ # Create an SSH key pair using Azure CLI

+ az sshkey create --name "mySSHKey" --resource-group "myResourceGroup"

+ # Create an SSH key pair using ssh-keygen

+The resource defined in the ARM template:

+1. Select **Deploy to Azure** to sign in and open a template.

+2. On the **Basics** page, leave the default values for the *OS Disk Size GB*, *Agent Count*, *Agent VM Size*, *OS Type*, and *Kubernetes Version*, and configure the following template parameters:

+ * **Resource group**: Select **Create new**. Enter a unique name for the resource group, such as *myResourceGroup*, then select **OK**.

+ * **SSH public key source**: Select **Use existing public key**.

+ * **Key pair name**: Copy and paste the *public* part of your SSH key pair (by default, the contents of *~/.ssh/id_rsa.pub*).

+3. Select **Review + Create** > **Create**.

+1. Install `kubectl` locally using the [`az aks install-cli`][az-aks-install-cli] command.

+ ```azurecli-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az-aks-get-credentials] command. This command downloads credentials and configures the Kubernetes CLI to use them.

+3. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command. This command returns a list of the cluster nodes.

+ ```azurecli-interactive

+ The following example output shows the three nodes created in the previous steps. Make sure the node status is *Ready*.

+1. Install `kubectl` locally using the [`Install-AzAksKubectl`][install-azakskubectl] cmdlet.

+ ```azurepowershell-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet. This command downloads credentials and configures the Kubernetes CLI to use them.

+3. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command. This command returns a list of the cluster nodes.

+ The following example output shows the three nodes created in the previous steps. Make sure the node status is *Ready*.

+ ```output

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` and copy in the following manifest:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+2. Deploy the application using the [`kubectl apply`][kubectl-apply] command and specify the name of your YAML manifest.

+ kubectl apply -f aks-store-quickstart.yaml

+ The following example output shows the deployments and

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+1. Check the status of the deployed pods using the [`kubectl get pods`][kubectl-get] command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ ```console

+ kubectl get service store-front --watch

+ ```

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+ ```

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+ ```

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-rm-template/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-rm-template/aks-store-application.png":::

+## Delete the cluster

+If you don't plan on going through the following tutorials, clean up unnecessary resources to avoid Azure charges.

+* Remove the resource group, container service, and all related resources using the [`az group delete`][az-group-delete] command.

+ ```azurecli-interactive

+ az group delete --name myResourceGroup --yes --no-wait

+ ```

+* Remove the resource group, container service, and all related resources using the [`Remove-AzResourceGroup`][remove-azresourcegroup] cmdlet

+ ```azurepowershell-interactive

+ Remove-AzResourceGroup -Name myResourceGroup

+ ```

+ > [!NOTE]

+ > The AKS cluster was created with a system-assigned managed identity, which is the default identity option used in this quickstart. The platform manages this identity so you don't need to manually remove it.

+To learn more about AKS and walk through a complete code to deployment example, continue to the Kubernetes cluster tutorial.

+[ssh-keys]: ../../virtual-machines/linux/create-ssh-keys-detailed.md

+ Title: 'Quickstart: Create an Azure Kubernetes Service (AKS) cluster using Terraform'

+description: Learn how to quickly create a Kubernetes cluster using Terraform and deploy an application in Azure Kubernetes Service (AKS).

+# Quickstart: Create an Azure Kubernetes Service (AKS) cluster using Terraform

+* Deploy an AKS cluster using Terraform.

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-cli.md).

+* [Install and configure Terraform](/azure/developer/terraform/quickstart-configure).

+* [Download kubectl](https://kubernetes.io/releases/download/).

+* Create a random value for the Azure resource group name using [random_pet](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/pet).

+* Create an Azure resource group using [azurerm_resource_group](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group).

+* Access the configuration of the AzureRM provider to get the Azure Object ID using [azurerm_client_config](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/client_config).

+* Create a Kubernetes cluster using [azurerm_kubernetes_cluster](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster).

+* Create an AzAPI resource [azapi_resource](https://registry.terraform.io/providers/Azure/azapi/latest/docs/resources/azapi_resource).

+* Create an AzAPI resource to generate an SSH key pair using [azapi_resource_action](https://registry.terraform.io/providers/Azure/azapi/latest/docs/resources/azapi_resource_action).

+1. Create a directory you can use to test the sample Terraform code and make it your current directory.

+1. Get the Azure resource group name using the following command.

+2. Display the name of your new Kubernetes cluster using the [`az aks list`](/cli/azure/aks#az-aks-list) command.

+ ```azurecli-interactive

+3. Get the Kubernetes configuration from the Terraform state and store it in a file that `kubectl` can read using the following command.

+4. Verify the previous command didn't add an ASCII EOT character using the following command.

+ * If you see `<< EOT` at the beginning and `EOT` at the end, remove these characters from the file. Otherwise, you may receive the following error message: `error: error loading config file "./azurek8s": yaml: line 2: mapping values are not allowed in this context`

+5. Set an environment variable so `kubectl` can pick up the correct config using the following command.

+6. Verify the health of the cluster using the `kubectl get nodes` command.

+* When you created the AKS cluster, monitoring was enabled to capture health metrics for both the cluster nodes and pods. These health metrics are available in the Azure portal. For more information on container health monitoring, see [Monitor Azure Kubernetes Service health](/azure/azure-monitor/insights/container-insights-overview).

+* Several key values classified as output when you applied the Terraform execution plan. For example, the host address, AKS cluster user name, and AKS cluster password are output.

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` and copy in the following manifest:

+ ```yaml

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: rabbitmq

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: rabbitmq

+ template:

+ metadata:

+ labels:

+ app: rabbitmq

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ env:

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ resources:

+ requests:

+ cpu: 10m

+ memory: 128Mi

+ limits:

+ cpu: 250m

+ memory: 256Mi

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ ports:

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: order-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3000

+ targetPort: 3000

+ selector:

+ app: order-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: product-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: product-service

+ template:

+ metadata:

+ labels:

+ app: product-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ resources:

+ requests:

+ cpu: 1m

+ memory: 1Mi

+ limits:

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ ports:

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: store-front

+ spec:

+ ports:

+ - port: 80

+ targetPort: 8080

+ selector:

+ app: store-front

+ type: LoadBalancer

+ ```

+ For a breakdown of YAML manifest files, see [Deployments and YAML manifests](../concepts-clusters-workloads.md#deployments-and-yaml-manifests).

+2. Deploy the application using the `kubectl apply` command and specify the name of your YAML manifest.

+ ```console

+ kubectl apply -f aks-store-quickstart.yaml

+ ```

+ The following example output shows the deployments and

+ ```output

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+ ```

+### Test the application

+When the application runs, a Kubernetes service exposes the application front end to the internet. This process can take a few minutes to complete.

+1. Check the status of the deployed pods using the `kubectl get pods` command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the `kubectl get service` command with the `--watch` argument.

+ ```azurecli-interactive

+ kubectl get service store-front --watch

+ ```

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-terraform/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-terraform/aks-store-application.png":::

+1. Get the service principal ID using the following command.

+ ```azurecli-interactive

+1. Delete the service principal using the [`az ad sp delete`](/cli/azure/ad/sp#az-ad-sp-delete) command.

+ ```azurecli-interactive

+[Troubleshoot common problems when using Terraform on Azure](/azure/developer/terraform/troubleshoot).

+> [Learn more about using AKS.](/azure/aks)

+<!-- LINKS - internal -->

+[kubernetes-concepts]: ../concepts-clusters-workloads.md

+[kubernetes-deployment]: ../concepts-clusters-workloads.md#deployments-and-yaml-manifests

+[intro-azure-linux]: ../../azure-linux/intro-azure-linux.md

+ Title: Kubernetes on Azure tutorial - Deploy an application to Azure Kubernetes Service (AKS)

+description: In this Azure Kubernetes Service (AKS) tutorial, you deploy a multi-container application to your cluster using images stored in Azure Container Registry.

+# Tutorial - Deploy an application to Azure Kubernetes Service (AKS)

+Kubernetes provides a distributed platform for containerized applications. You build and deploy your own applications and services into a Kubernetes cluster and let the cluster manage the availability and connectivity.

+In this tutorial, part four of seven, you deploy a sample application into a Kubernetes cluster. You learn how to:

+>

+> With AKS, you can use the following approaches for configuration management:

+>

+> * **GitOps**: Enables declarations of your cluster's state to automatically apply to the cluster. To learn how to use GitOps to deploy an application with an AKS cluster, see the [prerequisites for Azure Kubernetes Service clusters][gitops-flux-tutorial-aks] in the [GitOps with Flux v2][gitops-flux-tutorial] tutorial.

+>

+> * **DevOps**: Enables you to build, test, and deploy with continuous integration (CI) and continuous delivery (CD). To see examples of how to use DevOps to deploy an application with an AKS cluster, see [Build and deploy to AKS with Azure Pipelines](./devops-pipeline.md) or [GitHub Actions for deploying to Kubernetes](./kubernetes-action.md).

+In previous tutorials, you packaged an application into a container image, uploaded the image to Azure Container Registry, and created a Kubernetes cluster. To complete this tutorial, you need the pre-created `aks-store-quickstart.yaml` Kubernetes manifest file. This file download was included with the application source code in a previous tutorial. Make sure you cloned the repo and changed directories into the cloned repo. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

+This tutorial requires Azure CLI version 2.34.1 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+This tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+In these tutorials, your Azure Container Registry (ACR) instance stores the container images for the sample application. To deploy the application, you must update the image names in the Kubernetes manifest file to include your ACR login server name.

+1. Get your login server address using the [`az acr list`][az-acr-list] command and query for your login server.

+ ```azurecli-interactive

+ az acr list --resource-group myResourceGroup --query "[].{acrLoginServer:loginServer}" --output table

+ ```

+2. Make sure you're in the cloned *aks-store-demo* directory, and then open the manifest file with a text editor, such as `vi`:

+ ```azurecli-interactive

+ vi aks-store-quickstart.yaml

+ ```

+3. Update the `image` property for the containers by replacing *ghcr.io/azure-samples* with your ACR login server name.

+ ```yaml

+ containers:

+ ...

+ - name: order-service

+ image: <acrName>.azurecr.io/aks-store-demo/order-service:latest

+ ...

+ - name: product-service

+ image: <acrName>.azurecr.io/aks-store-demo/product-service:latest

+ ...

+ - name: store-front

+ image: <acrName>.azurecr.io/aks-store-demo/store-front:latest

+ ...

+ ```

+4. Save and close the file. In `vi`, use `:wq`.

+### [Azure PowerShell](#tab/azure-powershell)

+1. Get your login server address using the [`Get-AzContainerRegistry`][get-azcontainerregistry] cmdlet and query for your login server. Make sure you replace `<acrName>` with the name of your ACR instance.

+ ```azurepowershell-interactive

+ (Get-AzContainerRegistry -ResourceGroupName myResourceGroup -Name <acrName>).LoginServer

+ ```

+2. Make sure you're in the cloned *aks-store-demo* directory, and then open the manifest file with a text editor, such as `vi`:

+ ```azurepowershell-interactive

+ vi aks-store-quickstart.yaml

+ ```

+3. Update the `image` property for the containers by replacing *ghcr.io/azure-samples* with your ACR login server name.

+ ```yaml

+ containers:

+ ...

+ - name: order-service

+ image: <acrName>.azurecr.io/aks-store-demo/order-service:latest

+ ...

+ - name: product-service

+ image: <acrName>.azurecr.io/aks-store-demo/product-service:latest

+ ...

+ - name: store-front

+ image: <acrName>.azurecr.io/aks-store-demo/store-front:latest

+ ...

+ ```

+4. Save and close the file. In `vi`, use `:wq`.

+## Deploy the application

+* Deploy the application using the [`kubectl apply`][kubectl-apply] command, which parses the manifest file and creates the defined Kubernetes objects.

+ ```console

+ kubectl apply -f aks-store-quickstart.yaml

+ ```

+ The following example output shows the resources successfully created in the AKS cluster:

+ ```output

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+ ```

+1. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ ```console

+ kubectl get service store-front --watch

+ ```

+ Initially, the `EXTERNAL-IP` for the *store-front* service shows as *pending*.

+ ```output

+ store-front LoadBalancer 10.0.34.242 <pending> 80:30676/TCP 5s

+ ```

+2. When the `EXTERNAL-IP` address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ store-front LoadBalancer 10.0.34.242 52.179.23.131 80:30676/TCP 67s

+ ```

+3. View the application in action by opening a web browser to the external IP address of your service.

+In this tutorial, you deployed a sample Azure application to a Kubernetes cluster in AKS. You learned how to:

+In the next tutorial, you learn how to use PaaS services for stateful workloads in Kubernetes.

+> Use PaaS services for stateful workloads in AKS

+[aks-tutorial-paas]: ./tutorial-kubernetes-paas-services.md

+ Title: Kubernetes on Azure tutorial - Deploy an Azure Kubernetes Service (AKS) cluster

+description: In this Azure Kubernetes Service (AKS) tutorial, you create an AKS cluster and use kubectl to connect to the Kubernetes main node.

+# Tutorial - Deploy an Azure Kubernetes Service (AKS) cluster

+Kubernetes provides a distributed platform for containerized applications. With Azure Kubernetes Service (AKS), you can quickly create a production ready Kubernetes cluster.

+In this tutorial, part three of seven, you deploy a Kubernetes cluster in AKS. You learn how to:

+> * Deploy an AKS cluster that can authenticate to an Azure Container Registry (ACR).

+In previous tutorials, you created a container image and uploaded it to an ACR instance. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

+This tutorial requires Azure CLI version 2.0.53 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+### [Azure PowerShell](#tab/azure-powershell)

+This tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+## Create an AKS cluster

+AKS clusters can use [Kubernetes role-based access control (Kubernetes RBAC)][k8s-rbac], which allows you to define access to resources based on roles assigned to users. Permissions are combined when users are assigned multiple roles. Permissions can be scoped to either a single namespace or across the whole cluster. For more information, see [Control access to cluster resources using Kubernetes RBAC and Azure Active Directory identities in AKS][aks-k8s-rbac].

+For information about AKS resource limits and region availability, see [Quotas, virtual machine size restrictions, and region availability in AKS][quotas-skus-regions].

+> To ensure your cluster operates reliably, you should run at least two nodes.

+### [Azure CLI](#tab/azure-cli)

+To allow an AKS cluster to interact with other Azure resources, the Azure platform automatically creates a cluster identity. In this example, the cluster identity is [granted the right to pull images][container-registry-integration] from the ACR instance you created in the previous tutorial. To execute the command successfully, you need to have an **Owner** or **Azure account administrator** role in your Azure subscription.

+* Create an AKS cluster using the [`az aks create`][az aks create] command. The following example creates a cluster named *myAKSCluster* in the resource group named *myResourceGroup*. This resource group was created in the [previous tutorial][aks-tutorial-prepare-acr] in the *eastus* region.

+ ```azurecli-interactive

+ az aks create \

+ --resource-group myResourceGroup \

+ --name myAKSCluster \

+ --node-count 2 \

+ --generate-ssh-keys \

+ --attach-acr <acrName>

+ ```

+ > [!NOTE]

+ > If you already generated SSH keys, you may encounter an error similar to `linuxProfile.ssh.publicKeys.keyData is invalid`. To proceed, retry the command without the `--generate-ssh-keys` parameter.

+### [Azure PowerShell](#tab/azure-powershell)

+To allow an AKS cluster to interact with other Azure resources, the Azure platform automatically creates a cluster identity. In this example, the cluster identity is [granted the right to pull images][container-registry-integration] from the ACR instance you created in the previous tutorial. To execute the command successfully, you need to have an **Owner** or **Azure account administrator** role in your Azure subscription.

+* Create an AKS cluster using the [`New-AzAksCluster`][new-azakscluster] cmdlet. The following example creates a cluster named *myAKSCluster* in the resource group named *myResourceGroup*. This resource group was created in the [previous tutorial][aks-tutorial-prepare-acr] in the *eastus* region.

+ ```azurepowershell-interactive

+ New-AzAksCluster -ResourceGroupName myResourceGroup -Name myAKSCluster -NodeCount 2 -GenerateSshKey -AcrNameToAttach <acrName>

+ ```

+ > [!NOTE]

+ > If you already generated SSH keys, you may encounter an error similar to `linuxProfile.ssh.publicKeys.keyData is invalid`. To proceed, retry the command without the `-GenerateSshKey` parameter.

+You use the Kubernetes CLI, [`kubectl`][kubectl], to connect to your Kubernetes cluster. If you use the Azure Cloud Shell, `kubectl` is already installed. If you're running the commands locally, you can use the Azure CLI or Azure PowerShell to install `kubectl`.

+* Install `kubectl` locally using the [`az aks install-cli`][az aks install-cli] command.

+ ```azurecli-interactive

+ az aks install-cli

+ ```

+* Install `kubectl` locally using the [`Install-AzAksCliTool`][install-azaksclitool] cmdlet.

+ ```azurepowershell-interactive

+ Install-AzAksCliTool

+ ```

+1. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az aks get-credentials] command. The following example gets credentials for the AKS cluster named *myAKSCluster* in *myResourceGroup*.

+ ```azurecli-interactive

+ az aks get-credentials --resource-group myResourceGroup --name myAKSCluster

+ ```

+2. Verify connection to your cluster using the [`kubectl get nodes`][kubectl-get] command, which returns a list of cluster nodes.

+ ```azurecli-interactive

+ kubectl get nodes

+ ```

+ The following example output shows a list of the cluster nodes:

+ ```output

+ NAME STATUS ROLES AGE VERSION

+ aks-nodepool1-19366578-vmss000002 Ready agent 47h v1.25.6

+ aks-nodepool1-19366578-vmss000003 Ready agent 47h v1.25.6

+ ```

+### [Azure PowerShell](#tab/azure-powershell)

+1. Configure `kubectl` to connect to your Kubernetes cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet. The following example gets credentials for the AKS cluster named *myAKSCluster* in *myResourceGroup*.

+ ```azurepowershell-interactive

+ Import-AzAksCredential -ResourceGroupName myResourceGroup -Name myAKSCluster

+ ```

+2. Verify connection to your cluster using the [`kubectl get nodes`][kubectl-get] command, which returns a list of cluster nodes.

+ ```azurepowershell-interactive

+ kubectl get nodes

+ ```

+ The following example output shows a list of the cluster nodes:

+ ```output

+ NAME STATUS ROLES AGE VERSION

+ aks-nodepool1-19366578-vmss000002 Ready agent 47h v1.25.6

+ aks-nodepool1-19366578-vmss000003 Ready agent 47h v1.25.6

+ ```

+In the next tutorial, you learn how to deploy an application to your cluster.

+[install-azaksclitool]: /powershell/module/az.aks/install-azaksclitool

+[aks-k8s-rbac]: azure-ad-rbac.md

+ Title: Kubernetes on Azure tutorial - Use PaaS services with an Azure Kubernetes Service (AKS) cluster

+description: In this Azure Kubernetes Service (AKS) tutorial, you learn how to use the Azure Service Bus service with your AKS cluster.

+#Customer intent: As a developer, I want to learn how to use PaaS services with an Azure Kubernetes Service (AKS) cluster so that I can deploy and manage my applications.

+# Tutorial - Use PaaS services with an Azure Kubernetes Service (AKS) cluster

+With Kubernetes, you can use PaaS services, such as [Azure Service Bus][azure-service-bus], to develop and run your applications.

+In this tutorial, part five of seven, you create an Azure Service Bus namespace and queue to test your application. You learn how to:

+> [!div class="checklist"]

+>

+> * Create an Azure Service Bus namespace and queue.

+> * Update the Kubernetes manifest file to use the Azure Service Bus queue.

+> * Test the updated application by placing an order.

+## Before you begin

+In previous tutorials, you packaged an application into a container image, uploaded the image to Azure Container Registry, created a Kubernetes cluster, and deployed an application. To complete this tutorial, you need the pre-created `aks-store-quickstart.yaml` Kubernetes manifest file. This file download was included with the application source code in a previous tutorial. Make sure you cloned the repo and changed directories into the cloned repo. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

+### [Azure CLI](#tab/azure-cli)

+This tutorial requires Azure CLI version 2.34.1 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+### [Azure PowerShell](#tab/azure-powershell)

+This tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+## Create environment variables

+### [Azure CLI](#tab/azure-cli)

+* Create the following environment variables to use for the commands in this tutorial:

+ ```azurecli-interactive

+ LOC_NAME=eastus

+ RAND=$RANDOM

+ RG_NAME=myResourceGroup

+ AKS_NAME=myAKSCluster

+ SB_NS=sb-store-demo-$RAND

+ ```

+### [Azure PowerShell](#tab/azure-powershell)

+* Create the following environment variables to use for the commands in this tutorial:

+ ```azurepowershell-interactive

+ $LOC_NAME="eastus"

+ $rand=New-Object System.Random

+ $RAND=$rand.Next()

+ $RG_NAME="myResourceGroup"

+ $AKS_NAME="myAKSCluster"

+ $SB_NS="sb-store-demo-$RAND"

+ ```

+## Create Azure Service Bus namespace and queue

+In previous tutorials, you used a RabbitMQ container to store orders submitted by the `order-service`. In this tutorial, you use an Azure Service Bus namespace to provide a scoping container for the Service Bus resources within the application. You also use an Azure Service bus queue to send and receive messages between the application components. For more information on Azure Service Bus, see [Create an Azure Service Bus namespace and queue](../service-bus-messaging/service-bus-quickstart-cli.md).

+### [Azure CLI](#tab/azure-cli)

+1. Create an Azure Service Bus namespace using the [`az servicebus namespace create`][az-servicebus-namespace-create] command.

+ ```azurecli-interactive

+ az servicebus namespace create -n $SB_NS -g $RG_NAME -l $LOC_NAME

+ ```

+2. Create an Azure Service Bus queue using the [`az servicebus queue create`][az-servicebus-queue-create] command.

+ ```azurecli-interactive

+ az servicebus queue create -n orders -g $RG_NAME --namespace-name $SB_NS -g $RG_NAME

+ ```

+3. Create an Azure Service Bus authorization rule using the [`az servicebus queue authorization-rule create`][az-servicebus-queue-authorization-rule-create] command.

+ ```azurecli-interactive

+ az servicebus queue authorization-rule create \

+ --name sender \

+ --namespace-name $SB_NS \

+ --resource-group $RG_NAME \

+ --queue-name orders \

+ --rights Send

+ ```

+4. Get the Azure Service Bus credentials for later use using the [`az servicebus namespace show`][az-servicebus-namespace-show] and [`az servicebus queue authorization-rule keys list`][az-servicebus-queue-authorization-rule-keys-list] commands.

+ ```azurecli-interactive

+ az servicebus namespace show --name $SB_NS --resource-group $RG_NAME --query name -o tsv

+ az servicebus queue authorization-rule keys list --namespace-name $SB_NS --resource-group $RG_NAME --queue-name orders --name sender --query primaryKey -o tsv

+ ```

+### [Azure PowerShell](#tab/azure-powershell)

+1. Create an Azure Service Bus namespace using the [`New-AzServiceBusNamespace`][new-az-service-bus-namespace] cmdlet.

+ ```azurepowershell-interactive

+ New-AzServiceBusNamespace -Name $SB_NS -ResourceGroupName $RG_NAME -Location $LOC_NAME

+ ```

+2. Create an Azure Service Bus queue using the [`New-AzServiceBusQueue`][new-az-service-bus-queue] cmdlet.

+ ```azurepowershell-interactive

+ New-AzServiceBusQueue -Name orders -ResourceGroupName $RG_NAME -NamespaceName $SB_NS

+ ```

+3. Create an Azure Service Bus authorization rule using the [`New-AzServiceBusAuthorizationRule`][new-az-service-bus-authorization-rule] cmdlet.

+ ```azurepowershell-interactive

+ New-AzServiceBusAuthorizationRule `

+ -Name sender `

+ -NamespaceName $SB_NS `

+ -ResourceGroupName $RG_NAME `

+ -QueueName orders `

+ -Rights Send

+ ```

+4. Get the Azure Service Bus credentials for later use using the [`Get-AzServiceBusNamespace`][get-az-service-bus-namespace] and [`Get-AzServiceBusKey`][get-az-service-bus-key] cmdlets.

+ ```azurepowershell-interactive

+ (Get-AzServiceBusNamespace -Name $SB_NS -ResourceGroupName $RG_NAME).Name

+ (Get-AzServiceBusKey -NamespaceName $SB_NS -ResourceGroupName $RG_NAME -Name sender -QueueName orders).PrimaryKey

+ ```

+## Update Kubernetes manifest file

+### [Azure CLI](#tab/azure-cli)

+1. Configure `kubectl` to connect to your cluster using the [`az aks get-credentials`][az-aks-get-credentials] command.

+ ```azurecli-interactive

+ az aks get-credentials --resource-group myResourceGroup --name myAKSCluster

+ ```

+2. Open the `aks-store-quickstart.yaml` file in a text editor.

+3. Remove the existing `rabbitmq` Deployment, ConfigMap, and Service sections and replace the existing `order-service` Deployment section with the following content:

+ ```yaml

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: <REPLACE_WITH_YOUR_ACR_NAME>.azurecr.io/aks-store-demo/order-service:latest

+ ports:

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "<REPLACE_WITH_YOUR_SB_NS_HOSTNAME>" # Example: sb-store-demo-123456.servicebus.windows.net

+ - name: ORDER_QUEUE_PORT

+ value: "5671"

+ - name: ORDER_QUEUE_TRANSPORT

+ value: "tls"

+ - name: ORDER_QUEUE_USERNAME

+ value: "sender"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "<REPLACE_WITH_YOUR_SB_SENDER_PASSWORD>"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ ```

+ > [!NOTE]

+ > Directly adding sensitive information, such as API keys, to your Kubernetes manifest files isn't secure and may accidentally get committed to code repositories. We added it here for simplicity. For production workloads, use [Managed Identity](./use-managed-identity.md) to authenticate with Azure Service Bus or store your secrets in [Azure Key Vault](./csi-secrets-store-driver.md).

+4. Save and close the updated `aks-store-quickstart.yaml` file.

+### [Azure PowerShell](#tab/azure-powershell)

+1. Configure `kubectl` to connect to your cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet.

+ ```azurepowershell-interactive

+ Import-AzAksCredential -ResourceGroupName myResourceGroup -Name myAKSCluster

+ ```

+2. Open the `aks-store-quickstart.yaml` file in a text editor.

+3. Remove the existing `rabbitmq` Deployment, ConfigMap, and Service sections and replace the existing `order-service` Deployment section with the following content:

+ ```YAML

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: <REPLACE_WITH_YOUR_ACR_NAME>.azurecr.io/aks-store-demo/order-service:latest

+ ports:

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "<REPLACE_WITH_YOUR_SB_NS_HOSTNAME>" # Example: sb-store-demo-123456.servicebus.windows.net

+ - name: ORDER_QUEUE_PORT

+ value: "5671"

+ - name: ORDER_QUEUE_TRANSPORT

+ value: "tls"

+ - name: ORDER_QUEUE_USERNAME

+ value: "sender"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "<REPLACE_WITH_YOUR_SB_SENDER_PASSWORD>"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ ```

+ > [!NOTE]

+ > Directly adding sensitive information, such as API keys, to your Kubernetes manifest files isn't secure and may accidentally get committed to code repositories. We added it here for simplicity. For production workloads, use [Managed Identity](./use-managed-identity.md) to authenticate with Azure Service Bus or store your secrets in [Azure Key Vault](./csi-secrets-store-driver.md).

+4. Save and close the updated `aks-store-quickstart.yaml` file.

+## Deploy the updated application

+* Deploy the updated application using the `kubectl apply` command.

+ ```console

+ kubectl apply -f aks-store-quickstart.yaml

+ ```

+ The following example output shows the successfully updated resources:

+ ```output

+ deployment.apps/order-service configured

+ service/order-service unchanged

+ deployment.apps/product-service unchanged

+ service/product-service unchanged

+ deployment.apps/store-front configured

+ service/store-front unchanged

+ ```

+## Test the application

+### Place a sample order

+1. Get the external IP address of the `store-front` service using the `kubectl get service` command.

+ ```console

+ kubectl get service store-front

+ ```

+2. Navigate to the external IP address of the `store-front` service in your browser.

+3. Place an order by choosing a product and selecting **Add to cart**.

+4. Select **Cart** to view your order, and then select **Checkout**.

+### View the order in the Azure Service Bus queue

+1. Navigate to the Azure portal and open the Azure Service Bus namespace you created earlier.

+2. Under **Entities**, select **Queues**, and then select the **orders** queue.

+3. In the **orders** queue, select **Service Bus Explorer**.

+4. Select **Peek from start** to view the order you submitted.

+## Next steps

+In this tutorial, you used Azure Service Bus to update and test the sample application. You learned how to:

+> [!div class="checklist"]

+>

+> * Create an Azure Service Bus namespace and queue.

+> * Update the Kubernetes manifest file to use the Azure Service Bus queue.

+> * Test the updated application by placing an order.

+In the next tutorial, you learn how to scale an application in AKS.

+> [!div class="nextstepaction"]

+> [Scale applications in AKS][aks-tutorial-scale]

+<!-- LINKS - external -->

+<!-- LINKS - internal -->

+[aks-tutorial-prepare-app]: ./tutorial-kubernetes-prepare-app.md

+[azure-cli-install]: /cli/azure/install-azure-cli

+[azure-powershell-install]: /powershell/azure/install-az-ps

+[aks-tutorial-scale]: ./tutorial-kubernetes-scale.md

+[azure-service-bus]: ../service-bus-messaging/service-bus-messaging-overview.md

+[az-servicebus-namespace-create]: /cli/azure/servicebus/namespace#az_servicebus_namespace_create

+[az-servicebus-queue-create]: /cli/azure/servicebus/queue#az_servicebus_queue_create

+[az-servicebus-queue-authorization-rule-create]: /cli/azure/servicebus/queue/authorization-rule#az_servicebus_queue_authorization_rule_create

+[az-servicebus-namespace-show]: /cli/azure/servicebus/namespace#az_servicebus_namespace_show

+[az-servicebus-queue-authorization-rule-keys-list]: /cli/azure/servicebus/queue/authorization-rule/keys#az_servicebus_queue_authorization_rule_keys_list

+[new-az-service-bus-namespace]: /powershell/module/az.servicebus/new-azservicebusnamespace

+[new-az-service-bus-queue]: /powershell/module/az.servicebus/new-azservicebusqueue

+[new-az-service-bus-authorization-rule]: /powershell/module/az.servicebus/new-azservicebusauthorizationrule

+[get-az-service-bus-namespace]: /powershell/module/az.servicebus/get-azservicebusnamespace

+[get-az-service-bus-key]: /powershell/module/az.servicebus/get-azservicebuskey

+[import-azakscredential]: /powershell/module/az.aks/import-azakscredential

+[az-aks-get-credentials]: /cli/azure/aks#az_aks_get_credentials

+ Title: Kubernetes on Azure tutorial - Create an Azure Container Registry and build images

+description: In this Azure Kubernetes Service (AKS) tutorial, you create an Azure Container Registry instance and upload sample application container images.

+# Tutorial - Create an Azure Container Registry (ACR) and build images

+Azure Container Registry (ACR) is a private registry for container images. A private container registry allows you to securely build and deploy your applications and custom code.

+In this tutorial, part two of seven, you deploy an ACR instance and push a container image to it. You learn how to:

+> * Create an ACR instance.

+> * Use [ACR Tasks][acr-tasks] to build and push container images to ACR.

+> * View images in your registry.

+In the [previous tutorial][aks-tutorial-prepare-app], you used Docker to create a container image for a simple Azure Store Front application. If you haven't created the Azure Store Front app image, return to [Tutorial 1 - Prepare an application for AKS][aks-tutorial-prepare-app].

+This tutorial requires Azure CLI version 2.0.53 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+This tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+Before creating an ACR instance, you need a resource group. An Azure resource group is a logical container into which you deploy and manage Azure resources.

+1. Create a resource group using the [`az group create`][az-group-create] command.

+ ```azurecli-interactive

+ az group create --name myResourceGroup --location eastus

+ ```

+2. Create an ACR instance using the [`az acr create`][az-acr-create] command and provide your own unique registry name. The registry name must be unique within Azure and contain 5-50 alphanumeric characters. The rest of this tutorial uses `<acrName>` as a placeholder for the container registry name. The *Basic* SKU is a cost-optimized entry point for development purposes that provides a balance of storage and throughput.

+ ```azurecli-interactive

+ az acr create --resource-group myResourceGroup --name <acrName> --sku Basic

+ ```

+1. Create a resource group using the [`New-AzResourceGroup`][new-azresourcegroup] cmdlet.

+ ```azurepowershell-interactive

+ New-AzResourceGroup -Name myResourceGroup -Location eastus

+ ```

+2. Create an ACR instance using the [`New-AzContainerRegistry`][new-azcontainerregistry] cmdlet and provide your own unique registry name. The registry name must be unique within Azure and contain 5-50 alphanumeric characters. The rest of this tutorial uses `<acrName>` as a placeholder for the container registry name. The *Basic* SKU is a cost-optimized entry point for development purposes that provides a balance of storage and throughput.

+ ```azurepowershell-interactive

+ New-AzContainerRegistry -ResourceGroupName myResourceGroup -Name <acrName> -Location eastus -Sku Basic

+ ```

+## Build and push container images to registry

+* Build and push the images to your ACR using the [`az acr build`][az-acr-build] command.

+ > [!NOTE]

+ > In the following example, we don't build the `rabbitmq` image. This image is available from the Docker Hub public repository and doesn't need to be built or pushed to your ACR instance.

+ ```azurecli-interactive

+ az acr build --registry <acrName> --image aks-store-demo/product-service:latest ./src/product-service/

+ az acr build --registry <acrName> --image aks-store-demo/order-service:latest ./src/order-service/

+ az acr build --registry <acrName> --image aks-store-demo/store-front:latest ./src/store-front/

+ ```

+* View the images in your ACR instance using the [`az acr repository list`][az-acr-repository-list] command.

+ ```azurecli-interactive

+ az acr repository list --name <acrName> --output table

+ ```

+ The following example output lists the available images in your registry:

+ ```output

+ Result

+ -

+ aks-store-demo/product-service

+ aks-store-demo/order-service

+ aks-store-demo/store-front

+ ```

+* View the images in your ACR instance using the [`Get-AzContainerRegistryRepository`][get-azcontainerregistryrepository] cmdlet.

+ ```azurepowershell-interactive

+ Get-AzContainerRegistryRepository -RegistryName <acrName>

+ ```

+ The following example output lists the available images in your registry:

+ ```output

+ aks-store-demo/productservice

+ aks-store-demo/orderservice

+ aks-store-demo/storefront

+ ```

+In this tutorial, you created an ACR and pushed images to it to use in an AKS cluster. You learned how to:

+> * Create an ACR instance.

+> * Use [ACR Tasks][acr-tasks] to build and push container images to ACR.

+> * View images in your registry.

+In the next tutorial, you learn how to deploy a Kubernetes cluster in Azure.

+[get-azcontainerregistryrepository]: /powershell/module/az.containerregistry/get-azcontainerregistryrepository

+[acr-tasks]: ../container-registry/container-registry-tasks-overview.md

+[az-acr-build]: /cli/azure/acr#az_acr_build

+ Title: Kubernetes on Azure tutorial - Prepare an application for Azure Kubernetes Service (AKS)

+# Tutorial - Prepare an application for Azure Kubernetes Service (AKS)

+> * Clone a sample application source from GitHub.

+> * Create a container image from the sample application source.

+> * Test the multi-container application in a local Docker environment.

+> Azure Cloud Shell doesn't include the Docker components required to complete every step in these tutorials. Therefore, we recommend using a full Docker development environment.

+The [sample application][sample-application] used in this tutorial is a basic store front app including the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+1. Use [git][] to clone the sample application to your development environment.

+ ```console

+ git clone https://github.com/Azure-Samples/aks-store-demo.git

+ ```

+2. Change into the cloned directory.

+ ```console

+ cd aks-store-demo

+ ```

+## Review Docker Compose file

+The sample application you create in this tutorial uses the [*docker-compose-quickstart* YAML file](https://github.com/Azure-Samples/aks-store-demo/blob/main/docker-compose-quickstart.yml) in the [repository](https://github.com/Azure-Samples/aks-store-demo/tree/main) you cloned in the previous step.

+```yaml

+version: "3.7"

+ rabbitmq:

+ image: rabbitmq:3.11.17-management-alpine

+ container_name: 'rabbitmq'

+ restart: always

+ environment:

+ - "RABBITMQ_DEFAULT_USER=username"

+ - "RABBITMQ_DEFAULT_PASS=password"

+ ports:

+ - 15672:15672

+ - 5672:5672

+ healthcheck:

+ test: ["CMD", "rabbitmqctl", "status"]

+ interval: 30s

+ timeout: 10s

+ retries: 5

+ volumes:

+ - ./rabbitmq_enabled_plugins:/etc/rabbitmq/enabled_plugins

+ networks:

+ - backend_services

+ orderservice:

+ build: src/order-service

+ container_name: 'orderservice'

+ restart: always

+ ports:

+ - 3000:3000

+ healthcheck:

+ test: ["CMD", "wget", "-O", "", "-q", "http://orderservice:3000/health"]

+ interval: 30s

+ timeout: 10s

+ retries: 5

+ environment:

+ - ORDER_QUEUE_HOSTNAME=rabbitmq

+ - ORDER_QUEUE_PORT=5672

+ - ORDER_QUEUE_USERNAME=username

+ - ORDER_QUEUE_PASSWORD=password

+ - ORDER_QUEUE_NAME=orders

+ - ORDER_QUEUE_RECONNECT_LIMIT=3

+ networks:

+ - backend_services

+ depends_on:

+ rabbitmq:

+ condition: service_healthy

+ productservice:

+ build: src/product-service

+ container_name: 'productservice'

+ restart: always

+ ports:

+ - 3002:3002

+ healthcheck:

+ test: ["CMD", "wget", "-O", "", "-q", "http://productservice:3002/health"]

+ interval: 30s

+ timeout: 10s

+ retries: 5

+ networks:

+ - backend_services

+ storefront:

+ build: src/store-front

+ container_name: 'storefront'

+ restart: always

+ ports:

+ - 8080:8080

+ healthcheck:

+ test: ["CMD", "wget", "-O", "", "-q", "http://storefront:80/health"]

+ interval: 30s

+ timeout: 10s

+ retries: 5

+ environment:

+ - VUE_APP_PRODUCT_SERVICE_URL=http://productservice:3002/

+ - VUE_APP_ORDER_SERVICE_URL=http://orderservice:3000/

+ networks:

+ - backend_services

+ depends_on:

+ - productservice

+ - orderservice

+networks:

+ backend_

+ driver: bridge

+## Create container images and run application

+You can use [Docker Compose][docker-compose] to automate building container images and the deployment of multi-container applications.

+1. Create the container image, download the Redis image, and start the application using the `docker compose` command.

+ ```console

+ docker compose -f docker-compose-quickstart.yml up -d

+ ```

+2. View the created images using the [`docker images`][docker-images] command.

+ ```console

+ docker images

+ ```

+ The following condensed example output shows the created images:

+ ```output

+ REPOSITORY TAG IMAGE ID

+ aks-store-demo-productservice latest 2b66a7e91eca

+ aks-store-demo-orderservice latest 54ad5de546f9

+ aks-store-demo-storefront latest d9e3ac46a225

+ rabbitmq 3.11.17-management-alpine 79a570297657

+ ...

+ ```

+3. View the running containers using the [`docker ps`][docker-ps] command.

+ ```console

+ docker ps

+ ```

+ The following condensed example output shows four running containers:

+ ```output

+ CONTAINER ID IMAGE

+ 21574cb38c1f aks-store-demo-productservice

+ c30a5ed8d86a aks-store-demo-orderservice

+ d10e5244f237 aks-store-demo-storefront

+ 94e00b50b86a rabbitmq:3.11.17-management-alpine

+ ```

+On this page, you can view products, add them to your cart, and then place an order.

+## Clean up resources

+Since you validated the application's functionality, you can stop and remove the running containers. ***Do not delete the container images*** - you use them in the next tutorial.

+* Stop and remove the container instances and resources using the [`docker-compose down`][docker-compose-down] command.

+ ```console

+ docker compose down

+ ```

+> * Clone a sample application source from GitHub.

+> * Create a container image from the sample application source.

+> * Test the multi-container application in a local Docker environment.

+In the next tutorial, you learn how to store container images in an ACR.

+[sample-application]: https://github.com/Azure-Samples/aks-store-demo

+[aks-tutorial-prepare-acr]: ./tutorial-kubernetes-prepare-acr.md

+ Title: Kubernetes on Azure tutorial - Scale applications in Azure Kubernetes Service (AKS)

+# Tutorial - Scale applications in Azure Kubernetes Service (AKS)

+If you followed the previous tutorials, you have a working Kubernetes cluster and Azure Store Front app.

+In this tutorial, part six of seven, you scale out the pods in the app, try pod autoscaling, and scale the number of Azure VM nodes to change the cluster's capacity for hosting workloads. You learn how to:

+> * Configure autoscaling pods that run the app front end.

+In previous tutorials, you packaged an application into a container image, uploaded the image to Azure Container Registry, created an AKS cluster, deployed an application, and used Azure Service Bus to redeploy an updated application. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

+This tutorial requires Azure CLI version 2.34.1 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+1. View the pods in your cluster using the [`kubectl get`][kubectl-get] command.

+ The following example output shows the pods running the Azure Store Front app:

+ NAME READY STATUS RESTARTS AGE

+ order-service-848767080-tf34m 1/1 Running 0 31m

+ product-service-4019737227-2q2qz 1/1 Running 0 31m

+ store-front-2606967446-2q2qz 1/1 Running 0 31m

+2. Manually change the number of pods in the *store-front* deployment using the [`kubectl scale`][kubectl-scale] command.

+ kubectl scale --replicas=5 deployment.apps/store-front

+ The following example output shows the additional pods running the Azure Store Front app:

+ ```output

+ READY STATUS RESTARTS AGE

+ store-front-2606967446-2q2qzc 1/1 Running 0 15m

+ store-front-3309479140-2hfh0 1/1 Running 0 3m

+ store-front-3309479140-bzt05 1/1 Running 0 3m

+ store-front-3309479140-fvcvm 1/1 Running 0 3m

+ store-front-3309479140-hrbf2 1/1 Running 0 15m

+ store-front-3309479140-qphz8 1/1 Running 0 3m

+## Autoscale pods

+To use the horizontal pod autoscaler, all containers and pods must have defined CPU requests and limits. In the `aks-store-quickstart` deployment, the *front-end* container requests 1m CPU with a limit of 1000m CPU.

+...

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+...

+ cpu: 1m

+...

+ cpu: 1000m

+...

+1. Create a manifest file to define the autoscaler behavior and resource limits, as shown in the following condensed example manifest file `aks-store-quickstart-hpa.yaml`:

+ name: store-front-hpa

+ name: store-front

+ kubectl apply -f aks-store-quickstart-hpa.yaml

+ After a few minutes, with minimal load on the Azure Store Front app, the number of pod replicas decreases to three. You can use `kubectl get pods` again to see the unneeded pods being removed.

+> [!NOTE]

+> You can enable the Kubernetes-based Event-Driven Autoscaler (KEDA) AKS add-on to your cluster to drive scaling based on the number of events needing to be processed. For more information, see [Enable simplified application autoscaling with the Kubernetes Event-Driven Autoscaling (KEDA) add-on (Preview)][keda-addon].

+ ```azurecli-interactive

+* Scale your cluster nodes using the [`Get-AzAksCluster`][get-azakscluster] and [`Set-AzAksCluster`][set-azakscluster] cmdlets.

+ ```azurepowershell-interactive

+You can also autoscale the nodes in your cluster. For more information, see [Use the cluster autoscaler with node pools](./cluster-autoscaler.md#use-the-cluster-autoscaler-with-node-pools).

+In the next tutorial, you learn how to upgrade Kubernetes in your AKS cluster.

+> [Upgrade Kubernetes in Azure Kubernetes Service][aks-tutorial-upgrade-kubernetes]

+[aks-tutorial-upgrade-kubernetes]: ./tutorial-kubernetes-upgrade-cluster.md

+[keda-addon]: ./keda-about.md

+ Title: Kubernetes on Azure tutorial - Upgrade an Azure Kubernetes Service (AKS) cluster

+# Tutorial - Upgrade an Azure Kubernetes Service (AKS) cluster

+In this tutorial, part seven of seven, you upgrade an AKS cluster. You learn how to:

+In previous tutorials, you packaged an application into a container image and uploaded the container image to Azure Container Registry (ACR). You also created an AKS cluster and deployed an application to it. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

+If using Azure CLI, this tutorial requires Azure CLI version 2.34.1 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+If using Azure PowerShell, this tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+ ```azurecli-interactive

+ The following example output shows the current version as *1.26.6* and lists the available versions under `upgrades`:

+ "kubernetesVersion": "1.26.6",

+ "kubernetesVersion": "1.27.1"

+ "kubernetesVersion": "1.27.3"

+ ```azurepowershell-interactive

+ The following example output shows the current version as *1.26.6* and the location as *eastus*:

+ myAKSCluster 1.26.6 eastus

+ ```azurepowershell-interactive

+ Get-AzAksVersion -Location eastus | Where-Object OrchestratorVersion

+ The following example output shows the available versions under `OrchestratorVersion`:

+ Kubernetes 1.27.1

+ Kubernetes 1.27.3

+4. In **Kubernetes version**, select **Upgrade version**. This redirects you to a new page.

+## Upgrade an AKS cluster

+You can either [manually upgrade your cluster](#manually-upgrade-cluster) or [configure automatic cluster upgrades](#configure-automatic-cluster-upgrades). **We recommend you configure automatic cluster upgrades to ensure your cluster is always running the latest version of Kubernetes**.

+### Manually upgrade cluster

+#### [Azure CLI](#tab/azure-cli)

+ ```azurecli-interactive

+ The following example output shows the result of upgrading to *1.27.3*. Notice the `kubernetesVersion` now shows *1.27.3*:

+ "kubernetesVersion": "1.27.3",

+#### [Azure PowerShell](#tab/azure-powershell)

+ ```azurepowershell-interactive

+ The following example output shows the result of upgrading to *1.27.3*. Notice the `KubernetesVersion` now shows *1.27.3*:

+ KubernetesVersion : 1.27.3

+#### [Azure portal](#tab/azure-portal)

+3. In **Kubernetes version**, select **Upgrade version**. This redirects you to a new page.

+### Configure automatic cluster upgrades

+#### [Azure CLI](#tab/azure-cli)

+* Set an auto-upgrade channel on your cluster using the [`az aks update`][az-aks-update] command with the `--auto-upgrade-channel` parameter set to `patch`.

+ ```azurecli-interactive

+ az aks update --resource-group myResourceGroup --name myAKSCluster --auto-upgrade-channel patch

+ ```

+#### [Azure PowerShell](#tab/azure-powershell)

+* Set an auto-upgrade channel on your cluster using the [`Set-AzAksCluster`][set-azakscluster] cmdlet with the `-AutoUpgradeChannel` parameter set to `Patch`.

+ ```azurepowershell-interactive

+ Set-AzAksCluster -ResourceGroupName myResourceGroup -Name myAKSCluster -AutoUpgradeChannel Patch

+ ```

+#### [Azure portal](#tab/azure-portal)

+1. In the Azure portal, navigate to your AKS cluster.

+2. Under **Settings**, select **Cluster configuration**.

+3. In **Kubernetes version**, select **Upgrade version**.

+4. For **Automatic upgrade**, select **Enabled with patch (recommended)** > **Save**.

+For more information, see [Automatically upgrade an Azure Kubernetes Service (AKS) cluster][aks-auto-upgrade].

+#### Upgrade AKS node images

+AKS regularly provides new node images. Linux node images are updated weekly, and Windows node images are updated monthly. We recommend upgrading your node images frequently to use the latest AKS features and security updates. For more information, see [Upgrade node images in Azure Kubernetes Service (AKS)][node-image-upgrade]. To configure automatic node image upgrades, see [Automatically upgrade Azure Kubernetes Service (AKS) cluster node operating system images][auto-upgrade-node-image].

+ ```console

+ The following example output shows some of the above events listed during an upgrade:

+ ```azurecli-interactive

+ The following example output shows the AKS cluster runs *KubernetesVersion 1.27.3*:

+ myAKSCluster eastus myResourceGroup 1.27.3 1.27.3 Succeeded myaksclust-myresourcegroup-19da35-bd54a4be.hcp.eastus.azmk8s.io

+ ```azurepowershell-interactive

+ The following example output shows the AKS cluster runs *KubernetesVersion 1.27.3*:

+ myAKSCluster eastus 1.27.3 Succeeded

+As this tutorial is the last part of the series, you might want to delete your AKS cluster to avoid incurring Azure charges.

+3. On the popup that asks you to confirm the deletion of the cluster, select **Yes**.

+[aks-auto-upgrade]: ./auto-upgrade-cluster.md

+[auto-upgrade-node-image]: ./auto-upgrade-node-image.md

+[node-image-upgrade]: ./node-image-upgrade.md

+## Azure Virtual Network Manager

+Azure Virtual Network Manager is a management service that allows you to group, configure, deploy, and manage virtual networks globally across subscriptions. With Virtual Network Manager, you can define network groups to identify and logically segment your virtual networks. After that, you can determine the connectivity and security configurations you want and apply them across all the selected virtual networks in network groups at once. Azure Virtual Network Manager's security admin rule configuration allows you to define security policies at scale and apply them to multiple virtual networks at once.

+> [!NOTE]

+> Security admin rules of Azure Virtual Network Manager apply to Application Gateway subnets that only contain Application Gateways that have ["Network Isolation"](Application-gateway-private-deployment.md) enabled. Subnets that have any Application Gateway that does not have ["Network Isolation"](Application-gateway-private-deployment.md) enabled, will not have security admin rules.

+| SUSE Rancher | [K3s](https://rancher.com/products/k3s/) | [v1.27.4+k3s1](https://github.com/k3s-io/k3s/releases/tag/v1.27.4%2Bk3s1), [v1.26.7+k3s1](https://github.com/k3s-io/k3s/releases/tag/v1.26.7%2Bk3s1), [v1.25.12+k3s1](https://github.com/k3s-io/k3s/releases/tag/v1.25.12%2Bk3s1) |

+* Virtual machines (preview): Provision, resize, delete and manage virtual machines based on [VMware vSphere](./vmware-vsphere/overview.md) or [Azure Stack HCI](/azure-stack/hci/manage/azure-arc-vm-management-overview) and enable VM self-service through role-based access.

+* Perform virtual machine lifecycle and management operations for [VMware vSphere](./vmware-vsphere/overview.md) and [Azure Stack HCI](/azure-stack/hci/manage/azure-arc-vm-management-overview) environments.

+* Learn about [Azure Arc-enabled VMware vSphere](vmware-vsphere/overview.md).

+* Learn about [Azure Arc-enabled VM Management on Azure Stack HCI](/azure-stack/hci/manage/azure-arc-vm-management-overview).

+ * Azure Arc VM management on Azure Stack HCI

+* Custom locations: A deployment target where you can create Azure resources. It maps to different resource for different Azure services. For example, for Arc-enabled VMware, the custom locations resource maps to an instance of vCenter, and for Azure Arc VM management on Azure Stack HCI, it maps to an HCI cluster instance.

+## Example scenarios

+The following are just two examples of the many scenarios that can be enabled by using Arc resource bridge in a hybrid environment.

+### Apply Azure Policy and other Azure services to on-premises VMware VMs

+A customer deploys Arc Resource Bridge onto their on-premises VMware environment. They sign into the Azure portal and select the VMware VMs that they'd like to connect to Azure. Now they can manage these on-premises VMware VMs in Azure Resource Manager (ARM) as Arc-enabled machines, alongside their native Azure machines, achieving a single pane of glass to view their resources in a VMware/Azure hybrid environment. This includes deploying Azure services, such as Defender for Cloud and Azure Policy, to keep updated on the security and compliance posture of their on-premises VMware VMs in Azure.

+### Create physical HCI VMs on-premises from Azure

+A customer has multiple datacenter locations in Canada and New York. They install an Arc resource bridge in each datacenter and connect their Azure Stack HCI VMs to Azure in each location. They can then sign into Azure portal and see all their Arc-enabled VMs from the two physical locations together in one central cloud location. From the portal, the customer can choose to create a new VM; that VM is also created on-premises at the selected datacenter, allowing the customer to manage VMs in different physical locations centrally through Azure.

+## Version and region support

+In order to use Arc resource bridge in a region, Arc resource bridge and the Arc-enabled feature for a private cloud must be supported in the region. For example, to use Arc resource bridge with Azure Stack HCI in East US, Arc resource bridge and the Arc VM management feature for Azure Stack HCI must be supported in East US. To confirm feature availability across regions for each private cloud provider, review their deployment guide and other documentation. There could be instances where Arc resource bridge is available in a region where the private cloud feature is not yet available.

+When the Arc-enabled private cloud announces General Availability, the minimum supported version of Arc resource bridge will be 1.0.15.

+Generally, the latest released version and the previous three versions (n-3) of Arc resource bridge are supported. For example, if the current version is 1.0.18, then the typical n-3 supported versions are:

+* Current version: 1.0.18

+* n-1 version: 1.0.17

+* n-2 version: 1.0.16

+* n-3 version: 1.0.15

+There may be instances where supported versions are not sequential. For example, version 1.0.18 is released and later found to contain a bug; a hot fix is released in version 1.0.19 and version 1.0.18 is removed. In this scenario, n-3 supported versions become 1.0.19, 1.0.17, 1.0.16, 1.0.15.

+Arc resource bridge typically releases a new version on a monthly cadence, at the end of the month. Delays may occur that could push the release date further out. Regardless of when a new release comes out, if you are within n-3 supported versions (starting with 1.0.15), then your Arc resource bridge version is supported. To stay updated on releases, visit the [Arc resource bridge release notes](https://github.com/Azure/ArcResourceBridge/releases) on GitHub. To learn more about upgrade options, visit [Upgrade Arc resource bridge](upgrade.md).

+* Review the [system requirements](system-requirements.md) for deploying and managing Arc resource bridge.

+ Title: Overview of Azure Connected Machine agent to manage Windows and Linux machines

+description: This article provides a detailed overview of the Azure Connected Machine agent, which supports monitoring virtual machines hosted in hybrid environments.

+ms.

+# Overview of Azure Connected Machine agent to manage Windows and Linux machines

+The Azure Connected Machine agent enables you to manage your Windows and Linux machines hosted outside of Azure on your corporate network or other cloud providers.

+## Agent components

+The Azure Connected Machine agent package contains several logical components bundled together:

+* The Hybrid Instance Metadata service (HIMDS) manages the connection to Azure and the connected machine's Azure identity.

+* The guest configuration agent provides functionality such as assessing whether the machine complies with required policies and enforcing compliance.

+ Note the following behavior with Azure Policy [guest configuration](../../governance/machine-configuration/overview.md) for a disconnected machine:

+ * An Azure Policy assignment that targets disconnected machines is unaffected.

+ * Guest assignment is stored locally for 14 days. Within the 14-day period, if the Connected Machine agent reconnects to the service, policy assignments are reapplied.

+ * Assignments are deleted after 14 days, and aren't reassigned to the machine after the 14-day period.

+* The Extension agent manages VM extensions, including install, uninstall, and upgrade. Azure downloads extensions and copies them to the `%SystemDrive%\%ProgramFiles%\AzureConnectedMachineAgent\ExtensionService\downloads` folder on Windows, and to `/opt/GC_Ext/downloads` on Linux. On Windows, the extension installs to the following path `%SystemDrive%\Packages\Plugins\<extension>`, and on Linux the extension installs to `/var/lib/waagent/<extension>`.

+>[!NOTE]

+> The [Azure Monitor agent](../../azure-monitor/agents/azure-monitor-agent-overview.md) (AMA) is a separate agent that collects monitoring data, and it does not replace the Connected Machine agent; the AMA only replaces the Log Analytics agent, Diagnostics extension, and Telegraf agent for both Windows and Linux machines.

+## Agent resources

+The following information describes the directories and user accounts used by the Azure Connected Machine agent.

+### Windows agent installation details

+The Windows agent is distributed as a Windows Installer package (MSI). Download the Windows agent from the [Microsoft Download Center](https://aka.ms/AzureConnectedMachineAgent).

+Installing the Connected Machine agent for Window applies the following system-wide configuration changes:

+* The installation process creates the following folders during setup.

+ | Directory | Description |

+ |--|-|

+ | %ProgramFiles%\AzureConnectedMachineAgent | azcmagent CLI and instance metadata service executables.|

+ | %ProgramFiles%\AzureConnectedMachineAgent\ExtensionService\GC | Extension service executables.|

+ | %ProgramFiles%\AzureConnectedMachineAgent\GCArcService\GC | Guest configuration (policy) service executables.|

+ | %ProgramData%\AzureConnectedMachineAgent | Configuration, log and identity token files for azcmagent CLI and instance metadata service.|

+ | %ProgramData%\GuestConfig | Extension package downloads, guest configuration (policy) definition downloads, and logs for the extension and guest configuration services.|

+ | %SYSTEMDRIVE%\packages | Extension package executables |

+* Installing the agent creates the following Windows services on the target machine.

+ | Service name | Display name | Process name | Description |

+ |--|--|--|-|

+ | himds | Azure Hybrid Instance Metadata Service | himds | Synchronizes metadata with Azure and hosts a local REST API for extensions and applications to access the metadata and request Microsoft Entra managed identity tokens |

+ | GCArcService | Guest configuration Arc Service | gc_service | Audits and enforces Azure guest configuration policies on the machine. |

+ | ExtensionService | Guest configuration Extension Service | gc_service | Installs, updates, and manages extensions on the machine. |

+* Agent installation creates the following virtual service account.

+ | Virtual Account | Description |

+ ||-|

+ | NT SERVICE\\himds | Unprivileged account used to run the Hybrid Instance Metadata Service. |

+ > [!TIP]

+ > This account requires the "Log on as a service" right. This right is automatically granted during agent installation, but if your organization configures user rights assignments with Group Policy, you might need to adjust your Group Policy Object to grant the right to "NT SERVICE\\himds" or "NT SERVICE\\ALL SERVICES" to allow the agent to function.

+* Agent installation creates the following local security group.

+ | Security group name | Description |

+ ||-|

+ | Hybrid agent extension applications | Members of this security group can request Microsoft Entra tokens for the system-assigned managed identity |

+* Agent installation creates the following environmental variables

+ | Name | Default value | Description |

+ ||||

+ | IDENTITY_ENDPOINT | `http://localhost:40342/metadata/identity/oauth2/token` |

+ | IMDS_ENDPOINT | `http://localhost:40342` |

+* There are several log files available for troubleshooting, described in the following table.

+ | Log | Description |

+ |--|-|

+ | %ProgramData%\AzureConnectedMachineAgent\Log\himds.log | Records details of the heartbeat and identity agent component. |

+ | %ProgramData%\AzureConnectedMachineAgent\Log\azcmagent.log | Contains the output of the azcmagent tool commands. |

+ | %ProgramData%\GuestConfig\arc_policy_logs\gc_agent.log | Records details about the guest configuration (policy) agent component. |

+ | %ProgramData%\GuestConfig\ext_mgr_logs\gc_ext.log | Records details about extension manager activity (extension install, uninstall, and upgrade events). |

+ | %ProgramData%\GuestConfig\extension_logs | Directory containing logs for individual extensions. |

+* The process creates the local security group **Hybrid agent extension applications**.

+* After uninstalling the agent, the following artifacts remain.

+ * %ProgramData%\AzureConnectedMachineAgent\Log

+ * %ProgramData%\AzureConnectedMachineAgent

+ * %ProgramData%\GuestConfig

+ * %SystemDrive%\packages

+### Linux agent installation details

+The preferred package format for the distribution (`.rpm` or `.deb`) that's hosted in the Microsoft [package repository](https://packages.microsoft.com/) provides the Connected Machine agent for Linux. The shell script bundle [Install_linux_azcmagent.sh](https://aka.ms/azcmagent) installs and configures the agent.

+Installing, upgrading, and removing the Connected Machine agent isn't required after server restart.

+Installing the Connected Machine agent for Linux applies the following system-wide configuration changes.

+* Setup creates the following installation folders.

+ | Directory | Description |

+ |--|-|

+ | /opt/azcmagent/ | azcmagent CLI and instance metadata service executables. |

+ | /opt/GC_Ext/ | Extension service executables. |

+ | /opt/GC_Service/ | Guest configuration (policy) service executables. |

+ | /var/opt/azcmagent/ | Configuration, log and identity token files for azcmagent CLI and instance metadata service.|

+ | /var/lib/GuestConfig/ | Extension package downloads, guest configuration (policy) definition downloads, and logs for the extension and guest configuration services.|

+* Installing the agent creates the following daemons.

+ | Service name | Display name | Process name | Description |

+ |--|--|--|-|

+ | himdsd.service | Azure Connected Machine Agent Service | himds | This service implements the Hybrid Instance Metadata service (IMDS) to manage the connection to Azure and the connected machine's Azure identity.|

+ | gcad.service | GC Arc Service | gc_linux_service | Audits and enforces Azure guest configuration policies on the machine. |

+ | extd.service | Extension Service | gc_linux_service | Installs, updates, and manages extensions on the machine. |

+* There are several log files available for troubleshooting, described in the following table.

+ | Log | Description |

+ |--|-|

+ | /var/opt/azcmagent/log/himds.log | Records details of the heartbeat and identity agent component. |

+ | /var/opt/azcmagent/log/azcmagent.log | Contains the output of the azcmagent tool commands. |

+ | /var/lib/GuestConfig/arc_policy_logs | Records details about the guest configuration (policy) agent component. |

+ | /var/lib/GuestConfig/ext_mgr_logs | Records details about extension manager activity (extension install, uninstall, and upgrade events). |

+ | /var/lib/GuestConfig/extension_logs | Directory containing logs for individual extensions. |

+* Agent installation creates the following environment variables, set in `/lib/systemd/system.conf.d/azcmagent.conf`.

+ | Name | Default value | Description |

+ |||-|

+ | IDENTITY_ENDPOINT | `http://localhost:40342/metadata/identity/oauth2/token` |

+ | IMDS_ENDPOINT | `http://localhost:40342` |

+* After uninstalling the agent, the following artifacts remain.

+ * /var/opt/azcmagent

+ * /var/lib/GuestConfig

+## Agent resource governance

+The Azure Connected Machine agent is designed to manage agent and system resource consumption. The agent approaches resource governance under the following conditions:

+* The Guest Configuration agent can use up to 5% of the CPU to evaluate policies.

+* The Extension Service agent can use up to 5% of the CPU to install, upgrade, run, and delete extensions. Some extensions might apply more restrictive CPU limits once installed. The following exceptions apply:

+ | Extension type | Operating system | CPU limit |

+ | -- | - | |

+ | AzureMonitorLinuxAgent | Linux | 60% |

+ | AzureMonitorWindowsAgent | Windows | 100% |

+ | AzureSecurityLinuxAgent | Linux | 30% |

+ | LinuxOsUpdateExtension | Linux | 60% |

+ | MDE.Linux | Linux | 60% |

+ | MicrosoftDnsAgent | Windows | 100% |

+ | MicrosoftMonitoringAgent | Windows | 60% |

+ | OmsAgentForLinux | Windows | 60%|

+During normal operations, defined as the Azure Connected Machine agent being connected to Azure and not actively modifying an extension or evaluating a policy, you can expect the agent to consume the following system resources:

+| | Windows | Linux |

+| | - | -- |

+| **CPU usage (normalized to 1 core)** | 0.07% | 0.02% |

+| **Memory usage** | 57 MB | 42 MB |

+The performance data above was gathered in April 2023 on virtual machines running Windows Server 2022 and Ubuntu 20.04. Actual agent performance and resource consumption will vary based on the hardware and software configuration of your servers.

+## Instance metadata

+Metadata information about a connected machine is collected after the Connected Machine agent registers with Azure Arc-enabled servers. Specifically:

+* Operating system name, type, and version

+* Computer name

+* Computer manufacturer and model

+* Computer fully qualified domain name (FQDN)

+* Domain name (if joined to an Active Directory domain)

+* Active Directory and DNS fully qualified domain name (FQDN)

+* UUID (BIOS ID)

+* Connected Machine agent heartbeat

+* Connected Machine agent version

+* Public key for managed identity

+* Policy compliance status and details (if using guest configuration policies)

+* SQL Server installed (Boolean value)

+* Cluster resource ID (for Azure Stack HCI nodes)

+* Hardware manufacturer

+* Hardware model

+* CPU family, socket, physical core and logical core counts

+* Total physical memory

+* Serial number

+* SMBIOS asset tag

+* Cloud provider

+The agent requests the following metadata information from Azure:

+* Resource location (region)

+* Virtual machine ID

+* Tags

+* Microsoft Entra managed identity certificate

+* Guest configuration policy assignments

+* Extension requests - install, update, and delete.

+> [!NOTE]

+> Azure Arc-enabled servers doesn't store/process customer data outside the region the customer deploys the service instance in.

+## Next steps

+- [Connect your SCVMM server to Azure Arc](/azure/azure-arc/system-center-virtual-machine-manager/quickstart-connect-system-center-virtual-machine-manager-to-arc).

+- [Install Arc agent at scale for your SCVMM VMs](/azure/azure-arc/system-center-virtual-machine-manager/enable-guest-management-at-scale).

+- [Install Arc agent using a script for SCVMM VMs](/azure/azure-arc/system-center-virtual-machine-manager/install-arc-agents-using-script).

+ Title: Azure Arc agent

+description: Learn about Azure Arc agent

+# Azure Arc agent

+The Azure Connected Machine agent enables you to manage your Windows and Linux machines hosted outside of Azure on your corporate network or other cloud providers.

+## Agent components

+The Azure Connected Machine agent package contains several logical components bundled together:

+* The Hybrid Instance Metadata service (HIMDS) manages the connection to Azure and the connected machine's Azure identity.

+* The guest configuration agent provides functionality such as assessing whether the machine complies with required policies and enforcing compliance.

+ Note the following behavior with Azure Policy [guest configuration](../../governance/machine-configuration/overview.md) for a disconnected machine:

+ * An Azure Policy assignment that targets disconnected machines is unaffected.

+ * Guest assignment is stored locally for 14 days. Within the 14-day period, if the Connected Machine agent reconnects to the service, policy assignments are reapplied.

+ * Assignments are deleted after 14 days and aren't reassigned to the machine after the 14-day period.

+* The Extension agent manages VM extensions, including install, uninstall, and upgrade. Azure downloads extensions and copies them to the `%SystemDrive%\%ProgramFiles%\AzureConnectedMachineAgent\ExtensionService\downloads` folder on Windows, and to `/opt/GC_Ext/downloads` on Linux. On Windows, the extension installs to the path `%SystemDrive%\Packages\Plugins\<extension>`, and on Linux the extension installs to `/var/lib/waagent/<extension>`.

+>[!NOTE]

+> The [Azure Monitor agent](../../azure-monitor/agents/azure-monitor-agent-overview.md) (AMA) is a separate agent that collects monitoring data, and it does not replace the Connected Machine agent; the AMA only replaces the Log Analytics agent, Diagnostics extension, and Telegraf agent for both Windows and Linux machines.

+## Agent resources

+The following information describes the directories and user accounts used by the Azure Connected Machine agent.

+### Windows agent installation details

+The Windows agent is distributed as a Windows Installer package (MSI). Download the Windows agent from the [Microsoft Download Center](https://aka.ms/AzureConnectedMachineAgent).

+Installing the Connected Machine agent for Window applies the following system-wide configuration changes:

+* The installation process creates the following folders during setup.

+ | Directory | Description |

+ |--|-|

+ | %ProgramFiles%\AzureConnectedMachineAgent | azcmagent CLI and instance metadata service executables.|

+ | %ProgramFiles%\AzureConnectedMachineAgent\ExtensionService\GC | Extension service executables.|

+ | %ProgramFiles%\AzureConnectedMachineAgent\GCArcService\GC | Guest configuration (policy) service executables.|

+ | %ProgramData%\AzureConnectedMachineAgent | Configuration, log and identity token files for azcmagent CLI and instance metadata service.|

+ | %ProgramData%\GuestConfig | Extension package downloads, guest configuration (policy) definition downloads, and logs for the extension and guest configuration services.|

+ | %SYSTEMDRIVE%\packages | Extension package executables. |

+* Installing the agent creates the following Windows services on the target machine.

+ | Service name | Display name | Process name | Description |

+ |--|--|--|-|

+ | himds | Azure Hybrid Instance Metadata Service | himds | Synchronizes metadata with Azure and hosts a local REST API for extensions and applications to access the metadata and request Microsoft Entra managed identity tokens |

+ | GCArcService | Guest configuration Arc Service | gc_service | Audits and enforces Azure guest configuration policies on the machine. |

+ | ExtensionService | Guest configuration Extension Service | gc_service | Installs, updates, and manages extensions on the machine. |

+* Agent installation creates the following virtual service account.

+ | Virtual Account | Description |

+ ||-|

+ | NT SERVICE\\himds | Unprivileged account used to run the Hybrid Instance Metadata Service. |

+ > [!TIP]

+ > This account requires the *Log on as a service* right. This right is automatically granted during agent installation, but if your organization configures user rights assignments with Group Policy, you might need to adjust your Group Policy Object to grant the right to **NT SERVICE\\himds** or **NT SERVICE\\ALL SERVICES** to allow the agent to function.

+* Agent installation creates the following local security group.

+ | Security group name | Description |

+ ||-|

+ | Hybrid agent extension applications | Members of this security group can request Microsoft Entra tokens for the system-assigned managed identity |

+* Agent installation creates the following environmental variables

+ | Name | Default value | Description |

+ ||||

+ | IDENTITY_ENDPOINT | `http://localhost:40342/metadata/identity/oauth2/token` |

+ | IMDS_ENDPOINT | `http://localhost:40342` |

+* There are several log files available for troubleshooting, described in the following table.

+ | Log | Description |

+ |--|-|

+ | %ProgramData%\AzureConnectedMachineAgent\Log\himds.log | Records details of the heartbeat and identity agent component. |

+ | %ProgramData%\AzureConnectedMachineAgent\Log\azcmagent.log | Contains the output of the azcmagent tool commands. |

+ | %ProgramData%\GuestConfig\arc_policy_logs\gc_agent.log | Records details about the guest configuration (policy) agent component. |

+ | %ProgramData%\GuestConfig\ext_mgr_logs\gc_ext.log | Records details about extension manager activity (extension install, uninstall, and upgrade events). |

+ | %ProgramData%\GuestConfig\extension_logs | Directory containing logs for individual extensions. |

+* The process creates the local security group **Hybrid agent extension applications**.

+* After uninstalling the agent, the following artifacts remain:

+ * %ProgramData%\AzureConnectedMachineAgent\Log

+ * %ProgramData%\AzureConnectedMachineAgent

+ * %ProgramData%\GuestConfig

+ * %SystemDrive%\packages

+### Linux agent installation details

+The preferred package format for the distribution (`.rpm` or `.deb`) that's hosted in the Microsoft [package repository](https://packages.microsoft.com/) provides the Connected Machine agent for Linux. The shell script bundle [Install_linux_azcmagent.sh](https://aka.ms/azcmagent) installs and configures the agent.

+Installing, upgrading, and removing the Connected Machine agent isn't required after server restart.

+Installing the Connected Machine agent for Linux applies the following system-wide configuration changes.

+* Setup creates the following installation folders.

+ | Directory | Description |

+ |--|-|

+ | /opt/azcmagent/ | azcmagent CLI and instance metadata service executables. |

+ | /opt/GC_Ext/ | Extension service executables. |

+ | /opt/GC_Service/ | Guest configuration (policy) service executables. |

+ | /var/opt/azcmagent/ | Configuration, log and identity token files for azcmagent CLI and instance metadata service.|

+ | /var/lib/GuestConfig/ | Extension package downloads, guest configuration (policy) definition downloads, and logs for the extension and guest configuration services.|

+* Installing the agent creates the following daemons.

+ | Service name | Display name | Process name | Description |

+ |--|--|--|-|

+ | himdsd.service | Azure Connected Machine Agent Service | himds | This service implements the Hybrid Instance Metadata service (IMDS) to manage the connection to Azure and the connected machine's Azure identity.|

+ | gcad.service | GC Arc Service | gc_linux_service | Audits and enforces Azure guest configuration policies on the machine. |

+ | extd.service | Extension Service | gc_linux_service | Installs, updates, and manages extensions on the machine. |

+* There are several log files available for troubleshooting, described in the following table.

+ | Log | Description |

+ |--|-|

+ | /var/opt/azcmagent/log/himds.log | Records details of the heartbeat and identity agent component. |

+ | /var/opt/azcmagent/log/azcmagent.log | Contains the output of the azcmagent tool commands. |

+ | /var/lib/GuestConfig/arc_policy_logs | Records details about the guest configuration (policy) agent component. |

+ | /var/lib/GuestConfig/ext_mgr_logs | Records details about extension manager activity (extension install, uninstall, and upgrade events). |

+ | /var/lib/GuestConfig/extension_logs | Directory containing logs for individual extensions. |

+* Agent installation creates the following environment variables, set in `/lib/systemd/system.conf.d/azcmagent.conf`.

+ | Name | Default value | Description |

+ |||-|

+ | IDENTITY_ENDPOINT | `http://localhost:40342/metadata/identity/oauth2/token` |

+ | IMDS_ENDPOINT | `http://localhost:40342` |

+* After uninstalling the agent, the following artifacts remain:

+ * /var/opt/azcmagent

+ * /var/lib/GuestConfig

+## Agent resource governance

+The Azure Connected Machine agent is designed to manage agent and system resource consumption. The agent approaches resource governance under the following conditions:

+* The Guest Configuration agent can use up to 5% of the CPU to evaluate policies.

+* The Extension Service agent can use up to 5% of the CPU to install, upgrade, run, and delete extensions. Some extensions might apply more restrictive CPU limits once installed. The following exceptions apply:

+ | Extension type | Operating system | CPU limit |

+ | -- | - | |

+ | AzureMonitorLinuxAgent | Linux | 60% |

+ | AzureMonitorWindowsAgent | Windows | 100% |

+ | AzureSecurityLinuxAgent | Linux | 30% |

+ | LinuxOsUpdateExtension | Linux | 60% |

+ | MDE.Linux | Linux | 60% |

+ | MicrosoftDnsAgent | Windows | 100% |

+ | MicrosoftMonitoringAgent | Windows | 60% |

+ | OmsAgentForLinux | Windows | 60%|

+During normal operations, defined as the Azure Connected Machine agent being connected to Azure and not actively modifying an extension or evaluating a policy, you can expect the agent to consume the following system resources:

+| | Windows | Linux |

+| | - | -- |

+| **CPU usage (normalized to 1 core)** | 0.07% | 0.02% |

+| **Memory usage** | 57 MB | 42 MB |

+The performance data above was gathered in April 2023 on virtual machines running Windows Server 2022 and Ubuntu 20.04. The actual agent performance and resource consumption vary based on the hardware and software configuration of your servers.

+## Instance metadata

+Metadata information about a connected machine is collected after the Connected Machine agent registers with Azure Arc-enabled servers, specifically:

+* Operating system name, type, and version

+* Computer name

+* Computer manufacturer and model

+* Computer fully qualified domain name (FQDN)

+* Domain name (if joined to an Active Directory domain)

+* Active Directory and DNS fully qualified domain name (FQDN)

+* UUID (BIOS ID)

+* Connected Machine agent heartbeat

+* Connected Machine agent version

+* Public key for managed identity

+* Policy compliance status and details (if using guest configuration policies)

+* SQL Server installed (Boolean value)

+* Cluster resource ID (for Azure Stack HCI nodes)

+* Hardware manufacturer

+* Hardware model

+* CPU family, socket, physical core and logical core counts

+* Total physical memory

+* Serial number

+* SMBIOS asset tag

+* Cloud provider

+* Amazon Web Services (AWS) metadata, when running in AWS:

+ * Account ID

+ * Instance ID

+ * Region

+* Google Cloud Platform (GCP) metadata, when running in GCP:

+ * Instance ID

+ * Image

+ * Machine type

+ * Project ID

+ * Project number

+ * Service accounts

+ * Zone

+The agent requests the following metadata information from Azure:

+* Resource location (region)

+* Virtual machine ID

+* Tags

+* Microsoft Entra managed identity certificate

+* Guest configuration policy assignments

+* Extension requests - install, update, and delete.

+> [!NOTE]

+> Azure Arc-enabled servers don't store/process customer data outside the region the customer deploys the service instance in.

+## Next steps

+- [Connect VMware vCenter Server to Azure Arc](quick-start-connect-vcenter-to-arc-using-script.md).

+- [Install Arc agent at scale for your VMware VMs](enable-guest-management-at-scale.md).

+| [PowerShell](vminsights-enable-powershell.md) | Use a PowerShell script to enable multiple machines. Currently only supported for Log Analytics agent. |

+| [Manual install](vminsights-enable-hybrid.md) | Virtual machines or physical computers on-premises with other cloud environments.|

+- AgentId: The unique identifier for a machine running Azure Monitor Agent or the Log Analytics agent

+|AgentId | The unique identifier for a machine running Azure Monitor Agent or the Log Analytics agent |

+|AgentId | The unique identifier for a machine running Azure Monitor Agent or the Log Analytics agent |

+- Azure Monitor Agent is installed on Azure virtual machines by using a VM extension if the agent is already installed.

+ Title: Special cases to move Azure VMs to new subscription or resource group

+# Handling special cases when moving virtual machines to resource group or subscription

+This article describes special cases that require extra steps when moving a virtual machine to a new resource group or Azure subscription. If your virtual machine doesn't match any of these scenarios, you can move the virtual machine with the standard steps described in [Move resources to a new resource group or subscription](../move-resource-group-and-subscription.md).

+Virtual machines created from Marketplace resources with plans attached can't be moved across subscriptions. To work around this limitation, you can deprovision the virtual machine in the current subscription, and deploy it again in the new subscription. The following steps help you recreate the virtual machine in the new subscription. However, they might not work for all scenarios. If the plan is no longer available in the Marketplace, these steps won't work.

+# Move Azure resources to a new resource group or subscription

+# [Portal](#tab/Portal)

+To create a replica in an Azure region, go to your Web PubSub resource and find the **Replicas** blade on the Azure portal and click **Add** to create a replica.

+# [Bicep](#tab/Bicep)

+Use Visual Studio Code or your favorite editor to create a file with the following content and name it main.bicep:

+```bicep

+@description('The name for your Web PubSub service')

+param primaryName string = 'contoso'

+@description('The region in which to create your Web Pubsub service')

+param primaryLocation string = 'eastus'

+@description('Unit count of your Web PubSub service')

+param primaryCapacity int = 1

+resource primary 'Microsoft.SignalRService/webpubsub@2023-08-01-preview' = {

+ name: primaryName

+ location: primaryLocation

+ sku: {

+ capacity: primaryCapacity

+ name: 'Premium_P1'

+ }

+ properties: {

+ }

+}

+@description('The name for your Web PubSub replica')

+param replicaName string = 'contoso-westus'

+@description('The region in which to create the Web PubSub replica')

+param replicaLocation string = 'westus'

+@description('Unit count of the Web PubSub replica')

+param replicaCapacity int = 1

+@description('Whether to enable region endpoint for the replica')

+param regionEndpointEnabled string = 'Enabled'

+resource replica 'Microsoft.SignalRService/webpubsub/replicas@2023-08-01-preview' = {

+ parent: primary

+ name: replicaName

+ location: replicaLocation

+ sku: {

+ capacity: replicaCapacity

+ name: 'Premium_P1'

+ }

+ properties: {

+ regionEndpointEnabled: regionEndpointEnabled

+ }

+}

+```

+Deploy the Bicep file using Azure CLI

+ ```azurecli

+ az group create --name MyResourceGroup --location eastus

+ az deployment group create --resource-group MyResourceGroup --template-file main.bicep

+ ```

+# [CLI](#tab/CLI)

+Update **webpubsub** extension to the latest version, then run:

+ ```azurecli

+ az webpubsub replica create --sku Premium_P1 -l eastus --replica-name MyReplica --name MyWebPubSub -g MyResourceGroup

+ ```

+To delete a replica using the Azure CLI:

+ ```azurecli

+ az webpubsub replica delete --replica-name MyReplica --name MyWebPubSub -g MyResourceGroup

+ ```

+> * Due to the DNS cache, it might take several minutes for the DNS update to take effect.

+ If you use the Web PubSub event handler with Web PubSub C# server SDK or an Azure Function that utilizes the Web PubSub extension, you might encounter issues with the abuse protection once replicas are enabled. To address this, you can either **disable the abuse protection** or **upgrade to the latest SDK/extension versions**.

+| Region | West US |

+- **Authentication**. To use a pool without public IP addresses inside a [virtual network](./batch-virtual-network.md), the Batch client API must use Microsoft Entra authentication. Azure Batch support for Microsoft Entra ID is documented in [Authenticate Azure Batch services with Microsoft Entra ID](batch-aad-auth.md). If you aren't creating your pool within a virtual network, either Microsoft Entra authentication or key-based authentication can be used.

+(Azure AD ID). This identity is used to obtain Microsoft Entra tokens to authenticate with target

+- **Supported browsers** - The Chaos Studio portal experience has only been tested on the following browsers:

+- **Terraform** - At present Chaos Studio does not support terraform.

+- **Built-in roles** - Chaos Studio does not currently have its own built-in roles. Permissions may be attained to run a chaos experiment by either assigning an [Azure built-in role](chaos-studio-fault-providers.md) or a created custom role to the experiment's identity.

+<!-- LINKS - Internal -->

+[resource-manager template]: container-instances-multi-container-group.md

+| `CREATE INDEX` | Yes (including [named indexes](secondary-indexing.md) but full FROZEN collection is not supported) |

+Azure Cosmos DB offers multiple database APIs, which include NoSQL, MongoDB, PostgreSQL, Cassandra, Gremlin, and Table. By using these APIs, you can model real world data using documents, key-value, graph, and column-family data models. These APIs allow your applications to treat Azure Cosmos DB as if it were various other databases technologies, without the overhead of management, and scaling approaches. Azure Cosmos DB helps you to use the ecosystems, tools, and skills you already have for data modeling and querying with its various APIs.

+* In Azure Cosmos DB for NoSQL, the JSON object represents the data format. The partition key resides in a field in the document and sets which field is the partition key for the collection. Azure Cosmos DB doesn't have the concept of timestamp used for column family or version. As highlighted previously, it has change feed support through which one can track/record changes performed on a container. The following is an example of a document.

+There's a [capacity calculator](estimate-ru-with-capacity-planner.md) available to assist with sizing exercise for RUs.

+Physical partitions each contain a replica of your data and an instance of the Azure Cosmos DB database engine. This structure makes your data durable and highly available and throughput is divided equally amongst the local physical partitions. Physical partitions are automatically created and configured, and it's not possible to control their size, location, or which logical partitions they contain. Logical partitions aren't split between physical partitions.

+As with HBase RowKey, partition key design is important for Azure Cosmos DB. HBase's Row Key works by sorting data and storing continuous data, and Azure Cosmos DB's Partition Key is a different mechanism because it hash-distributes data. Assuming your application using HBase is optimized for data access patterns to HBase, using the same RowKey for the partition Key won't give good performance results. Given that it's sorted data on HBase, the [Azure Cosmos DB composite index](../index-policy.md#composite-indexes) may be useful. It's required if you want to use the ORDER BY clause in more than one field. You can also improve the performance of many equal and range queries by defining a composite index.

+Azure Cosmos DB doesn't require any configuration such as cluster component redundancy. It provides a comprehensive SLA for high availability, consistency, and latency. See [SLA for Azure Cosmos DB](https://azure.microsoft.com/support/legal/sla/cosmos-db/v1_3/) for more detail.

+### Offline and online migration

+When performing offline data migration, it depends on the version of HBase you're currently running and the tools available. See the [Data Migration](#migrate-your-data) section for more details.

+If you're using Apache Phoenix on HBase cluster, you need to collect data from Phoenix as well.

+| Custom tool with Azure Cosmos DB bulk executor library | All versions | Most flexible to create custom data migration tools using libraries. Requires more effort to setup. |

+The mappings for code migration are shown here, but the HBase RowKeys and Azure Cosmos DB Partition Keys used in these examples aren't always well designed. Design according to the actual data model of the migration source.

+Azure Cosmos DB provides type safety via data model. We use data model named ΓÇÿFamilyΓÇÖ.

+In Azure Cosmos DB, updates are treated as Upsert operations. That is, if the document doesn't exist, it will be inserted.

+HBase offers several server-side programming features. If you're using these features, you will also need to migrate their processing.

+ * Endpoint is a feature for extending HBase RPC. It's a function similar to an RDBMS stored procedure.

+Data security is a shared responsibility of the customer and the database provider. For on-premises solutions, customers have to provide everything from endpoint protection to physical hardware security, which is not an easy task. If you choose a PaaS cloud database provider such as Azure Cosmos DB, customer involvement will be reduced. Azure Cosmos DB runs on the Azure platform, so it can be enhanced in a different way than HBase. Azure Cosmos DB doesn't require any extra components to be installed for security. We recommend that you consider migrating your database system security implementation using the following checklist:

+| Automatic data backups | There's no automatic backup function. You need to implement data backup yourself. | Azure Cosmos DB is backed up regularly and stored in the geo redundant storage. |

+| Protect and isolate sensitive data | For example, if you're using Apache Ranger, you can use Ranger policy to apply the policy to the table. | You can separate personal and other sensitive data into specific containers and read / write, or limit read-only access to specific users. |

+Azure Cosmos DB automatically backs up data at periodic intervals, which doesn't affect the performance or availability of database operations. Backups are stored in Azure storage and can be used to recover data if needed. There are two types of Azure Cosmos DB backups:

+|**Sample with Azure App Service** | [How to use Spring and Azure Cosmos DB with App Service on Linux](/azure/developer/java/spring-framework/configure-spring-app-with-cosmos-db-on-app-service-linux) |

+Azure Cosmos DB traces follow the [OpenTelemetry database specification](https://github.com/open-telemetry/opentelemetry-specification) and also provide several custom attributes. You may see different attributes depending on the operation of your request, and these attributes are core attributes for all requests.

+[Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/), is a fully managed cloud service that lets you create workflows using algorithms in a simple drag-and-drop interface, code your own algorithms in [R](https://en.wikipedia.org/wiki/R_\(programming_language\)), or use some of the already-built and ready to use APIs such as: [Text Analytics](https://gallery.cortanaanalytics.com/MachineLearningAPI/Text-Analytics-2), Content Moderator, or [Recommendations](https://gallery.azure.ai/Solution/Recommendations-Solution).

+The ultimate goal of unit economics, as a derivative of activity-based costing methodology, is to factor in the whole picture of your business's cost. This article focuses on capturing how you can factor your Microsoft Cloud costs into those efforts. As your FinOps practice matures, consider the manual processes and steps outside of the cloud that might be important for calculating units that are critical for your business to track the most accurate cost per unit.

+ - If you don't see the data you need in metrics, also check [logs and traces in Azure Monitor](../../azure-monitor/overview.md#data-platform). It might not be a direct correlation to usage but might be able to give you some indication of usage.

+- **Integration runtimes and sharing**. Integration runtimes don't change often and are similar across all stages in your CI/CD. So Data Factory expects you to have the same name, type and sub-type of integration runtime across all stages of CI/CD. If you want to share integration runtimes across all stages, consider using a ternary factory just to contain the shared integration runtimes. You can use this shared factory in all of your environments as a linked integration runtime type.

+description: Learn about self-hosted integration runtime auto-update and expire notification.

+This article describes how to let self-hosted integration runtime auto-update to the latest version and how Azure Data Factory (ADF) manages the versions of self-hosted integration runtime.

+You can check the version either in your self-hosted integration runtime client or in the ADF portal:

+Generally, when you install a self-hosted integration runtime in your local machine or an Azure VM, you have two options to manage the version of self-hosted integration runtime: auto-update or maintain manually. Typically, ADF releases two new versions of self-hosted integration runtime every month, which includes new features released, bugs fixed, and enhancements. So we recommend users to update to the latest version.

+The most convenient way is to enable auto-update when you create or edit self-hosted integration runtime. The self-hosted integration runtime is automatically update to newer version. You can also schedule the update at the most suitable time slot as you wish.

+To ensure the stability of self-hosted integration runtime, although we release two versions, we'll only push one version every month. So sometimes you find that the auto-updated version is the previous version of the actual latest version. If you want to get the latest version, you can go to [download center](https://www.microsoft.com/download/details.aspx?id=39717) and do so manually. Additionally, **auto-update** to a new version is managed internally. You can't change it.

+The self-hosted integration runtime **Auto update** page in the ADF portal shows the newer version if current version is old. When your self-hosted integration runtime is online, this version is auto-update version and will automatically update your self-hosted integration runtime in the scheduled time. But if your self-hosted integration runtime is offline, the page only shows the latest version.

+If you have multiple nodes, and for some reasons that some of them aren't auto-updated successfully. Then these nodes roll back to the version, which was the same across all nodes before the auto-update.

+If you want to manually control which version of self-hosted integration runtime, you can disable the setting of auto-update and install it manually. Each version of self-hosted integration runtime expires in one year. The expiring message is shown in ADF portal and self-hosted integration runtime client **90 days** before expiration.

+To apply the 2309 update, your device must be running version 2303 or later.

+ - You can update to 2303 from 2207 or later, and then install 2309.

+|**2.**|Wi-Fi |Starting this release, Wi-Fi functionality for Azure Stack Edge Mini R has been deprecated. | |

+az dedicated-hsm update --resource-group myRG ΓÇô-name hsm1

+az dedicated-hsm delete --resource-group myRG ΓÇô-name hsm1

+* [Monitoring](monitoring.md)

+|(Preview) Azure API Management APIs should be onboarded to Defender for APIs. | Onboarding APIs to Defender for APIs requires compute and memory utilization on the Azure API Management service. Monitor performance of your Azure API Management service while onboarding APIs, and scale out your Azure API Management resources as needed.|High|

+|(Preview) API endpoints that are unused should be disabled and removed from the Azure API Management service|As a security best practice, API endpoints that haven't received traffic for 30 days are considered unused, and should be removed from the Azure API Management service. Keeping unused API endpoints might pose a security risk. These might be APIs that should have been deprecated from the Azure API Management service, but have accidentally been left active. Such APIs typically do not receive the most up-to-date security coverage.|Low|

+|(Preview) API endpoints in Azure API Management should be authenticated|API endpoints published within Azure API Management should enforce authentication to help minimize security risk. Authentication mechanisms are sometimes implemented incorrectly or are missing. This allows attackers to exploit implementation flaws and to access data. For APIs published in Azure API Management, this recommendation assesses authentication though verifying the presence of Azure API Management subscription keys for APIs or products where subscription is required, and the execution of policies for validating [JWT](/azure/api-management/validate-jwt-policy), [client certificates](/azure/api-management/validate-client-certificate-policy), and [Microsoft Entra](/azure/api-management/validate-azure-ad-token-policy) tokens. If none of these authentication mechanisms are executed during the API call, the API will receive this recommendation.|High|

+<!-- Please don't adjust this next line without getting approval from the Defender for Cloud documentation team. It is necessary for proper RSS functionality. -->

+| [Changes to how Microsoft Defender for Cloud's costs are presented in Microsoft Cost Management](#changes-to-how-microsoft-defender-for-clouds-costs-are-presented-in-microsoft-cost-management) | October 25, 2023 | November 2023 |

+## Changes to how Microsoft Defender for Cloud's costs are presented in Microsoft Cost Management

+**Annoucement date: October 26, 2023**

+**Estimated date for change: November 2023**

+In November there will be a change as to how Microsoft Defender for Cloud's costs are presented in Cost Management and in Subscriptions invoices.

+Costs will be presented for each protected resource instead of as an aggregation of all resources on the subscription.

+If a resource has a tag applied, which are often used by organizations to perform financial chargeback processes, it will be added to the appropriate billing lines.

+[Azure for Students](https://azure.microsoft.com/free/students/) is a program offered by Microsoft Azure to provide students at Higher Education Institutions with access to various Azure cloud services and resources. Joining Azure for Students will grant you $100 in Azure credits with no credit card required. This credit will be valid for one year and you can renew each year you're an active student to get an addition $100. You also receive access to a set of free services you can deploy once you join. Select versions of Azure Virtual Machines, Azure SQL Databases, Azure Blob Storage and more are all free with your subscription.

+Azure for Students is available only to students who meet the following requirements:

+* You must affirm that you're age 18 or older and attend an accredited, degree-granting, two-year or four-year educational institution where youΓÇÖre a full-time student.

+* You must verify your academic status through your institutionΓÇÖs email address.

+* See the Azure for Students Offer for detailed terms of use.

+* This offer isn't available for use in a massive open online course (MOOC) or in other professional trainings from for-profit organizations.

+This offer is limited to one Azure for Student subscription per eligible customer. It's nontransferable and can't be combined with any other offer, unless otherwise permitted by Microsoft.

+You need to verify yourself as an active student to become eligible for the offer. This verification process takes place as part of the signup flow for Azure for Students here.

+ Title: Azure Event GridΓÇÖs MQTT broker feature - Monitor data reference

+description: This article provides reference documentation for metrics and diagnostic logs for Azure Event GridΓÇÖs MQTT broker feature.

+# Monitor data reference for Azure Event GridΓÇÖs MQTT broker feature (Preview)

+This article provides a reference of log and metric data collected to analyze the performance and availability of MQTT broker.

+| MQTT.Throughput | MQTT: Throughput | Count | Total | The total bytes published to or delivered by the namespace. This metric includes all the MQTT packets that your MQTT clients send to the MQTT broker, regardless of their success. | Direction |

+| MQTT.SuccessfulSubscriptionOperations | MQTT: Successful Subscription Operations | Count | Total | The number of successful subscription operations (Subscribe, Unsubscribe). This metric is incremented for every topic filter within your subscription request that gets accepted by MQTT broker. | OperationType, Protocol |

+| MQTT.FailedSubscriptionOperations | MQTT: Failed Subscription Operations | Count | Total | The number of failed subscription operations (Subscribe, Unsubscribe). This metric is incremented for every topic filter within your subscription request that gets rejected by MQTT broker. | OperationType, Protocol, Error |

+| MQTT.Connections | MQTT: Active Connections | Count | Total | The number of active connections in the namespace. The value for this metric is a point-in-time value. Connections that were active immediately after that point-in-time might not be reflected in the metric. | Protocol |

+| Mqtt.DroppedSessions | MQTT: Dropped Sessions | Count | Total | The number of dropped sessions in the namespace. The value for this metric is a point-in-time value. Sessions that were dropped immediately after that point-in-time might not be reflected in the metric. | DropReason |

+| OperationType | The type of the operation. The available values include: <br><br>- Publish: PUBLISH requests sent from MQTT clients to MQTT broker. <br>- Deliver: PUBLISH requests sent from MQTT broker to MQTT clients. <br>- Subscribe: SUBSCRIBE requests by MQTT clients. <br>- Unsubscribe: UNSUBSCRIBE requests by MQTT clients. <br>- Connect: CONNECT requests by MQTT clients. |

+| Error | Error occurred during the operation. The available values include: <br><br>-QuotaExceeded: the client exceeded one or more of the throttling limits that resulted in a failure <br>- AuthenticationError: a failure because of any authentication reasons. In case of failed MQTT routing messages, the EventGrid Data Sender role for the custom topic configured as the destination for MQTT routed messages was deleted. This error doesn't apply for namespace topics since they don't need a permission to route MQTT messages. In that case for MQTT message routing, MQTT broker drops the MQTT message that was meant to be routed. <br>- AuthorizationError: a failure because of any authorization reasons.<br>- ClientError: the client sent a bad request or used one of the unsupported features that resulted in a failure. <br>-TopicNotFoundError: The custom topic that is configured to receive all the MQTT routed messages was deleted. This error doesn't apply for namespace topics since they can't be deleted if they're used as the destination for MQTT routed messages. In that case, MQTT broker drops the MQTT message that was meant to be routed.<br>-TooManyRequests: the number of MQTT routed messages per second exceeds the limit of the destination (namespace topic or custom topic) for MQTT routed messages. In that case, Event Grid retries to route the MQTT message. <br>- ServiceError: a failure because of an unexpected server error or for a server's operational reason. In that case for MQTT message routing, Event Grid retries to route the MQTT message. |

+| Direction | The direction of the operation. The available values are: <br><br>- Inbound: inbound throughput to MQTT broker. <br>- Outbound: outbound throughput from MQTT broker. |

+description: 'Describes the main concepts for access control for MQTT clients in Azure Event GridΓÇÖs MQTT broker feature.'

+Access control enables you to manage the authorization of clients to publish or subscribe to topics, using a role-based access control model. Given the enormous scale of IoT environments, assigning permission for each client to each topic is incredibly tedious. Azure Event GridΓÇÖs MQTT broker feature tackles this scale challenge through grouping clients and topics into client groups and topic spaces.

+The variables represent either client authentication names or client attributes. During communication with MQTT broker, each client would replace the variable in the MQTT topic with a substituted value. For example, the variable ${client.authenticationName} would be replaced with the authentication name of each client: machine1, machine2, etc. MQTT broker would allow access only to the clients that have a substituted value that matches either their authentication name or the value of the specified attribute.

+1. The vehicle publishes telemetry and events messages through an MQTT client with defined topics to the **Azure Event GridΓÇÖs MQTT broker feature** in the *vehicle messaging services*.

+1. The factory system triggers the **device provisioning client** to connect to the *device registration* and provision the device. The device retrieves connection information to the *MQTT broker*.

+1. The *device registration* application creates the device identity with MQTT broker.

+1. The factory system triggers the device to establish a connection to the *MQTT broker* for the first time.

+1. The *MQTT broker* manages authorization for allowed topics using the local registry.

+1. The vehicle publishes a message to the *MQTT broker*.

+ 1. Messages that don't require more processing are published to the *MQTT broker*.

+description: 'Describes how MQTT clients are authenticated and mTLS connection is established when a client connects to Azure Event GridΓÇÖs MQTT broker feature.'

+To establish a secure connection with MQTT broker, you can use either MQTTS over port 8883 or MQTT over web sockets on port 443. It's important to note that only secure connections are supported. The following steps are to establish secure connection prior to the client authentication.

+1. The client initiates the handshake with MQTT broker. It sends a hello packet with supported TLS version, cipher suites.

+ Title: 'Overview of Azure Event GridΓÇÖs MQTT broker feature (preview)'

+description: 'Describes the main concepts for the Azure Event GridΓÇÖs MQTT broker feature.'

+# Overview of the Azure Event GridΓÇÖs MQTT broker feature (Preview)

+Azure Event Grid enables your MQTT clients to communicate with each other and with Azure services, to support your Internet of Things (IoT) solutions.

+Azure Event GridΓÇÖs MQTT broker feature enables you to accomplish the following scenarios:

+The MQTT broker is ideal for the implementation of automotive and mobility scenarios, among others. See [the reference architecture](mqtt-automotive-connectivity-and-data-solution.md) to learn how to build secure and scalable solutions for connecting millions of vehicles to the cloud, using AzureΓÇÖs messaging and data analytics services.

+The following are a list of key concepts involved in Azure Event GridΓÇÖs MQTT broker feature.

+MQTT is a publish-subscribe messaging transport protocol that was designed for constrained environments. It has become the go-to communication standard for IoT scenarios due to efficiency, scalability, and reliability. MQTT broker enables clients to publish and subscribe to messages over MQTT v3.1.1, MQTT v3.1.1 over WebSockets, MQTT v5, and MQTT v5 over WebSockets protocols. The following list shows some of the feature highlights of MQTT broker:

+ - **Message expiry interval** allows you to declare to MQTT broker when to disregard a message that is no longer relevant or valid. For example, disregard stale commands or alerts.

+- MQTT broker is adding more MQTT v5 features in the future to align more with the MQTT specifications. The following items detail the current differences between features supported by MQTT broker and the MQTT v5 specifications: Will message, Retain flag, Message ordering and QoS 2 aren't supported.

+- MQTT broker is adding more MQTT v3.1.1 features in the future to align more with the MQTT specifications. The following items detail the current differences between features supported by MQTT broker and the MQTT v3.1.1 specification: Will message, Retain flag, Message ordering and QoS 2 aren't supported.

+[Learn more about the MQTT broker and current limitations.](mqtt-support.md)

+- The many-to-one messaging pattern enables clients to offload the burden of managing the high number of connections to MQTT broker.

+Event Grid Namespace is a management container for the resources supporting the MQTT broker functionality, along with the resources supporting the [pull delivery functionality](pull-delivery-overview.md). Your MQTT client can connect to MQTT broker and publish/subscribe to messages, while MQTT broker authenticates your clients, authorizes publish/subscribe requests, and forwards messages to interested clients. Learn more about [the namespace concept.](mqtt-event-grid-namespace-terminology.md)

+Event Grid has a client registry that stores information about the clients permitted to connect to it. Before a client can connect, there must be an entry for that client in the client registry. As a client connects to MQTT broker, it needs to authenticate with MQTT broker based on credentials stored in the identity registry. MQTT broker supports X.509 certificate authentication that is the industry authentication standard in IoT devices and [Microsoft Entra](mqtt-client-azure-ad-token-and-rbac.md) that is Azure's authentication standard for applications.[Learn more about MQTT client authentication.](mqtt-client-authentication.md)

+Use the following articles to learn more about the MQTT broker and its main concepts.

+description: 'Quickstart guide to use Azure Event GridΓÇÖs MQTT broker feature and Azure CLI to publish and subscribe MQTT messages on a topic'

+Azure Event GridΓÇÖs MQTT broker feature supports messaging using the MQTT protocol. Clients (both devices and cloud applications) can publish and subscribe MQTT messages over flexible hierarchical topics for scenarios such as high scale broadcast, and command & control.

+description: 'Quickstart guide to use Azure Event GridΓÇÖs MQTT broker feature and Azure portal to publish and subscribe MQTT messages on a topic.'

+1. Create an Event Grid namespace and enable MQTT

+1. After the deployment succeeds, select **Go to resource** to navigate to the Event Grid Namespace Overview page for your namespace.

+1. Select **Connect** to connect the client to the MQTT broker.

+## MQTT message routing behavior

+While routing MQTT messages to namespace topics or custom topics, Event Grid provides durable delivery as it tries to deliver each message **at least once** immediately. If there's a failure, Event Grid either retries delivery or drops the message that was meant to be routed. Event Grid doesn't guarantee order for event delivery, so subscribers might receive them out of order.

+The following table describes the behavior of MQTT message routing based on different errors.

+| Error| Error description | Behavior |

+| --| --|--|

+| TopicNotFoundError | The custom topic that is configured to receive all the MQTT routed messages was deleted. This error doesn't apply for namespace topics since they can't be deleted if they're used as the destination for MQTT routed messages. | Event Grid drops the MQTT message that was meant to be routed.|

+| AuthenticationError | The EventGrid Data Sender role for the custom topic configured as the destination for MQTT routed messages was deleted. This error doesn't apply for namespace topics since they don't need a permission to route MQTT messages. | Event Grid drops the MQTT message that was meant to be routed.|

+| TooManyRequests | The number of MQTT routed messages per second exceeds the limit of the destination (namespace topic or custom topic) for MQTT routed messages. | Event Grid retries to route the MQTT message.|

+| ServiceError | An unexpected server error for a server's operational reason. | Event Grid retries to route the MQTT message.|

+

+During retries, Event Grid uses an exponential backoff retry policy for MQTT message routing. Event Grid retries delivery on the following schedule on a best effort basis:

+- 10 seconds

+- 30 seconds

+- 1 minute

+- 5 minutes

+- 10 minutes

+- 30 minutes

+- 1 hour

+- 3 hours

+- 6 hours

+- Every 12 hours

+If a routed MQTT message that was queued for redelivery succeeded, Event Grid attempts to remove the message from the retry queue on a best effort basis, but duplicates might still be received.

+ Title: 'MQTT Features Support by Azure Event GridΓÇÖs MQTT broker feature'

+description: 'Describes the MQTT features supported by Azure Event GridΓÇÖs MQTT broker feature.'

+# MQTT features supported by Azure Event GridΓÇÖs MQTT broker feature

+MQTT is a publish-subscribe messaging transport protocol that was designed for constrained environments. ItΓÇÖs efficient, scalable, and reliable, which made it the gold standard for communication in IoT scenarios. MQTT broker supports clients that publish and subscribe to messages over MQTT v3.1.1, MQTT v3.1.1 over WebSockets, MQTT v5, and MQTT v5 over WebSockets. MQTT broker also supports cross MQTT version (MQTT 3.1.1 and MQTT 5) communication.

+While connecting to MQTT broker, use the following ports during communication over MQTT:

+Multi-session support enables your application MQTT clients to have more scalable and reliable implementation by connecting to MQTT broker with multiple active sessions at the same time.

+- Provide the Username property in the CONNECT packet to signify your client authentication name.

+For example, the following combinations of Username and ClientIds in the CONNECT packet enable the client "Mgmt-application" to connect to MQTT broker over three independent sessions:

+- If a client tries to take over another client's active session by presenting its session name with a different authentication name, its connection request is rejected with an unauthorized error. For example, if Client B tries to connect to session 123 that is assigned at that time for client A, Client B's connection request is rejected. That being said, if the same client tries to reconnect with the same session names and the same authentication name, it is able to take over its existing session.

+- If a client resource is deleted without ending its session, other clients can't use its session name until the session expires. For example, If client B creates a session with session name 123 then client B gets deleted, client A can't connect to session 123 until it expires.

+- The limit for the number of sessions per client applies to online and offline sessions at any point in time. For example, consider a namespace with the maximum client sessions per authentication name is set to 1. If client A connects with a persistent session 123 then gets disconnected, client A won't be able to connect with a new session 456 since its session 123 is still active even if it's offline. Accordingly, we recommend that the same client always reconnects with the same static session names as opposed to generating a new session name with every reconnect.

+Azure Event GridΓÇÖs MQTT broker feature supports the following MQTT features:

+MQTT broker supports QoS 0 and 1, which define the guarantee of message delivery on PUBLISH and SUBSCRIBE packets between clients and MQTT broker. QoS 0 guarantees at-most-once delivery; messages with QoS 0 arenΓÇÖt acknowledged by the subscriber nor get retransmitted by the publisher. QoS 1 guarantees at-least-once delivery; messages are acknowledged by the subscriber and get retransmitted by the publisher if they didnΓÇÖt get acknowledged. QoS enables your clients to control the efficiency and reliability of the communication.

+MQTT broker supports persistent sessions for MQTT v3.1.1 such that MQTT broker preserves information about a clientΓÇÖs session in case of disconnections to ensure reliability of the communication. This information includes the clientΓÇÖs subscriptions and missed/ unacknowledged QoS 1 messages. Clients can configure a persistent session through setting the cleanSession flag in the CONNECT packet to false.

+MQTT v5 has introduced the clean start and session expiry features as an improvement over MQTT v3.1.1 in handling session persistence. Clean Start is a feature that allows a client to start a new session with MQTT broker, discarding any previous session data. Session Expiry allows a client to inform MQTT broker when an inactive session is considered expired and automatically removed. In the CONNECT packet, a client can set Clean Start flag to true and/or short session expiry interval for security reasons or to avoid any potential data conflicts that might have occurred during the previous session. A client can also set a clean start to false and/or long session expiry interval to ensure the reliability and efficiency of persistent sessions.

+MQTT broker maintains a queue of messages for each active MQTT session that isn't connected, until the client connects with MQTT broker again to receive the messages in the queue. If a client doesn't connect to receive the queued QOS1 messages, the session queue starts accumulating the messages until it reaches its limit: 100 messages or 1 MB. Once the queue reaches its limit during the lifespan of the session, the session is terminated.

+MQTT broker supports user properties on MQTT v5 PUBLISH packets that allow you to add custom key-value pairs in the message header to provide more context about the message. The use cases for user properties are versatile. You can use this feature to include the purpose or origin of the message so the receiver can handle the message without parsing the payload, saving computing resources. For example, a message with a user property indicating its purpose as a "warning" could trigger different handling logic than one with the purpose of "information."

+In MQTT v5, message expiry interval allows messages to have a configurable lifespan. The message expiry interval is defined as the time interval between the time a message is published to MQTT broker and the time when the MQTT broker needs to discard the message if it hasn't been delivered. This feature is useful in scenarios where messages are only valid for a certain amount of time, such as time-sensitive commands, real-time data streaming, or security alerts. By setting a message expiry interval, MQTT broker can automatically remove outdated messages, ensuring that only relevant information is available to subscribers. If a message's expiry interval is set to zero, it means the message should never expire.

+In MQTT v5, topic aliases allow a client to use a shorter alias in place of the full topic name in the published message. MQTT broker maintains a mapping between the topic alias and the actual topic name. This feature can save network bandwidth and reduce the size of the message header, particularly for topics with long names. It's useful in scenarios where the same topic is repeatedly published in multiple messages, such as in sensor networks. MQTT broker supports up to 10 topic aliases. A client can use a Topic Alias field in the PUBLISH packet to replace the full topic name with the corresponding alias.

+In MQTT v5, flow control refers to the mechanism for managing the rate and size of messages that a client can handle. Flow control can be configured by setting the Maximum Packet Size and Receive Maximum parameters in the CONNECT packet. The Receive Maximum parameter allows the client to limit the number of messages sent by the broker to the number of messages that the client is able to handle. The Maximum Packet Size parameter defines the maximum size of packets that the client can receive. MQTT broker has a message size limit of 512 KiB. This feature ensures reliability and stability of the communication for constrained devices with limited processing speed or storage capabilities.

+For MQTT v5, MQTT broker is able to send negative acknowledgments (NACKs) and server-initiated disconnect packets that provide the client with more information about failures for message delivery or connection. These features help the client diagnose the reason behind a failure and take appropriate mitigating actions. MQTT broker uses the reason codes that are defined in the [MQTT v5 Specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html)

+MQTT broker is adding more MQTT v5 and MQTT v3.1.1 features in the future to align more with the MQTT specifications. The following list details the current differences between features supported by the MQTT broker and the MQTT specifications:

+Learn more about MQTT broker:

+# Topic Spaces in Azure Event GridΓÇÖs MQTT broker feature

+Topic spaces are used to simplify access control management by enabling you to grant publish or subscribe access to a group of topics at once instead of managing access for each individual topic. To publish or subscribe to any MQTT topic, you need to:

+MQTT broker supports all the MQTT wildcards defined by the [MQTT specification](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html) as follows:

+For more information about wildcards, see [Topic Wildcards in the MQTT spec](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html).

+- Topics that start with $ are reserved for internal use.

+ Title: Azure Event GridΓÇÖs MQTT broker feature - Troubleshooting guide

+description: This article provides guidance on how to troubleshoot MQTT broker related issues.

+# Guide to troubleshoot issues with Azure Event GridΓÇÖs MQTT broker feature

+This guide provides you with information on what you can do to troubleshoot things before you submit a support ticket.

+**Data distribution using push and pull delivery (preview) modes**. At any point in a data pipeline, HTTP applications can consume messages using push or pull APIs. The source of the data might include MQTT clientsΓÇÖ data, but also includes the following data sources that send their events over HTTP:

+- **X.509 certificate [authentication](mqtt-client-authentication.md)** - authenticate your devices using the IoT industry's standard mechanism for authentication.

+- **[AAD authentication](mqtt-client-azure-ad-token-and-rbac.md)** - authenticate your applications using the Azure's standard mechanism for authentication.

+A multitenant SaaS provider or platform can publish their events to Event Grid through a feature called [Partner Events](partner-events-overview.md). You can [subscribe to those events](subscribe-to-partner-events.md) and automate tasks, for example. Events from the following partners are currently available:

+- [Microsoft Graph API](subscribe-to-graph-api-events.md). Through Microsoft Graph API you can get events from [Azure AD](azure-active-directory-events.md), [Microsoft Outlook](outlook-events.md), [Teams](teams-events.md), Conversations, security alerts, and Universal Print.

+Azure Event Grid uses a pay-per-event pricing model. You only pay for what you use. For the push-style delivery that is generally available, the first 100,000 operations per month are free. Examples of operations include event publication, event delivery, delivery attempts, event filter evaluations that refer to event data properties (sometimes referred to as Advanced Filters), and events sent to a dead letter location. For details, see the [pricing page](https://azure.microsoft.com/pricing/details/event-grid/).

+Event Grid operations involving Namespaces and its resources, including MQTT and pull HTTP delivery operations, are in public preview and are available at no charge today.

+- [Pull delivery overview](pull-delivery-overview.md).

+- [Push delivery overview](push-delivery-overview.md).

+- Quickstart: [Publish and subscribe to app events using namespace topics](publish-events-using-namespace-topics.md).

+see [Understand applicability in Azure Policy](./policy-applicability.md). Azure Policy exemptions work with

+[Resource Manager modes](./definition-structure.md#resource-manager-modes), Microsoft.Kubneretes.Data, Microsoft.KeyVault.Data and Microsoft.Network.Data and don't work with the other

+A policy exemption is created as a child object on the resource hierarchy or the individual resource granted the exemption. Exemptions cannot be created at the Resource Provider mode component level.

+If the parent resource to which the exemption applies is removed, then the exemption is removed as well.

+( `policyDefinitionReferenceId`: `requiredTags`) and the **Allowed locations** built-in policy definition ( `policyDefinitionReferenceId` : `allowedLocations`):

+information. In the example, properties **requestedBy**, **approvedBy**, **approvedOn**, and

+Exemptions support an optional property `resourceSelectors`. This property works the same way in exemptions as it does in assignments, allowing for gradual rollout or rollback of an _exemption_ to certain subsets of resources in a controlled manner based on resource type, resource location, or whether the resource has a location. More details about how to use resource selectors can be found in the [assignment structure](assignment-structure.md#resource-selectors-preview). Here is an example exemption JSON, which uses resource selectors. In this example, only resources in `westcentralus` will be exempt from the policy assignment:

+Regions can be added or removed from the `resourceLocation` list in the example. Resource selectors allow for greater flexibility of where and how exemptions can be created and managed.

+## Exemption creation and management

+Exemptions are recommended for time-bound or specific scenarios where a resource or resource hierarchy should still be tracked and would otherwise be evaluated, but there's a specific reason it shouldn't be assessed for compliance. For example, if an environment has the built-in definition `Storage accounts should disable public network access` (ID: `b2982f36-99f2-4db5-8eff-283140c09693`) assigned with _effect_ set to _audit_. Upon compliance assessment, resource "StorageAcc1" is non-compliant, but StorageAcc1 must have public network access enable for business purposes. At that time, a request should be submitted to create an exemption resource that targets StorageAcc1. Once the exemption is created, StorageAcc1 will be shown as _exempt_ in compliance review.

+Regularly revisit your exemptions to ensure that all eligible items are appropriately exempted and promptly remove any no longer qualifying for exemption. At that time, exemption resources that have expired could be deleted as well.

+- Leverage [Azure Resource Graph queries on exemptions](../samples/resource-graph-samples.md#azure-policy-exemptions).

+- Learn about [the difference between exclusions and exemptions](./scope.md#scope-comparison).

+## Resource manager modes

+### -IfNotExists policy effects

+The applicability of `AuditIfNotExists` and `DeployIfNotExists` policies is based off the entire `if` condition of the policy rule. When the `if` evaluates to false, the policy isn't applicable.

+### All other policy effects

+## Resource provider modes

+### Microsoft.KeyVault.Data, Microsoft.ManagedHSM.Data, and Microsoft.DataFactory.Data

+## Not Applicable Resources

+There could be situations in which resources are applicable to an assignment based on conditions or scope, but they shouldn't be applicable due to business reasons. At that time, it would be best to apply [exclusions](./assignment-structure.md#excluded-scopes) or [exemptions](./exemption-structure.md). To learn more on when to use either, review [scope comparison](./scope.md#scope-comparison)

+> [!NOTE]

+> By design, Azure Policy does not evaluate resources under the `Microsoft.Resources` resource provider (RP) from

+policy evaluation, except for subscriptions and resource groups.

+- Learn how to [mark resources as not applicable](./assignment-structure.md#excluded-scopes).

+- Lean more on [applicability limitations](https://github.com/azure/azure-policy#known-issues)

+description: When a resource is non-compliant, there are many possible reasons. Discover what caused the non-compliance with the policy.

+understand which change altered a previously compliant resource to make it non-compliant. There are

+- [Change history (Preview)](#change-history-preview)

+- Resource details such as name, type, location, and resource ID.

+- Compliance state and timestamp of the last evaluation for the current policy assignment.

+- A list of reasons for the resource non-compliance.

+> the _Non-compliant_ resource is `Microsoft.Compute/virtualMachines` then the user must have the

+> `Microsoft.Compute/virtualMachines/read` operation. If the user doesn't have the needed

+1. From the **Resource compliance** tab of the **Policy compliance** page, select and hold (or

+ right-click) or select the ellipsis of a resource in a **compliance state** that's

+ :::image type="content" source="../media/determine-non-compliance/view-compliance-details.png" alt-text="Screenshot of the View compliance details link on the Resource compliance tab." :::

+ to the current policy assignment. In this example, the field `Microsoft.Sql/servers/version` is

+ :::image type="content" source="../media/determine-non-compliance/compliance-details-pane.png" alt-text="Screenshot of the Compliance details pane and reasons for non-compliance that current value is 12 and target value is 14." :::

+ For an `auditIfNotExists` or `deployIfNotExists` policy definition, the details include the

+ Example partial `deployIfNotExists` definition:

+ "if": {

+ "field": "type",

+ "equals": "[parameters('resourceType')]"

+ },

+ "then": {

+ "effect": "deployIfNotExists",

+ "details": {

+ "type": "Microsoft.Insights/metricAlerts",

+ "existenceCondition": {

+ "field": "name",

+ "equals": "[concat(parameters('alertNamePrefix'), '-', resourcegroup().name, '-', field('name'))]"

+ },

+ "existenceScope": "subscription",

+ "deployment": {

+ ...

+ }

+ }

+ :::image type="content" source="../media/determine-non-compliance/compliance-details-pane-existence.png" alt-text="Screenshot of Compliance details pane for ifNotExists including evaluated resource count." :::

+history (Preview)](#change-history-preview).

+| No related resources match the effect details in the policy definition. | A resource of the type defined in `then.details.type` and related to the resource defined in the **if** portion of the policy rule doesn't exist. |

+| NonModifiablePolicyAlias | NonModifiableAliasConflict: The alias '{alias}' isn't modifiable in requests using API version '{apiVersion}'. This error happens when a request using an API version where the alias doesn't support the 'modify' effect or only supports the 'modify' effect with a different token type. |

+| AppendPoliciesNotApplicable | AppendPoliciesUnableToAppend: The aliases: '{ aliases }' aren't modifiable in requests using API version: '{ apiVersion }'. This can happen in requests using API versions for which the aliases don't support the 'modify' effect, or support the 'modify' effect with a different token type. |

+| ConflictingAppendPolicies | ConflictingAppendPolicies: Found conflicting policy assignments that modify the '{notApplicableFields}' field. Policy identifiers: '{policy}'. Contact the subscription administrator to update the policy assignments. |

+| AppendPoliciesFieldsExist | AppendPoliciesFieldsExistWithDifferentValues: Policy assignments attempted to append fields which already exist in the request with different values. Fields: '{existingFields}'. Policy identifiers: '{policy}'. Contact the subscription administrator to update the policies. |

+| AppendPoliciesUndefinedFields | AppendPoliciesUndefinedFields: Found policy definition that refers to an undefined field property for API version '{apiVersion}'. Fields: '{nonExistingFields}'. Policy identifiers: '{policy}'. Contact the subscription administrator to update the policies. |

+| MissingRegistrationForType | MissingRegistrationForResourceType: The subscription isn't registered for the resource type '{ResourceType}'. Check that the resource type exists and that the resource type is registered. |

+| InvalidResourceNameWildcardPosition | The policy assignment '{0}' associated with the policy definition '{1}' couldn't be evaluated. The resource name '{2}' within an ifNotExists condition contains the wildcard '?' character in an invalid position. Wildcards can only be located at the end of the name in a segment by themselves (ex. TopLevelResourceName/?). Either fix the policy or remove the policy assignment to unblock. |

+| TooManyResourceNameSegments | The policy assignment '{0}' associated with the policy definition '{1}' couldn't be evaluated. The resource name '{2}' within an ifNotExists condition contains too many name segments. The number of name segments must be equal to or less than the number of type segments (excluding the resource provider namespace). Either fix the policy definition or remove the policy assignment to unblock. |

+| InvalidPolicyFieldPath | The field path '{0}' within the policy definition is invalid. Field paths must contain no empty segments. They might contain only alphanumeric characters with the exception of the '.' character for splitting segments and the '[*]' character sequence to access array properties. |

+| ConstraintStatusStale | There's an existing 'Audit' status, but Gatekeeper hasn't performed an audit within the last hour. |

+| ConstraintNotProcessed | There's no status and Gatekeeper hasn't performed an audit within the last hour. |

+| InvalidConstraint/Template | The resource was rejected because of one of the following reasons: invalid constraint template Rego content, invalid YAML, or a parameter type mismatch between constraint and constraint template (providing a string value when an integer was expected). |

+_Non-compliant_ resource to open a deeper view. The **Component Compliance** tab shows more information specific to the Resource Provider mode on the assigned policy with the

+settings evaluated inside the virtual machine and you need to view per-setting details. For

+_Non-compliant_, you need to know which specific settings are out of compliance and why.

+Begin by following the same steps in the [Compliance details](#compliance-details) section to view policy compliance details.

+[Azure Automanage](../../../automanage/index.yml)

+information, select each assignment using the link in the column **Name**.

+## Change history (Preview)

+1. From the **Resource compliance** tab of the **Policy compliance** page, select a resource.

+ :::image type="content" source="../media/determine-non-compliance/change-history-tab.png" alt-text="Screenshot of the Change History tab and detected change times on Resource Compliance page." :::

+ :::image type="content" source="../media/determine-non-compliance/change-history-visual-diff.png" alt-text="Screenshot of the Change History Visual Diff of the before and after state of properties on the Change history page." :::

+The _visual diff_ aides in identifying changes to a resource. The changes detected might not be

+> and Azure Policy's [Change history](../policy/how-to/determine-non-compliance.md#change-history-preview)

+because you get resource groups that you can access, without any indication that the result might be

+HDInsight on AKS is a modern, reliable, secure, and fully managed Platform as a Service (PaaS) that runs on Azure Kubernetes Service (AKS). HDInsight on AKS allows you to deploy popular Open-Source Analytics workloads like Apache Spark™, Apache Flink®️, and Trino without the overhead of managing and monitoring containers.

+* **Cluster pools** are a logical grouping of clusters that help build robust interoperability across multiple cluster types and allow enterprises to have the clusters in the same virtual network. Cluster pools provide rapid and cost-effective access to all the cluster types created on-demand and at scale. One cluster pool corresponds to one cluster in AKS infrastructure.

+* **Clusters** are individual compute workloads, such as Apache Spark, Apache Flink, and Trino that can be created rapidly in few minutes with preset configurations.

+[HDInsight](/azure/hdinsight/) offers Apache Kafka®, Apache HBase® and other analytics workloads in Platform as a Service (PaaS) formfactor.

+* [Apache Flink connecting to Apache HBase](./flink/use-flink-to-sink-kafka-message-into-hbase.md)

+* [Apache Flink connecting to Apache Kafka](./flink/join-stream-kafka-table-filesystem.md)

+* Apache Spark connecting to Apache HBase

+* Apache Spark connecting to Apache Kafka

+> [!Note]

+> - The Trino brand and trademarks are owned and managed by the [Trino Software Foundation](https://trino.io/foundation.html). No endorsement by The Trino Software Foundation is implied by the use of these marks.

+> - Apache Spark, Spark and the Spark logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+> - Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+> - Apache, Apache Flink, Flink and the Flink logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+> - Apache HBase, HBase and the HBase logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+> - Apache®, Apache Spark™, Apache HBase®, Apache Kafka®, and Apache Flink® are either registered trademarks or trademarks of the [Apache Software Foundation](https://www.apache.org/) in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

+ Title: Trademarks

+description: The Trademark and Brand Guidelines detail how you can help us protect MicrosoftΓÇÖs brand assets.

+# Trademarks

+Product names, logos and other material used on this Azure HDInsight on AKS learn pages are registered trademarks of various entities including, but not limited to, the following trademark owners and names:

+- The [Trino Software Foundation](https://trino.io/foundation.html) owns and manages the Trino brand and trademarks. The use of these marks does not imply endorsement by The Trino Software Foundation.

+- Apache Spark, Spark and the Spark logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+- Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+- Apache, Apache Flink, Flink and the Flink logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+- Apache HBase, HBase and the HBase logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+- Apache®, Apache Spark™, Apache HBase®, Apache Kafka®, and Apache Flink® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. The use of these marks does not imply endorsement by The Apache Software Foundation.

+## Release date: September 7, 2023

+This release applies to HDInsight 4.x and 5.x HDInsight release will be available to all regions over several days. This release is applicable for image number **2308221128**. [How to check the image number?](./view-hindsight-cluster-image-version.md)

+HDInsight uses safe deployment practices, which involve gradual region deployment. It might take up to 10 business days for a new release or a new version to be available in all regions.

+**OS versions**

+* HDInsight 4.0: Ubuntu 18.04.5 LTS Linux Kernel 5.4

+* HDInsight 5.0: Ubuntu 18.04.5 LTS Linux Kernel 5.4

+* HDInsight 5.1: Ubuntu 18.04.5 LTS Linux Kernel 5.4

+For workload specific versions, see

+* [HDInsight 5.x component versions](./hdinsight-5x-component-versioning.md)

+* [HDInsight 4.x component versions](./hdinsight-40-component-versioning.md)

+> [!IMPORTANT]

+> This release addresses the following CVEs released by [MSRC](https://msrc.microsoft.com/update-guide/vulnerability) on September 12, 2023. The action is to update to the latest image **2308221128**. Customers are advised to plan accordingly.

+| CVE | Severity| CVE Title| Remark |

+| - | - | - | - |

+| [CVE-2023-38156](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-38156) | Important | Azure HDInsight Apache Ambari Elevation of Privilege Vulnerability |Included on 2308221128 image |

+| [CVE-2023-36419](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36419) | Important | Azure HDInsight Apache Oozie Workflow Scheduler Elevation of Privilege Vulnerability | Apply [Script action](https://hdiconfigactions2.blob.core.windows.net/msrc-script/script_action.sh) on your clusters |

+##  Coming soon

+* The max length of cluster name will be changed to 45 from 59 characters, to improve the security posture of clusters. This change will be implemented by September 30, 2023.

+* Cluster permissions for secure storage

+ * Customers can specify (during cluster creation) whether a secure channel should be used for HDInsight cluster nodes to contact the storage account.

+* In-line quota update.

+ * Request quotas increase directly from the My Quota page, which will be a direct API call, which is faster. If the APdI call fails, then customers need to create a new support request for quota increase.

+* HDInsight Cluster Creation with Custom VNets.

+ * To improve the overall security posture of the HDInsight clusters, HDInsight clusters using custom VNETs need to ensure that the user needs to have permission for `Microsoft Network/virtualNetworks/subnets/join/action` to perform create operations. Customers would need to plan accordingly as this change would be a mandatory check to avoid cluster creation failures before September 30, 2023.ΓÇ»

+* Basic and Standard A-series VMs Retirement.

+ * On August 31, 2024, we'll retire Basic and Standard A-series VMs. Before that date, you need to migrate your workloads to Av2-series VMs, which provide more memory per vCPU and faster storage on solid-state drives (SSDs). To avoid service disruptions, [migrate your workloads](https://aka.ms/Av1retirement) from Basic and Standard A-series VMs to Av2-series VMs before August 31, 2024.

+* Non-ESP ABFS clusters [Cluster Permissions for Word Readable]

+ * Plan to introduce a change in non-ESP ABFS clusters, which restricts non-Hadoop group users from executing Hadoop commands for storage operations. This change to improve cluster security posture. Customers need to plan for the updates before September 30, 2023.ΓÇ»

+If you have any more questions, contact [Azure Support](https://ms.portal.azure.com/#view/Microsoft_Azure_Support/HelpAndSupportBlade/~/overview).

+You can always ask us about HDInsight on [Azure HDInsight - Microsoft Q&A](/answers/tags/168/azure-hdinsight)

+YouΓÇÖre welcome to add more proposals and ideas and other topics here and vote for them - [HDInsight Community (azure.com)](https://feedback.azure.com/d365community/search/?q=HDInsight).

+ > [!NOTE]

+ > We advise customers to use to latest versions of HDInsight [Images](./view-hindsight-cluster-image-version.md) as they bring in the best of open source updates, Azure updates and security fixes. For more information, see [Best practices](./hdinsight-overview-before-you-start.md).

+HDInsight uses safe deployment practices, which involve gradual region deployment. it might take up to 10 business days for a new release or a new version to be available in all regions.

+| CVE | Severity| CVE Title|

+| - | - | - |

+| [CVE-2023-35393](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-35393) | Important|Azure Apache Hive Spoofing Vulnerability |

+| [CVE-2023-35394](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-35394) | Important|Azure HDInsight Jupyter Notebook Spoofing Vulnerability |

+| [CVE-2023-36877](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36877) | Important|Azure Apache Oozie Spoofing Vulnerability |

+| [CVE-2023-36881](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36881) | Important|Azure Apache Ambari Spoofing Vulnerability |

+| [CVE-2023-38188](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-38188) | Important|Azure Apache Hadoop Spoofing Vulnerability |

+## Coming soon

+HDInsight uses safe deployment practices, which involve gradual region deployment. it might take up to 10 business days for a new release or a new version to be available in all regions.

+HDInsight uses safe deployment practices, which involve gradual region deployment. it might take up to 10 business days for a new release or a new version to be available in all regions.

+HDInsight uses safe deployment practices, which involve gradual region deployment. It might take up to 10 business days for a new release or a new version to be available in all regions.

+HDInsight uses safe deployment practices, which involve gradual region deployment. It might take up to 10 business days for a new release or a new version to be available in all regions.

+HDInsight cluster comes with predefined disk space based on SKU. This space might not be sufficient in large job scenarios.

+HDInsight uses safe deployment practices, which involve gradual region deployment. It might take up to 10 business days for a new release or a new version to be available in all regions.

+HDInsight cluster comes with predefined disk space based on SKU. This space might not be sufficient in large job scenarios.

+| LLAP status service driver might get stuck with wrong Yarn app ID|[HIVE-21866](https://issues.apache.org/jira/browse/HIVE-21866)|

+As customer scenarios grow more mature and diverse, we've identified some limitations with Interactive Query (LLAP) load-based Autoscale. These limitations are caused by the nature of LLAP query dynamics, future load prediction accuracy issues, and issues in the LLAP scheduler's task redistribution. Due to these limitations, users might see their queries run slower on LLAP clusters when Autoscale is enabled. The effect on performance can outweigh the cost benefits of Autoscale.

+As customer scenarios grow more mature and diverse, we've identified some limitations with Interactive Query (LLAP) load-based Autoscale. These limitations are caused by the nature of LLAP query dynamics, future load prediction accuracy issues, and issues in the LLAP scheduler's task redistribution. Due to these limitations, users might see their queries run slower on LLAP clusters when Autoscale is enabled. The effect on performance can outweigh the cost benefits of Autoscale.

+HDInsight now uses Azure virtual machines to provision the cluster. The service will gradually migrate to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+HDInsight now uses Azure virtual machines to provision the cluster. The service is gradually migrating to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+As customer scenarios grow more mature and diverse, we've identified some limitations with Interactive Query (LLAP) load-based Autoscale. These limitations are caused by the nature of LLAP query dynamics, future load prediction accuracy issues, and issues in the LLAP scheduler's task redistribution. Due to these limitations, users might see their queries run slower on LLAP clusters when Autoscale is enabled. The impact on performance can outweigh the cost benefits of Autoscale.

+HDInsight now uses Azure virtual machines to provision the cluster. The service is gradually migrating to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+HDInsight now uses Azure virtual machines to provision the cluster. Starting from this release, the service will gradually migrate to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+HDInsight now uses Azure virtual machines to provision the cluster. Starting from this release, the service will gradually migrate to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+HDInsight now uses Azure virtual machines to provision the cluster. Starting from this release, the service will gradually migrate to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+A fault domain is a logical grouping of underlying hardware in an Azure data center. Each fault domain shares a common power source and network switch. Before HDInsight Kafka might store all partition replicas in the same fault domain. Starting from this release, HDInsight now supports automatically distribution of Kafka partitions based on Azure fault domains.

+HDInsight now uses Azure virtual machines to provision the cluster. Starting from this release, the service will gradually migrate to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+An issue has been fixed in the Azure portal, where users were experiencing an error when they were creating an Azure HDInsight cluster using an SSH authentication type of public key. When users clicked **Review + Create**, they would receive the error "Must not contain any three consecutive characters from SSH username." This issue has been fixed, but it might require that you refresh your browser cache by hitting CTRL + F5 to load the corrected view. The workaround to this issue was to create a cluster with an ARM template.

+- [HDFS-8496](https://issues.apache.org/jira/browse/HDFS-8496): Calling stopWriter() with FSDatasetImpl lock held might block other threads (cmccabe).

+- [HBASE-16135](https://issues.apache.org/jira/browse/HBASE-16135): PeerClusterZnode under rs of removed peer might never be deleted.

+- [*HIVE-18587*](https://issues.apache.org/jira/browse/HIVE-18587): insert DML event might attempt to calculate a checksum on directories.

+- [*HIVE-18587*](https://issues.apache.org/jira/browse/HIVE-18587): insert DML event might attempt to calculate a checksum on directories.

+- There's a small possibility that some Mahout jobs might encounter "ClassNotFoundException" or "could not load class" errors related to "org.apache.commons.httpclient", "net.java.dev.jets3t", or related class name prefixes. If these errors happen, you might consider whether to manually install the needed jars in your classpath for the job, if the risk of security issues in the obsolete library is acceptable in your environment.

+- There's an even smaller possibility that some Mahout jobs might encounter crashes in Mahout's hbase-client code calls to the hadoop-common libraries, due to binary compatibility problems. Regrettably, there's no way to resolve this issue except revert to the HDP-2.4.2 version of Mahout, which might have security issues. Again, this should be unusual, and is unlikely to occur in any given Mahout job suite.

+- [RANGER-1984](https://issues.apache.org/jira/browse/RANGER-1984): HBase audit log records might not show all tags associated with accessed column.

+Fixed issues represent selected issues that were previously logged via Hortonworks Support, but are now addressed in the current release. These issues might have been reported in previous versions within the Known Issues section; meaning they were reported by customers or identified by Hortonworks Quality Engineering team.

+| BUG-99883 | [HIVE-19073](https://issues.apache.org/jira/browse/HIVE-19073), [HIVE-19145](https://issues.apache.org/jira/browse/HIVE-19145) | StatsOptimizer might mangle constant columns |

+| BUG-95542 | [HBASE-16135](https://issues.apache.org/jira/browse/HBASE-16135) | PeerClusterZnode under rs of removed peer might never be deleted |

+| BUG-96748 | [HIVE-18587](https://issues.apache.org/jira/browse/HIVE-18587) | insert DML event might attempt to calculate a checksum on directories |

+| BUG-97100 | [RANGER-1984](https://issues.apache.org/jira/browse/RANGER-1984) | HBase audit log records might not show all tags associated with accessed column |

+ >The noted values are working examples and might not be indicative of your environment. Ensure that the way you set these properties matches how your environment is configured.

+In this release HDI 5.1 version is moved to General Availability (GA) stage.

+## Release date: October 26, 2023

+This release applies to HDInsight 4.x and 5.x HDInsight release will be available to all regions over several days. This release is applicable for image number **2310140056**. [How to check the image number?](./view-hindsight-cluster-image-version.md)

+HDInsight uses safe deployment practices, which involve gradual region deployment. it might take up to 10 business days for a new release or a new version to be available in all regions.

+## What's new

+* HDInsight announces the General availability of HDInsight 5.1 starting October 26, 2023. This release brings in a full stack refresh to the [open source components](./hdinsight-5x-component-versioning.md#open-source-components-available-with-hdinsight-5x) and the integrations from Microsoft.

+ * Latest Open Source Versions ΓÇô [HDInsight 5.1](./hdinsight-5x-component-versioning.md) comes with the latest stable [open-source version](./hdinsight-5x-component-versioning.md#open-source-components-available-with-hdinsight-5x) available. Customers can benefit from all latest open source features, Microsoft performance improvements, and Bug fixes.

+ * Secure ΓÇô The latest versions come with the most recent security fixes, both open-source security fixes and security improvements by Microsoft.

+ * Lower TCO ΓÇô With performance enhancements customers can lower the operating cost, along with [enhanced autoscale](https://techcommunity.microsoft.com/t5/analytics-on-azure-blog/enhanced-autoscale-capabilities-in-hdinsight-clusters/ba-p/3811271).

+

+* Cluster permissions for secure storage

+ * Customers can specify (during cluster creation) whether a secure channel should be used for HDInsight cluster nodes to connect the storage account.

+

+* HDInsight Cluster Creation with Custom VNets.

+ * To improve the overall security posture of the HDInsight clusters, HDInsight clusters using custom VNETs need to ensure that the user needs to have permission for `Microsoft Network/virtualNetworks/subnets/join/action` to perform create operations. Customer might face creation failures if this check is not enabled.

+

+ * Non-ESP ABFS clusters [Cluster Permissions for Word Readable]

+ * Non-ESP ABFS clusters restrict non-Hadoop group users from executing Hadoop commands for storage operations. This change improves cluster security posture.ΓÇ»

+* The max length of cluster name will be changed to 45 from 59 characters, to improve the security posture of clusters. This change will be rolled out to all regions starting upcoming release.

+ * On August 31, 2024, we will retire Basic and Standard A-series VMs. Before that date, you need to migrate your workloads to Av2-series VMs, which provide more memory per vCPU and faster storage on solid-state drives (SSDs).

+ * To avoid service disruptions, [migrate your workloads](https://aka.ms/Av1retirement) from Basic and Standard A-series VMs to Av2-series VMs before August 31, 2024.

+We are listening: YouΓÇÖre welcome to add more ideas and other topics here and vote for them - [HDInsight Ideas](https://feedback.azure.com/d365community/search/?q=HDInsight) and follow us for more updates on [AzureHDInsight Community](https://www.linkedin.com/groups/14313521/)

+> [!NOTE]

+> This release addresses the following CVEs released by [MSRC](https://msrc.microsoft.com/update-guide/vulnerability) on September 12, 2023. The action is to update to the latest image 2308221128 or 2310140056. Customers are advised to plan accordingly.

+| CVE | Severity | CVE Title | Remark |

+| - | - | - | - |

+| [CVE-2023-38156](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-38156) | Important | Azure HDInsight Apache Ambari Elevation of Privilege Vulnerability |Included on image 2308221128 or 2310140056 |

+| [CVE-2023-36419](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36419) | Important | Azure HDInsight Apache Oozie Workflow Scheduler Elevation of Privilege Vulnerability | Apply [Script action](https://hdiconfigactions2.blob.core.windows.net/msrc-script/script_action.sh) on your clusters, or update to 2310140056 image |

+> [!NOTE]

+> We advise customers to use to latest versions of HDInsight [Images](./view-hindsight-cluster-image-version.md) as they bring in the best of open source updates, Azure updates and security fixes. For more information, see [Best practices](./hdinsight-overview-before-you-start.md).

+ Title: Migrate Apache Kafka workloads to Azure HDInsight 5.1

+description: Learn how to migrate Apache Kafka workloads on HDInsight 4.0 to HDInsight 5.1.

+# Migrate Apache Kafka workloads to Azure HDInsight 5.1

+Azure HDInsight 5.1 offers the latest open-source components with significant enhancements in performance, connectivity, and security. This document explains how to migrate Apache Kafka workloads from HDInsight 4.0 (Kafka 2.1) to HDInsight 5.1 (Kafka 3.2).

+## Apache Kafka versions

+### Apache Kafka 3.2.0

+

+If you migrate from Kafka to 3.2.0, you can take advantage of the following new features:

+- Support Automated consumer offsets sync across cluster in MM 2.0, making it easier to migrate or failover consumers across clusters. (KIP-545)

+- Hint to the partition leader to recover the partition: A new feature that allows the controller to communicate to a newly elected topic partition leader whether it needs to recover its state (KIP-704)

+- Supports TLS 1.2 by default for secure communication

+- Zookeeper Dependency Removal: Producers and consumers no longer need the zookeeper parameter. Use the `--bootstrap-server` option instead of `--zookeeper` with CLI commands. (KIP-500)

+- Configurable backlog size for creating Acceptor: A new configuration that allows setting the size of the SYN backlog for TCPΓÇÖs acceptor sockets on the brokers (KIP-764)

+- Top-level error code field to DescribeLogDirsResponse: A new error code that makes DescribeLogDirs API consistent with other APIs and allows returning other errors besides CLUSTER_AUTHORIZATION_FAILED (KIP-784)

+For a complete list of updates, see [Apache Kafka 3.2.0 release notes](https://archive.apache.org/dist/kafka/3.2.0/RELEASE_NOTES.html).

+## Kafka client compatibility

+New Kafka brokers support older clients. [KIP-35 - Retrieving protocol version](https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version) introduced a mechanism for dynamically determining the functionality of a Kafka broker and [KIP-97: Improved Kafka Client RPC Compatibility Policy](https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy) introduced a new compatibility policy and guarantees for the Java client. Previously, a Kafka client had to interact with a broker of the same version or a newer version. Now, newer versions of the Java clients and other clients that support KIP-35 such as `librdkafka` can fall back to older request types or throw appropriate errors if functionality isn't available.

+> [!NOTE]

+> Recommended to use kafka client version same as the cluster versions. For more information, see [Compatibility Matrix](https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix).

+## General migration process

+The following migration guidance assumes an Apache Kafka 2.1.1 cluster deployed on HDInsight 4.0 in a single virtual network. The existing broker has some topics and is being actively used by producers and consumers.

+Upgrading the Kafka version on an existing cluster isn't supported. After you create a cluster with HDI 5.1, migrate your Kafka clients to use the new cluster.

+To complete the migration, do the following steps:

+1. **Deploy a new HDInsight 5.1 cluster and clients for test.** Deploy a new HDInsight 5.1 Kafka cluster. If multiple Kafka cluster versions can be selected, it's recommended to select the latest version. After deployment, set some parameters as needed and create a topic with the same name as your existing environment. Also, set TLS and bring-your-own-key (BYOK) encryption as needed. Then check if it works correctly with the new cluster.

+ :::image type="content" source="./media/migrate-5-1-versions/deploy-new-hdinsight-clusters.png" alt-text="Screenshot shows how to Deploy new HDInsight 5.1 clusters." lightbox="./media/migrate-5-1-versions/deploy-new-hdinsight-clusters.png":::

+1. **Switch the cluster for the producer application, and wait until all the queue data is consumed by the current consumers.** When the new HDInsight 5.1 Kafka cluster is ready, switch the existing producer destination to the new cluster. Leave it as it is until the existing Consumer app has consumed all the data from the existing cluster.

+ :::image type="content" source="./media/migrate-5-1-versions/switch-cluster-producer-app.png" alt-text="Screenshot shows how to Switch cluster for producer app." lightbox="./media/migrate-5-1-versions/switch-cluster-producer-app.png":::

+1. **Switch the cluster on the consumer application.** After confirming that the existing consumer application has finished consuming all data from the existing cluster, switch the connection to the new cluster.

+ :::image type="content" source="./media/migrate-5-1-versions/switch-cluster-consumer-app.png" alt-text="Screenshot shows how to Switch cluster on consumer app." lightbox="./media/migrate-5-1-versions/switch-cluster-consumer-app.png":::

+1. **Remove the old cluster and test applications as needed.** Once the switch is complete and working properly, remove the old HDInsight 4.0 Kafka cluster and the producers and consumers used in the test as needed.

+## Next steps

+* [Performance optimization for Apache Kafka HDInsight clusters](apache-kafka-performance-tuning.md)

+* [Quickstart: Create Apache Kafka cluster in Azure HDInsight using Azure portal](apache-kafka-get-started.md)

+ * Searches for the attribute that failed validation returns the study/series/instance if the value is corrected in one of the few ways [mentioned below](#search-results-might-be-incomplete-for-extended-query-tags-with-validation-warnings).

+In the v1 API and continued for v2, if an [extended query tag](dicom-extended-query-tags-overview.md) has any errors, because one or more of the existing instances had a tag value that couldn't be indexed, then subsequent search queries containing the extended query tag return `erroneous-dicom-attributes` as detailed in the [documentation](dicom-extended-query-tags-overview.md#tag-query-status). However, tags (also known as attributes) with validation warnings from STOW-RS are **not** included in this header. If a store request results in validation warnings for [searchable attributes](dicom-services-conformance-statement-v2.md#searchable-attributes) at the time the instance was stored, those attributes may not be used to search for the stored instance. However, any [searchable attributes](dicom-services-conformance-statement-v2.md#searchable-attributes) that failed validation will be able to return results if the values are overwritten by instances in the same study/series that are stored after the failed one, or if the values are already stored correctly by a previous instance. If the attribute values are not overwritten, then they will not produce any search results.

+An attribute can be corrected in the following ways:

+- Delete the stored instance and upload a new instance with the corrected data

+- Upload a new instance in the same study/series with corrected data

+In the v1 API and continued for v2, if an [extended query tag](dicom-extended-query-tags-overview.md) has any errors, because one or more of the existing instances had a tag value that couldn't be indexed, then subsequent search queries containing the extended query tag returns `erroneous-dicom-attributes` as detailed in the [documentation](dicom-extended-query-tags-overview.md#tag-query-status). However, tags (also known as attributes) with validation warnings from STOW-RS are **not** included in this header. If a store request results in validation warnings for [searchable attributes](#searchable-attributes) at the time the [instance was stored](#store-changes-from-v1), those attributes may not be used to search for the stored instance. However, any [searchable attributes](#searchable-attributes) that failed validation will be able to return results if the values are overwritten by instances in the same study/series that are stored after the failed one, or if the values are already stored correctly by a previous instance. If the attribute values are not overwritten, then they will not produce any search results.

+An attribute can be corrected in the following ways:

+- Delete the stored instance and upload a new instance with the corrected data

+- Upload a new instance in the same study/series with corrected data

+- Monitoring and diagnostics for hundreds of thousands of connected IoT devices.

+Azure IoT Central is an application platform as a service (aPaaS) that manages scalability and HADR for you. An IoT Central application can scale to support hundreds of thousands of connected devices. For more information about device and message pricing, see [Azure IoT Central pricing](https://azure.microsoft.com/pricing/details/iot-central/). For more information about the service level agreement, see [SLA for Azure IoT Central](https://azure.microsoft.com/support/legal/sla/iot-central/v1_0/).

+| Item | Quota or limit |

+| - | -- |

+| Number of devices in an application | 200,000 |

+| Number of IoT Central simulated devices in an application | 100 |

+| Number of messages per second per application | 200 | Individual devices can temporarily send up to 10 messages per second. |

+| Item | Quota or limit |

+| - | -- |

+| Query API requests per second | 1 |

+| Other API requests per second | 20 |

+For large volumes of export data, you may experience up to 60 seconds of latency. Typically, the latency is much lower than this.

+| Number of device groups in an application | 500 | For performance reasons, you shouldn't exceed this limit. |

+| Number of devices registrations per minute | 200 | The underlying DPS instance sets this quota. |

+| Number of rules in an application | 50 | This quota is fixed and can't be changed. |

+An IoT Central application lets you monitor and manage hundreds of thousands of devices throughout their life cycle. This guide is for device developers who implement the code to run on devices that connect to IoT Central.

+An IoT Central application lets you monitor and manage hundreds of thousands of devices throughout their life cycle.

+ IoT Central provides a scalable and reliable infrastructure to capture streaming data from hundreds of thousands of connected devices. Sometimes, you need to run stream computations over the hot or warm data paths to meet business requirements. You can also merge IoT data with data in external stores such as Azure Data explorer to provide enhanced diagnostics.

+See [Configure RBAC for batch endpoint invoke](#configure-rbac-for-batch-endpoints-invoke) for a detailed list of RBAC permissions.

+### Prerequisites

+* This example assumes that you have a model correctly deployed as a batch endpoint. Particularly, we are using the *heart condition classifier* created in the tutorial [Using MLflow models in batch deployments](how-to-mlflow-batch.md).

+## Configure RBAC for Batch Endpoints invoke

+Batch Endpoints exposes a durable API consumers can use to generate jobs. The invoker request proper permission to be able to generate those jobs. You can either use one of the [built-in security roles](../role-based-access-control/built-in-roles.md) or you can create a custom role for the purposes.

+To successfully invoke a batch endpoint you need the following explicit actions granted to the identity used to invoke the endpoints. See [Steps to assign an Azure role](../role-based-access-control/role-assignments-steps.md) for instructions to assign them.

+```json

+"actions": [

+ "Microsoft.MachineLearningServices/workspaces/read",

+ "Microsoft.MachineLearningServices/workspaces/data/versions/write",

+ "Microsoft.MachineLearningServices/workspaces/datasets/registered/read",

+ "Microsoft.MachineLearningServices/workspaces/datasets/registered/write",

+ "Microsoft.MachineLearningServices/workspaces/datasets/unregistered/read",

+ "Microsoft.MachineLearningServices/workspaces/datasets/unregistered/write",

+ "Microsoft.MachineLearningServices/workspaces/datastores/read",

+ "Microsoft.MachineLearningServices/workspaces/datastores/write",

+ "Microsoft.MachineLearningServices/workspaces/datastores/listsecrets/action",

+ "Microsoft.MachineLearningServices/workspaces/listStorageAccountKeys/action",

+ "Microsoft.MachineLearningServices/workspaces/batchEndpoints/read",

+ "Microsoft.MachineLearningServices/workspaces/batchEndpoints/deployments/read",

+ "Microsoft.MachineLearningServices/workspaces/computes/read",

+ "Microsoft.MachineLearningServices/workspaces/computes/listKeys/action",

+ "Microsoft.MachineLearningServices/workspaces/metadata/secrets/read",

+ "Microsoft.MachineLearningServices/workspaces/metadata/snapshots/read",

+ "Microsoft.MachineLearningServices/workspaces/metadata/artifacts/read",

+ "Microsoft.MachineLearningServices/workspaces/metadata/artifacts/write",

+ "Microsoft.MachineLearningServices/workspaces/experiments/read",

+ "Microsoft.MachineLearningServices/workspaces/experiments/runs/submit/action",

+ "Microsoft.MachineLearningServices/workspaces/experiments/runs/read",

+ "Microsoft.MachineLearningServices/workspaces/experiments/runs/write",

+ "Microsoft.MachineLearningServices/workspaces/metrics/resource/write",

+ "Microsoft.MachineLearningServices/workspaces/modules/read",

+ "Microsoft.MachineLearningServices/workspaces/models/read",

+ "Microsoft.MachineLearningServices/workspaces/endpoints/pipelines/read",

+ "Microsoft.MachineLearningServices/workspaces/endpoints/pipelines/write",

+ "Microsoft.MachineLearningServices/workspaces/environments/read",

+ "Microsoft.MachineLearningServices/workspaces/environments/write",

+ "Microsoft.MachineLearningServices/workspaces/environments/build/action"

+ "Microsoft.MachineLearningServices/workspaces/environments/readSecrets/action"

+]

+```

+You can compare the input data, parameters, output settings, run settings of the two pipeline jobs or components using compare feature. To learn more, see [how to enable and use the graph compare feature](./how-to-use-pipeline-ui.md#compare-different-pipelines-to-debug-failure-or-other-unexpected-issues-preview)

+To identify any changes in pipeline topology, pipeline input/output, or pipeline settings between two pipelines, select **Compare graph** after adding two pipeline jobs to the compare list.

+Furthermore, you can compare two components to observe if there have been any changes in the component input/output, component setting or source code. To do this, select **Compare details** after adding two components to the compare list.

+When developing flows, you can not only use the built-in tools provided by Prompt flow, but also develop your own custom tool. In this document, we guide you through the process of developing your own tool package, offering detailed steps and advice on how to utilize your creation.

+After successful installation, your custom "tool" can show up in the tool list:

+Your tool package should be a python package. To develop your custom tool, follow the steps **Create your own tool package** and **build and share the tool package** in [Create and Use Tool package](https://microsoft.github.io/promptflow/how-to-guides/develop-a-tool/create-and-use-tool-package.html). You can also [Add a tool icon](https://microsoft.github.io/promptflow/how-to-guides/develop-a-tool/add-a-tool-icon.html) and [Add Category and tags](https://microsoft.github.io/promptflow/how-to-guides/develop-a-tool/add-category-and-tags-for-tool.html) for your tool.

+To add the custom tool to your tool list, it's necessary to create a runtime, which is based on a customized environment where your custom tool is preinstalled. Here we use [my-tools-package](https://pypi.org/project/my-tools-package/) as an example to prepare the runtime.

+ 1. Create a customized environment in Azure Machine Learning studio by going to **Environments** then select **Create**. In the settings tab under *Select environment source*, choose " Create a new docker content."

+ RUN pip install my-tools-package==0.0.1

+ It takes several minutes to create the environment. After it succeeded, you can copy the Azure Container Registry (ACR) from environment detail page for the next step.

+### Prepare compute instance runtime

+1. Create a compute instance runtime using the customized environment created in step 2.

+ 1. Create a new compute instance. Existing compute instance created long time ago can possibly hit unexpected issue.

+ 2. Create runtime on CI with customized environment.

+1. Install Prompt flow for VS Code extension

+ :::image type="content" source="./media/how-to-custom-tool-package-creation-and-usage/prompt-flow-vs-code-extension.png" alt-text="Screenshot of Prompt flow VS Code extension."lightbox ="./media/how-to-custom-tool-package-creation-and-usage/prompt-flow-vs-code-extension.png":::

+2. Go to terminal and install your tool package in conda environment of the extension. Assume your conda env name is `prompt-flow`.

+3. Go to the extension and open one flow folder. Select 'flow.dag.yaml' and preview the flow. Next, select `+` button and you can see your tools. You need to **reload the windows** to clean previous cache if you don't see your tool in the list.

+ 3. Run this script in your conda environment. It returns the metadata of all tools installed in your local environment, and you should verify that your tools are listed.

+- If you encounter a `403 Forbidden Error`, it's likely due to a naming conflict with an existing package. You need to choose a different name. Package names must be unique on PyPI to avoid confusion and conflicts among users. Before creating a new package, it's recommended to search PyPI (https://pypi.org/) to verify that your chosen name isn't already taken. If the name you want is unavailable, consider selecting an alternative name or a variation that clearly differentiates your package from the existing one.

+> [!IMPORTANT]

+> For more information, you can refer to [the Prompt flow CLI documentation for Azure](https://microsoft.github.io/promptflow/reference/pfazure-command-reference.html).

+During iterative development, as you refine and fine-tune your flow or prompts, it could be beneficial to carry out multiple iterations locally within your code repository. The community version, **Prompt flow VS Code extension** and **Prompt flow local SDK & CLI** is provided to facilitate pure local development and testing without Azure binding.

+- [Prompt flow CLI documentation for Azure](https://microsoft.github.io/promptflow/reference/pfazure-command-reference.html)

+ Title: Embedding tool in Azure Machine Learning prompt flow (preview)

+description: Prompt flow embedding tool uses OpenAI's embedding models to convert text into dense vector representations for various NLP tasks.

+# Embedding tool (preview)

+OpenAI's embedding models convert text into dense vector representations for various NLP tasks. See the [OpenAI Embeddings API](https://platform.openai.com/docs/api-reference/embeddings) for more information.

+> [!IMPORTANT]

+> Prompt flow is currently in public preview. This preview is provided without a service-level agreement, and is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+## Prerequisites

+Create OpenAI resources:

+- **OpenAI**

+ Sign up account [OpenAI website](https://openai.com/)

+ Login and [Find personal API key](https://platform.openai.com/account/api-keys)

+- **Azure OpenAI (AOAI)**

+ Create Azure OpenAI resources with [instruction](../../../ai-services/openai/how-to/create-resource.md)

+## **Connections**

+Set up connections to provide resources in embedding tool.

+| Type | Name | API KEY | API Type | API Version |

+|-|-|-|-|-|

+| OpenAI | Required | Required | - | - |

+| AzureOpenAI | Required | Required | Required | Required |

+## Inputs

+| Name | Type | Description | Required |

+||-|--|-|

+| input | string | the input text to embed | Yes |

+| connection | string | the connection for the embedding tool use to provide resources | Yes |

+| model/deployment_name | string | instance of the text-embedding engine to use. Fill in model name if you use OpenAI connection, or deployment name if use Azure OpenAI connection. | Yes |

+## Outputs

+| Return Type | Description |

+|-||

+| list | The vector representations for inputs |

+There is an example response returned by the embedding tool:

+<details>

+ <summary>Output</summary>

+

+```

+[-0.005744616035372019,

+-0.007096089422702789,

+-0.00563855143263936,

+-0.005272455979138613,

+-0.02355326898396015,

+0.03955197334289551,

+-0.014260607771575451,

+-0.011810848489403725,

+-0.023170066997408867,

+-0.014739611186087132,

+...]

+```

+</details>

+> [!NOTE]

+> We now remove the `embedding` option from LLM tool api setting. You can use embedding api with [Embedding tool](embedding-tool.md).

+| prompt | string | text prompt for the language model. | Yes |

+ Title: More tools in Prompt flow

+description: More tools in Prompt flow are displayed in the table, along with instructions for custom tool package creation and tool package usage.

+# More tools in Prompt flow (preview)

+This table provides an index of more tools. If existing tools can't meet your requirements, you can [develop your own custom tool and make a tool package](https://microsoft.github.io/promptflow/how-to-guides/develop-a-tool/create-and-use-tool-package.html).

+> [!IMPORTANT]

+> Prompt flow is currently in public preview. This preview is provided without a service-level agreement, and is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+| Tool name | Description | Environment | Package Name |

+||--|-|--|

+| [Python](./python-tool.md) | Run Python code. | Default | -- |

+| [LLM](./llm-tool.md) | Use Open AI's Large Language Model for text completion or chat. | Default | -- |

+| [Prompt](./prompt-tool.md) | Craft prompt using Jinja as the templating language. | Default | -- |

+| [Embedding](./embedding-tool.md) | Use Open AI's embedding model to create an embedding vector representing the input text. | Default | -- |

+| [Open Source LLM](./open-source-llm-tool.md) | Use an Open Source model from the Azure Model catalog, deployed to an Azure Machine Learning Online Endpoint for LLM Chat or Completion API calls. | Default | -- |

+| [Serp API](./serp-api-tool.md) | Use Serp API to obtain search results from a specific search engine. | Default | -- |

+| [Content Safety (Text)](./content-safety-text-tool.md) | Use Azure Content Safety to detect harmful content. | Default | [promptflow-contentsafety](https://pypi.org/project/promptflow-contentsafety/) |

+| [Faiss Index Lookup](./faiss-index-lookup-tool.md) | Search vector based query from the FAISS index file. | Default | [promptflow-vectordb](https://pypi.org/project/promptflow-vectordb/) |

+| [Vector DB Lookup](./vector-db-lookup-tool.md) | Search vector based query from existing Vector Database. | Default | [promptflow-vectordb](https://pypi.org/project/promptflow-vectordb/) |

+| [Vector Index Lookup](./vector-index-lookup-tool.md) | Search text or vector based query from Azure Machine Learning Vector Index. | Default | [promptflow-vectordb](https://pypi.org/project/promptflow-vectordb/) |

+To discover more custom tools that developed by the open source community, see [more custom tools](https://microsoft.github.io/promptflow/integrations/tools/https://docsupdatetracker.net/index.html).

+For the tools that should be utilized in the custom environment, see [Custom tool package creation and usage](../how-to-custom-tool-package-creation-and-usage.md#prepare-runtime) to prepare the runtime. Then the tools can be displayed in the tool list.

+ Title: Open source LLM tool in Azure Machine Learning prompt flow (preview)

+description: The Prompt flow Open Source LLM tool enables you to utilize various Open Source and Foundational Models.

+# Open Source LLM (preview)

+The Prompt flow Open Source LLM tool enables you to utilize various Open Source and Foundational Models, such as [Falcon](https://aka.ms/AAlc25c) or [Llama 2](https://aka.ms/AAlc258) for natural language processing, in PromptFlow.

+Here's how it looks in action on the Visual Studio Code Prompt flow extension. In this example, the tool is being used to call a LlaMa-2 chat endpoint and asking "What is CI?".

+This Prompt flow supports two different LLM API types:

+- **Chat**: Shown in the example above. The chat API type facilitates interactive conversations with text-based inputs and responses.

+- **Completion**: The Completion API type is used to generate single response text completions based on provided prompt input.

+> [!IMPORTANT]

+> Prompt flow is currently in public preview. This preview is provided without a service-level agreement, and is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+## Quick Overview: How do I use Open Source LLM Tool?

+1. Choose a Model from the Azure Machine Learning Model Catalog and deploy.

+2. Setup and select the connections to the model deployment.

+3. Configure the tool with the model settings.

+4. [Prepare the Prompt](./prompt-tool.md#how-to-write-prompt).

+5. Run the flow.

+## Prerequisites: Model Deployment

+1. Pick the model that matched your scenario from the [Azure Machine Learning model catalog](https://ml.azure.com/model/catalog).

+2. Use the "Deploy" button to deploy the model to an Azure Machine Learning Online Inference endpoint.

+To learn more, see [Deploying foundation models to endpoints for inferencing.](../../how-to-use-foundation-models.md#deploying-foundation-models-to-endpoints-for-inferencing)

+## Prerequisites: Prompt flow Connections

+In order for Prompt flow to use your deployed model, you need to set up a Connection. Explicitly, the Open Source LLM tool uses the CustomConnection.

+1. To learn how to create a custom connection, see [create a connection](https://microsoft.github.io/promptflow/how-to-guides/manage-connections.html#create-a-connection).

+ The keys to set are:

+ 1. **endpoint_url**

+ - This value can be found at the previously created Inferencing endpoint.

+ 2. **endpoint_api_key**

+ - Ensure to set this key as a secret value.

+ - This value can be found at the previously created Inferencing endpoint.

+ 3. **model_family**

+ - Supported values: LLAMA, DOLLY, GPT2, or FALCON

+ - This value is dependent on the type of deployment you're targeting.

+## Running the Tool: Inputs

+The Open Source LLM tool has many parameters, some of which are required. See the below table for details, you can match these parameters to the screenshot above for visual clarity.

+| Name | Type | Description | Required |

+|||-|-|

+| api | string | This parameter is the API mode and depends on the model used and the scenario selected. *Supported values: (Completion \| Chat)* | Yes |

+| connection | CustomConnection | This parameter is the name of the connection, which points to the Online Inferencing endpoint. | Yes |

+| model_kwargs | dictionary | This input is used to provide configuration specific to the model used. For example, the Llama-02 model uses {\"temperature\":0.4}. *Default: {}* | No |

+| deployment_name | string | The name of the deployment to target on the Online Inferencing endpoint. If no value is passed, the Inferencing load balancer traffic settings are used. | No |

+| prompt | string | The text prompt that the language model uses to generate its response. | Yes |

+## Outputs

+| API | Return Type | Description |

+||-||

+| Completion | string | The text of one predicted completion |

+| Chat | string | The text of one response int the conversation |

+### Types

+| Type | Python example | Description |

+|--||--|

+| int | param: int | Integer type |

+| bool | param: bool | Boolean type |

+| string | param: str | String type |

+| double | param: float | Double type |

+| list | param: list or param: List[T] | List type |

+| object | param: dict or param: Dict[K, V] | Object type |

+| [Connection](../concept-connections.md) | param: CustomConnection | Connection type will be handled specially |

+Parameters with `Connection` type annotation will be treated as connection inputs, which means:

+- Prompt flow extension will show a selector to select the connection.

+- During execution time, prompt flow will try to find the connection with the name same from parameter value passed in.

+> [!Note]

+> `Union[...]` type annotation is supported **ONLY** for connection type, for example, `param: Union[CustomConnection, OpenAIConnection]`.

+ *The sample in the next section defines python tool "my_python_tool" which decorated with @tool*

+This snippet shows the basic structure of a tool function. Prompt flow will read the function and extract inputs from function parameters and type annotations.

+from promptflow.connections import CustomConnection

+# Please update the function name/signature per need

+def my_python_tool(message: str, my_conn: CustomConnection) -> str:

+ my_conn_dict = dict(my_conn)

+ # Do some function call with my_conn_dict...

+| Name | Type | Sample Value in Flow Yaml | Value passed to function|

+||--|-| |

+| message | string | "world" | "world" |

+| my_conn | CustomConnection | "my_conn" | CustomConnection object |

+Prompt flow will try to find the connection named 'my_conn' during execution time.

+This article contains reference information for [configuring Kubernetes with Azure Machine Learning](./how-to-attach-kubernetes-anywhere.md).

+|Oct 11, 2023 | 1.1.35| Fix vulnerable image. Bug fixes. |

+|Aug 25, 2023 | 1.1.34| Fix vulnerable image. Return more detailed identity error. Bug fixes. |

+| Mar 27, 2023| 1.1.25 | Add Azure Machine Learning job throttle. Fast fail for training job when SSH setup failed. Reduce Prometheus scrape interval to 30s. Improve error messages for inference. Fix vulnerable image. |

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=start-spark-session)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=root-dir)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=init-prj-ws-client)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=init-fs-ws-client)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=init-fs-core-sdk)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=redis-settings)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=provision-redis)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=reuse-redis)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=retrieve-uai)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=attach-online-store)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=enable-accounts-material)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=start-accounts-backfill)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=track-accounts-backfill)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=enable-transact-material)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=start-transact-material)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=track-transact-material)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=parse-feat-list)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=init-online-lookup)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=online-feat-loockup)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=define-endpoint)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=create-endpoint)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=get-endpoint-identity)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=endpoint-redis-rbac)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=endpoint-fs-rbac)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=define-online-deployment)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=begin-online-deployment)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=test-online-deployment)]

+ :::image type="content" source="./media/tutorial-discover-aws/search-subscription.png" alt-text="Screenshot of Search box to search for the Azure subscription.":::

+ :::image type="content" source="../../includes/role-based-access-control/media/add-role-assignment-page.png" alt-text="Screenshot of Add role assignment page in Azure portal.":::

+> If you have already created a project, you can use the same project to register additional appliances to discover and assess more no of servers. [Learn more](create-manage-projects.md#find-a-project).

+ :::image type="content" source="./media/tutorial-discover-physical/key-format.png" alt-text="Screenshot of SSH private key supported format.":::

+* Appliance can connect to only those SQL Server instances to which it has network line of sight, whereas software inventory by itself might not need network line of sight.

+ :::image type="content" source="./media/tutorial-discover-gcp/search-subscription.png" alt-text="Screenshot of Search box to search for the Azure subscription.":::

+ :::image type="content" source="../../includes/role-based-access-control/media/add-role-assignment-page.png" alt-text="Screenshot of role assignment page in Azure portal.":::

+> If you have already created a project, you can use the same project to register additional appliances to discover and assess more no of servers. [Learn more](create-manage-projects.md#find-a-project).

+ :::image type="content" source="./media/tutorial-discover-physical/key-format.png" alt-text="Screenshot of SSH private key supported format.":::

+* Appliance can connect to only those SQL Server instances to which it has network line of sight, whereas software inventory by itself might not need network line of sight.

+**Servers** | All Windows and Linux OS versions are supported for discovery of configuration and performance metadata. <br /><br /> For application discovery on servers, all Windows and Linux OS versions are supported. Check the [OS versions supported for agentless dependency analysis](migrate-support-matrix-hyper-v.md#dependency-analysis-requirements-agentless).<br /><br /> To discover ASP.NET web apps running on IIS web server, check [supported Windows OS and IIS versions](migrate-support-matrix-vmware.md#web-apps-discovery-requirements). For discovery of installed applications and for agentless dependency analysis, Windows servers must have PowerShell version 2.0 or later installed.<br /><br /> To discover Java web apps running on Apache Tomcat web server, check [supported Linux OS and Tomcat versions](migrate-support-matrix-vmware.md#web-apps-discovery-requirements).

+ :::image type="content" source="./media/tutorial-discover-hyper-v/search-subscription.png" alt-text="Screenshot of Search box to search for the Azure subscription.":::

+ :::image type="content" source="../../includes/role-based-access-control/media/add-role-assignment-page.png" alt-text="Screenshot of add role assignment page in Azure portal.":::

+ :::image type="content" source="./media/tutorial-discover-hyper-v/added-tool.png" alt-text="Screenshot showing Azure Migrate: Discovery and assessment tool added by default.":::

+> If you have already created a project, you can use the same project to register additional appliances to discover and assess more no of servers. [Learn more](create-manage-projects.md#find-a-project)

+* Appliance can connect to only those SQL Server instances to which it has network line of sight, whereas software inventory by itself might not need network line of sight.

+ :::image type="content" source="./media/tutorial-discover-vmware/guest-operations.png" alt-text="Screenshot that shows the vSphere web client and how to create a new account and select user roles and privileges.":::

+* Appliance can connect to only those SQL Server instances to which it has network line of sight, whereas software inventory by itself might not need network line of sight.

+Details such as OS license support status, inventory, database instances, etc. are displayed.

+In this tutorial, you use Azure Network Watcher [next hop](network-watcher-next-hop-overview.md) tool to troubleshoot and diagnose a VM routing problem that's preventing it from correctly communicating with other resources. Next hop shows you that a [custom route](../virtual-network/virtual-networks-udr-overview.md?toc=/azure/network-watcher/toc.json#custom-routes) caused the routing problem.

+1. In the search box at the top of the portal, enter ***myResourceGroup***. Select **myResourceGroup** from the search results.

+## Next step

+> [!div class="nextstepaction"]

+> [Diagnose a virtual machine network routing problem](diagnose-vm-network-routing-problem.md)

+ - Standard_E4as_v5

+ - Standard_E8as_v5

+ - Standard_E16as_v5

+ Refer to the [QuickStart guide](./quickstarts-virtual-machine-deployment-cli.md) to deploy a VNF using the image you created.

+Before proceeding with this how-to guide, ensure you have completed all steps in the Azure Operator Nexus virtual machine [QuickStart guide](./quickstarts-virtual-machine-deployment-cli.md).

+> The resourceId argument must be specified in the form of an ARM ID, and it must be a valid resource ID for the target object. If the resourceId is incorrect or invalid, the placement hint will not work correctly, and the VM scheduling might fail.

+> In this and the following examples, only variations of the `--placement-hints` argument are provided. For the actual creation of the VM with placement hints, you should add `--placement-hints` to the CLI illustrated in the VM [QuickStart guide](./quickstarts-virtual-machine-deployment-cli.md).

+> The current placement hint configuration with the Affinity hintType ensures that the virtual machine is scheduled exclusively on the specified rack with the provided rack ID. However, it's important to note that the rack affinity cannot be specified for more than one rack with `Hard` scheduling execution. This limitation might influence your deployment strategy, particularly if you are considering placing VMs on multiple racks and allowing the scheduler to select from them.

+This placement hint uses both the `AntiAffinity` hintType and `Hard` schedulingExecution to prevent the virtual machine from being scheduled on the specified rack identified by the rack ID. In this configuration, the scheduler strictly follows these placement hints. However, if the rack ID is incorrect or there's not enough capacity on other racks, the VM placement might fail due to the strict application of the `Hard` scheduling rule

+This placement hint uses both the `AntiAffinity` hintType and `Hard` schedulingExecution to prevent the virtual machine from being scheduled on the specified bare-metal machine identified by the bare-metal machine ID. In this configuration, the scheduler strictly follows these placement hints. However, if the bare-metal machine ID is incorrect or there's not enough capacity on other bare-metal machines, the VM placement might fail due to the strict application of the `Hard` scheduling rule

+ Title: Create an Azure Operator Nexus virtual machine by using Azure Resource Manager template (ARM template)

+description: Learn how to create an Azure Operator Nexus virtual machine (VM) for virtual network function (VNF) workloads by using Azure Resource Manager template (ARM template).

+# Quickstart: Create an Azure Operator Nexus virtual machine by using Azure Resource Manager template (ARM template)

+* Deploy an Azure Nexus virtual machine using an Azure Resource Manager template.

+This quick-start guide is designed to help you get started with using Nexus virtual machines to host virtual network functions (VNFs). By following the steps outlined in this guide, you're able to quickly and easily create a customized Nexus virtual machine that meets your specific needs and requirements. Whether you're a beginner or an expert in Nexus networking, this guide is here to help. You learn everything you need to know to create and customize Nexus virtual machines for hosting virtual network functions.

+## Before you begin

+* Complete the [prerequisites](./quickstarts-tenant-workload-prerequisites.md) for deploying a Nexus virtual machine.

+## Review the template

+Before deploying the virtual machine template, let's review the content to understand its structure.

+Once you have reviewed and saved the template file named ```virtual-machine-arm-template.json```, proceed to the next section to deploy the template.

+## Deploy the template

+1. Create a file named ```virtual-machine-parameters.json``` and add the required parameters in JSON format. You can use the following example as a starting point. Replace the values with your own.

+2. Deploy the template.

+```azurecli

+ az deployment group create --resource-group myResourceGroup --template-file virtual-machine-arm-template.json --parameters @virtual-machine-parameters.json

+```

+## Review deployed resources

+## Clean up resources

+## Next steps

+You've successfully created a Nexus virtual machine. You can now use the virtual machine to host virtual network functions (VNFs).

+ Title: Create an Azure Operator Nexus virtual machine by using Bicep

+description: Learn how to create an Azure Operator Nexus virtual machine (VM) for virtual network function (VNF) workloads by using Bicep

+# Quickstart: Create an Azure Operator Nexus virtual machine by using Bicep

+* Deploy an Azure Nexus virtual machine using Bicep

+This quick-start guide is designed to help you get started with using Nexus virtual machines to host virtual network functions (VNFs). By following the steps outlined in this guide, you're able to quickly and easily create a customized Nexus virtual machine that meets your specific needs and requirements. Whether you're a beginner or an expert in Nexus networking, this guide is here to help. You learn everything you need to know to create and customize Nexus virtual machines for hosting virtual network functions.

+## Before you begin

+* Complete the [prerequisites](./quickstarts-tenant-workload-prerequisites.md) for deploying a Nexus virtual machine.

+## Review the template

+Before deploying the virtual machine template, let's review the content to understand its structure.

+Once you have reviewed and saved the template file named ```virtual-machine-bicep-template.bicep```, proceed to the next section to deploy the template.

+## Deploy the template

+1. Create a file named ```virtual-machine-parameters.json``` and add the required parameters in JSON format. You can use the following example as a starting point. Replace the values with your own.

+2. Deploy the template.

+```azurecli

+ az deployment group create --resource-group myResourceGroup --template-file virtual-machine-bicep-template.bicep --parameters @virtual-machine-parameters.json

+```

+## Review deployed resources

+## Clean up resources

+## Next steps

+You've successfully created a Nexus virtual machine. You can now use the virtual machine to host virtual network functions (VNFs).

+ Title: Create an Azure Operator Nexus virtual machine by using Azure CLI

+description: Learn how to create an Azure Operator Nexus virtual machine (VM) for virtual network function (VNF) workloads

+# Quickstart: Create an Azure Operator Nexus virtual machine by using Azure CLI

+* Deploy an Azure Nexus virtual machine using Azure CLI

+This quick-start guide is designed to help you get started with using Nexus virtual machines to host virtual network functions (VNFs). By following the steps outlined in this guide, you're able to quickly and easily create a customized Nexus virtual machine that meets your specific needs and requirements. Whether you're a beginner or an expert in Nexus networking, this guide is here to help. You learn everything you need to know to create and customize Nexus virtual machines for hosting virtual network functions.

+## Before you begin

+* Complete the [prerequisites](./quickstarts-tenant-workload-prerequisites.md) for deploying a Nexus virtual machine.

+## Create a Nexus virtual machine

+The following example creates a virtual machine named *myNexusVirtualMachine* in resource group *myResourceGroup* in the *eastus* location.

+Before you run the commands, you need to set several variables to define the configuration for your virtual machine. Here are the variables you need to set, along with some default values you can use for certain variables:

+| Variable | Description |

+| -- | |

+| LOCATION | The Azure region where you want to create your virtual machine. |

+| RESOURCE_GROUP | The name of the Azure resource group where you want to create the virtual machine. |

+| SUBSCRIPTION | The ID of your Azure subscription. |

+| CUSTOM_LOCATION | This argument specifies a custom location of the Nexus instance. |

+| CSN_ARM_ID | CSN ID is the unique identifier for the cloud services network you want to use. |

+| L3_NETWORK_ID | L3 Network ID is the unique identifier for the network interface to be used by the virtual machine. |

+| NETWORK_INTERFACE_NAME | The name of the L3 network interface for the virtual machine. |

+| ADMIN_USERNAME | The username for the virtual machine administrator. |

+| SSH_PUBLIC_KEY | The SSH public key that is used for secure communication with the virtual machine. |

+| CPU_CORES | The number of CPU cores for the virtual machine (even number, max 46 vCPUs) |

+| MEMORY_SIZE | The amount of memory (in GB, max 224 GB) for the virtual machine. |

+| VM_DISK_SIZE | The size (in GB) of the virtual machine disk. |

+| VM_IMAGE | The URL of the virtual machine image. |

+| ACR_URL | The URL of the Azure Container Registry. |

+| ACR_USERNAME | The username for the Azure Container Registry. |

+| ACR_PASSWORD | The password for the Azure Container Registry. |

+Once you've defined these variables, you can run the Azure CLI command to create the virtual machine. Add the ```--debug``` flag at the end to provide more detailed output for troubleshooting purposes.

+To define these variables, use the following set commands and replace the example values with your preferred values. You can also use the default values for some of the variables, as shown in the following example:

+```bash

+# Azure parameters

+RESOURCE_GROUP="myResourceGroup"

+SUBSCRIPTION="<Azure subscription ID>"

+CUSTOM_LOCATION="/subscriptions/<subscription_id>/resourceGroups/<managed_resource_group>/providers/microsoft.extendedlocation/customlocations/<custom-location-name>"

+LOCATION="$(az group show --name $RESOURCE_GROUP --query location --subscription $SUBSCRIPTION -o tsv)"

+# VM parameters

+VM_NAME="myNexusVirtualMachine"

+# VM credentials

+ADMIN_USERNAME="azureuser"

+SSH_PUBLIC_KEY="$(cat ~/.ssh/id_rsa.pub)"

+# Network parameters

+CSN_ARM_ID="/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.NetworkCloud/cloudServicesNetworks/<csn-name>"

+L3_NETWORK_ID="/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.NetworkCloud/l3Networks/<l3Network-name>"

+NETWORK_INTERFACE_NAME="mgmt0"

+# VM Size parameters

+CPU_CORES=4

+MEMORY_SIZE=12

+VM_DISK_SIZE="64"

+# Virtual Machine Image parameters

+VM_IMAGE="<VM image, example: myacr.azurecr.io/ubuntu:20.04>"

+ACR_URL="<Azure container registry URL, example: myacr.azurecr.io>"

+ACR_USERNAME="<Azure container registry username>"

+ACR_PASSWORD="<Azure container registry password>"

+```

+> [!IMPORTANT]

+> It is essential that you replace the placeholders for CUSTOM_LOCATION, CSN_ARM_ID, L3_NETWORK_ID and ACR parameters with your actual values before running these commands.

+After defining these variables, you can create the virtual machine by executing the following Azure CLI command.

+```bash

+az networkcloud virtualmachine create \

+ --name "$VM_NAME" \

+ --resource-group "$RESOURCE_GROUP" \

+ --subscription "$SUBSCRIPTION" \

+ --extended-location name="$CUSTOM_LOCATION" type="CustomLocation" \

+ --location "$LOCATION" \

+ --admin-username "$ADMIN_USERNAME" \

+ --csn "attached-network-id=$CSN_ARM_ID" \

+ --cpu-cores $CPU_CORES \

+ --memory-size $MEMORY_SIZE \

+ --network-attachments '[{"attachedNetworkId":"'$L3_NETWORK_ID'","ipAllocationMethod":"Dynamic","defaultGateway":"True","networkAttachmentName":"'$NETWORK_INTERFACE_NAME'"}]'\

+ --storage-profile create-option="Ephemeral" delete-option="Delete" disk-size="$VM_DISK_SIZE" \

+ --vm-image "$VM_IMAGE" \

+ --ssh-key-values "$SSH_PUBLIC_KEY" \

+ --vm-image-repository-credentials registry-url="$ACR_URL" username="$ACR_USERNAME" password="$ACR_PASSWORD"

+```

+After a few minutes, the command completes and returns information about the virtual machine. You've created the virtual machine. You're now ready to use them.

+## Review deployed resources

+## Clean up resources

+## Next steps

+You've successfully created a Nexus virtual machine. You can now use the virtual machine to host virtual network functions (VNFs).

+ Title: Create an Azure Operator Nexus virtual machine by using Azure PowerShell

+description: Learn how to create an Azure Operator Nexus virtual machine (VM) for virtual network function (VNF) workloads using PowerShell

+# Quickstart: Create an Azure Operator Nexus virtual machine by using Azure PowerShell

+* Deploy an Azure Nexus virtual machine using Azure PowerShell

+This quick-start guide is designed to help you get started with using Nexus virtual machines to host virtual network functions (VNFs). By following the steps outlined in this guide, you're able to quickly and easily create a customized Nexus virtual machine that meets your specific needs and requirements. Whether you're a beginner or an expert in Nexus networking, this guide is here to help. You learn everything you need to know to create and customize Nexus virtual machines for hosting virtual network functions.

+## Before you begin

+* Complete the [prerequisites](./quickstarts-tenant-workload-prerequisites.md) for deploying a Nexus virtual machine.

+## Create a Nexus virtual machine

+The following example creates a virtual machine named *myNexusVirtualMachine* in resource group *myResourceGroup* in the *eastus* location.

+Before you run the commands, you need to set several variables to define the configuration for your virtual machine. Here are the variables you need to set, along with some default values you can use for certain variables:

+| Variable | Description |

+| -- | |

+| LOCATION | The Azure region where you want to create your virtual machine. |

+| RESOURCE_GROUP | The name of the Azure resource group where you want to create the virtual machine. |

+| SUBSCRIPTION | The ID of your Azure subscription. |

+| CUSTOM_LOCATION | This argument specifies a custom location of the Nexus instance. |

+| CSN_ARM_ID | CSN ID is the unique identifier for the cloud services network you want to use. |

+| L3_NETWORK_ID | L3 Network ID is the unique identifier for the network interface to be used by the virtual machine. |

+| NETWORK_INTERFACE_NAME | The name of the L3 network interface for the virtual machine. |

+| ADMIN_USERNAME | The username for the virtual machine administrator. |

+| SSH_PUBLIC_KEY | The SSH public key that is used for secure communication with the virtual machine. |

+| CPU_CORES | The number of CPU cores for the virtual machine (even number, max 46 vCPUs) |

+| MEMORY_SIZE | The amount of memory (in GB, max 224 GB) for the virtual machine. |

+| VM_DISK_SIZE | The size (in GB) of the virtual machine disk. |

+| VM_IMAGE | The URL of the virtual machine image. |

+| ACR_URL | The URL of the Azure Container Registry. |

+| ACR_USERNAME | The username for the Azure Container Registry. |

+| ACR_PASSWORD | The password for the Azure Container Registry. |

+| VMDEVICEMODEL | The VMDeviceModel defaults to T2, available options T2(Modern) and T1(Transitional). |

+| USERDATA | The base64 encoded string of cloud-init userdata. |

+| BOOTMETHOD | The Method used to boot the virutalmachine UEFI or BIOS. |

+| OS_DISK_CREATE_OPTION | The OS disk create specifies ephemeral disk option. |

+| OS_DISK_DELETE_OPTION | The OS disk delete specifies delete disk option. |

+| IP_AllOCATION_METHOD | The IpAllocationMethod valid for L3Networks specify Dynamic or Static or Disabled. |

+| NETWORKATTACHMENTNAME | The name of the Network to attach for workload. |

+| NETWORKDATA | The base64 encoded string of cloud-init network data. |

+Once you've defined these variables, you can run the Azure PowerShell command to create the virtual machine. Add the ```-Debug``` flag at the end to provide more detailed output for troubleshooting purposes.

+To define these variables, use the following set commands and replace the example values with your preferred values. You can also use the default values for some of the variables, as shown in the following example:

+```azurepowershell-interactive

+# Azure parameters

+$RESOURCE_GROUP="myResourceGroup"

+$SUBSCRIPTION="<Azure subscription ID>"

+$CUSTOM_LOCATION="/subscriptions/<subscription_id>/resourceGroups/<managed_resource_group>/providers/microsoft.extendedlocation/customlocations/<custom-location-name>"

+$CUSTOM_LOCATION_TYPE="CustomLocation"

+$LOCATION="<ClusterAzureRegion>"

+# VM parameters

+$VM_NAME="myNexusVirtualMachine"

+$BOOT_METHOD="UEFI"

+$OS_DISK_CREATE_OPTION="Ephemeral"

+$OS_DISK_DELETE_OPTION="Delete"

+$NETWORKDATA="bmV0d29ya0RhdGVTYW1wbGU="

+$VMDEVICEMODEL="T2"

+$USERDATA=""

+# VM credentials

+$ADMIN_USERNAME="admin"

+$SSH_PUBLIC_KEY = @{

+ KeyData = "$(cat ~/.ssh/id_rsa.pub)"

+}

+# Network parameters

+$CSN_ARM_ID="/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.NetworkCloud/cloudServicesNetworks/<csn-name>"

+$L3_NETWORK_ID="/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.NetworkCloud/l3Networks/<l3Network-name>"

+$IP_AllOCATION_METHOD="Dynamic"

+$CSN_ATTACHMENT_DEFAULTGATEWAY="False"

+$CSN_ATTACHMENT_NAME="<l3Network-name>"

+$ISOLATE_EMULATOR_THREAD="True"

+$VIRTIOINTERFACE="Modern"

+$NETWORKATTACHMENTNAME="mgmt0"

+# VM Size parameters

+$CPU_CORES=4

+$MEMORY_SIZE=12

+$VM_DISK_SIZE="64"

+# Virtual Machine Image parameters

+$VM_IMAGE="<VM image, example: myacr.azurecr.io/ubuntu:20.04>"

+$ACR_URL="<Azure container registry URL, example: myacr.azurecr.io>"

+$ACR_USERNAME="<Azure container registry username>"

+$NETWORKATTACHMENT = New-AzNetworkCloudNetworkAttachmentObject `

+-AttachedNetworkId $L3_NETWORK_ID `

+-IpAllocationMethod $IP_AllOCATION_METHOD `

+-DefaultGateway "True" `

+-Name $NETWORKATTACHMENTNAME

+$SECUREPASSWORD = ConvertTo-SecureString "<YourPassword>" -asplaintext -force

+```

+> [!IMPORTANT]

+> It is essential that you replace the placeholders for CUSTOM_LOCATION, CSN_ARM_ID, L3_NETWORK_ID and ACR parameters with your actual values before running these commands.

+After defining these variables, you can create the virtual machine by executing the following Azure PowerShell command.

+```azurepowershell-interactive

+New-AzNetworkCloudVirtualMachine -Name $VM_NAME `

+-ResourceGroupName $RESOURCE_GROUP `

+-AdminUsername $ADMIN_USERNAME `

+-CloudServiceNetworkAttachmentAttachedNetworkId $CSN_ARM_ID `

+-CloudServiceNetworkAttachmentIPAllocationMethod $IP_AllOCATION_METHOD `

+-CpuCore $CPU_CORES `

+-ExtendedLocationName $CUSTOM_LOCATION `

+-ExtendedLocationType $CUSTOM_LOCATION_TYPE `

+-Location $LOCATION `

+-SubscriptionId $SUBSCRIPTION `

+-MemorySizeGb $MEMORY_SIZE `

+-OSDiskSizeGb $VM_DISK_SIZE `

+-VMImage $VM_IMAGE `

+-BootMethod $BOOT_METHOD `

+-CloudServiceNetworkAttachmentDefaultGateway $CSN_ATTACHMENT_DEFAULTGATEWAY `

+-CloudServiceNetworkAttachmentName $CSN_ATTACHMENT_NAME `

+-IsolateEmulatorThread $ISOLATE_EMULATOR_THREAD `

+-NetworkAttachment $NETWORKATTACHMENT `

+-NetworkData $NETWORKDATA `

+-OSDiskCreateOption $OS_DISK_CREATE_OPTION `

+-OSDiskDeleteOption $OS_DISK_DELETE_OPTION `

+-SshPublicKey $SSH_PUBLIC_KEY `

+-UserData $USERDATA `

+-VMDeviceModel $VMDEVICEMODEL `

+-VMImageRepositoryCredentialsUsername $ACR_USERNAME `

+-VMImageRepositoryCredentialsPassword $SECUREPASSWORD `

+-VMImageRepositoryCredentialsRegistryUrl $ACR_URL

+```

+After a few minutes, the command completes and returns information about the virtual machine. You've created the virtual machine. You're now ready to use them.

+## Review deployed resources

+## Clean up resources

+## Next steps

+You've successfully created a Nexus virtual machine. You can now use the virtual machine to host virtual network functions (VNFs).

+Azure Orbital Ground Station interfaces with your Azure resources using virtual network (VNET) injection, which is used in both uplink and downlink directions. This page describes how to ensure your subnet and Azure Orbital Ground Station resources are configured correctly.

+In this how-to guide, you'll learn how to:

+> [!div class="checklist"]

+> * Prepare the subnet for VNET injection

+> * Prepare endpoints

+> * Verify the contact profile

+> * Find IPs of scheduled contacts

+Ensure the objects comply with the recommendations in this article. Note that these steps don't have to be followed in order.

+## Create and prepare subnet for VNET injection

+- An entire subnet with no existing IPs allocated or in use that can be dedicated to the Azure Orbital Ground Station service, in your virtual network within your resource group. If you need to make a new subnet, follow instructions to [add a subnet](../virtual-network/virtual-network-manage-subnet.md?tabs=azure-portal#add-a-subnet).

+Follow instructions to [add a subnet delegation](../virtual-network/manage-subnet-delegation.md#delegate-a-subnet-to-an-azure-service) in your virtual network. Delegate your subnet to the service named: **Microsoft.Orbital/orbitalGateways**.

+The following is an example of a typical VNET setup with a subnet delegated to Azure Orbital Ground Station:

+Set the MTU of all desired endpoints to at least **3650** by sending appropriate commands to your virtual machine within your resource group.

+## Verify the contact profile

+### Region

+The VNET/subnet must be in the same region as the contact profile.

+### Subnet ID

+1. Go to overview page of your contact profile and select **JSON view**. Find the **networkConfigurations** section, then identify the "**subnetId**".

+2. Go to overview page of your virtual network and select **JSON view**. Find the section for your **delegated subnet**, then identify the "**id**".

+3. Verify that these IDs are identical.

+### Link flows: IP Address and Port

+The links/channels must be set up in the following manner, based on direction and TCP or UDP preference.

+> [!NOTE]

+> These settings are for managed modems only.

+#### Uplink

+| Setting | TCP Client | TCP Server | UDP Client | UDP Server |

+|:|:|:-|:|:|

+| Contact Profile: Link/Channel **IP Address** | Blank | Routable IP from delegated subnet | Blank | Not applicable |

+| Contact Profile: Link/Channel **Port** | Unique port in 49152-65535 | Unique port in 49152-65535 | Unique port in 49152-65535 | Not applicable |

+| **Output** | | | | |

+| Contact Resource: **destinationIP** | Connect to this IP | Not applicable | Connect to this IP | Not applicable |

+| Contact Resource: **sourceIP** | Not applicable | Link comes from one of these IPs | Not applicable | Not applicable |

+#### Downlink

+| Setting | TCP Client | TCP Server | UDP Client | UDP Server |

+|:|:|:-|:|:-|

+| Contact Profile: Link/Channel **IP Address** | Blank | Routable IP from delegated subnet | Not applicable | Routable IP from delegated subnet |

+| Contact Profile: Link/Channel **Port** | Unique port in 49152-65535 | Unique port in 49152-65535 | Not applicable | Unique port in 49152-65535 |

+| **Output** | | | | |

+| Contact Resource: **destinationIP** | Connect to this IP | Not applicable | Not applicable | Not applicable |

+| Contact Resource: **sourceIP** | Not applicable | Link comes from one of these IPs | Not applicable | Link comes from one of these IPs |

+> You can have multiple links/channels in a contact profile, and you can have multiple IPs. However the combination of port/protocol must be unique. You can't have two identical ports, even if you have two different destination IPs.

+For more information, learn about [contact profiles](/azure/orbital/concepts-contact-profile) and [how to configure a contact profile](/azure/orbital/contact-profile).

+## Find IPs of a scheduled contact

+The Azure Orbital Ground Station platform prereserves IPs in the subnet when a contact is scheduled. These IPs represent the platform-side endpoints for each link. IPs are unique between contacts, and if multiple concurrent contacts are using the same subnet, Microsoft guarantees those IPs to be distinct. The service fails to schedule the contact and an error is returned if the service runs out of IPs or can't allocate an IP.

+When you create a contact, you can find these IPs by viewing the contact properties.

+To view the contact properties, go to the contact resource overview page and select **JSON view** in the portal or use the **GET contact** API call. Make sure to use the current API version of 2022-11-01. The parameters of interest are below:

+| **Parameter** | **Usage** |

+||--|

+| antennaConfiguration.destinationIp | Connect to this IP when you configure the link as **tcp/udp client**. |

+| antennaConfiguration.sourceIps | Data comes from this IP when you configure the link as **tcp/udp server**. |

+You can use this information to set up network policies or to distinguish between simultaneous contacts to the same endpoint.

+> [!NOTE]

+> - The source and destination IPs are always taken from the subnet address range.

+> - Only one destination IP is present. Any link in client mode should connect to this IP and the links are differentiated based on port.

+> - Many source IPs can be present. Links in server mode will connect to your specified IP address in the contact profile. The flows will originate from the source IPs present in this field and target the port as per the link details in the contact profile. There is no fixed assignment of link to source IP so please make sure to allow all IPs in any networking setup or firewalls.

+For more information, learn about [contacts](/azure/orbital/concepts-contact) and [how to schedule a contact](/azure/orbital/schedule-contact).

+This article describes how to use Azure services like Azure Functions, and how to install connectors to Azure resources for Confluent Cloud.

+Azure Native ISV Services enable you to easily provision, manage, and tightly integrate *independent software vendor* (ISV) software and services on Azure. Azure Native ISV Services is developed and managed by Microsoft and the ISV. Currently, several services are publicly available across these areas: observability, data, networking, and storage. For a list of all our current ISV partner services, see [Extend Azure with Azure Native ISV Services](partners.md).

+A list of features of any Azure Native ISV Service listed follows.

+- Logs and metrics: Seamlessly direct logs and metrics from Azure Monitor to the Azure Native ISV Service using just a few gestures. You can configure autodiscovery of resources to monitor, and set up automatic log forwarding and metrics shipping. You can easily do the setup in Azure, without needing to create more infrastructure or write custom code.

+- Virtual network injection: Provides private data plane access to Azure Native ISV services from customersΓÇÖ virtual networks.

+- Unified billing: Get a single bill for all resources that you consume on Azure for the Qumulo service.

+- Private access: The service is directly connected to your own virtual network, sometimes called *VNet injection*.

+ > [Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/qumulo1584033880660.qumulo-saas-mpp?tab=Overview)

+- **[Cost Savings](../flexible-server/how-to-deploy-on-azure-free-account.md)** ΓÇô Flexible server allows you to stop and start server on-demand to lower your TCO. Your compute tier billing is stopped immediately, which allows you to have significant cost savings during development, testing and for time-bound predictable production workloads.

+- **[Support for new PG versions](../flexible-server/concepts-supported-versions.md)** - Flexible server currently supports PG version 11 and onwards till version 14. Newer community versions of PostgreSQL will be supported only in flexible server.

+The following table gives an overview of Offline and Online modes of migration.

+| Online | - Very minimal downtime to application.<br />- Ideal for large databases and for customers having limited downtime requirements.<br />| - Replication used in online migration has multiple restrictions listed in this [doc](https://www.postgresql.org/docs/current/logical-replication-restrictions.html) (e.g Primary Keys needed in all tables)<br />- Tough and more complex to execute than offline migration.ΓÇ»<br />- Greater chances of failure due to complexity of migration.ΓÇ»<br />There's an impact on the source server's storage and compute if the migration runs for a long time. The impact needs to be monitored closely during migration. |

+## Pre-migration validations

+We have noticed many migrations fail due to setup issues on source and target server. Most of the issues can be categorized into the following buckets:

+* Issues related to authentication/permissions for the migration user on source and target server.

+* [Prerequisites](#migration-prerequisites) not being taken care of, before running the migration.

+* Unsupported features/configurations between the source and target.

+Pre-migration validation helps you verify if the migration setup is intact to perform a successful migration. Checks are done against a rule set and any potential problems along with the remedial actions are shown to take corrective measures.

+### How to use pre-migration validation?

+A new parameter called **Migration option** is introduced while creating a migration.

+You can pick any of the following options

+* **Validate** - Use this option to check your server and database readiness for migration to the target. **This option will not start data migration and will not require any downtime to your servers.**

+The result of the Validate option can be

+ - **Succeeded** - No issues were found and you can plan for the migration

+ - **Failed** - There were errors found during validation, which can fail the migration. Go through the list of errors along with their suggested workarounds and take corrective measures before planning the migration.

+ - **Warning** - Warnings are informative messages that you need to keep in mind while planning the migration.

+ Plan your migrations better by performing pre-migration validations in advance to know the potential issues you might encounter while performing migrations.

+* **Migrate** - Use this option to kickstart the migration without going through validation process. It's recommended to perform validation before triggering a migration to increase the chances for a successful migration. Once validation is done, you can use this option to start the migration process.

+* **Validate and Migrate** - In this option, validations are performed and then migration gets triggered if all checks are in **succeeded** or **warning** state. Validation failures don't start the migration between source and target servers.

+We recommend customers to use pre-migration validations in the following way:

+1) Choose **Validate** option and run pre-migration validation on an advanced date of your planned migration.

+2) Analyze the output and take any remedial actions of any errors.

+3) Run Step 1 again till the validation is successful

+4) Start the migration using the **Validate and Migrate** option on the planned date and time.

+> [!NOTE]

+> Pre-migration validations is enabled for flexible servers in North Europe region. It will be enabled for flexible servers in other Azure regions soon. This functionality is available only in Azure portal. Support for CLI will be introduced at a later point in time.

+- You can have only one active migration or validation to your flexible server.

+- The source and target server must be in the same Azure region. Cross region migrations are enabled only for servers in China regions.

+- Azure Active Directory users present on your source server won't be migrated to target server. To mitigate this limitation, manually create all Azure Active Directory users on your target server using this [link](../flexible-server/how-to-manage-azure-ad-users.md) before triggering a migration. If Azure Active Directory users aren't created on target server, migration fails with appropriate error message.

+- If the target flexible server uses SCRAM-SHA-256 password encryption method, connection to flexible server using the users/roles on single server fails since the passwords are encrypted using md5 algorithm. To mitigate this limitation, choose the option **MD5** for **password_encryption** server parameter on your flexible server.

+- If your single server is public access (case #1 and case #2 in the above table), there's nothing needed from your end. The single to flex migration tool automatically establishes connection between single and flexible server and the migration goes through.

+- If your single server is in private access, then the only supported scenario is when your Flexible server is inside a VNet. If your flexible server is deployed in the same VNet as the private end point of your Single server, connections between single server and flexible server should automatically work provided there's no network security group(NSGs) blocking the connectivity between subnets. If flexible server is deployed in another VNet, [peering should be established between the VNets](../../virtual-network/tutorial-connect-virtual-networks-portal.md) for the connection to work between Single and Flexible server.

+The migration tool automatically allows lists all extensions used by your single server databases on your flexible server except for the ones whose libraries need to be loaded at the server start.

+Execute the following query on your source server to get the list of Azure Active Directory users.

+Create the Azure Active Directory users on your target flexible server using this [link](../flexible-server/how-to-manage-azure-ad-users.md) before creating a migration.

+#### Database migration planning

+The first step in the database migration planning is to run pre-migration validation on your source and target server to check for any errors in the migration setup. Analyze the validation report and take any remedial actions if needed. Keep running pre migration validation until it results in **Succeeded** state. Now with the migration setup ready, you can move on to the next phase of planning.

+The next phase of planning involves downtime incurred by applications for performing offline migration using the single to flex migration tool.

+##### How to calculate the downtime?

+In most cases, the non-prod servers (dev, UAT, test, staging) are migrated using offline migrations. Since these servers have less data than the production servers, the migration completes fast. For migration of production server, you need to know the time it would take to complete the migration to plan for it in advance.

+The time taken for an offline migration to complete is dependent on several factors that includes the number of databases, size of databases, number of tables inside each database, number of indexes, and the distribution of data across tables. It also depends on the SKU of the source and target server, and the IOPS available on the source and target server. Given the many factors that can affect the migration time, it's hard to estimate the total time for the offline migration to complete. The best approach would be to try it on a server restored from the primary server.

+For calculating the total downtime to perform offline migration of production server, the following phases are considered.

+- **Migration of PITR** - The best way to get a good estimate on the time taken to migrate your production database server would be to take a point-in time restore of your production server and run the offline migration on this newly restored server.

+- **Migration of Buffer** - After completing the above step, you can plan for actual production migration during a time period when the application traffic is low. This migration can be planned on the same day or probably a week away. By this time, the size of the source server might have increased. Update your estimated migration time for your production server based on the amount of this increase. If the increase is significant, you can consider doing another test using the PITR server. But for most servers the size increase shouldn't be significant enough.

+- **Data Validation** - Once the offline migration completes for the production server, you need to verify if the data in flexible server is an exact copy of the single server. Customers can use opensource/thirdparty tools or can do the validation manually. Prepare the validation steps that you would like to do in advance of the actual migration. Validation can include:

+ * Row count match for all the tables involved in the migration.

+ * Matching counts for all the database object (tables, sequences, extensions, procedures, indexes)

+ * Comparing max or min IDs of key application related columns

+> [!NOTE]

+> The size of databases is not the right metric for validation.The source server might have bloats/dead tuples which can bump up the size on the source server. Also, the storage containers used in single and flexible servers are completely different. It is completely normal to have size differences between source and target servers. If there is an issue in the first three steps of validation, it indicates a problem with the migration.

+- **Migration of server settings** - The server parameters, firewall rules (if applicable), tags, alerts need to be manually copied from single server to flexible server.

+- **Changing connection strings** - Post successful validation, application should change their connection strings to point to flexible server. This activity is coordinated with the application team to make changes to all the references of connection strings pointing to single server. In the flexible server the user parameter in the connection string no longer needs to be in the **username@servername** format. You should just use the **user=username** format for this parameter in the connection string

+For example

+Psql -h **mysingleserver**.postgres.database.azure.com -u **user1@mysingleserver** -d db1

+should now be of the format

+Psql -h **myflexserver**.postgres.database.azure.com -u user1 -d db1

+**Total planned downtime** = **Time to migrate PITR** + **time to migrate Buffer** + **time for Validation** + **time to migrate server settings** + **time to switch connection strings to the flexible server.**

+While most frequently a migration runs without a hitch, it's good practice to plan for contingencies if more time is required for debugging or if a migration needs to be restarted.

+Trigger the migration of your production databases using the **Migrate** or **Validate and Migrate** option in the migration tool. The migration requires close monitoring, and the monitoring user interface of the migration tool comes in handy. Check the migration status over the period of time to ensure there is progress and wait for the migration to complete.

+In general, a powerful SKU is recommended for the target as the migration tool runs out of a container on the Flexible server. A powerful SKU enables a greater number of tables to be migrated in parallel. You can scale the SKU back to your preferred configuration after the migration. This section contains steps to improve the migration speed in case the data distribution among the tables is skewed and/or a more powerful SKU doesn't have a significant impact on the migration speed.

+If the data distribution on the source is highly skewed, with most of the data present in one table, the allocated compute for migration isn't fully utilized and it creates a bottleneck. So, we split large tables into smaller chunks, which are then migrated in parallel. This is applicable to tables that have more than 10000000 (10m) tuples. Splitting the table into smaller chunks is possible if one of the following conditions is satisfied.

+2. If the table doesn't have a simple primary key or unique index of type int or big int, but has a column that meets the data type criteria, the column can be converted into a unique index using the below command. This command does not require a lock on the table.

+3. If the table has neither a simple int/big int primary key or unique index nor any column that meets the data type criteria, you can add such a column using [ALTER](https://www.postgresql.org/docs/current/sql-altertable.html) and drop it post-migration. Running the ALTER command requires a lock on the table.

+If any of the above conditions are satisfied, the table is migrated in multiple partitions in parallel, which should provide a marked increase in the migration speed.

+In summary, the Single to Flexible migration tool migrates a table in parallel threads and reduce the migration time if:

+3. The SKU used has idle cores, which can be used for migrating the table in parallel.

+**Migration Option** gives you the option to perform validations before triggering a migration. You can pick any of the following options

+ - **Validate** - Checks your server and database readiness for migration to the target.

+ - **Migrate** - Skips validations and starts migrations.

+ - **Validate and Migrate** - Performs validation before triggering a migration. Migration gets triggered only if there are no validation failures.

+It is always a good practice to choose **Validate** or **Validate and Migrate** option to perform pre-migration validations before running the migration. To learn more about the pre-migration validation refer to this [documentation](./concepts-single-to-flexible.md#pre-migration-validations).

+**Migration mode** gives you the option to pick the mode for the migration. **Offline** is the default option. Support for online migrations will be introduced later in the tool.

+After you make the **Subscription** and **Resource Group** selections, the dropdown list for server names shows Single Servers under that resource group across regions. Select the source that you want to migrate databases from. Note that you can migrate databases from a Single Server to a target Flexible Server in the same region. Cross region migrations are supported only in China regions.

+### Review

+The **Review** tab summarizes all the details for creating the validation or migration. Review the details and click on the start button.

+After you click the start button, a notification appears in a few seconds to say that the validation or migration creation is successful. You are redirected automatically to the **Migration** blade of Flexible Server. This has a new entry for the recently created validation or migration.

+The grid that displays the migrations has these columns: **Name**, **Status**, **Source DB server**, **Resource group**, **Region**, **Databases**, and **Start time**. The entries are displayed in the descending order of the start time with the most recent entry on the top.

+You can use the refresh button to refresh the status of the validation or migration.

+You can also select the migration name in the grid to see the associated details.

+As soon as the validation or migration is created, it moves to the **InProgress** state and **PerformingPreRequisiteSteps** substate. It takes 2-3 minutes for the workflow to set up the migration infrastructure and network connections.

+Let us look at how to monitor migrations for each of the **Migration Option**.

+### Validate

+After the **PerformingPreRequisiteSteps** substate is completed, the validation moves to the substate of **Validation in Progress** where checks are done on the source and target server to assess the readiness for migration.

+The validation moves to the **Succeeded** state if all validations are either in **Succeeded** or **Warning** state.

+The validation grid has the following columns

+- **Finding** - Represents the validation rules that are used to check readiness for migration.

+- **Finding Status** - Represents the result for each rule and can have any of the three values

+ - **Succeeded** - If no errors were found.

+ - **Failed** - If there are validation errors.

+ - **Warning** - If there are validation warnings.

+- **Impacted Object** - Represents the object name for which the errors or warnings are raised.

+- **Object Type** - This can have the value **Database** for database level validations and **Instance** for server level validations.

+The validation moves to **Validation Failed** state if there are any errors in the validation. Click on the **Finding** in the grid whose status is **Failed** and a fan-out pane appears giving the details and the corrective action you should take to avoid this error.

+### Migrate

+### Validate and Migrate

+In this option, validations are performed first before migration starts. After the **PerformingPreRequisiteSteps** sub state is completed, the workflow moves into the sub state of **Validation in Progress**.

+- If validation has errors, the migration will move into a **Failed** state.

+- If validation completes without any error, the migration will start and the workflow will move into the sub state of **Migrating Data**.

+You can see the results of validation under the **Validation** tab and monitor the migration under the **Migration** tab.

+- **Validation Failed** : The validation has failed.

+- **Validation in Progress**: Validation is in progress.

+You can cancel any ongoing validations or migrations. The workflow must be in the **InProgress** state to be canceled. You can't cancel a validation or migration that's in the **Succeeded** or **Failed** state.

+Canceling a validation stops any further validation activity and the validation moves to a **Cancelled** state.

+Canceling a migration stops further migration activity on your target server and moves to a **Cancelled** state. It doesn't drop or roll back any changes on your target server. Be sure to drop the databases on your target server involved in a canceled migration.

+| Azure Database for MySQL - Single Server (Microsoft.DBforMySQL/servers) | mysqlServer | privatelink.mysql.database.azure.com | mysql.database.azure.com |

+| Azure Database for MySQL - Flexible Server (Microsoft.DBforMySQL/flexibleServers) | mysqlServer | privatelink.mysql.database.azure.com | mysql.database.azure.com |

+| Azure Database for MySQL - Single Server (Microsoft.DBforMySQL/servers) | mysqlServer | privatelink.mysql.database.usgovcloudapi.net | mysql.database.usgovcloudapi.net |

+| Azure Database for MySQL - Flexible Server (Microsoft.DBforMySQL/flexibleServers) | mysqlServer | privatelink.mysql.database.usgovcloudapi.net | mysql.database.usgovcloudapi.net |

+| Azure Database for MySQL - Single Server (Microsoft.DBforMySQL/servers) | mysqlServer | privatelink.mysql.database.chinacloudapi.cn | mysql.database.chinacloudapi.cn |

+| Azure Database for MySQL - Flexible Server (Microsoft.DBforMySQL/flexibleServers) | mysqlServer | privatelink.mysql.database.chinacloudapi.cn | mysql.database.chinacloudapi.cn |

+| Central US | Germany West Central | Israel Central | | Japan East |

+| Israel | Israel Central|

+Reference the connection details and sample code in the following tables, according to your connection's authentication type and client type, to connect your compute services to Azure Cosmos DB for Apache Cassandra.

+| Java - Spring Boot |  |  |  |  |

+## Default environment variable names or application properties and Sample code

+### System-assigned managed identity

+#### SpringBoot client type

+Using a system-assigned managed identity as the authentication type is only available for Spring Cloud Azure version 4.0 or higher.

+| Default environment variable name | Description | Example value |

+|--|--|--|

+| spring.cloud.azure.cosmos.credential.managed-identity-enabled | Whether to enable managed identity | `true` |

+| spring.cloud.azure.cosmos.database | Your database | `https://management.azure.com/.default` |

+| spring.cloud.azure.cosmos.endpoint | Your resource endpoint | `https://<database-server>.documents.azure.com:443/` |

+#### Other client types

+#### Sample code

+Refer to the steps and code below to Connect to Azure Cosmos DB for NoSQL.

+### User-assigned managed identity

+#### SpringBoot client type

+Using a user-assigned managed identity as the authentication type is only available for Spring Cloud Azure version 4.0 or higher.

+| Default environment variable name | Description | Example value |

+|--|--|--|

+| spring.cloud.azure.cosmos.credential.managed-identity-enabled | Whether to enable managed identity | `true` |

+| spring.cloud.azure.cosmos.database | Your database | `https://management.azure.com/.default` |

+| spring.cloud.azure.cosmos.endpoint | Your resource endpoint | `https://<database-server>.documents.azure.com:443/` |

+| spring.cloud.azure.cosmos.credential.client-id | Your client ID | `<client-ID>` |

+#### Other client types

+| AZURE_COSMOS_CLIENTID | Your client ID | `<client-ID>` |

+#### Sample code

+Refer to the steps and code below to Connect to Azure Cosmos DB for NoSQL.

+### Connection string

+#### SpringBoot client type

+| Default environment variable name | Description | Example value |

+|--|-|-|

+| azure.cosmos.key | The access key for your database for Spring Cloud Azure version below 4.0 | `<access-key>` |

+| azure.cosmos.database | Your database for Spring Cloud Azure version below 4.0 | `<database-name>` |

+| azure.cosmos.uri | Your database URI for Spring Cloud Azure version below 4.0 | `https://<database-server>.documents.azure.com:443/` |

+| spring.cloud.azure.cosmos.key | The access key for your database for Spring Cloud Azure version over 4.0 | `<access-key>` |

+| spring.cloud.azure.cosmos.database| Your database for Spring Cloud Azure version over 4.0 | `<database-name>` |

+| spring.cloud.azure.cosmos.endpoint| Your database URI for Spring Cloud Azure version over 4.0 | `https://<database-server>.documents.azure.com:443/` |

+#### Other client types

+| Default environment variable name | Description | Example value |

+|--|-|-|

+| AZURE_COSMOS_CONNECTIONSTRING | Azure Cosmos DB for NoSQL connection string | `AccountEndpoint=https://<database-server>.documents.azure.com:443/;AccountKey=<account-key>` |

+#### Sample code

+Refer to the steps and code below to Connect to Azure Cosmos DB for NoSQL.

+#### SpringBoot client type

+| Default environment variable name | Description | Example value |

+|--|--|--|

+| spring.cloud.azure.cosmos.credential.client-id | Your client ID | `<client-ID>` |

+| spring.cloud.azure.cosmos.credential.client-secret | Your client secret | `<client-secret>` |

+| spring.cloud.azure.cosmos.profile.tenant-id | Your tenant ID | `<tenant-ID>` |

+| spring.cloud.azure.cosmos.database | Your database | `<database-name>` |

+| spring.cloud.azure.cosmos.endpoint | Your resource endpoint | `https://<database-server>.documents.azure.com:443/` |

+#### Other client types

+#### Sample code

+Refer to the steps and code below to Connect to Azure Cosmos DB for NoSQL.

+This page shows supported authentication methods and clients, and shows sample code you can use to connect Azure Web PubSub to other cloud services using Service Connector. You might still be able to connect to App Configuration using other methods. For more information about naming conventions, check the [Service Connector internals](concept-service-connector-internals.md#configuration-naming-convention) article.

+## Default environment variable names or application properties and sample code

+Use the environment variable names and application properties listed below, according to your connection's authentication type and client type, to connect compute services to Web PubSub using .NET, Java, Node.js, or Python. For each example below, replace the placeholder texts `<name>`, `<client-id>`, `<client-secret`, `<access-key>`, and `<tenant-id>` with your own resource name, client ID, client secret, access-key, and tenant ID.

+#### Sample code

+Refer to the steps and code below to connect to Azure Web PubSub using a system-assigned managed identity.

+#### Sample code

+Refer to the steps and code below to connect to Azure Web PubSub using a user-assigned managed identity.

+### Connection string

+#### Sample code

+Refer to the steps and code below to connect to Azure Web PubSub using a connection string.

+#### Sample code

+Refer to the steps and code below to connect to Azure Web PubSub using a service principal.

+* [Azure Cosmos DB for NoSQL](../service-connector/how-to-integrate-cosmos-sql.md?tabs=spring-apps#default-environment-variable-names-or-application-properties-and-sample-code)

+#### Configure APM integration on the service instance level (recommended)

+To enable APM monitoring in your VMware Spring Cloud Gateway, you can create APM configuration on the service instance level and bind it to Spring Cloud Gateway. In this way, you can conveniently configure the APM only once and bind the same APM to Spring Cloud Gateway and to your apps.

+##### [Azure portal](#tab/Azure-portal)

+Use the following steps to set up APM by using the Azure portal:

+1. Configure APM on the service instance level with the APM name, type, and properties. For more information, see the [Manage APMs on the service instance level (recommended)](./how-to-enterprise-configure-apm-integration-and-ca-certificates.md#manage-apms-on-the-service-instance-level-recommended) section of [How to configure APM integration and CA certificates](./how-to-enterprise-configure-apm-integration-and-ca-certificates.md).

+ :::image type="content" source="media/how-to-configure-enterprise-spring-cloud-gateway/service-level-apm-configure.png" alt-text="Screenshot of Azure portal that shows the Azure Spring Apps APM editor page." lightbox="media/how-to-configure-enterprise-spring-cloud-gateway/service-level-apm-configure.png":::

+1. Select **Spring Cloud Gateway** on the navigation pane, then select **APM**.

+1. Choose the APM name in the **APM reference names** list. The list includes all the APM names configured in step 1.

+1. Select **Save** to bind APM reference names to Spring Cloud Gateway. Your gateway restarts to enable APM monitoring.

+##### [Azure CLI](#tab/Azure-CLI)

+Use the following steps to set up APM in Spring Cloud Gateway by using the Azure CLI:

+1. Configure APM on the service instance level with the APM name, type, and properties. For more information, see the [Manage APMs on the service instance level (recommended)](./how-to-enterprise-configure-apm-integration-and-ca-certificates.md#manage-apms-on-the-service-instance-level-recommended) section of [How to configure APM integration and CA certificates](./how-to-enterprise-configure-apm-integration-and-ca-certificates.md).

+1. Use the following command to update gateway with APM reference names:

+ ```azurecli

+ az spring gateway update \

+ --resource-group <resource-group-name> \

+ --service <Azure-Spring-Apps-instance-name> \

+ --apms <APM-reference-name>

+ ```

+ The value for `--apms` is a space-separated list of APM reference names, which you created in step 1.

+ > [!NOTE]

+ > Spring Cloud Gateway is deprecating APM types. Use APM reference names to configure APM in a gateway.

+#### Manage APM in VMware Spring Cloud Gateway (deprecated)

+² When applied to an account with a hierarchical namespace enabled, a delete action removes empty directories. If the directory isn't empty, then the delete action removes objects that meet the policy conditions within the first lifecycle policy execution cycle. If that action results in an empty directory that also meets the policy conditions, then that directory will be removed within the next execution cycle, and so on.

+The platform runs the lifecycle policy once a day. When you configure or edit a lifecycle policy., it can take up to 24 hours for changes to go into effect and for the first execution to start. The time taken for policy actions to complete depends on the number of blobs evaluated and operated on.

+>

+> Additionally, rotating or regenerating access keys revokes shared access signatures (SAS) that are generated based on that key. After access key rotation, you must regenerate **account** and **service** SAS tokens to avoid disruptions to applications. Note that **user delegation** SAS tokens are secured with Microsoft Entra credentials and aren't affected by key rotation.

+<!--

+10/26/23: 100 (869/0)

+Prev score: 87 (850/8)

+-->

+- **Offline transfer using shippable devices** - Use physical shippable devices when you want to do offline one-time bulk data transfer. This use case involves copying data to either a disk or specialized device, and then shipping it to a secure Microsoft facility where the data is uploaded. You can purchase and ship your own disks, or you order a Microsoft-supplied disk or device. Microsoft-supplied solutions for offline transfer include Azure [Data Box](../../databox/data-box-overview.md), [Data Box Disk](../../databox/data-box-disk-overview.md), and [Data Box Heavy](../../databox/data-box-heavy-overview.md).

+- **Network Transfer** - You transfer your data to Azure over your network connection. This transfer can be done in many ways.

+ - **Hybrid migration service** - [Azure Storage Mover](../../storage-mover/service-overview.md) is a new, fully managed migration service that enables you to migrate your files and folders to Azure Storage while minimizing downtime for your workload. Azure Storage Mover is a hybrid cloud service consisting of a cloud service component and an on-premises migration agent virtual machine (VM). Storage Mover is used for migration scenarios such as *lift-and-shift*, and for cloud migrations that you repeat occasionally.

+ - **Graphical interface** - If you occasionally transfer just a few files and don't need to automate the data transfer, you can choose a graphical interface tool such as Azure Storage Explorer or a web-based exploration tool in Azure portal.

+ - **Scripted or programmatic transfer** - You can use optimized software tools that we provide or call our REST APIs/SDKs directly. The available scriptable tools are AzCopy, Azure PowerShell, and Azure CLI. For programmatic interface, use one of the SDKs for .NET, Java, Python, Node/JS, C++, Go, PHP or Ruby.

+- Is your available network bandwidth limited or nonexistent, and you want to transfer large datasets?

+You can also provide information specific to your scenario and review a list of optimal data transfer solutions. To view the list, navigate to your Azure Storage account within the Azure portal and select the **Data transfer** feature. After providing the network bandwidth in your environment, the size of the data you want to transfer, and the frequency of data transfer, you're shown a list of solutions corresponding to the information that you have provided.

+- Learn about:

+ - [Azure Storage Mover](../../storage-mover/service-overview.md), a hybrid migration service.

+ - [Cloud migration using Azure Storage Mover](../../storage-mover/migration-basics.md).

+- [Learn about Azure Data Factory](../../data-factory/copy-activity-overview.md).

+Microsoft has developed many proven practices for developing high-performance applications with Queue Storage. This checklist identifies key practices that developers can follow to optimize performance. Keep these practices in mind while you're designing your application and throughout the process.

+| &nbsp; |.NET configuration |[For .NET Framework applications, have you configured your client to use a sufficient number of concurrent connections?](#increase-default-connection-limit) |

+| &nbsp; |.NET configuration |[For .NET Framework applications, have you configured .NET to use a sufficient number of threads?](#increase-minimum-number-of-threads) |

+If your application approaches or exceeds any of the scalability targets, it might encounter increased transaction latencies or throttling. When Azure Storage throttles your application, the service begins to return 503 (`Server Busy`) or 500 (`Operation Timeout`) error codes. Avoiding these errors by staying within the limits of the scalability targets is an important part of enhancing your application's performance.

+- If your application is approaching the bandwidth targets, consider compressing data on the client side to reduce the bandwidth required to send the data to Azure Storage. While compressing data might save bandwidth and improve network performance, it can also have negative effects on performance. Evaluate the performance impact of the additional processing requirements for data compression and decompression on the client side. Keep in mind that storing compressed data can make troubleshooting more difficult because it might be more challenging to view the data using standard tools.

+- If your application is approaching the scalability targets, then make sure that you're using an exponential backoff for retries. It's best to try to avoid reaching the scalability targets by implementing the recommendations described in this article. However, using an exponential backoff for retries prevents your application from retrying rapidly, which could make throttling worse. For more information, see the [Timeout and Server Busy errors](#timeout-and-server-busy-errors) section.

+The physical network constraints of the application might have a significant impact on performance. The following sections describe some of limitations users might encounter.

+For bandwidth, the problem is often the capabilities of the client. Larger Azure instances have NICs with greater capacity, so you should consider using a larger instance or more VMs if you need higher network limits from a single machine. If you're accessing Azure Storage from an on-premises application, then the same rule applies: understand the network capabilities of the client device and the network connectivity to the Azure Storage location and either improve them as needed or design your application to work within their capabilities.

+As with any network usage, keep in mind that network conditions resulting in errors and packet loss slows effective throughput. Using Wireshark or Network Monitor might help in diagnosing this issue.

+If client applications access Azure Storage but aren't hosted within Azure, such as mobile device apps or on-premises enterprise services, then locating the storage account in a region near to those clients might reduce latency. If your clients are broadly distributed (for example, some in North America, and some in Europe), then consider using one storage account per region. This approach is easier to implement if the data the application stores is specific to individual users, and doesn't require replicating data between storage accounts.

+Typically, a web browser won't allow JavaScript in a page that is hosted by a website on one domain to perform certain operations, such as write operations, to another domain. Known as the same-origin policy, this policy prevents a malicious script on one page from obtaining access to data on another web page. However, the same-origin policy can be a limitation when building a solution in the cloud. Cross-origin resource sharing (CORS) is a browser feature that enables the target domain to communicate to the browser that it trusts requests originating in the source domain.

+For projects using .NET Framework, this section lists some quick configuration settings that you can use to make significant performance improvements. If you're using a language other than .NET, check to see if similar concepts apply in your chosen language.

+> [!NOTE]

+> This section applies to projects using .NET Framework, as connection pooling is controlled by the ServicePointManager class. .NET Core introduced a significant change around connection pool management, where connection pooling happens at the HttpClient level and the pool size is not limited by default. This means that HTTP connections are automatically scaled to satisfy your workload. Using the latest version of .NET is recommended, when possible, to take advantage of performance enhancements.

+For projects using .NET Framework, you can use the following code to increase the default connection limit (which is usually 2 in a client environment or 10 in a server environment) to 100. Typically, you should set the value to approximately the number of threads used by your application. Set the connection limit before opening any connections.

+To learn more about connection pool limits in .NET Framework, see [.NET Framework Connection Pool Limits and the new Azure SDK for .NET](https://devblogs.microsoft.com/azure-sdk/net-framework-connection-pool-limits/).

+For other programming languages, see the documentation to determine how to set the connection limit.

+### Increase minimum number of threads

+If you're using synchronous calls together with asynchronous tasks, you might want to increase the number of threads in the thread pool:

+For more information, see the [ThreadPool.SetMinThreads](/dotnet/api/system.threading.threadpool.setminthreads) method.

+While parallelism can be great for performance, be careful about using unbounded parallelism, meaning that there's no limit enforced on the number of threads or parallel requests. Be sure to limit parallel requests to upload or download data, to access multiple partitions in the same storage account, or to access multiple items in the same partition. If parallelism is unbounded, your application can exceed the client device's capabilities or the storage account's scalability targets, resulting in longer latencies and throttling.

+For best performance, always use the latest client libraries and tools provided by Microsoft. Azure Storage client libraries are available for various languages. Azure Storage also supports PowerShell and Azure CLI. Microsoft actively develops these client libraries and tools with performance in mind, keeps them up-to-date with the latest service versions, and ensures that they handle many of the proven performance practices internally. For more information, see the [Azure Storage reference documentation](./reference.md).

+Azure Storage returns an error when the service can't process a request. Understanding the errors that might be returned by Azure Storage in a given scenario is helpful for optimizing performance.

+Azure Storage might throttle your application if it approaches the scalability limits. In some cases, Azure Storage might be unable to handle a request due to some transient condition. In both cases, the service might return a 503 (`Server Busy`) or 500 (`Timeout`) error. These errors can also occur if the service is rebalancing data partitions to allow for higher throughput. The client application should typically retry the operation that causes one of these errors. However, if Azure Storage is throttling your application because it's exceeding scalability targets, or even if the service was unable to serve the request for some other reason, aggressive retries might make the problem worse. Using an exponential back off retry policy is recommended, and the client libraries default to this behavior. For example, your application might retry after 2 seconds, then 4 seconds, then 10 seconds, then 30 seconds, and then give up completely. In this way, your application significantly reduces its load on the service, rather than exacerbating behavior that could lead to throttling.

+Connectivity errors can be retried immediately, because they aren't the result of throttling and are expected to be transient.

+The client libraries handle retries with an awareness of which errors can be retried and which can't. However, if you're calling the Azure Storage REST API directly, there are some errors that you shouldn't retry. For example, a 400 (`Bad Request`) error indicates that the client application sent a request that couldn't be processed because it wasn't in the expected form. Resending this request results the same response every time, so there's no point in retrying it. If you're calling the Azure Storage REST API directly, be aware of potential errors and whether they should be retried.

+Nagle's algorithm is widely implemented across TCP/IP networks as a means to improve network performance. However, it isn't optimal in all circumstances (such as highly interactive environments). Nagle's algorithm has a negative impact on the performance of requests to Azure Table Storage, and you should disable it if possible.

+Most applications poll for messages from a queue, which can be one of the largest sources of transactions for that application. Select your polling interval wisely: polling too frequently could cause your application to approach the scalability targets for the queue. However, at 200,000 transactions for $0.01 (at the time of writing), a single processor polling once every second for a month would cost less than 15 cents so cost isn't typically a factor that affects your choice of polling interval.

+Microsoft has developed many proven practices for developing high-performance applications with Table storage. This checklist identifies key practices that developers can follow to optimize performance. Keep these practices in mind while you're designing your application and throughout the process.

+| &nbsp; |.NET configuration |[For .NET Framework applications, have you configured your client to use a sufficient number of concurrent connections?](#increase-default-connection-limit) |

+| &nbsp; |.NET configuration |[For .NET Framework applications, have you configured .NET to use a sufficient number of threads?](#increase-minimum-number-of-threads) |

+| &nbsp; |Limiting returned data |[Are you using filtering to avoid returning entities that aren't needed?](#limiting-the-amount-of-data-returned) |

+| &nbsp; |Limiting returned data |[Are you using projection to avoid returning properties that aren't needed?](#limiting-the-amount-of-data-returned) |

+| &nbsp; |Insert, update, and delete |[Have you considered storing series of data that are frequently retrieved together in a single entity as properties instead of multiple entities?](#storing-data-series-in-a-single-entity) |

+| &nbsp; |Insert, update, and delete |[For entities that are retrieved together and can be written in batches (for example, time series data), have you considered using blobs instead of tables?](#storing-structured-data-in-blobs) |

+If your application approaches or exceeds any of the scalability targets, it might encounter increased transaction latencies or throttling. When Azure Storage throttles your application, the service begins to return 503 (Server busy) or 500 (Operation timeout) error codes. Avoiding these errors by staying within the limits of the scalability targets is an important part of enhancing your application's performance.

+ While compressing data might save bandwidth and improve network performance, it can also have negative effects on performance. Evaluate the performance impact of the additional processing requirements for data compression and decompression on the client side. Keep in mind that storing compressed data can make troubleshooting more difficult because it may be more challenging to view the data using standard tools.

+Azure Storage load balances as the traffic to your storage account increases, but if the traffic exhibits sudden bursts, you might not be able to get this volume of throughput immediately. Expect to see throttling and/or timeouts during the burst as Azure Storage automatically load balances your table. Ramping up slowly generally provides better results, as the system has time to load balance appropriately.

+The physical network constraints of the application might have a significant impact on performance. The following sections describe some of limitations users might encounter.

+For bandwidth, the problem is often the capabilities of the client. Larger Azure instances have NICs with greater capacity, so you should consider using a larger instance or more VMs if you need higher network limits from a single machine. If you're accessing Azure Storage from an on premises application, then the same rule applies: understand the network capabilities of the client device and the network connectivity to the Azure Storage location and either improve them as needed or design your application to work within their capabilities.

+As with any network usage, keep in mind that network conditions resulting in errors and packet loss slows effective throughput. Using WireShark or NetMon may help in diagnosing this issue.

+If client applications will access Azure Storage but aren't hosted within Azure, such as mobile device apps or on premises enterprise services, then locating the storage account in a region near to those clients might reduce latency. If your clients are broadly distributed (for example, some in North America, and some in Europe), then consider using one storage account per region. This approach is easier to implement if the data the application stores is specific to individual users, and doesn't require replicating data between storage accounts.

+Typically, a web browser won't allow JavaScript in a page that is hosted by a website on one domain to perform certain operations, such as write operations, to another domain. Known as the same-origin policy, this policy prevents a malicious script on one page from obtaining access to data on another web page. However, the same-origin policy can be a limitation when building a solution in the cloud. Cross-origin resource sharing (CORS) is a browser feature that enables the target domain to communicate to the browser that it trusts requests originating in the source domain.

+For projects using .NET Framework, this section lists some quick configuration settings that you can use to make significant performance improvements. If you're using a language other than .NET, check to see if similar concepts apply in your chosen language.

+> [!NOTE]

+> This section applies to projects using .NET Framework, as connection pooling is controlled by the ServicePointManager class. .NET Core introduced a significant change around connection pool management, where connection pooling happens at the HttpClient level and the pool size is not limited by default. This means that HTTP connections are automatically scaled to satisfy your workload. Using the latest version of .NET is recommended, when possible, to take advantage of performance enhancements.

+For projects using .NET Framework, you can use the following code to increase the default connection limit (which is usually 2 in a client environment or 10 in a server environment) to 100. Typically, you should set the value to approximately the number of threads used by your application. Set the connection limit before opening any connections.

+To learn more about connection pool limits in .NET Framework, see [.NET Framework Connection Pool Limits and the new Azure SDK for .NET](https://devblogs.microsoft.com/azure-sdk/net-framework-connection-pool-limits/).

+For other programming languages, see the documentation to determine how to set the connection limit.

+If you're using synchronous calls together with asynchronous tasks, you might want to increase the number of threads in the thread pool:

+While parallelism can be great for performance, be careful about using unbounded parallelism, meaning that there's no limit enforced on the number of threads or parallel requests. Be sure to limit parallel requests to upload or download data, to access multiple partitions in the same storage account, or to access multiple items in the same partition. If parallelism is unbounded, your application can exceed the client device's capabilities or the storage account's scalability targets, resulting in longer latencies and throttling.

+For best performance, always use the latest client libraries and tools provided by Microsoft. Azure Storage client libraries are available for various languages. Azure Storage also supports PowerShell and Azure CLI. Microsoft actively develops these client libraries and tools with performance in mind, keeps them up-to-date with the latest service versions, and ensures that they handle many of the proven performance practices internally.

+Azure Storage returns an error when the service can't process a request. Understanding the errors returned by Azure Storage in a given scenario is helpful for optimizing performance.

+Azure Storage might throttle your application if it approaches the scalability limits. In some cases, Azure Storage might be unable to handle a request due to some transient condition. In both cases, the service might return a 503 (Server Busy) or 500 (Timeout) error. These errors can also occur if the service is rebalancing data partitions to allow for higher throughput. The client application should typically retry the operation that causes one of these errors. However, if Azure Storage is throttling your application because it's exceeding scalability targets, or even if the service was unable to serve the request for some other reason, aggressive retries might make the problem worse. Using an exponential back off retry policy is recommended, and the client libraries default to this behavior. For example, your application might retry after 2 seconds, then 4 seconds, then 10 seconds, then 30 seconds, and then give up completely. In this way, your application significantly reduces its load on the service, rather than exacerbating behavior that could lead to throttling.

+Connectivity errors can be retried immediately, because they aren't the result of throttling and are expected to be transient.

+The client libraries handle retries with an awareness of which errors can be retried and which can't. However, if you're calling the Azure Storage REST API directly, there are some errors that you shouldn't retry. For example, a 400 (Bad Request) error indicates that the client application sent a request that couldn't be processed because it wasn't in the expected form. Resending this request results the same response every time, so there's no point in retrying it. If you're calling the Azure Storage REST API directly, be aware of potential errors and whether they should be retried.

+Nagle's algorithm is widely implemented across TCP/IP networks as a means to improve network performance. However, it isn't optimal in all circumstances (such as highly interactive environments). Nagle's algorithm has a negative impact on the performance of requests to the Azure Table service, and you should disable it if possible.

+- Scalability limit: Access to entities stored in a single partition can't be load-balanced because partitions support atomic batch transactions. For this reason, the scalability target for an individual table partition is lower than for the Table service as a whole.

+- Locate data that your client application frequently updates or queries in the same logical unit of work in the same partition. For example, locate data in the same partition if your application is aggregating writes or you're performing atomic batch operations. Also, data in a single partition can be more efficiently queried in a single query than data across partitions.

+- Locate data that your client application doesn't insert, update, or query in the same logical unit of work (that is, in a single query or batch update) in separate partitions. Keep in mind that there's no limit to the number of partition keys in a single table, so having millions of partition keys isn't a problem and won't impact performance. For example, if your application is a popular website with user sign in, using the User ID as the partition key could be a good choice.

+A hot partition is one that is receiving a disproportionate percentage of the traffic to an account, and can't be load balanced because it's a single partition. In general, hot partitions are created one of two ways:

+The "Append Only" pattern is one where all (or nearly all) of the traffic to a given partition key increases and decreases according to the current time. For example, suppose that your application uses the current date as a partition key for log data. This design results in all of the inserts going to the last partition in your table, and the system can't load balance properly. If the volume of traffic to that partition exceeds the partition-level scalability target, then it results in throttling. It's better to ensure that traffic is sent to multiple partitions, to enable load balance the requests across your table.

+If your partitioning scheme results in a single partition that just has data that is far more used than other partitions, you might also see throttling as that partition approaches the scalability target for a single partition. It's better to make sure that your partition scheme results in no single partition approaching the scalability targets.

+- **Table queries:** A table query is a query that retrieves a set of entities that doesn't share a common partition key. These queries aren't efficient and you should avoid them if possible.

+Another key factor in query efficiency is the number of entities returned as compared to the number of entities scanned to find the returned set. If your application performs a table query with a filter for a property value that only 1% of the data shares, the query scans 100 entities for every one entity it returns. The table scalability targets discussed previously all relate to the number of entities scanned, and not the number of entities returned: a low query density can easily cause the Table service to throttle your application because it must scan so many entities to retrieve the entity you're looking for. For more information on how to avoid throttling, see the section titled [Denormalization](#denormalization).

+When you know that a query returns entities that you don't need in the client application, consider using a filter to reduce the size of the returned set. While the entities not returned to the client still count toward the scalability limits, your application performance improves because of the reduced network payload size and the reduced number of entities that your client application must process. Keep in mind that the scalability targets relate to the number of entities scanned, so a query that filters out many entities might still result in throttling, even if few entities are returned. For more information on making queries efficient, see the section titled [Query density](#query-density).

+Unlike working with relational databases, the proven practices for efficiently querying table data lead to denormalizing your data. That is, duplicating the same data in multiple entities (one for each key you might use to find the data) to minimize the number of entities that a query must scan to find the data the client needs, rather than having to scan large numbers of entities to find the data your application needs. For example, in an e-commerce website, you might want to find an order both by the customer ID (give me this customer's orders) and by the date (give me orders on a date). In Table Storage, it's best to store the entity (or a reference to it) twice ΓÇô once with Table Name, PK, and RK to facilitate finding by customer ID, once to facilitate finding it by the date.

+Batch transactions are known as entity group transactions in Azure Storage. All operations within an entity group transaction must be on a single partition in a single table. Where possible, use entity group transactions to perform inserts, updates, and deletes in batches. Using entity group transactions reduces the number of round trips from your client application to the server, reduces the number of billable transactions (an entity group transaction counts as a single transaction for billing purposes and can contain up to 100 storage operations), and enables atomic updates (all operations succeed or all fail within an entity group transaction). Environments with high latencies such as mobile devices benefit greatly from using entity group transactions.

+- **InsertOrMerge**: Use this operation when you want to upload a subset of the entity's properties, but aren't sure whether the entity already exists. If the entity exists, this call updates the properties included in the **Upsert** operation, and leaves all existing properties as they are, if the entity doesn't exist, it inserts the new entity. This is similar to using projection in a query, in that you only need to upload the properties that are changing.

+- **InsertOrReplace**: Use this operation when you want to upload an entirely new entity, but you aren't sure whether it already exists. Use this operation when you know that the newly uploaded entity is entirely correct because it completely overwrites the old entity. For example, you want to update the entity that stores a user's current location regardless of whether or not the application has previously stored location data for the user; the new location entity is complete, and you don't need any information from any previous entity.

+Alternatively, your application could store the CPU usage for each hour as a separate property of a single entity: to update each hour, your application can use a single **InsertOrMerge Upsert** call to update the value for the most recent hour. To plot the data, the application only needs to retrieve a single entity instead of 24, making for an efficient query. For more information on query efficiency, see the section titled [Query scope](#query-scope).

+If you're performing batch inserts and then retrieving ranges of entities together, consider using blobs instead of tables. A good example is a log file. You can batch several minutes of logs and insert them, and then retrieve several minutes of logs at a time. In this case, performance is better if you use blobs instead of tables, since you can significantly reduce the number of objects written to or read, and also possibly the number of requests that need made.