+ > If you intend to purchase a commitment tier for disconnected container usage, you will need to request separate access and select the **Commitment tier disconnected containers** pricing tier. For more information, see [disconnected containers](./containers/disconnected-containers.md).

+## Purchase a commitment tier pricing plan for disconnected containers

+Commitment plans for disconnected containers have a calendar year commitment period. These are different plans than web and connected container commitment plans. When you purchase a commitment plan, you'll be charged the full price immediately. During the commitment period, you can't change your commitment plan, however you can purchase additional unit(s) at a pro-rated price for the remaining days in the year. You have until midnight (UTC) on the last day of your commitment, to end a commitment plan.

+You can choose a different commitment plan in the **Commitment Tier pricing** settings of your resource.

+## Overage pricing for disconnected containers

+To use a disconnected container beyond the quota initially purchased with your disconnected container commitment plan, you can purchase additional quota by updating your commitment plan at any time.

+To purchase additional quota, go to your resource in Azure portal and adjust the "unit count" of your disconnected container commitment plan using the slider. This will add additional monthly quota and you will be charged a pro-rated price based on the remaining days left in the current billing cycle.

+## Purchase a commitment tier pricing plan for disconnected containers

+Commitment plans for disconnected containers have a calendar year commitment period. These are different plans than web and connected container commitment plans. When you purchase a commitment plan, you'll be charged the full price immediately. During the commitment period, you can't change your commitment plan, however you can purchase additional unit(s) at a pro-rated price for the remaining days in the year. You have until midnight (UTC) on the last day of your commitment, to end a commitment plan.

+You can choose a different commitment plan in the **Commitment Tier pricing** settings of your resource. For more information about commitment tier pricing plans, see [purchase commitment tier pricing](../commitment-tier.md).

+## Overage pricing for disconnected containers

+To use a disconnected container beyond the quota initially purchased with your disconnected container commitment plan, you can purchase additional quota by updating your commitment plan at any time.

+To purchase additional quota, go to your resource in Azure portal and adjust the "unit count" of your disconnected container commitment plan using the slider. This will add additional monthly quota and you will be charged a pro-rated price based on the remaining days left in the current billing cycle.

+* [Change or end a commitment plan](../../../ai-services/containers/disconnected-containers.md#purchase-a-commitment-tier-pricing-plan-for-disconnected-containers)

+[Make sure `kubectl` is installed and pointed to your AKS cluster.][kubectl] If you use the Azure Cloud Shell, `kubectl` is already installed.

+ Title: 'Quickstart: Create an Azure Kubernetes Service (AKS) cluster using the Bicep extensibility Kubernetes provider'

+description: Learn how to quickly create a Kubernetes cluster using the Bicep extensibility Kubernetes provider and deploy an application in Azure Kubernetes Service (AKS).

+# Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using the Bicep extensibility Kubernetes provider (Preview)

+Azure Kubernetes Service (AKS) is a managed Kubernetes service that lets you quickly deploy and manage clusters. In this quickstart, you:

+* Deploy an AKS cluster using the Bicep extensibility Kubernetes provider (preview).

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-cli.md).

+* To set up your environment for Bicep development, see [Install Bicep tools](../../azure-resource-manager/bicep/install.md). After completing the steps, you have [Visual Studio Code](https://code.visualstudio.com/) and the [Bicep extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-bicep). You also have either the latest [Azure CLI](/cli/azure/) version or the latest [Azure PowerShell module](/powershell/azure/new-azureps-module-az).

+* To create an AKS cluster using a Bicep file, you provide an SSH public key. If you need this resource, see the following section. Otherwise, skip to [Review the Bicep file](#review-the-bicep-file).

+* Make sure the identity you use to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity.md).

+* To deploy a Bicep file, you need write access on the resources you deploy and access to all operations on the `Microsoft.Resources/deployments` resource type. For example, to deploy a virtual machine, you need `Microsoft.Compute/virtualMachines/write` and `Microsoft.Resources/deployments/*` permissions. For a list of roles and permissions, see [Azure built-in roles](../../role-based-access-control/built-in-roles.md).

+2. Create an SSH key pair using the [`az sshkey create`][az-sshkey-create] Azure CLI command or the `ssh-keygen` command.

+ ```azurecli-interactive

+ # Create an SSH key pair using Azure CLI

+ az sshkey create --name "mySSHKey" --resource-group "myResourceGroup"

+ # Create an SSH key pair using ssh-keygen

+The Bicep file used to create an AKS cluster is from [Azure Quickstart Templates](https://azure.microsoft.com/resources/templates/aks/). For more AKS samples, see [AKS quickstart templates][aks-quickstart-templates].

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` in the same folder as `main.bicep` and copy in the following manifest:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+2. Open `main.bicep` in Visual Studio Code.

+3. Press <kbd>Ctrl+Shift+P</kbd> to open **Command Palette**.

+4. Search for **bicep**, and then select **Bicep: Import Kubernetes Manifest**.

+ :::image type="content" source="./media/quick-kubernetes-deploy-bicep-extensibility-kubernetes-provider/bicep-extensibility-kubernetes-provider-import-kubernetes-manifest.png" alt-text="Screenshot of Visual Studio Code import Kubernetes Manifest." lightbox="./media/quick-kubernetes-deploy-bicep-extensibility-kubernetes-provider/bicep-extensibility-kubernetes-provider-import-kubernetes-manifest.png":::

+5. Select `aks-store-quickstart.yaml` from the prompt. This process creates an `aks-store-quickstart.bicep` file in the same folder.

+6. Open `main.bicep` and add the following Bicep at the end of the file to reference the newly created `aks-store-quickstart.bicep` module:

+ module kubernetes './aks-store-quickstart.bicep' = {

+7. Save both `main.bicep` and `aks-store-quickstart.bicep`.

+### [Azure CLI](#tab/azure-cli)

+1. Create an Azure resource group using the [`az group create`][az-group-create] command.

+ ```azurecli-interactive

+ ```

+2. Deploy the Bicep file using the [`az deployment group create`][az-deployment-group-create] command.

+ ```azurecli-interactive

+### [Azure PowerShell](#tab/azure-powershell)

+1. Create an Azure resource group using the [`New-AzResourceGroup`][new-azresourcegroup] cmdlet.

+ ```azurepowershell-interactive

+2. Deploy the Bicep file using the [`New-AzResourceGroupDeployment`][new-azresourcegroupdeployment] cmdlet.

+ ```azurepowershell-interactive

+ New-AzResourceGroupDeployment -ResourceGroupName myResourceGroup -TemplateFile ./main.bicep -clusterName=<cluster-name> -dnsPrefix=<dns-prefix> -linuxAdminUsername=<linux-admin-username> -sshRSAPublicKey="<ssh-key>"

+ ```

+Provide the following values in the commands:

+* **Cluster name**: Enter a unique name for the AKS cluster, such as *myAKSCluster*.

+* **DNS prefix**: Enter a unique DNS prefix for your cluster, such as *myakscluster*.

+* **Linux Admin Username**: Enter a username to connect using SSH, such as *azureuser*.

+* **SSH RSA Public Key**: Copy and paste the *public* part of your SSH key pair (by default, the contents of *~/.ssh/id_rsa.pub*).

+It takes a few minutes to create the AKS cluster. Wait for the cluster successfully deploy before you move on to the next step.

+1. Sign in to the [Azure portal](https://portal.azure.com/).

+2. On the Azure portal menu or from the **Home** page, navigate to your AKS cluster.

+3. Under **Kubernetes resources**, select **Services and ingresses**.

+4. Find the **store-front** service and copy the value for **External IP**.

+5. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-bicep-extensibility-kubernetes-provider/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-bicep-extensibility-kubernetes-provider/aks-store-application.png":::

+## Delete the cluster

+If you don't plan on going through the following tutorials, clean up unnecessary resources to avoid Azure charges.

+* Remove the resource group, container service, and all related resources using the [`az group delete`][az-group-delete] command.

+ ```azurecli-interactive

+ az group delete --name myResourceGroup --yes --no-wait

+ ```

+* Remove the resource group, container service, and all related resources using the [`Remove-AzResourceGroup`][remove-azresourcegroup] cmdlet.

+ ```azurepowershell-interactive

+ Remove-AzResourceGroup -Name myResourceGroup

+ ```

+> The AKS cluster was created with a system-assigned managed identity, which is the default identity option used in this quickstart. The platform manages this identity so you don't need to manually remove it.

+To learn more about AKS and walk through a complete code to deployment example, continue to the Kubernetes cluster tutorial.

+> [AKS tutorial][aks-tutorial]

+[az-deployment-group-create]: /cli/azure/group/deployment#az_deployment_group_create

+[new-azresourcegroup]: /powershell/module/az.resources/new-azresourcegroup

+[new-azresourcegroupdeployment]: /powershell/module/az.resources/new-azresourcegroupdeployment

+[az-sshkey-create]: /cli/azure/sshkey#az_sshkey_create

+ Title: 'Quickstart: Create an Azure Kubernetes Service (AKS) cluster using Bicep'

+description: Learn how to quickly create a Kubernetes cluster using a Bicep file and deploy an application in Azure Kubernetes Service (AKS).

+* Deploy an AKS cluster using Bicep.

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-cli.md).

+* [!INCLUDE [About Bicep](../../../includes/resource-manager-quickstart-bicep-introduction.md)]

+* This article requires Azure CLI version 2.0.64 or later. If you're using Azure Cloud Shell, the latest version is already installed.

+* This article requires an existing Azure resource group. If you need to create one, you can use the [`az group create`][az-group-create] command.

+* If you're running PowerShell locally, install the `Az PowerShell` module. If using Azure Cloud Shell, the latest version is already installed.

+* You need the Bicep CLI. For more information, see [Azure PowerShell](../../azure-resource-manager/bicep/install.md#azure-powershell).

+* This article requires an existing Azure resource group. If you need to create one, you can use the [`New-AzAksCluster`][new-az-aks-cluster] cmdlet.

+* To create an AKS cluster using a Bicep file, you provide an SSH public key. If you need this resource, see the following section. Otherwise, skip to [Review the Bicep file](#review-the-bicep-file).

+* Make sure the identity you use to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity.md).

+* To deploy a Bicep file, you need write access on the resources you deploy and access to all operations on the `Microsoft.Resources/deployments` resource type. For example, to deploy a virtual machine, you need `Microsoft.Compute/virtualMachines/write` and `Microsoft.Resources/deployments/*` permissions. For a list of roles and permissions, see [Azure built-in roles](../../role-based-access-control/built-in-roles.md).

+ ```azurecli-interactive

+ ### [Azure CLI](#tab/azure-cli)

+ ```azurecli-interactive

+ ### [Azure PowerShell](#tab/azure-powershell)

+ ```azurepowershell-interactive

+1. Install `kubectl` locally using the [`az aks install-cli`][az-aks-install-cli] command.

+ ```azurecli-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az-aks-get-credentials] command. This command downloads credentials and configures the Kubernetes CLI to use them.

+3. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command. This command returns a list of the cluster nodes.

+ ```azurecli-interactive

+ The following example output shows the single node created in the previous steps. Make sure the node status is *Ready*.

+1. Install `kubectl` locally using the [`Install-AzAksKubectl`][install-azakskubectl] cmdlet.

+ ```azurepowershell-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet. This command downloads credentials and configures the Kubernetes CLI to use them.

+3. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command. This command returns a list of the cluster nodes.

+ The following example output shows the three nodes created in the previous steps. Make sure the node status is *Ready*.

+ ```output

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` and copy in the following manifest:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+2. Deploy the application using the [`kubectl apply`][kubectl-apply] command and specify the name of your YAML manifest.

+ kubectl apply -f aks-store-quickstart.yaml

+ The following example output shows the deployments and

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+1. Check the status of the deployed pods using the [`kubectl get pods`][kubectl-get] command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ ```console

+ kubectl get service store-front --watch

+ ```

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+ ```

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+ ```

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-bicep/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-bicep/aks-store-application.png":::

+## Delete the cluster

+If you don't plan on going through the following tutorials, clean up unnecessary resources to avoid Azure charges.

+* Remove the resource group, container service, and all related resources using the [`az group delete`][az-group-delete] command.

+ ```azurecli-interactive

+ az group delete --name myResourceGroup --yes --no-wait

+ ```

+* Remove the resource group, container service, and all related resources using the [`Remove-AzResourceGroup`][remove-azresourcegroup] cmdlet

+ ```azurepowershell-interactive

+ Remove-AzResourceGroup -Name myResourceGroup

+ ```

+ > [!NOTE]

+ > The AKS cluster was created with a system-assigned managed identity, which is the default identity option used in this quickstart. The platform manages this identity so you don't need to manually remove it.

+[az-sshkey-create]: /cli/azure/sshkey#az_sshkey_create

+description: Learn how to quickly create a Kubernetes cluster, deploy an application, and monitor performance in Azure Kubernetes Service (AKS) using Azure CLI.

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+ ```azurecli-interactive

+ ```azurecli-interactive

+ The following example output resembles successful creation of the resource group:

+ ```output

+* Create an AKS cluster using the [`az aks create`][az-aks-create] command with the `--enable-addons monitoring` and `--enable-msi-auth-for-monitoring` parameters to enable [Azure Monitor Container insights][azure-monitor-containers] with managed identity authentication (preview).

+ az aks create -g myResourceGroup -n myAKSCluster --enable-managed-identity --node-count 1 --enable-addons monitoring --enable-msi-auth-for-monitoring --generate-ssh-keys

+ ```azurecli-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az-aks-get-credentials] command. This command downloads credentials and configures the Kubernetes CLI to use them.

+ The following example output shows the single node created in the previous steps. Make sure the node status is *Ready*.

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` and copy in the following manifest:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+ ```azurecli-interactive

+ kubectl apply -f aks-store-quickstart.yaml

+ The following example output shows the deployments and

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+1. Check the status of the deployed pods using the [`kubectl get pods`][kubectl-get] command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ kubectl get service store-front --watch

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-portal/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-portal/aks-store-application.png":::

+To learn more about AKS and walk through a complete code-to-deployment example, continue to the Kubernetes cluster tutorial.

+[intro-azure-linux]: ../../azure-linux/intro-azure-linux.md

+ Title: 'Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using the Azure portal'

+# Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using Azure portal

+Azure Kubernetes Service (AKS) is a managed Kubernetes service that lets you quickly deploy and manage clusters. In this quickstart, you:

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* If you're unfamiliar with the Azure Cloud Shell, review [Overview of Azure Cloud Shell](../../cloud-shell/overview.md).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-portal.md).

+* The identity you use to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity.md).

+ :::image type="content" source="media/quick-kubernetes-deploy-portal/create-cluster-basics.png" alt-text="Screenshot of Create AKS cluster - provide basic information." lightbox="media/quick-kubernetes-deploy-portal/create-cluster-basics.png":::

+ > :::image type="content" source="media/quick-kubernetes-deploy-portal/cluster-preset-options.png" alt-text="Screenshot of Create AKS cluster - portal preset options." lightbox="media/quick-kubernetes-deploy-portal/cluster-preset-options.png":::

+1. Select **Next: Node pools** when complete.

+ - The default value for **Resource identity** is **System-assigned managed identity**. Managed identities provide an identity for applications to use when connecting to resources that support Microsoft Entra authentication. For more details about managed identities, see [What are managed identities for Azure resources?](../../active-directory/managed-identities-azure-resources/overview.md).

+ :::image type="content" source="media/quick-kubernetes-deploy-portal/aks-cloud-shell.png" alt-text="Screenshot of Open the Azure Cloud Shell in the portal option." lightbox="media/quick-kubernetes-deploy-portal/aks-cloud-shell.png":::

+ >

+1. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az-aks-get-credentials] command. This command downloads credentials and configures the Kubernetes CLI to use them.

+2. Verify the connection to your cluster using `kubectl get` to return a list of the cluster nodes.

+ ```azurecli-interactive

+ kubectl get nodes

+ ```

+ The following example output shows the single node created in the previous steps. Make sure the node status is *Ready*.

+ ```output

+ NAME STATUS ROLES AGE VERSION

+ aks-nodepool1-31718369-0 Ready agent 6m44s v1.15.10

+ ```

+1. Configure `kubectl` to connect to your Kubernetes cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet. This command downloads credentials and configures the Kubernetes CLI to use them.

+2. Verify the connection to your cluster using `kubectl get` to return a list of the cluster nodes.

+ ```azurepowershell-interactive

+ The following example output shows the single node created in the previous steps. Make sure the node status is *Ready*.

+ aks-nodepool1-31718369-0 Ready agent 6m44s v1.15.10

+## Deploy the application

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A Kubernetes manifest file defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. In the Cloud Shell, open an editor and create a file named `aks-store-quickstart.yaml`.

+2. Paste the following manifest into the editor:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+3. Deploy the application using the `kubectl apply` command and specify the name of your YAML manifest:

+ The following example output shows the deployments and

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+1. Check the status of the deployed pods using the [`kubectl get pods`][kubectl-get] command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ ```azurecli-interactive

+ kubectl get service store-front --watch

+ ```

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+ ```

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+ ```

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-portal/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-portal/aks-store-application.png":::

+## Delete the cluster

+If you don't plan on going through the following tutorials, clean up unnecessary resources to avoid Azure charges.

+1. In the Azure portal, navigate to your AKS cluster resource group.

+2. Select **Delete resource group**.

+3. Enter the name of the resource group to delete, and then select **Delete** > **Delete**.

+ > [!NOTE]

+ > The AKS cluster was created with a system-assigned managed identity. This identity is managed by the platform and doesn't require removal.

+In this quickstart, you deployed a Kubernetes cluster and deployed a simple multi-container application to it.

+To learn more about AKS and walk through a complete code-to-deployment example, continue to the Kubernetes cluster tutorial.

+[intro-azure-linux]: ../../azure-linux/intro-azure-linux.md

+ Title: 'Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using Azure PowerShell'

+# Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using Azure PowerShell

+Azure Kubernetes Service (AKS) is a managed Kubernetes service that lets you quickly deploy and manage clusters. In this quickstart, you:

+* Deploy an AKS cluster using Azure PowerShell.

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-powershell.md).

+* If you're running PowerShell locally, install the `Az PowerShell` module and connect to your Azure account using the [`Connect-AzAccount`](/powershell/module/az.accounts/Connect-AzAccount) cmdlet. For more information about installing the Az PowerShell module, see [Install Azure PowerShell][install-azure-powershell].

+* The identity you use to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity.md).

+* If you have multiple Azure subscriptions, select the appropriate subscription ID in which the resources should be billed using the

+[`Set-AzContext`](/powershell/module/az.accounts/set-azcontext) cmdlet.

+> [!NOTE]

+> If you plan to run the commands locally instead of in Azure Cloud Shell, make sure you run the commands with administrative privileges.

+An [Azure resource group][azure-resource-group] is a logical group in which Azure resources are deployed and managed. When you create a resource group, you're prompted to specify a location. This location is the storage location of your resource group metadata and where your resources run in Azure if you don't specify another region during resource creation.

+The following example creates a resource group named *myResourceGroup* in the *eastus* location.

+* Create a resource group using the [`New-AzResourceGroup`][new-azresourcegroup] cmdlet.

+ ```azurepowershell-interactive

+ New-AzResourceGroup -Name myResourceGroup -Location eastus

+ ```

+ The following example output resembles successful creation of the resource group:

+ ```output

+ ResourceGroupName : myResourceGroup

+ Location : eastus

+ ProvisioningState : Succeeded

+ Tags :

+ ResourceId : /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup

+ ```

+The following example creates a cluster named *myAKSCluster* with one node and enables a system-assigned managed identity.

+* Create an AKS cluster using the [`New-AzAksCluster`][new-azakscluster] cmdlet with the `-WorkspaceResourceId` parameter to enable [Azure Monitor container insights][azure-monitor-containers].

+ After a few minutes, the command completes and returns information about the cluster.

+ > [!NOTE]

+ > When you create an AKS cluster, a second resource group is automatically created to store the AKS resources. For more information, see [Why are two resource groups created with AKS?](../faq.md#why-are-two-resource-groups-created-with-aks)

+1. Install `kubectl` locally using the `Install-AzAksCliTool` cmdlet.

+ ```azurepowershell-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet. This command downloads credentials and configures the Kubernetes CLI to use them.

+3. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command. This command returns a list of the cluster nodes.

+ The following example output shows the single node created in the previous steps. Make sure the node status is *Ready*.

+ ```output

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` and copy in the following manifest:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+1. Deploy the application using the [`kubectl apply`][kubectl-apply] command and specify the name of your YAML manifest.

+ ```console

+ kubectl apply -f aks-store-quickstart.yaml

+ The following example output shows the deployments and

+ ```output

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+1. Check the status of the deployed pods using the [`kubectl get pods`][kubectl-get] command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ ```azurecli-interactive

+ kubectl get service store-front --watch

+ ```

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+ ```

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+ ```

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-powershell/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-powershell/aks-store-application.png":::

+If you don't plan on going through the following tutorials, clean up unnecessary resources to avoid Azure charges.

+* Remove the resource group, container service, and all related resources using the [`Remove-AzResourceGroup`][remove-azresourcegroup] cmdlet

+ ```azurepowershell-interactive

+ Remove-AzResourceGroup -Name myResourceGroup

+ ```

+ > [!NOTE]

+ > The AKS cluster was created with system-assigned managed identity (default identity option used in this quickstart), the identity is managed by the platform and doesn't require removal.

+To learn more about AKS and walk through a complete code to deployment example, continue to the Kubernetes cluster tutorial.

+[azure-resource-group]: ../../azure-resource-manager/management/overview.md

+ Title: 'Quickstart: Create an Azure Kubernetes Service (AKS) cluster using an ARM template'

+description: Learn how to quickly create a Kubernetes cluster using an Azure Resource Manager template and deploy an application in Azure Kubernetes Service (AKS).

+Azure Kubernetes Service (AKS) is a managed Kubernetes service that lets you quickly deploy and manage clusters. In this quickstart, you:

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-cli.md).

+* [!INCLUDE [About Azure Resource Manager](../../../includes/resource-manager-quickstart-introduction.md)]

+* If your environment meets the prerequisites and you're familiar with ARM templates, select **Deploy to Azure** to open the template in the Azure portal.

+ [](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fquickstarts%2Fmicrosoft.kubernetes%2Faks%2Fazuredeploy.json)

+* This article requires Azure CLI version 2.0.64 or later. If using Azure Cloud Shell, the latest version is already installed.

+* If you're running PowerShell locally, install the `Az PowerShell` module. If using Azure Cloud Shell, the latest version is already installed.

+* To create an AKS cluster using an ARM template, you provide an SSH public key. If you need this resource, see the following section. Otherwise, skip [Review the template](#review-the-template).

+* Make sure the identity you use to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity.md).

+* To deploy an ARM template, you need write access on the resources you're deploying and access to all operations on the `Microsoft.Resources/deployments` resource type. For example, to deploy a virtual machine, you need `Microsoft.Compute/virtualMachines/write` and `Microsoft.Resources/deployments/*` permissions. For a list of roles and permissions, see [Azure built-in roles](../../role-based-access-control/built-in-roles.md).

+2. Create an SSH key pair using the [`az sshkey create`][az-sshkey-create] Azure CLI command or the `ssh-keygen` command.

+ ```azurecli-interactive

+ # Create an SSH key pair using Azure CLI

+ az sshkey create --name "mySSHKey" --resource-group "myResourceGroup"

+ # Create an SSH key pair using ssh-keygen

+The resource defined in the ARM template:

+1. Select **Deploy to Azure** to sign in and open a template.

+2. On the **Basics** page, leave the default values for the *OS Disk Size GB*, *Agent Count*, *Agent VM Size*, *OS Type*, and *Kubernetes Version*, and configure the following template parameters:

+ * **Resource group**: Select **Create new**. Enter a unique name for the resource group, such as *myResourceGroup*, then select **OK**.

+ * **SSH public key source**: Select **Use existing public key**.

+ * **Key pair name**: Copy and paste the *public* part of your SSH key pair (by default, the contents of *~/.ssh/id_rsa.pub*).

+3. Select **Review + Create** > **Create**.

+1. Install `kubectl` locally using the [`az aks install-cli`][az-aks-install-cli] command.

+ ```azurecli-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az-aks-get-credentials] command. This command downloads credentials and configures the Kubernetes CLI to use them.

+3. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command. This command returns a list of the cluster nodes.

+ ```azurecli-interactive

+ The following example output shows the three nodes created in the previous steps. Make sure the node status is *Ready*.

+1. Install `kubectl` locally using the [`Install-AzAksKubectl`][install-azakskubectl] cmdlet.

+ ```azurepowershell-interactive

+2. Configure `kubectl` to connect to your Kubernetes cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet. This command downloads credentials and configures the Kubernetes CLI to use them.

+3. Verify the connection to your cluster using the [`kubectl get`][kubectl-get] command. This command returns a list of the cluster nodes.

+ The following example output shows the three nodes created in the previous steps. Make sure the node status is *Ready*.

+ ```output

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` and copy in the following manifest:

+ name: rabbitmq

+ app: rabbitmq

+ app: rabbitmq

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ cpu: 10m

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ name: order-service

+ type: ClusterIP

+ - name: http

+ port: 3000

+ targetPort: 3000

+ app: order-service

+ name: product-service

+ app: product-service

+ app: product-service

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ cpu: 1m

+ memory: 1Mi

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+ name: store-front

+ targetPort: 8080

+ app: store-front

+ type: LoadBalancer

+2. Deploy the application using the [`kubectl apply`][kubectl-apply] command and specify the name of your YAML manifest.

+ kubectl apply -f aks-store-quickstart.yaml

+ The following example output shows the deployments and

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+1. Check the status of the deployed pods using the [`kubectl get pods`][kubectl-get] command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ ```console

+ kubectl get service store-front --watch

+ ```

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+ ```

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+ ```

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-rm-template/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-rm-template/aks-store-application.png":::

+## Delete the cluster

+If you don't plan on going through the following tutorials, clean up unnecessary resources to avoid Azure charges.

+* Remove the resource group, container service, and all related resources using the [`az group delete`][az-group-delete] command.

+ ```azurecli-interactive

+ az group delete --name myResourceGroup --yes --no-wait

+ ```

+* Remove the resource group, container service, and all related resources using the [`Remove-AzResourceGroup`][remove-azresourcegroup] cmdlet

+ ```azurepowershell-interactive

+ Remove-AzResourceGroup -Name myResourceGroup

+ ```

+ > [!NOTE]

+ > The AKS cluster was created with a system-assigned managed identity, which is the default identity option used in this quickstart. The platform manages this identity so you don't need to manually remove it.

+To learn more about AKS and walk through a complete code to deployment example, continue to the Kubernetes cluster tutorial.

+[ssh-keys]: ../../virtual-machines/linux/create-ssh-keys-detailed.md

+ Title: 'Quickstart: Create an Azure Kubernetes Service (AKS) cluster using Terraform'

+description: Learn how to quickly create a Kubernetes cluster using Terraform and deploy an application in Azure Kubernetes Service (AKS).

+# Quickstart: Create an Azure Kubernetes Service (AKS) cluster using Terraform

+* Deploy an AKS cluster using Terraform.

+* Run a sample multi-container application with a group of microservices and web front ends simulating a retail scenario.

+> [!NOTE]

+> This sample application is just for demo purposes and doesn't represent all the best practices for Kubernetes applications.

+## Before you begin

+* This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)][kubernetes-concepts].

+* You need an Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* To learn more about creating a Windows Server node pool, see [Create an AKS cluster that supports Windows Server containers](quick-windows-container-deploy-cli.md).

+* [Install and configure Terraform](/azure/developer/terraform/quickstart-configure).

+* [Download kubectl](https://kubernetes.io/releases/download/).

+* Create a random value for the Azure resource group name using [random_pet](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/pet).

+* Create an Azure resource group using [azurerm_resource_group](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group).

+* Access the configuration of the AzureRM provider to get the Azure Object ID using [azurerm_client_config](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/client_config).

+* Create a Kubernetes cluster using [azurerm_kubernetes_cluster](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster).

+* Create an AzAPI resource [azapi_resource](https://registry.terraform.io/providers/Azure/azapi/latest/docs/resources/azapi_resource).

+* Create an AzAPI resource to generate an SSH key pair using [azapi_resource_action](https://registry.terraform.io/providers/Azure/azapi/latest/docs/resources/azapi_resource_action).

+1. Create a directory you can use to test the sample Terraform code and make it your current directory.

+1. Get the Azure resource group name using the following command.

+2. Display the name of your new Kubernetes cluster using the [`az aks list`](/cli/azure/aks#az-aks-list) command.

+ ```azurecli-interactive

+3. Get the Kubernetes configuration from the Terraform state and store it in a file that `kubectl` can read using the following command.

+4. Verify the previous command didn't add an ASCII EOT character using the following command.

+ * If you see `<< EOT` at the beginning and `EOT` at the end, remove these characters from the file. Otherwise, you may receive the following error message: `error: error loading config file "./azurek8s": yaml: line 2: mapping values are not allowed in this context`

+5. Set an environment variable so `kubectl` can pick up the correct config using the following command.

+6. Verify the health of the cluster using the `kubectl get nodes` command.

+* When you created the AKS cluster, monitoring was enabled to capture health metrics for both the cluster nodes and pods. These health metrics are available in the Azure portal. For more information on container health monitoring, see [Monitor Azure Kubernetes Service health](/azure/azure-monitor/insights/container-insights-overview).

+* Several key values classified as output when you applied the Terraform execution plan. For example, the host address, AKS cluster user name, and AKS cluster password are output.

+To deploy the application, you use a manifest file to create all the objects required to run the [AKS Store application](https://github.com/Azure-Samples/aks-store-demo). A [Kubernetes manifest file][kubernetes-deployment] defines a cluster's desired state, such as which container images to run. The manifest includes the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+> [!NOTE]

+> We don't recommend running stateful containers, such as Rabbit MQ, without persistent storage for production. These are used here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.

+1. Create a file named `aks-store-quickstart.yaml` and copy in the following manifest:

+ ```yaml

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: rabbitmq

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: rabbitmq

+ template:

+ metadata:

+ labels:

+ app: rabbitmq

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: rabbitmq

+ image: mcr.microsoft.com/mirror/docker/library/rabbitmq:3.10-management-alpine

+ ports:

+ - containerPort: 5672

+ name: rabbitmq-amqp

+ - containerPort: 15672

+ name: rabbitmq-http

+ env:

+ - name: RABBITMQ_DEFAULT_USER

+ value: "username"

+ - name: RABBITMQ_DEFAULT_PASS

+ value: "password"

+ resources:

+ requests:

+ cpu: 10m

+ memory: 128Mi

+ limits:

+ cpu: 250m

+ memory: 256Mi

+ volumeMounts:

+ - name: rabbitmq-enabled-plugins

+ mountPath: /etc/rabbitmq/enabled_plugins

+ subPath: enabled_plugins

+ volumes:

+ - name: rabbitmq-enabled-plugins

+ configMap:

+ name: rabbitmq-enabled-plugins

+ items:

+ - key: rabbitmq_enabled_plugins

+ path: enabled_plugins

+

+ apiVersion: v1

+ data:

+ rabbitmq_enabled_plugins: |

+ [rabbitmq_management,rabbitmq_prometheus,rabbitmq_amqp1_0].

+ kind: ConfigMap

+ metadata:

+ name: rabbitmq-enabled-plugins

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: rabbitmq

+ spec:

+ selector:

+ app: rabbitmq

+ ports:

+ - name: rabbitmq-amqp

+ port: 5672

+ targetPort: 5672

+ - name: rabbitmq-http

+ port: 15672

+ targetPort: 15672

+ type: ClusterIP

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: ghcr.io/azure-samples/aks-store-demo/order-service:latest

+ ports:

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "rabbitmq"

+ - name: ORDER_QUEUE_PORT

+ value: "5672"

+ - name: ORDER_QUEUE_USERNAME

+ value: "username"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "password"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ initContainers:

+ - name: wait-for-rabbitmq

+ image: busybox

+ command: ['sh', '-c', 'until nc -zv rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done;']

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: order-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3000

+ targetPort: 3000

+ selector:

+ app: order-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: product-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: product-service

+ template:

+ metadata:

+ labels:

+ app: product-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: product-service

+ image: ghcr.io/azure-samples/aks-store-demo/product-service:latest

+ ports:

+ - containerPort: 3002

+ resources:

+ requests:

+ cpu: 1m

+ memory: 1Mi

+ limits:

+ cpu: 1m

+ memory: 7Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: product-service

+ spec:

+ type: ClusterIP

+ ports:

+ - name: http

+ port: 3002

+ targetPort: 3002

+ selector:

+ app: product-service

+

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: store-front

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: store-front

+ template:

+ metadata:

+ labels:

+ app: store-front

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ ports:

+ - containerPort: 8080

+ name: store-front

+ env:

+ - name: VUE_APP_ORDER_SERVICE_URL

+ value: "http://order-service:3000/"

+ - name: VUE_APP_PRODUCT_SERVICE_URL

+ value: "http://product-service:3002/"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 200Mi

+ limits:

+ cpu: 1000m

+ memory: 512Mi

+

+ apiVersion: v1

+ kind: Service

+ metadata:

+ name: store-front

+ spec:

+ ports:

+ - port: 80

+ targetPort: 8080

+ selector:

+ app: store-front

+ type: LoadBalancer

+ ```

+ For a breakdown of YAML manifest files, see [Deployments and YAML manifests](../concepts-clusters-workloads.md#deployments-and-yaml-manifests).

+2. Deploy the application using the `kubectl apply` command and specify the name of your YAML manifest.

+ ```console

+ kubectl apply -f aks-store-quickstart.yaml

+ ```

+ The following example output shows the deployments and

+ ```output

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+ ```

+### Test the application

+When the application runs, a Kubernetes service exposes the application front end to the internet. This process can take a few minutes to complete.

+1. Check the status of the deployed pods using the `kubectl get pods` command. Make all pods are `Running` before proceeding.

+2. Check for a public IP address for the store-front application. Monitor progress using the `kubectl get service` command with the `--watch` argument.

+ ```azurecli-interactive

+ kubectl get service store-front --watch

+ ```

+ The **EXTERNAL-IP** output for the `store-front` service initially shows as *pending*:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 <pending> 80:30025/TCP 4h4m

+3. Once the **EXTERNAL-IP** address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

+ store-front LoadBalancer 10.0.100.10 20.62.159.19 80:30025/TCP 4h5m

+4. Open a web browser to the external IP address of your service to see the Azure Store app in action.

+ :::image type="content" source="media/quick-kubernetes-deploy-terraform/aks-store-application.png" alt-text="Screenshot of AKS Store sample application." lightbox="media/quick-kubernetes-deploy-terraform/aks-store-application.png":::

+1. Get the service principal ID using the following command.

+ ```azurecli-interactive

+1. Delete the service principal using the [`az ad sp delete`](/cli/azure/ad/sp#az-ad-sp-delete) command.

+ ```azurecli-interactive

+[Troubleshoot common problems when using Terraform on Azure](/azure/developer/terraform/troubleshoot).

+> [Learn more about using AKS.](/azure/aks)

+<!-- LINKS - internal -->

+[kubernetes-concepts]: ../concepts-clusters-workloads.md

+[kubernetes-deployment]: ../concepts-clusters-workloads.md#deployments-and-yaml-manifests

+[intro-azure-linux]: ../../azure-linux/intro-azure-linux.md

+ Title: Kubernetes on Azure tutorial - Deploy an application to Azure Kubernetes Service (AKS)

+description: In this Azure Kubernetes Service (AKS) tutorial, you deploy a multi-container application to your cluster using images stored in Azure Container Registry.

+# Tutorial - Deploy an application to Azure Kubernetes Service (AKS)

+Kubernetes provides a distributed platform for containerized applications. You build and deploy your own applications and services into a Kubernetes cluster and let the cluster manage the availability and connectivity.

+In this tutorial, part four of seven, you deploy a sample application into a Kubernetes cluster. You learn how to:

+>

+> With AKS, you can use the following approaches for configuration management:

+>

+> * **GitOps**: Enables declarations of your cluster's state to automatically apply to the cluster. To learn how to use GitOps to deploy an application with an AKS cluster, see the [prerequisites for Azure Kubernetes Service clusters][gitops-flux-tutorial-aks] in the [GitOps with Flux v2][gitops-flux-tutorial] tutorial.

+>

+> * **DevOps**: Enables you to build, test, and deploy with continuous integration (CI) and continuous delivery (CD). To see examples of how to use DevOps to deploy an application with an AKS cluster, see [Build and deploy to AKS with Azure Pipelines](./devops-pipeline.md) or [GitHub Actions for deploying to Kubernetes](./kubernetes-action.md).

+In previous tutorials, you packaged an application into a container image, uploaded the image to Azure Container Registry, and created a Kubernetes cluster. To complete this tutorial, you need the pre-created `aks-store-quickstart.yaml` Kubernetes manifest file. This file download was included with the application source code in a previous tutorial. Make sure you cloned the repo and changed directories into the cloned repo. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

+This tutorial requires Azure CLI version 2.34.1 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+This tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+In these tutorials, your Azure Container Registry (ACR) instance stores the container images for the sample application. To deploy the application, you must update the image names in the Kubernetes manifest file to include your ACR login server name.

+1. Get your login server address using the [`az acr list`][az-acr-list] command and query for your login server.

+ ```azurecli-interactive

+ az acr list --resource-group myResourceGroup --query "[].{acrLoginServer:loginServer}" --output table

+ ```

+2. Make sure you're in the cloned *aks-store-demo* directory, and then open the manifest file with a text editor, such as `vi`:

+ ```azurecli-interactive

+ vi aks-store-quickstart.yaml

+ ```

+3. Update the `image` property for the containers by replacing *ghcr.io/azure-samples* with your ACR login server name.

+ ```yaml

+ containers:

+ ...

+ - name: order-service

+ image: <acrName>.azurecr.io/aks-store-demo/order-service:latest

+ ...

+ - name: product-service

+ image: <acrName>.azurecr.io/aks-store-demo/product-service:latest

+ ...

+ - name: store-front

+ image: <acrName>.azurecr.io/aks-store-demo/store-front:latest

+ ...

+ ```

+4. Save and close the file. In `vi`, use `:wq`.

+### [Azure PowerShell](#tab/azure-powershell)

+1. Get your login server address using the [`Get-AzContainerRegistry`][get-azcontainerregistry] cmdlet and query for your login server. Make sure you replace `<acrName>` with the name of your ACR instance.

+ ```azurepowershell-interactive

+ (Get-AzContainerRegistry -ResourceGroupName myResourceGroup -Name <acrName>).LoginServer

+ ```

+2. Make sure you're in the cloned *aks-store-demo* directory, and then open the manifest file with a text editor, such as `vi`:

+ ```azurepowershell-interactive

+ vi aks-store-quickstart.yaml

+ ```

+3. Update the `image` property for the containers by replacing *ghcr.io/azure-samples* with your ACR login server name.

+ ```yaml

+ containers:

+ ...

+ - name: order-service

+ image: <acrName>.azurecr.io/aks-store-demo/order-service:latest

+ ...

+ - name: product-service

+ image: <acrName>.azurecr.io/aks-store-demo/product-service:latest

+ ...

+ - name: store-front

+ image: <acrName>.azurecr.io/aks-store-demo/store-front:latest

+ ...

+ ```

+4. Save and close the file. In `vi`, use `:wq`.

+## Deploy the application

+* Deploy the application using the [`kubectl apply`][kubectl-apply] command, which parses the manifest file and creates the defined Kubernetes objects.

+ ```console

+ kubectl apply -f aks-store-quickstart.yaml

+ ```

+ The following example output shows the resources successfully created in the AKS cluster:

+ ```output

+ deployment.apps/rabbitmq created

+ service/rabbitmq created

+ deployment.apps/order-service created

+ service/order-service created

+ deployment.apps/product-service created

+ service/product-service created

+ deployment.apps/store-front created

+ service/store-front created

+ ```

+1. Monitor progress using the [`kubectl get service`][kubectl-get] command with the `--watch` argument.

+ ```console

+ kubectl get service store-front --watch

+ ```

+ Initially, the `EXTERNAL-IP` for the *store-front* service shows as *pending*.

+ ```output

+ store-front LoadBalancer 10.0.34.242 <pending> 80:30676/TCP 5s

+ ```

+2. When the `EXTERNAL-IP` address changes from *pending* to an actual public IP address, use `CTRL-C` to stop the `kubectl` watch process.

+ The following example output shows a valid public IP address assigned to the service:

+ ```output

+ store-front LoadBalancer 10.0.34.242 52.179.23.131 80:30676/TCP 67s

+ ```

+3. View the application in action by opening a web browser to the external IP address of your service.

+In this tutorial, you deployed a sample Azure application to a Kubernetes cluster in AKS. You learned how to:

+In the next tutorial, you learn how to use PaaS services for stateful workloads in Kubernetes.

+> Use PaaS services for stateful workloads in AKS

+[aks-tutorial-paas]: ./tutorial-kubernetes-paas-services.md

+ Title: Kubernetes on Azure tutorial - Deploy an Azure Kubernetes Service (AKS) cluster

+description: In this Azure Kubernetes Service (AKS) tutorial, you create an AKS cluster and use kubectl to connect to the Kubernetes main node.

+# Tutorial - Deploy an Azure Kubernetes Service (AKS) cluster

+Kubernetes provides a distributed platform for containerized applications. With Azure Kubernetes Service (AKS), you can quickly create a production ready Kubernetes cluster.

+In this tutorial, part three of seven, you deploy a Kubernetes cluster in AKS. You learn how to:

+> * Deploy an AKS cluster that can authenticate to an Azure Container Registry (ACR).

+In previous tutorials, you created a container image and uploaded it to an ACR instance. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

+This tutorial requires Azure CLI version 2.0.53 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+### [Azure PowerShell](#tab/azure-powershell)

+This tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+## Create an AKS cluster

+AKS clusters can use [Kubernetes role-based access control (Kubernetes RBAC)][k8s-rbac], which allows you to define access to resources based on roles assigned to users. Permissions are combined when users are assigned multiple roles. Permissions can be scoped to either a single namespace or across the whole cluster. For more information, see [Control access to cluster resources using Kubernetes RBAC and Azure Active Directory identities in AKS][aks-k8s-rbac].

+For information about AKS resource limits and region availability, see [Quotas, virtual machine size restrictions, and region availability in AKS][quotas-skus-regions].

+> To ensure your cluster operates reliably, you should run at least two nodes.

+### [Azure CLI](#tab/azure-cli)

+To allow an AKS cluster to interact with other Azure resources, the Azure platform automatically creates a cluster identity. In this example, the cluster identity is [granted the right to pull images][container-registry-integration] from the ACR instance you created in the previous tutorial. To execute the command successfully, you need to have an **Owner** or **Azure account administrator** role in your Azure subscription.

+* Create an AKS cluster using the [`az aks create`][az aks create] command. The following example creates a cluster named *myAKSCluster* in the resource group named *myResourceGroup*. This resource group was created in the [previous tutorial][aks-tutorial-prepare-acr] in the *eastus* region.

+ ```azurecli-interactive

+ az aks create \

+ --resource-group myResourceGroup \

+ --name myAKSCluster \

+ --node-count 2 \

+ --generate-ssh-keys \

+ --attach-acr <acrName>

+ ```

+ > [!NOTE]

+ > If you already generated SSH keys, you may encounter an error similar to `linuxProfile.ssh.publicKeys.keyData is invalid`. To proceed, retry the command without the `--generate-ssh-keys` parameter.

+### [Azure PowerShell](#tab/azure-powershell)

+To allow an AKS cluster to interact with other Azure resources, the Azure platform automatically creates a cluster identity. In this example, the cluster identity is [granted the right to pull images][container-registry-integration] from the ACR instance you created in the previous tutorial. To execute the command successfully, you need to have an **Owner** or **Azure account administrator** role in your Azure subscription.

+* Create an AKS cluster using the [`New-AzAksCluster`][new-azakscluster] cmdlet. The following example creates a cluster named *myAKSCluster* in the resource group named *myResourceGroup*. This resource group was created in the [previous tutorial][aks-tutorial-prepare-acr] in the *eastus* region.

+ ```azurepowershell-interactive

+ New-AzAksCluster -ResourceGroupName myResourceGroup -Name myAKSCluster -NodeCount 2 -GenerateSshKey -AcrNameToAttach <acrName>

+ ```

+ > [!NOTE]

+ > If you already generated SSH keys, you may encounter an error similar to `linuxProfile.ssh.publicKeys.keyData is invalid`. To proceed, retry the command without the `-GenerateSshKey` parameter.

+You use the Kubernetes CLI, [`kubectl`][kubectl], to connect to your Kubernetes cluster. If you use the Azure Cloud Shell, `kubectl` is already installed. If you're running the commands locally, you can use the Azure CLI or Azure PowerShell to install `kubectl`.

+* Install `kubectl` locally using the [`az aks install-cli`][az aks install-cli] command.

+ ```azurecli-interactive

+ az aks install-cli

+ ```

+* Install `kubectl` locally using the [`Install-AzAksCliTool`][install-azaksclitool] cmdlet.

+ ```azurepowershell-interactive

+ Install-AzAksCliTool

+ ```

+1. Configure `kubectl` to connect to your Kubernetes cluster using the [`az aks get-credentials`][az aks get-credentials] command. The following example gets credentials for the AKS cluster named *myAKSCluster* in *myResourceGroup*.

+ ```azurecli-interactive

+ az aks get-credentials --resource-group myResourceGroup --name myAKSCluster

+ ```

+2. Verify connection to your cluster using the [`kubectl get nodes`][kubectl-get] command, which returns a list of cluster nodes.

+ ```azurecli-interactive

+ kubectl get nodes

+ ```

+ The following example output shows a list of the cluster nodes:

+ ```output

+ NAME STATUS ROLES AGE VERSION

+ aks-nodepool1-19366578-vmss000002 Ready agent 47h v1.25.6

+ aks-nodepool1-19366578-vmss000003 Ready agent 47h v1.25.6

+ ```

+### [Azure PowerShell](#tab/azure-powershell)

+1. Configure `kubectl` to connect to your Kubernetes cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet. The following example gets credentials for the AKS cluster named *myAKSCluster* in *myResourceGroup*.

+ ```azurepowershell-interactive

+ Import-AzAksCredential -ResourceGroupName myResourceGroup -Name myAKSCluster

+ ```

+2. Verify connection to your cluster using the [`kubectl get nodes`][kubectl-get] command, which returns a list of cluster nodes.

+ ```azurepowershell-interactive

+ kubectl get nodes

+ ```

+ The following example output shows a list of the cluster nodes:

+ ```output

+ NAME STATUS ROLES AGE VERSION

+ aks-nodepool1-19366578-vmss000002 Ready agent 47h v1.25.6

+ aks-nodepool1-19366578-vmss000003 Ready agent 47h v1.25.6

+ ```

+In the next tutorial, you learn how to deploy an application to your cluster.

+[install-azaksclitool]: /powershell/module/az.aks/install-azaksclitool

+[aks-k8s-rbac]: azure-ad-rbac.md

+ Title: Kubernetes on Azure tutorial - Use PaaS services with an Azure Kubernetes Service (AKS) cluster

+description: In this Azure Kubernetes Service (AKS) tutorial, you learn how to use the Azure Service Bus service with your AKS cluster.

+#Customer intent: As a developer, I want to learn how to use PaaS services with an Azure Kubernetes Service (AKS) cluster so that I can deploy and manage my applications.

+# Tutorial - Use PaaS services with an Azure Kubernetes Service (AKS) cluster

+With Kubernetes, you can use PaaS services, such as [Azure Service Bus][azure-service-bus], to develop and run your applications.

+In this tutorial, part five of seven, you create an Azure Service Bus namespace and queue to test your application. You learn how to:

+> [!div class="checklist"]

+>

+> * Create an Azure Service Bus namespace and queue.

+> * Update the Kubernetes manifest file to use the Azure Service Bus queue.

+> * Test the updated application by placing an order.

+## Before you begin

+In previous tutorials, you packaged an application into a container image, uploaded the image to Azure Container Registry, created a Kubernetes cluster, and deployed an application. To complete this tutorial, you need the pre-created `aks-store-quickstart.yaml` Kubernetes manifest file. This file download was included with the application source code in a previous tutorial. Make sure you cloned the repo and changed directories into the cloned repo. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

+### [Azure CLI](#tab/azure-cli)

+This tutorial requires Azure CLI version 2.34.1 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+### [Azure PowerShell](#tab/azure-powershell)

+This tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+## Create environment variables

+### [Azure CLI](#tab/azure-cli)

+* Create the following environment variables to use for the commands in this tutorial:

+ ```azurecli-interactive

+ LOC_NAME=eastus

+ RAND=$RANDOM

+ RG_NAME=myResourceGroup

+ AKS_NAME=myAKSCluster

+ SB_NS=sb-store-demo-$RAND

+ ```

+### [Azure PowerShell](#tab/azure-powershell)

+* Create the following environment variables to use for the commands in this tutorial:

+ ```azurepowershell-interactive

+ $LOC_NAME="eastus"

+ $rand=New-Object System.Random

+ $RAND=$rand.Next()

+ $RG_NAME="myResourceGroup"

+ $AKS_NAME="myAKSCluster"

+ $SB_NS="sb-store-demo-$RAND"

+ ```

+## Create Azure Service Bus namespace and queue

+In previous tutorials, you used a RabbitMQ container to store orders submitted by the `order-service`. In this tutorial, you use an Azure Service Bus namespace to provide a scoping container for the Service Bus resources within the application. You also use an Azure Service bus queue to send and receive messages between the application components. For more information on Azure Service Bus, see [Create an Azure Service Bus namespace and queue](../service-bus-messaging/service-bus-quickstart-cli.md).

+### [Azure CLI](#tab/azure-cli)

+1. Create an Azure Service Bus namespace using the [`az servicebus namespace create`][az-servicebus-namespace-create] command.

+ ```azurecli-interactive

+ az servicebus namespace create -n $SB_NS -g $RG_NAME -l $LOC_NAME

+ ```

+2. Create an Azure Service Bus queue using the [`az servicebus queue create`][az-servicebus-queue-create] command.

+ ```azurecli-interactive

+ az servicebus queue create -n orders -g $RG_NAME --namespace-name $SB_NS -g $RG_NAME

+ ```

+3. Create an Azure Service Bus authorization rule using the [`az servicebus queue authorization-rule create`][az-servicebus-queue-authorization-rule-create] command.

+ ```azurecli-interactive

+ az servicebus queue authorization-rule create \

+ --name sender \

+ --namespace-name $SB_NS \

+ --resource-group $RG_NAME \

+ --queue-name orders \

+ --rights Send

+ ```

+4. Get the Azure Service Bus credentials for later use using the [`az servicebus namespace show`][az-servicebus-namespace-show] and [`az servicebus queue authorization-rule keys list`][az-servicebus-queue-authorization-rule-keys-list] commands.

+ ```azurecli-interactive

+ az servicebus namespace show --name $SB_NS --resource-group $RG_NAME --query name -o tsv

+ az servicebus queue authorization-rule keys list --namespace-name $SB_NS --resource-group $RG_NAME --queue-name orders --name sender --query primaryKey -o tsv

+ ```

+### [Azure PowerShell](#tab/azure-powershell)

+1. Create an Azure Service Bus namespace using the [`New-AzServiceBusNamespace`][new-az-service-bus-namespace] cmdlet.

+ ```azurepowershell-interactive

+ New-AzServiceBusNamespace -Name $SB_NS -ResourceGroupName $RG_NAME -Location $LOC_NAME

+ ```

+2. Create an Azure Service Bus queue using the [`New-AzServiceBusQueue`][new-az-service-bus-queue] cmdlet.

+ ```azurepowershell-interactive

+ New-AzServiceBusQueue -Name orders -ResourceGroupName $RG_NAME -NamespaceName $SB_NS

+ ```

+3. Create an Azure Service Bus authorization rule using the [`New-AzServiceBusAuthorizationRule`][new-az-service-bus-authorization-rule] cmdlet.

+ ```azurepowershell-interactive

+ New-AzServiceBusAuthorizationRule `

+ -Name sender `

+ -NamespaceName $SB_NS `

+ -ResourceGroupName $RG_NAME `

+ -QueueName orders `

+ -Rights Send

+ ```

+4. Get the Azure Service Bus credentials for later use using the [`Get-AzServiceBusNamespace`][get-az-service-bus-namespace] and [`Get-AzServiceBusKey`][get-az-service-bus-key] cmdlets.

+ ```azurepowershell-interactive

+ (Get-AzServiceBusNamespace -Name $SB_NS -ResourceGroupName $RG_NAME).Name

+ (Get-AzServiceBusKey -NamespaceName $SB_NS -ResourceGroupName $RG_NAME -Name sender -QueueName orders).PrimaryKey

+ ```

+## Update Kubernetes manifest file

+### [Azure CLI](#tab/azure-cli)

+1. Configure `kubectl` to connect to your cluster using the [`az aks get-credentials`][az-aks-get-credentials] command.

+ ```azurecli-interactive

+ az aks get-credentials --resource-group myResourceGroup --name myAKSCluster

+ ```

+2. Open the `aks-store-quickstart.yaml` file in a text editor.

+3. Remove the existing `rabbitmq` Deployment, ConfigMap, and Service sections and replace the existing `order-service` Deployment section with the following content:

+ ```yaml

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: <REPLACE_WITH_YOUR_ACR_NAME>.azurecr.io/aks-store-demo/order-service:latest

+ ports:

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "<REPLACE_WITH_YOUR_SB_NS_HOSTNAME>" # Example: sb-store-demo-123456.servicebus.windows.net

+ - name: ORDER_QUEUE_PORT

+ value: "5671"

+ - name: ORDER_QUEUE_TRANSPORT

+ value: "tls"

+ - name: ORDER_QUEUE_USERNAME

+ value: "sender"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "<REPLACE_WITH_YOUR_SB_SENDER_PASSWORD>"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ ```

+ > [!NOTE]

+ > Directly adding sensitive information, such as API keys, to your Kubernetes manifest files isn't secure and may accidentally get committed to code repositories. We added it here for simplicity. For production workloads, use [Managed Identity](./use-managed-identity.md) to authenticate with Azure Service Bus or store your secrets in [Azure Key Vault](./csi-secrets-store-driver.md).

+4. Save and close the updated `aks-store-quickstart.yaml` file.

+### [Azure PowerShell](#tab/azure-powershell)

+1. Configure `kubectl` to connect to your cluster using the [`Import-AzAksCredential`][import-azakscredential] cmdlet.

+ ```azurepowershell-interactive

+ Import-AzAksCredential -ResourceGroupName myResourceGroup -Name myAKSCluster

+ ```

+2. Open the `aks-store-quickstart.yaml` file in a text editor.

+3. Remove the existing `rabbitmq` Deployment, ConfigMap, and Service sections and replace the existing `order-service` Deployment section with the following content:

+ ```YAML

+ apiVersion: apps/v1

+ kind: Deployment

+ metadata:

+ name: order-service

+ spec:

+ replicas: 1

+ selector:

+ matchLabels:

+ app: order-service

+ template:

+ metadata:

+ labels:

+ app: order-service

+ spec:

+ nodeSelector:

+ "kubernetes.io/os": linux

+ containers:

+ - name: order-service

+ image: <REPLACE_WITH_YOUR_ACR_NAME>.azurecr.io/aks-store-demo/order-service:latest

+ ports:

+ - containerPort: 3000

+ env:

+ - name: ORDER_QUEUE_HOSTNAME

+ value: "<REPLACE_WITH_YOUR_SB_NS_HOSTNAME>" # Example: sb-store-demo-123456.servicebus.windows.net

+ - name: ORDER_QUEUE_PORT

+ value: "5671"

+ - name: ORDER_QUEUE_TRANSPORT

+ value: "tls"

+ - name: ORDER_QUEUE_USERNAME

+ value: "sender"

+ - name: ORDER_QUEUE_PASSWORD

+ value: "<REPLACE_WITH_YOUR_SB_SENDER_PASSWORD>"

+ - name: ORDER_QUEUE_NAME

+ value: "orders"

+ - name: FASTIFY_ADDRESS

+ value: "0.0.0.0"

+ resources:

+ requests:

+ cpu: 1m

+ memory: 50Mi

+ limits:

+ cpu: 75m

+ memory: 128Mi

+ ```

+ > [!NOTE]

+ > Directly adding sensitive information, such as API keys, to your Kubernetes manifest files isn't secure and may accidentally get committed to code repositories. We added it here for simplicity. For production workloads, use [Managed Identity](./use-managed-identity.md) to authenticate with Azure Service Bus or store your secrets in [Azure Key Vault](./csi-secrets-store-driver.md).

+4. Save and close the updated `aks-store-quickstart.yaml` file.

+## Deploy the updated application

+* Deploy the updated application using the `kubectl apply` command.

+ ```console

+ kubectl apply -f aks-store-quickstart.yaml

+ ```

+ The following example output shows the successfully updated resources:

+ ```output

+ deployment.apps/order-service configured

+ service/order-service unchanged

+ deployment.apps/product-service unchanged

+ service/product-service unchanged

+ deployment.apps/store-front configured

+ service/store-front unchanged

+ ```

+## Test the application

+### Place a sample order

+1. Get the external IP address of the `store-front` service using the `kubectl get service` command.

+ ```console

+ kubectl get service store-front

+ ```

+2. Navigate to the external IP address of the `store-front` service in your browser.

+3. Place an order by choosing a product and selecting **Add to cart**.

+4. Select **Cart** to view your order, and then select **Checkout**.

+### View the order in the Azure Service Bus queue

+1. Navigate to the Azure portal and open the Azure Service Bus namespace you created earlier.

+2. Under **Entities**, select **Queues**, and then select the **orders** queue.

+3. In the **orders** queue, select **Service Bus Explorer**.

+4. Select **Peek from start** to view the order you submitted.

+## Next steps

+In this tutorial, you used Azure Service Bus to update and test the sample application. You learned how to:

+> [!div class="checklist"]

+>

+> * Create an Azure Service Bus namespace and queue.

+> * Update the Kubernetes manifest file to use the Azure Service Bus queue.

+> * Test the updated application by placing an order.

+In the next tutorial, you learn how to scale an application in AKS.

+> [!div class="nextstepaction"]

+> [Scale applications in AKS][aks-tutorial-scale]

+<!-- LINKS - external -->

+<!-- LINKS - internal -->

+[aks-tutorial-prepare-app]: ./tutorial-kubernetes-prepare-app.md

+[azure-cli-install]: /cli/azure/install-azure-cli

+[azure-powershell-install]: /powershell/azure/install-az-ps

+[aks-tutorial-scale]: ./tutorial-kubernetes-scale.md

+[azure-service-bus]: ../service-bus-messaging/service-bus-messaging-overview.md

+[az-servicebus-namespace-create]: /cli/azure/servicebus/namespace#az_servicebus_namespace_create

+[az-servicebus-queue-create]: /cli/azure/servicebus/queue#az_servicebus_queue_create

+[az-servicebus-queue-authorization-rule-create]: /cli/azure/servicebus/queue/authorization-rule#az_servicebus_queue_authorization_rule_create

+[az-servicebus-namespace-show]: /cli/azure/servicebus/namespace#az_servicebus_namespace_show

+[az-servicebus-queue-authorization-rule-keys-list]: /cli/azure/servicebus/queue/authorization-rule/keys#az_servicebus_queue_authorization_rule_keys_list

+[new-az-service-bus-namespace]: /powershell/module/az.servicebus/new-azservicebusnamespace

+[new-az-service-bus-queue]: /powershell/module/az.servicebus/new-azservicebusqueue

+[new-az-service-bus-authorization-rule]: /powershell/module/az.servicebus/new-azservicebusauthorizationrule

+[get-az-service-bus-namespace]: /powershell/module/az.servicebus/get-azservicebusnamespace

+[get-az-service-bus-key]: /powershell/module/az.servicebus/get-azservicebuskey

+[import-azakscredential]: /powershell/module/az.aks/import-azakscredential

+[az-aks-get-credentials]: /cli/azure/aks#az_aks_get_credentials

+ Title: Kubernetes on Azure tutorial - Create an Azure Container Registry and build images

+description: In this Azure Kubernetes Service (AKS) tutorial, you create an Azure Container Registry instance and upload sample application container images.

+# Tutorial - Create an Azure Container Registry (ACR) and build images

+Azure Container Registry (ACR) is a private registry for container images. A private container registry allows you to securely build and deploy your applications and custom code.

+In this tutorial, part two of seven, you deploy an ACR instance and push a container image to it. You learn how to:

+> * Create an ACR instance.

+> * Use [ACR Tasks][acr-tasks] to build and push container images to ACR.

+> * View images in your registry.

+In the [previous tutorial][aks-tutorial-prepare-app], you used Docker to create a container image for a simple Azure Store Front application. If you haven't created the Azure Store Front app image, return to [Tutorial 1 - Prepare an application for AKS][aks-tutorial-prepare-app].

+This tutorial requires Azure CLI version 2.0.53 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+This tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+Before creating an ACR instance, you need a resource group. An Azure resource group is a logical container into which you deploy and manage Azure resources.

+1. Create a resource group using the [`az group create`][az-group-create] command.

+ ```azurecli-interactive

+ az group create --name myResourceGroup --location eastus

+ ```

+2. Create an ACR instance using the [`az acr create`][az-acr-create] command and provide your own unique registry name. The registry name must be unique within Azure and contain 5-50 alphanumeric characters. The rest of this tutorial uses `<acrName>` as a placeholder for the container registry name. The *Basic* SKU is a cost-optimized entry point for development purposes that provides a balance of storage and throughput.

+ ```azurecli-interactive

+ az acr create --resource-group myResourceGroup --name <acrName> --sku Basic

+ ```

+1. Create a resource group using the [`New-AzResourceGroup`][new-azresourcegroup] cmdlet.

+ ```azurepowershell-interactive

+ New-AzResourceGroup -Name myResourceGroup -Location eastus

+ ```

+2. Create an ACR instance using the [`New-AzContainerRegistry`][new-azcontainerregistry] cmdlet and provide your own unique registry name. The registry name must be unique within Azure and contain 5-50 alphanumeric characters. The rest of this tutorial uses `<acrName>` as a placeholder for the container registry name. The *Basic* SKU is a cost-optimized entry point for development purposes that provides a balance of storage and throughput.

+ ```azurepowershell-interactive

+ New-AzContainerRegistry -ResourceGroupName myResourceGroup -Name <acrName> -Location eastus -Sku Basic

+ ```

+## Build and push container images to registry

+* Build and push the images to your ACR using the [`az acr build`][az-acr-build] command.

+ > [!NOTE]

+ > In the following example, we don't build the `rabbitmq` image. This image is available from the Docker Hub public repository and doesn't need to be built or pushed to your ACR instance.

+ ```azurecli-interactive

+ az acr build --registry <acrName> --image aks-store-demo/product-service:latest ./src/product-service/

+ az acr build --registry <acrName> --image aks-store-demo/order-service:latest ./src/order-service/

+ az acr build --registry <acrName> --image aks-store-demo/store-front:latest ./src/store-front/

+ ```

+* View the images in your ACR instance using the [`az acr repository list`][az-acr-repository-list] command.

+ ```azurecli-interactive

+ az acr repository list --name <acrName> --output table

+ ```

+ The following example output lists the available images in your registry:

+ ```output

+ Result

+ -

+ aks-store-demo/product-service

+ aks-store-demo/order-service

+ aks-store-demo/store-front

+ ```

+* View the images in your ACR instance using the [`Get-AzContainerRegistryRepository`][get-azcontainerregistryrepository] cmdlet.

+ ```azurepowershell-interactive

+ Get-AzContainerRegistryRepository -RegistryName <acrName>

+ ```

+ The following example output lists the available images in your registry:

+ ```output

+ aks-store-demo/productservice

+ aks-store-demo/orderservice

+ aks-store-demo/storefront

+ ```

+In this tutorial, you created an ACR and pushed images to it to use in an AKS cluster. You learned how to:

+> * Create an ACR instance.

+> * Use [ACR Tasks][acr-tasks] to build and push container images to ACR.

+> * View images in your registry.

+In the next tutorial, you learn how to deploy a Kubernetes cluster in Azure.

+[get-azcontainerregistryrepository]: /powershell/module/az.containerregistry/get-azcontainerregistryrepository

+[acr-tasks]: ../container-registry/container-registry-tasks-overview.md

+[az-acr-build]: /cli/azure/acr#az_acr_build

+ Title: Kubernetes on Azure tutorial - Prepare an application for Azure Kubernetes Service (AKS)

+# Tutorial - Prepare an application for Azure Kubernetes Service (AKS)

+> * Clone a sample application source from GitHub.

+> * Create a container image from the sample application source.

+> * Test the multi-container application in a local Docker environment.

+> Azure Cloud Shell doesn't include the Docker components required to complete every step in these tutorials. Therefore, we recommend using a full Docker development environment.

+The [sample application][sample-application] used in this tutorial is a basic store front app including the following Kubernetes deployments and

+* **Store front**: Web application for customers to view products and place orders.

+* **Product service**: Shows product information.

+* **Order service**: Places orders.

+* **Rabbit MQ**: Message queue for an order queue.

+1. Use [git][] to clone the sample application to your development environment.

+ ```console

+ git clone https://github.com/Azure-Samples/aks-store-demo.git

+ ```

+2. Change into the cloned directory.

+ ```console

+ cd aks-store-demo

+ ```

+## Review Docker Compose file

+The sample application you create in this tutorial uses the [*docker-compose-quickstart* YAML file](https://github.com/Azure-Samples/aks-store-demo/blob/main/docker-compose-quickstart.yml) in the [repository](https://github.com/Azure-Samples/aks-store-demo/tree/main) you cloned in the previous step.

+```yaml

+version: "3.7"

+ rabbitmq:

+ image: rabbitmq:3.11.17-management-alpine

+ container_name: 'rabbitmq'

+ restart: always

+ environment:

+ - "RABBITMQ_DEFAULT_USER=username"

+ - "RABBITMQ_DEFAULT_PASS=password"

+ ports:

+ - 15672:15672

+ - 5672:5672

+ healthcheck:

+ test: ["CMD", "rabbitmqctl", "status"]

+ interval: 30s

+ timeout: 10s

+ retries: 5

+ volumes:

+ - ./rabbitmq_enabled_plugins:/etc/rabbitmq/enabled_plugins

+ networks:

+ - backend_services

+ orderservice:

+ build: src/order-service

+ container_name: 'orderservice'

+ restart: always

+ ports:

+ - 3000:3000

+ healthcheck:

+ test: ["CMD", "wget", "-O", "", "-q", "http://orderservice:3000/health"]

+ interval: 30s

+ timeout: 10s

+ retries: 5

+ environment:

+ - ORDER_QUEUE_HOSTNAME=rabbitmq

+ - ORDER_QUEUE_PORT=5672

+ - ORDER_QUEUE_USERNAME=username

+ - ORDER_QUEUE_PASSWORD=password

+ - ORDER_QUEUE_NAME=orders

+ - ORDER_QUEUE_RECONNECT_LIMIT=3

+ networks:

+ - backend_services

+ depends_on:

+ rabbitmq:

+ condition: service_healthy

+ productservice:

+ build: src/product-service

+ container_name: 'productservice'

+ restart: always

+ ports:

+ - 3002:3002

+ healthcheck:

+ test: ["CMD", "wget", "-O", "", "-q", "http://productservice:3002/health"]

+ interval: 30s

+ timeout: 10s

+ retries: 5

+ networks:

+ - backend_services

+ storefront:

+ build: src/store-front

+ container_name: 'storefront'

+ restart: always

+ ports:

+ - 8080:8080

+ healthcheck:

+ test: ["CMD", "wget", "-O", "", "-q", "http://storefront:80/health"]

+ interval: 30s

+ timeout: 10s

+ retries: 5

+ environment:

+ - VUE_APP_PRODUCT_SERVICE_URL=http://productservice:3002/

+ - VUE_APP_ORDER_SERVICE_URL=http://orderservice:3000/

+ networks:

+ - backend_services

+ depends_on:

+ - productservice

+ - orderservice

+networks:

+ backend_

+ driver: bridge

+## Create container images and run application

+You can use [Docker Compose][docker-compose] to automate building container images and the deployment of multi-container applications.

+1. Create the container image, download the Redis image, and start the application using the `docker compose` command.

+ ```console

+ docker compose -f docker-compose-quickstart.yml up -d

+ ```

+2. View the created images using the [`docker images`][docker-images] command.

+ ```console

+ docker images

+ ```

+ The following condensed example output shows the created images:

+ ```output

+ REPOSITORY TAG IMAGE ID

+ aks-store-demo-productservice latest 2b66a7e91eca

+ aks-store-demo-orderservice latest 54ad5de546f9

+ aks-store-demo-storefront latest d9e3ac46a225

+ rabbitmq 3.11.17-management-alpine 79a570297657

+ ...

+ ```

+3. View the running containers using the [`docker ps`][docker-ps] command.

+ ```console

+ docker ps

+ ```

+ The following condensed example output shows four running containers:

+ ```output

+ CONTAINER ID IMAGE

+ 21574cb38c1f aks-store-demo-productservice

+ c30a5ed8d86a aks-store-demo-orderservice

+ d10e5244f237 aks-store-demo-storefront

+ 94e00b50b86a rabbitmq:3.11.17-management-alpine

+ ```

+On this page, you can view products, add them to your cart, and then place an order.

+## Clean up resources

+Since you validated the application's functionality, you can stop and remove the running containers. ***Do not delete the container images*** - you use them in the next tutorial.

+* Stop and remove the container instances and resources using the [`docker-compose down`][docker-compose-down] command.

+ ```console

+ docker compose down

+ ```

+> * Clone a sample application source from GitHub.

+> * Create a container image from the sample application source.

+> * Test the multi-container application in a local Docker environment.

+In the next tutorial, you learn how to store container images in an ACR.

+[sample-application]: https://github.com/Azure-Samples/aks-store-demo

+[aks-tutorial-prepare-acr]: ./tutorial-kubernetes-prepare-acr.md

+ Title: Kubernetes on Azure tutorial - Scale applications in Azure Kubernetes Service (AKS)

+# Tutorial - Scale applications in Azure Kubernetes Service (AKS)

+If you followed the previous tutorials, you have a working Kubernetes cluster and Azure Store Front app.

+In this tutorial, part six of seven, you scale out the pods in the app, try pod autoscaling, and scale the number of Azure VM nodes to change the cluster's capacity for hosting workloads. You learn how to:

+> * Configure autoscaling pods that run the app front end.

+In previous tutorials, you packaged an application into a container image, uploaded the image to Azure Container Registry, created an AKS cluster, deployed an application, and used Azure Service Bus to redeploy an updated application. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

+This tutorial requires Azure CLI version 2.34.1 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+1. View the pods in your cluster using the [`kubectl get`][kubectl-get] command.

+ The following example output shows the pods running the Azure Store Front app:

+ NAME READY STATUS RESTARTS AGE

+ order-service-848767080-tf34m 1/1 Running 0 31m

+ product-service-4019737227-2q2qz 1/1 Running 0 31m

+ store-front-2606967446-2q2qz 1/1 Running 0 31m

+2. Manually change the number of pods in the *store-front* deployment using the [`kubectl scale`][kubectl-scale] command.

+ kubectl scale --replicas=5 deployment.apps/store-front

+ The following example output shows the additional pods running the Azure Store Front app:

+ ```output

+ READY STATUS RESTARTS AGE

+ store-front-2606967446-2q2qzc 1/1 Running 0 15m

+ store-front-3309479140-2hfh0 1/1 Running 0 3m

+ store-front-3309479140-bzt05 1/1 Running 0 3m

+ store-front-3309479140-fvcvm 1/1 Running 0 3m

+ store-front-3309479140-hrbf2 1/1 Running 0 15m

+ store-front-3309479140-qphz8 1/1 Running 0 3m

+## Autoscale pods

+To use the horizontal pod autoscaler, all containers and pods must have defined CPU requests and limits. In the `aks-store-quickstart` deployment, the *front-end* container requests 1m CPU with a limit of 1000m CPU.

+...

+ - name: store-front

+ image: ghcr.io/azure-samples/aks-store-demo/store-front:latest

+ - containerPort: 8080

+ name: store-front

+...

+ cpu: 1m

+...

+ cpu: 1000m

+...

+1. Create a manifest file to define the autoscaler behavior and resource limits, as shown in the following condensed example manifest file `aks-store-quickstart-hpa.yaml`:

+ name: store-front-hpa

+ name: store-front

+ kubectl apply -f aks-store-quickstart-hpa.yaml

+ After a few minutes, with minimal load on the Azure Store Front app, the number of pod replicas decreases to three. You can use `kubectl get pods` again to see the unneeded pods being removed.

+> [!NOTE]

+> You can enable the Kubernetes-based Event-Driven Autoscaler (KEDA) AKS add-on to your cluster to drive scaling based on the number of events needing to be processed. For more information, see [Enable simplified application autoscaling with the Kubernetes Event-Driven Autoscaling (KEDA) add-on (Preview)][keda-addon].

+ ```azurecli-interactive

+* Scale your cluster nodes using the [`Get-AzAksCluster`][get-azakscluster] and [`Set-AzAksCluster`][set-azakscluster] cmdlets.

+ ```azurepowershell-interactive

+You can also autoscale the nodes in your cluster. For more information, see [Use the cluster autoscaler with node pools](./cluster-autoscaler.md#use-the-cluster-autoscaler-with-node-pools).

+In the next tutorial, you learn how to upgrade Kubernetes in your AKS cluster.

+> [Upgrade Kubernetes in Azure Kubernetes Service][aks-tutorial-upgrade-kubernetes]

+[aks-tutorial-upgrade-kubernetes]: ./tutorial-kubernetes-upgrade-cluster.md

+[keda-addon]: ./keda-about.md

+ Title: Kubernetes on Azure tutorial - Upgrade an Azure Kubernetes Service (AKS) cluster

+# Tutorial - Upgrade an Azure Kubernetes Service (AKS) cluster

+In this tutorial, part seven of seven, you upgrade an AKS cluster. You learn how to:

+In previous tutorials, you packaged an application into a container image and uploaded the container image to Azure Container Registry (ACR). You also created an AKS cluster and deployed an application to it. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

+If using Azure CLI, this tutorial requires Azure CLI version 2.34.1 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

+If using Azure PowerShell, this tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

+ ```azurecli-interactive

+ The following example output shows the current version as *1.26.6* and lists the available versions under `upgrades`:

+ "kubernetesVersion": "1.26.6",

+ "kubernetesVersion": "1.27.1"

+ "kubernetesVersion": "1.27.3"

+ ```azurepowershell-interactive

+ The following example output shows the current version as *1.26.6* and the location as *eastus*:

+ myAKSCluster 1.26.6 eastus

+ ```azurepowershell-interactive

+ Get-AzAksVersion -Location eastus | Where-Object OrchestratorVersion

+ The following example output shows the available versions under `OrchestratorVersion`:

+ Kubernetes 1.27.1

+ Kubernetes 1.27.3

+4. In **Kubernetes version**, select **Upgrade version**. This redirects you to a new page.

+## Upgrade an AKS cluster

+You can either [manually upgrade your cluster](#manually-upgrade-cluster) or [configure automatic cluster upgrades](#configure-automatic-cluster-upgrades). **We recommend you configure automatic cluster upgrades to ensure your cluster is always running the latest version of Kubernetes**.

+### Manually upgrade cluster

+#### [Azure CLI](#tab/azure-cli)

+ ```azurecli-interactive

+ The following example output shows the result of upgrading to *1.27.3*. Notice the `kubernetesVersion` now shows *1.27.3*:

+ "kubernetesVersion": "1.27.3",

+#### [Azure PowerShell](#tab/azure-powershell)

+ ```azurepowershell-interactive

+ The following example output shows the result of upgrading to *1.27.3*. Notice the `KubernetesVersion` now shows *1.27.3*:

+ KubernetesVersion : 1.27.3

+#### [Azure portal](#tab/azure-portal)

+3. In **Kubernetes version**, select **Upgrade version**. This redirects you to a new page.

+### Configure automatic cluster upgrades

+#### [Azure CLI](#tab/azure-cli)

+* Set an auto-upgrade channel on your cluster using the [`az aks update`][az-aks-update] command with the `--auto-upgrade-channel` parameter set to `patch`.

+ ```azurecli-interactive

+ az aks update --resource-group myResourceGroup --name myAKSCluster --auto-upgrade-channel patch

+ ```

+#### [Azure PowerShell](#tab/azure-powershell)

+* Set an auto-upgrade channel on your cluster using the [`Set-AzAksCluster`][set-azakscluster] cmdlet with the `-AutoUpgradeChannel` parameter set to `Patch`.

+ ```azurepowershell-interactive

+ Set-AzAksCluster -ResourceGroupName myResourceGroup -Name myAKSCluster -AutoUpgradeChannel Patch

+ ```

+#### [Azure portal](#tab/azure-portal)

+1. In the Azure portal, navigate to your AKS cluster.

+2. Under **Settings**, select **Cluster configuration**.

+3. In **Kubernetes version**, select **Upgrade version**.

+4. For **Automatic upgrade**, select **Enabled with patch (recommended)** > **Save**.

+For more information, see [Automatically upgrade an Azure Kubernetes Service (AKS) cluster][aks-auto-upgrade].

+#### Upgrade AKS node images

+AKS regularly provides new node images. Linux node images are updated weekly, and Windows node images are updated monthly. We recommend upgrading your node images frequently to use the latest AKS features and security updates. For more information, see [Upgrade node images in Azure Kubernetes Service (AKS)][node-image-upgrade]. To configure automatic node image upgrades, see [Automatically upgrade Azure Kubernetes Service (AKS) cluster node operating system images][auto-upgrade-node-image].

+ ```console

+ The following example output shows some of the above events listed during an upgrade:

+ ```azurecli-interactive

+ The following example output shows the AKS cluster runs *KubernetesVersion 1.27.3*:

+ myAKSCluster eastus myResourceGroup 1.27.3 1.27.3 Succeeded myaksclust-myresourcegroup-19da35-bd54a4be.hcp.eastus.azmk8s.io

+ ```azurepowershell-interactive

+ The following example output shows the AKS cluster runs *KubernetesVersion 1.27.3*:

+ myAKSCluster eastus 1.27.3 Succeeded

+As this tutorial is the last part of the series, you might want to delete your AKS cluster to avoid incurring Azure charges.

+3. On the popup that asks you to confirm the deletion of the cluster, select **Yes**.

+[aks-auto-upgrade]: ./auto-upgrade-cluster.md

+[auto-upgrade-node-image]: ./auto-upgrade-node-image.md

+[node-image-upgrade]: ./node-image-upgrade.md

+## Azure Virtual Network Manager

+Azure Virtual Network Manager is a management service that allows you to group, configure, deploy, and manage virtual networks globally across subscriptions. With Virtual Network Manager, you can define network groups to identify and logically segment your virtual networks. After that, you can determine the connectivity and security configurations you want and apply them across all the selected virtual networks in network groups at once. Azure Virtual Network Manager's security admin rule configuration allows you to define security policies at scale and apply them to multiple virtual networks at once.

+> [!NOTE]

+> Security admin rules of Azure Virtual Network Manager apply to Application Gateway subnets that only contain Application Gateways that have ["Network Isolation"](Application-gateway-private-deployment.md) enabled. Subnets that have any Application Gateway that does not have ["Network Isolation"](Application-gateway-private-deployment.md) enabled, will not have security admin rules.

+| SUSE Rancher | [K3s](https://rancher.com/products/k3s/) | [v1.27.4+k3s1](https://github.com/k3s-io/k3s/releases/tag/v1.27.4%2Bk3s1), [v1.26.7+k3s1](https://github.com/k3s-io/k3s/releases/tag/v1.26.7%2Bk3s1), [v1.25.12+k3s1](https://github.com/k3s-io/k3s/releases/tag/v1.25.12%2Bk3s1) |

+* Virtual machines (preview): Provision, resize, delete and manage virtual machines based on [VMware vSphere](./vmware-vsphere/overview.md) or [Azure Stack HCI](/azure-stack/hci/manage/azure-arc-vm-management-overview) and enable VM self-service through role-based access.

+* Perform virtual machine lifecycle and management operations for [VMware vSphere](./vmware-vsphere/overview.md) and [Azure Stack HCI](/azure-stack/hci/manage/azure-arc-vm-management-overview) environments.

+* Learn about [Azure Arc-enabled VMware vSphere](vmware-vsphere/overview.md).

+* Learn about [Azure Arc-enabled VM Management on Azure Stack HCI](/azure-stack/hci/manage/azure-arc-vm-management-overview).

+ * Azure Arc VM management on Azure Stack HCI

+* Custom locations: A deployment target where you can create Azure resources. It maps to different resource for different Azure services. For example, for Arc-enabled VMware, the custom locations resource maps to an instance of vCenter, and for Azure Arc VM management on Azure Stack HCI, it maps to an HCI cluster instance.

+## Example scenarios

+The following are just two examples of the many scenarios that can be enabled by using Arc resource bridge in a hybrid environment.

+### Apply Azure Policy and other Azure services to on-premises VMware VMs

+A customer deploys Arc Resource Bridge onto their on-premises VMware environment. They sign into the Azure portal and select the VMware VMs that they'd like to connect to Azure. Now they can manage these on-premises VMware VMs in Azure Resource Manager (ARM) as Arc-enabled machines, alongside their native Azure machines, achieving a single pane of glass to view their resources in a VMware/Azure hybrid environment. This includes deploying Azure services, such as Defender for Cloud and Azure Policy, to keep updated on the security and compliance posture of their on-premises VMware VMs in Azure.

+### Create physical HCI VMs on-premises from Azure

+A customer has multiple datacenter locations in Canada and New York. They install an Arc resource bridge in each datacenter and connect their Azure Stack HCI VMs to Azure in each location. They can then sign into Azure portal and see all their Arc-enabled VMs from the two physical locations together in one central cloud location. From the portal, the customer can choose to create a new VM; that VM is also created on-premises at the selected datacenter, allowing the customer to manage VMs in different physical locations centrally through Azure.

+## Version and region support

+In order to use Arc resource bridge in a region, Arc resource bridge and the Arc-enabled feature for a private cloud must be supported in the region. For example, to use Arc resource bridge with Azure Stack HCI in East US, Arc resource bridge and the Arc VM management feature for Azure Stack HCI must be supported in East US. To confirm feature availability across regions for each private cloud provider, review their deployment guide and other documentation. There could be instances where Arc resource bridge is available in a region where the private cloud feature is not yet available.

+When the Arc-enabled private cloud announces General Availability, the minimum supported version of Arc resource bridge will be 1.0.15.

+Generally, the latest released version and the previous three versions (n-3) of Arc resource bridge are supported. For example, if the current version is 1.0.18, then the typical n-3 supported versions are:

+* Current version: 1.0.18

+* n-1 version: 1.0.17

+* n-2 version: 1.0.16

+* n-3 version: 1.0.15

+There may be instances where supported versions are not sequential. For example, version 1.0.18 is released and later found to contain a bug; a hot fix is released in version 1.0.19 and version 1.0.18 is removed. In this scenario, n-3 supported versions become 1.0.19, 1.0.17, 1.0.16, 1.0.15.

+Arc resource bridge typically releases a new version on a monthly cadence, at the end of the month. Delays may occur that could push the release date further out. Regardless of when a new release comes out, if you are within n-3 supported versions (starting with 1.0.15), then your Arc resource bridge version is supported. To stay updated on releases, visit the [Arc resource bridge release notes](https://github.com/Azure/ArcResourceBridge/releases) on GitHub. To learn more about upgrade options, visit [Upgrade Arc resource bridge](upgrade.md).

+* Review the [system requirements](system-requirements.md) for deploying and managing Arc resource bridge.

+ Title: Overview of Azure Connected Machine agent to manage Windows and Linux machines

+description: This article provides a detailed overview of the Azure Connected Machine agent, which supports monitoring virtual machines hosted in hybrid environments.

+ms.

+# Overview of Azure Connected Machine agent to manage Windows and Linux machines

+The Azure Connected Machine agent enables you to manage your Windows and Linux machines hosted outside of Azure on your corporate network or other cloud providers.

+## Agent components

+The Azure Connected Machine agent package contains several logical components bundled together:

+* The Hybrid Instance Metadata service (HIMDS) manages the connection to Azure and the connected machine's Azure identity.

+* The guest configuration agent provides functionality such as assessing whether the machine complies with required policies and enforcing compliance.

+ Note the following behavior with Azure Policy [guest configuration](../../governance/machine-configuration/overview.md) for a disconnected machine:

+ * An Azure Policy assignment that targets disconnected machines is unaffected.

+ * Guest assignment is stored locally for 14 days. Within the 14-day period, if the Connected Machine agent reconnects to the service, policy assignments are reapplied.

+ * Assignments are deleted after 14 days, and aren't reassigned to the machine after the 14-day period.

+* The Extension agent manages VM extensions, including install, uninstall, and upgrade. Azure downloads extensions and copies them to the `%SystemDrive%\%ProgramFiles%\AzureConnectedMachineAgent\ExtensionService\downloads` folder on Windows, and to `/opt/GC_Ext/downloads` on Linux. On Windows, the extension installs to the following path `%SystemDrive%\Packages\Plugins\<extension>`, and on Linux the extension installs to `/var/lib/waagent/<extension>`.

+>[!NOTE]

+> The [Azure Monitor agent](../../azure-monitor/agents/azure-monitor-agent-overview.md) (AMA) is a separate agent that collects monitoring data, and it does not replace the Connected Machine agent; the AMA only replaces the Log Analytics agent, Diagnostics extension, and Telegraf agent for both Windows and Linux machines.

+## Agent resources

+The following information describes the directories and user accounts used by the Azure Connected Machine agent.

+### Windows agent installation details

+The Windows agent is distributed as a Windows Installer package (MSI). Download the Windows agent from the [Microsoft Download Center](https://aka.ms/AzureConnectedMachineAgent).

+Installing the Connected Machine agent for Window applies the following system-wide configuration changes:

+* The installation process creates the following folders during setup.

+ | Directory | Description |

+ |--|-|

+ | %ProgramFiles%\AzureConnectedMachineAgent | azcmagent CLI and instance metadata service executables.|

+ | %ProgramFiles%\AzureConnectedMachineAgent\ExtensionService\GC | Extension service executables.|

+ | %ProgramFiles%\AzureConnectedMachineAgent\GCArcService\GC | Guest configuration (policy) service executables.|

+ | %ProgramData%\AzureConnectedMachineAgent | Configuration, log and identity token files for azcmagent CLI and instance metadata service.|

+ | %ProgramData%\GuestConfig | Extension package downloads, guest configuration (policy) definition downloads, and logs for the extension and guest configuration services.|

+ | %SYSTEMDRIVE%\packages | Extension package executables |

+* Installing the agent creates the following Windows services on the target machine.

+ | Service name | Display name | Process name | Description |

+ |--|--|--|-|

+ | himds | Azure Hybrid Instance Metadata Service | himds | Synchronizes metadata with Azure and hosts a local REST API for extensions and applications to access the metadata and request Microsoft Entra managed identity tokens |

+ | GCArcService | Guest configuration Arc Service | gc_service | Audits and enforces Azure guest configuration policies on the machine. |

+ | ExtensionService | Guest configuration Extension Service | gc_service | Installs, updates, and manages extensions on the machine. |

+* Agent installation creates the following virtual service account.

+ | Virtual Account | Description |

+ ||-|

+ | NT SERVICE\\himds | Unprivileged account used to run the Hybrid Instance Metadata Service. |

+ > [!TIP]

+ > This account requires the "Log on as a service" right. This right is automatically granted during agent installation, but if your organization configures user rights assignments with Group Policy, you might need to adjust your Group Policy Object to grant the right to "NT SERVICE\\himds" or "NT SERVICE\\ALL SERVICES" to allow the agent to function.

+* Agent installation creates the following local security group.

+ | Security group name | Description |

+ ||-|

+ | Hybrid agent extension applications | Members of this security group can request Microsoft Entra tokens for the system-assigned managed identity |

+* Agent installation creates the following environmental variables

+ | Name | Default value | Description |

+ ||||

+ | IDENTITY_ENDPOINT | `http://localhost:40342/metadata/identity/oauth2/token` |

+ | IMDS_ENDPOINT | `http://localhost:40342` |

+* There are several log files available for troubleshooting, described in the following table.

+ | Log | Description |

+ |--|-|

+ | %ProgramData%\AzureConnectedMachineAgent\Log\himds.log | Records details of the heartbeat and identity agent component. |

+ | %ProgramData%\AzureConnectedMachineAgent\Log\azcmagent.log | Contains the output of the azcmagent tool commands. |

+ | %ProgramData%\GuestConfig\arc_policy_logs\gc_agent.log | Records details about the guest configuration (policy) agent component. |

+ | %ProgramData%\GuestConfig\ext_mgr_logs\gc_ext.log | Records details about extension manager activity (extension install, uninstall, and upgrade events). |

+ | %ProgramData%\GuestConfig\extension_logs | Directory containing logs for individual extensions. |

+* The process creates the local security group **Hybrid agent extension applications**.

+* After uninstalling the agent, the following artifacts remain.

+ * %ProgramData%\AzureConnectedMachineAgent\Log

+ * %ProgramData%\AzureConnectedMachineAgent

+ * %ProgramData%\GuestConfig

+ * %SystemDrive%\packages

+### Linux agent installation details

+The preferred package format for the distribution (`.rpm` or `.deb`) that's hosted in the Microsoft [package repository](https://packages.microsoft.com/) provides the Connected Machine agent for Linux. The shell script bundle [Install_linux_azcmagent.sh](https://aka.ms/azcmagent) installs and configures the agent.

+Installing, upgrading, and removing the Connected Machine agent isn't required after server restart.

+Installing the Connected Machine agent for Linux applies the following system-wide configuration changes.

+* Setup creates the following installation folders.

+ | Directory | Description |

+ |--|-|

+ | /opt/azcmagent/ | azcmagent CLI and instance metadata service executables. |

+ | /opt/GC_Ext/ | Extension service executables. |

+ | /opt/GC_Service/ | Guest configuration (policy) service executables. |

+ | /var/opt/azcmagent/ | Configuration, log and identity token files for azcmagent CLI and instance metadata service.|

+ | /var/lib/GuestConfig/ | Extension package downloads, guest configuration (policy) definition downloads, and logs for the extension and guest configuration services.|

+* Installing the agent creates the following daemons.

+ | Service name | Display name | Process name | Description |

+ |--|--|--|-|

+ | himdsd.service | Azure Connected Machine Agent Service | himds | This service implements the Hybrid Instance Metadata service (IMDS) to manage the connection to Azure and the connected machine's Azure identity.|

+ | gcad.service | GC Arc Service | gc_linux_service | Audits and enforces Azure guest configuration policies on the machine. |

+ | extd.service | Extension Service | gc_linux_service | Installs, updates, and manages extensions on the machine. |

+* There are several log files available for troubleshooting, described in the following table.

+ | Log | Description |

+ |--|-|

+ | /var/opt/azcmagent/log/himds.log | Records details of the heartbeat and identity agent component. |

+ | /var/opt/azcmagent/log/azcmagent.log | Contains the output of the azcmagent tool commands. |

+ | /var/lib/GuestConfig/arc_policy_logs | Records details about the guest configuration (policy) agent component. |

+ | /var/lib/GuestConfig/ext_mgr_logs | Records details about extension manager activity (extension install, uninstall, and upgrade events). |

+ | /var/lib/GuestConfig/extension_logs | Directory containing logs for individual extensions. |

+* Agent installation creates the following environment variables, set in `/lib/systemd/system.conf.d/azcmagent.conf`.

+ | Name | Default value | Description |

+ |||-|

+ | IDENTITY_ENDPOINT | `http://localhost:40342/metadata/identity/oauth2/token` |

+ | IMDS_ENDPOINT | `http://localhost:40342` |

+* After uninstalling the agent, the following artifacts remain.

+ * /var/opt/azcmagent

+ * /var/lib/GuestConfig

+## Agent resource governance

+The Azure Connected Machine agent is designed to manage agent and system resource consumption. The agent approaches resource governance under the following conditions:

+* The Guest Configuration agent can use up to 5% of the CPU to evaluate policies.

+* The Extension Service agent can use up to 5% of the CPU to install, upgrade, run, and delete extensions. Some extensions might apply more restrictive CPU limits once installed. The following exceptions apply:

+ | Extension type | Operating system | CPU limit |

+ | -- | - | |

+ | AzureMonitorLinuxAgent | Linux | 60% |

+ | AzureMonitorWindowsAgent | Windows | 100% |

+ | AzureSecurityLinuxAgent | Linux | 30% |

+ | LinuxOsUpdateExtension | Linux | 60% |

+ | MDE.Linux | Linux | 60% |

+ | MicrosoftDnsAgent | Windows | 100% |

+ | MicrosoftMonitoringAgent | Windows | 60% |

+ | OmsAgentForLinux | Windows | 60%|

+During normal operations, defined as the Azure Connected Machine agent being connected to Azure and not actively modifying an extension or evaluating a policy, you can expect the agent to consume the following system resources:

+| | Windows | Linux |

+| | - | -- |

+| **CPU usage (normalized to 1 core)** | 0.07% | 0.02% |

+| **Memory usage** | 57 MB | 42 MB |

+The performance data above was gathered in April 2023 on virtual machines running Windows Server 2022 and Ubuntu 20.04. Actual agent performance and resource consumption will vary based on the hardware and software configuration of your servers.

+## Instance metadata

+Metadata information about a connected machine is collected after the Connected Machine agent registers with Azure Arc-enabled servers. Specifically:

+* Operating system name, type, and version

+* Computer name

+* Computer manufacturer and model

+* Computer fully qualified domain name (FQDN)

+* Domain name (if joined to an Active Directory domain)

+* Active Directory and DNS fully qualified domain name (FQDN)

+* UUID (BIOS ID)

+* Connected Machine agent heartbeat

+* Connected Machine agent version

+* Public key for managed identity

+* Policy compliance status and details (if using guest configuration policies)

+* SQL Server installed (Boolean value)

+* Cluster resource ID (for Azure Stack HCI nodes)

+* Hardware manufacturer

+* Hardware model

+* CPU family, socket, physical core and logical core counts

+* Total physical memory

+* Serial number

+* SMBIOS asset tag

+* Cloud provider

+The agent requests the following metadata information from Azure:

+* Resource location (region)

+* Virtual machine ID

+* Tags

+* Microsoft Entra managed identity certificate

+* Guest configuration policy assignments

+* Extension requests - install, update, and delete.

+> [!NOTE]

+> Azure Arc-enabled servers doesn't store/process customer data outside the region the customer deploys the service instance in.

+## Next steps

+- [Connect your SCVMM server to Azure Arc](/azure/azure-arc/system-center-virtual-machine-manager/quickstart-connect-system-center-virtual-machine-manager-to-arc).

+- [Install Arc agent at scale for your SCVMM VMs](/azure/azure-arc/system-center-virtual-machine-manager/enable-guest-management-at-scale).

+- [Install Arc agent using a script for SCVMM VMs](/azure/azure-arc/system-center-virtual-machine-manager/install-arc-agents-using-script).

+ Title: Azure Arc agent

+description: Learn about Azure Arc agent

+# Azure Arc agent

+The Azure Connected Machine agent enables you to manage your Windows and Linux machines hosted outside of Azure on your corporate network or other cloud providers.

+## Agent components

+The Azure Connected Machine agent package contains several logical components bundled together:

+* The Hybrid Instance Metadata service (HIMDS) manages the connection to Azure and the connected machine's Azure identity.

+* The guest configuration agent provides functionality such as assessing whether the machine complies with required policies and enforcing compliance.

+ Note the following behavior with Azure Policy [guest configuration](../../governance/machine-configuration/overview.md) for a disconnected machine:

+ * An Azure Policy assignment that targets disconnected machines is unaffected.

+ * Guest assignment is stored locally for 14 days. Within the 14-day period, if the Connected Machine agent reconnects to the service, policy assignments are reapplied.

+ * Assignments are deleted after 14 days and aren't reassigned to the machine after the 14-day period.

+* The Extension agent manages VM extensions, including install, uninstall, and upgrade. Azure downloads extensions and copies them to the `%SystemDrive%\%ProgramFiles%\AzureConnectedMachineAgent\ExtensionService\downloads` folder on Windows, and to `/opt/GC_Ext/downloads` on Linux. On Windows, the extension installs to the path `%SystemDrive%\Packages\Plugins\<extension>`, and on Linux the extension installs to `/var/lib/waagent/<extension>`.

+>[!NOTE]

+> The [Azure Monitor agent](../../azure-monitor/agents/azure-monitor-agent-overview.md) (AMA) is a separate agent that collects monitoring data, and it does not replace the Connected Machine agent; the AMA only replaces the Log Analytics agent, Diagnostics extension, and Telegraf agent for both Windows and Linux machines.

+## Agent resources

+The following information describes the directories and user accounts used by the Azure Connected Machine agent.

+### Windows agent installation details

+The Windows agent is distributed as a Windows Installer package (MSI). Download the Windows agent from the [Microsoft Download Center](https://aka.ms/AzureConnectedMachineAgent).

+Installing the Connected Machine agent for Window applies the following system-wide configuration changes:

+* The installation process creates the following folders during setup.

+ | Directory | Description |

+ |--|-|

+ | %ProgramFiles%\AzureConnectedMachineAgent | azcmagent CLI and instance metadata service executables.|

+ | %ProgramFiles%\AzureConnectedMachineAgent\ExtensionService\GC | Extension service executables.|

+ | %ProgramFiles%\AzureConnectedMachineAgent\GCArcService\GC | Guest configuration (policy) service executables.|

+ | %ProgramData%\AzureConnectedMachineAgent | Configuration, log and identity token files for azcmagent CLI and instance metadata service.|

+ | %ProgramData%\GuestConfig | Extension package downloads, guest configuration (policy) definition downloads, and logs for the extension and guest configuration services.|

+ | %SYSTEMDRIVE%\packages | Extension package executables. |

+* Installing the agent creates the following Windows services on the target machine.

+ | Service name | Display name | Process name | Description |

+ |--|--|--|-|

+ | himds | Azure Hybrid Instance Metadata Service | himds | Synchronizes metadata with Azure and hosts a local REST API for extensions and applications to access the metadata and request Microsoft Entra managed identity tokens |

+ | GCArcService | Guest configuration Arc Service | gc_service | Audits and enforces Azure guest configuration policies on the machine. |

+ | ExtensionService | Guest configuration Extension Service | gc_service | Installs, updates, and manages extensions on the machine. |

+* Agent installation creates the following virtual service account.

+ | Virtual Account | Description |

+ ||-|

+ | NT SERVICE\\himds | Unprivileged account used to run the Hybrid Instance Metadata Service. |

+ > [!TIP]

+ > This account requires the *Log on as a service* right. This right is automatically granted during agent installation, but if your organization configures user rights assignments with Group Policy, you might need to adjust your Group Policy Object to grant the right to **NT SERVICE\\himds** or **NT SERVICE\\ALL SERVICES** to allow the agent to function.

+* Agent installation creates the following local security group.

+ | Security group name | Description |

+ ||-|

+ | Hybrid agent extension applications | Members of this security group can request Microsoft Entra tokens for the system-assigned managed identity |

+* Agent installation creates the following environmental variables

+ | Name | Default value | Description |

+ ||||

+ | IDENTITY_ENDPOINT | `http://localhost:40342/metadata/identity/oauth2/token` |

+ | IMDS_ENDPOINT | `http://localhost:40342` |

+* There are several log files available for troubleshooting, described in the following table.

+ | Log | Description |

+ |--|-|

+ | %ProgramData%\AzureConnectedMachineAgent\Log\himds.log | Records details of the heartbeat and identity agent component. |

+ | %ProgramData%\AzureConnectedMachineAgent\Log\azcmagent.log | Contains the output of the azcmagent tool commands. |

+ | %ProgramData%\GuestConfig\arc_policy_logs\gc_agent.log | Records details about the guest configuration (policy) agent component. |

+ | %ProgramData%\GuestConfig\ext_mgr_logs\gc_ext.log | Records details about extension manager activity (extension install, uninstall, and upgrade events). |

+ | %ProgramData%\GuestConfig\extension_logs | Directory containing logs for individual extensions. |

+* The process creates the local security group **Hybrid agent extension applications**.

+* After uninstalling the agent, the following artifacts remain:

+ * %ProgramData%\AzureConnectedMachineAgent\Log

+ * %ProgramData%\AzureConnectedMachineAgent

+ * %ProgramData%\GuestConfig

+ * %SystemDrive%\packages

+### Linux agent installation details

+The preferred package format for the distribution (`.rpm` or `.deb`) that's hosted in the Microsoft [package repository](https://packages.microsoft.com/) provides the Connected Machine agent for Linux. The shell script bundle [Install_linux_azcmagent.sh](https://aka.ms/azcmagent) installs and configures the agent.

+Installing, upgrading, and removing the Connected Machine agent isn't required after server restart.

+Installing the Connected Machine agent for Linux applies the following system-wide configuration changes.

+* Setup creates the following installation folders.

+ | Directory | Description |

+ |--|-|

+ | /opt/azcmagent/ | azcmagent CLI and instance metadata service executables. |

+ | /opt/GC_Ext/ | Extension service executables. |

+ | /opt/GC_Service/ | Guest configuration (policy) service executables. |

+ | /var/opt/azcmagent/ | Configuration, log and identity token files for azcmagent CLI and instance metadata service.|

+ | /var/lib/GuestConfig/ | Extension package downloads, guest configuration (policy) definition downloads, and logs for the extension and guest configuration services.|

+* Installing the agent creates the following daemons.

+ | Service name | Display name | Process name | Description |

+ |--|--|--|-|

+ | himdsd.service | Azure Connected Machine Agent Service | himds | This service implements the Hybrid Instance Metadata service (IMDS) to manage the connection to Azure and the connected machine's Azure identity.|

+ | gcad.service | GC Arc Service | gc_linux_service | Audits and enforces Azure guest configuration policies on the machine. |

+ | extd.service | Extension Service | gc_linux_service | Installs, updates, and manages extensions on the machine. |

+* There are several log files available for troubleshooting, described in the following table.

+ | Log | Description |

+ |--|-|

+ | /var/opt/azcmagent/log/himds.log | Records details of the heartbeat and identity agent component. |

+ | /var/opt/azcmagent/log/azcmagent.log | Contains the output of the azcmagent tool commands. |

+ | /var/lib/GuestConfig/arc_policy_logs | Records details about the guest configuration (policy) agent component. |

+ | /var/lib/GuestConfig/ext_mgr_logs | Records details about extension manager activity (extension install, uninstall, and upgrade events). |

+ | /var/lib/GuestConfig/extension_logs | Directory containing logs for individual extensions. |

+* Agent installation creates the following environment variables, set in `/lib/systemd/system.conf.d/azcmagent.conf`.

+ | Name | Default value | Description |

+ |||-|

+ | IDENTITY_ENDPOINT | `http://localhost:40342/metadata/identity/oauth2/token` |

+ | IMDS_ENDPOINT | `http://localhost:40342` |

+* After uninstalling the agent, the following artifacts remain:

+ * /var/opt/azcmagent

+ * /var/lib/GuestConfig

+## Agent resource governance

+The Azure Connected Machine agent is designed to manage agent and system resource consumption. The agent approaches resource governance under the following conditions:

+* The Guest Configuration agent can use up to 5% of the CPU to evaluate policies.

+* The Extension Service agent can use up to 5% of the CPU to install, upgrade, run, and delete extensions. Some extensions might apply more restrictive CPU limits once installed. The following exceptions apply:

+ | Extension type | Operating system | CPU limit |

+ | -- | - | |

+ | AzureMonitorLinuxAgent | Linux | 60% |

+ | AzureMonitorWindowsAgent | Windows | 100% |

+ | AzureSecurityLinuxAgent | Linux | 30% |

+ | LinuxOsUpdateExtension | Linux | 60% |

+ | MDE.Linux | Linux | 60% |

+ | MicrosoftDnsAgent | Windows | 100% |

+ | MicrosoftMonitoringAgent | Windows | 60% |

+ | OmsAgentForLinux | Windows | 60%|

+During normal operations, defined as the Azure Connected Machine agent being connected to Azure and not actively modifying an extension or evaluating a policy, you can expect the agent to consume the following system resources:

+| | Windows | Linux |

+| | - | -- |

+| **CPU usage (normalized to 1 core)** | 0.07% | 0.02% |

+| **Memory usage** | 57 MB | 42 MB |

+The performance data above was gathered in April 2023 on virtual machines running Windows Server 2022 and Ubuntu 20.04. The actual agent performance and resource consumption vary based on the hardware and software configuration of your servers.

+## Instance metadata

+Metadata information about a connected machine is collected after the Connected Machine agent registers with Azure Arc-enabled servers, specifically:

+* Operating system name, type, and version

+* Computer name

+* Computer manufacturer and model

+* Computer fully qualified domain name (FQDN)

+* Domain name (if joined to an Active Directory domain)

+* Active Directory and DNS fully qualified domain name (FQDN)

+* UUID (BIOS ID)

+* Connected Machine agent heartbeat

+* Connected Machine agent version

+* Public key for managed identity

+* Policy compliance status and details (if using guest configuration policies)

+* SQL Server installed (Boolean value)

+* Cluster resource ID (for Azure Stack HCI nodes)

+* Hardware manufacturer

+* Hardware model

+* CPU family, socket, physical core and logical core counts

+* Total physical memory

+* Serial number

+* SMBIOS asset tag

+* Cloud provider

+* Amazon Web Services (AWS) metadata, when running in AWS:

+ * Account ID

+ * Instance ID

+ * Region

+* Google Cloud Platform (GCP) metadata, when running in GCP:

+ * Instance ID

+ * Image

+ * Machine type

+ * Project ID

+ * Project number

+ * Service accounts

+ * Zone

+The agent requests the following metadata information from Azure:

+* Resource location (region)

+* Virtual machine ID

+* Tags

+* Microsoft Entra managed identity certificate

+* Guest configuration policy assignments

+* Extension requests - install, update, and delete.

+> [!NOTE]

+> Azure Arc-enabled servers don't store/process customer data outside the region the customer deploys the service instance in.

+## Next steps

+- [Connect VMware vCenter Server to Azure Arc](quick-start-connect-vcenter-to-arc-using-script.md).

+- [Install Arc agent at scale for your VMware VMs](enable-guest-management-at-scale.md).

+| [PowerShell](vminsights-enable-powershell.md) | Use a PowerShell script to enable multiple machines. Currently only supported for Log Analytics agent. |

+| [Manual install](vminsights-enable-hybrid.md) | Virtual machines or physical computers on-premises with other cloud environments.|

+- AgentId: The unique identifier for a machine running Azure Monitor Agent or the Log Analytics agent

+|AgentId | The unique identifier for a machine running Azure Monitor Agent or the Log Analytics agent |

+|AgentId | The unique identifier for a machine running Azure Monitor Agent or the Log Analytics agent |

+- Azure Monitor Agent is installed on Azure virtual machines by using a VM extension if the agent is already installed.

+ Title: Special cases to move Azure VMs to new subscription or resource group

+# Handling special cases when moving virtual machines to resource group or subscription

+This article describes special cases that require extra steps when moving a virtual machine to a new resource group or Azure subscription. If your virtual machine doesn't match any of these scenarios, you can move the virtual machine with the standard steps described in [Move resources to a new resource group or subscription](../move-resource-group-and-subscription.md).

+Virtual machines created from Marketplace resources with plans attached can't be moved across subscriptions. To work around this limitation, you can deprovision the virtual machine in the current subscription, and deploy it again in the new subscription. The following steps help you recreate the virtual machine in the new subscription. However, they might not work for all scenarios. If the plan is no longer available in the Marketplace, these steps won't work.

+# Move Azure resources to a new resource group or subscription

+# [Portal](#tab/Portal)

+To create a replica in an Azure region, go to your Web PubSub resource and find the **Replicas** blade on the Azure portal and click **Add** to create a replica.

+# [Bicep](#tab/Bicep)

+Use Visual Studio Code or your favorite editor to create a file with the following content and name it main.bicep:

+```bicep

+@description('The name for your Web PubSub service')

+param primaryName string = 'contoso'

+@description('The region in which to create your Web Pubsub service')

+param primaryLocation string = 'eastus'

+@description('Unit count of your Web PubSub service')

+param primaryCapacity int = 1

+resource primary 'Microsoft.SignalRService/webpubsub@2023-08-01-preview' = {

+ name: primaryName

+ location: primaryLocation

+ sku: {

+ capacity: primaryCapacity

+ name: 'Premium_P1'

+ }

+ properties: {

+ }

+}

+@description('The name for your Web PubSub replica')

+param replicaName string = 'contoso-westus'

+@description('The region in which to create the Web PubSub replica')

+param replicaLocation string = 'westus'

+@description('Unit count of the Web PubSub replica')

+param replicaCapacity int = 1

+@description('Whether to enable region endpoint for the replica')

+param regionEndpointEnabled string = 'Enabled'

+resource replica 'Microsoft.SignalRService/webpubsub/replicas@2023-08-01-preview' = {

+ parent: primary

+ name: replicaName

+ location: replicaLocation

+ sku: {

+ capacity: replicaCapacity

+ name: 'Premium_P1'

+ }

+ properties: {

+ regionEndpointEnabled: regionEndpointEnabled

+ }

+}

+```

+Deploy the Bicep file using Azure CLI

+ ```azurecli

+ az group create --name MyResourceGroup --location eastus

+ az deployment group create --resource-group MyResourceGroup --template-file main.bicep

+ ```

+# [CLI](#tab/CLI)

+Update **webpubsub** extension to the latest version, then run:

+ ```azurecli

+ az webpubsub replica create --sku Premium_P1 -l eastus --replica-name MyReplica --name MyWebPubSub -g MyResourceGroup

+ ```

+To delete a replica using the Azure CLI:

+ ```azurecli

+ az webpubsub replica delete --replica-name MyReplica --name MyWebPubSub -g MyResourceGroup

+ ```

+> * Due to the DNS cache, it might take several minutes for the DNS update to take effect.

+ If you use the Web PubSub event handler with Web PubSub C# server SDK or an Azure Function that utilizes the Web PubSub extension, you might encounter issues with the abuse protection once replicas are enabled. To address this, you can either **disable the abuse protection** or **upgrade to the latest SDK/extension versions**.

+| Region | West US |

+- **Authentication**. To use a pool without public IP addresses inside a [virtual network](./batch-virtual-network.md), the Batch client API must use Microsoft Entra authentication. Azure Batch support for Microsoft Entra ID is documented in [Authenticate Azure Batch services with Microsoft Entra ID](batch-aad-auth.md). If you aren't creating your pool within a virtual network, either Microsoft Entra authentication or key-based authentication can be used.

+(Azure AD ID). This identity is used to obtain Microsoft Entra tokens to authenticate with target

+- **Supported browsers** - The Chaos Studio portal experience has only been tested on the following browsers:

+- **Terraform** - At present Chaos Studio does not support terraform.

+- **Built-in roles** - Chaos Studio does not currently have its own built-in roles. Permissions may be attained to run a chaos experiment by either assigning an [Azure built-in role](chaos-studio-fault-providers.md) or a created custom role to the experiment's identity.

+<!-- LINKS - Internal -->

+[resource-manager template]: container-instances-multi-container-group.md

+| `CREATE INDEX` | Yes (including [named indexes](secondary-indexing.md) but full FROZEN collection is not supported) |

+Azure Cosmos DB offers multiple database APIs, which include NoSQL, MongoDB, PostgreSQL, Cassandra, Gremlin, and Table. By using these APIs, you can model real world data using documents, key-value, graph, and column-family data models. These APIs allow your applications to treat Azure Cosmos DB as if it were various other databases technologies, without the overhead of management, and scaling approaches. Azure Cosmos DB helps you to use the ecosystems, tools, and skills you already have for data modeling and querying with its various APIs.

+* In Azure Cosmos DB for NoSQL, the JSON object represents the data format. The partition key resides in a field in the document and sets which field is the partition key for the collection. Azure Cosmos DB doesn't have the concept of timestamp used for column family or version. As highlighted previously, it has change feed support through which one can track/record changes performed on a container. The following is an example of a document.

+There's a [capacity calculator](estimate-ru-with-capacity-planner.md) available to assist with sizing exercise for RUs.

+Physical partitions each contain a replica of your data and an instance of the Azure Cosmos DB database engine. This structure makes your data durable and highly available and throughput is divided equally amongst the local physical partitions. Physical partitions are automatically created and configured, and it's not possible to control their size, location, or which logical partitions they contain. Logical partitions aren't split between physical partitions.

+As with HBase RowKey, partition key design is important for Azure Cosmos DB. HBase's Row Key works by sorting data and storing continuous data, and Azure Cosmos DB's Partition Key is a different mechanism because it hash-distributes data. Assuming your application using HBase is optimized for data access patterns to HBase, using the same RowKey for the partition Key won't give good performance results. Given that it's sorted data on HBase, the [Azure Cosmos DB composite index](../index-policy.md#composite-indexes) may be useful. It's required if you want to use the ORDER BY clause in more than one field. You can also improve the performance of many equal and range queries by defining a composite index.

+Azure Cosmos DB doesn't require any configuration such as cluster component redundancy. It provides a comprehensive SLA for high availability, consistency, and latency. See [SLA for Azure Cosmos DB](https://azure.microsoft.com/support/legal/sla/cosmos-db/v1_3/) for more detail.

+### Offline and online migration

+When performing offline data migration, it depends on the version of HBase you're currently running and the tools available. See the [Data Migration](#migrate-your-data) section for more details.

+If you're using Apache Phoenix on HBase cluster, you need to collect data from Phoenix as well.

+| Custom tool with Azure Cosmos DB bulk executor library | All versions | Most flexible to create custom data migration tools using libraries. Requires more effort to setup. |

+The mappings for code migration are shown here, but the HBase RowKeys and Azure Cosmos DB Partition Keys used in these examples aren't always well designed. Design according to the actual data model of the migration source.

+Azure Cosmos DB provides type safety via data model. We use data model named ΓÇÿFamilyΓÇÖ.

+In Azure Cosmos DB, updates are treated as Upsert operations. That is, if the document doesn't exist, it will be inserted.

+HBase offers several server-side programming features. If you're using these features, you will also need to migrate their processing.

+ * Endpoint is a feature for extending HBase RPC. It's a function similar to an RDBMS stored procedure.

+Data security is a shared responsibility of the customer and the database provider. For on-premises solutions, customers have to provide everything from endpoint protection to physical hardware security, which is not an easy task. If you choose a PaaS cloud database provider such as Azure Cosmos DB, customer involvement will be reduced. Azure Cosmos DB runs on the Azure platform, so it can be enhanced in a different way than HBase. Azure Cosmos DB doesn't require any extra components to be installed for security. We recommend that you consider migrating your database system security implementation using the following checklist:

+| Automatic data backups | There's no automatic backup function. You need to implement data backup yourself. | Azure Cosmos DB is backed up regularly and stored in the geo redundant storage. |

+| Protect and isolate sensitive data | For example, if you're using Apache Ranger, you can use Ranger policy to apply the policy to the table. | You can separate personal and other sensitive data into specific containers and read / write, or limit read-only access to specific users. |

+Azure Cosmos DB automatically backs up data at periodic intervals, which doesn't affect the performance or availability of database operations. Backups are stored in Azure storage and can be used to recover data if needed. There are two types of Azure Cosmos DB backups:

+|**Sample with Azure App Service** | [How to use Spring and Azure Cosmos DB with App Service on Linux](/azure/developer/java/spring-framework/configure-spring-app-with-cosmos-db-on-app-service-linux) |

+Azure Cosmos DB traces follow the [OpenTelemetry database specification](https://github.com/open-telemetry/opentelemetry-specification) and also provide several custom attributes. You may see different attributes depending on the operation of your request, and these attributes are core attributes for all requests.

+[Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/), is a fully managed cloud service that lets you create workflows using algorithms in a simple drag-and-drop interface, code your own algorithms in [R](https://en.wikipedia.org/wiki/R_\(programming_language\)), or use some of the already-built and ready to use APIs such as: [Text Analytics](https://gallery.cortanaanalytics.com/MachineLearningAPI/Text-Analytics-2), Content Moderator, or [Recommendations](https://gallery.azure.ai/Solution/Recommendations-Solution).

+The ultimate goal of unit economics, as a derivative of activity-based costing methodology, is to factor in the whole picture of your business's cost. This article focuses on capturing how you can factor your Microsoft Cloud costs into those efforts. As your FinOps practice matures, consider the manual processes and steps outside of the cloud that might be important for calculating units that are critical for your business to track the most accurate cost per unit.

+ - If you don't see the data you need in metrics, also check [logs and traces in Azure Monitor](../../azure-monitor/overview.md#data-platform). It might not be a direct correlation to usage but might be able to give you some indication of usage.

+- **Integration runtimes and sharing**. Integration runtimes don't change often and are similar across all stages in your CI/CD. So Data Factory expects you to have the same name, type and sub-type of integration runtime across all stages of CI/CD. If you want to share integration runtimes across all stages, consider using a ternary factory just to contain the shared integration runtimes. You can use this shared factory in all of your environments as a linked integration runtime type.

+description: Learn about self-hosted integration runtime auto-update and expire notification.

+This article describes how to let self-hosted integration runtime auto-update to the latest version and how Azure Data Factory (ADF) manages the versions of self-hosted integration runtime.

+You can check the version either in your self-hosted integration runtime client or in the ADF portal:

+Generally, when you install a self-hosted integration runtime in your local machine or an Azure VM, you have two options to manage the version of self-hosted integration runtime: auto-update or maintain manually. Typically, ADF releases two new versions of self-hosted integration runtime every month, which includes new features released, bugs fixed, and enhancements. So we recommend users to update to the latest version.

+The most convenient way is to enable auto-update when you create or edit self-hosted integration runtime. The self-hosted integration runtime is automatically update to newer version. You can also schedule the update at the most suitable time slot as you wish.

+To ensure the stability of self-hosted integration runtime, although we release two versions, we'll only push one version every month. So sometimes you find that the auto-updated version is the previous version of the actual latest version. If you want to get the latest version, you can go to [download center](https://www.microsoft.com/download/details.aspx?id=39717) and do so manually. Additionally, **auto-update** to a new version is managed internally. You can't change it.

+The self-hosted integration runtime **Auto update** page in the ADF portal shows the newer version if current version is old. When your self-hosted integration runtime is online, this version is auto-update version and will automatically update your self-hosted integration runtime in the scheduled time. But if your self-hosted integration runtime is offline, the page only shows the latest version.

+If you have multiple nodes, and for some reasons that some of them aren't auto-updated successfully. Then these nodes roll back to the version, which was the same across all nodes before the auto-update.

+If you want to manually control which version of self-hosted integration runtime, you can disable the setting of auto-update and install it manually. Each version of self-hosted integration runtime expires in one year. The expiring message is shown in ADF portal and self-hosted integration runtime client **90 days** before expiration.

+To apply the 2309 update, your device must be running version 2303 or later.

+ - You can update to 2303 from 2207 or later, and then install 2309.

+|**2.**|Wi-Fi |Starting this release, Wi-Fi functionality for Azure Stack Edge Mini R has been deprecated. | |

+az dedicated-hsm update --resource-group myRG ΓÇô-name hsm1

+az dedicated-hsm delete --resource-group myRG ΓÇô-name hsm1

+* [Monitoring](monitoring.md)

+|(Preview) Azure API Management APIs should be onboarded to Defender for APIs. | Onboarding APIs to Defender for APIs requires compute and memory utilization on the Azure API Management service. Monitor performance of your Azure API Management service while onboarding APIs, and scale out your Azure API Management resources as needed.|High|

+|(Preview) API endpoints that are unused should be disabled and removed from the Azure API Management service|As a security best practice, API endpoints that haven't received traffic for 30 days are considered unused, and should be removed from the Azure API Management service. Keeping unused API endpoints might pose a security risk. These might be APIs that should have been deprecated from the Azure API Management service, but have accidentally been left active. Such APIs typically do not receive the most up-to-date security coverage.|Low|

+|(Preview) API endpoints in Azure API Management should be authenticated|API endpoints published within Azure API Management should enforce authentication to help minimize security risk. Authentication mechanisms are sometimes implemented incorrectly or are missing. This allows attackers to exploit implementation flaws and to access data. For APIs published in Azure API Management, this recommendation assesses authentication though verifying the presence of Azure API Management subscription keys for APIs or products where subscription is required, and the execution of policies for validating [JWT](/azure/api-management/validate-jwt-policy), [client certificates](/azure/api-management/validate-client-certificate-policy), and [Microsoft Entra](/azure/api-management/validate-azure-ad-token-policy) tokens. If none of these authentication mechanisms are executed during the API call, the API will receive this recommendation.|High|

+<!-- Please don't adjust this next line without getting approval from the Defender for Cloud documentation team. It is necessary for proper RSS functionality. -->

+| [Changes to how Microsoft Defender for Cloud's costs are presented in Microsoft Cost Management](#changes-to-how-microsoft-defender-for-clouds-costs-are-presented-in-microsoft-cost-management) | October 25, 2023 | November 2023 |

+## Changes to how Microsoft Defender for Cloud's costs are presented in Microsoft Cost Management

+**Annoucement date: October 26, 2023**

+**Estimated date for change: November 2023**

+In November there will be a change as to how Microsoft Defender for Cloud's costs are presented in Cost Management and in Subscriptions invoices.

+Costs will be presented for each protected resource instead of as an aggregation of all resources on the subscription.

+If a resource has a tag applied, which are often used by organizations to perform financial chargeback processes, it will be added to the appropriate billing lines.

+[Azure for Students](https://azure.microsoft.com/free/students/) is a program offered by Microsoft Azure to provide students at Higher Education Institutions with access to various Azure cloud services and resources. Joining Azure for Students will grant you $100 in Azure credits with no credit card required. This credit will be valid for one year and you can renew each year you're an active student to get an addition $100. You also receive access to a set of free services you can deploy once you join. Select versions of Azure Virtual Machines, Azure SQL Databases, Azure Blob Storage and more are all free with your subscription.

+Azure for Students is available only to students who meet the following requirements:

+* You must affirm that you're age 18 or older and attend an accredited, degree-granting, two-year or four-year educational institution where youΓÇÖre a full-time student.

+* You must verify your academic status through your institutionΓÇÖs email address.

+* See the Azure for Students Offer for detailed terms of use.

+* This offer isn't available for use in a massive open online course (MOOC) or in other professional trainings from for-profit organizations.

+This offer is limited to one Azure for Student subscription per eligible customer. It's nontransferable and can't be combined with any other offer, unless otherwise permitted by Microsoft.

+You need to verify yourself as an active student to become eligible for the offer. This verification process takes place as part of the signup flow for Azure for Students here.

+ Title: Azure Event GridΓÇÖs MQTT broker feature - Monitor data reference

+description: This article provides reference documentation for metrics and diagnostic logs for Azure Event GridΓÇÖs MQTT broker feature.

+# Monitor data reference for Azure Event GridΓÇÖs MQTT broker feature (Preview)

+This article provides a reference of log and metric data collected to analyze the performance and availability of MQTT broker.

+| MQTT.Throughput | MQTT: Throughput | Count | Total | The total bytes published to or delivered by the namespace. This metric includes all the MQTT packets that your MQTT clients send to the MQTT broker, regardless of their success. | Direction |

+| MQTT.SuccessfulSubscriptionOperations | MQTT: Successful Subscription Operations | Count | Total | The number of successful subscription operations (Subscribe, Unsubscribe). This metric is incremented for every topic filter within your subscription request that gets accepted by MQTT broker. | OperationType, Protocol |

+| MQTT.FailedSubscriptionOperations | MQTT: Failed Subscription Operations | Count | Total | The number of failed subscription operations (Subscribe, Unsubscribe). This metric is incremented for every topic filter within your subscription request that gets rejected by MQTT broker. | OperationType, Protocol, Error |

+| MQTT.Connections | MQTT: Active Connections | Count | Total | The number of active connections in the namespace. The value for this metric is a point-in-time value. Connections that were active immediately after that point-in-time might not be reflected in the metric. | Protocol |

+| Mqtt.DroppedSessions | MQTT: Dropped Sessions | Count | Total | The number of dropped sessions in the namespace. The value for this metric is a point-in-time value. Sessions that were dropped immediately after that point-in-time might not be reflected in the metric. | DropReason |

+| OperationType | The type of the operation. The available values include: <br><br>- Publish: PUBLISH requests sent from MQTT clients to MQTT broker. <br>- Deliver: PUBLISH requests sent from MQTT broker to MQTT clients. <br>- Subscribe: SUBSCRIBE requests by MQTT clients. <br>- Unsubscribe: UNSUBSCRIBE requests by MQTT clients. <br>- Connect: CONNECT requests by MQTT clients. |

+| Error | Error occurred during the operation. The available values include: <br><br>-QuotaExceeded: the client exceeded one or more of the throttling limits that resulted in a failure <br>- AuthenticationError: a failure because of any authentication reasons. In case of failed MQTT routing messages, the EventGrid Data Sender role for the custom topic configured as the destination for MQTT routed messages was deleted. This error doesn't apply for namespace topics since they don't need a permission to route MQTT messages. In that case for MQTT message routing, MQTT broker drops the MQTT message that was meant to be routed. <br>- AuthorizationError: a failure because of any authorization reasons.<br>- ClientError: the client sent a bad request or used one of the unsupported features that resulted in a failure. <br>-TopicNotFoundError: The custom topic that is configured to receive all the MQTT routed messages was deleted. This error doesn't apply for namespace topics since they can't be deleted if they're used as the destination for MQTT routed messages. In that case, MQTT broker drops the MQTT message that was meant to be routed.<br>-TooManyRequests: the number of MQTT routed messages per second exceeds the limit of the destination (namespace topic or custom topic) for MQTT routed messages. In that case, Event Grid retries to route the MQTT message. <br>- ServiceError: a failure because of an unexpected server error or for a server's operational reason. In that case for MQTT message routing, Event Grid retries to route the MQTT message. |

+| Direction | The direction of the operation. The available values are: <br><br>- Inbound: inbound throughput to MQTT broker. <br>- Outbound: outbound throughput from MQTT broker. |

+description: 'Describes the main concepts for access control for MQTT clients in Azure Event GridΓÇÖs MQTT broker feature.'

+Access control enables you to manage the authorization of clients to publish or subscribe to topics, using a role-based access control model. Given the enormous scale of IoT environments, assigning permission for each client to each topic is incredibly tedious. Azure Event GridΓÇÖs MQTT broker feature tackles this scale challenge through grouping clients and topics into client groups and topic spaces.

+The variables represent either client authentication names or client attributes. During communication with MQTT broker, each client would replace the variable in the MQTT topic with a substituted value. For example, the variable ${client.authenticationName} would be replaced with the authentication name of each client: machine1, machine2, etc. MQTT broker would allow access only to the clients that have a substituted value that matches either their authentication name or the value of the specified attribute.

+1. The vehicle publishes telemetry and events messages through an MQTT client with defined topics to the **Azure Event GridΓÇÖs MQTT broker feature** in the *vehicle messaging services*.

+1. The factory system triggers the **device provisioning client** to connect to the *device registration* and provision the device. The device retrieves connection information to the *MQTT broker*.

+1. The *device registration* application creates the device identity with MQTT broker.

+1. The factory system triggers the device to establish a connection to the *MQTT broker* for the first time.

+1. The *MQTT broker* manages authorization for allowed topics using the local registry.

+1. The vehicle publishes a message to the *MQTT broker*.

+ 1. Messages that don't require more processing are published to the *MQTT broker*.

+description: 'Describes how MQTT clients are authenticated and mTLS connection is established when a client connects to Azure Event GridΓÇÖs MQTT broker feature.'

+To establish a secure connection with MQTT broker, you can use either MQTTS over port 8883 or MQTT over web sockets on port 443. It's important to note that only secure connections are supported. The following steps are to establish secure connection prior to the client authentication.

+1. The client initiates the handshake with MQTT broker. It sends a hello packet with supported TLS version, cipher suites.

+ Title: 'Overview of Azure Event GridΓÇÖs MQTT broker feature (preview)'

+description: 'Describes the main concepts for the Azure Event GridΓÇÖs MQTT broker feature.'

+# Overview of the Azure Event GridΓÇÖs MQTT broker feature (Preview)

+Azure Event Grid enables your MQTT clients to communicate with each other and with Azure services, to support your Internet of Things (IoT) solutions.

+Azure Event GridΓÇÖs MQTT broker feature enables you to accomplish the following scenarios:

+The MQTT broker is ideal for the implementation of automotive and mobility scenarios, among others. See [the reference architecture](mqtt-automotive-connectivity-and-data-solution.md) to learn how to build secure and scalable solutions for connecting millions of vehicles to the cloud, using AzureΓÇÖs messaging and data analytics services.

+The following are a list of key concepts involved in Azure Event GridΓÇÖs MQTT broker feature.

+MQTT is a publish-subscribe messaging transport protocol that was designed for constrained environments. It has become the go-to communication standard for IoT scenarios due to efficiency, scalability, and reliability. MQTT broker enables clients to publish and subscribe to messages over MQTT v3.1.1, MQTT v3.1.1 over WebSockets, MQTT v5, and MQTT v5 over WebSockets protocols. The following list shows some of the feature highlights of MQTT broker:

+ - **Message expiry interval** allows you to declare to MQTT broker when to disregard a message that is no longer relevant or valid. For example, disregard stale commands or alerts.

+- MQTT broker is adding more MQTT v5 features in the future to align more with the MQTT specifications. The following items detail the current differences between features supported by MQTT broker and the MQTT v5 specifications: Will message, Retain flag, Message ordering and QoS 2 aren't supported.

+- MQTT broker is adding more MQTT v3.1.1 features in the future to align more with the MQTT specifications. The following items detail the current differences between features supported by MQTT broker and the MQTT v3.1.1 specification: Will message, Retain flag, Message ordering and QoS 2 aren't supported.

+[Learn more about the MQTT broker and current limitations.](mqtt-support.md)

+- The many-to-one messaging pattern enables clients to offload the burden of managing the high number of connections to MQTT broker.

+Event Grid Namespace is a management container for the resources supporting the MQTT broker functionality, along with the resources supporting the [pull delivery functionality](pull-delivery-overview.md). Your MQTT client can connect to MQTT broker and publish/subscribe to messages, while MQTT broker authenticates your clients, authorizes publish/subscribe requests, and forwards messages to interested clients. Learn more about [the namespace concept.](mqtt-event-grid-namespace-terminology.md)

+Event Grid has a client registry that stores information about the clients permitted to connect to it. Before a client can connect, there must be an entry for that client in the client registry. As a client connects to MQTT broker, it needs to authenticate with MQTT broker based on credentials stored in the identity registry. MQTT broker supports X.509 certificate authentication that is the industry authentication standard in IoT devices and [Microsoft Entra](mqtt-client-azure-ad-token-and-rbac.md) that is Azure's authentication standard for applications.[Learn more about MQTT client authentication.](mqtt-client-authentication.md)

+Use the following articles to learn more about the MQTT broker and its main concepts.

+description: 'Quickstart guide to use Azure Event GridΓÇÖs MQTT broker feature and Azure CLI to publish and subscribe MQTT messages on a topic'

+Azure Event GridΓÇÖs MQTT broker feature supports messaging using the MQTT protocol. Clients (both devices and cloud applications) can publish and subscribe MQTT messages over flexible hierarchical topics for scenarios such as high scale broadcast, and command & control.

+description: 'Quickstart guide to use Azure Event GridΓÇÖs MQTT broker feature and Azure portal to publish and subscribe MQTT messages on a topic.'

+1. Create an Event Grid namespace and enable MQTT

+1. After the deployment succeeds, select **Go to resource** to navigate to the Event Grid Namespace Overview page for your namespace.

+1. Select **Connect** to connect the client to the MQTT broker.

+## MQTT message routing behavior

+While routing MQTT messages to namespace topics or custom topics, Event Grid provides durable delivery as it tries to deliver each message **at least once** immediately. If there's a failure, Event Grid either retries delivery or drops the message that was meant to be routed. Event Grid doesn't guarantee order for event delivery, so subscribers might receive them out of order.

+The following table describes the behavior of MQTT message routing based on different errors.

+| Error| Error description | Behavior |

+| --| --|--|

+| TopicNotFoundError | The custom topic that is configured to receive all the MQTT routed messages was deleted. This error doesn't apply for namespace topics since they can't be deleted if they're used as the destination for MQTT routed messages. | Event Grid drops the MQTT message that was meant to be routed.|

+| AuthenticationError | The EventGrid Data Sender role for the custom topic configured as the destination for MQTT routed messages was deleted. This error doesn't apply for namespace topics since they don't need a permission to route MQTT messages. | Event Grid drops the MQTT message that was meant to be routed.|

+| TooManyRequests | The number of MQTT routed messages per second exceeds the limit of the destination (namespace topic or custom topic) for MQTT routed messages. | Event Grid retries to route the MQTT message.|

+| ServiceError | An unexpected server error for a server's operational reason. | Event Grid retries to route the MQTT message.|

+

+During retries, Event Grid uses an exponential backoff retry policy for MQTT message routing. Event Grid retries delivery on the following schedule on a best effort basis:

+- 10 seconds

+- 30 seconds

+- 1 minute

+- 5 minutes

+- 10 minutes

+- 30 minutes

+- 1 hour

+- 3 hours

+- 6 hours

+- Every 12 hours

+If a routed MQTT message that was queued for redelivery succeeded, Event Grid attempts to remove the message from the retry queue on a best effort basis, but duplicates might still be received.

+ Title: 'MQTT Features Support by Azure Event GridΓÇÖs MQTT broker feature'

+description: 'Describes the MQTT features supported by Azure Event GridΓÇÖs MQTT broker feature.'

+# MQTT features supported by Azure Event GridΓÇÖs MQTT broker feature

+MQTT is a publish-subscribe messaging transport protocol that was designed for constrained environments. ItΓÇÖs efficient, scalable, and reliable, which made it the gold standard for communication in IoT scenarios. MQTT broker supports clients that publish and subscribe to messages over MQTT v3.1.1, MQTT v3.1.1 over WebSockets, MQTT v5, and MQTT v5 over WebSockets. MQTT broker also supports cross MQTT version (MQTT 3.1.1 and MQTT 5) communication.

+While connecting to MQTT broker, use the following ports during communication over MQTT:

+Multi-session support enables your application MQTT clients to have more scalable and reliable implementation by connecting to MQTT broker with multiple active sessions at the same time.

+- Provide the Username property in the CONNECT packet to signify your client authentication name.

+For example, the following combinations of Username and ClientIds in the CONNECT packet enable the client "Mgmt-application" to connect to MQTT broker over three independent sessions:

+- If a client tries to take over another client's active session by presenting its session name with a different authentication name, its connection request is rejected with an unauthorized error. For example, if Client B tries to connect to session 123 that is assigned at that time for client A, Client B's connection request is rejected. That being said, if the same client tries to reconnect with the same session names and the same authentication name, it is able to take over its existing session.

+- If a client resource is deleted without ending its session, other clients can't use its session name until the session expires. For example, If client B creates a session with session name 123 then client B gets deleted, client A can't connect to session 123 until it expires.

+- The limit for the number of sessions per client applies to online and offline sessions at any point in time. For example, consider a namespace with the maximum client sessions per authentication name is set to 1. If client A connects with a persistent session 123 then gets disconnected, client A won't be able to connect with a new session 456 since its session 123 is still active even if it's offline. Accordingly, we recommend that the same client always reconnects with the same static session names as opposed to generating a new session name with every reconnect.

+Azure Event GridΓÇÖs MQTT broker feature supports the following MQTT features:

+MQTT broker supports QoS 0 and 1, which define the guarantee of message delivery on PUBLISH and SUBSCRIBE packets between clients and MQTT broker. QoS 0 guarantees at-most-once delivery; messages with QoS 0 arenΓÇÖt acknowledged by the subscriber nor get retransmitted by the publisher. QoS 1 guarantees at-least-once delivery; messages are acknowledged by the subscriber and get retransmitted by the publisher if they didnΓÇÖt get acknowledged. QoS enables your clients to control the efficiency and reliability of the communication.

+MQTT broker supports persistent sessions for MQTT v3.1.1 such that MQTT broker preserves information about a clientΓÇÖs session in case of disconnections to ensure reliability of the communication. This information includes the clientΓÇÖs subscriptions and missed/ unacknowledged QoS 1 messages. Clients can configure a persistent session through setting the cleanSession flag in the CONNECT packet to false.

+MQTT v5 has introduced the clean start and session expiry features as an improvement over MQTT v3.1.1 in handling session persistence. Clean Start is a feature that allows a client to start a new session with MQTT broker, discarding any previous session data. Session Expiry allows a client to inform MQTT broker when an inactive session is considered expired and automatically removed. In the CONNECT packet, a client can set Clean Start flag to true and/or short session expiry interval for security reasons or to avoid any potential data conflicts that might have occurred during the previous session. A client can also set a clean start to false and/or long session expiry interval to ensure the reliability and efficiency of persistent sessions.

+MQTT broker maintains a queue of messages for each active MQTT session that isn't connected, until the client connects with MQTT broker again to receive the messages in the queue. If a client doesn't connect to receive the queued QOS1 messages, the session queue starts accumulating the messages until it reaches its limit: 100 messages or 1 MB. Once the queue reaches its limit during the lifespan of the session, the session is terminated.

+MQTT broker supports user properties on MQTT v5 PUBLISH packets that allow you to add custom key-value pairs in the message header to provide more context about the message. The use cases for user properties are versatile. You can use this feature to include the purpose or origin of the message so the receiver can handle the message without parsing the payload, saving computing resources. For example, a message with a user property indicating its purpose as a "warning" could trigger different handling logic than one with the purpose of "information."

+In MQTT v5, message expiry interval allows messages to have a configurable lifespan. The message expiry interval is defined as the time interval between the time a message is published to MQTT broker and the time when the MQTT broker needs to discard the message if it hasn't been delivered. This feature is useful in scenarios where messages are only valid for a certain amount of time, such as time-sensitive commands, real-time data streaming, or security alerts. By setting a message expiry interval, MQTT broker can automatically remove outdated messages, ensuring that only relevant information is available to subscribers. If a message's expiry interval is set to zero, it means the message should never expire.

+In MQTT v5, topic aliases allow a client to use a shorter alias in place of the full topic name in the published message. MQTT broker maintains a mapping between the topic alias and the actual topic name. This feature can save network bandwidth and reduce the size of the message header, particularly for topics with long names. It's useful in scenarios where the same topic is repeatedly published in multiple messages, such as in sensor networks. MQTT broker supports up to 10 topic aliases. A client can use a Topic Alias field in the PUBLISH packet to replace the full topic name with the corresponding alias.

+In MQTT v5, flow control refers to the mechanism for managing the rate and size of messages that a client can handle. Flow control can be configured by setting the Maximum Packet Size and Receive Maximum parameters in the CONNECT packet. The Receive Maximum parameter allows the client to limit the number of messages sent by the broker to the number of messages that the client is able to handle. The Maximum Packet Size parameter defines the maximum size of packets that the client can receive. MQTT broker has a message size limit of 512 KiB. This feature ensures reliability and stability of the communication for constrained devices with limited processing speed or storage capabilities.

+For MQTT v5, MQTT broker is able to send negative acknowledgments (NACKs) and server-initiated disconnect packets that provide the client with more information about failures for message delivery or connection. These features help the client diagnose the reason behind a failure and take appropriate mitigating actions. MQTT broker uses the reason codes that are defined in the [MQTT v5 Specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html)

+MQTT broker is adding more MQTT v5 and MQTT v3.1.1 features in the future to align more with the MQTT specifications. The following list details the current differences between features supported by the MQTT broker and the MQTT specifications:

+Learn more about MQTT broker:

+# Topic Spaces in Azure Event GridΓÇÖs MQTT broker feature

+Topic spaces are used to simplify access control management by enabling you to grant publish or subscribe access to a group of topics at once instead of managing access for each individual topic. To publish or subscribe to any MQTT topic, you need to:

+MQTT broker supports all the MQTT wildcards defined by the [MQTT specification](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html) as follows:

+For more information about wildcards, see [Topic Wildcards in the MQTT spec](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html).

+- Topics that start with $ are reserved for internal use.

+ Title: Azure Event GridΓÇÖs MQTT broker feature - Troubleshooting guide

+description: This article provides guidance on how to troubleshoot MQTT broker related issues.

+# Guide to troubleshoot issues with Azure Event GridΓÇÖs MQTT broker feature

+This guide provides you with information on what you can do to troubleshoot things before you submit a support ticket.

+**Data distribution using push and pull delivery (preview) modes**. At any point in a data pipeline, HTTP applications can consume messages using push or pull APIs. The source of the data might include MQTT clientsΓÇÖ data, but also includes the following data sources that send their events over HTTP:

+- **X.509 certificate [authentication](mqtt-client-authentication.md)** - authenticate your devices using the IoT industry's standard mechanism for authentication.

+- **[AAD authentication](mqtt-client-azure-ad-token-and-rbac.md)** - authenticate your applications using the Azure's standard mechanism for authentication.

+A multitenant SaaS provider or platform can publish their events to Event Grid through a feature called [Partner Events](partner-events-overview.md). You can [subscribe to those events](subscribe-to-partner-events.md) and automate tasks, for example. Events from the following partners are currently available:

+- [Microsoft Graph API](subscribe-to-graph-api-events.md). Through Microsoft Graph API you can get events from [Azure AD](azure-active-directory-events.md), [Microsoft Outlook](outlook-events.md), [Teams](teams-events.md), Conversations, security alerts, and Universal Print.

+Azure Event Grid uses a pay-per-event pricing model. You only pay for what you use. For the push-style delivery that is generally available, the first 100,000 operations per month are free. Examples of operations include event publication, event delivery, delivery attempts, event filter evaluations that refer to event data properties (sometimes referred to as Advanced Filters), and events sent to a dead letter location. For details, see the [pricing page](https://azure.microsoft.com/pricing/details/event-grid/).

+Event Grid operations involving Namespaces and its resources, including MQTT and pull HTTP delivery operations, are in public preview and are available at no charge today.

+- [Pull delivery overview](pull-delivery-overview.md).

+- [Push delivery overview](push-delivery-overview.md).

+- Quickstart: [Publish and subscribe to app events using namespace topics](publish-events-using-namespace-topics.md).

+see [Understand applicability in Azure Policy](./policy-applicability.md). Azure Policy exemptions work with

+[Resource Manager modes](./definition-structure.md#resource-manager-modes), Microsoft.Kubneretes.Data, Microsoft.KeyVault.Data and Microsoft.Network.Data and don't work with the other

+A policy exemption is created as a child object on the resource hierarchy or the individual resource granted the exemption. Exemptions cannot be created at the Resource Provider mode component level.

+If the parent resource to which the exemption applies is removed, then the exemption is removed as well.

+( `policyDefinitionReferenceId`: `requiredTags`) and the **Allowed locations** built-in policy definition ( `policyDefinitionReferenceId` : `allowedLocations`):

+information. In the example, properties **requestedBy**, **approvedBy**, **approvedOn**, and

+Exemptions support an optional property `resourceSelectors`. This property works the same way in exemptions as it does in assignments, allowing for gradual rollout or rollback of an _exemption_ to certain subsets of resources in a controlled manner based on resource type, resource location, or whether the resource has a location. More details about how to use resource selectors can be found in the [assignment structure](assignment-structure.md#resource-selectors-preview). Here is an example exemption JSON, which uses resource selectors. In this example, only resources in `westcentralus` will be exempt from the policy assignment:

+Regions can be added or removed from the `resourceLocation` list in the example. Resource selectors allow for greater flexibility of where and how exemptions can be created and managed.

+## Exemption creation and management

+Exemptions are recommended for time-bound or specific scenarios where a resource or resource hierarchy should still be tracked and would otherwise be evaluated, but there's a specific reason it shouldn't be assessed for compliance. For example, if an environment has the built-in definition `Storage accounts should disable public network access` (ID: `b2982f36-99f2-4db5-8eff-283140c09693`) assigned with _effect_ set to _audit_. Upon compliance assessment, resource "StorageAcc1" is non-compliant, but StorageAcc1 must have public network access enable for business purposes. At that time, a request should be submitted to create an exemption resource that targets StorageAcc1. Once the exemption is created, StorageAcc1 will be shown as _exempt_ in compliance review.

+Regularly revisit your exemptions to ensure that all eligible items are appropriately exempted and promptly remove any no longer qualifying for exemption. At that time, exemption resources that have expired could be deleted as well.

+- Leverage [Azure Resource Graph queries on exemptions](../samples/resource-graph-samples.md#azure-policy-exemptions).

+- Learn about [the difference between exclusions and exemptions](./scope.md#scope-comparison).

+## Resource manager modes

+### -IfNotExists policy effects

+The applicability of `AuditIfNotExists` and `DeployIfNotExists` policies is based off the entire `if` condition of the policy rule. When the `if` evaluates to false, the policy isn't applicable.

+### All other policy effects

+## Resource provider modes

+### Microsoft.KeyVault.Data, Microsoft.ManagedHSM.Data, and Microsoft.DataFactory.Data

+## Not Applicable Resources

+There could be situations in which resources are applicable to an assignment based on conditions or scope, but they shouldn't be applicable due to business reasons. At that time, it would be best to apply [exclusions](./assignment-structure.md#excluded-scopes) or [exemptions](./exemption-structure.md). To learn more on when to use either, review [scope comparison](./scope.md#scope-comparison)

+> [!NOTE]

+> By design, Azure Policy does not evaluate resources under the `Microsoft.Resources` resource provider (RP) from

+policy evaluation, except for subscriptions and resource groups.

+- Learn how to [mark resources as not applicable](./assignment-structure.md#excluded-scopes).

+- Lean more on [applicability limitations](https://github.com/azure/azure-policy#known-issues)

+description: When a resource is non-compliant, there are many possible reasons. Discover what caused the non-compliance with the policy.

+understand which change altered a previously compliant resource to make it non-compliant. There are

+- [Change history (Preview)](#change-history-preview)

+- Resource details such as name, type, location, and resource ID.

+- Compliance state and timestamp of the last evaluation for the current policy assignment.

+- A list of reasons for the resource non-compliance.

+> the _Non-compliant_ resource is `Microsoft.Compute/virtualMachines` then the user must have the

+> `Microsoft.Compute/virtualMachines/read` operation. If the user doesn't have the needed

+1. From the **Resource compliance** tab of the **Policy compliance** page, select and hold (or

+ right-click) or select the ellipsis of a resource in a **compliance state** that's

+ :::image type="content" source="../media/determine-non-compliance/view-compliance-details.png" alt-text="Screenshot of the View compliance details link on the Resource compliance tab." :::

+ to the current policy assignment. In this example, the field `Microsoft.Sql/servers/version` is

+ :::image type="content" source="../media/determine-non-compliance/compliance-details-pane.png" alt-text="Screenshot of the Compliance details pane and reasons for non-compliance that current value is 12 and target value is 14." :::

+ For an `auditIfNotExists` or `deployIfNotExists` policy definition, the details include the

+ Example partial `deployIfNotExists` definition:

+ "if": {

+ "field": "type",

+ "equals": "[parameters('resourceType')]"

+ },

+ "then": {

+ "effect": "deployIfNotExists",

+ "details": {

+ "type": "Microsoft.Insights/metricAlerts",

+ "existenceCondition": {

+ "field": "name",

+ "equals": "[concat(parameters('alertNamePrefix'), '-', resourcegroup().name, '-', field('name'))]"

+ },

+ "existenceScope": "subscription",

+ "deployment": {

+ ...

+ }

+ }

+ :::image type="content" source="../media/determine-non-compliance/compliance-details-pane-existence.png" alt-text="Screenshot of Compliance details pane for ifNotExists including evaluated resource count." :::

+history (Preview)](#change-history-preview).

+| No related resources match the effect details in the policy definition. | A resource of the type defined in `then.details.type` and related to the resource defined in the **if** portion of the policy rule doesn't exist. |

+| NonModifiablePolicyAlias | NonModifiableAliasConflict: The alias '{alias}' isn't modifiable in requests using API version '{apiVersion}'. This error happens when a request using an API version where the alias doesn't support the 'modify' effect or only supports the 'modify' effect with a different token type. |

+| AppendPoliciesNotApplicable | AppendPoliciesUnableToAppend: The aliases: '{ aliases }' aren't modifiable in requests using API version: '{ apiVersion }'. This can happen in requests using API versions for which the aliases don't support the 'modify' effect, or support the 'modify' effect with a different token type. |

+| ConflictingAppendPolicies | ConflictingAppendPolicies: Found conflicting policy assignments that modify the '{notApplicableFields}' field. Policy identifiers: '{policy}'. Contact the subscription administrator to update the policy assignments. |

+| AppendPoliciesFieldsExist | AppendPoliciesFieldsExistWithDifferentValues: Policy assignments attempted to append fields which already exist in the request with different values. Fields: '{existingFields}'. Policy identifiers: '{policy}'. Contact the subscription administrator to update the policies. |

+| AppendPoliciesUndefinedFields | AppendPoliciesUndefinedFields: Found policy definition that refers to an undefined field property for API version '{apiVersion}'. Fields: '{nonExistingFields}'. Policy identifiers: '{policy}'. Contact the subscription administrator to update the policies. |

+| MissingRegistrationForType | MissingRegistrationForResourceType: The subscription isn't registered for the resource type '{ResourceType}'. Check that the resource type exists and that the resource type is registered. |

+| InvalidResourceNameWildcardPosition | The policy assignment '{0}' associated with the policy definition '{1}' couldn't be evaluated. The resource name '{2}' within an ifNotExists condition contains the wildcard '?' character in an invalid position. Wildcards can only be located at the end of the name in a segment by themselves (ex. TopLevelResourceName/?). Either fix the policy or remove the policy assignment to unblock. |

+| TooManyResourceNameSegments | The policy assignment '{0}' associated with the policy definition '{1}' couldn't be evaluated. The resource name '{2}' within an ifNotExists condition contains too many name segments. The number of name segments must be equal to or less than the number of type segments (excluding the resource provider namespace). Either fix the policy definition or remove the policy assignment to unblock. |

+| InvalidPolicyFieldPath | The field path '{0}' within the policy definition is invalid. Field paths must contain no empty segments. They might contain only alphanumeric characters with the exception of the '.' character for splitting segments and the '[*]' character sequence to access array properties. |

+| ConstraintStatusStale | There's an existing 'Audit' status, but Gatekeeper hasn't performed an audit within the last hour. |

+| ConstraintNotProcessed | There's no status and Gatekeeper hasn't performed an audit within the last hour. |

+| InvalidConstraint/Template | The resource was rejected because of one of the following reasons: invalid constraint template Rego content, invalid YAML, or a parameter type mismatch between constraint and constraint template (providing a string value when an integer was expected). |

+_Non-compliant_ resource to open a deeper view. The **Component Compliance** tab shows more information specific to the Resource Provider mode on the assigned policy with the

+settings evaluated inside the virtual machine and you need to view per-setting details. For

+_Non-compliant_, you need to know which specific settings are out of compliance and why.

+Begin by following the same steps in the [Compliance details](#compliance-details) section to view policy compliance details.

+[Azure Automanage](../../../automanage/index.yml)

+information, select each assignment using the link in the column **Name**.

+## Change history (Preview)

+1. From the **Resource compliance** tab of the **Policy compliance** page, select a resource.

+ :::image type="content" source="../media/determine-non-compliance/change-history-tab.png" alt-text="Screenshot of the Change History tab and detected change times on Resource Compliance page." :::

+ :::image type="content" source="../media/determine-non-compliance/change-history-visual-diff.png" alt-text="Screenshot of the Change History Visual Diff of the before and after state of properties on the Change history page." :::

+The _visual diff_ aides in identifying changes to a resource. The changes detected might not be

+> and Azure Policy's [Change history](../policy/how-to/determine-non-compliance.md#change-history-preview)

+because you get resource groups that you can access, without any indication that the result might be

+HDInsight on AKS is a modern, reliable, secure, and fully managed Platform as a Service (PaaS) that runs on Azure Kubernetes Service (AKS). HDInsight on AKS allows you to deploy popular Open-Source Analytics workloads like Apache Spark™, Apache Flink®️, and Trino without the overhead of managing and monitoring containers.

+* **Cluster pools** are a logical grouping of clusters that help build robust interoperability across multiple cluster types and allow enterprises to have the clusters in the same virtual network. Cluster pools provide rapid and cost-effective access to all the cluster types created on-demand and at scale. One cluster pool corresponds to one cluster in AKS infrastructure.

+* **Clusters** are individual compute workloads, such as Apache Spark, Apache Flink, and Trino that can be created rapidly in few minutes with preset configurations.

+[HDInsight](/azure/hdinsight/) offers Apache Kafka®, Apache HBase® and other analytics workloads in Platform as a Service (PaaS) formfactor.

+* [Apache Flink connecting to Apache HBase](./flink/use-flink-to-sink-kafka-message-into-hbase.md)

+* [Apache Flink connecting to Apache Kafka](./flink/join-stream-kafka-table-filesystem.md)

+* Apache Spark connecting to Apache HBase

+* Apache Spark connecting to Apache Kafka

+> [!Note]

+> - The Trino brand and trademarks are owned and managed by the [Trino Software Foundation](https://trino.io/foundation.html). No endorsement by The Trino Software Foundation is implied by the use of these marks.

+> - Apache Spark, Spark and the Spark logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+> - Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+> - Apache, Apache Flink, Flink and the Flink logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+> - Apache HBase, HBase and the HBase logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+> - Apache®, Apache Spark™, Apache HBase®, Apache Kafka®, and Apache Flink® are either registered trademarks or trademarks of the [Apache Software Foundation](https://www.apache.org/) in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

+ Title: Trademarks

+description: The Trademark and Brand Guidelines detail how you can help us protect MicrosoftΓÇÖs brand assets.

+# Trademarks

+Product names, logos and other material used on this Azure HDInsight on AKS learn pages are registered trademarks of various entities including, but not limited to, the following trademark owners and names:

+- The [Trino Software Foundation](https://trino.io/foundation.html) owns and manages the Trino brand and trademarks. The use of these marks does not imply endorsement by The Trino Software Foundation.

+- Apache Spark, Spark and the Spark logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+- Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+- Apache, Apache Flink, Flink and the Flink logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+- Apache HBase, HBase and the HBase logo are trademarks of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+- Apache®, Apache Spark™, Apache HBase®, Apache Kafka®, and Apache Flink® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. The use of these marks does not imply endorsement by The Apache Software Foundation.

+## Release date: September 7, 2023

+This release applies to HDInsight 4.x and 5.x HDInsight release will be available to all regions over several days. This release is applicable for image number **2308221128**. [How to check the image number?](./view-hindsight-cluster-image-version.md)

+HDInsight uses safe deployment practices, which involve gradual region deployment. It might take up to 10 business days for a new release or a new version to be available in all regions.

+**OS versions**

+* HDInsight 4.0: Ubuntu 18.04.5 LTS Linux Kernel 5.4

+* HDInsight 5.0: Ubuntu 18.04.5 LTS Linux Kernel 5.4

+* HDInsight 5.1: Ubuntu 18.04.5 LTS Linux Kernel 5.4

+For workload specific versions, see

+* [HDInsight 5.x component versions](./hdinsight-5x-component-versioning.md)

+* [HDInsight 4.x component versions](./hdinsight-40-component-versioning.md)

+> [!IMPORTANT]

+> This release addresses the following CVEs released by [MSRC](https://msrc.microsoft.com/update-guide/vulnerability) on September 12, 2023. The action is to update to the latest image **2308221128**. Customers are advised to plan accordingly.

+| CVE | Severity| CVE Title| Remark |

+| - | - | - | - |

+| [CVE-2023-38156](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-38156) | Important | Azure HDInsight Apache Ambari Elevation of Privilege Vulnerability |Included on 2308221128 image |

+| [CVE-2023-36419](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36419) | Important | Azure HDInsight Apache Oozie Workflow Scheduler Elevation of Privilege Vulnerability | Apply [Script action](https://hdiconfigactions2.blob.core.windows.net/msrc-script/script_action.sh) on your clusters |

+##  Coming soon

+* The max length of cluster name will be changed to 45 from 59 characters, to improve the security posture of clusters. This change will be implemented by September 30, 2023.

+* Cluster permissions for secure storage

+ * Customers can specify (during cluster creation) whether a secure channel should be used for HDInsight cluster nodes to contact the storage account.

+* In-line quota update.

+ * Request quotas increase directly from the My Quota page, which will be a direct API call, which is faster. If the APdI call fails, then customers need to create a new support request for quota increase.

+* HDInsight Cluster Creation with Custom VNets.

+ * To improve the overall security posture of the HDInsight clusters, HDInsight clusters using custom VNETs need to ensure that the user needs to have permission for `Microsoft Network/virtualNetworks/subnets/join/action` to perform create operations. Customers would need to plan accordingly as this change would be a mandatory check to avoid cluster creation failures before September 30, 2023.ΓÇ»

+* Basic and Standard A-series VMs Retirement.

+ * On August 31, 2024, we'll retire Basic and Standard A-series VMs. Before that date, you need to migrate your workloads to Av2-series VMs, which provide more memory per vCPU and faster storage on solid-state drives (SSDs). To avoid service disruptions, [migrate your workloads](https://aka.ms/Av1retirement) from Basic and Standard A-series VMs to Av2-series VMs before August 31, 2024.

+* Non-ESP ABFS clusters [Cluster Permissions for Word Readable]

+ * Plan to introduce a change in non-ESP ABFS clusters, which restricts non-Hadoop group users from executing Hadoop commands for storage operations. This change to improve cluster security posture. Customers need to plan for the updates before September 30, 2023.ΓÇ»

+If you have any more questions, contact [Azure Support](https://ms.portal.azure.com/#view/Microsoft_Azure_Support/HelpAndSupportBlade/~/overview).

+You can always ask us about HDInsight on [Azure HDInsight - Microsoft Q&A](/answers/tags/168/azure-hdinsight)

+YouΓÇÖre welcome to add more proposals and ideas and other topics here and vote for them - [HDInsight Community (azure.com)](https://feedback.azure.com/d365community/search/?q=HDInsight).

+ > [!NOTE]

+ > We advise customers to use to latest versions of HDInsight [Images](./view-hindsight-cluster-image-version.md) as they bring in the best of open source updates, Azure updates and security fixes. For more information, see [Best practices](./hdinsight-overview-before-you-start.md).

+HDInsight uses safe deployment practices, which involve gradual region deployment. it might take up to 10 business days for a new release or a new version to be available in all regions.

+| CVE | Severity| CVE Title|

+| - | - | - |

+| [CVE-2023-35393](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-35393) | Important|Azure Apache Hive Spoofing Vulnerability |

+| [CVE-2023-35394](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-35394) | Important|Azure HDInsight Jupyter Notebook Spoofing Vulnerability |

+| [CVE-2023-36877](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36877) | Important|Azure Apache Oozie Spoofing Vulnerability |

+| [CVE-2023-36881](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36881) | Important|Azure Apache Ambari Spoofing Vulnerability |

+| [CVE-2023-38188](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-38188) | Important|Azure Apache Hadoop Spoofing Vulnerability |

+## Coming soon

+HDInsight uses safe deployment practices, which involve gradual region deployment. it might take up to 10 business days for a new release or a new version to be available in all regions.

+HDInsight uses safe deployment practices, which involve gradual region deployment. it might take up to 10 business days for a new release or a new version to be available in all regions.

+HDInsight uses safe deployment practices, which involve gradual region deployment. It might take up to 10 business days for a new release or a new version to be available in all regions.

+HDInsight uses safe deployment practices, which involve gradual region deployment. It might take up to 10 business days for a new release or a new version to be available in all regions.

+HDInsight cluster comes with predefined disk space based on SKU. This space might not be sufficient in large job scenarios.

+HDInsight uses safe deployment practices, which involve gradual region deployment. It might take up to 10 business days for a new release or a new version to be available in all regions.

+HDInsight cluster comes with predefined disk space based on SKU. This space might not be sufficient in large job scenarios.

+| LLAP status service driver might get stuck with wrong Yarn app ID|[HIVE-21866](https://issues.apache.org/jira/browse/HIVE-21866)|

+As customer scenarios grow more mature and diverse, we've identified some limitations with Interactive Query (LLAP) load-based Autoscale. These limitations are caused by the nature of LLAP query dynamics, future load prediction accuracy issues, and issues in the LLAP scheduler's task redistribution. Due to these limitations, users might see their queries run slower on LLAP clusters when Autoscale is enabled. The effect on performance can outweigh the cost benefits of Autoscale.

+As customer scenarios grow more mature and diverse, we've identified some limitations with Interactive Query (LLAP) load-based Autoscale. These limitations are caused by the nature of LLAP query dynamics, future load prediction accuracy issues, and issues in the LLAP scheduler's task redistribution. Due to these limitations, users might see their queries run slower on LLAP clusters when Autoscale is enabled. The effect on performance can outweigh the cost benefits of Autoscale.

+HDInsight now uses Azure virtual machines to provision the cluster. The service will gradually migrate to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+HDInsight now uses Azure virtual machines to provision the cluster. The service is gradually migrating to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+As customer scenarios grow more mature and diverse, we've identified some limitations with Interactive Query (LLAP) load-based Autoscale. These limitations are caused by the nature of LLAP query dynamics, future load prediction accuracy issues, and issues in the LLAP scheduler's task redistribution. Due to these limitations, users might see their queries run slower on LLAP clusters when Autoscale is enabled. The impact on performance can outweigh the cost benefits of Autoscale.

+HDInsight now uses Azure virtual machines to provision the cluster. The service is gradually migrating to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+HDInsight now uses Azure virtual machines to provision the cluster. Starting from this release, the service will gradually migrate to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+HDInsight now uses Azure virtual machines to provision the cluster. Starting from this release, the service will gradually migrate to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+HDInsight now uses Azure virtual machines to provision the cluster. Starting from this release, the service will gradually migrate to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+A fault domain is a logical grouping of underlying hardware in an Azure data center. Each fault domain shares a common power source and network switch. Before HDInsight Kafka might store all partition replicas in the same fault domain. Starting from this release, HDInsight now supports automatically distribution of Kafka partitions based on Azure fault domains.

+HDInsight now uses Azure virtual machines to provision the cluster. Starting from this release, the service will gradually migrate to [Azure virtual machine scale sets](../virtual-machine-scale-sets/overview.md). The entire process might take months. After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. No breaking change is expected.

+An issue has been fixed in the Azure portal, where users were experiencing an error when they were creating an Azure HDInsight cluster using an SSH authentication type of public key. When users clicked **Review + Create**, they would receive the error "Must not contain any three consecutive characters from SSH username." This issue has been fixed, but it might require that you refresh your browser cache by hitting CTRL + F5 to load the corrected view. The workaround to this issue was to create a cluster with an ARM template.

+- [HDFS-8496](https://issues.apache.org/jira/browse/HDFS-8496): Calling stopWriter() with FSDatasetImpl lock held might block other threads (cmccabe).

+- [HBASE-16135](https://issues.apache.org/jira/browse/HBASE-16135): PeerClusterZnode under rs of removed peer might never be deleted.

+- [*HIVE-18587*](https://issues.apache.org/jira/browse/HIVE-18587): insert DML event might attempt to calculate a checksum on directories.

+- [*HIVE-18587*](https://issues.apache.org/jira/browse/HIVE-18587): insert DML event might attempt to calculate a checksum on directories.

+- There's a small possibility that some Mahout jobs might encounter "ClassNotFoundException" or "could not load class" errors related to "org.apache.commons.httpclient", "net.java.dev.jets3t", or related class name prefixes. If these errors happen, you might consider whether to manually install the needed jars in your classpath for the job, if the risk of security issues in the obsolete library is acceptable in your environment.

+- There's an even smaller possibility that some Mahout jobs might encounter crashes in Mahout's hbase-client code calls to the hadoop-common libraries, due to binary compatibility problems. Regrettably, there's no way to resolve this issue except revert to the HDP-2.4.2 version of Mahout, which might have security issues. Again, this should be unusual, and is unlikely to occur in any given Mahout job suite.

+- [RANGER-1984](https://issues.apache.org/jira/browse/RANGER-1984): HBase audit log records might not show all tags associated with accessed column.

+Fixed issues represent selected issues that were previously logged via Hortonworks Support, but are now addressed in the current release. These issues might have been reported in previous versions within the Known Issues section; meaning they were reported by customers or identified by Hortonworks Quality Engineering team.

+| BUG-99883 | [HIVE-19073](https://issues.apache.org/jira/browse/HIVE-19073), [HIVE-19145](https://issues.apache.org/jira/browse/HIVE-19145) | StatsOptimizer might mangle constant columns |

+| BUG-95542 | [HBASE-16135](https://issues.apache.org/jira/browse/HBASE-16135) | PeerClusterZnode under rs of removed peer might never be deleted |

+| BUG-96748 | [HIVE-18587](https://issues.apache.org/jira/browse/HIVE-18587) | insert DML event might attempt to calculate a checksum on directories |

+| BUG-97100 | [RANGER-1984](https://issues.apache.org/jira/browse/RANGER-1984) | HBase audit log records might not show all tags associated with accessed column |

+ >The noted values are working examples and might not be indicative of your environment. Ensure that the way you set these properties matches how your environment is configured.

+In this release HDI 5.1 version is moved to General Availability (GA) stage.

+## Release date: October 26, 2023

+This release applies to HDInsight 4.x and 5.x HDInsight release will be available to all regions over several days. This release is applicable for image number **2310140056**. [How to check the image number?](./view-hindsight-cluster-image-version.md)

+HDInsight uses safe deployment practices, which involve gradual region deployment. it might take up to 10 business days for a new release or a new version to be available in all regions.

+## What's new

+* HDInsight announces the General availability of HDInsight 5.1 starting October 26, 2023. This release brings in a full stack refresh to the [open source components](./hdinsight-5x-component-versioning.md#open-source-components-available-with-hdinsight-5x) and the integrations from Microsoft.

+ * Latest Open Source Versions ΓÇô [HDInsight 5.1](./hdinsight-5x-component-versioning.md) comes with the latest stable [open-source version](./hdinsight-5x-component-versioning.md#open-source-components-available-with-hdinsight-5x) available. Customers can benefit from all latest open source features, Microsoft performance improvements, and Bug fixes.

+ * Secure ΓÇô The latest versions come with the most recent security fixes, both open-source security fixes and security improvements by Microsoft.

+ * Lower TCO ΓÇô With performance enhancements customers can lower the operating cost, along with [enhanced autoscale](https://techcommunity.microsoft.com/t5/analytics-on-azure-blog/enhanced-autoscale-capabilities-in-hdinsight-clusters/ba-p/3811271).

+

+* Cluster permissions for secure storage

+ * Customers can specify (during cluster creation) whether a secure channel should be used for HDInsight cluster nodes to connect the storage account.

+

+* HDInsight Cluster Creation with Custom VNets.

+ * To improve the overall security posture of the HDInsight clusters, HDInsight clusters using custom VNETs need to ensure that the user needs to have permission for `Microsoft Network/virtualNetworks/subnets/join/action` to perform create operations. Customer might face creation failures if this check is not enabled.

+

+ * Non-ESP ABFS clusters [Cluster Permissions for Word Readable]

+ * Non-ESP ABFS clusters restrict non-Hadoop group users from executing Hadoop commands for storage operations. This change improves cluster security posture.ΓÇ»

+* The max length of cluster name will be changed to 45 from 59 characters, to improve the security posture of clusters. This change will be rolled out to all regions starting upcoming release.

+ * On August 31, 2024, we will retire Basic and Standard A-series VMs. Before that date, you need to migrate your workloads to Av2-series VMs, which provide more memory per vCPU and faster storage on solid-state drives (SSDs).

+ * To avoid service disruptions, [migrate your workloads](https://aka.ms/Av1retirement) from Basic and Standard A-series VMs to Av2-series VMs before August 31, 2024.

+We are listening: YouΓÇÖre welcome to add more ideas and other topics here and vote for them - [HDInsight Ideas](https://feedback.azure.com/d365community/search/?q=HDInsight) and follow us for more updates on [AzureHDInsight Community](https://www.linkedin.com/groups/14313521/)

+> [!NOTE]

+> This release addresses the following CVEs released by [MSRC](https://msrc.microsoft.com/update-guide/vulnerability) on September 12, 2023. The action is to update to the latest image 2308221128 or 2310140056. Customers are advised to plan accordingly.

+| CVE | Severity | CVE Title | Remark |

+| - | - | - | - |

+| [CVE-2023-38156](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-38156) | Important | Azure HDInsight Apache Ambari Elevation of Privilege Vulnerability |Included on image 2308221128 or 2310140056 |

+| [CVE-2023-36419](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36419) | Important | Azure HDInsight Apache Oozie Workflow Scheduler Elevation of Privilege Vulnerability | Apply [Script action](https://hdiconfigactions2.blob.core.windows.net/msrc-script/script_action.sh) on your clusters, or update to 2310140056 image |

+> [!NOTE]

+> We advise customers to use to latest versions of HDInsight [Images](./view-hindsight-cluster-image-version.md) as they bring in the best of open source updates, Azure updates and security fixes. For more information, see [Best practices](./hdinsight-overview-before-you-start.md).

+ Title: Migrate Apache Kafka workloads to Azure HDInsight 5.1

+description: Learn how to migrate Apache Kafka workloads on HDInsight 4.0 to HDInsight 5.1.

+# Migrate Apache Kafka workloads to Azure HDInsight 5.1

+Azure HDInsight 5.1 offers the latest open-source components with significant enhancements in performance, connectivity, and security. This document explains how to migrate Apache Kafka workloads from HDInsight 4.0 (Kafka 2.1) to HDInsight 5.1 (Kafka 3.2).

+## Apache Kafka versions

+### Apache Kafka 3.2.0

+

+If you migrate from Kafka to 3.2.0, you can take advantage of the following new features:

+- Support Automated consumer offsets sync across cluster in MM 2.0, making it easier to migrate or failover consumers across clusters. (KIP-545)

+- Hint to the partition leader to recover the partition: A new feature that allows the controller to communicate to a newly elected topic partition leader whether it needs to recover its state (KIP-704)

+- Supports TLS 1.2 by default for secure communication

+- Zookeeper Dependency Removal: Producers and consumers no longer need the zookeeper parameter. Use the `--bootstrap-server` option instead of `--zookeeper` with CLI commands. (KIP-500)

+- Configurable backlog size for creating Acceptor: A new configuration that allows setting the size of the SYN backlog for TCPΓÇÖs acceptor sockets on the brokers (KIP-764)

+- Top-level error code field to DescribeLogDirsResponse: A new error code that makes DescribeLogDirs API consistent with other APIs and allows returning other errors besides CLUSTER_AUTHORIZATION_FAILED (KIP-784)

+For a complete list of updates, see [Apache Kafka 3.2.0 release notes](https://archive.apache.org/dist/kafka/3.2.0/RELEASE_NOTES.html).

+## Kafka client compatibility

+New Kafka brokers support older clients. [KIP-35 - Retrieving protocol version](https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version) introduced a mechanism for dynamically determining the functionality of a Kafka broker and [KIP-97: Improved Kafka Client RPC Compatibility Policy](https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy) introduced a new compatibility policy and guarantees for the Java client. Previously, a Kafka client had to interact with a broker of the same version or a newer version. Now, newer versions of the Java clients and other clients that support KIP-35 such as `librdkafka` can fall back to older request types or throw appropriate errors if functionality isn't available.

+> [!NOTE]

+> Recommended to use kafka client version same as the cluster versions. For more information, see [Compatibility Matrix](https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix).

+## General migration process

+The following migration guidance assumes an Apache Kafka 2.1.1 cluster deployed on HDInsight 4.0 in a single virtual network. The existing broker has some topics and is being actively used by producers and consumers.

+Upgrading the Kafka version on an existing cluster isn't supported. After you create a cluster with HDI 5.1, migrate your Kafka clients to use the new cluster.

+To complete the migration, do the following steps:

+1. **Deploy a new HDInsight 5.1 cluster and clients for test.** Deploy a new HDInsight 5.1 Kafka cluster. If multiple Kafka cluster versions can be selected, it's recommended to select the latest version. After deployment, set some parameters as needed and create a topic with the same name as your existing environment. Also, set TLS and bring-your-own-key (BYOK) encryption as needed. Then check if it works correctly with the new cluster.

+ :::image type="content" source="./media/migrate-5-1-versions/deploy-new-hdinsight-clusters.png" alt-text="Screenshot shows how to Deploy new HDInsight 5.1 clusters." lightbox="./media/migrate-5-1-versions/deploy-new-hdinsight-clusters.png":::

+1. **Switch the cluster for the producer application, and wait until all the queue data is consumed by the current consumers.** When the new HDInsight 5.1 Kafka cluster is ready, switch the existing producer destination to the new cluster. Leave it as it is until the existing Consumer app has consumed all the data from the existing cluster.

+ :::image type="content" source="./media/migrate-5-1-versions/switch-cluster-producer-app.png" alt-text="Screenshot shows how to Switch cluster for producer app." lightbox="./media/migrate-5-1-versions/switch-cluster-producer-app.png":::

+1. **Switch the cluster on the consumer application.** After confirming that the existing consumer application has finished consuming all data from the existing cluster, switch the connection to the new cluster.

+ :::image type="content" source="./media/migrate-5-1-versions/switch-cluster-consumer-app.png" alt-text="Screenshot shows how to Switch cluster on consumer app." lightbox="./media/migrate-5-1-versions/switch-cluster-consumer-app.png":::

+1. **Remove the old cluster and test applications as needed.** Once the switch is complete and working properly, remove the old HDInsight 4.0 Kafka cluster and the producers and consumers used in the test as needed.

+## Next steps

+* [Performance optimization for Apache Kafka HDInsight clusters](apache-kafka-performance-tuning.md)

+* [Quickstart: Create Apache Kafka cluster in Azure HDInsight using Azure portal](apache-kafka-get-started.md)

+ * Searches for the attribute that failed validation returns the study/series/instance if the value is corrected in one of the few ways [mentioned below](#search-results-might-be-incomplete-for-extended-query-tags-with-validation-warnings).

+In the v1 API and continued for v2, if an [extended query tag](dicom-extended-query-tags-overview.md) has any errors, because one or more of the existing instances had a tag value that couldn't be indexed, then subsequent search queries containing the extended query tag return `erroneous-dicom-attributes` as detailed in the [documentation](dicom-extended-query-tags-overview.md#tag-query-status). However, tags (also known as attributes) with validation warnings from STOW-RS are **not** included in this header. If a store request results in validation warnings for [searchable attributes](dicom-services-conformance-statement-v2.md#searchable-attributes) at the time the instance was stored, those attributes may not be used to search for the stored instance. However, any [searchable attributes](dicom-services-conformance-statement-v2.md#searchable-attributes) that failed validation will be able to return results if the values are overwritten by instances in the same study/series that are stored after the failed one, or if the values are already stored correctly by a previous instance. If the attribute values are not overwritten, then they will not produce any search results.

+An attribute can be corrected in the following ways:

+- Delete the stored instance and upload a new instance with the corrected data

+- Upload a new instance in the same study/series with corrected data

+In the v1 API and continued for v2, if an [extended query tag](dicom-extended-query-tags-overview.md) has any errors, because one or more of the existing instances had a tag value that couldn't be indexed, then subsequent search queries containing the extended query tag returns `erroneous-dicom-attributes` as detailed in the [documentation](dicom-extended-query-tags-overview.md#tag-query-status). However, tags (also known as attributes) with validation warnings from STOW-RS are **not** included in this header. If a store request results in validation warnings for [searchable attributes](#searchable-attributes) at the time the [instance was stored](#store-changes-from-v1), those attributes may not be used to search for the stored instance. However, any [searchable attributes](#searchable-attributes) that failed validation will be able to return results if the values are overwritten by instances in the same study/series that are stored after the failed one, or if the values are already stored correctly by a previous instance. If the attribute values are not overwritten, then they will not produce any search results.

+An attribute can be corrected in the following ways:

+- Delete the stored instance and upload a new instance with the corrected data

+- Upload a new instance in the same study/series with corrected data

+- Monitoring and diagnostics for hundreds of thousands of connected IoT devices.

+Azure IoT Central is an application platform as a service (aPaaS) that manages scalability and HADR for you. An IoT Central application can scale to support hundreds of thousands of connected devices. For more information about device and message pricing, see [Azure IoT Central pricing](https://azure.microsoft.com/pricing/details/iot-central/). For more information about the service level agreement, see [SLA for Azure IoT Central](https://azure.microsoft.com/support/legal/sla/iot-central/v1_0/).

+| Item | Quota or limit |

+| - | -- |

+| Number of devices in an application | 200,000 |

+| Number of IoT Central simulated devices in an application | 100 |

+| Number of messages per second per application | 200 | Individual devices can temporarily send up to 10 messages per second. |

+| Item | Quota or limit |

+| - | -- |

+| Query API requests per second | 1 |

+| Other API requests per second | 20 |

+For large volumes of export data, you may experience up to 60 seconds of latency. Typically, the latency is much lower than this.

+| Number of device groups in an application | 500 | For performance reasons, you shouldn't exceed this limit. |

+| Number of devices registrations per minute | 200 | The underlying DPS instance sets this quota. |

+| Number of rules in an application | 50 | This quota is fixed and can't be changed. |

+An IoT Central application lets you monitor and manage hundreds of thousands of devices throughout their life cycle. This guide is for device developers who implement the code to run on devices that connect to IoT Central.

+An IoT Central application lets you monitor and manage hundreds of thousands of devices throughout their life cycle.

+ IoT Central provides a scalable and reliable infrastructure to capture streaming data from hundreds of thousands of connected devices. Sometimes, you need to run stream computations over the hot or warm data paths to meet business requirements. You can also merge IoT data with data in external stores such as Azure Data explorer to provide enhanced diagnostics.

+See [Configure RBAC for batch endpoint invoke](#configure-rbac-for-batch-endpoints-invoke) for a detailed list of RBAC permissions.

+### Prerequisites

+* This example assumes that you have a model correctly deployed as a batch endpoint. Particularly, we are using the *heart condition classifier* created in the tutorial [Using MLflow models in batch deployments](how-to-mlflow-batch.md).

+## Configure RBAC for Batch Endpoints invoke

+Batch Endpoints exposes a durable API consumers can use to generate jobs. The invoker request proper permission to be able to generate those jobs. You can either use one of the [built-in security roles](../role-based-access-control/built-in-roles.md) or you can create a custom role for the purposes.

+To successfully invoke a batch endpoint you need the following explicit actions granted to the identity used to invoke the endpoints. See [Steps to assign an Azure role](../role-based-access-control/role-assignments-steps.md) for instructions to assign them.

+```json

+"actions": [

+ "Microsoft.MachineLearningServices/workspaces/read",

+ "Microsoft.MachineLearningServices/workspaces/data/versions/write",

+ "Microsoft.MachineLearningServices/workspaces/datasets/registered/read",

+ "Microsoft.MachineLearningServices/workspaces/datasets/registered/write",

+ "Microsoft.MachineLearningServices/workspaces/datasets/unregistered/read",

+ "Microsoft.MachineLearningServices/workspaces/datasets/unregistered/write",

+ "Microsoft.MachineLearningServices/workspaces/datastores/read",

+ "Microsoft.MachineLearningServices/workspaces/datastores/write",

+ "Microsoft.MachineLearningServices/workspaces/datastores/listsecrets/action",

+ "Microsoft.MachineLearningServices/workspaces/listStorageAccountKeys/action",

+ "Microsoft.MachineLearningServices/workspaces/batchEndpoints/read",

+ "Microsoft.MachineLearningServices/workspaces/batchEndpoints/deployments/read",

+ "Microsoft.MachineLearningServices/workspaces/computes/read",

+ "Microsoft.MachineLearningServices/workspaces/computes/listKeys/action",

+ "Microsoft.MachineLearningServices/workspaces/metadata/secrets/read",

+ "Microsoft.MachineLearningServices/workspaces/metadata/snapshots/read",

+ "Microsoft.MachineLearningServices/workspaces/metadata/artifacts/read",

+ "Microsoft.MachineLearningServices/workspaces/metadata/artifacts/write",

+ "Microsoft.MachineLearningServices/workspaces/experiments/read",

+ "Microsoft.MachineLearningServices/workspaces/experiments/runs/submit/action",

+ "Microsoft.MachineLearningServices/workspaces/experiments/runs/read",

+ "Microsoft.MachineLearningServices/workspaces/experiments/runs/write",

+ "Microsoft.MachineLearningServices/workspaces/metrics/resource/write",

+ "Microsoft.MachineLearningServices/workspaces/modules/read",

+ "Microsoft.MachineLearningServices/workspaces/models/read",

+ "Microsoft.MachineLearningServices/workspaces/endpoints/pipelines/read",

+ "Microsoft.MachineLearningServices/workspaces/endpoints/pipelines/write",

+ "Microsoft.MachineLearningServices/workspaces/environments/read",

+ "Microsoft.MachineLearningServices/workspaces/environments/write",

+ "Microsoft.MachineLearningServices/workspaces/environments/build/action"

+ "Microsoft.MachineLearningServices/workspaces/environments/readSecrets/action"

+]

+```

+You can compare the input data, parameters, output settings, run settings of the two pipeline jobs or components using compare feature. To learn more, see [how to enable and use the graph compare feature](./how-to-use-pipeline-ui.md#compare-different-pipelines-to-debug-failure-or-other-unexpected-issues-preview)

+To identify any changes in pipeline topology, pipeline input/output, or pipeline settings between two pipelines, select **Compare graph** after adding two pipeline jobs to the compare list.

+Furthermore, you can compare two components to observe if there have been any changes in the component input/output, component setting or source code. To do this, select **Compare details** after adding two components to the compare list.

+When developing flows, you can not only use the built-in tools provided by Prompt flow, but also develop your own custom tool. In this document, we guide you through the process of developing your own tool package, offering detailed steps and advice on how to utilize your creation.

+After successful installation, your custom "tool" can show up in the tool list:

+Your tool package should be a python package. To develop your custom tool, follow the steps **Create your own tool package** and **build and share the tool package** in [Create and Use Tool package](https://microsoft.github.io/promptflow/how-to-guides/develop-a-tool/create-and-use-tool-package.html). You can also [Add a tool icon](https://microsoft.github.io/promptflow/how-to-guides/develop-a-tool/add-a-tool-icon.html) and [Add Category and tags](https://microsoft.github.io/promptflow/how-to-guides/develop-a-tool/add-category-and-tags-for-tool.html) for your tool.

+To add the custom tool to your tool list, it's necessary to create a runtime, which is based on a customized environment where your custom tool is preinstalled. Here we use [my-tools-package](https://pypi.org/project/my-tools-package/) as an example to prepare the runtime.

+ 1. Create a customized environment in Azure Machine Learning studio by going to **Environments** then select **Create**. In the settings tab under *Select environment source*, choose " Create a new docker content."

+ RUN pip install my-tools-package==0.0.1

+ It takes several minutes to create the environment. After it succeeded, you can copy the Azure Container Registry (ACR) from environment detail page for the next step.

+### Prepare compute instance runtime

+1. Create a compute instance runtime using the customized environment created in step 2.

+ 1. Create a new compute instance. Existing compute instance created long time ago can possibly hit unexpected issue.

+ 2. Create runtime on CI with customized environment.

+1. Install Prompt flow for VS Code extension

+ :::image type="content" source="./media/how-to-custom-tool-package-creation-and-usage/prompt-flow-vs-code-extension.png" alt-text="Screenshot of Prompt flow VS Code extension."lightbox ="./media/how-to-custom-tool-package-creation-and-usage/prompt-flow-vs-code-extension.png":::

+2. Go to terminal and install your tool package in conda environment of the extension. Assume your conda env name is `prompt-flow`.

+3. Go to the extension and open one flow folder. Select 'flow.dag.yaml' and preview the flow. Next, select `+` button and you can see your tools. You need to **reload the windows** to clean previous cache if you don't see your tool in the list.

+ 3. Run this script in your conda environment. It returns the metadata of all tools installed in your local environment, and you should verify that your tools are listed.

+- If you encounter a `403 Forbidden Error`, it's likely due to a naming conflict with an existing package. You need to choose a different name. Package names must be unique on PyPI to avoid confusion and conflicts among users. Before creating a new package, it's recommended to search PyPI (https://pypi.org/) to verify that your chosen name isn't already taken. If the name you want is unavailable, consider selecting an alternative name or a variation that clearly differentiates your package from the existing one.

+> [!IMPORTANT]

+> For more information, you can refer to [the Prompt flow CLI documentation for Azure](https://microsoft.github.io/promptflow/reference/pfazure-command-reference.html).

+During iterative development, as you refine and fine-tune your flow or prompts, it could be beneficial to carry out multiple iterations locally within your code repository. The community version, **Prompt flow VS Code extension** and **Prompt flow local SDK & CLI** is provided to facilitate pure local development and testing without Azure binding.

+- [Prompt flow CLI documentation for Azure](https://microsoft.github.io/promptflow/reference/pfazure-command-reference.html)

+ Title: Embedding tool in Azure Machine Learning prompt flow (preview)

+description: Prompt flow embedding tool uses OpenAI's embedding models to convert text into dense vector representations for various NLP tasks.

+# Embedding tool (preview)

+OpenAI's embedding models convert text into dense vector representations for various NLP tasks. See the [OpenAI Embeddings API](https://platform.openai.com/docs/api-reference/embeddings) for more information.

+> [!IMPORTANT]

+> Prompt flow is currently in public preview. This preview is provided without a service-level agreement, and is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+## Prerequisites

+Create OpenAI resources:

+- **OpenAI**

+ Sign up account [OpenAI website](https://openai.com/)

+ Login and [Find personal API key](https://platform.openai.com/account/api-keys)

+- **Azure OpenAI (AOAI)**

+ Create Azure OpenAI resources with [instruction](../../../ai-services/openai/how-to/create-resource.md)

+## **Connections**

+Set up connections to provide resources in embedding tool.

+| Type | Name | API KEY | API Type | API Version |

+|-|-|-|-|-|

+| OpenAI | Required | Required | - | - |

+| AzureOpenAI | Required | Required | Required | Required |

+## Inputs

+| Name | Type | Description | Required |

+||-|--|-|

+| input | string | the input text to embed | Yes |

+| connection | string | the connection for the embedding tool use to provide resources | Yes |

+| model/deployment_name | string | instance of the text-embedding engine to use. Fill in model name if you use OpenAI connection, or deployment name if use Azure OpenAI connection. | Yes |

+## Outputs

+| Return Type | Description |

+|-||

+| list | The vector representations for inputs |

+There is an example response returned by the embedding tool:

+<details>

+ <summary>Output</summary>

+

+```

+[-0.005744616035372019,

+-0.007096089422702789,

+-0.00563855143263936,

+-0.005272455979138613,

+-0.02355326898396015,

+0.03955197334289551,

+-0.014260607771575451,

+-0.011810848489403725,

+-0.023170066997408867,

+-0.014739611186087132,

+...]

+```

+</details>

+> [!NOTE]

+> We now remove the `embedding` option from LLM tool api setting. You can use embedding api with [Embedding tool](embedding-tool.md).

+| prompt | string | text prompt for the language model. | Yes |

+ Title: More tools in Prompt flow

+description: More tools in Prompt flow are displayed in the table, along with instructions for custom tool package creation and tool package usage.

+# More tools in Prompt flow (preview)

+This table provides an index of more tools. If existing tools can't meet your requirements, you can [develop your own custom tool and make a tool package](https://microsoft.github.io/promptflow/how-to-guides/develop-a-tool/create-and-use-tool-package.html).

+> [!IMPORTANT]

+> Prompt flow is currently in public preview. This preview is provided without a service-level agreement, and is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+| Tool name | Description | Environment | Package Name |

+||--|-|--|

+| [Python](./python-tool.md) | Run Python code. | Default | -- |

+| [LLM](./llm-tool.md) | Use Open AI's Large Language Model for text completion or chat. | Default | -- |

+| [Prompt](./prompt-tool.md) | Craft prompt using Jinja as the templating language. | Default | -- |

+| [Embedding](./embedding-tool.md) | Use Open AI's embedding model to create an embedding vector representing the input text. | Default | -- |

+| [Open Source LLM](./open-source-llm-tool.md) | Use an Open Source model from the Azure Model catalog, deployed to an Azure Machine Learning Online Endpoint for LLM Chat or Completion API calls. | Default | -- |

+| [Serp API](./serp-api-tool.md) | Use Serp API to obtain search results from a specific search engine. | Default | -- |

+| [Content Safety (Text)](./content-safety-text-tool.md) | Use Azure Content Safety to detect harmful content. | Default | [promptflow-contentsafety](https://pypi.org/project/promptflow-contentsafety/) |

+| [Faiss Index Lookup](./faiss-index-lookup-tool.md) | Search vector based query from the FAISS index file. | Default | [promptflow-vectordb](https://pypi.org/project/promptflow-vectordb/) |

+| [Vector DB Lookup](./vector-db-lookup-tool.md) | Search vector based query from existing Vector Database. | Default | [promptflow-vectordb](https://pypi.org/project/promptflow-vectordb/) |

+| [Vector Index Lookup](./vector-index-lookup-tool.md) | Search text or vector based query from Azure Machine Learning Vector Index. | Default | [promptflow-vectordb](https://pypi.org/project/promptflow-vectordb/) |

+To discover more custom tools that developed by the open source community, see [more custom tools](https://microsoft.github.io/promptflow/integrations/tools/https://docsupdatetracker.net/index.html).

+For the tools that should be utilized in the custom environment, see [Custom tool package creation and usage](../how-to-custom-tool-package-creation-and-usage.md#prepare-runtime) to prepare the runtime. Then the tools can be displayed in the tool list.

+ Title: Open source LLM tool in Azure Machine Learning prompt flow (preview)

+description: The Prompt flow Open Source LLM tool enables you to utilize various Open Source and Foundational Models.

+# Open Source LLM (preview)

+The Prompt flow Open Source LLM tool enables you to utilize various Open Source and Foundational Models, such as [Falcon](https://aka.ms/AAlc25c) or [Llama 2](https://aka.ms/AAlc258) for natural language processing, in PromptFlow.

+Here's how it looks in action on the Visual Studio Code Prompt flow extension. In this example, the tool is being used to call a LlaMa-2 chat endpoint and asking "What is CI?".

+This Prompt flow supports two different LLM API types:

+- **Chat**: Shown in the example above. The chat API type facilitates interactive conversations with text-based inputs and responses.

+- **Completion**: The Completion API type is used to generate single response text completions based on provided prompt input.

+> [!IMPORTANT]

+> Prompt flow is currently in public preview. This preview is provided without a service-level agreement, and is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+## Quick Overview: How do I use Open Source LLM Tool?

+1. Choose a Model from the Azure Machine Learning Model Catalog and deploy.

+2. Setup and select the connections to the model deployment.

+3. Configure the tool with the model settings.

+4. [Prepare the Prompt](./prompt-tool.md#how-to-write-prompt).

+5. Run the flow.

+## Prerequisites: Model Deployment

+1. Pick the model that matched your scenario from the [Azure Machine Learning model catalog](https://ml.azure.com/model/catalog).

+2. Use the "Deploy" button to deploy the model to an Azure Machine Learning Online Inference endpoint.

+To learn more, see [Deploying foundation models to endpoints for inferencing.](../../how-to-use-foundation-models.md#deploying-foundation-models-to-endpoints-for-inferencing)

+## Prerequisites: Prompt flow Connections

+In order for Prompt flow to use your deployed model, you need to set up a Connection. Explicitly, the Open Source LLM tool uses the CustomConnection.

+1. To learn how to create a custom connection, see [create a connection](https://microsoft.github.io/promptflow/how-to-guides/manage-connections.html#create-a-connection).

+ The keys to set are:

+ 1. **endpoint_url**

+ - This value can be found at the previously created Inferencing endpoint.

+ 2. **endpoint_api_key**

+ - Ensure to set this key as a secret value.

+ - This value can be found at the previously created Inferencing endpoint.

+ 3. **model_family**

+ - Supported values: LLAMA, DOLLY, GPT2, or FALCON

+ - This value is dependent on the type of deployment you're targeting.

+## Running the Tool: Inputs

+The Open Source LLM tool has many parameters, some of which are required. See the below table for details, you can match these parameters to the screenshot above for visual clarity.

+| Name | Type | Description | Required |

+|||-|-|

+| api | string | This parameter is the API mode and depends on the model used and the scenario selected. *Supported values: (Completion \| Chat)* | Yes |

+| connection | CustomConnection | This parameter is the name of the connection, which points to the Online Inferencing endpoint. | Yes |

+| model_kwargs | dictionary | This input is used to provide configuration specific to the model used. For example, the Llama-02 model uses {\"temperature\":0.4}. *Default: {}* | No |

+| deployment_name | string | The name of the deployment to target on the Online Inferencing endpoint. If no value is passed, the Inferencing load balancer traffic settings are used. | No |

+| prompt | string | The text prompt that the language model uses to generate its response. | Yes |

+## Outputs

+| API | Return Type | Description |

+||-||

+| Completion | string | The text of one predicted completion |

+| Chat | string | The text of one response int the conversation |

+### Types

+| Type | Python example | Description |

+|--||--|

+| int | param: int | Integer type |

+| bool | param: bool | Boolean type |

+| string | param: str | String type |

+| double | param: float | Double type |

+| list | param: list or param: List[T] | List type |

+| object | param: dict or param: Dict[K, V] | Object type |

+| [Connection](../concept-connections.md) | param: CustomConnection | Connection type will be handled specially |

+Parameters with `Connection` type annotation will be treated as connection inputs, which means:

+- Prompt flow extension will show a selector to select the connection.

+- During execution time, prompt flow will try to find the connection with the name same from parameter value passed in.

+> [!Note]

+> `Union[...]` type annotation is supported **ONLY** for connection type, for example, `param: Union[CustomConnection, OpenAIConnection]`.

+ *The sample in the next section defines python tool "my_python_tool" which decorated with @tool*

+This snippet shows the basic structure of a tool function. Prompt flow will read the function and extract inputs from function parameters and type annotations.

+from promptflow.connections import CustomConnection

+# Please update the function name/signature per need

+def my_python_tool(message: str, my_conn: CustomConnection) -> str:

+ my_conn_dict = dict(my_conn)

+ # Do some function call with my_conn_dict...

+| Name | Type | Sample Value in Flow Yaml | Value passed to function|

+||--|-| |

+| message | string | "world" | "world" |

+| my_conn | CustomConnection | "my_conn" | CustomConnection object |

+Prompt flow will try to find the connection named 'my_conn' during execution time.

+This article contains reference information for [configuring Kubernetes with Azure Machine Learning](./how-to-attach-kubernetes-anywhere.md).

+|Oct 11, 2023 | 1.1.35| Fix vulnerable image. Bug fixes. |

+|Aug 25, 2023 | 1.1.34| Fix vulnerable image. Return more detailed identity error. Bug fixes. |

+| Mar 27, 2023| 1.1.25 | Add Azure Machine Learning job throttle. Fast fail for training job when SSH setup failed. Reduce Prometheus scrape interval to 30s. Improve error messages for inference. Fix vulnerable image. |

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=start-spark-session)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=root-dir)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=init-prj-ws-client)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=init-fs-ws-client)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=init-fs-core-sdk)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=redis-settings)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=provision-redis)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=reuse-redis)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=retrieve-uai)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=attach-online-store)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=enable-accounts-material)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=start-accounts-backfill)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=track-accounts-backfill)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=enable-transact-material)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=start-transact-material)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=track-transact-material)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=parse-feat-list)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=init-online-lookup)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=online-feat-loockup)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=define-endpoint)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=create-endpoint)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=get-endpoint-identity)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=endpoint-redis-rbac)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=endpoint-fs-rbac)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=define-online-deployment)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=begin-online-deployment)]

+ [!notebook-python[] (~/azureml-examples-temp-fix/sdk/python/featurestore_sample/notebooks/sdk_only/5. Enable online store and run online inference.ipynb?name=test-online-deployment)]

+ :::image type="content" source="./media/tutorial-discover-aws/search-subscription.png" alt-text="Screenshot of Search box to search for the Azure subscription.":::

+ :::image type="content" source="../../includes/role-based-access-control/media/add-role-assignment-page.png" alt-text="Screenshot of Add role assignment page in Azure portal.":::

+> If you have already created a project, you can use the same project to register additional appliances to discover and assess more no of servers. [Learn more](create-manage-projects.md#find-a-project).

+ :::image type="content" source="./media/tutorial-discover-physical/key-format.png" alt-text="Screenshot of SSH private key supported format.":::

+* Appliance can connect to only those SQL Server instances to which it has network line of sight, whereas software inventory by itself might not need network line of sight.

+ :::image type="content" source="./media/tutorial-discover-gcp/search-subscription.png" alt-text="Screenshot of Search box to search for the Azure subscription.":::

+ :::image type="content" source="../../includes/role-based-access-control/media/add-role-assignment-page.png" alt-text="Screenshot of role assignment page in Azure portal.":::

+> If you have already created a project, you can use the same project to register additional appliances to discover and assess more no of servers. [Learn more](create-manage-projects.md#find-a-project).

+ :::image type="content" source="./media/tutorial-discover-physical/key-format.png" alt-text="Screenshot of SSH private key supported format.":::

+* Appliance can connect to only those SQL Server instances to which it has network line of sight, whereas software inventory by itself might not need network line of sight.

+**Servers** | All Windows and Linux OS versions are supported for discovery of configuration and performance metadata. <br /><br /> For application discovery on servers, all Windows and Linux OS versions are supported. Check the [OS versions supported for agentless dependency analysis](migrate-support-matrix-hyper-v.md#dependency-analysis-requirements-agentless).<br /><br /> To discover ASP.NET web apps running on IIS web server, check [supported Windows OS and IIS versions](migrate-support-matrix-vmware.md#web-apps-discovery-requirements). For discovery of installed applications and for agentless dependency analysis, Windows servers must have PowerShell version 2.0 or later installed.<br /><br /> To discover Java web apps running on Apache Tomcat web server, check [supported Linux OS and Tomcat versions](migrate-support-matrix-vmware.md#web-apps-discovery-requirements).

+ :::image type="content" source="./media/tutorial-discover-hyper-v/search-subscription.png" alt-text="Screenshot of Search box to search for the Azure subscription.":::

+ :::image type="content" source="../../includes/role-based-access-control/media/add-role-assignment-page.png" alt-text="Screenshot of add role assignment page in Azure portal.":::

+ :::image type="content" source="./media/tutorial-discover-hyper-v/added-tool.png" alt-text="Screenshot showing Azure Migrate: Discovery and assessment tool added by default.":::

+> If you have already created a project, you can use the same project to register additional appliances to discover and assess more no of servers. [Learn more](create-manage-projects.md#find-a-project)

+* Appliance can connect to only those SQL Server instances to which it has network line of sight, whereas software inventory by itself might not need network line of sight.

+ :::image type="content" source="./media/tutorial-discover-vmware/guest-operations.png" alt-text="Screenshot that shows the vSphere web client and how to create a new account and select user roles and privileges.":::

+* Appliance can connect to only those SQL Server instances to which it has network line of sight, whereas software inventory by itself might not need network line of sight.

+Details such as OS license support status, inventory, database instances, etc. are displayed.

+In this tutorial, you use Azure Network Watcher [next hop](network-watcher-next-hop-overview.md) tool to troubleshoot and diagnose a VM routing problem that's preventing it from correctly communicating with other resources. Next hop shows you that a [custom route](../virtual-network/virtual-networks-udr-overview.md?toc=/azure/network-watcher/toc.json#custom-routes) caused the routing problem.

+1. In the search box at the top of the portal, enter ***myResourceGroup***. Select **myResourceGroup** from the search results.

+## Next step

+> [!div class="nextstepaction"]

+> [Diagnose a virtual machine network routing problem](diagnose-vm-network-routing-problem.md)

+ - Standard_E4as_v5

+ - Standard_E8as_v5

+ - Standard_E16as_v5

+ Refer to the [QuickStart guide](./quickstarts-virtual-machine-deployment-cli.md) to deploy a VNF using the image you created.

+Before proceeding with this how-to guide, ensure you have completed all steps in the Azure Operator Nexus virtual machine [QuickStart guide](./quickstarts-virtual-machine-deployment-cli.md).

+> The resourceId argument must be specified in the form of an ARM ID, and it must be a valid resource ID for the target object. If the resourceId is incorrect or invalid, the placement hint will not work correctly, and the VM scheduling might fail.

+> In this and the following examples, only variations of the `--placement-hints` argument are provided. For the actual creation of the VM with placement hints, you should add `--placement-hints` to the CLI illustrated in the VM [QuickStart guide](./quickstarts-virtual-machine-deployment-cli.md).

+> The current placement hint configuration with the Affinity hintType ensures that the virtual machine is scheduled exclusively on the specified rack with the provided rack ID. However, it's important to note that the rack affinity cannot be specified for more than one rack with `Hard` scheduling execution. This limitation might influence your deployment strategy, particularly if you are considering placing VMs on multiple racks and allowing the scheduler to select from them.

+This placement hint uses both the `AntiAffinity` hintType and `Hard` schedulingExecution to prevent the virtual machine from being scheduled on the specified rack identified by the rack ID. In this configuration, the scheduler strictly follows these placement hints. However, if the rack ID is incorrect or there's not enough capacity on other racks, the VM placement might fail due to the strict application of the `Hard` scheduling rule

+This placement hint uses both the `AntiAffinity` hintType and `Hard` schedulingExecution to prevent the virtual machine from being scheduled on the specified bare-metal machine identified by the bare-metal machine ID. In this configuration, the scheduler strictly follows these placement hints. However, if the bare-metal machine ID is incorrect or there's not enough capacity on other bare-metal machines, the VM placement might fail due to the strict application of the `Hard` scheduling rule

+ Title: Create an Azure Operator Nexus virtual machine by using Azure Resource Manager template (ARM template)

+description: Learn how to create an Azure Operator Nexus virtual machine (VM) for virtual network function (VNF) workloads by using Azure Resource Manager template (ARM template).

+# Quickstart: Create an Azure Operator Nexus virtual machine by using Azure Resource Manager template (ARM template)

+* Deploy an Azure Nexus virtual machine using an Azure Resource Manager template.

+This quick-start guide is designed to help you get started with using Nexus virtual machines to host virtual network functions (VNFs). By following the steps outlined in this guide, you're able to quickly and easily create a customized Nexus virtual machine that meets your specific needs and requirements. Whether you're a beginner or an expert in Nexus networking, this guide is here to help. You learn everything you need to know to create and customize Nexus virtual machines for hosting virtual network functions.

+## Before you begin

+* Complete the [prerequisites](./quickstarts-tenant-workload-prerequisites.md) for deploying a Nexus virtual machine.

+## Review the template

+Before deploying the virtual machine template, let's review the content to understand its structure.

+Once you have reviewed and saved the template file named ```virtual-machine-arm-template.json```, proceed to the next section to deploy the template.

+## Deploy the template

+1. Create a file named ```virtual-machine-parameters.json``` and add the required parameters in JSON format. You can use the following example as a starting point. Replace the values with your own.

+2. Deploy the template.

+```azurecli

+ az deployment group create --resource-group myResourceGroup --template-file virtual-machine-arm-template.json --parameters @virtual-machine-parameters.json

+```

+## Review deployed resources

+## Clean up resources

+## Next steps

+You've successfully created a Nexus virtual machine. You can now use the virtual machine to host virtual network functions (VNFs).

+ Title: Create an Azure Operator Nexus virtual machine by using Bicep

+description: Learn how to create an Azure Operator Nexus virtual machine (VM) for virtual network function (VNF) workloads by using Bicep

+# Quickstart: Create an Azure Operator Nexus virtual machine by using Bicep

+* Deploy an Azure Nexus virtual machine using Bicep

+This quick-start guide is designed to help you get started with using Nexus virtual machines to host virtual network functions (VNFs). By following the steps outlined in this guide, you're able to quickly and easily create a customized Nexus virtual machine that meets your specific needs and requirements. Whether you're a beginner or an expert in Nexus networking, this guide is here to help. You learn everything you need to know to create and customize Nexus virtual machines for hosting virtual network functions.

+## Before you begin

+* Complete the [prerequisites](./quickstarts-tenant-workload-prerequisites.md) for deploying a Nexus virtual machine.

+## Review the template

+Before deploying the virtual machine template, let's review the content to understand its structure.

+Once you have reviewed and saved the template file named ```virtual-machine-bicep-template.bicep```, proceed to the next section to deploy the template.

+## Deploy the template

+1. Create a file named ```virtual-machine-parameters.json``` and add the required parameters in JSON format. You can use the following example as a starting point. Replace the values with your own.

+2. Deploy the template.

+```azurecli

+ az deployment group create --resource-group myResourceGroup --template-file virtual-machine-bicep-template.bicep --parameters @virtual-machine-parameters.json

+```

+## Review deployed resources

+## Clean up resources

+## Next steps

+You've successfully created a Nexus virtual machine. You can now use the virtual machine to host virtual network functions (VNFs).

+ Title: Create an Azure Operator Nexus virtual machine by using Azure CLI

+description: Learn how to create an Azure Operator Nexus virtual machine (VM) for virtual network function (VNF) workloads

+# Quickstart: Create an Azure Operator Nexus virtual machine by using Azure CLI

+* Deploy an Azure Nexus virtual machine using Azure CLI

+This quick-start guide is designed to help you get started with using Nexus virtual machines to host virtual network functions (VNFs). By following the steps outlined in this guide, you're able to quickly and easily create a customized Nexus virtual machine that meets your specific needs and requirements. Whether you're a beginner or an expert in Nexus networking, this guide is here to help. You learn everything you need to know to create and customize Nexus virtual machines for hosting virtual network functions.

+## Before you begin

+* Complete the [prerequisites](./quickstarts-tenant-workload-prerequisites.md) for deploying a Nexus virtual machine.

+## Create a Nexus virtual machine

+The following example creates a virtual machine named *myNexusVirtualMachine* in resource group *myResourceGroup* in the *eastus* location.

+Before you run the commands, you need to set several variables to define the configuration for your virtual machine. Here are the variables you need to set, along with some default values you can use for certain variables:

+| Variable | Description |

+| -- | |

+| LOCATION | The Azure region where you want to create your virtual machine. |

+| RESOURCE_GROUP | The name of the Azure resource group where you want to create the virtual machine. |

+| SUBSCRIPTION | The ID of your Azure subscription. |

+| CUSTOM_LOCATION | This argument specifies a custom location of the Nexus instance. |

+| CSN_ARM_ID | CSN ID is the unique identifier for the cloud services network you want to use. |

+| L3_NETWORK_ID | L3 Network ID is the unique identifier for the network interface to be used by the virtual machine. |

+| NETWORK_INTERFACE_NAME | The name of the L3 network interface for the virtual machine. |

+| ADMIN_USERNAME | The username for the virtual machine administrator. |

+| SSH_PUBLIC_KEY | The SSH public key that is used for secure communication with the virtual machine. |

+| CPU_CORES | The number of CPU cores for the virtual machine (even number, max 46 vCPUs) |

+| MEMORY_SIZE | The amount of memory (in GB, max 224 GB) for the virtual machine. |

+| VM_DISK_SIZE | The size (in GB) of the virtual machine disk. |

+| VM_IMAGE | The URL of the virtual machine image. |

+| ACR_URL | The URL of the Azure Container Registry. |

+| ACR_USERNAME | The username for the Azure Container Registry. |

+| ACR_PASSWORD | The password for the Azure Container Registry. |

+Once you've defined these variables, you can run the Azure CLI command to create the virtual machine. Add the ```--debug``` flag at the end to provide more detailed output for troubleshooting purposes.

+To define these variables, use the following set commands and replace the example values with your preferred values. You can also use the default values for some of the variables, as shown in the following example:

+```bash

+# Azure parameters

+RESOURCE_GROUP="myResourceGroup"

+SUBSCRIPTION="<Azure subscription ID>"

+CUSTOM_LOCATION="/subscriptions/<subscription_id>/resourceGroups/<managed_resource_group>/providers/microsoft.extendedlocation/customlocations/<custom-location-name>"

+LOCATION="$(az group show --name $RESOURCE_GROUP --query location --subscription $SUBSCRIPTION -o tsv)"

+# VM parameters

+VM_NAME="myNexusVirtualMachine"

+# VM credentials

+ADMIN_USERNAME="azureuser"

+SSH_PUBLIC_KEY="$(cat ~/.ssh/id_rsa.pub)"

+# Network parameters

+CSN_ARM_ID="/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.NetworkCloud/cloudServicesNetworks/<csn-name>"

+L3_NETWORK_ID="/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.NetworkCloud/l3Networks/<l3Network-name>"

+NETWORK_INTERFACE_NAME="mgmt0"

+# VM Size parameters

+CPU_CORES=4

+MEMORY_SIZE=12

+VM_DISK_SIZE="64"

+# Virtual Machine Image parameters

+VM_IMAGE="<VM image, example: myacr.azurecr.io/ubuntu:20.04>"

+ACR_URL="<Azure container registry URL, example: myacr.azurecr.io>"

+ACR_USERNAME="<Azure container registry username>"

+ACR_PASSWORD="<Azure container registry password>"

+```

+> [!IMPORTANT]

+> It is essential that you replace the placeholders for CUSTOM_LOCATION, CSN_ARM_ID, L3_NETWORK_ID and ACR parameters with your actual values before running these commands.

+After defining these variables, you can create the virtual machine by executing the following Azure CLI command.

+```bash

+az networkcloud virtualmachine create \

+ --name "$VM_NAME" \

+ --resource-group "$RESOURCE_GROUP" \

+ --subscription "$SUBSCRIPTION" \

+ --extended-location name="$CUSTOM_LOCATION" type="CustomLocation" \

+ --location "$LOCATION" \

+ --admin-username "$ADMIN_USERNAME" \

+ --csn "attached-network-id=$CSN_ARM_ID" \

+ --cpu-cores $CPU_CORES \

+ --memory-size $MEMORY_SIZE \

+ --network-attachments '[{"attachedNetworkId":"'$L3_NETWORK_ID'","ipAllocationMethod":"Dynamic","defaultGateway":"True","networkAttachmentName":"'$NETWORK_INTERFACE_NAME'"}]'\

+ --storage-profile create-option="Ephemeral" delete-option="Delete" disk-size="$VM_DISK_SIZE" \

+ --vm-image "$VM_IMAGE" \

+ --ssh-key-values "$SSH_PUBLIC_KEY" \

+ --vm-image-repository-credentials registry-url="$ACR_URL" username="$ACR_USERNAME" password="$ACR_PASSWORD"

+```

+After a few minutes, the command completes and returns information about the virtual machine. You've created the virtual machine. You're now ready to use them.

+## Review deployed resources

+## Clean up resources

+## Next steps

+You've successfully created a Nexus virtual machine. You can now use the virtual machine to host virtual network functions (VNFs).

+ Title: Create an Azure Operator Nexus virtual machine by using Azure PowerShell

+description: Learn how to create an Azure Operator Nexus virtual machine (VM) for virtual network function (VNF) workloads using PowerShell

+# Quickstart: Create an Azure Operator Nexus virtual machine by using Azure PowerShell

+* Deploy an Azure Nexus virtual machine using Azure PowerShell

+This quick-start guide is designed to help you get started with using Nexus virtual machines to host virtual network functions (VNFs). By following the steps outlined in this guide, you're able to quickly and easily create a customized Nexus virtual machine that meets your specific needs and requirements. Whether you're a beginner or an expert in Nexus networking, this guide is here to help. You learn everything you need to know to create and customize Nexus virtual machines for hosting virtual network functions.

+## Before you begin

+* Complete the [prerequisites](./quickstarts-tenant-workload-prerequisites.md) for deploying a Nexus virtual machine.

+## Create a Nexus virtual machine

+The following example creates a virtual machine named *myNexusVirtualMachine* in resource group *myResourceGroup* in the *eastus* location.

+Before you run the commands, you need to set several variables to define the configuration for your virtual machine. Here are the variables you need to set, along with some default values you can use for certain variables:

+| Variable | Description |

+| -- | |

+| LOCATION | The Azure region where you want to create your virtual machine. |

+| RESOURCE_GROUP | The name of the Azure resource group where you want to create the virtual machine. |

+| SUBSCRIPTION | The ID of your Azure subscription. |

+| CUSTOM_LOCATION | This argument specifies a custom location of the Nexus instance. |

+| CSN_ARM_ID | CSN ID is the unique identifier for the cloud services network you want to use. |

+| L3_NETWORK_ID | L3 Network ID is the unique identifier for the network interface to be used by the virtual machine. |

+| NETWORK_INTERFACE_NAME | The name of the L3 network interface for the virtual machine. |

+| ADMIN_USERNAME | The username for the virtual machine administrator. |

+| SSH_PUBLIC_KEY | The SSH public key that is used for secure communication with the virtual machine. |

+| CPU_CORES | The number of CPU cores for the virtual machine (even number, max 46 vCPUs) |

+| MEMORY_SIZE | The amount of memory (in GB, max 224 GB) for the virtual machine. |

+| VM_DISK_SIZE | The size (in GB) of the virtual machine disk. |

+| VM_IMAGE | The URL of the virtual machine image. |

+| ACR_URL | The URL of the Azure Container Registry. |

+| ACR_USERNAME | The username for the Azure Container Registry. |

+| ACR_PASSWORD | The password for the Azure Container Registry. |

+| VMDEVICEMODEL | The VMDeviceModel defaults to T2, available options T2(Modern) and T1(Transitional). |

+| USERDATA | The base64 encoded string of cloud-init userdata. |

+| BOOTMETHOD | The Method used to boot the virutalmachine UEFI or BIOS. |

+| OS_DISK_CREATE_OPTION | The OS disk create specifies ephemeral disk option. |

+| OS_DISK_DELETE_OPTION | The OS disk delete specifies delete disk option. |

+| IP_AllOCATION_METHOD | The IpAllocationMethod valid for L3Networks specify Dynamic or Static or Disabled. |

+| NETWORKATTACHMENTNAME | The name of the Network to attach for workload. |

+| NETWORKDATA | The base64 encoded string of cloud-init network data. |

+Once you've defined these variables, you can run the Azure PowerShell command to create the virtual machine. Add the ```-Debug``` flag at the end to provide more detailed output for troubleshooting purposes.

+To define these variables, use the following set commands and replace the example values with your preferred values. You can also use the default values for some of the variables, as shown in the following example:

+```azurepowershell-interactive

+# Azure parameters

+$RESOURCE_GROUP="myResourceGroup"

+$SUBSCRIPTION="<Azure subscription ID>"

+$CUSTOM_LOCATION="/subscriptions/<subscription_id>/resourceGroups/<managed_resource_group>/providers/microsoft.extendedlocation/customlocations/<custom-location-name>"

+$CUSTOM_LOCATION_TYPE="CustomLocation"

+$LOCATION="<ClusterAzureRegion>"

+# VM parameters

+$VM_NAME="myNexusVirtualMachine"

+$BOOT_METHOD="UEFI"

+$OS_DISK_CREATE_OPTION="Ephemeral"

+$OS_DISK_DELETE_OPTION="Delete"

+$NETWORKDATA="bmV0d29ya0RhdGVTYW1wbGU="

+$VMDEVICEMODEL="T2"

+$USERDATA=""

+# VM credentials

+$ADMIN_USERNAME="admin"

+$SSH_PUBLIC_KEY = @{

+ KeyData = "$(cat ~/.ssh/id_rsa.pub)"

+}

+# Network parameters

+$CSN_ARM_ID="/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.NetworkCloud/cloudServicesNetworks/<csn-name>"

+$L3_NETWORK_ID="/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.NetworkCloud/l3Networks/<l3Network-name>"

+$IP_AllOCATION_METHOD="Dynamic"

+$CSN_ATTACHMENT_DEFAULTGATEWAY="False"

+$CSN_ATTACHMENT_NAME="<l3Network-name>"

+$ISOLATE_EMULATOR_THREAD="True"

+$VIRTIOINTERFACE="Modern"

+$NETWORKATTACHMENTNAME="mgmt0"

+# VM Size parameters

+$CPU_CORES=4

+$MEMORY_SIZE=12

+$VM_DISK_SIZE="64"

+# Virtual Machine Image parameters

+$VM_IMAGE="<VM image, example: myacr.azurecr.io/ubuntu:20.04>"

+$ACR_URL="<Azure container registry URL, example: myacr.azurecr.io>"

+$ACR_USERNAME="<Azure container registry username>"

+$NETWORKATTACHMENT = New-AzNetworkCloudNetworkAttachmentObject `

+-AttachedNetworkId $L3_NETWORK_ID `

+-IpAllocationMethod $IP_AllOCATION_METHOD `

+-DefaultGateway "True" `

+-Name $NETWORKATTACHMENTNAME

+$SECUREPASSWORD = ConvertTo-SecureString "<YourPassword>" -asplaintext -force

+```

+> [!IMPORTANT]

+> It is essential that you replace the placeholders for CUSTOM_LOCATION, CSN_ARM_ID, L3_NETWORK_ID and ACR parameters with your actual values before running these commands.

+After defining these variables, you can create the virtual machine by executing the following Azure PowerShell command.

+```azurepowershell-interactive

+New-AzNetworkCloudVirtualMachine -Name $VM_NAME `

+-ResourceGroupName $RESOURCE_GROUP `

+-AdminUsername $ADMIN_USERNAME `

+-CloudServiceNetworkAttachmentAttachedNetworkId $CSN_ARM_ID `

+-CloudServiceNetworkAttachmentIPAllocationMethod $IP_AllOCATION_METHOD `

+-CpuCore $CPU_CORES `

+-ExtendedLocationName $CUSTOM_LOCATION `

+-ExtendedLocationType $CUSTOM_LOCATION_TYPE `

+-Location $LOCATION `

+-SubscriptionId $SUBSCRIPTION `

+-MemorySizeGb $MEMORY_SIZE `

+-OSDiskSizeGb $VM_DISK_SIZE `

+-VMImage $VM_IMAGE `

+-BootMethod $BOOT_METHOD `

+-CloudServiceNetworkAttachmentDefaultGateway $CSN_ATTACHMENT_DEFAULTGATEWAY `

+-CloudServiceNetworkAttachmentName $CSN_ATTACHMENT_NAME `

+-IsolateEmulatorThread $ISOLATE_EMULATOR_THREAD `

+-NetworkAttachment $NETWORKATTACHMENT `

+-NetworkData $NETWORKDATA `

+-OSDiskCreateOption $OS_DISK_CREATE_OPTION `

+-OSDiskDeleteOption $OS_DISK_DELETE_OPTION `

+-SshPublicKey $SSH_PUBLIC_KEY `

+-UserData $USERDATA `

+-VMDeviceModel $VMDEVICEMODEL `

+-VMImageRepositoryCredentialsUsername $ACR_USERNAME `

+-VMImageRepositoryCredentialsPassword $SECUREPASSWORD `

+-VMImageRepositoryCredentialsRegistryUrl $ACR_URL

+```

+After a few minutes, the command completes and returns information about the virtual machine. You've created the virtual machine. You're now ready to use them.

+## Review deployed resources

+## Clean up resources

+## Next steps

+You've successfully created a Nexus virtual machine. You can now use the virtual machine to host virtual network functions (VNFs).

+Azure Orbital Ground Station interfaces with your Azure resources using virtual network (VNET) injection, which is used in both uplink and downlink directions. This page describes how to ensure your subnet and Azure Orbital Ground Station resources are configured correctly.

+In this how-to guide, you'll learn how to:

+> [!div class="checklist"]

+> * Prepare the subnet for VNET injection

+> * Prepare endpoints

+> * Verify the contact profile

+> * Find IPs of scheduled contacts

+Ensure the objects comply with the recommendations in this article. Note that these steps don't have to be followed in order.

+## Create and prepare subnet for VNET injection

+- An entire subnet with no existing IPs allocated or in use that can be dedicated to the Azure Orbital Ground Station service, in your virtual network within your resource group. If you need to make a new subnet, follow instructions to [add a subnet](../virtual-network/virtual-network-manage-subnet.md?tabs=azure-portal#add-a-subnet).

+Follow instructions to [add a subnet delegation](../virtual-network/manage-subnet-delegation.md#delegate-a-subnet-to-an-azure-service) in your virtual network. Delegate your subnet to the service named: **Microsoft.Orbital/orbitalGateways**.

+The following is an example of a typical VNET setup with a subnet delegated to Azure Orbital Ground Station:

+Set the MTU of all desired endpoints to at least **3650** by sending appropriate commands to your virtual machine within your resource group.

+## Verify the contact profile

+### Region

+The VNET/subnet must be in the same region as the contact profile.

+### Subnet ID

+1. Go to overview page of your contact profile and select **JSON view**. Find the **networkConfigurations** section, then identify the "**subnetId**".

+2. Go to overview page of your virtual network and select **JSON view**. Find the section for your **delegated subnet**, then identify the "**id**".

+3. Verify that these IDs are identical.

+### Link flows: IP Address and Port

+The links/channels must be set up in the following manner, based on direction and TCP or UDP preference.

+> [!NOTE]

+> These settings are for managed modems only.

+#### Uplink

+| Setting | TCP Client | TCP Server | UDP Client | UDP Server |

+|:|:|:-|:|:|

+| Contact Profile: Link/Channel **IP Address** | Blank | Routable IP from delegated subnet | Blank | Not applicable |

+| Contact Profile: Link/Channel **Port** | Unique port in 49152-65535 | Unique port in 49152-65535 | Unique port in 49152-65535 | Not applicable |

+| **Output** | | | | |

+| Contact Resource: **destinationIP** | Connect to this IP | Not applicable | Connect to this IP | Not applicable |

+| Contact Resource: **sourceIP** | Not applicable | Link comes from one of these IPs | Not applicable | Not applicable |

+#### Downlink

+| Setting | TCP Client | TCP Server | UDP Client | UDP Server |

+|:|:|:-|:|:-|

+| Contact Profile: Link/Channel **IP Address** | Blank | Routable IP from delegated subnet | Not applicable | Routable IP from delegated subnet |

+| Contact Profile: Link/Channel **Port** | Unique port in 49152-65535 | Unique port in 49152-65535 | Not applicable | Unique port in 49152-65535 |

+| **Output** | | | | |

+| Contact Resource: **destinationIP** | Connect to this IP | Not applicable | Not applicable | Not applicable |

+| Contact Resource: **sourceIP** | Not applicable | Link comes from one of these IPs | Not applicable | Link comes from one of these IPs |

+> You can have multiple links/channels in a contact profile, and you can have multiple IPs. However the combination of port/protocol must be unique. You can't have two identical ports, even if you have two different destination IPs.

+For more information, learn about [contact profiles](/azure/orbital/concepts-contact-profile) and [how to configure a contact profile](/azure/orbital/contact-profile).

+## Find IPs of a scheduled contact

+The Azure Orbital Ground Station platform prereserves IPs in the subnet when a contact is scheduled. These IPs represent the platform-side endpoints for each link. IPs are unique between contacts, and if multiple concurrent contacts are using the same subnet, Microsoft guarantees those IPs to be distinct. The service fails to schedule the contact and an error is returned if the service runs out of IPs or can't allocate an IP.

+When you create a contact, you can find these IPs by viewing the contact properties.

+To view the contact properties, go to the contact resource overview page and select **JSON view** in the portal or use the **GET contact** API call. Make sure to use the current API version of 2022-11-01. The parameters of interest are below:

+| **Parameter** | **Usage** |

+||--|

+| antennaConfiguration.destinationIp | Connect to this IP when you configure the link as **tcp/udp client**. |

+| antennaConfiguration.sourceIps | Data comes from this IP when you configure the link as **tcp/udp server**. |

+You can use this information to set up network policies or to distinguish between simultaneous contacts to the same endpoint.

+> [!NOTE]

+> - The source and destination IPs are always taken from the subnet address range.

+> - Only one destination IP is present. Any link in client mode should connect to this IP and the links are differentiated based on port.

+> - Many source IPs can be present. Links in server mode will connect to your specified IP address in the contact profile. The flows will originate from the source IPs present in this field and target the port as per the link details in the contact profile. There is no fixed assignment of link to source IP so please make sure to allow all IPs in any networking setup or firewalls.

+For more information, learn about [contacts](/azure/orbital/concepts-contact) and [how to schedule a contact](/azure/orbital/schedule-contact).

+This article describes how to use Azure services like Azure Functions, and how to install connectors to Azure resources for Confluent Cloud.

+Azure Native ISV Services enable you to easily provision, manage, and tightly integrate *independent software vendor* (ISV) software and services on Azure. Azure Native ISV Services is developed and managed by Microsoft and the ISV. Currently, several services are publicly available across these areas: observability, data, networking, and storage. For a list of all our current ISV partner services, see [Extend Azure with Azure Native ISV Services](partners.md).

+A list of features of any Azure Native ISV Service listed follows.

+- Logs and metrics: Seamlessly direct logs and metrics from Azure Monitor to the Azure Native ISV Service using just a few gestures. You can configure autodiscovery of resources to monitor, and set up automatic log forwarding and metrics shipping. You can easily do the setup in Azure, without needing to create more infrastructure or write custom code.

+- Virtual network injection: Provides private data plane access to Azure Native ISV services from customersΓÇÖ virtual networks.

+- Unified billing: Get a single bill for all resources that you consume on Azure for the Qumulo service.

+- Private access: The service is directly connected to your own virtual network, sometimes called *VNet injection*.

+ > [Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/qumulo1584033880660.qumulo-saas-mpp?tab=Overview)

+- **[Cost Savings](../flexible-server/how-to-deploy-on-azure-free-account.md)** ΓÇô Flexible server allows you to stop and start server on-demand to lower your TCO. Your compute tier billing is stopped immediately, which allows you to have significant cost savings during development, testing and for time-bound predictable production workloads.

+- **[Support for new PG versions](../flexible-server/concepts-supported-versions.md)** - Flexible server currently supports PG version 11 and onwards till version 14. Newer community versions of PostgreSQL will be supported only in flexible server.