+[az-aks-get-credentials]: /cli/azure/aks#az_aks_get_credentials

+For Arc-enabled VMware vSphere (preview), manual upgrade is available, and cloud-managed upgrade is supported for appliances on version 1.0.15 and higher. When Arc-enabled VMware vSphere announces General Availability, appliances on version 1.0.15 and higher will receive cloud-managed upgrade as the default experience. Appliances that are below version 1.0.15 must be manually upgraded. A manual upgrade only upgrades the appliance to the next version, not the latest version. If you have multiple versions to upgrade, then another option is to review the steps for [performing a recovery](/azure/azure-arc/vmware-vsphere/recover-from-resource-bridge-deletion), then delete the appliance VM and perform the recovery steps. This will deploy a new Arc resource bridge using the latest version and reconnect pre-existing Azure resources.

+ - On the Enterprise and Enterprise Flash tier, the Used Memory value includes the memory in both the primary and replica nodes. This can make the metric appear twice as large as expected.

+[Learn more about Dapr service invocation.](https://docs.dapr.io/developing-applications/building-blocks/bindings/)

+[Learn more about Dapr.](https://docs.dapr.io/)

+1. <a name="custom-props"></a>(Optional) In the **Advanced options** at **Details tab** the **Custom properties** section is added, if you've configured action groups for this alert rule. In this ection you can add your own properties to include in the alert notification payload. You can use these properties in the actions called by the action group, such as webhook, Azure function or logic app actions.

+2. [`ama-metrics-prometheus-config`](https://aka.ms/azureprometheus-addon-rs-configmap) (**Recommended**)

+3. [`ama-metrics-prometheus-config-node`](https://aka.ms/azureprometheus-addon-ds-configmap) (**Advanced**)

+4. [`ama-metrics-prometheus-config-node-windows`](https://aka.ms/azureprometheus-addon-ds-configmap-windows) (**Advanced**)

+- ama-metrics-prometheus-config (**Recommended**) - When a configmap with this name is created, scrape jobs defined in it are run from the Azure monitor metrics replica pod running in the cluster.

+- ama-metrics-prometheus-config-node (**Advanced**) - When a configmap with this name is created, scrape jobs defined in it are run from each **Linux** DaemonSet pod running in the cluster. For more information, see [Advanced Setup](#advanced-setup-configure-custom-prometheus-scrape-jobs-for-the-daemonset).

+- ama-metrics-prometheus-config-node-windows (**Advanced**) - When a configmap with this name is created, scrape jobs defined in it are run from each **windows** DaemonSet. For more information, see [Advanced Setup](#advanced-setup-configure-custom-prometheus-scrape-jobs-for-the-daemonset).

+> [!NOTE]

+> ### Advanced setup: Configure custom Prometheus scrape jobs for the DaemonSet

+>

+> The `ama-metrics` Replica pod consumes the custom Prometheus config and scrapes the specified targets. For a cluster with a large number of nodes and pods and a large volume of metrics to scrape, some of the applicable custom scrape targets can be off-loaded from the single `ama-metrics` Replica pod to the `ama-metrics` DaemonSet pod.

+>

+> The [ama-metrics-prometheus-config-node configmap](https://aka.ms/azureprometheus-addon-ds-configmap), is similar to the replica-set configmap, and can be created to have static scrape configs on each node. The scrape config should only target a single node and shouldn't use service discovery/pod annotations. Otherwise, each node tries to scrape all targets and makes many calls to the Kubernetes API server.

+>

+> Custom scrape targets can follow the same format by using `static_configs` with targets and using the `$NODE_IP` environment variable and specifying the port to scrape. Each pod of the DaemonSet takes the config, scrapes the metrics, and sends them for that node.

+>

+> Example:- The following `node-exporter` config is one of the default targets for the DaemonSet pods. It uses the `$NODE_IP` environment variable, which is already set for every `ama-metrics` add-on container to target a specific port on the node.

+ ```yaml

+ - job_name: nodesample

+ scrape_interval: 30s

+ scheme: http

+ metrics_path: /metrics

+ relabel_configs:

+ - source_labels: [__metrics_path__]

+ regex: (.*)

+ target_label: metrics_path

+ - source_labels: [__address__]

+ replacement: '$NODE_NAME'

+ target_label: instance

+ static_configs:

+ - targets: ['$NODE_IP:9100']

+ ```

+- Microsoft Entra application with access to the DCR

+## Limitations

+- If you're duplicating IP ranges either with VMs or Azure Virtual Machine Scale Sets across subnets and virtual networks, VM insights Map might display incorrect information. This issue is known. We're investigating options to improve this experience.

+- The Map feature currently only supports IPv4. We're investigating support for IPv6. We also support IPv4 that's tunnelled inside IPv6.

+- A map for a resource group or other large group might be difficult to view. Although we've made improvements to Map to handle large and complex configurations, we realize a map can have many nodes, connections, and nodes working as a cluster. We're committed to continuing to enhance support to increase scalability.

+- In the Free pricing tier, the VM insights Map feature supports only five machines that are connected to a Log Analytics workspace.

+> [!NOTE]

+> The network chart on the Performance tab looks different from the network chart on the Azure VM overview page because the overview page displays charts based on the host's measurement of activity in the guest VM. The network chart on the Azure VM overview only displays network traffic that will be billed. Inter-virtual network traffic isn't included. The data and charts shown for VM insights are based on data from the guest VM. The network chart displays all TCP/IP traffic that's inbound and outbound to that VM, including inter-virtual network traffic.

+ Title: Configure customer-managed keys (preview) for experiment encryption

+description: Learn how to configure customer-managed keys (preview) for your Azure Chaos Studio experiment resource using Azure Blob Storage

+

+# Configure customer-managed keys (preview) for Azure Chaos Studio using Azure Blob Storage

+

+Azure Chaos Studio automatically encrypts all data stored in your experiment resource with keys that Microsoft provides (service-managed keys). As an optional feature, you can add a second layer of security by also providing your own (customer-managed) encryption key(s). Customer-managed keys offer greater flexibility for controlling access and key-rotation policies.

+

+When you use customer-managed encryption keys, you need to specify a user-assigned managed identity (UMI) to retrieve the key. The UMI you create needs to match the UMI that you use for the Chaos Studio experiment.

+

+When configured, Azure Chaos Studio uses Azure Storage, which uses the customer-managed key to encrypt all of your experiment execution and result data within your own Storage account.

+## Prerequisites

+

+- An Azure account with an active subscription. If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.

+

+- An existing user-assigned managed identity. For more information about creating a user-assigned managed identity, see [Manage user-assigned managed identities](../active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities.md?pivots=identity-mi-methods-azp#create-a-user-assigned-managed-identity).

+- A public-access enabled Azure storage account

+

+## Limitations

+

+- Azure Chaos Studio experiments can't automatically rotate the customer-managed key to use the latest version of the encryption key. You would do key rotation directly in your chosen Azure Storage account.

+- You will need to use our **2023-10-27-preview REST API** to create and use CMK-enabled experiments ONLY. There is **no** support for CMK-enabled experiments in our GA-stable REST API until H1 2024.

+- Azure Chaos Studio currently **only supports creating Chaos Studio Customer-Managed-Key experiments via the Command Line using our 2023-10-27-preview REST API**. As a result, you **cannot** create a Chaos Studio experiment with CMK enabled via the Azure portal. We plan to add this functionality in H1 of 2024.

+- The storage account must have **public access from all networks** enabled for Azure Chaos Studio experiments to be able to use it. If you have a hard requirement from your organization, reach out to your CSA for potential solutions.

+## Configure your Azure storage account

+

+When creating and/or updating your storage account to use for a CMK experiment, you need to navigate to the encryption tab and set the Encryption type to Customer-managed keys (CMK) and fill out all required information.

+> [!NOTE]

+> The User-assigned managed identity that you use should match the one you use for the corresponding Chaos Studio CMK-enabled experiment.

+

+## Use customer-managed keys with Azure Chaos Studio

+

+You can only configure customer-managed encryption keys when you create a new Azure Chaos Studio experiment resource. When you specify the encryption key details, you also have to select a user-assigned managed identity to retrieve the key from Azure Key Vault.

+> [!NOTE]

+> The UMI should be the SAME user-assigned managed identity you use with your Chaos Studio experiment resource, otherwise the Chaos Studio CMK experiment fails to Create/Update.

+

+# [Azure CLI](#tab/azure-cli)

+

+The following code sample shows an example PUT command for creating or updating a Chaos Studio experiment resource to enable customer-managed keys:

+> [!NOTE]

+>The two parameters specific to CMK experiments are under the "CustomerDataStorage" block, in which we ask for the Subscription ID of the Azure Blob Storage Account you want to use to storage your experiment data and the name of the Blob Storage container to use or create.

+

+```HTTP

+PUT https://management.azure.com/subscriptions/<yourSubscriptionID>/resourceGroups/exampleRG/providers/Microsoft.Chaos/experiments/exampleExperiment?api-version=2023-10-27-preview

+{

+ "location": "eastus2euap",

+ "identity": {

+ "type": "SystemAssigned"

+ },

+ "properties": {

+ "steps": [

+ {

+ "name": "step1",

+ "branches": [

+ {

+ "name": "branch1",

+ "actions": [

+ {

+ "type": "continuous",

+ "name": "urn:csci:microsoft:virtualMachine:shutdown/1.0",

+ "selectorId": "selector1",

+ "duration": "PT10M",

+ "parameters": [

+ {

+ "key": "abruptShutdown",

+ "value": "false"

+ }

+ ]

+ }

+ ]

+ }

+ ]

+ }

+ ],

+ "selectors": [

+ {

+ "type": "List",

+ "id": "selector1",

+ "targets": [

+ {

+ "type": "ChaosTarget",

+ "id": "/subscriptions/6b052e15-03d3-4f17-b2e1-be7f07588291/resourceGroups/exampleRG/providers/Microsoft.Compute/virtualMachines/exampleVM/providers/Microsoft.Chaos/targets/Microsoft-VirtualMachine"

+ }

+ ]

+ }

+ ],

+ "customerDataStorage": {

+ "storageAccountResourceId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/exampleRG/providers/Microsoft.Storage/storageAccounts/exampleStorage",

+ "blobContainerName": "azurechaosstudioexperiments"

+ }

+ }

+}

+```

+## Disable CMK on a Chaos Studio experiment

+

+If you run the same PUT command from the previous example on an existing CMK-enabled experiment resource, but leave the fields in "customerDataStorage" empty, CMK is disabled on an experiment.

+## Re-enable CMK on a Chaos Studio experiment

+

+If you run the same PUT command from the previous example on an existing experiment resource using the 2023-10-27-preview REST API and populate the fields in "customerDataStorage", CMK is re-enabled on an experiment.

+## Change the user-assigned managed identity for retrieving the encryption key

+

+You can change the managed identity for customer-managed keys for an existing Chaos Studio experiment at any time. The outcome would be identical to updating the User-assigned Managed identity for any Chaos Studio experiment.

+> [!NOTE]

+>If the User-Assigned Managed Identity does NOT have the correct permissions to retrieve the CMK from your key vault and write to the Blob Storage, the PUT command to update the UMI fails.

+### List whether an experiment is CMK-enabled or not

+

+Using the "Get Experiment" command from the 2023-10-27-preview REST API, the response shows you whether the "CustomerDataStorage" properties have been populated or not, which is how you can tell whether an experiment has CMK enabled or not.

+

+## Update the customer-managed encryption key being used by your Azure Storage Account

+

+You can change the key that you're using at any time, since Azure Chaos Studio is using your own Azure Storage account for encryption using your CMK.

+

+## Frequently asked questions

+

+### Is there an extra charge to enable customer-managed keys?

+

+While there's no charge associated directly from Azure Chaos Studio, the use of Azure Blob Storage and Azure Key Vault could carry some additional cost subject to those services' individual pricing.

+

+### Are customer-managed keys supported for existing Azure Chaos Studio experiments?

+

+This feature is currently only available for Azure Chaos Studio experiments created using our **2023-10-27-preview** REST API.

+| jsonSpec | A JSON-formatted Chaos Mesh spec that uses the [NetworkChaos kind](https://chaos-mesh.org/docs/simulate-network-chaos-on-kubernetes/#create-experiments-using-the-yaml-files). You can use a YAML-to-JSON converter like [Convert YAML To JSON](https://www.convertjson.com/yaml-to-json.htm) to convert the Chaos Mesh YAML to JSON and minify it. Use single-quotes within the JSON or escape the quotes with a backslash character. Only include the YAML under the `jsonSpec` property. Don't include information like metadata and kind. Specifying duration within the `jsonSpec` isn't necessary, but will be used if available. |

+ "value": "{\"action\":\"delay\",\"mode\":\"one\",\"selector\":{\"namespaces\":[\"default\"]},\"delay\":{\"latency\":\"200ms\",\"correlation\":\"100\",\"jitter\":\"0ms\"}}}"

+| jsonSpec | A JSON-formatted Chaos Mesh spec that uses the [PodChaos kind](https://chaos-mesh.org/docs/simulate-pod-chaos-on-kubernetes/#create-experiments-using-yaml-configuration-files). You can use a YAML-to-JSON converter like [Convert YAML To JSON](https://www.convertjson.com/yaml-to-json.htm) to convert the Chaos Mesh YAML to JSON and minify it. Use single-quotes within the JSON or escape the quotes with a backslash character. Only include the YAML under the `jsonSpec` property. Don't include information like metadata and kind. Specifying duration within the `jsonSpec` isn't necessary, but will be used if available. |

+ "value": "{\"action\":\"pod-failure\",\"mode\":\"all\",\"selector\":{\"namespaces\":[\"default\"]}}"

+| jsonSpec | A JSON-formatted Chaos Mesh spec that uses the [StressChaos kind](https://chaos-mesh.org/docs/simulate-heavy-stress-on-kubernetes/#create-experiments-using-the-yaml-file). You can use a YAML-to-JSON converter like [Convert YAML To JSON](https://www.convertjson.com/yaml-to-json.htm) to convert the Chaos Mesh YAML to JSON and minify it. Use single-quotes within the JSON or escape the quotes with a backslash character. Only include the YAML under the `jsonSpec` property. Don't include information like metadata and kind. Specifying duration within the `jsonSpec` isn't necessary, but will be used if available. |

+ "value": "{\"mode\":\"one\",\"selector\":{\"namespaces\":[\"default\"]},\"stressors\":{\"cpu\":{\"workers\":1,\"load\":50},\"memory\":{\"workers\":4,\"size\":\"256MB\"}}"

+| jsonSpec | A JSON-formatted Chaos Mesh spec that uses the [IOChaos kind](https://chaos-mesh.org/docs/simulate-io-chaos-on-kubernetes/#create-experiments-using-the-yaml-files). You can use a YAML-to-JSON converter like [Convert YAML To JSON](https://www.convertjson.com/yaml-to-json.htm) to convert the Chaos Mesh YAML to JSON and minify it. Use single-quotes within the JSON or escape the quotes with a backslash character. Only include the YAML under the `jsonSpec` property. Don't include information like metadata and kind. Specifying duration within the `jsonSpec` isn't necessary, but will be used if available. |

+ "value": "{\"action\":\"latency\",\"mode\":\"one\",\"selector\":{\"app\":\"etcd\"},\"volumePath\":\"\/var\/run\/etcd\",\"path\":\"\/var\/run\/etcd\/**\/*\",\"delay\":\"100ms\",\"percent\":50}"

+| jsonSpec | A JSON-formatted Chaos Mesh spec that uses the [TimeChaos kind](https://chaos-mesh.org/docs/simulate-time-chaos-on-kubernetes/#create-experiments-using-the-yaml-file). You can use a YAML-to-JSON converter like [Convert YAML To JSON](https://www.convertjson.com/yaml-to-json.htm) to convert the Chaos Mesh YAML to JSON and minify it. Use single-quotes within the JSON or escape the quotes with a backslash character. Only include the YAML under the `jsonSpec` property. Don't include information like metadata and kind. Specifying duration within the `jsonSpec` isn't necessary, but will be used if available. |

+ "value": "{\"mode\":\"one\",\"selector\":{\"namespaces\":[\"default\"]},\"timeOffset\":\"-10m100ns\"}"

+| jsonSpec | A JSON-formatted Chaos Mesh spec that uses the [KernelChaos kind](https://chaos-mesh.org/docs/simulate-kernel-chaos-on-kubernetes/#configuration-file). You can use a YAML-to-JSON converter like [Convert YAML To JSON](https://www.convertjson.com/yaml-to-json.htm) to convert the Chaos Mesh YAML to JSON and minify it. Use single-quotes within the JSON or escape the quotes with a backslash character. Only include the YAML under the `jsonSpec` property. Don't include information like metadata and kind. Specifying duration within the `jsonSpec` isn't necessary, but will be used if available. |

+ "value": "{\"mode\":\"one\",\"selector\":{\"namespaces\":[\"default\"]},\"failKernRequest\":{\"callchain\":[{\"funcname\":\"__x64_sys_mount\"}],\"failtype\":0}}"

+| jsonSpec | A JSON-formatted Chaos Mesh spec that uses the [HTTPChaos kind](https://chaos-mesh.org/docs/simulate-http-chaos-on-kubernetes/#create-experiments). You can use a YAML-to-JSON converter like [Convert YAML To JSON](https://www.convertjson.com/yaml-to-json.htm) to convert the Chaos Mesh YAML to JSON and minify it. Use single-quotes within the JSON or escape the quotes with a backslash character. Only include the YAML under the `jsonSpec` property. Don't include information like metadata and kind. Specifying duration within the `jsonSpec` isn't necessary, but will be used if available. |

+ "value": "{\"mode\":\"all\",\"selector\":{\"namespaces\":[\"default\"]},\"target\":\"Request\",\"port\":80,\"method\":\"GET\",\"path\":\"/api\",\"abort\":true}"

+| jsonSpec | A JSON-formatted Chaos Mesh spec that uses the [DNSChaos kind](https://chaos-mesh.org/docs/simulate-dns-chaos-on-kubernetes/#create-experiments-using-the-yaml-file). You can use a YAML-to-JSON converter like [Convert YAML To JSON](https://www.convertjson.com/yaml-to-json.htm) to convert the Chaos Mesh YAML to JSON and minify it. Use single-quotes within the JSON or escape the quotes with a backslash character. Only include the YAML under the `jsonSpec` property. Don't include information like metadata and kind. Specifying duration within the `jsonSpec` isn't necessary, but will be used if available. |

+ "value": "{\"action\":\"random\",\"mode\":\"all\",\"patterns\":[\"google.com\",\"chaos-mesh.*\",\"github.?om\"],\"selector\":{\"namespaces\":[\"default\"]}}"

+| version | Certificate version that should be disabled. If not specified, the latest version is disabled. |

+- **Terraform** - Chaos Studio doesn't support Terraform at this time.

+- **PowerShell modules** - Chaos Studio doesn't have dedicated PowerShell modules at this time. For PowerShell, use our REST API

+- **Azure CLI** - Chaos Studio doesn't have dedicated AzCLI modules at this time. Use our REST API from AzCLI

+- **Azure Policy** - Chaos Studio doesn't support the applicable built-in policies for our service (audit policy for customer-managed keys and Private Link) at this time.

+- **Private Link** To use Private Link for Agent Service, you need to have your subscription allowlisted and use our preview API version. We don't support Azure portal UI experiments for Agent-based experiments using Private Link. These restrictions do NOT apply to our Service-direct faults

+- **Customer-Managed Keys** You need to use our 2023-10-27-preview REST API via a CLI to create CMK-enabled experiments. We don't support portal UI experiments using CMK at this time.

+- **Lockbox** At present, we don't have integration with Customer Lockbox.

+- **Java SDK** At present, we don't have a dedicated Java SDK. If this is something you would use, reach out to us with your feature request.

+- **Built-in roles** - Chaos Studio doesn't currently have its own built-in roles. Permissions can be attained to run a chaos experiment by either assigning an [Azure built-in role](chaos-studio-fault-providers.md) or a created custom role to the experiment's identity.

+- **Agent Service Tags** Currently we don't have service tags available for our Agent-based faults.

+1. Confirm that the desired resource is listed. Select **Review + Enable**, then **Enable**.

+1. A notification appears and indicates that the resource selected was successfully enabled.

+ 

+1. Select **Create** > **New experiment**.

+1. In the Chaos Studio experiment designer, give a friendly name to your **Step** and **Branch**. Select **Add action > Add fault**.

+Navigate to the Chaos Studio overview page and click on the **Targets** blade under the "Experiments Management" section. Find the target platform. If it is already enabled as a target for agent-based experiments, you will need to disable it as a target and then "enable for agent-based targets" to bring up the Chaos Studio agent target configuration pane.

+See screenshot below for an example:

+At this point, the Agent target configuration page seen in the screenshot should come up . After configuring your managed identity, make sure Application Insights is "Enabled" and then select your desired Application Insights Account and enter the Instrumentation Key you copied in Step 1. Once you have filled out the required information, you can click "Review+Create" to deploy your resource.

+

+ If you receive a PowerShell parsing error, switch to a Bash terminal as recommended for this tutorial or surround the referenced JSON file in single quotes (`--body '@target.json'`).

+1. Create the capabilities by replacing `$RESOURCE_ID` with the resource ID of the target VM or virtual machine scale set. Replace `$CAPABILITY` with the [name of the fault capability you're enabling](chaos-studio-fault-library.md) (for example, `CPUPressure-1.0`).

+1. Select the **Experiments** tab in Chaos Studio. In this view, you can see and manage all your chaos experiments. Select **Create** > **New experiment**.

+1. You're now in the Chaos Studio experiment designer. You can build your experiment by adding steps, branches, and faults. Give a friendly name to your **Step** and **Branch**. Then select **Add action > Add fault**.

+ 1. Use a [JSON string escape tool like this one](https://www.freeformatter.com/json-escape.html) to escape the JSON spec, or change the double-quotes to single-quotes.

+ ```json

+ {'action':'pod-failure','mode':'all','selector':{'namespaces':['default']}}

+ ```

+1. Confirm that the desired resource is listed. Select **Review + Enable**, then **Enable**.

+1. Select the **Experiments** tab in Chaos Studio. In this view, you can see and manage all your chaos experiments. Select **Create** > **New experiment**.

+1. You're now in the Chaos Studio experiment designer. The experiment designer allows you to build your experiment by adding steps, branches, and faults. Give a friendly name to your **Step** and **Branch** and select **Add action > Add fault**.

+1. In Chaos Studio, go to **Experiments** > **Create** > **New experiment**.

+1. Confirm that the desired resource is listed. Select **Review + Enable**, then **Enable**.

+1. Select the **Experiments** tab in Chaos Studio. In this view, you can see and manage all your chaos experiments. Select **Create** > **New experiment**.

+1. You're now in the Chaos Studio experiment designer. The experiment designer allows you to build your experiment by adding steps, branches, and faults. Give a friendly name to your **Step** and **Branch** and select **Add action > Add fault**.

+- [Outbound call to a phone number](../../../quickstarts/telephony/pstn-call.md)

+ Title: Theming the UI Library

+# Theming the UI Library

+Azure Communication Services UI Library is a set of components, icons and composites designed to make it easier for you to build high-quality user interfaces for your projects. The UI Library uses components and icons from [Fluent UI](https://developer.microsoft.com/fluentui), the cross-platform design system that's used by Microsoft. As a result, the components are built with usability, accessibility, and localization in mind.

+The UI Library is fully documented for developers on a separate site. Our documentation is interactive and designed to make it easy to understand how the APIs work by giving you the ability to try them out directly from a web page. See the [UI Library documentation](https://azure.github.io/communication-ui-library/?path=/docs/overview--page) for more information.

+3. Now if you navigate back to the "Events" option in left panel of your Azure Communication Services resource, you should be able to see the new event subscription with the Advanced Messaging events.

+ --object-id <key-vault-key-id>

+ Title: Save on select Azure SQL Services in Poland Central for a limited time

+description: Learn how to save up to 66 percent on select Azure SQL Services in Poland Central for a limited time.

+# Save on select Azure SQL Services in Poland Central for a limited time

+Save up to 66 percent compared to pay-as-you-go pricing when you purchase one or three-year reserved capacity for select [Azure SQL Database](/azure/azure-sql/database/reserved-capacity-overview), [SQL Managed Instances](/azure/azure-sql/database/reserved-capacity-overview), and [Azure Database for MySQL](../../mysql/single-server/concept-reserved-pricing.md) in Poland Central for a limited time. This offer is available between November 1, 2023 – March 31, 2024.

+## Purchase the limited time offer

+To take advantage of this limited-time offer, [purchase](https://aka.ms/reservations) a one or three-year term for select Azure SQL Databases, SQL Managed Instances, and Azure Database for MySQL in the Poland Central region.

+### Buy a reservation

+1. Sign in to the [Azure portal](https://portal.azure.com/).

+1. Select **All services** > **Reservations**.

+1. Select **Add** and then select a qualified product listed in the [Terms and conditions of the limited time offer](#terms-and-conditions-of-the-limited-time-offer) section.

+1. Select the [scope](prepare-buy-reservation.md#reservation-scoping-options), and then a billing subscription that you want to use for the reservation. You can change the reservation scope after purchase.

+1. Set the **Region** to **Poland Central**.

+1. Select a reservation term and billing frequency.

+1. Select **Add to cart**.

+1. In the cart, you can change the quantity. After you review your cart and you're ready to purchase, select **Next: Review + buy**.

+1. Select **Buy now**.

+You can view the reservation in the [Reservations](https://portal.azure.com/#blade/Microsoft_Azure_Billing/SubscriptionsBlade/Reservations) page in the Azure portal.

+## Charge back limited time offer costs

+Enterprise Agreement and Microsoft Customer Agreement billing readers can view amortized cost data for reservations. They can use the cost data to charge back the monetary value for a subscription, resource group, resource, or a tag to their partners. In amortized data, the effective price is the prorated hourly reservation cost. The cost is the total cost of reservation usage by the resource on that day. Users with an individual subscription can get the amortized cost data from their usage file. For more information, see [Charge back Azure Reservation costs](charge-back-usage.md).

+## Terms and conditions of the limited time offer

+These terms and conditions (hereinafter referred to as "terms") govern the limited time offer ("offer") provided by Microsoft to customers purchasing a one or three year reserved capacity for Azure SQL Databases, SQL Managed Instances, and Azure Database for MySQL in Poland Central between November 1, 2023 (12 AM Pacific Standard Time) ΓÇô March 31, 2024 (11:59 PM Pacific Standard Time), for any of the following

+- Azure Database for MySQL Single Server General Purpose - Compute Gen5

+- Azure Database for MySQL Single Server Memory Optimized - Compute Gen5

+- SQL Database Single/Elastic Pool Business Critical - Compute DC-Series

+- SQL Database Single/Elastic Pool Business Critical - Compute Gen5

+- SQL Database Single/Elastic Pool Business Critical - Compute M Series

+- SQL Database Single/Elastic Pool General Purpose - Compute DC-Series

+- SQL Database Single/Elastic Pool General Purpose - Compute FSv2 Series

+- SQL Database Single/Elastic Pool General Purpose - Compute Gen5

+- SQL Database SingleDB/Elastic Pool Hyperscale - Compute Gen5

+- SQL Managed Instance Business Critical - Compute Gen5

+- SQL Managed Instance General Purpose - Compute Gen5

+The 66 percent saving is based on one Azure Database for MySQL Single Server Memory Optimized - Compute Gen5 in the Poland Central region running for 36 months at a pay-as-you-go rate; reduced rate for a three-year reserved capacity. Actual savings might vary based on location, term commitment, instance type, or usage. The savings doesn't include operating system costs. For more information about pricing, see [Poland Central SQL Services reservation savings](/legal/cost-management-billing/reservations/poland-central-limited-time-sql-services).

+**Eligibility** - The Offer is open to individuals who meet the following criteria:

+- To buy a reservation, you must have the owner role or reservation purchaser role on an Azure subscription that's of one of the following types:

+ - Enterprise (MS-AZR-0017P or MS-AZR-0148P)

+ - Pay-As-You-Go (MS-AZR-0003P or MS-AZR-0023P)

+ - Microsoft Customer Agreement

+- Cloud solution providers can use the Azure portal or [Partner Center](/partner-center/azure-reservations) to purchase Azure Reservations. You can't purchase a reservation if you have a custom role that mimics the owner role or reservation purchaser role on an Azure subscription. You must use the built-in owner or built-in reservation purchaser role.

+- For more information about who can purchase a reservation, see [Buy an Azure reservation](prepare-buy-reservation.md).

+**Offer details** - For Azure SQL Database or SQL Managed Instance, you make a commitment for SQL Database or SQL Managed Instance use for one or three years to get a significant discount on the compute costs. To purchase reserved capacity, you need to specify the Azure region, deployment type, performance tier, and term.

+You don't need to assign the reservation to a specific database or managed instance. Matching existing deployments that are already running or ones that are newly deployed automatically get the benefit. Hence, by purchasing a reserved capacity, existing resources infrastructure wouldn't be modified and thus no failover/downtime is triggered on existing resources. By purchasing a reservation, you commit to usage for the compute costs for one or three years. As soon as you buy a reservation, the compute charges that match the reservation attributes are no longer charged at the pay-as-you go rates.

+For more information, see [Save compute costs with reserved capacity - Azure SQL Database & SQL Managed Instance](/azure/azure-sql/database/reserved-capacity-overview).

+For Azure Database for MySQL, you make an upfront commitment on MySQL server for a one or three year period to get a significant discount on the compute costs. To purchase Azure Database for MySQL reserved capacity, you need to specify the Azure region, deployment type, performance tier, and term.

+You don't need to assign the reservation to specific Azure Database for MySQL servers. An already running Azure Database for MySQL or ones that are newly deployed, automatically get the benefit of reserved pricing. By purchasing a reservation, you're prepaying for the compute costs for one or three years. As soon as you buy a reservation, the Azure database for MySQL compute charges that match the reservation attributes are no longer charged at the pay-as-you go rates. A reservation doesn't cover software, networking, or storage charges associated with the MySQL Database server. At the end of the reservation term, the billing benefit expires. Azure Database for MySQL usage is then billed at the pay-as-you go price.

+For more information, seeΓÇ»[Prepay for compute with reserved capacity - Azure Database for MySQL](/azure/mysql/single-server/concept-reserved-pricing).

+- Additional taxes might apply.

+- Payment will be processed using the payment method on file for the selected subscriptions.

+- Estimated savings are calculated based on your current on-demand rate.

+**Qualifying purchase** - To be eligible for the 66% discount, customers must make a purchase of the one or three year reserved capacity for Azure SQL Databases, SQL Managed Instances, and Azure Database for MySQL in Poland Central for one of the following qualified services in Poland Central between November 1, 2023, and March 31, 2024.

+- Azure Database for MySQL Single Server General Purpose - Compute Gen5

+- Azure Database for MySQL Single Server Memory Optimized - Compute Gen5

+- SQL Database Single/Elastic Pool Business Critical - Compute DC-Series

+- SQL Database Single/Elastic Pool Business Critical - Compute Gen5

+- SQL Database Single/Elastic Pool Business Critical - Compute M Series

+- SQL Database Single/Elastic Pool General Purpose - Compute DC-Series

+- SQL Database Single/Elastic Pool General Purpose - Compute FSv2 Series

+- SQL Database Single/Elastic Pool General Purpose - Compute Gen5

+- SQL Database SingleDB/Elastic Pool Hyperscale - Compute Gen5

+- SQL Managed Instance Business Critical - Compute Gen5

+- SQL Managed Instance General Purpose - Compute Gen5

+**Discount limitations**

+- A reservation discount is "use-it-or-lose-it." So, if you don't have matching resources for any hour, then you lose a reservation quantity for that hour. You can't carry forward unused reserved hours.

+- When you shut down a resource, the reservation discount automatically applies to another matching resource in the specified scope. If no matching resources are found in the specified scope, then the reserved hours are lost.

+- Stopped resources are billed and continue to use reservation hours. Deallocate or delete resources or scale-in other resources to use your available reservation hours with other workloads.

+- For more information about how reservation discounts are applied, see [How a reservation discount is applied](reservation-discount-application.md).

+**Exchanges and refunds** - The offer follows standard exchange and refund policies for reservations. For more information about exchanges and refunds, see [Self-service exchanges and refunds for Azure Reservations](exchange-and-refund-azure-reservations.md).

+**Renewals**

+- The renewal price **will not be** the limited time offer price, but the price available at time of renewal.

+- For more information about renewals, see [Automatically renew Azure reservations](reservation-renew.md).

+**Termination or modification** - Microsoft reserves the right to modify, suspend, or terminate the offer at any time without prior notice.

+If you have purchased the one or three year reserved capacity for Azure SQL Databases, SQL Managed Instances, and Azure Database for MySQL qualified services in Poland Central between November 1, 2023, and March 31, 2024 you'll continue to get the discount throughout the purchased term length, even if the offer is canceled.

+By participating in the offer, customers agree to be bound by these terms and the decisions of Microsoft. Microsoft reserves the right to disqualify any customer who violates these terms or engages in any fraudulent or harmful activities related to the offer.

+## Next steps

+- [Understand Azure reservation discount](reservation-discount-application.md)

+- [Purchase reserved capacity in the Azure portal](https://portal.azure.com/#view/Microsoft_Azure_Reservations/ReservationsBrowseBlade)

+>

+> - This price includes free vulnerability assessments for 20 unique images per charged resource, whereby the count will be based on the previous month's consumption. Every subsequent scan will be charged at $0.29 per image digest. The majority of customers are not expected to incur any additional image scan charges. For subscriptions that are both under the Defender CSPM and Defender for Containers plans, free vulnerability assessment will be calculated based on free image scans provided via the Defender for Containers plan, as specified [in the Microsoft Defender for Cloud pricing page](https://azure.microsoft.com/pricing/details/defender-for-cloud/).

+- Watch video: [Predict future security incidents! Cloud Security Posture Management with Microsoft Defender](https://www.youtube.com/watch?v=jF3NSR_OepI)

+- Learn about Defender for Cloud's [Defender plans](defender-for-cloud-introduction.md#protect-cloud-workloads).

+ > [!NOTE]

+ > If you select **Management account** to create a connector to a management account, then the tab to onboard with Terraform is not visible in the UI, but you can still onboard using Terraform, similar to what's covered at [Onboarding your AWS/GCP environment to Microsoft Defender for Cloud with Terraform - Microsoft Community Hub](https://techcommunity.microsoft.com/t5/microsoft-defender-for-cloud/onboarding-your-aws-gcp-environment-to-microsoft-defender-for/ba-p/3798664).

+1. In your VM options, change your boot options from **Firmware** to **BIOS**. Make sure that you're not booting from EFI.

+|`[-ts <TIMES>]` `[--time_span <TIMES>]` | Defines the time span for which the rule is active, using the following syntax: `hh:mm-hh:mm, hh:mm-hh:mm` |

+|`[-ts <TIMES>]` `[--time_span <TIMES>]` | Defines the time span for which the rule is active, using the following syntax: `hh:mm-hh:mm, hh:mm-hh:mm` |

+|`[-ts <TIMES>]` `[--time_span <TIMES>]` | Defines the time span for which the rule is active, using the following syntax: `hh:mm-hh:mm, hh:mm-hh:mm` |

+ Title: Write event messages into Azure Data Lake Storage Gen2 with Apache Flink® DataStream API

+description: Learn how to write event messages into Azure Data Lake Storage Gen2 with Apache Flink® DataStream API

+# Write event messages into Azure Data Lake Storage Gen2 with Apache Flink® DataStream API

+* [Apache Flink cluster on HDInsight on AKS ](../flink/flink-create-cluster-portal.md)

+* [Apache Kafka cluster on HDInsight](../../hdinsight/kafk)

+ * You're required to ensure the network settings are taken care as described on [Using Apache Kafka on HDInsight](../flink/process-and-consume-data.md); that's to make sure HDInsight on AKS and HDInsight clusters are in the same Virtual Network

+> Replace [Apache Kafka on HDInsight cluster](../../hdinsight/kafk) bootStrapServers with your own brokers for Kafka 2.4 or 3.2

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Kafka, Kafka, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Incorporate Apache Flink® DataStream into Azure Databricks Delta Lake Table

+description: Learn about incorporate Apache Flink® DataStream into Azure Databricks Delta Lake Table

+# Incorporate Apache Flink® DataStream into Azure Databricks Delta Lake Tables

+This example shows how to sink stream data in Azure ADLS Gen2 from Apache Flink cluster on HDInsight on AKS into Delta Lake tables using Azure Databricks Auto Loader.

+- [Apache Flink 1.16.0 on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+- [Apache Kafka 3.2 on HDInsight](../../hdinsight/kafk)

+### Create Apache Kafka® table on Apache Flink® SQL

+### Reference

+* Apache, Apache Kafka, Kafka, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Process real-time IoT data on Apache Flink® with Azure HDInsight on AKS

+description: How to integrate Azure IoT Hub and Apache Flink®

+# Process real-time IoT data on Apache Flink® with Azure HDInsight on AKS

+2. [Create Flink cluster on HDInsight on AKS](./flink-create-cluster-portal.md)

+### Reference

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Kafka, Kafka, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: How to perform Change Data Capture of SQL Server with Apache Flink® DataStream API and DataStream Source.

+description: Learn how to perform Change Data Capture of SQL Server with Apache Flink® DataStream API and DataStream Source.

+# Change Data Capture of SQL Server with Apache Flink® DataStream API and DataStream Source on HDInsight on AKS

+* [Apache Flink cluster on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+* [Apache Kafka cluster on HDInsight](../../hdinsight/kafk)

+ * You're required to ensure the network settings are taken care as described on [Using HDInsight Kafka](../flink/process-and-consume-data.md); that's to make sure HDInsight on AKS and HDInsight clusters are in the same VNet

+* [Kafka Connector](https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kafka/#behind-the-scene)

+* Apache, Apache Kafka, Kafka, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Using Azure Cosmos DB for Apache Cassandra® with HDInsight on AKS for Apache Flink®

+description: Learn how to Sink Apache Kafka® message into Azure Cosmos DB for Apache Cassandra®, with Apache Flink® running on HDInsight on AKS.

+# Sink Apache Kafka® messages into Azure Cosmos DB for Apache Cassandra, with Apache Flink® on HDInsight on AKS

+This example uses [Apache Flink](../flink/flink-overview.md) to sink [HDInsight for Apache Kafka](/azure/hdinsight/kafka/apache-kafka-introduction) messages into [Azure Cosmos DB for Apache Cassandra](/azure/cosmos-db/cassandra/introduction)

+This example is prominent when Engineers prefer real-time aggregated data for analysis. With access to historical aggregated data, you can build machine learning (ML) models to build insights or actions. You can also ingest IoT data into Apache Flink to aggregate data in real-time and store it in Apache Cassandra.

+* [Apache Flink 1.16.0 on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+* [Apache Kafka 3.2 on HDInsight](../../hdinsight/kafk)

+* An Ubuntu VM for maven project development environment in the same VNet as HDInsight on AKS cluster.

+> * Replace Kafka Broker IPs with your Kafka cluster broker IPs

+## Sink Kafka Topics into Cosmos DB for Apache Cassandra

+Check job on Flink Web UI on HDInsight on AKS Cluster

+* Apache, Apache Kafka, Kafka, Apache Flink, Flink, Apache Cassandra, Cassandra and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: How to create Apache Kafka table on an Apache Flink® on HDInsight on AKS

+description: Learn how to create Apache Kafka table on Apache Flink®

+# Create Apache Kafka® table on Apache Flink® on HDInsight on AKS

+* [Apache Kafka cluster on HDInsight](../../hdinsight/kafk)

+* [Apache Flink cluster on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+## Create a Kafka table on Flink SQL

+We're using the **Kafka 3.2.0** dependencies in the below step, You're required to update the command based on your Kafka version on HDInsight cluster.

+* Apache, Apache Kafka, Kafka, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Use DataStream API for MongoDB as a source and sink with Apache Flink®

+description: Learn how to use Apache Flink® DataStream API on HDInsight on AKS for MongoDB as a source and sink

+# Use Apache Flink® DataStream API on HDInsight on AKS for MongoDB as a source and sink

+* [Flink cluster 1.16.0 on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+* We use the [MongoDB Connector](https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/mongodb/)

+### Reference

+- Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Microsoft Fabric with Apache Flink® in HDInsight on AKS

+description: An introduction to lakehouse on Microsoft Fabric with Apache Flink® on HDInsight on AKS

+# Connect to OneLake in Microsoft Fabric with HDInsight on AKS cluster for Apache Flink®

+This example demonstrates on how to use HDInsight on AKS cluster for Apache Flink® with [Microsoft Fabric](/fabric/get-started/microsoft-fabric-overview).

+* [Flink cluster on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+Here, we use the packaged jar and submit to Flink cluster in HDInsight on AKS

+* Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Table API and SQL - Use Delta Catalog type with Hive with Apache Flink® on Azure HDInsight on AKS

+description: Learn about how to create Delta Catalog with Apache Flink® on Azure HDInsight on AKS

+# Create Delta Catalog with Apache Flink® on Azure HDInsight on AKS

+### Reference

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Table API and SQL - Use Iceberg Catalog type with Hive in Apache Flink® on HDInsight on AKS

+description: Learn how to create Iceberg Catalog in Apache Flink® on HDInsight on AKS

+# Create Iceberg Catalog in Apache Flink® on HDInsight on AKS

+[Apache Iceberg](https://iceberg.apache.org/) is an open table format for huge analytic datasets. Iceberg adds tables to compute engines like Apache Flink, using a high-performance table format that works just like a SQL table. Apache Iceberg [supports](https://iceberg.apache.org/multi-engine-support/#apache-flink) both Apache FlinkΓÇÖs DataStream API and Table API.

+In this article, we learn how to use Iceberg Table managed in Hive catalog, with Apache Flink on HDInsight on AKS cluster

+### Reference

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Hive, Hive, Apache Iceberg, Iceberg, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Troubleshoot Apache Flink® on HDInsight on AKS

+description: Learn to troubleshoot Apache Flink® cluster configurations on HDInsight on AKS

+# Troubleshoot Apache Flink® cluster configurations on HDInsight on AKS

+1. Contact [support team](../hdinsight-aks-support-help.md) with this information.

+### Reference

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Apache Flink® Configuration Management in HDInsight on AKS

+# Apache Flink® Configuration management in HDInsight on AKS

+HDInsight on AKS provides a set of default configurations of Apache Flink for most properties and a few based on common application profiles. However, in case you're required to tweak Flink configuration properties to improve performance for certain applications with state usage, parallelism, or memory settings, you can change certain properties at cluster level using **Configuration management** section in HDInsight on AKS cluster.

+By default Apache Flink clusters in HDInsight on AKS use Rocks db

+- [Apache Flink Configurations](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/)

+- Apache, Apache Kafka, Kafka, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Create an Apache Flink® cluster in HDInsight on AKS using Azure portal

+description: Creating an Apache Flink cluster in HDInsight on AKS with Azure portal.

+# Create an Apache Flink® cluster in HDInsight on AKS with Azure portal

+Complete the following steps to create an Apache Flink cluster on Azure portal.

+> [!NOTE]

+> Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: How to connect Apache Flink® on HDInsight on AKS with Azure Event Hubs for Apache Kafka®

+description: Learn how to connect Apache Flink® on HDInsight on AKS with Azure Event Hubs for Apache Kafka®

+# Connect Apache Flink® on HDInsight on AKS with Azure Event Hubs for Apache Kafka®

+In this article, we explore how to connect [Azure Event Hubs](/azure/event-hubs/event-hubs-about) with [Apache Flink on HDInsight on AKS](./flink-overview.md) and cover the following

+### Reference

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Kafka, Kafka, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Apache Flink® job management in HDInsight on AKS

+# Apache Flink® job management in HDInsight on AKS clusters

+HDInsight on AKS provides a feature to manage and submit Apache Flink® jobs directly through the Azure portal (user-friendly interface) and ARM Rest APIs.

+This feature empowers users to efficiently control and monitor their Apache Flink jobs without requiring deep cluster-level knowledge.

+HDInsight on AKS supports user friendly ARM Rest APIs to submit job and manage job. Using this Flink REST API, you can seamlessly integrate Flink job operations into your Azure Pipeline. Whether you're launching new jobs, updating running jobs, or performing various job operations, this streamlined approach eliminates manual steps and empowers you to manage your Flink cluster efficiently.

+ | jobType | Type of Job. It should be ΓÇ£FlinkJobΓÇ¥ | | Yes|

+### Reference

+- [Apache Flink Job Scheduling](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/internals/job_scheduling/)

+- Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Azure Data Factory Managed Airflow with Apache Flink® on HDInsight on AKS

+description: Learn how to perform Apache Flink® job orchestration using Azure Data Factory Managed Airflow

+# Apache Flink® job orchestration using Azure Data Factory Managed Airflow

+This article covers managing a Flink job using [Azure REST API](flink-job-management.md#arm-rest-api) and orchestration data pipeline with Azure Data Factory Managed Airflow. [Azure Data Factory Managed Airflow](/azure/data-factory/concept-managed-airflow) service is a simple and efficient way to create and manage [Apache Airflow](https://airflow.apache.org/) environments, enabling you to run data pipelines at scale easily.

+Apache Airflow is an open-source platform that programmatically creates, schedules, and monitors complex data workflows. It allows you to define a set of tasks, called operators that can be combined into directed acyclic graphs (DAGs) to represent data pipelines.

+1. Upload your Flink Job jar to the storage account. It can be the primary storage account associated with the Flink cluster or any other storage account, where you should assign the "Storage Blob Data Owner" role to the user-assigned MSI used for the cluster in this storage account.

+1. Create [Microsoft Entra service principal](/cli/azure/ad/sp/) to access Key Vault – Grant permission to access Azure Key Vault with the “Key Vault Secrets Officer” role, and make a note of ‘appId’ ‘password’, and ‘tenant’ from the response. We need to use the same for Airflow to use Key Vault storage as backends for storing sensitive information.

+1. Create Managed Airflow enable with [Azure Key Vault](/azure/data-factory/enable-azure-key-vault-for-managed-airflow) to store and manage your sensitive information in a secure and centralized manner. By doing this, you can use variables and connections, and they automatically be stored in Azure Key Vault. The name of connections and variables need to be prefixed by variables_prefix  defined in AIRFLOW__SECRETS__BACKEND_KWARGS. For example, If variables_prefix has a value as  hdinsight-aks-variables then for a variable key of hello, you would want to store your Variable at hdinsight-aks-variable -hello.

+- get `OAuth Token`

+### Reference

+- Refer to the [sample code](https://github.com/Azure-Samples/hdinsight-aks/blob/main/flink/airflow-python-sample-code).

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Airflow, Airflow, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: What is Apache Flink® in Azure HDInsight on AKS? (Preview)

+description: An introduction to Apache Flink® in Azure HDInsight on AKS.

+# What is Apache Flink® in Azure HDInsight on AKS? (Preview)

+Apache Flink clusters in HDInsight on AKS are a fully managed service. Benefits of creating a Flink cluster in HDInsight on AKS are listed here.

+| Feature | Description |

+| | |

+| Ease creation |You can create a new Flink cluster in HDInsight in minutes using the Azure portal, Azure PowerShell, or the SDK. See [Get started with Apache Flink cluster in HDInsight on AKS](flink-create-cluster-portal.md). |

+| Ease of use | Flink clusters in HDInsight on AKS include portal based configuration management, and scaling. In addition to this with job management API, you use the REST API or Azure portal for job management.|

+| REST APIs | Flink clusters in HDInsight on AKS include [Job management API](flink-job-management.md), a REST API-based Flink job submission method to remotely submit and monitor jobs on Azure portal.|

+| Deployment Type | Flink can execute applications in Session mode or Application mode. Currently HDInsight on AKS supports only Session clusters. You can run multiple Flink jobs on a Session cluster. App mode is on the roadmap for HDInsight on AKS clusters|

+| Support for Metastore | Flink clusters in HDInsight on AKS can support catalogs with [Hive Metastore](hive-dialect-flink.md) in different open file formats with remote checkpoints to Azure Data Lake Storage Gen2.|

+| Support for Azure Storage | Flink clusters in HDInsight can use Azure Data Lake Storage Gen2 as File sink. For more information on Data Lake Storage Gen2, see [Azure Data Lake Storage Gen2](../../storage/blobs/data-lake-storage-introduction.md).|

+| Integration with Azure services | Flink cluster in HDInsight on AKS comes with an integration to Kafka along with [Azure Event Hubs](flink-how-to-setup-event-hub.md) and [Azure HDInsight](process-and-consume-data.md). You can build streaming applications using the Event Hubs or HDInsight. |

+| Adaptability | HDInsight on AKS allows you to scale the Flink cluster nodes based on schedule with the Autoscale feature. See [Automatically scale Azure HDInsight on AKS clusters](../hdinsight-on-aks-autoscale-clusters.md). |

+| State Backend | HDInsight on AKS uses the [RocksDB](http://rocksdb.org) as default StateBackend. RocksDB is an embeddable persistent key-value store for fast storage.|

+| Checkpoints | Checkpointing is enabled in HDInsight on AKS clusters by default. Default settings on HDInsight on AKS maintain the last five checkpoints in persistent storage. In case, your job fails, the job can be restarted from the latest checkpoint.|

+| Incremental Checkpoints | RocksDB supports Incremental Checkpoints. We encourage the use of incremental checkpoints for large state, you need to enable this feature manually. Setting a default in your `flink-conf.yaml: state.backend.incremental: true` enables incremental checkpoints, unless the application overrides this setting in the code. This statement is true by default. You can alternatively configure this value directly in the code (overrides the config default) ``EmbeddedRocksDBStateBackend` backend = new `EmbeddedRocksDBStateBackend(true);`` . By default, we preserve the last five checkpoints in the checkpoint dir configured. This value can be changed by changing the configuration on configuration management section `state.checkpoints.num-retained: 5`|

+Apache Flink clusters in HDInsight include the following components, they are available on the clusters by default.

+* [DataStreamAPI](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/overview/#what-is-a-datastream)

+* [TableAPI & SQL](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/overview/#table-api--sql).

+Refer to the [Roadmap](../whats-new.md#coming-soon) on what's coming soon!

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Kafka, Kafka, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Table API and SQL in Apache Flink® clusters on HDInsight on AKS

+description: Learn about Table API and SQL in Apache Flink® clusters on HDInsight on AKS

+# Table API and SQL in Apache Flink® clusters on HDInsight on AKS

+## Using Flink SQL Client in HDInsight on AKS clusters

+## Adding Dependencies

+## Hive Metastore in Apache Flink® clusters on HDInsight on AKS

+> HDInsight on AKS clusters comes with an integrated option of Hive Metastore for Apache Flink. You can opt for Hive Metastore during [cluster creation](../flink/flink-create-cluster-portal.md)

+ > HDInsight on AKS supports **Hive 3.1.2** and **Hadoop 3.3.2**. The `hive-conf-dir` is set to location `/opt/hive-conf`

+> There is a known limitation in Apache Flink. The last ΓÇÿnΓÇÖ columns are chosen for partitions, irrespective of user defined partition column. [FLINK-32596](https://issues.apache.org/jira/browse/FLINK-32596) The partition key will be wrong when use Flink dialect to create Hive table.

+### Reference

+- [Apache Flink Table API & SQL](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/overview/#table-api--sql)

+- Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: How to enter the Apache Flink® CLI client using Secure Shell (SSH) on HDInsight on AKS clusters with Azure portal

+description: How to enter Apache Flink® SQL & DStream CLI client using webssh on HDInsight on AKS clusters with Azure portal

+# Access Apache Flink® CLI client using Secure Shell (SSH) on HDInsight on AKS clusters with Azure portal

+This example guides how to enter the Apache Flink CLI client on HDInsight on AKS clusters using SSH on Azure portal, we cover both SQL and Flink DataStream

+* Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Fraud detection with the Apache Flink® DataStream API

+description: Learn about Fraud detection with the Apache Flink® DataStream API

+# Fraud detection with the Apache Flink® DataStream API

+* [Flink cluster 1.16.0 on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+* Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Hive dialect in Apache Flink® clusters on HDInsight on AKS

+description: how to use Hive dialect in Apache Flink® clusters on HDInsight on AKS

+# Hive dialect in Apache Flink® clusters on HDInsight on AKS

+In this article, learn how to use Hive dialect in Apache Flink clusters on HDInsight on AKS.

+The user cannot change the default `flink` dialect to hive dialect for their usage on HDInsight on AKS clusters. All the SQL operations fail once changed to hive dialect with the following error.

+### Reference

+- [Hive Dialect in Apache Flink](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/hive/hive_dialect/#hive-dialect)

+- Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Integration of Azure Data Explorer and Apache Flink®

+description: Integration of Azure Data Explorer and Apache Flink® in HDInsight on AKS

+# Integration of Azure Data Explorer and Apache Flink®

+ADX helps users in analysis of large volumes of data from streaming applications, websites, IoT devices, etc. Integrating Apache Flink with ADX helps you to process real-time data and analyze it in ADX.

+- [Create Apache Flink cluster on HDInsight on AKS](./flink-create-cluster-portal.md)

+1. [Create Flink cluster](./flink-create-cluster-portal.md).

+### Reference

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Enrich the events from Apache Kafka® with the attributes from FileSystem with Apache Flink®

+description: Learn how to join stream from Kafka with table from fileSystem using Apache Flink® DataStream API

+# Enrich the events from Apache Kafka® with attributes from ADLS Gen2 with Apache Flink®

+* [Flink cluster on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+* [Kafka cluster on HDInsight](../../hdinsight/kafk)

+ * You're required to ensure the network settings are taken care as described on [Using Kafka on HDInsight](../flink/process-and-consume-data.md); that's to make sure HDInsight on AKS and HDInsight clusters are in the same VNet

+- [Flink Examples](https://github.com/flink-extended/)

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Kafka, Kafka, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Change Data Capture (CDC) of PostgreSQL table using Apache Flink®

+description: Learn how to perform CDC on PostgreSQL table using Apache Flink®

+# Change Data Capture (CDC) of PostgreSQL table using Apache Flink®

+* [Apache Flink Cluster on HDInsight on AKS](./flink-create-cluster-portal.md)

+- [Apache Flink Website](https://flink.apache.org/)

+- [PostgreSQL CDC Connector](https://ververica.github.io/flink-cdc-connectors/release-2.1/content/connectors/postgres-cdc.html) is licensed under [Apache 2.0 License](https://github.com/ververica/flink-cdc-connectors/blob/master/LICENSE)

+- Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Using Apache Kafka® on HDInsight with Apache Flink® on HDInsight on AKS

+description: Learn how to use Apache Kafka® on HDInsight with Apache Flink® on HDInsight on AKS

+# Using Apache Kafka® on HDInsight with Apache Flink® on HDInsight on AKS

+This example uses HDInsight on AKS clusters running Flink 1.16.0 to process streaming data consuming and producing Kafka topic.

+## Apache Kafka Connector

+Kafka sink provides a builder class to construct an instance of a KafkaSink. We use the same to construct our Sink and use it along with Flink cluster running on HDInsight on AKS

+* Apache, Apache Kafka, Kafka, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Use Elasticsearch along with Apache Flink® on HDInsight on AKS

+description: Learn how to use Elasticsearch along Apache Flink® on HDInsight on AKS

+# Using Elasticsearch with Apache Flink® on HDInsight on AKS

+Apache Flink for real-time analytics can be used to build a dashboard application that visualizes the streaming data using Elasticsearch and Kibana.

+In this article, learn how to Use Elastic along Apache Flink® on HDInsight on AKS.

+* [Create Flink 1.16.0 cluster](./flink-create-cluster-portal.md)

+* Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Change Data Capture (CDC) of SQL Server using Apache Flink®

+description: Learn how to perform CDC of SQL Server using Apache Flink®

+# Change Data Capture (CDC) of SQL Server using Apache Flink®

+Apache Flink supports to interpret Debezium JSON and Avro messages as INSERT/UPDATE/DELETE messages into Flink SQL system.

+ * [Flink cluster on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+### SQLServer CDC Connector

+The SQLServer CDC connector is a Flink Source connector, which reads database snapshot first and then continues to read change events with exactly once processing even failures happen. This example uses Flink CDC to create a SQLServerCDC table on FLINK SQL

+### Use SSH to use Flink SQL client

+* Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Use Apache NiFi with HDInsight on AKS clusters running Apache Flink® to publish into ADLS Gen2

+description: Learn how to use Apache NiFi to consume processed Apache Kafka® topic from Apache Flink® on HDInsight on AKS clusters and publish into ADLS Gen2

+# Use Apache NiFi to consume processed Apache Kafka® topics from Apache Flink® and publish into ADLS Gen2

+* [Flink cluster on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+* [Kafka cluster on HDInsight](../../hdinsight/kafk)

+ * You're required to ensure the network settings are taken care as described on [Using Kafka on HDInsight](../flink/process-and-consume-data.md); that's to make sure HDInsight on AKS and HDInsight clusters are in the same VNet

+> Setup a HDInsight cluster with [Apache Kafka](../../hdinsight/kafk) and replace broker list with your own list before you get started for both Kafka 2.4 and 3.2.

+**Kafka 2.4.1**

+**Kafka 3.2.0**

+## Process streaming data from Kafka cluster on HDInsight with Flink cluster on HDInsight on AKS

+You can replace 2.4.1 with 3.2.0 in case you're using Kafka 3.2.0 on HDInsight, where applicable on the pom.xml

+## Submit streaming job to Flink cluster on HDInsight on AKS

+Now, lets submit streaming job as mentioned in the previous step into Flink cluster

+## Check the topic on Kafka cluster

+Check the topic on Kafka.

+* Apache, Apache Kafka, Kafka, Apache Flink, Flink,Apache NiFi, NiFi and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: How to use Azure Pipelines with Apache Flink® on HDInsight on AKS

+description: Learn how to use Azure Pipelines with Apache Flink®

+# How to use Azure Pipelines with Apache Flink® on HDInsight on AKS

+In this article, you'll learn how to use Azure Pipelines with HDInsight on AKS to submit Flink jobs with the cluster's REST API. We guide you through the process using a sample YAML pipeline and a PowerShell script, both of which streamline the automation of the REST API interactions.

+### Reference

+- [Apache Flink Website](https://flink.apache.org/)

+> [!NOTE]

+> Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: How to use Apache Flink® CLI to submit jobs

+description: Learn how to use Apache Flink® CLI to submit jobs

+# Apache Flink® Command-Line Interface (CLI) on HDInsight on AKS clusters

+This command installs Flink CLI in the user's home directory (`$HOME/flink-cli`). The script can also be downloaded and run locally. You might have to restart your shell in order for changes to take effect.

+| savepoint | This action can be used to *create or disposing savepoints* for a given job. It might be necessary to specify a savepoint directory besides the JobID. |

+### Reference

+- [Apache Flink Website](https://flink.apache.org/)

+- Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: How to use Apache Flink® on HDInsight on AKS with Flink/Delta connector

+description: Learn how to use Flink/Delta Connector

+# How to use Flink/Delta Connector

+## What is Flink/Delta connector

+Flink/Delta Connector is a JVM library to read and write data from Apache Flink applications to Delta tables utilizing the Delta Standalone JVM library. The connector provides exactly once delivery guarantee.

+We are using the following connector, to match with the Apache Flink version running on HDInsight on AKS cluster.

+* [Create Flink 1.16.0 cluster](./flink-create-cluster-portal.md)

+* Apache, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Write messages to Apache HBase® with Apache Flink® DataStream API

+description: Learn how to write messages to Apache HBase with Apache Flink DataStream API

+# Write messages to Apache HBase® with Apache Flink® DataStream API

+* [Apache Flink cluster on HDInsight on AKS](../flink/flink-create-cluster-portal.md)

+* [Apache Kafka cluster on HDInsight](../flink/process-and-consume-data.md)

+* [Apache HBase 2.4.11 clusteron HDInsight](../../hdinsight/hbase/apache-hbase-tutorial-get-started-linux.md#create-apache-hbase-cluster)

+ * You're required to ensure HDInsight on AKS cluster can connect to HDInsight cluster, with same virtual network.

+**Use pipeline to produce Apache Kafka topic**

+**Create HBase table on HDInsight cluster**

+* Apache, Apache Kafka, Kafka, Apache HBase, HBase, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Use Hive Catalog, Hive Read & Write demo on Apache Flink®

+description: Learn how to use Hive Catalog, Hive Read & Write demo on Apache Flink® on HDInsight on AKS

+# How to use Hive Catalog with Apache Flink® on HDInsight on AKS

+This example uses HiveΓÇÖs Metastore as a persistent catalog with Apache FlinkΓÇÖs Hive Catalog. We use this functionality for storing Kafka table and MySQL table metadata on Flink across sessions. Flink uses Kafka table registered in Hive Catalog as a source, perform some lookup and sink result to MySQL database

+* [Apache Flink Cluster on HDInsight on AKS with Hive Metastore 3.1.2](../flink/flink-create-cluster-portal.md)

+* [Apache Kafka cluster on HDInsight](../../hdinsight/kafk)

+ * You're required to ensure the network settings are complete as described on [Using Kafka](../flink/process-and-consume-data.md); that's to make sure HDInsight on AKS and HDInsight clusters are in the same VNet

+## Apache Hive on Apache Flink

+For more information, see [Apache Hive](https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/)

+Move the jar flink-table-planner_2.12-1.16.0-0.0.18.jar located in webssh pod's /opt to /lib and move out the jar flink-table-planner-loader-1.16.0-0.0.18.jar from /lib. Refer to [issue](https://issues.apache.org/jira/browse/FLINK-25128) for more details. Perform the following steps to move the planner jar.

+* Apache, Apache Hive, Hive, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Use Hive Metastore with Apache Flink® DataStream API

+description: Use Hive Metastore with Apache Flink® DataStream API

+# Use Hive Metastore with Apache Flink® DataStream API

+## Supported Hive versions for Apache Flink clusters on HDInsight on AKS

+- [Hive read & write](https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/hive_read_write/)

+- Apache, Apache Hive, Hive, Apache Flink, Flink, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ * Apache SparkΓäó

+ * Apache Flink®

+## Reference

+* Apache, Apache Spark, Spark, and associated open source project names are [trademarks](./trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: How to use Delta Lake in Azure HDInsight on AKS with Apache SparkΓäó cluster.

+description: Learn how to use Delta Lake scenario in Azure HDInsight on AKS with Apache SparkΓäó cluster.

+# Use Delta Lake in Azure HDInsight on AKS with Apache SparkΓäó cluster (Preview)

+[Azure HDInsight on AKS](../overview.md) is a managed cloud-based service for big data analytics that helps organizations process large amounts data. This tutorial shows how to use Delta Lake in Azure HDInsight on AKS with Apache SparkΓäó cluster.

+1. Create an [Apache SparkΓäó cluster in Azure HDInsight on AKS](./create-spark-cluster.md)

+ :::image type="content" source="./media/azure-hdinsight-spark-on-aks-delta-lake/create-spark-cluster.png" alt-text="Screenshot showing spark cluster creation." lightbox="./media/azure-hdinsight-spark-on-aks-delta-lake/create-spark-cluster.png":::

+Delta Lake with Apache Spark Compatibility matrix - [Delta Lake](https://docs.delta.io/latest/releases.html), change Delta Lake version based on Apache Spark Version.

+## Reference

+* Apache, Apache Spark, Spark, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: Configuration management in HDInsight on AKS with Apache SparkΓäó

+description: Learn how to perform Configuration management in HDInsight on AKS with Apache SparkΓäó cluster

+# Configuration management in HDInsight on AKS with Apache SparkΓäó cluster

+Azure HDInsight on AKS is a managed cloud-based service for big data analytics that helps organizations process large amounts data. This tutorial shows how to use configuration management in Azure HDInsight on AKS with Apache SparkΓäó cluster.

+Configuration management is used to add specific configurations into the Apache Spark cluster.

+## Reference

+* Apache, Apache Spark, Spark, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+

+1. Create an HDInsight on AKS cluster with Apache SparkΓäó. Follow these instructions: Set up clusters in HDInsight on AKS.

+1. Create a new Notebook and select type as **pyspark**.

+## Reference

+* Apache, Apache Spark, Spark, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: What is Apache SparkΓäó in HDInsight on AKS? (Preview)

+description: An introduction to Apache SparkΓäó in HDInsight on AKS

+# What is Apache SparkΓäó in HDInsight on AKS? (Preview)

+Apache SparkΓäó is a parallel processing framework that supports in-memory processing to boost the performance of big-data analytic applications.

+Apache SparkΓäó provides primitives for in-memory cluster computing. A Spark job can load and cache data into memory and query it repeatedly. In-memory computing is faster than disk-based applications, such as Hadoop, which shares data through Hadoop distributed file system (HDFS). Apache Spark allows integration with the Scala and Python programming languages to let you manipulate distributed data sets like local collections. There's no need to structure everything as map and reduce operations.

+## Apache Spark cluster with HDInsight on AKS

+Apache SparkΓäó in Azure HDInsight on AKS is the managed spark service in Microsoft Azure. With Apache Spark in Azure HDInsight on AKS, you can store and process your data all within Azure. Spark clusters in HDInsight are compatible with or [Azure Data Lake Storage Gen2](../../storage/blobs/data-lake-storage-introduction.md), allows you to apply Spark processing on your existing data stores.

+Apache Spark on AKS in HDInsight composed of multiple components as pods.

+## Apache Spark service components

+**Client Interfaces:** Apache Spark clusters in HDInsight on AKS, provides various client interfaces. Livy Server, Jupyter Notebook, Spark History Server, provides Spark services to HDInsight on AKS users.

+## Reference

+* Apache, Apache Spark, Spark, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: How to submit and manage jobs on an Apache SparkΓäó cluster in Azure HDInsight on AKS

+description: Learn how to submit and manage jobs on an Apache SparkΓäó cluster in HDInsight on AKS

+# Submit and manage jobs on an Apache SparkΓäó cluster in HDInsight on AKS

+An Apache SparkΓäó cluster on HDInsight on AKS. For more information, seeΓÇ»[Create an Apache Spark cluster](./create-spark-cluster.md).

+1. Navigate to the Apache SparkΓäó cluster page and open the **Overview** tab. Click on Jupyter, it asks you to authenticate and open the Jupyter web page.

+Apache Spark clusters in HDInsight on AKS includeΓÇ»[Apache Zeppelin notebooks](https://zeppelin.apache.org/). Use the notebooks to run Apache Spark jobs. In this article, you learn how to use the Zeppelin notebook on an HDInsight on AKS cluster.

+1. Navigate to the Apache Spark cluster Overview page and select Zeppelin notebook from Cluster dashboards. It prompts to authenticate and open the Zeppelin page.

+## Monitor queries on an Apache Spark cluster in HDInsight on AKS

+## Reference

+* Apache, Apache Spark, Spark, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+ Title: How to use Hive metastore in Apache SparkΓäó

+description: Learn how to use Hive metastore in Apache SparkΓäó

+# How to use Hive metastore with Apache SparkΓäó cluster

+1. Configure Metastore while you create a HDInsight on AKS cluster with Apache SparkΓäó

+1. The rest of the details are to be filled in as per the cluster creation rules for [Apache Spark cluster in HDInsight on AKS](./create-spark-cluster.md).

+

+## Reference

+* Apache, Apache Spark, Spark, and associated open source project names are [trademarks](../trademarks.md) of the [Apache Software Foundation](https://www.apache.org/) (ASF).

+description: Learn how to register a client application for the DICOM service in Microsoft Entra ID.

+In this article, you learn how to register a client application for the DICOM&reg; service. You can find more information on [Register an application with the Microsoft identity platform](../../active-directory/develop/quickstart-register-app.md).

+1. Select **App registrations**.

+ [ ](media/register-application-one.png#lightbox)

+1. Select **New registration**.

+1. For **Supported account types**, select **Accounts in this organizational directory only**. Leave the other options as is.

+ [ ](media/register-application-two.png#lightbox)

+1. Select **Register**.

+After you register a new application, you can find the **Application (client) ID** and **Directory (tenant) ID** from the **Overview** menu option. Make a note of the values for use later.

+[ ](media/register-application-three.png#lightbox)

+## Authentication setting: Confidential vs. public

+Select **Authentication** to review the settings. The default value for **Allow public client flows** is **No**.

+[ ](media/register-application-five.png#lightbox)

+If you change the default value to **Yes** for the **Allow public client flows** option in the **Advanced** setting, the application registration is a **public client application** and a certificate or secret isn't required. The **Yes** value is useful when you want to use the client application in your mobile app or a JavaScript app where you don't want to store any secrets.

+> [!NOTE]

+> For Postman, select **Mobile and desktop applications**. Enter `https://www.getpostman.com/oauth2/callback` in the **Custom redirect URIs** section. Select **Configure** to save the setting.

+[ ](media/register-application-five-bravo.png#lightbox)

+Select **Certificates & secrets** and select **New client secret**.

+[ ](media/register-application-six.png#lightbox)

+The following steps are required for the DICOM service. In addition, user access permissions or role assignments for Azure Health Data Services are managed through role-based access control (RBAC). For more information, see [Configure Azure RBAC for Azure Health Data Services](./../configure-azure-rbac.md).

+1. Select the **API permissions** pane.

+ [ ](./media/dicom-add-apis-permissions.png#lightbox)

+1. Select **Add a permission**.

+ Add a permission to the DICOM service by searching for **Azure API for DICOM** under **APIs my organization uses**.

+ [ ](./media/dicom-search-apis-permissions.png#lightbox)

+ The search result for Azure API for DICOM only returns if you've already deployed the DICOM service in the workspace.

+ If you're referencing a different resource application, select your DICOM API resource application registration that you created previously under **APIs my organization uses**.

+1. Select scopes (permissions) that the confidential client application asks for on behalf of a user. Select **Dicom.ReadWrite**, and then select **Add permissions**.

+ [ ](./media/dicom-select-scopes-new.png#lightbox)

+Your application registration is now finished.

+description: This article demonstrates how to use Azure Data Factory and Microsoft Fabric to perform analytics on DICOM data.

+# Get started using DICOM data in analytics workloads

+This article describes how to get started by using DICOM&reg; data in analytics workloads with Azure Data Factory and Microsoft Fabric.

+Before you get started, ensure that you've done the following steps:

+* Deploy an instance of the [DICOM service](deploy-dicom-services-in-azure.md).

+* Create a [storage account with Azure Data Lake Storage Gen2 capabilities](../../storage/blobs/create-data-lake-storage-account.md) by enabling a hierarchical namespace:

+ * Create a container to store DICOM metadata, for example, named `dicom`.

+* Create a [Data Factory](../../data-factory/quickstart-create-data-factory.md) instance:

+ * Enable a [system-assigned managed identity](../../data-factory/data-factory-service-identity.md).

+* Create a [lakehouse](/fabric/data-engineering/tutorial-build-lakehouse) in Fabric.

+* Add role assignments to the Data Factory system-assigned managed identity for the DICOM service and the Data Lake Storage Gen2 storage account:

+ * Add the **Storage Blob Data Contributor** role to grant permission to the Data Lake Storage Gen2 account.

+## Configure a Data Factory pipeline for the DICOM service

+In this example, a Data Factory [pipeline](../../data-factory/concepts-pipelines-activities.md) is used to write DICOM attributes for instances, series, and studies into a storage account in a [Delta table](https://delta.io/) format.

+From the Azure portal, open the Data Factory instance and select **Launch studio** to begin.

+Data Factory pipelines read from _data sources_ and write to _data sinks_, which are typically other Azure services. These connections to other services are managed as _linked services_.

+The pipeline in this example reads data from a DICOM service and writes its output to a storage account, so a linked service must be created for both.

+#### Create a linked service for the DICOM service

+1. In Azure Data Factory Studio, select **Manage** from the menu on the left. Under **Connections**, select **Linked services** and then select **New**.

+ :::image type="content" source="media/data-factory-linked-services.png" alt-text="Screenshot that shows the Linked services screen in Data Factory." lightbox="media/data-factory-linked-services.png":::

+1. On the **New linked service** pane, search for **REST**. Select the **REST** tile and then select **Continue**.

+ :::image type="content" source="media/data-factory-rest.png" alt-text="Screenshot that shows the New linked service pane with the REST tile selected." lightbox="media/data-factory-rest.png":::

+1. Enter a **Name** and **Description** for the linked service.

+ :::image type="content" source="media/data-factory-linked-service-dicom.png" alt-text="Screenshot that shows the New linked service pane with DICOM service details." lightbox="media/data-factory-linked-service-dicom.png":::

+1. In the **Base URL** field, enter the service URL for your DICOM service. For example, a DICOM service named `contosoclinic` in the `contosohealth` workspace has the service URL `https://contosohealth-contosoclinic.dicom.azurehealthcareapis.com`.

+1. For **Authentication type**, select **System Assigned Managed Identity**.

+1. For **AAD resource**, enter `https://dicom.healthcareapis.azure.com`. This URL is the same for all DICOM service instances.

+1. After you fill in the required fields, select **Test connection** to ensure the identity's roles are correctly configured.

+1. When the connection test is successful, select **Create**.

+#### Create a linked service for Azure Data Lake Storage Gen2

+1. In Data Factory Studio, select **Manage** from the menu on the left. Under **Connections**, select **Linked services** and then select **New**.

+1. On the **New linked service** pane, search for **Azure Data Lake Storage Gen2**. Select the **Azure Data Lake Storage Gen2** tile and then select **Continue**.

+ :::image type="content" source="media/data-factory-adls.png" alt-text="Screenshot that shows the New linked service pane with the Azure Data Lake Storage Gen2 tile selected." lightbox="media/data-factory-adls.png":::

+1. Enter a **Name** and **Description** for the linked service.

+ :::image type="content" source="media/data-factory-linked-service-adls.png" alt-text="Screenshot that shows the New linked service pane with Data Lake Storage Gen2 details." lightbox="media/data-factory-linked-service-adls.png":::

+1. For **Authentication type**, select **System Assigned Managed Identity**.

+1. Enter the storage account details by entering the URL to the storage account manually. Or you can select the Azure subscription and storage account from dropdowns.

+1. After you fill in the required fields, select **Test connection** to ensure the identity's roles are correctly configured.

+1. When the connection test is successful, select **Create**.

+Data Factory pipelines are a collection of _activities_ that perform a task, like copying DICOM metadata to Delta tables. This section details the creation of a pipeline that regularly synchronizes DICOM data to Delta tables as data is added to, updated in, and deleted from a DICOM service.

+1. Select **Author** from the menu on the left. In the **Factory Resources** pane, select the plus sign (+) to add a new resource. Select **Pipeline** and then select **Template gallery** from the menu.

+ :::image type="content" source="media/data-factory-create-pipeline-menu.png" alt-text="Screenshot that shows Template gallery selected under Pipeline." lightbox="media/data-factory-create-pipeline-menu.png":::

+1. In the **Template gallery**, search for **DICOM**. Select the **Copy DICOM Metadata Changes to ADLS Gen2 in Delta Format** tile and then select **Continue**.

+ :::image type="content" source="media/data-factory-gallery-dicom.png" alt-text="Screenshot that shows the DICOM template selected in the Template gallery." lightbox="media/data-factory-gallery-dicom.png":::

+1. In the **Inputs** section, select the linked services previously created for the DICOM service and Data Lake Storage Gen2 account.

+ :::image type="content" source="media/data-factory-create-pipeline.png" alt-text="Screenshot that shows the Inputs section with linked services selected." lightbox="media/data-factory-create-pipeline.png":::

+1. Select **Use this template** to create the new pipeline.

+## Schedule a pipeline

+Pipelines are scheduled by _triggers_. There are different types of triggers. _Schedule triggers_ allow pipelines to be triggered on a wall-clock schedule. _Manual triggers_ trigger pipelines on demand.

+In this example, a _tumbling window trigger_ is used to periodically run the pipeline given a starting point and regular time interval. For more information about triggers, see [Pipeline execution and triggers in Azure Data Factory or Azure Synapse Analytics](../../data-factory/concepts-pipeline-execution-triggers.md).

+1. Select **Author** from the menu on the left. Select the pipeline for the DICOM service and select **Add trigger** and **New/Edit** from the menu bar.

+ :::image type="content" source="media/data-factory-add-trigger.png" alt-text="Screenshot that shows the pipeline view of Data Factory Studio with the Add trigger button on the menu bar selected." lightbox="media/data-factory-add-trigger.png":::

+1. On the **Add triggers** pane, select the **Choose trigger** dropdown and then select **New**.

+1. Enter a **Name** and **Description** for the trigger.

+ :::image type="content" source="media/data-factory-new-trigger.png" alt-text="Screenshot that shows the New trigger pane with the Name, Description, Type, Date, and Recurrence fields." lightbox="media/data-factory-new-trigger.png":::

+1. Select **Tumbling window** as the **Type**.

+1. To configure a pipeline that runs hourly, set the **Recurrence** to **1 Hour**.

+1. Expand the **Advanced** section and enter a **Delay** of **15 minutes**. This setting allows any pending operations at the end of an hour to complete before processing.

+1. Set **Max concurrency** to **1** to ensure consistency across tables.

+1. Select **OK** to continue configuring the trigger run parameters.

+Triggers define when to run a pipeline. They also include [parameters](../../data-factory/how-to-use-trigger-parameterization.md) that are passed to the pipeline execution. The **Copy DICOM Metadata Changes to Delta** template defines a few parameters that are described in the following table. If no value is supplied during configuration, the listed default value is used for each parameter.

+| BatchSize | The maximum number of changes to retrieve at a time from the change feed (maximum 200) | `200` |

+| ApiVersion | The API version for the Azure DICOM service (minimum 2) | `2` |

+| StartTime | The inclusive start time for DICOM changes | `0001-01-01T00:00:00Z` |

+| EndTime | The exclusive end time for DICOM changes | `9999-12-31T23:59:59Z` |

+| ContainerName | The container name for the resulting Delta tables | `dicom` |

+| InstanceTablePath | The path that contains the Delta table for DICOM SOP instances within the container| `instance` |

+| SeriesTablePath | The path that contains the Delta table for DICOM series within the container | `series` |

+| StudyTablePath | The path that contains the Delta table for DICOM studies within the container | `study` |

+| RetentionHours | The maximum retention in hours for data in the Delta tables | `720` |

+1. On the **Trigger Run Parameters** pane, enter the **ContainerName** value that matches the name of the storage container created in the prerequisites.

+ :::image type="content" source="media/data-factory-trigger-parameters.png" alt-text="Screenshot that shows the Trigger Run Parameters pane, with StartTime and EndTime values entered." lightbox="media/data-factory-trigger-parameters.png":::

+1. For **StartTime**, use the system variable `@formatDateTime(trigger().outputs.windowStartTime)`.

+1. For **EndTime**, use the system variable `@formatDateTime(trigger().outputs.windowEndTime)`.

+ > [!NOTE]

+ > Only tumbling window triggers support the system variables:

+ > * `@trigger().outputs.windowStartTime` and

+ > * `@trigger().outputs.windowEndTime`

+ >

+ > Schedule triggers use different system variables:

+ > * `@trigger().scheduledTime` and

+ > * `@trigger().startTime`

+ >

+ > Learn more about [trigger types](../../data-factory/concepts-pipeline-execution-triggers.md#trigger-type-comparison).

+1. Select **Save** to create the new trigger. Select **Publish** to begin your trigger running on the defined schedule.

+ :::image type="content" source="media/data-factory-publish.png" alt-text="Screenshot that shows the Publish button on the main menu bar." lightbox="media/data-factory-publish.png":::

+After the trigger is published, it can be triggered manually by using the **Trigger now** option. If the start time was set for a value in the past, the pipeline starts immediately.

+## Monitor pipeline runs

+You can monitor trigger runs and their associated pipeline runs on the **Monitor** tab. Here, you can browse when each pipeline ran and how long it took to run. You can also potentially debug any problems that arose.

+[Fabric](https://www.microsoft.com/microsoft-fabric) is an all-in-one analytics solution that sits on top of [Microsoft OneLake](/fabric/onelake/onelake-overview). With the use of a [Fabric lakehouse](/fabric/data-engineering/lakehouse-overview), you can manage, structure, and analyze data in OneLake in a single location. Any data outside of OneLake, written to Data Lake Storage Gen2, can be connected to OneLake as shortcuts to take advantage of Fabric's suite of tools.

+### Create shortcuts

+1. Go to the lakehouse created in the prerequisites. In the **Explorer** view, select the ellipsis menu (**...**) next to the **Tables** folder.

+1. Select **New shortcut** to create a new shortcut to the storage account that contains the DICOM analytics data.

+ :::image type="content" source="media/fabric-create-shortcut.png" alt-text="Screenshot that shows the New shortcut option in the Explorer view." lightbox="media/fabric-create-shortcut.png":::

+1. Select **Azure Data Lake Storage Gen2** as the source for the shortcut.

+ :::image type="content" source="media/fabric-new-shortcut.png" alt-text="Screenshot that shows the New shortcut view with the Azure Data Lake Storage Gen2 tile." lightbox="media/fabric-new-shortcut.png":::

+1. Under **Connection settings**, enter the **URL** you used in the [Linked services](#create-a-linked-service-for-azure-data-lake-storage-gen2) section.

+ :::image type="content" source="media/fabric-connection-settings.png" alt-text="Screenshot that shows the connection settings for the Azure Data Lake Storage Gen2 account." lightbox="media/fabric-connection-settings.png":::

+1. Select an existing connection or create a new connection by selecting the **Authentication kind** you want to use.

+ > [!NOTE]

+ > There are a few options for authenticating between Data Lake Storage Gen2 and Fabric. You can use an organizational account or a service principal. We don't recommend using account keys or shared access signature tokens.

+1. Select **Next**.

+1. Enter a **Shortcut Name** that represents the data created by the Data Factory pipeline. For example, for the `instance` Delta table, the shortcut name should probably be **instance**.

+1. Enter the **Sub Path** that matches the `ContainerName` parameter from [run parameters](#configure-trigger-run-parameters) configuration and the name of the table for the shortcut. For example, use `/dicom/instance` for the Delta table with the path `instance` in the `dicom` container.

+1. Select **Create** to create the shortcut.

+1. Repeat steps 2 to 9 to add the remaining shortcuts to the other Delta tables in the storage account (for example, `series` and `study`).

+After you've created the shortcuts, expand a table to show the names and types of the columns.

+### Run notebooks

+After the tables are created in the lakehouse, you can query them from [Fabric notebooks](/fabric/data-engineering/how-to-use-notebook). You can create notebooks directly from the lakehouse by selecting **Open Notebook** from the menu bar.

+On the notebook page, the contents of the lakehouse can still be viewed on the left side, including the newly added tables. At the top of the page, select the language for the notebook. The language can also be configured for individual cells. The following example uses Spark SQL.

+#### Query tables by using Spark SQL

+In the cell editor, enter a Spark SQL query like a `SELECT` statement.

+This query selects all the contents from the `instance` table. When you're ready, select **Run cell** to run the query.

+After a few seconds, the results of the query appear in a table underneath the cell like the example shown here. The amount of time might be longer if this Spark query is the first in the session because the Spark context needs to be initialized.

+* Use Data Factory templates to create a pipeline from the DICOM service to a Data Lake Storage Gen2 account.

+* Configure a trigger to extract DICOM metadata on an hourly schedule.

+* Use shortcuts to connect DICOM data in a storage account to a Fabric lakehouse.

+* Use notebooks to query for DICOM data in the lakehouse.

+## Next steps

+Learn more about Data Factory pipelines:

+* [Pipelines and activities in Data Factory](../../data-factory/concepts-pipelines-activities.md)

+* [Use Microsoft Fabric notebooks](/fabric/data-engineering/how-to-use-notebook)

+ Title: Migrate to Transport Layer Security (TLS) 1.2 for Azure Blob Storage

+description: Avoid disruptions to client applications that connect to your storage account by migrating to Transport Layer Security (TLS) version 1.2.

+ms.devlang: csharp

+# Migrate to TLS 1.2 for Azure Blob Storage

+On **Nov 1, 2024**, Azure Blob Storage will stop supporting versions 1.0 and 1.1 of Transport Layer Security (TLS). TLS 1.2 will become the new minimum TLS version. This change impacts all existing and new blob storage accounts, using TLS 1.0 and 1.1 in all clouds. Storage accounts already using TLS 1.2 aren't impacted by this change.

+To avoid disruptions to applications that connect to your storage account, you must ensure that your account requires clients to send and receive data by using TLS **1.2**, and remove dependencies on TLS version 1.0 and 1.1.

+## About Transport Layer Security

+Transport Layer Security (TLS) is an internet security protocol that establishes encryption channels over networks to encrypt communication between your applications and servers. When storage data is accessed via HTTPS connections, communication between client applications and the storage account is encrypted using TLS.

+TLS encrypts data sent over the internet to prevent malicious users from accessing private, sensitive information. The client and server perform a TLS handshake to verify each other's identity and determine how they communicate. During the handshake, each party identifies which TLS versions they use. The client and server can communicate if they both support a common version.

+## Why use TLS 1.2?

+We're recommending that customers secure their infrastructure by using TLS 1.2 with Azure Storage. The older TLS versions (1.0 and 1.1) are being deprecated and removed to meet evolving technology and regulatory standards (FedRamp, NIST), and provide improved security for our customers.

+TLS 1.2 is more secure and faster than TLS 1.0 and 1.1, which don't support modern cryptographic algorithms and cipher suites. While many customers using Azure storage are already using TLS 1.2, we're sharing further guidance to accelerate this transition for customers that are still using TLS 1.0 or 1.1.

+## Configure clients to use TLS 1.2

+First, identify each client that makes requests to the Blob Storage service of your account. Then, ensure that each client uses TLS 1.2 to make those requests.

+For each client application, we recommend the following tasks.

+- Update the operating system to the latest version.

+- Update your development libraries and frameworks to their latest versions. (For example, Python 3.6 and 3.7 support TLS 1.2).

+- Fix hardcoded instances of older security protocols TLS 1.0 and 1.1.

+- Configure clients to use a TLS 1.2. See [Configure Transport Layer Security (TLS) for a client application](transport-layer-security-configure-client-version.md?toc=/azure/storage/blobs/toc.json&bc=/azure/storage/blobs/breadcrumb/toc.json).

+For more detailed guidance, see the [checklist to deprecate older TLS versions in your environment](/security/engineering/solving-tls1-problem#figure-1-security-protocol-support-by-os-version).

+> [!IMPORTANT]

+> Notify your customers and partners of your product or service's migration to TLS 1.2 so that they can make the necessary changes to their applications.

+## Enforce TLS 1.2 as the minimum allowed version

+In advance of the deprecation date, you can enable Azure policy to enforce minimum TLS version.

+To understand how configuring the minimum TLS version might affect client applications, we recommend that you enable logging for your Azure Storage account and analyze the logs after an interval of time to detect what versions of TLS client applications are using.

+When you're confident that traffic from clients using older versions of TLS is minimal, or that it's acceptable to fail requests made with an older version of TLS, then you can begin enforcement of a minimum TLS version on your storage account.

+To learn how to detect the TLS versions used by client applications, and then enforce TLS 1.2 as the minimum allowed version, see [Enforce a minimum required version of Transport Layer Security (TLS) for incoming requests for Azure Storage](transport-layer-security-configure-minimum-version.md?toc=/azure/storage/blobs/toc.json&bc=/azure/storage/blobs/breadcrumb/toc.json#detect-the-tls-version-used-by-client-applications).

+## Quick Tips

+- Windows 8+ has TLS 1.2 enabled by default.

+- Windows Server 2016+ has TLS 1.2 enabled by default.

+- When possible, avoid hardcoding the protocol version. Instead, configure your applications to always defer to your operating system's default TLS version.

+- For example, you can enable the SystemDefaultTLSVersion flag in .NET Framework applications to defer to your operating system's default version. This approach lets your applications take advantage of future TLS versions.

+- If you can't avoid hardcoding, specify TLS 1.2.

+- Upgrade applications that target .NET Framework 4.5 or earlier. Instead, use .NET Framework 4.7 or later because these versions support TLS 1.2.

+ For example, Visual Studio 2013 doesn't support TLS 1.2. Instead, use at least the latest release of Visual Studio 2017.

+- Use [Qualys SSL Labs](https://www.ssllabs.com/) to identify which TLS version is requested by clients connecting to your application.

+- Use [Fiddler](https://www.telerik.com/fiddler) to identify which TLS version your client uses when you send out HTTPS requests.

+## Next steps

+- [Solving the TLS 1.0 Problem, 2nd Edition](/security/engineering/solving-tls1-problem) – deep dive into migrating to TLS 1.2.

+- [How to enable TLS 1.2 on clients](/mem/configmgr/core/plan-design/security/enable-tls-1-2-client) – for Microsoft Configuration Manager.

+- [Configure Transport Layer Security (TLS) for a client application](transport-layer-security-configure-client-version.md?toc=/azure/storage/blobs/toc.json&bc=/azure/storage/blobs/breadcrumb/toc.json) – contains instructions to update TLS version in PowerShell

+- [Enable support for TLS 1.2 in your environment for Microsoft Entra ID TLS 1.1 and 1.0 deprecation](/troubleshoot/azure/active-directory/enable-support-tls-environment) – contains information on updating TLS version for WinHTTP.

+- [Transport Layer Security (TLS) best practices with the .NET Framework](/dotnet/framework/network-programming/tls) – best practices when configuring security protocols for applications targeting .NET Framework.

+- [TLS best practices with the .NET Framework](https://github.com/dotnet/docs/issues/4675) – GitHub to ask questions about best practices with .NET Framework.

+- [Troubleshooting TLS 1.2 compatibility with PowerShell](https://github.com/microsoft/azure-devops-tls12) – probe to check TLS 1.2 compatibility and identify issues when incompatible with PowerShell

+### Upload Certificate to Key vault via Azure CLI

+> You must upload the certificate as a secret. You must use Azure CLI to upload certificates as secrets to your key vault.

+> Your Azure Stream Analytics job will fail when the certificate used for authentication expires. To resolve this, you must update/replace the certificate in your key vault and restart your Azure Stream Analytics job

+You can visit this page to get guidance on setting up Azure CLI: [Get started with Azure CLI](https://learn.microsoft.com/cli/azure/get-started-with-azure-cli#how-to-sign-into-the-azure-cli)

+Below are some steps you can follow to upload the your certificate to Azure CLI using your powershell

+**Login to Azure CLI:**

+az login

+**Connect to your subscription containing your key vault:**

+```azurecli-interactive

+az account set --subscription <subscription name>

+```

+**The following command can upload the certificate as a secret to your key vault:**

+```azurecli-interactive

+az keyvault secret set --vault-name <your key vault> --name <name of the secret> --file <file path to secret>

+```

+* Azure Stream Analytics does not support authentication to confluent cloud using OAuth or SAML single sign-on (SSO). You must use API Key via the SASL_SSL protocol

+Azure Stream Analytics lets you connect directly to Kafka clusters to ingest data. The solution is low code and entirely managed by the Azure Stream Analytics team at Microsoft, allowing it to meet business compliance standards. The ASA Kafka input is backward compatible and supports all versions with the latest client release starting from version 0.10. Users can connect to Kafka clusters inside a VNET and Kafka clusters with a public endpoint, depending on the configurations. The configuration relies on existing Kafka configuration conventions. Supported compression types are None, Gzip, Snappy, LZ4, and Zstd.

+The ASA Kafka input is a librdkafka-based client, and to connect to confluent cloud, you need TLS certificates that confluent cloud uses for server auth.

+### Upload Certificate to Key vault via Azure CLI

+> You must have "**Key Vault Administrator**" permissions access to your Key vault for this command to work properly

+> You must upload the certificate as a secret. You must use Azure CLI to upload certificates as secrets to your key vault.

+> Your Azure Stream Analytics job will fail when the certificate used for authentication expires. To resolve this, you must update/replace the certificate in your key vault and restart your Azure Stream Analytics job

+You can visit this page to get guidance on setting up Azure CLI: [Get started with Azure CLI](https://learn.microsoft.com/cli/azure/get-started-with-azure-cli#how-to-sign-into-the-azure-cli)

+**Login to Azure CLI:**

+az login

+**Connect to your subscription containing your key vault:**

+```azurecli-interactive

+az account set --subscription <subscription name>

+```

+**The following command can upload the certificate as a secret to your key vault:**

+```azurecli-interactive

+az keyvault secret set --vault-name <your key vault> --name <name of the secret> --file <file path to secret>

+```

+* When configuring your Azure Stream Analytics jobs to use VNET/SWIFT, your job must be configured with at least six (6) streaming units or one (1) V2 streaming unit. .

+> For direct help with using the Azure Stream Analytics Kafka input, please reach out to [askasa@microsoft.com](mailto:askasa@microsoft.com).

+- Change the [size of a virtual machine](./resize-vm.md).

+## ExpressRoute gateway performance

+## ExpressRoute limits in Virtual WAN

+| Maximum number of circuits connected to the same virtual hub's ExpressRoute gateway | Limit |

+| | |

+| Maximum number of circuits in the same peering location connected to the same virtual hub | 4 |

+| Maximum number of circuits in different peering locations connected to the same virtual hub | 8 |

+The above two limits hold true regardless of the number of ExpressRoute gateway scale units deployed. For ExpressRoute circuit route limits, please see [ExpressRoute Circuit Route Advertisement Limits](../azure-resource-manager/management/azure-subscription-service-limits.md#route-advertisement-limits).