+* Assign `Cognitive Services User` role to your account. Go to the [Azure Portal](https://portal.azure.com/), navigate to your Content Safety resource or Azure AI Services resource, and select **Access Control** in the left navigation bar, then select **+ Add role assignment**, choose the `Cognitive Services User` role and select the member of your account that you need to assign this role to, then review and assign. It might take few minutes for the assignment to take effect.

+> [!IMPORTANT]

+> * You must assign the `Cognitive Services User` role to your Azure account to use the studio experience. Go to the [Azure Portal](https://portal.azure.com/), navigate to your Content Safety resource or Azure AI Services resource, and select **Access Control** in the left navigation bar, then select **+ Add role assignment**, choose the `Cognitive Services User` role and select the member of your account that you need to assign this role to, then review and assign. It might take few minutes for the assignment to take effect.

+### Global batch model availability

+### Region and model support

+The following models support global batch:

+| Model | Version | Input format |

+|||

+|`gpt-4o` | 2024-05-13 |text + image |

+|`gpt-4` | turbo-2024-04-09 | text |

+|`gpt-4` | 0613 | text |

+| `gpt-35-turbo` | 0125 | text |

+| `gpt-35-turbo` | 1106 | text |

+| `gpt-35-turbo` | 0613 | text |

+Global batch is currently supported in the following regions:

+- East US

+- West US

+- Sweden Central

+| `gpt-4o-mini` ^**1** (2024-07-18) | North Central US <br> Sweden Central | Input: 128,000 <br> Output: 16,384 <br> Training example context length: 64,536 | Oct 2023 |

+**¹** GPT-4 and GPT-4o mini fine-tuning is currently in public preview. See our [GPT-4 & GPT-4o mini fine-tuning safety evaluation guidance](/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo%2Cpython-new&pivots=programming-language-python#safety-evaluation-gpt-4-fine-tuningpublic-preview) for more information.

+ Title: 'How to use global batch processing with Azure OpenAI Service'

+description: Learn how to use global batch with Azure OpenAI Service

+recommendations: false

+zone_pivot_groups: openai-fine-tuning-batch

+# Getting started with Azure OpenAI global batch deployments (preview)

+The Azure OpenAI Batch API is designed to handle large-scale and high-volume processing tasks efficiently. Process asynchronous groups of requests with separate quota, with 24-hour target turnaround, at [50% less cost than global standard](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/). With batch processing, rather than send one request at a time you send a large number of requests in a single file. Global batch requests have a separate enqueued token quota avoiding any disruption of your online workloads.

+Key use cases include:

+* **Large-Scale Data Processing:** Quickly analyze extensive datasets in parallel.

+* **Content Generation:** Create large volumes of text, such as product descriptions or articles.

+* **Document Review and Summarization:** Automate the review and summarization of lengthy documents.

+* **Customer Support Automation:** Handle numerous queries simultaneously for faster responses.

+* **Data Extraction and Analysis:** Extract and analyze information from vast amounts of unstructured data.

+* **Natural Language Processing (NLP) Tasks:** Perform tasks like sentiment analysis or translation on large datasets.

+* **Marketing and Personalization:** Generate personalized content and recommendations at scale.

+> [!IMPORTANT]

+> We aim to process batch requests within 24 hours; we do not expire the jobs that take longer. You can [cancel](#cancel-batch) the job anytime. When you cancel the job, any remaining work is cancelled and any already completed work is returned. You will be charged for any completed work.

+>

+> Data stored at rest remains in the designated Azure geography, while data may be processed for inferencing in any Azure OpenAI location.ΓÇ»[Learn more about data residency](https://azure.microsoft.com/explore/global-infrastructure/data-residency/).ΓÇ»

+## Global batch support

+### Region and model support

+Global batch is currently supported in the following regions:

+- East US

+- West US

+- Sweden Central

+The following models support global batch:

+| Model | Version | Supported |

+|||

+|`gpt-4o` | 2024-05-13 |Yes (text + vision) |

+|`gpt-4` | turbo-2024-04-09 | Yes (text only) |

+|`gpt-4` | 0613 | Yes |

+| `gpt-35-turbo` | 0125 | Yes |

+| `gpt-35-turbo` | 1106 | Yes |

+| `gpt-35-turbo` | 0613 | Yes |

+Refer to the [models page](../concepts/models.md) for the most up-to-date information on regions/models where global batch is currently supported.

+### API Versions

+- `2024-07-01-preview`

+### Not supported

+The following aren't currently supported:

+- Integration with the Assistants API.

+- Integration with Azure OpenAI On Your Data feature.

+### Global batch deployment

+In the Studio UI the deployment type will appear as `Global-Batch`.

+> [!TIP]

+> Each line of your input file for batch processing has a `model` attribute that requires a global batch **deployment name**. For a given input file, all names must be the same deployment name. This is different from OpenAI where the concept of model deployments does not exist.

+## Batch object

+|Property | Type | Definition|

+||||

+| `id` | string | |

+| `object` | string| `batch` |

+| `endpoint` | string | The API endpoint used by the batch |

+| `errors` | object | |

+| `input_file_id` | string | The ID of the input file for the batch |

+| `completion_window` | string | The time frame within which the batch should be processed |

+| `status` | string | The current status of the batch. Possible values: `validating`, `failed`, `in_progress`, `finalizing`, `completed`, `expired`, `cancelling`, `cancelled`. |

+| `output_file_id` | string |The ID of the file containing the outputs of successfully executed requests. |

+| `error_file_id` | string | The ID of the file containing the outputs of requests with errors. |

+| `created_at` | integer | A timestamp when this batch was created (in unix epochs). |

+| `in_progress_at` | integer | A timestamp when this batch started progressing (in unix epochs). |

+| `expires_at` | integer | A timestamp when this batch will expire (in unix epochs). |

+| `finalizing_at` | integer | A timestamp when this batch started finalizing (in unix epochs). |

+| `completed_at` | integer | A timestamp when this batch started finalizing (in unix epochs). |

+| `failed_at` | integer | A timestamp when this batch failed (in unix epochs) |

+| `expired_at` | integer | A timestamp when this batch expired (in unix epochs).|

+| `cancelling_at` | integer | A timestamp when this batch started `cancelling` (in unix epochs). |

+| `cancelled_at` | integer | A timestamp when this batch was `cancelled` (in unix epochs). |

+| `request_counts` | object | Object structure:<br><br> `total` *integer* <br> The total number of requests in the batch. <br>`completed` *integer* <br> The number of requests in the batch that have been completed successfully. <br> `failed` *integer* <br> The number of requests in the batch that have failed.

+| `metadata` | map | A set of key-value pairs that can be attached to the batch. This property can be useful for storing additional information about the batch in a structured format. |

+## Frequently asked questions (FAQ)

+### Can images be used with the batch API?

+This capability is limited to certain multi-modal models. Currently only GPT-4o support images as part of batch requests. Images can be provided as input either via [image url or a base64 encoded representation of the image](#input-format). Images for batch are currently not supported with GPT-4 Turbo.

+### Can I use the batch API with fine-tuned models?

+This is currently not supported.

+### Can I use the batch API for embeddings models?

+This is currently not supported.

+### Does content filtering work with Global Batch deployment?

+Yes. Similar to other deployment types, you can create content filters and associate them with the Global Batch deployment type.

+### Can I request additional quota?

+Yes, from the quota page in the Studio UI. Default quota allocation can be found in the [quota and limits article](../quotas-limits.md#global-batch-quota).

+### What happens if the API doesn't complete my request within the 24 hour time frame?

+We aim to process these requests within 24 hours; we don't expire the jobs that take longer. You can cancel the job anytime. When you cancel the job, any remaining work is cancelled and any already completed work is returned. You'll be charged for any completed work.

+### How many requests can I queue using batch?

+There's no fixed limit on the number of requests you can batch, however, it will depend on your enqueued token quota. Your enqueued token quota includes the maximum number of input tokens you can enqueue at one time.

+Once your batch request is completed, your batch rate limit is reset, as your input tokens are cleared. The limit depends on the number of global requests in the queue. If the Batch API queue processes your batches quickly, your batch rate limit is reset more quickly.

+## Troubleshooting

+A job is successful when `status` is `Completed`. Successful jobs will still generate an error_file_id, but it will be associated with an empty file with zero bytes.

+When a job failure occurs, you'll find details about the failure in the `errors` property:

+```json

+"value": [

+ {

+ "cancelled_at": null,

+ "cancelling_at": null,

+ "completed_at": "2024-06-27T06:50:01.6603753+00:00",

+ "completion_window": null,

+ "created_at": "2024-06-27T06:37:07.3746615+00:00",

+ "error_file_id": "file-f13a58f6-57c7-44d6-8ceb-b89682588072",

+ "expired_at": null,

+ "expires_at": "2024-06-28T06:37:07.3163459+00:00",

+ "failed_at": null,

+ "finalizing_at": "2024-06-27T06:49:59.1994732+00:00",

+ "id": "batch_50fa47a0-ef19-43e5-9577-a4679b92faff",

+ "in_progress_at": "2024-06-27T06:39:57.455977+00:00",

+ "input_file_id": "file-42147e78ea42488682f4fd1d73028e72",

+ "errors": {

+ "object": ΓÇ£listΓÇ¥,

+ "data": [

+ {

+ ΓÇ£codeΓÇ¥: ΓÇ£empty_fileΓÇ¥,

+ ΓÇ£messageΓÇ¥: ΓÇ£The input file is empty. Please ensure that the batch contains at least one request.ΓÇ¥

+ }

+ ]

+ },

+ "metadata": null,

+ "object": "batch",

+ "output_file_id": "file-22d970b7-376e-4223-a307-5bb081ea24d7",

+ "request_counts": {

+ "total": 10,

+ "completed": null,

+ "failed": null

+ },

+ "status": "Failed"

+ }

+```

+### Error codes

+|Error code | Definition|

+|||

+|`invalid_json_line`| A line (or multiple) in your input file wasn't able to be parsed as valid json.<br><br> Please ensure no typos, proper opening and closing brackets, and quotes as per JSON standard, and resubmit the request.|

+| `too_many_tasks` |The number of requests in the input file exceeds the maximum allowed value of 100,000.<br><br>Please ensure your total requests are under 100,000 and resubmit the job.|

+| `url_mismatch` | Either a row in your input file has a URL that doesnΓÇÖt match the rest of the rows, or the URL specified in the input file doesnΓÇÖt match the expected endpoint URL. <br><br>Please ensure all request URLs are the same, and that they match the endpoint URL associated with your Azure OpenAI deployment.|

+|`model_not_found`|The Azure OpenAI model deployment name that was specified in the `model` property of the input file wasn't found.<br><br> Please ensure this name points to a valid Azure OpenAI model deployment.|

+| `duplicate_custom_id` | The custom ID for this request is a duplicate of the custom ID in another request. |

+|`empty_batch` | Please check your input file to ensure that the custom ID parameter is unique for each request in the batch.|

+|`model_mismatch`| The Azure OpenAI model deployment name that was specified in the `model` property of this request in the input file doesn't match the rest of the file.<br><br>Please ensure that all requests in the batch point to the same AOAI model deployment in the `model` property of the request.|

+|`invalid_request`| The schema of the input line is invalid or the deployment SKU is invalid. <br><br>Please ensure the properties of the request in your input file match the expected input properties, and that the Azure OpenAI deployment SKU is `globalbatch` for batch API requests.|

+### Known issues

+- Resources deployed with Azure CLI won't work out-of-box with Azure OpenAI global batch. This is due to an issue where resources deployed using this method have endpoint subdomains that don't follow the `https://your-resource-name.openai.azure.com` pattern. A workaround for this issue is to deploy a new Azure OpenAI resource using one of the other common deployment methods which will properly handle the subdomain setup as part of the deployment process.

+## See also

+* Learn more about Azure OpenAI [deployment types](./deployment-types.md)

+* Learn more about Azure OpenAI [quotas and limits](../quotas-limits.md)

+| **Offering** | **Global-Batch** | **Global-Standard** | **Standard** | **Provisioned** |

+||:|:|:|:|

+| **Best suited for** | Offline scoring <br><br> Workloads that are not latency sensitive and can be completed in hours.<br><br> For use cases that do not have data processing residency requirements.| Recommended starting place for customers. <br><br>Global-Standard will have the higher default quota and larger number of models available than Standard. <br><br> For production applications that do not have data processing residency requirements. | For customers with data residency requirements. Optimized for low to medium volume. | Real-time scoring for large consistent volume. Includes the highest commitments and limits.|

+| **How it works** | Offline processing via files |Traffic may be routed anywhere in the world | | |

+| **Getting started** | [Global-Batch](./batch.md) | [Model deployment](./create-resource.md) | [Model deployment](./create-resource.md) | [Provisioned onboarding](./provisioned-throughput-onboarding.md) |

+| **Cost** | [Least expensive option](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) <br> 50% less cost compared to Global Standard prices. Access to all new models with larger quota allocations. | [Global deployment pricing](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) | [Regional pricing](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) | May experience cost savings for consistent usage |

+| **What you get** |[Significant discount compared to Global Standard](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) | Easy access to all new models with highest default pay-per-call limits.<br><br> Customers with high volume usage may see higher latency variability | Easy access with [SLA on availability](https://azure.microsoft.com/support/legal/sl#estimate-provisioned-throughput-and-cost) |

+| **What you don’t get** |❌Real-time call performance <br><br>❌Data processing guarantee<br> <br> Data stored at rest remains in the designated Azure geography, while data may be processed for inferencing in any Azure OpenAI location. [Learn more about data residency](https://azure.microsoft.com/explore/global-infrastructure/data-residency/) |❌Data processing guarantee<br> <br> Data stored at rest remains in the designated Azure geography, while data may be processed for inferencing in any Azure OpenAI location. [Learn more about data residency](https://azure.microsoft.com/explore/global-infrastructure/data-residency/) | ❌High volume w/consistent low latency | ❌Pay-per-call flexibility |

+| **Per-call Latency** | Not Applicable (file based async process) | Optimized for real-time calling & low to medium volume usage. Customers with high volume usage may see higher latency variability. Threshold set per model | Optimized for real-time calling & low to medium volume usage. Customers with high volume usage may see higher latency variability. Threshold set per model | Optimized for real-time. |

+| **Sku Name in code** | `GlobalBatch` | `GlobalStandard` | `Standard` | `ProvisionedManaged` |

+| **Billing model** | Pay-per-token |Pay-per-token | Pay-per-token | Monthly Commitments |

+> Data stored at rest remains in the designated Azure geography, while data may be processed for inferencing in any Azure OpenAI location. [Learn more about data residency](https://azure.microsoft.com/explore/global-infrastructure/data-residency/).

+## Global batch

+> [!IMPORTANT]

+> Data stored at rest remains in the designated Azure geography, while data may be processed for inferencing in any Azure OpenAI location. [Learn more about data residency](https://azure.microsoft.com/explore/global-infrastructure/data-residency/).

+[Global batch](./batch.md) is designed to handle large-scale and high-volume processing tasks efficiently. Process asynchronous groups of requests with separate quota, with 24-hour target turnaround, at [50% less cost than global standard](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/). With batch processing, rather than send one request at a time you send a large number of requests in a single file. Global batch requests have a separate enqueued token quota avoiding any disruption of your online workloads.

+Key use cases include:

+* **Large-Scale Data Processing:** Quickly analyze extensive datasets in parallel.

+* **Content Generation:** Create large volumes of text, such as product descriptions or articles.

+* **Document Review and Summarization:** Automate the review and summarization of lengthy documents.

+* **Customer Support Automation:** Handle numerous queries simultaneously for faster responses.

+* **Data Extraction and Analysis:** Extract and analyze information from vast amounts of unstructured data.

+* **Natural Language Processing (NLP) Tasks:** Perform tasks like sentiment analysis or translation on large datasets.

+* **Marketing and Personalization:** Generate personalized content and recommendations at scale.

+## Retrieve batch job output file

+## August 2024

+### Global batch deployments are now available

+The Azure OpenAI Batch API is designed to handle large-scale and high-volume processing tasks efficiently. Process asynchronous groups of requests with separate quota, with 24-hour target turnaround, at [50% less cost than global standard](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/). With batch processing, rather than send one request at a time you send a large number of requests in a single file. Global batch requests have a separate enqueued token quota avoiding any disruption of your online workloads.

+Key use cases include:

+* **Large-Scale Data Processing:** Quickly analyze extensive datasets in parallel.

+* **Content Generation:** Create large volumes of text, such as product descriptions or articles.

+* **Document Review and Summarization:** Automate the review and summarization of lengthy documents.

+* **Customer Support Automation:** Handle numerous queries simultaneously for faster responses.

+* **Data Extraction and Analysis:** Extract and analyze information from vast amounts of unstructured data.

+* **Natural Language Processing (NLP) Tasks:** Perform tasks like sentiment analysis or translation on large datasets.

+* **Marketing and Personalization:** Generate personalized content and recommendations at scale.

+For more information on [getting started with global batch deployments](./how-to/batch.md).

+### GPT-4o mini is now available for fine-tuning

+GPT-4o mini fine-tuning is [now available in public preview](./concepts/models.md#fine-tuning-models) in Sweden Central and in North Central US.

+In Azure AI Studio, you can use synthetic data generation to efficiently produce predictions for your datasets. In this article, you're introduced to the concept of synthetic data generation and how it can be used in machine learning.

+In machine learning, synthetic data is valuable for several reasons:

+- [Azure AI FAQ article](../faq.yml)

+description: Learn about the regions where each model is available for deployment in serverless API endpoints via Azure AI Studio.

+# Region availability for models in serverless API endpoints

+- The token must have the audience set to exactly `https://*.asazure.windows.net`. Note that `*` isn't a placeholder or a wildcard, and the audience must have the `*` character as the subdomain. Custom audiences, such as https://customersubdomain.asazure.windows.net, are not supported. Specifying an invalid audience results in authentication failure.

+## Set up diagnostic settings

+To learn how to set up diagnostic settings using the Azure portal, Azure CLI, PowerShell, or Azure Resource Manager, see [Create diagnostic settings in Azure Monitor](/azure/azure-monitor/essentials/create-diagnostic-settings).

+To learn how to set up diagnostics logging, see [Set up diagnostic logging](analysis-services-logging.md).

+ > [!NOTE]

+ > Where noted, certain features are available only in the extension's pre-release version. When installing the extension from the [Visual Studio Code Marketplace](https://marketplace.visualstudio.com/items?itemName=apidev.azure-api-center&ssr=false#overview), you can choose to install the release version or a pre-release version. Switch between the two versions at any time by using the extension's **Manage** button context menu in the Extensions view.

+Use the power of GitHub Copilot with the Azure API Center extension for Visual Studio Code to create an OpenAPI specification file from your API code. Right-click on the API code, select **Copilot** from the options, and select **Generate API documentation**. This will create an OpenAPI specification file.

+> [!NOTE]

+> This feature is available in the pre-release version of the API Center extension.

+ > Using the pre-release version of the extension, you can generate API documentation in Markdown, a format that's easy to maintain and share with end users. Right-click on the definition, and select **Generate Markdown**.

+* **Managed** - The managed gateway is the default gateway component that is deployed in Azure for every API Management instance in every service tier. A standalone managed gateway can also be associated with a [workspace](workspaces-overview.md) in an API Management instance. With the managed gateway, all API traffic flows through Azure regardless of where backends implementing the APIs are hosted.

+* **Workspace** - the managed gateway available in a [workspace](workspaces-overview.md) in select service tiers

+| Feature support | Classic | V2 | Consumption | Self-hosted | Workspace |

+| [Custom domains](configure-custom-domain.md) | ✔️ | ✔️ | ✔️ | ✔️ | ❌ |

+| [Built-in cache](api-management-howto-cache.md) | ✔️ | ✔️ | ❌ | ❌ | ✔️ |

+| [External Redis-compatible cache](api-management-howto-cache-external.md) | ✔️ | ✔️ |✔️ | ✔️ | ❌ |

+| [Virtual network injection](virtual-network-concepts.md) | Developer, Premium | ❌ | ❌ | ✔️^1,2 | ✔️ |

+| [Inbound private endpoints](private-endpoint.md) | Developer, Basic, Standard, Premium | ❌ | ❌ | ❌ | ❌ |

+| [Outbound virtual network integration](integrate-vnet-outbound.md) | ❌ | Standard V2 | ❌ | ❌ | ✔️ |

+| [Availability zones](zone-redundancy.md) | Premium | Γ£ö∩╕ų | Γ¥î | Γ£ö∩╕Ź | Γ£ö∩╕ų |

+| [Multi-region deployment](api-management-howto-deploy-multi-region.md) | Premium | Γ¥î | Γ¥î | Γ£ö∩╕Ź | Γ¥î |

+| [CA root certificates](api-management-howto-ca-certificates.md) for certificate validation | Γ£ö∩╕Å | Γ£ö∩╕Å | Γ¥î | Γ£ö∩╕Å⁴ | Γ¥î |

+| [Managed domain certificates](configure-custom-domain.md?tabs=managed#domain-certificate-options) | Developer, Basic, Standard, Premium | ❌ | ✔️ | ❌ | ❌ |

+| [TLS settings](api-management-howto-manage-protocols-ciphers.md) | ✔️ | ✔️ | ✔️ | ✔️ | ❌ |

+| **HTTP/2** (Client-to-gateway) | Γ£ö∩╕Å⁵ | Γ£ö∩╕Å⁵ |Γ¥î | Γ£ö∩╕Å | Γ¥î |

+| **HTTP/2** (Gateway-to-backend) | ❌ | ❌ | ❌ | ✔️ | ❌ |

+| API threat detection with [Defender for APIs](protect-with-defender-for-apis.md) | ✔️ | ✔️ | ❌ | ❌ | ❌ |

+³ Two zones are enabled by default; not configurable.<br/>

+⁴ CA root certificates for self-hosted gateway are managed separately per gateway<br/>

+⁵ Client protocol needs to be enabled.

+| Feature support | Classic | V2 | Consumption | Self-hosted | Workspace |

+| | | -- | -- | - | -- |

+| [OpenAPI specification](import-api-from-oas.md) | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| [WSDL specification](import-soap-api.md) | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| WADL specification | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| [Logic App](import-logic-app-as-api.md) | ✔️ | ✔️ | ✔️ |✔️ | ✔️ |

+| [App Service](import-app-service-as-api.md) | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| [Function App](import-function-app-as-api.md) | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| [Container App](import-container-app-with-oas.md) | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| [Service Fabric](../service-fabric/service-fabric-api-management-overview.md) | Developer, Premium | ❌ |❌ | ❌ | ❌ |

+| [Pass-through GraphQL](graphql-apis-overview.md) | ✔️ | ✔️ |✔️ | ✔️ | ✔️ |

+| [Synthetic GraphQL](graphql-apis-overview.md)| Γ£ö∩╕Å | Γ£ö∩╕Å | Γ£ö∩╕Ź | Γ£ö∩╕Ź | Γ¥î |

+| [Pass-through WebSocket](websocket-api.md) | ✔️ | ✔️ | ❌ | ✔️ | ❌ |

+| [Pass-through gRPC](grpc-api.md) | ❌ | ❌ | ❌ | ✔️ | ❌ |

+| [OData](import-api-from-odata.md) | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| [Azure OpenAI](azure-openai-api-from-specification.md) | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| [Circuit breaker in backend](backends.md#circuit-breaker) | ✔️ | ✔️ | ❌ | ✔️ | ✔️ |

+| [Load-balanced backend pool](backends.md#load-balanced-pool) | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| Feature support | Classic | V2 | Consumption | Self-hosted¹ | Workspace |

+| | | -- | -- | - | -- |

+| [Dapr integration](api-management-policies.md#integration-and-external-communication) | ❌ | ❌ |❌ | ✔️ | ❌ |

+| [GraphQL resolvers](api-management-policies.md#graphql-resolvers) and [GraphQL validation](api-management-policies.md#content-validation)| ✔️ | ✔️ |✔️ | ❌ | ❌ |

+| [Get authorization context](get-authorization-context-policy.md) | ✔️ | ✔️ |✔️ | ❌ | ❌ |

+| [Quota and rate limit](api-management-policies.md#rate-limiting-and-quotas) | Γ£ö∩╕Å | Γ£ö∩╕Ų | Γ£ö∩╕ų | Γ£ö∩╕Å⁴ | Γ£ö∩╕Å |

+| Feature support | Classic | V2 | Consumption | Self-hosted | Workspace |

+| | | -- | -- | - | -- |

+| [API analytics](howto-use-analytics.md) | Γ£ö∩╕Å | Γ£ö∩╕Ź | Γ¥î | Γ¥î | Γ¥î |

+| [Application Insights](api-management-howto-app-insights.md) | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| [Logging through Event Hubs](api-management-howto-log-event-hubs.md) | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+| [Metrics in Azure Monitor](api-management-howto-use-azure-monitor.md#view-metrics-of-your-apis) | ✔️ | ✔️ |✔️ | ✔️ | ❌ |

+| [OpenTelemetry Collector](how-to-deploy-self-hosted-gateway-kubernetes-opentelemetry.md) | ❌ | ❌ | ❌ | ✔️ | ❌ |

+| [Request logs in Azure Monitor and Log Analytics](api-management-howto-use-azure-monitor.md#resource-logs) | Γ£ö∩╕Å | Γ£ö∩╕Å | Γ¥î | Γ¥î² | Γ¥î |

+| [Local metrics and logs](how-to-configure-local-metrics-logs.md) | ❌ | ❌ | ❌ | ✔️ | ❌ |

+| [Request tracing](api-management-howto-api-inspector.md) | Γ£ö∩╕Å | Γ¥î³ | Γ£ö∩╕Å | Γ£ö∩╕Å | Γ¥î |

+| Feature support | Classic | V2 | Consumption | Self-hosted | Workspace |

+| | | -- | -- | - | -- |

+| [Credential manager](credentials-overview.md) | ✔️ | ✔️ | ✔️ | ❌ | ❌ |

+### Workspace gateway

+Scale capacity by adding and removing scale [units](upgrade-and-scale.md) in the workspace gateway.

+> [!NOTE]

+> In an API Management [workspace](workspaces-overview.md), a workspace owner can independently integrate Application Insights and enable Application Insights logging for the workspace's APIs. The general guidance to integrate a workspace with Application Insights is similar to the guidance for an API Management instance; however, configuration is scoped to the workspace only. Currently, you must integrate Application Insights in a workspace by configuring an instrumentation key or connection string.

+ > If your Application Insights resource is in a different tenant, then you must create the logger using the [REST API](#create-a-connection-using-the-rest-api-bicep-or-arm-template) as shown in a later section of this article.

+Use the API Management [Logger - Create or Update](/rest/api/apimanagement/current-preview/logger/create-or-update) REST API with the following request body.

+If you are configuring the logger for a workspace, use the [Workspace Logger - Create or Update](/rest/api/apimanagement/workspace-logger/create-or-update?view=rest-apimanagement-2023-09-01-preview&preserve-view=true) REST API.

+Use the API Management [Logger - Create or Update](/rest/api/apimanagement/current-preview/logger/create-or-update) REST API with the following request body.

+Use the API Management [Logger - Create or Update](/rest/api/apimanagement/current-preview/logger/create-or-update) REST API with the following request body.

+## Related content

+> * Currently, autoscale is not supported for the [workspace gateway](workspaces-overview.md#workspace-gateway) in API Management workspaces.

+> * In [workspaces](workspaces-overview.md), the managed gateway doesn't support changes to the default protocol and cipher configuration.

+|`context`|[`Api`](#ref-context-api): [`IApi`](#ref-iapi)<br /><br /> [`Deployment`](#ref-context-deployment)<br /><br /> Elapsed: `TimeSpan` - time interval between the value of `Timestamp` and current time<br /><br /> [`GraphQL`](#ref-context-graphql)<br /><br />[`LastError`](#ref-context-lasterror)<br /><br /> [`Operation`](#ref-context-operation)<br /><br /> [`Request`](#ref-context-request)<br /><br /> `RequestId`: `Guid` - unique request identifier<br /><br /> [`Response`](#ref-context-response)<br /><br /> [`Subscription`](#ref-context-subscription)<br /><br /> `Timestamp`: `DateTime` - point in time when request was received<br /><br /> `Tracing`: `bool` - indicates if tracing is on or off <br /><br /> [User](#ref-context-user)<br /><br /> [`Variables`](#ref-context-variables): `IReadOnlyDictionary<string, object>`<br /><br /> `void Trace(message: string)` <br /><br /> [`Workspace`](#ref-context-workspace) |

+|<a id="ref-context-api"></a>`context.Api`|`Id`: `string`<br /><br /> `IsCurrentRevision`: `bool`<br /><br /> `Name`: `string`<br /><br /> `Path`: `string`<br /><br /> `Revision`: `string`<br /><br /> `ServiceUrl`: [`IUrl`](#ref-iurl)<br /><br /> `Version`: `string` |

+|<a id="ref-context-product"></a>`context.Product`|`ApprovalRequired`: `bool`<br /><br /> `Groups`: `IEnumerable<`[`IGroup`](#ref-igroup)`>`<br /><br /> `Id`: `string`<br /><br /> `Name`: `string`<br /><br /> `State`: `enum ProductState {NotPublished, Published}`<br /><br /> `SubscriptionsLimit`: `int?`<br /><br /> `SubscriptionRequired`: `bool`|

+|<a id="ref-context-workspace"></a>`context.Workspace`| `Id`: `string`<br /><br /> `Name`: `string`|

+ Title: How to use role-based access control in Azure API Management | Microsoft Docs

+A workspace collaborator must be assigned both a workspace-scoped role and a service-scoped role.

+Depending on how workspace collaborators use or manage the workspace, we recommend also assigning one of the following Azure-provided RBAC roles at the scope of the [workspace gateway](workspaces-overview.md#workspace-gateway): **Reader**, **Contributor**, or **Owner**.

+## Built-in developer portal roles

+|Role |Scope |Description |

+||||

+|API Management Developer Portal Content Editor | service | Can customize the developer portal, edit its content, and publish it using Azure Resource Manager APIs. |

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Policy scopes:**](./api-management-howto-policies.md#scopes) global, product, API, operation

+You can import an Azure OpenAI API directly to API Management from the Azure OpenAI Service.

+When you import the API, API Management automatically configures:

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Policy scopes:**](./api-management-howto-policies.md#scopes) global, product, API, operation

+- [**Policy scopes:**](./api-management-howto-policies.md#scopes) global, product, API, operation

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, self-hosted, workspace

+| [Workspaces preview breaking changes][workspaces2024] | June 14, 2024 |

+| [Workspaces preview breaking changes, part 2][workspaces2025march] | March 31, 2025 |

+[workspaces2025march]: ./workspaces-breaking-changes-march-2025.md

+ Title: Azure API Management workspaces preview - breaking changes (June 2024) | Microsoft Docs

+> [!IMPORTANT]

+> If you created workspaces after the generally available release of workspaces in July 2024, your workspaces shouldn't be affected by these changes.

+>

+After 14 June 2024, as part of our development of [workspaces](../workspaces-overview.md) in Azure API Management, we're introducing several breaking changes.

+* [Workspaces breaking changes, part 2 (March 2025)](workspaces-breaking-changes-march-2025.md)

+ Title: Azure API Management workspaces preview - breaking changes (March 2025)

+description: Azure API Management is removing support for preview workspaces. If your service uses preview workspaces, migrate your workspaces to the generally available version.

+# Workspaces breaking changes, part 2 (March 2025)

+> [!IMPORTANT]

+> These breaking changes apply only to *preview* workspaces in Azure API Management. If you created workspaces after the generally available release in July 2024 and use workspaces with workspace gateways, your workspaces shouldn't be affected by these changes.

+>

+Azure API Management [workspaces](../workspaces-overview.md) are now generally available, and we introduced several feature updates with that release. As part of our continued development of workspaces, we're removing support for preview workspaces (created before July 2024). If you created preview workspaces in Azure API Management and want to continue using them, you need to migrate your workspaces to the generally available version.

+After 31 March 2025, your preview workspaces and APIs managed in them may stop working if you haven't migrated to the latest workspace capabilities. APIs and resources managed outside workspaces aren't affected by this change.

+## Is my service affected by these changes?

+Your service may be affected by these changes if you created preview workspaces in your API Management instance, before the generally available release of workspaces in July 2024. Workspaces created after the generally available release date that use workspace gateways for API runtime aren't affected by the breaking changes.

+## Breaking changes

+The following are breaking changes that require you to take action to migrate your preview workspaces to the generally available version:

+* **Workspace API gateway is required** - Each workspace must be associated with a workspace API gateway that isolates the workspace's runtime traffic. In preview, workspaces shared a gateway with the service.

+* **Service-level managed identities are not supported** - To improve the security of workspaces, system-assigned and user-assigned managed identities enabled at the service level can't be used in workspaces. Currently, related API Management features that depend on managed identities, such as storing named values and certificates in Azure Key Vault, and using the `authentication-managed-identity` policy, aren't supported in workspaces.

+> [!NOTE]

+> These breaking changes are in addition to the [June 2024 breaking changes](workspaces-breaking-changes-june-2024.md) for preview workspaces that were announced previously.

+## What is the deadline for the change?

+The breaking changes will be enforced in preview workspaces after 31 March 2025. We strongly recommend that you make all required changes to the configuration of your preview workspaces before that date.

+## What do I need to do?

+If your workspaces are affected by these changes, you need to migrate your workspaces to align with the generally available capabilities. The following sections provide guidance on how to migrate your workspaces.

+### Use Premium tier for your API Management instance

+Ensure that your API Management instance is running in the **Premium** tier to continue using workspaces. As announced [previously](workspaces-breaking-changes-june-2024.md), if your instance is in the **Standard** or **Developer** tier, you need to upgrade to the **Premium** tier.

+### Confirm the region for your instance

+Adding a workspace gateway to a workspace requires that the gateway is in the same region as your instance. Currently, workspace gateways are supported in a [subset of regions](../workspaces-overview.md#workspace-gateway) in which API Management is available. The regions with support for workspace gateways will be updated over time.

+To determine if a preview workspace is in a supported region:

+1. In the [Azure portal](https://portal.azure.com), navigate to your API Management instance.

+1. In the left menu, under **APIs**, select **Workspaces**, and select a workspace.

+1. If your workspace is in a region that doesn't support workspace gateways, you'll see a message in the portal similar to "Workspaces are currently unavailable in the region of your API Management service".

+ * If you see this message, you can [move your API Management instance](../api-management-howto-migrate.md) to a supported region.

+ * If you don't see this message, your workspace is in a supported region and you can proceed to add a workspace gateway.

+### Add a workspace gateway to your workspace

+The following are abbreviated steps to add a workspace gateway to a workspace. For gateway networking options, prerequisites, and detailed instructions, see [Create and manage a workspace](../how-to-create-workspace.md).

+> [!NOTE]

+> * The workspace gateway incurs additional charges. For more information, see [API Management pricing](https://aka.ms/apimpricing).

+> * API Management currently supports a dedicated gateway per workspace only. If this is impacting your migration plans, see the workspaces roadmap in the [workspaces GA announcement](https://aka.ms/apim/workspaces/ga-announcement).

+1. In the [Azure portal](https://portal.azure.com), navigate to your API Management instance.

+1. In the left menu, under **APIs**, select **Workspaces**.

+1. Select a workspace.

+1. In the left menu, under **Deployment + infrastructure**, select **Gateways** > **+ Add**.

+1. Complete the wizard to create a gateway. Currently, provisioning of the gateway can take from several minutes to up to 3 hours or longer.

+1. After your gateway is provisioned, go to the gateway's **Overview** page. Note the value of **Runtime hostname**. Use this value to update your client apps that call your workspace's APIs.

+1. Repeat the preceding steps for your remaining workspaces.

+### Update client apps to use the new gateway hostname

+After adding a gateway to your workspace, you need to update your client apps that call the workspace's APIs to use the new gateway hostname instead of the gateway hostname of your API Management instance.

+> [!NOTE]

+> To help you migrate your workspaces, APIs in workspaces can still be accessed at runtime through October 2024 using the gateway hostname of your API Management instance, even if a workspace gateway is associated with a workspace. We strongly recommend that you complete migration before this date. If your workspace gateways are configured with private inbound access and private outbound access, make sure that connectivity to your API Management instance's built-in gateway is also secured.

+### Update dependencies on service-level managed identities

+If you're using service-level managed identities in the configuration of workspace entities (for example, named values or certificates), you need to update the configurations. Recommended steps vary depending on the entity. Example: Update named values to use secret values instead of secrets stored in Azure Key Vault.

+## Help and support

+If you have questions, get answers from community experts in [Microsoft Q&A](https://aka.ms/apim/azureqa/change/captcha-2022). If you have a support plan and you need technical help, create a [support request](https://portal.azure.com/#view/Microsoft_Azure_Support/HelpAndSupportBlade/~/overview).

+## More information

+* [Workspaces overview](../workspaces-overview.md)

+* [Workspaces breaking changes (June 2024)](workspaces-breaking-changes-june-2024.md)

+## Related content

+See all [upcoming breaking changes and feature retirements](overview.md).

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+> [!NOTE]

+> Currently, custom domain names aren't supported in a [workspace gateway](workspaces-overview.md#workspace-gateway).

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+1. Configure your desired the values for the remaining settings, or use the default values. Select **Add**.

+ > [!NOTE]

+ > If you want to allow guest users as well as signed-in users to access the developer portal on WordPress, you can enable unauthenticated access. In **Restrict access**, select **Allow unauthenticated access**. [Learn more](../app-service/overview-authentication-authorization.md#authorization-behavior)

+Access the WordPress site to see your new API Management developer portal deployed on WordPress and hosted on App Service.

+1. In a new browser window, navigate to your WordPress site, substituting the name of your app service in the following URL: `https://<yourapp-service-name>.azurewebsites.net`.

+1. When prompted, sign in using Microsoft Entra ID credentials for a developer account. If unauthenticated access to the developer portal is enabled, select **Sign in** on the home page of the developer portal.

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Policy scopes:**](./api-management-howto-policies.md#scopes) global, product, API, operation

+description: Learn how to create a workspace and a workspace gateway in Azure API Management. Workspaces allow decentralized API development teams to own and productize their own APIs.

+# Create and manage a workspace in Azure API Management

+Set up a [workspace](workspaces-overview.md) to enable an API team to manage and productize their own APIs, while providing the API platform team with the tools to observe, govern, and maintain the API Management platform. After you create a workspace and assign permissions, workspace collaborators can create and manage their own APIs, products, subscriptions, and related resources.

+Follow the steps in this article to:

+* Create an API Management workspace and a workspace gateway using the Azure portal

+* Optionally, isolate the workspace gateway in an Azure virtual network

+* Assign permissions to the workspace

+> Currently, creating a workspace gateway is a long-running operation that can take up to 3 hours or more to complete.

+* An API Management instance. If you need to, [create one](get-started-create-service-instance.md) in a supported tier.

+* **Owner** or **Contributor** role on the resource group where the API Management instance is deployed, or equivalent permissions to create resources in the resource group.

+* (Optional) An existing or new Azure virtual network and subnet to isolate the workspace gateway's inbound and outbound traffic. For configuration options and requirements, see [Network resource requirements for workspace gateways](virtual-network-workspaces-resources.md).

+

+1. In the left menu, under **APIs**, select **Workspaces** > **+ Add**.

+1. On the **Basics** tab, enter a descriptive **Display name**, resource **Name**, and optional **Description** for the workspace. Select **Next**.

+1. On the **Gateway** tab, configure settings for the workspace gateway:

+ * In **Gateway details**, enter a gateway name and select the number of scale **Units**. The gateway costs are based on the number of units you select. For more information, see [API Management pricing](https://aka.ms/apimpricing).

+ * In **Network**, select a **Network configuration** for your workspace gateway.

+ > [!IMPORTANT]

+ > Plan your workspace's network configuration carefully. You can't change the network configuration after you create the workspace.

+ * If you select a network configuration that includes private inbound or private outbound network access, select a **Virtual network** and **Subnet** to isolate the workspace gateway, or create a new one. For network requirements, see [Network resource requirements for workspace gateways](virtual-network-workspaces-resources.md).

+1. Select **Next**. After validation completes, select **Create**.

+It can take from several minutes to up to several hours to create the workspace, workspace gateway, and related resources. To track the deployment progress in the Azure portal, go to the gateway's resource group. In the left menu, under **Settings**, select **Deployments**.

+After the deployment completes, the new workspace appears in the list on the **Workspaces** page. Select the workspace to manage its settings and resources.

+> [!NOTE]

+> * To view the gateway runtime hostname and other gateway details, select the workspace in the portal. Under **Deployment + infrastructure**, select **Gateways**, and select the name of the workspace's gateway.

+> * While the workspace gateway is being created, runtime calls to the workspace's APIs won't succeed.

+To manage the workspace gateway, we recommend also assigning workspace users an Azure-provided RBAC role scoped to the workspace gateway.

+1. In the menu for your API Management instance, under **APIs**, select **Workspaces** > the name of the workspace that you created.

+1. Assign one of the following workspace-scoped roles to the workspace members so that they can manage workspace APIs and other resources.

+### Assign a gateway-scoped role

+1. Sign in to the [Azure portal](https://portal.azure.com), and navigate to your API Management instance.

+1. In the left menu, under **APIs**, select **Workspaces** > the name of your workspace.

+1. In the left menu of the workspace, select **Gateways**, and select the workspace gateway.

+1. In the left menu, select **Access control (IAM)** > **+ Add**.

+1. Assign one of the following roles to each member of the workspace. At minimum, we recommend assigning the **Reader** role to view the gateway's settings. **Owners** and **Contributors** can manage the gateway's settings including scaling the gateway.

+

+ * **Owner**

+ * **Contributor**

+ * **Reader**

+## Get started with your workspace

+Depending on their role in the workspace, users might have permissions to create APIs, products, subscriptions, and other resources, or they might have read-only access to some or all of them.

+To get started managing, protecting, and publishing APIs in a workspace, see the following guidance.

+|Resource |Guide |

+|||

+|APIs | [Tutorial: Import and publish your first API](import-and-publish.md) |

+|Products | [Tutorial: Create and publish a product](api-management-howto-add-products.md) |

+|Subscriptions | [Subscriptions in Azure API Management](api-management-subscriptions.md)<br/><br/>[Create subscriptions in API Management](api-management-howto-create-subscriptions.md) |

+|Policies | [Tutorial: Transform and protect your API](transform-api.md)<br/><br/>[Policies in Azure API Management](api-management-howto-policies.md)<br/><br/>[Set or edit API Management policies](set-edit-policies.md) |

+|Named values | [Manage secrets using named values](api-management-howto-properties.md) |

+| Backends | [Use backends in Azure API Management](backends.md) |

+|Policy fragments | [Reuse policy configurations in your API Management policy definitions](policy-fragments.md) |

+| Schemas | [Validate content](validate-content-policy.md) |

+| Groups | [Create and use groups to manage developer accounts](api-management-howto-create-groups.md) |

+| Notifications | [How to configure notifications and notification templates](api-management-howto-configure-notifications.md) |

+## Related content

+* Learn more about [workspaces in Azure API Management](workspaces-overview.md).

+* [Use a virtual network to secure inbound or outbound traffic for Azure API Management](virtual-network-concepts.md)

+This article shows how to import an Azure Container App to Azure API Management and test the imported API using the Azure portal.

+In this article, you learn how to:

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Policy scopes:**](./api-management-howto-policies.md#scopes) global, product, API, operation

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Policy scopes:**](./api-management-howto-policies.md#scopes) global, product, API, operation

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+| **[API Inspector](api-management-howto-api-inspector.md)** | Testing and debugging | Instant | Last 100 traces | Turned on per request | Request traces | Managed, Self-hosted, Azure Arc, Workspace |

+| **[Azure Application Insights](api-management-howto-app-insights.md)** | Reporting, monitoring, and debugging | Seconds | 90 days/5GB (upgrade to extend) | Custom | Logs, metrics | Managed¹, Self-hosted¹, Azure Arc¹, Workspace¹ |

+* Connections aren't supported on the [self-hosted gateway](self-hosted-gateway-overview.md) or on a [workspace gateway](workspaces-overview.md#workspace-gateway).

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Policy scopes:**](./api-management-howto-policies.md#scopes) global, product, API, operation

+- [**Gateways:**](api-management-gateways-overview.md) classic, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- **[Gateways:](api-management-gateways-overview.md)** dedicated, consumption, self-hosted, workspace

+- [**Policy scopes:**](./api-management-howto-policies.md#scopes) global, product, API, operation

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+### Add or remove units - workspace gateway

+1. Navigate to your API Management instance in the [Azure portal](https://portal.azure.com/).

+1. In the left menu, under **APIs**, select **Workspaces** > the name of your workspace.

+1. In the left menu, under **Deployment + infrastructure**, select **Gateways** > the name of your gateway.

+1. In the left menu, under **Deployment and infrastructure**, select **Scale**.

+1. Specify the new number of **Units** - use the slider, or select or type the number.

+1. Select **Save**.

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+ Title: Azure API Management virtual network injection - network resources

+ Title: Azure API Management workspace gateways - VNet integration - network resources

+description: Learn about requirements for network resources when you integrate your API Management workspace gateway in an Azure virtual network.

+# Network resource requirements for integration of a workspace gateway into a virtual network

+Network isolation is an optional feature of an API Management [workspace gateway](workspaces-overview.md#workspace-gateway). This article provides network resource requirements when you integrate your gateway in an Azure virtual network. Some requirements differ depending on the desired inbound and outbound access mode. The following modes are supported:

+* Public inbound access, private outbound access (Public/Private)

+* Private inbound access, private outbound access (Private/Private)

+For information about networking options in API Management, see [Use a virtual network to secure inbound or outbound traffic for Azure API Management](virtual-network-concepts.md).

+## Network location

+* The virtual network must be in the same region and Azure subscription as the API Management instance.

+## Subnet size

+* The subnet size must be `/24` (256 IP addresses).

+* The subnet can't be shared with another Azure resource, including another workspace gateway.

+## Subnet delegation

+The subnet must be delegated as follows to enable the desired inbound and outbound access.

+For information about configuring subnet delegation, see [Add or remove a subnet delegation](../virtual-network/manage-subnet-delegation.md).

+#### [Public/Private](#tab/external)

+For Public/Private mode, the subnet needs to be delegated to the **Microsoft.Web/serverFarms** service.

+> [!NOTE]

+> You might need to register the `Microsoft.Web/serverFarms` resource provider in the subscription so that you can delegate the subnet to the service.

+#### [Private/Private](#tab/internal)

+For Private/Private mode, the subnet needs to be delegated to the **Microsoft.Web/hostingEnvironments** service.

+> [!NOTE]

+> You might need to register the `Microsoft.Web/hostingEnvironments` resource provider in the subscription so that you can delegate the subnet to the service.

+## Network security group (NSG) rules

+A network security group (NSG) must be attached to the subnet to explicitly allow inbound connectivity. Configure the following rules in the NSG. Set the priority of these rules higher than that of the default rules.

+#### [Public/Private](#tab/external)

+| Source / Destination Port(s) | Direction | Transport protocol | Source | Destination | Purpose |

+||--|--||-|--|

+| */80 | Inbound | TCP | AzureLoadBalancer | Workspace gateway subnet range | Allow internal health ping traffic |

+| */80,443 | Inbound | TCP | Internet | Workspace gateway subnet range | Allow inbound traffic |

+#### [Private/Private](#tab/internal)

+| Source / Destination Port(s) | Direction | Transport protocol | Source | Destination | Purpose |

+||--|--||-|--|

+| */80 | Inbound | TCP | AzureLoadBalancer | Workspace gateway subnet range | Allow internal health ping traffic |

+| */80,443 | Inbound | TCP | Virtual network | Workspace gateway subnet range | Allow inbound traffic |

+## DNS settings for Private/Private configuration

+In the Private/Private network configuration, you have to manage your own DNS to enable inbound access to your workspace gateway.

+We recommend:

+1. Configure an Azure [DNS private zone](../dns/private-dns-overview.md).

+1. Link the Azure DNS private zone to the VNet into which you've deployed your workspace gateway.

+Learn how to [set up a private zone in Azure DNS](../dns/private-dns-getstarted-portal.md).

+### Access on default hostname

+When you create an API Management workspace, the workspace gateway is assigned a default hostname. The hostname is visible in the Azure portal on the workspace gateway's **Overview** page, along with its private virtual IP address. The default hostname is in the format `<gateway-name>-<random hash>.gateway.<region>-<number>.azure-api.net`. Example: `team-workspace-123456abcdef.gateway.uksouth-01.azure-api.net`.

+> [!NOTE]

+> The workspace gateway only responds to requests to the hostname configured on its endpoint, not its private VIP address.

+### Configure DNS record

+Create an A record in your DNS server to access the workspace from within your VNet. Map the endpoint record to the private VIP address of your workspace gateway.

+For testing purposes, you might update the hosts file on a virtual machine in a subnet connected to the VNet in which API Management is deployed. Assuming the private virtual IP address for your workspace gateway is 10.1.0.5, you can map the hosts file as shown in the following example. The hosts mapping file is at `%SystemDrive%\drivers\etc\hosts` (Windows) or `/etc/hosts` (Linux, macOS).

+| Internal virtual IP address | Gateway hostname |

+| -- | -- |

+| 10.1.0.5 | `teamworkspace.gateway.westus.azure-api.net` |

+## Related content

+* [Use a virtual network to secure inbound or outbound traffic for Azure API Management](virtual-network-concepts.md)

+* [Workspaces in Azure API Management](workspaces-overview.md)

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+description: Learn about Azure API Management workspaces. With workspaces, decentralized API development teams manage and productize APIs in a common service infrastructure.

+#customer intent: As administrator of an API Management instance, I want to learn about using workspaces to manage APIs in a decentralized way, so that I can enable my development teams to manage and productize their own APIs.

+# What are workspaces in Azure API Management?

+In API Management, *workspaces* bring a new level of autonomy to an organization's API teams, enabling them to create, manage, and publish APIs faster, more reliably, securely, and productively within an API Management service. By providing isolated administrative access and API runtime, workspaces empower API teams while allowing the API platform team to retain oversight. This includes central monitoring, enforcement of API policies and compliance, and publishing APIs for discovery through a unified developer portal.

+Workspaces function like "folders" within an API Management service:

+* Each workspace contains APIs, products, subscriptions, named values, and related resources.

+* Access to resources within a workspace is managed through Azure's role-based access control (RBAC) with built-in or custom roles assignable to Microsoft Entra accounts.

+* Each workspace is associated with a *workspace gateway* for routing API traffic to the backend services of APIs in the workspace.

+## Federated API management with workspaces

+Workspaces add first-class support for a *federated model* of managing APIs in API Management, in addition to already supported centralized and siloed models. See the following table for a comparison of these models.

+|Model|Description |

+|||

+|**Centralized**<br/><br/>:::image type="content" source="media/workspaces-overview/centralized.png" alt-text="Diagram of the centralized model of Azure API Management." border="false" lightbox="media/workspaces-overview/centralized.png"::: |**Pros**<br/>ΓÇó Centralized API governance and observability<br/>ΓÇó Unified developer portal for effective API discovery and onboarding<br/>ΓÇó Cost-efficiency of the infrastructure<br/><br/>**Cons**<br/>ΓÇó No segregation of administrative permissions between teams<br/>ΓÇó API gateway is a single point of failure<br/>ΓÇó Inability to attribute runtime issues to specific teams<br/>ΓÇó Burden on platform team to facilitate collaboration may reduce API growth |

+|**Siloed**<br/><br/>:::image type="content" source="media/workspaces-overview/siloed.png" alt-text="Diagram of the siloed model of Azure API Management." border="false" lightbox="media/workspaces-overview/siloed.png"::: |**Pros**<br/>ΓÇó Segregation of administrative permissions between teams increases productivity and security<br/>ΓÇó Segregation of API runtime between teams increases API reliability, resiliency, and security<br/>ΓÇó Runtime issues are contained and attributable to specific teams<br/><br/>**Cons**<br/>ΓÇó Lack of centralized API governance and observability<br/>ΓÇó Lack of unified developer portal<br/>ΓÇó Increased cost and harder platform managementΓÇï |

+|**Federated**<br/><br/>:::image type="content" source="media/workspaces-overview/federated.png" alt-text="Diagram of the federated model of Azure API Management." border="false" lightbox="media/workspaces-overview/federated.png"::: |**Pros**<br/>ΓÇó Centralized API governance and observability<br/>ΓÇó Unified developer portal for effective API discovery and onboarding<br/>ΓÇó Segregation of administrative permissions between teams increases productivity and security<br/>ΓÇó Segregation of API runtime between teams increases API reliability, resiliency, and security<br/>ΓÇó Runtime issues are contained and attributable to specific teams<br/><br/>**Cons**<br/>ΓÇó Platform cost and management difficulty greater than in the centralized model but lower than in the siloed model |

+An organization that manages APIs using Azure API Management may have multiple development teams that develop, define, maintain, and productize different sets of APIs. Workspaces allow these teams to use API Management to manage, access, and secure their APIs separately, and independently of managing the service infrastructure.

+1. A central API platform team that manages the API Management instance creates a workspace and assigns permissions to workspace collaborators using RBAC roles - for example, permissions to create or read resources in the workspace. A dedicated API gateway is also created for the workspace.

+1. The central API platform team manages the infrastructure of the service, such as monitoring, resiliency, and enforcement of all-APIs policies.

+## API management in a workspace

+Teams manage their own APIs, products, subscriptions, backends, policies, loggers, and other resources within workspaces. See the API Management [REST API reference](/rest/api/apimanagement/workspace?view=rest-apimanagement-2023-09-01-preview&preserve-view=true) for a full list of resources and operations supported in workspaces.

+While workspaces are managed independently from the API Management service and other workspaces, by design they can reference selected service-level resources. See [Workspaces and other API Management features](#workspaces-and-other-api-management-features), later in this article.

+## Workspace gateway

+Each workspace can be associated with workspace gateways to enable runtime of APIs managed within the workspace. The workspace gateway is a standalone Azure resource with the same core functionality as the gateway built into your API Management service.

+Workspace gateways are managed independently from the API Management service and from each other. They ensure isolation of runtime between workspaces, increasing API reliability, resiliency, and security and enabling attribution of runtime issues to workspaces.

+* For information on the cost of workspace gateways, see [API Management pricing](https://aka.ms/apimpricing).

+* For a detailed comparison of API Management gateways, see [API Management gateways overview](api-management-gateways-overview.md).

+### Gateway hostname

+Each association of a workspace to a workspace gateway creates a unique hostname for APIs managed in that workspace. Default hostnames follow the pattern `<workspace-name>-<hash>.gateway.<region>.azure-api.net`. Currently, custom hostnames aren't supported for workspace gateways.

+> Through October 2024, APIs in workspaces can be accessed at runtime using the gateway hostname of your API Management instance in addition to the hostname of the workspace gateway.

+### Network isolation

+A workspace gateway can optionally be configured in a private virtual network to isolate inbound and/or outbound traffic. If configured, the workspace gateway must use a dedicated subnet in the virtual network.

+For detailed requirements, see [Network resource requirements for workspace gateways](virtual-network-workspaces-resources.md).

+### Scale capacity

+Manage gateway capacity by manually adding or removing scale units, similar to the [units](upgrade-and-scale.md) that can be added to the API Management instance in certain service tiers. The costs of a workspace gateway are based on the number of units you select.

+### Regional availability

+Workspace gateways need to be in the same Azure region and subscription as the API Management service.

+> [!NOTE]

+> Starting in August 2024, workspace gateway support will be rolled out in the following regions. These regions are a subset of those where API Management is available.

+* West US

+* North Central US

+* UK South

+* France Central

+* North Europe

+* East Asia

+* Southeast Asia

+* Australia East

+* Japan East

+### Gateway constraints

+The following constraints currently apply to workspace gateways:

+* A gateway can be associated only with one workspace

+* A workspace can't be associated with a self-hosted gateway

+* Workspace gateways don't support inbound private endpoints

+* APIs in workspace gateways can't be assigned custom hostnames

+* APIs in workspaces aren't covered by Defender for APIs

+* Workspace gateways don't support the API Management service's credential manager

+* Workspace gateways support only internal cache; external cache isn't supported

+* Workspace gateways don't support synthetic GraphQL APIs and WebSocket APIs

+* Workspace gateways don't support APIs created from Azure resources such as Azure OpenAI Service, App Service, Function Apps, and so on

+* Request metrics can't be split by workspace in Azure Monitor; all workspace metrics are aggregated at the service level

+* Azure Monitor logs are aggregated at the service level; workspace-level logs aren't available

+* Workspace gateways don't support CA certificates

+* Workspace gateways don't support autoscaling

+* Workspace gateways don't support managed identities, including related features like storing secrets in Azure Key Vault and using the `authentication-managed-identity` policy

+## RBAC roles for workspaces

+To manage APIs and other resources in the workspace, workspace members must be assigned roles (or equivalent permissions using custom roles) scoped to the API Management service, the workspace, and the workspace gateway. The service-scoped role enables referencing certain service-level resources from workspace-level resources. For example, organize a user into a workspace-level group to control API and product visibility.

+Workspaces are designed to be self-contained to maximize segregation of administrative access and API runtime. There are several exceptions to ensure higher productivity and enable platform-wide governance, observability, reusability, and API discovery.

+* **Resource references** - Resources in a workspace can reference other resources in the workspace and selected resources from the service level, such as users, authorization servers, or built-in user groups. They can't reference resources from another workspace.

+ For security reasons, it's not possible to reference service-level resources from workspace-level policies (for example, named values) or by resource names, such as `backend-id` in the [set-backend-service](set-backend-service-policy.md) policy.

+ > [!IMPORTANT]

+ > All resources in an API Management service (for example, APIs, products, tags, or subscriptions) need to have unique names, even if they are located in different workspaces. There can't be any resources of the same type and with the same Azure resource name in the same workspace, in other workspaces, or on the service level.

+ >

+* **Developer portal** - Workspaces are an administrative concept and aren't surfaced as such to developer portal consumers, including through the developer portal UI and the underlying API. APIs and products within a workspace can be published to the developer portal, just like APIs and products on the service level.

+ > [!NOTE]

+ > API Management supports assigning authorization servers defined on the service level to APIs within workspaces.

+ >

+## Migrate from preview workspaces

+If you created preview workspaces in Azure API Management and want to continue using them, migrate your workspaces to the generally available version by associating a workspace gateway with each workspace.

+For details and to learn about other changes that could affect your preview workspaces, see [Workspaces breaking changes (March 2025)](breaking-changes/workspaces-breaking-changes-march-2025.md).

+## Deleting a workspace

+Deleting a workspace deletes all its child resources (APIs, products, and so on) and its associated gateway, if you're deleting the workspace using the Azure portal interface. It doesn't delete the API Management instance or other workspaces.

+

+* [Workspaces breaking changes - March 2025](breaking-changes/workspaces-breaking-changes-march-2025.md)

+* [Limits - API Management workspaces](../azure-resource-manager/management/azure-subscription-service-limits.md?toc=/azure/api-management/toc.json&bc=/azure/api-management/breadcrumb/toc.json#limitsapi-management-workspaces)

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+- [**Gateways:**](api-management-gateways-overview.md) classic, v2, consumption, self-hosted, workspace

+> Since scaling is blocked during the migration, you should scale your environment to the desired size before starting the migration. If you have auto-scaling enabled, if a scaling event occurs before the migration starts, you have to wait until the scaling event completes before starting the migration. You should disable auto-scaling before starting the migration to avoid this issue. If you need to scale your environment after the migration, you can do so once the migration is complete.

+Since scaling is blocked during the migration, you should scale your environment to the desired size before starting the migration. If you need to scale your environment after the migration, you can do so once the migration is complete. If you have auto-scaling enabled, if a scaling event occurs before the migration starts, your migration is blocked until the scaling event completes. You should disable auto-scaling before starting the migration to avoid this issue.

+> Since scaling is blocked during the migration, you should scale your environment to the desired size before starting the migration. If you have auto-scaling enabled, if a scaling event occurs before the migration starts, you have to wait until the scaling event completes before starting the migration. You should disable auto-scaling before starting the migration to avoid this issue. If you need to scale your environment after the migration, you can do so once the migration is complete.

+Since scaling is blocked during the migration, you should scale your environment to the desired size before starting the migration. If you need to scale your environment after the migration, you can do so once the migration is complete. If you have auto-scaling enabled, if a scaling event occurs before the migration starts, your migration is blocked until the scaling event completes. You should disable auto-scaling before starting the migration to avoid this issue.

+|Azure Application Gateway version compatibility |[v1](../../application-gateway/overview.md), [v2](../../application-gateway/overview-v2.md) |[v1](../../application-gateway/overview.md), [v2](../../application-gateway/overview-v2.md) |[v2](../../application-gateway/overview-v2.md) |

+> The resource specific option is currently available in all **clouds**.<br>

+ Title: Configure Application Gateway for Containers for Prometheus and Grafana

+description: Configure Application Gateway for Containers metrics to be sent to Prometheus and displayed on Grafana.

+# Configure Application Gateway for Containers for Prometheus and Grafana

+Establishing monitoring for Application Gateway for Containers is crucial part of successful operations. Firstly, it allows you to visualize how traffic is controlled, providing actionable insights that help optimize performance and troubleshoot issues promptly. Secondly, monitoring enhances security measures by providing valuable insights during investigations, ensuring that your gateway remains secure and resilient against threats. Implementing monitoring for your Application Gateway for Containers not only supports ongoing performance optimization but also strengthens your overall security posture by enabling proactive detection and response capabilities.

+You can monitor Azure Application Gateway for Containers resources in the following ways. Refer to the diagram.

+- [Backend Health Metrics](../../application-gateway/for-containers/application-gateway-for-containers-metrics.md): ALB Controller's metric and backend health endpoints exposes several metrics and summary of backend health. The metrics endpoint enables exposure to Prometheus.

+

+- [Metrics](../../application-gateway/for-containers/application-gateway-for-containers-metrics.md): Metrics and Activity Logs are exposed through Azure Monitor to monitor the performance of your Application Gateway for Containers deployments. The metrics contain numerical values in an ordered set of time-series data.

+

+- [Diagnostic Logs](../../application-gateway/for-containers/diagnostics.md): Access Logs audit all requests made to Application Gateway for Containers. Logs can provide several characteristics, such as the client's IP, requested URL, request latencies, return code, and bytes in and out. An access log is collected every 60 seconds.

+[](./media/prometheus-grafana/design-arch.png#lightbox)

+## Learn About the Services

+- [What is Azure Managed Prometheus?](../../azure-monitor/essentials/prometheus-metrics-overview.md)

+ - Why use Prometheus: Azure Prometheus offers native integration and management capabilities, simplifying the setup and management of monitoring infrastructure.

+- [What is Azure Managed Grafana?](../../managed-grafan)

+ - Why use Grafana: Azure Managed Grafana lets you bring together all your telemetry data into one place and Built-in support for Azure Monitor and Azure Data Explorer using Microsoft Entra identities.

+- [What is Azure Log Analytics Workspace?](../../azure-monitor/logs/log-analytics-workspace-overview.md)

+ - Why use Log Analytics Workspace: Log Analytics workspace scales with your business needs, handling large volumes of log data efficiently and detects and diagnose issues quickly.

+

+## Prerequisites

+- An Azure account for work or school and an active subscription. You can create an account for free.

+- Active Kubernetes cluster.

+- Active Application Gateway for Container deployment.

+- Active Resource Group with contributor permission.

+ > [!TIP]

+ > Alternative to Contributor role, you may also want to leverage the following:

+ > - Custom Role with 'microsoft.monitor/accounts/write'.

+ > - Read access.

+ > - Grafana Admin.

+ > - Log Analytics Contributor.

+ > - Monitoring Contributor permissions.

+ > [Learn more about custom roles here](https://aka.ms/custom-roles).

+

+

+## Create new Applications for Configuration

+Complete the steps to configure prometheus and grafana.

+1. Sign in to the [Azure portal](https://portal.azure.com) with your Azure account.

+2. In **Search resources, service, and docs**, type **Application Gateways for Containers** and select your Kubernetes Cluster name.

+

+ [  ](./media/prometheus-grafana/configure.png#lightbox)

+

+3. Under insights and select **Configure Monitoring**.

+

+ [  ](./media/prometheus-grafana/grafana-container.png#lightbox)

+ Create new instances of Log analytics, Azure Monitor (Prometheus), and Managed Grafana to store current Kubernetes cluster metrics.

+4. In **Search resources, service, and docs**, type **Managed Prometheus** and select.

+

+ [  ](./media/prometheus-grafana/managed-prometheus.png#lightbox)

+

+5. Follow the steps to enable Azure Monitor to enable Managed Prometheus service by selecting **Create**.

+6. Create Azure Monitor Workspace Instance:

+ 1. In the **Create** an Azure Monitor Workspace page, select a subscription and resource group.

+ 2. Provide a name and a region for the workspace.

+ 3. Select **Review + create** to create the workspace.

+7. Add Prometheus Config Map to your cluster:

+ 1. Copy this file to notepad or Visual Studio Code: https://github.com/Azure/prometheus-collector/blob/main/otelcollector/configmaps/ama-metrics-settings-configmap.yaml.

+ 2. Modify line 35 to set podannotationnamespaceregex from ΓÇ£ΓÇ¥ to "azure-alb-system".

+ ```Bash

+ # Example Kusto Query

+ podannotationnamespaceregex = "azure-alb-system"

+ ```

+ 3. Save the file as configprometheus.yaml.

+ 4. Add file into CLI (command-line interfaces) under manage files.

+ 5. Run the following command:

+ ```Bash

+ # Run the Following Command in Bash

+ kubectl apply -f configprometheus.yaml

+ ```

+8. [Create a managed Grafana](../../managed-grafan).

+ Link a Grafana Workspace:

+ - In **Search resources, service, and docs**, type **Azure Monitor**.

+ - Select your monitor workspace.

+ - Select **Linked Grafana Workspaces**.

+ 

+9. Select a Grafana workspace.

+10. Select **Link**.

+## Configure Kubernetes cluster for logging

+We created the resources and now we combine all resources and configure prometheus.

+1. Cluster configuration

+ 1. In **Search resources, service, and docs**, search for your kubernetes cluster.

+ 2. Search for insights and Select on **Configure Monitoring**.

+2. Specify each instance:

+ - Log analytics workspace: Use the default new log analytics workspace created for you.

+ - Managed Prometheus: Select on **ΓÇ£Enable Prometheus metricsΓÇ¥** checkbox.

+ - Select on advanced setting: specify the Azure monitor workspace recently created.

+ - Grafana Workspace: Select on **Enable Grafana** checkbox.

+ - Select on advanced setting: specify the Grafana instance recently created.

+ - Select **ΓÇ£ConfigureΓÇ¥**.

+ > [!NOTE]

+ > Check for ama-metrics under workloads in your kubernetes cluster.

+ > [  ](./media/prometheus-grafana/notes-image.png#lightbox)

+

+## Enable diagnostic logs for Application Gateway for Containers

+Activity logging is automatically enabled for every Resource Manager resource. For Access Logs, you must enable access logging to start collecting the data available through those logs. To enable logging, you may configure diagnostic settings in Azure Monitor.

+1. [Create a log analytics workspace](../../azure-monitor/logs/quick-create-workspace.md).

+2. Send logs from Application Gateway for Containers to log analytics workspace:

+ 1. Enter **Application Gateway for Containers** in the search box. Select your active Application Gateway for Container resource.

+ 2. Search and select Diagnostic Setting under Monitoring. Add diagnostic setting.

+ 3. Select a name, check box **allLogs** which includes the Application Gateway for Container Access Logs, and select **Send to Log analytics Workspace** with your desired subscription and recently made log analytics workspace.

+ [  ](./media/prometheus-grafana/logs-all.png#lightbox)

+3. Select **Save**.

+## Access Grafana dashboard

+In this section, we enter Grafana default dashboards.

+1. In **Search resources, service, and docs**, select your **Managed Grafana**.

+2. Select the grafana resource used for configuring monitoring in the cluster.

+3. Select on Endpoint URL in the overview.

+ 

+4. After entering your user credentials, refer to the Grafana introduction.

+5. Select on the left side bar to access default dashboards under dashboards.

+ 

+## Graph Prometheus metrics on Grafana

+In this section, we visualize a sample metric from Prometheus metrics. Refer to all Prometheus metrics availabilities here: [Prometheus Metrics](../../application-gateway/for-containers/application-gateway-for-containers-metrics.md).

+1. In the right top corner, Select **Add Dashboard**.

+2. Select **Add Visualization**.

+3. Search for prometheus under data source.

+

+4. Select desired metric. For Example: alb_controller_total_unhealthy_endpoints that gives any unhealthy endpoints of your backend service.

+5. Choose app as alb-controller.

+6. Select name of the panel, type of visualization, and time range.

+ 

+7. **Save + Apply** of your panel to add into your dashboard.

+ > [!NOTE]

+ > Add a custom legend by {{variable_name}}.

+## Graph access logs and metrics on Grafana

+In this section, we visualize a sample logs from Log Analytics Workspace. Refer to all diagnostic Logs availabilities here: [Diagnostic Logs](../../application-gateway/for-containers/diagnostics.md).

+### Workspace for logs

+1. In the right top corner, Select **Add + Add Dashboard**.

+2. Select **Add Visualization**.

+3. Search for Azure Monitor under data source + **Add**.

+

+4. Change service as **Logs**.

+5. Type:

+ ```kusto

+ // Example Kusto Query

+ AGCAccessLogs

+ | project BackendResponseLatency, TimeGenerated

+ ```

+6. Select a **Time Series** as a visualization.

+7. Select name, description, and time range of the panel.

+

+8. **Save + Apply** to your dashboard.

+### Workspace for metrics

+1. In the right top corner, select **Add + Add Dashboard**.

+2. Select **Add Visualization**.

+3. Search for Azure Monitor under data source+ **Add**.

+4. Change service as Metrics.

+5. Select your application gateway for containers instance.

+](./media/prometheus-grafana/metrics-logs-datasource.png#lightbox)

+6. Select metric namespace as microsoft.servicenetworking/trafficcontrollers.

+7. Choose a metric such as **total requests** and type of data visualization.

+[  ](./media/prometheus-grafana/metrics-logs.png#lightbox)

+8. Select a name, description, and time range of the panel.

+9. **Save + Apply** to your dashboard.

+Congratulations! You set up a monitoring service to enhance your health tracking!

+The Azure OpenAI extension for Azure Functions implements a set of triggers and bindings that enable you to easily integrate features and behaviors of [Azure OpenAI Service](../ai-services/openai/overview.md) into your function code executions.

+In the Turbine.cs project file, replace the contents of the generated class library code with the following code:

+One way to detect attacks is through activity monitoring and logging analytics. Functions integrates with Application Insights to collect log, performance, and error data for your function app. Application Insights automatically detects performance anomalies and includes powerful analytics tools to help you diagnose issues and understand how your functions are used. To learn more, see [Monitor Azure Functions](functions-monitoring.md).

+Functions also integrates with Azure Monitor Logs to enable you to consolidate function app logs with system events for easier analysis. You can use diagnostic settings to configure the streaming export of platform logs and metrics for your functions to the destination of your choice, such as a Logs Analytics workspace. To learn more, see [Monitoring Azure Functions with Azure Monitor Logs](functions-monitor-log-analytics.md).

+By default, clients can connect to function endpoints by using either HTTP or HTTPS. You should redirect HTTP to HTTPS because HTTPS uses the SSL/TLS protocol to provide a secure connection, which is both encrypted and authenticated. To learn how, see [Enforce HTTPS](../app-service/configure-ssl-bindings.md#enforce-https).

+As with any application or service, the goal is to run your function app with the lowest possible permissions.

+Connection strings and other credentials stored in application settings give all of the functions in the function app the same set of permissions in the associated resource. Consider minimizing the number of functions with access to specific credentials by moving functions that don't use those credentials to a separate function app. You can always use techniques such as [function chaining](/training/modules/chain-azure-functions-data-using-bindings/) to pass data between functions in different function apps.

+To be able to connect to the various services and resources needed to run your code, function apps need to be able to access secrets, such as connection strings and service keys. This section describes how to store secrets required by your functions.

+App settings and connection strings are stored encrypted in Azure. They're decrypted only before being injected into your app's process memory when the app starts. The encryption keys are rotated regularly. If you prefer to manage the secure storage of your secrets, the app settings should instead be references to Azure Key Vault secrets.

+You can also encrypt settings by default in the `local.settings.json` file when developing functions on your local computer. For more information, see [Encrypt the local settings file](functions-run-local.md#encrypt-the-local-settings-file).

+While application settings are sufficient for most functions, you may want to share the same secrets across multiple services. In this case, redundant storage of secrets results in more potential vulnerabilities. A more secure approach is to use a central secret storage service and use references to this service instead of the secrets themselves.

+[Azure Key Vault](../key-vault/general/overview.md) is a service that provides centralized secrets management, with full control over access policies and audit history. You can use a Key Vault reference in place of a connection string or key in your application settings. To learn more, see [Use Key Vault references for App Service and Azure Functions](../app-service/app-service-key-vault-references.md?toc=/azure/azure-functions/toc.json).

+Consider setting a usage quota for functions running in a Consumption plan. When you set a daily GB-sec limit on the total execution of functions in your function app, execution is stopped when the limit is reached. This could potentially help mitigate against malicious code executing your functions. To learn how to estimate consumption for your functions, see [Estimating Consumption plan costs](functions-consumption-costs.md).

+While it seems basic, it's important to write good error handling in your functions. Unhandled errors bubble up to the host and are handled by the runtime. Different bindings handle the processing of errors differently. To learn more, see [Azure Functions error handling](functions-bindings-error-pages.md).

+Azure Functions tooling integration makes it easy to publish local function project code to Azure. It's important to understand how deployment works when considering security for an Azure Functions topology.

+### Secure the `scm` endpoint

+Every function app has a corresponding `scm` service endpoint that is used by the Advanced Tools (Kudu) service for deployments and other App Service [site extensions](https://github.com/projectkudu/kudu/wiki/Azure-Site-Extensions). The `scm` endpoint for a function app is always a URL in the form `https://<FUNCTION_APP_NAME>.scm.azurewebsites.net`. When you use network isolation to secure your functions, you must also account for this endpoint.

+By having a separate `scm` endpoint, you can control deployments and other Advanced Tools functionalities for function apps that are isolated or running in a virtual network. The `scm` endpoint supports both basic authentication (using deployment credentials) and single sign-on with your Azure portal credentials. To learn more, see [Accessing the Kudu service](https://github.com/projectkudu/kudu/wiki/Accessing-the-kudu-service).

+Access restrictions allow you to define lists of allow/deny rules to control traffic to your app. Rules are evaluated in priority order. If no rules are defined, your app will accept traffic from any address. To learn more, see [Azure App Service Access Restrictions](../app-service/app-service-ip-restrictions.md?toc=/azure/azure-functions/toc.json).

+|Models available|US Gov Arizona:<br>&nbsp;&nbsp;&nbsp;GPT-4o (2024-05-13)&nbsp;&nbsp;&nbsp;GPT-4 (1106-Preview)<br>&nbsp;&nbsp;&nbsp;GPT-3.5-Turbo (0125)&nbsp;&nbsp;&nbsp;GPT-3.5-Turbo (1106)<br>&nbsp;&nbsp;&nbsp;text-embedding-ada-002 (version 2)<br><br>US Gov Virginia:<br>&nbsp;&nbsp;&nbsp;GPT-4o (2024-05-13)&nbsp;&nbsp;&nbsp;GPT-4 (1106-Preview)<br>&nbsp;&nbsp;&nbsp;GPT-3.5-Turbo (0125)<br>&nbsp;&nbsp;&nbsp;text-embedding-ada-002 (version 2)<br><br>Learn more about the different capabilities of each model in [Azure OpenAI Service models](../ai-services/openai/concepts/models.md)|

+* To request quota increases for the pay-as-you-go consumption model, apply at [https://aka.ms/AOAIGovQuota](https://aka.ms/AOAIGovQuota)

+| July 2024 | **Windows**<ul><li>Security hardening of Agent data folder.</li><li>Fixed credential leaks in agent logs.</li><li>Various bug fix for AzureWatson.</li><li>Added columns to the Windows Event table: Keywords, UserName, Opcode, Correlation, ProcessId, ThreadId, EventRecordId.</li><li>AMA: Support for private preview of Agent side transformation.</li><li>AMA: Support AMA Client Installer for selected partners.</li></ul>**Linux Features**<ul><li>Enable Dynamic Linking of OpenSSL 1.1 in all regions</li><li>Add Computer field to Custom Logs</li><li>Add EventHub upload support for Custom Logs </li><li>Reliability improvement for upload task scheduling</li><li>Added support for SUSE15 SP5, Ubuntu 24, and AWS 3 distributions</li></ul>**Linux Fixes**<ul><li>Fix Direct upload to storage for perf counters when no other destination is configured. You don't see perf counters If storage was the only configured destination for perf counters, they wouldn't see perf counters in their blob or table.</li><li>Fluent-Bit updated to version 3.0.7. This fixes the issue with Fluent-Bit creating junk files in the root directory on process shutdown.</li><li>Fix proxy for system-wide proxy using http(s)_proxy env var </li><li>Support for syslog hostnames that are up to 255characters</li><li>Stop sending rows longer than 1MB. This exceeds ingestion limits and destabilizes the agent. Now the row is gracefully dropped and a diagnostic message is written.</li><li>Set max disk space used for rsyslog spooling to 1GB. There was no limit before which could lead to high memory usage.</li><li>Use random available TCP port when there is a port conflict with AMA port 28230 and 28330 . This resolved issues where port 28230 and 28330 were already in uses by the customer which prevented data upload to Azure.</li></ul>| 1.29 | 1.32.2 |

+> Creating a cluster involves multiple resources and operation typically complete in two hours.

+> - Linking a workspace to a cluster involves syncing multiple backend components and cache hydration, which typically complete in two hours.

+> - When linking a Log Analytics workspace workspace, the workspace billing plan in changed to *LACluster*, and you should remove SKU in workspace template to prevent conflict during workspace deployment.

+ A large volume cannot be resized to more than 30% of its lowest provisioned size. This limit is adjustable via [a support request](azure-netapp-files-resource-limits.md#resource-limits).

+* [API Management workspaces](#limitsapi-management-workspaces)

+### Limits - API Management workspaces

+| | Mute remote participants | ✔️ |

+zone_pivot_groups: acs-plat-web-ios-android-windows

+You need to collect consent from all participants in the call before you can transcribe them. Microsoft Teams allows users to start transcription in the meetings or calls. You would receive event when transcription has started on you can check the transcription state, if transcription started before you joined the call or meeting.

+## Support

+The following tables define support of call transcription in Azure Communication Services.

+## Identities and call types

+The following tables show support of transcription for specific call type and identity.

+|Identities | Teams meeting | Room | 1:1 call | Group call | 1:1 Teams interop call | Group Teams interop call |

+|--|||-|||--|

+|Communication Services user | ✔️ | | | | ✔️ | ✔️ |

+|Microsoft 365 user | ✔️ | | | | ✔️ | ✔️ |

+## Operations

+The following tables show support of individual APIs in calling SDK to individual identity types.

+|Operations | Communication Services user | Microsoft 365 user |

+|--||-|

+|Get event that transcription has started | ✔️ | ✔️ |

+|Get transcription state | ✔️ | ✔️ |

+|Start or stop transcription | | |

+## SDKs

+The following tables show support of transcription in individual Azure Communication Services SDKs.

+| Platforms | Web | Web UI | iOS | iOS UI | Android | Android UI | Windows |

+|-|--|--|--|--||||

+|Is Supported | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |

+azd config set aspire.dashboard on

+Log in to the Azure CLI:

+1. In the terminal, log in to your Azure subscription.

+ Title: Microservice APIs powered by Dapr

+# Microservice APIs powered by Dapr

+Azure Container Apps provides APIs powered by [Distributed Application Runtime (Dapr)][dapr-concepts] that help you write and implement simple, portable, resilient, and secured microservices. Dapr works together with Azure Container Apps as an abstraction layer to provide a low-maintenance and scalable platform. Azure Container Apps offers a selection of fully managed Dapr APIs, components, and features, catered specifically to microservice scenarios. Simply [enable and configure Dapr][dapr-enable] as usual in your container app environment.

+## How the microservices APIs work with your container app

+Configure microservices APIs for your container apps environment with a [Dapr-enabled container app][dapr-enable], a [Dapr component configured for your solution][dapr-components], and a Dapr sidecar invoking communication between them. The following diagram demonstrates these core concepts, using the pub/sub API as an example.

+## Next steps

+description: Deploy a sample application to Azure Container Apps that leverages the Dapr Bindings API.

+> * Run the application locally with the Dapr CLI.

+1. Clone the [sample application](https://github.com/Azure-Samples/bindings-dapr-nodejs-cron-postgres) to your local machine.

+### Run the application using the Dapr CLI

+1. Run the JavaScript service application.

+ The `dapr run` command runs the binding application locally. Once the application is running successfully, the terminal window shows the output binding data.

+## Deploy the application template using Azure Developer CLI

+Now that you've run the application locally, let's deploy the bindings application to Azure Container Apps using [`azd`](/azure/developer/azure-developer-cli/overview). During deployment, we will swap the local containerized PostgreSQL for an Azure PostgreSQL component.

+1. Run `azd up` to provision the infrastructure and deploy the application to Azure Container Apps in a single command.

+1. Clone the [sample application](https://github.com/Azure-Samples/bindings-dapr-python-cron-postgres) to your local machine.

+### Run the application using the Dapr CLI

+1. Run the Python service application.

+ The `dapr run` command runs the binding application locally. Once the application is running successfully, the terminal window shows the output binding data.

+## Deploy the application template using Azure Developer CLI

+Now that you've run the application locally, let's deploy the bindings application to Azure Container Apps using [`azd`](/azure/developer/azure-developer-cli/overview). During deployment, we will swap the local containerized PostgreSQL for an Azure PostgreSQL component.

+1. Run `azd up` to provision the infrastructure and deploy the application to Azure Container Apps in a single command.

+1. Clone the [sample application](https://github.com/Azure-Samples/bindings-dapr-csharp-cron-postgres) to your local machine.

+### Run the application using the Dapr CLI

+1. Run the .NET service application.

+ The `dapr run` command runs the binding application locally. Once the application is running successfully, the terminal window shows the output binding data.

+## Deploy the application template using Azure Developer CLI

+Now that you've run the application locally, let's deploy the bindings application to Azure Container Apps using [`azd`](/azure/developer/azure-developer-cli/overview). During deployment, we will swap the local containerized PostgreSQL for an Azure PostgreSQL component.

+1. Run `azd up` to provision the infrastructure and deploy the application to Azure Container Apps in a single command.

+- Learn more about [deploying microservices using Dapr to Azure Container Apps](./microservices-dapr.md).

+- [Scale your applications using KEDA scalers](./dapr-keda-scaling.md)

+description: Enable two sample applications to send and receive messages and leverage the Dapr pub/sub API.

+# Tutorial: Microservices communication using Dapr Publish and Subscribe

+In this tutorial, you create publisher and subscriber microservices that leverage [the Dapr Pub/sub API](./dapr-overview.md#supported-dapr-apis-components-and-tooling) to communicate using messages for event-driven architectures. You'll:

+> * Create a publisher microservice and a subscriber microservice that leverage the [Dapr pub/sub API](https://docs.dapr.io/developing-applications/building-blocks/pubsub/pubsub-overview/) to communicate using messages for event-driven architectures.

+1. A message generator `checkout` service (publisher) that generates messages of a specific topic.

+1. An `order-processor` service (subscriber) that listens for messages from the `checkout` service of a specific topic.

+1. Clone the [sample application](https://github.com/Azure-Samples/pubsub-dapr-nodejs-servicebus) to your local machine.

+### Run the applications using the Dapr CLI

+Start by running the `order-processor` subscriber service.

+1. Run the `order-processor` service.

+1. Run the `checkout` service.

+## Deploy the application template using Azure Developer CLI

+Deploy the application to Azure Container Apps using [`azd`](/azure/developer/azure-developer-cli/overview).

+1. Run `azd up` to provision the infrastructure and deploy the application to Azure Container Apps in a single command.

+1. Clone the [sample application](https://github.com/Azure-Samples/pubsub-dapr-python-servicebus) to your local machine.

+### Run the applications using the Dapr CLI

+Start by running the `order-processor` subscriber service.

+1. Run the `order-processor` service.

+1. Run the `checkout` service.

+## Deploy the application template using Azure Developer CLI

+Deploy the application to Azure Container Apps using [`azd`](/azure/developer/azure-developer-cli/overview).

+1. Run `azd up` to provision the infrastructure and deploy the application to Azure Container Apps in a single command.

+1. Clone the [sample application](https://github.com/Azure-Samples/pubsub-dapr-csharp-servicebus) to your local machine.

+### Run the applications using the Dapr CLI

+Start by running the `order-processor` subscriber service

+1. Run the `order-processor` service.

+1. Run the `checkout` service.

+## Deploy the application template using Azure Developer CLI

+Deploy the application to Azure Container Apps using [`azd`](/azure/developer/azure-developer-cli/overview).

+1. Run `azd up` to provision the infrastructure and deploy the application to Azure Container Apps in a single command.

+- Learn more about [deploying applications to Azure Container Apps](./microservices-dapr.md).

+- [Scale your applications using KEDA scalers](./dapr-keda-scaling.md)

+# Tutorial: Microservices communication using Dapr Service Invocation

+In this tutorial, you create and run two microservices that communicate securely using auto-mTLS and reliably using built-in retries via [the Dapr Service Invocation API](./dapr-overview.md#supported-dapr-apis-components-and-tooling). You'll:

+> * Run the application locally.

+1. A `checkout` service that uses HTTP proxying on a loop to invoke a request on the `order-processor` service.

+1. An `order-processor` service that receives the request from the `checkout` service.

+1. Clone the [sample applications](https://github.com/Azure-Samples/svc-invoke-dapr-nodejs) to your local machine.

+### Run the applications using the Dapr CLI

+1. Run the `order-processor` service.

+1. Run the `checkout` service.

+## Deploy the application template using Azure Developer CLI

+Deploy the application to Azure Container Apps using [`azd`](/azure/developer/azure-developer-cli/overview).

+1. Run `azd up` to provision the infrastructure and deploy the application to Azure Container Apps in a single command.

+1. Clone the [sample applications](https://github.com/Azure-Samples/svc-invoke-dapr-python) to your local machine.

+### Run the applications using the Dapr CLI

+1. Run the `order-processor` service.

+1. Run the `checkout` service.

+## Deploy the application template using Azure Developer CLI

+Deploy the application to Azure Container Apps using [`azd`](/azure/developer/azure-developer-cli/overview).

+1. Run `azd up` to provision the infrastructure and deploy the application to Azure Container Apps in a single command.

+1. Clone the [sample applications](https://github.com/Azure-Samples/svc-invoke-dapr-csharp) to your local machine.

+### Run the applications using the Dapr CLI

+Start by running the `order-processor` callee service.

+1. Run the `order-processor` service.

+1. Run the `checkout` service.

+## Deploy the application template using Azure Developer CLI

+Deploy the application to Azure Container Apps using [`azd`](/azure/developer/azure-developer-cli/overview).

+1. Run `azd up` to provision the infrastructure and deploy the application to Azure Container Apps in a single command.

+- [Dapr integration](./dapr-overview.md)

+Log in to the Azure CLI:

+> [!NOTE]

+> You can now view the **Service tenant ID** for subscriptions billed to your account on the **Azure Subscriptions** page under __Cost Management + Billing.__

+> [!NOTE]

+> Enrollments don't automatically transfer if a new enrollment number is generated at renewal. You must include your prior enrollment number in your renewal paperwork to facilitate an automatic transfer.

+ - **Self Service**: Leverage RedWolf to run tests yourself. For more information about RedWolf's self-service, see [Self Service](https://www.redwolfsecurity.com/self-serve-testing/).

+| CIS Azure Kubernetes Service (AKS Benchmark) | California Consumer Privacy Act (CCPA) | CIS GCP Foundations|

+| CMMC |CIS Amazon Elastic Kubernetes Service (EKS) Benchmark| CIS Google Cloud Platform Foundation Benchmark|

+| FedRAMP ΓÇÿHΓÇÖ & ΓÇÿMΓÇÖ | CIS AWS Foundations | CIS Google Kubernetes Engine (GKE) Benchmark|

+| HIPAA/HITRUST | CRI Profile | CRI Profile|

+| ISO/IEC 27001 | CSA Cloud Controls Matrix (CCM) | CSA Cloud Controls Matrix (CCM)|

+| New Zealand ISM Restricted | GDPR | Cybersecurity Maturity Model Certification (CMMC)|

+| NIST SP 800-171 | ISO/IEC 27001 | FFIEC Cybersecurity Assessment Tool (CAT)|

+| NIST SP 800-53 | ISO/IEC 27002 | GDPR|

+| PCI DSS | NIST Cybersecurity Framework (CSF) | ISO/IEC 27001|

+| RMIT Malaysia | NIST SP 800-172 | ISO/IEC 27002|

+| SOC 2 | PCI DSS | ISO/IEC 27017|

+## Dell Edge Gateway 3200 - Bill of materials

+ Title: Dell PowerEdge R660 for operational technology (OT) monitoring - Microsoft Defender for IoT

+description: Learn about the Dell PowerEdge R660 appliance's configuration when used for OT monitoring with Microsoft Defender for IoT in enterprise deployments.

+# Dell PowerEdge R660

+This article describes the Dell PowerEdge R660 appliance, supported for operational technology (OT) sensors in an enterprise deployment.

+The Dell PowerEdge R660 is also available for the on-premises management console.

+|Appliance characteristic | Description|

+|||

+|**Hardware profile** | R660 |

+|**Performance** | Max bandwidth: 3 Gbps<br>Max devices: 12,000 |

+|**Physical Specifications** | Mounting: 1U with rail kit<br>Ports: 6x RJ45 1 GbE|

+|**Status** | Supported, available as a preconfigured appliance|

+The following image shows a view of the Dell PowerEdge R660 front panel:

+The following image shows a view of the Dell PowerEdge R660 back panel:

+## Specifications

+|Component| Technical specifications|

+|:-|:-|

+|Chassis| 1U rack server|

+|Dimensions| Height: 1.68 in / 42.8 mm <br>Width: 18.97 in / 482.0 cm<br>Depth: 23.04 in / 585.3 mm (without bezel) 23.57 in / 598.9 mm (with bezel)|

+|Processor| Intel Xeon E-2434 3.4 GHz <br>8M Cache<br> 4C/8T, Turbo, HT (55 W) DDR5-4800|

+|Memory| 128 GB |

+|Storage| 7.2 TB Hard Drive |

+|Network controller| - PowerEdge R660 Motherboard with Broadcom 5720 Dual Port 1 Gb On-Board LOM, <br>- PCIe Blank Filler, Low Profile. <br>- Intel Ethernet i350 Quad Port 1 GbE BASE-T Adapter, PCIe Low Profile, V2|

+|Management|iDRAC Group Manager, Disabled|

+|Rack support| ReadyRails Sliding Rails With Cable Management Arm|

+## Dell PowerEdge R660 - Bill of Materials

+### Components

+|Quantity|PN| Module| Description|

+|-||-||

+|1| 210-BFUZ | Base | PowerEdge R660xs |

+|1| 461-AAIG | Trusted platform module | Trusted platform module 2.0 V3 |

+|1| 470-AFQI | Chassis configuration | 2.5" Chassis with up to 8 Hard Drives (SAS/SATA), 2 CPU |

+|1| 338-CKVW | Processor | Intel Xeon Silver 4410T 2.7 G 10C/20T, 16 GT/s, 27 M caches, Turbo, HT (150 W) DDR5-4000 |

+|1| 338-CKVW | Additional processor | Intel Xeon Silver 4410T 2.7 G 10C/20T, 16 GT/s, 27 M caches, Turbo, HT (150 W) DDR5-4000 |

+|1| 379-BDCO | Additional processor | Additional processor selected |

+|1| 338-CHQT | Processor thermal configuration | Heatsink for 2 CPU configuration (CPU less than or equal to 150 W)|

+|1| 370-AAIP | Memory configuration type | Performance Optimized |

+|1| 370-AHCL | Memory DIMM type and speed | 4800-MT/s RDIMMs |

+|4| 370-AGZP | Memory capacity | 32 GB RDIMM, 4,800 MT/s dual rank |

+|1| 780-BCDS | RAID configuration | unconfigured RAID |

+|1| 405-AAZB | RAID controller | PERC H755 SAS Front |

+|1| 750-ACFR | RAID controller | Front PERC Mechanical Parts, front load |

+|6| 161-BCBX | Hard drives | 2.4 TB Hard Drive SAS ISE 12 Gbps 10k 512e 2.5in Hot Plug |

+|1| 384-BBBH | BIOS and Advanced System Configuration Settings | Power Saving BIOS Settings |

+|1| 387-BBEY | Advanced System Configurations | No Energy Star |

+|1| 384-BDJC | Fans | Standard Fan X7 |

+|1| 528-CTIC | Embedded Systems Management | iDRAC9, Enterprise 16G |

+|1| 450-AKLF | Power supply | Dual, Redundant(1+1), Hot-Plug Power Supply, 1100 W MM(100-240Vac) Titanium |

+|2| 450-AADY | Power cords | C13 to C14, PDU Style, 10 AMP, 6.5 Feet (2 m), Power Cord |

+|1| 330-BCCE | PCIe Riser | Riser Config 6, Low profile, 1x 16 LP slots (Gen 5) + 1x8 LP Slot (Gen 5), 2 CPU |

+|1| 384-BDKV | Motherboard | PowerEdge R660xs Motherboard with Broadcom 5720 Dual Port 1 Gb On-Board LOM |

+|1| 540-BCOB | Network daughter card | Broadcom 5720 Quad Port 1 GbE BASE-T Adapter, OCP NIC 3.0 |

+|1| 350-BCEL | Quick sync | Quick Sync 2 (At-the-box mgmt) |

+|1| 379-BCSF | Password | iDRAC, Factory Generated Password |

+|1| 379-BCQX | IDRAC service module | iDRAC Service Module (ISM), NOT Installed |

+|1| 379-BCQV | Group manager | iDRAC group manager, Enabled |

+|1| 325-BEVH | Bezel | PowerEdge 1U Standard Bezel |

+|1| 350-BEUF | Bezel | Dell Luggage Tag, 0/6/8/10 |

+|1| 770-BCJI | Rack rails | A11 drop-in/stab-in Combo Rails Without Cable Management Arm |

+|1| 340-DLRR | Shipping | PowerEdge R660XS Shipping EMEA1 (English/French/German/Spanish/Russian/Hebrew) |

+|1| 340-DFKP | Shipping material | PowerEdge R660xs, 8x2.5, Short Drive Shipping Material |

+|1| 389-FBMD | Regulatory |PowerEdge R660xs HS5610 Label, CE and CCC Marking, for below 1,300 W PSU |

+|1| 683-11870 | Dell

+### Software

+|Quantity|PN| Module| Description|

+|-||-||

+|1| 800-BBDM | Advanced system configuration | UEFI BIOS Boot Mode with GPT Partition |

+|1| 528-COYT | Embedded Systems Management | Secured Component Verification |

+|1| 611-BBBF | Operating system | No operating system |

+|1| 605-BBFN | OS media kits | No media required |

+|1| 631-AACK | System documentation | No Systems Documentation, No OpenManage DVD Kit |

+### Service

+|Quantity|PN| Module| Description|

+|-||-||

+|1| 293-10049 | Shipping Box Labels - Standard | Order Configuration Shipbox Label (Ship Date, Model, Processor Speed, HDD Size, RAM) |

+|1| 865-BBLL | Dell

+|1| 865-BBLM | Dell

+|1| 709-BBIX | Dell

+## Install Defender for IoT software on the DELL R660

+This procedure describes how to install Defender for IoT software on the Dell R660.

+The installation process takes about 20 minutes. During the installation, the system restarts several times.

+To install Defender for IoT software:

+1. Connect the screen and keyboard to the appliance, and then connect to the CLI.

+1. Connect an external CD or disk-on-key that contains the software you downloaded from the Azure portal.

+1. Start the appliance.

+1. Continue with the generic procedure for installing Defender for IoT software. For more information, see [Defender for IoT software installation](../how-to-install-software.md).

+## Next steps

+Continue learning about the system requirements for physical or virtual appliances. For more information, see [Which appliances do I need?](../ot-appliance-sizing.md).

+Then, use any of the following procedures to continue:

+- [Download software for an OT sensor](../ot-deploy/install-software-ot-sensor.md#download-software-files-from-the-azure-portal)

+- [Download software files for an on-premises management console](../legacy-central-management/install-software-on-premises-management-console.md#download-software-files-from-the-azure-portal)

+In this article, you learn how to build custom Azure Resource Manager (ARM) and Bicep container images to deploy your [environment definitions](configure-environment-definition.md) in Azure Deployment Environments (ADE).

+The ADE extensibility model enables you to create custom container images to use with your environment definitions. By using the extensibility model, you can create your own custom container images, and store them in a container registry like Azure Container Registry (ACR) or Docker Hub. You can then reference these images in your environment definitions to deploy your environments.

+You must build your Docker image and push it to your container registry to make it available for use in ADE.

+You can build your image using the Docker CLI, or by using a script provided by ADE.

+In order to use custom images, you need to store them in a container registry. Azure Container Instances (ACR) is highly recommended for that. Due to its tight integration with ADE, the image can be published without allowing public anonymous pull access.

+It's also possible to store the image in a different container registry such as Docker Hub, but in that case it needs to be publicly accessible.

+> [!Caution]

+> Enabling anonymous (unauthenticated) pull access makes all registry content publicly available for read (pull) actions.

+To use a custom image stored in ACR, you need to ensure that ADE has appropriate permissions to access your image. Anonymous pull access is disabled by default in ACR.

+#### Use a public registry with anonymous pull

+#### Use ACR with secured access

+By default, access to pull or push content from an Azure Container Registry is only available to authenticated users. You can further secure access to ACR by limiting access from certain networks and assigning specific roles.

+##### Limit network access

+To secure network access to your ACR, you can limit access to your own networks, or disable public network access entirely. If you limit network access, you must enable the firewall exception *Allow trusted Microsoft services to access this container registry*.

+To disable access from public networks:

+1. [Create an ACR instance](/azure/container-registry/container-registry-get-started-azure-cli) or use an existing one.

+1. In the Azure portal, go to the ACR that you want to configure.

+1. On the left menu, under **Settings**, select **Networking**.

+1. On the Networking page, on the **Public access** tab, under **Public network access**, select **Disabled**.

+ :::image type="content" source="media/how-to-configure-extensibility-bicep-container-image/container-registry-network-settings.png" alt-text="Screenshot of the Azure portal, showing the ACR network settings, with Public access and Disabled highlighted.":::

+1. Under **Firewall exception**, check that **Allow trusted Microsoft services to access this container registry** is selected, and then select **Save**.

+ :::image type="content" source="media/how-to-configure-extensibility-bicep-container-image/container-registry-network-disable-public.png" alt-text="Screenshot of the ACR network settings, with Allow trusted Microsoft services to access this container registry and Save highlighted.":::

+##### Assign the AcrPull role

+Creating environments by using container images uses the ADE infrastructure, including projects and environment types. Each project has one or more project environment types, which need read access to the container image that defines the environment to be deployed. To access the images within your ACR securely, assign the AcrPull role to each project environment type.

+To assign the AcrPull role to the Project Environment Type:

+1. In the Azure portal, go to the ACR that you want to configure.

+1. On the left menu, select **Access Control (IAM)**.

+1. Select **Add** > **Add role assignment**.

+1. Assign the following role. For detailed steps, see [Assign Azure roles using the Azure portal](../role-based-access-control/role-assignments-portal.yml).

+ | Setting | Value |

+ | | |

+ | **Role** | Select **AcrPull**. |

+ | **Assign access to** | Select **User, group, or service principal**. |

+ | **Members** | Enter the name of the project environment type that needs to access the image in the container. |

+ The project environment type displays like the following example:

+ :::image type="content" source="media/how-to-configure-extensibility-bicep-container-image/container-registry-access-control.png" alt-text="Screenshot of the Select members pane, showing a list of project environment types with part of the name highlighted.":::

+In this configuration, ADE uses the Managed Identity for the PET, whether system assigned or user assigned.

+> [!Tip]

+> This role assignment has to be made for every project environment type. It can be automated through the Azure CLI.

+When you're ready to push your image to your registry, run the following command:

+```docker

+docker push {YOUR_REGISTRY}.azurecr.io/{YOUR_IMAGE_LOCATION}:{YOUR_TAG}

+```

+The Kafka compression for Event Hubs is only supported in Premium and Dedicated tiers currently.

+### Release date: March 20, 2024

+**This release applies to the following**

+- Cluster Pool Version: 1.1

+- Cluster Version: 1.1.1

+- AKS version: 1.27

+### New Features

+**Apache Flink Application Mode Cluster**

+Application mode clusters are designed to support dedicated resources for large and long-running jobs. When you have resource-intensive or extensive data processing tasks, you can use the [Application Mode Cluster](https://flink.apache.org/2020/07/14/application-deployment-in-flink-current-state-and-the-new-application-mode/#application-mode). This mode allows you to allocate dedicated resources for specific Apache Flink applications, ensuring that they have the necessary computing power and memory to handle large workloads effectively.

+For more information, see [Apache Flink Application Mode cluster on HDInsight on AKS](../flink/application-mode-cluster-on-hdinsight-on-aks.md).

+**Private Clusters for HDInsight on AKS**

+With private clusters, and outbound cluster settings you can now control ingress and egress traffic from HDInsight on AKS cluster pools and clusters.

+- Use Azure Firewall or Network Security Groups (NSGs) to control the egress traffic, when you opt to use outbound cluster pool with load balancer.

+- Use Outbound cluster pool with User defined routing to control egress traffic at the subnet level.

+- Use Private AKS cluster feature - To ensure AKS control plane, or API server has internal IP addresses. The network traffic between AKS Control plane / API server and HDInsight on AKS node pools (clusters) remains on the private network only.

+- Avoid creating public IPs for the cluster. Use private ingress feature on your clusters.

+For more information, see [Control network traffic from HDInsight on AKS Cluster pools and cluster](../control-egress-traffic-from-hdinsight-on-aks-clusters.md).

+**In place Upgrade**

+Upgrade your clusters and cluster pools with the latest software updates. This means that you can enjoy the latest cluster package hotfixes, security updates, and AKS patches, without recreating clusters. For more information, see [Upgrade your HDInsight on AKS clusters and cluster pools](../in-place-upgrade.md).

+> [!IMPORTANT]

+> To take benefit of all these **latest features**, you are required to create a new cluster pool with 1.1 and cluster version 1.1.1.

+### Known issues

+- **Workload identity limitation:**

+ - There's a known [limitation](/azure/aks/workload-identity-overview#limitations) when transitioning to workload identity. This limitation is due to the permission-sensitive nature of FIC operations. Users can't perform deletion of a cluster by deleting the resource group. Cluster deletion requests must be triggered by the application/user/principal with FIC/delete permissions. In case, the FIC deletion fails, the high-level cluster deletion also fails.

+ - **User Assigned Managed Identities (UAMI)** support ΓÇô There's a limit of 20 FICs per UAMI. You can only create 20 Federated Credentials on an identity. In HDInsight on AKS cluster, FIC (Federated Identity Credential) and SA have one-to-one mapping and only 20 SAs can be created against an MSI. If you want to create more clusters, then you are required to provide different MSIs to overcome the limitation.

+ - Creation of federated identity credentials is currently not supported on user-assigned managed identities created in [these regions](/entra/workload-id/workload-identity-federation-considerations#unsupported-regions-user-assigned-managed-identities)

+

+### Operating System version

+- Mariner OS 2.0

+**Workload versions**

+|Workload|Version|

+| -- | -- |

+|Trino | 426 |

+|Flink | 1.17.0 |

+|Apache Spark | 3.3.1 |

+**Supported Java and Scala versions**

+|Workload |Java|Scala|

+| -- | -- | -- |

+|Trino |Open JDK 17.0.7  |- |

+|Flink |Open JDK 11.0.21 |2.12.7 |

+|Spark |Open JDK 1.8.0_345  |2.12.15 |

+The preview is available in the following [regions](../overview.md#region-availability-public-preview).

+If you have any more questions, contact [Azure Support](https://ms.portal.azure.com/#view/Microsoft_Azure_Support/HelpAndSupportBlade/~/overview) or refer to the [Support options](../hdinsight-aks-support-help.md) page. If you have product specific feedback, write us on [aka.ms/askhdinsight](https://forms.office.com/pages/responsepage.aspx?id=v4j5cvGGr0GRqy180BHbR6HHTBN7UDpEhLm8BJmDhGJURDhLWEhBVE5QN0FQRUpHWDg4ODlZSDA4RCQlQCN0PWcu).

+### Release date: Aug 05, 2024

+- Cluster Pool Version: 1.2

+- Cluster Version: 1.2.1

+**MSI based SQL authentication**

+Users can now authenticate external Azure SQL DB Metastore with MSI instead of User ID password authentication. This feature helps to further secure the cluster connection with Metastore.

+**Configurable VM SKUs for Head node, SSH node**

+This functionality allows users to choose specific SKUs for head nodes, worker nodes, and SSH nodes, offering the flexibility to select according to the use case and the potential to lower total cost of ownership (TCO).

+**Multiple MSI in cluster**

+Users can configure multiple MSI for cluster admins operations and for job related resource access. This feature allows users to demarcate and control the access to the cluster and data lying in the storage account.

+For example, one MSI for access to data in storage account and dedicated MSI for cluster operations.

+### Updated

+**Script action**

+Script Action now can be added with Sudo user permission. Users can now install multiple dependencies including custom jars to customize the clusters as required.

+**Library Management**

+Maven repository shortcut feature added to the Library Management in this release. User can now install Maven dependencies directly from the open-source repositories.

+**Spark 3.4**

+Spark 3.4 update brings a range of new features includes

+* API enhancements

+* Structured streaming improvements

+* Improved usability and developer experience

+> To take benefit of all these **latest features**, you are required to create a new cluster pool with 1.2 and cluster version 1.2.1

+|Trino | 440 |

+|Apache Spark | 3.4 |

+|Trino |Open JDK 21.0.2  |- |

+> [!Important]

+> On 31 August 2024, Azure is retiring the Classic Azure Monitor experience on HDInsight.

+## **July 2024**

+### FHIR service

+**Bug Fixes**

+**Fixed: Exporting Data as SMART User**

+Exporting data as a SMART user no longer requires write scopes. Previously, it was necessary to grant "write" privileges to a SMART user for exporting data, which implied higher privilege levels. To initiate an export job as a SMART user, ensure the user is a member of the FHIR export role in RBAC and requests the "read" SMART clinical scope.

+**Fixed: Updating Status Code from HTTP 500 to HTTP 400**

+During a patch operation, if the payload requested an update for a resource type other than Parameter, an internal server error (HTTP 500) was initially thrown. This has been updated to throw an HTTP 400 error instead.

+| `allowNegativeVersions`| Allows FHIR server assigning negative versions for resource records with explicit lastUpdated value and no version specified when input does not fit in contiguous space of positive versions existing in the store. | 0..1 | To enable this feature pass true. By default it is false. |

+ "valueString": "<Use "InitialLoad" for initial mode import / Use "IncrementalLoad" for incremental mode import>"

+ },

+ {

+ "name": "allowNegativeVersions",

+ "valueBoolean": true

+#### Import Operation enhancement

+The FHIR service now allows ingestion of data without specifying a version at the resource level. The order of resources is maintained using the lastUpdated value. This enhancement introduces the "allowNegativeVersions" flag. Setting flag true allows the FHIR service to assign negative versions for resource records with an explicit lastUpdated value and no version specified.

+#### Bug Fixes

+- **Fixed inclusion of soft deleted resources when using _security:not search parameter**

+When using the _security:not search parameter in search operations, IDs for soft-deleted resources were being included in the search results. We have fixed the issue so that soft-deleted resources are now excluded from search results.

+- **Exporting Data as SMART User**

+Exporting data as a SMART user no longer requires write scopes. Previously, it was necessary to grant "write" privileges to a SMART user for exporting data, which implied higher privilege levels. To initiate an export job as a SMART user, ensure the user is a member of the FHIR export role in RBAC and requests the "read" SMART clinical scope.

+Updating Status Code from HTTP 500 to HTTP 400

+- **Updating Status Code from HTTP 500 to HTTP 400**

+During a patch operation, if the payload requested an update for a resource type other than parameter, an internal server error (HTTP 500) was initially thrown. This has been updated to throw an HTTP 400 error instead.

+#### Performance enhancement

+Query optimization is added when searching FHIR resources with a data range. This query optimization will help with efficient querying as one combined CTE is generated.

+* HTTPS probes doesn't support mutual authentication with a client certificate.

+* HTTP probes doesn't support using hostnames to probes backends

+* HTTP probes doesn't support probing on the following ports due to security concerns: 19, 21, 25, 70, 110, 119, 143, 220, 993.

+* A model to package. This example, uses an MLflow model registered in the workspace.

+ > [!CAUTION]

+ > Model packaging is not supported for models in the Azure AI model catalog, including large language models. Models in the Azure AI model catalog are optimized for inference on Azure AI deployment targets and are not suitable for packaging.

+> [!NOTE]

+> Although you can retrieve scoring URI, swagger URI and other information from Azure Machine Learning studio (UI), using Test tab on Azure Machine Learning studio isn't supported for Azure Container Instance or Azure Kubernetes Service based web services. Instead, use code based approach to consume the web service as described in the later section of this article. To fully utilize Test tab to test the deployments, consider [migrating to v2 Managed online endpoint](../migrate-to-v2-deploy-endpoints.md). For more, see [endpoints for inferencing](../concept-endpoints.md).

+ Title: Train models with the Azure Machine Learning Python SDK (v1)

+1. **Attach**: Attach the DSVM to your workspace [using Azure Machine Learning studio](../how-to-create-attach-compute-studio.md#other-compute-targets).

+1. **Attach**: Attach the HDInsight cluster to your workspace [using Azure Machine Learning studio](../how-to-create-attach-compute-studio.md#other-compute-targets).

+Azure Modeling and Simulation Workbench is a secure, on-demand service that provides a fully managed engineering design and simulation environment for safe and efficient user collaboration. The service incorporates many infrastructure services required to build a successful environment for engineering development, such as: workload specific VMs, scheduler, orchestration, license server, remote connectivity, high performance storage, network configurations, security, and access controls.

+## August 2024

+- **Major version upgrade support for Burstable compute tier**

+

+ Azure Database for MySQL now offers major version upgrades for Burstable SKU compute tiers. This support automatically upgrades the compute tier to General Purpose SKU before performing the upgrade, ensuring sufficient resources. Customers can choose to revert back to Burstable SKU after the upgrade. Additional costs may apply. [Learn more](how-to-upgrade.md#perform-a-planned-major-version-upgrade-from-mysql-57-to-mysql-80-using-the-azure-portal-for-burstable-sku-servers)

+

+Azure Operator Nexus is built on basic constructs like compute servers, storage appliances, and network fabric devices. These compute servers, also called bare-metal machines (BMMs), represent the physical machines on the rack. They run the Azure Linux (formerly CBL-Mariner) operating system and provide closed integration support for high-performance workloads.

+Through the strategic placement of software components and workloads in a NUMA-aware way, Operators can enhance the performance of network functions, such as virtualized routers and firewalls. This placement leads to improved service delivery and responsiveness in their cloud environments.

+Huge page usage in workloads refers to the utilization of large memory pages, typically 2 MB or 1 GB in size, instead of the standard 4-KB pages. This approach helps reduce memory overhead and improves the overall system performance. It reduces the translation look-aside buffer (TLB) miss rate and improves memory access efficiency.

+Workloads that involve large data sets or intensive memory operations, such as network packet processing, can benefit from huge page usage because it enhances memory performance and reduces memory-related bottlenecks. As a result, users see improved throughput and reduced latency.

+### Azure Linux

+Azure Operator Nexus runs Microsoft's own Linux distribution called [Azure Linux (formerly CBL-Mariner)](https://github.com/microsoft/azurelinux) on the bare-metal hosts in the operator's facilities. The same Linux distribution supports Azure cloud infrastructure and edge services. It includes a small set of core packages by default.

+Azure Linux is a lightweight operating system. It consumes limited system resources and is engineered to be efficient. For example, it has a fast startup time with a small footprint and locked-down packages to reduce the threat landscape.

+The reliable and distributed cloud services model supports the operators' network functions. Operators can interact with Azure Operator Nexus to provision the network fabric via zero-touch provisioning (ZTP). They can also perform complex network implementations via a workflow-driven API model.

+In a near-edge environment (also known as an instance), the compute servers (also known as bare-metal machines) represent the physical machines on the rack. They run the Azure Linux operating system and provide support for running high-performance workloads.

+| 1.28.0 | 5 | Calico v3.27.3<br>metrics-server v0.7.1<br>Multus v4.0.0<br>azure-arc-servers v1.1.0<br>CoreDNS v1.9.4<br>etcd v3.5.13<br>sriov-dp v3.11.0-68<br>Csi-nfs v4.7.0<br>csi-volume v0.1.0 | Azure Linux 2.0 | No breaking changes | Beginning with this version bundle, volume orchestration connectivity is TLS encrypted |

+| 1.28.0 | 4 | Calico v3.27.2<br>metrics-server v0.6.3<br>Multus v3.8.0<br>azure-arc-servers v1.0.0<br>CoreDNS v1.9.3<br>etcd v3.5.6-5<br>sriov-dp v3.10.0-60<br>Csi-nfs v4.6.0 | Azure Linux 2.0 | No breaking changes | |

+| 1.27.9 | 1 | Calico v3.27.3<br>metrics-server v0.7.1<br>Multus v4.0.0<br>azure-arc-servers v1.1.0<br>CoreDNS v1.9.4<br>etcd v3.5.13<br>sriov-dp v3.11.0-68<br>Csi-nfs v4.7.0<br>csi-volume v0.1.0 | Azure Linux 2.0 | No breaking changes | Beginning with this version bundle, volume orchestration connectivity is TLS encrypted and cluster nodes are Azure Arc-enabled |

+| 1.27.3 | 5 | Calico v3.27.3<br>metrics-server v0.7.1<br>Multus v4.0.0<br>azure-arc-servers v1.1.0<br>CoreDNS v1.9.4<br>etcd v3.5.13<br>sriov-dp v3.11.0-68<br>Csi-nfs v4.7.0<br>csi-volume v0.1.0 | Azure Linux 2.0 | No breaking changes | Beginning with this version bundle, volume orchestration connectivity is TLS encrypted |

+| 1.27.3 | 4 | Calico v3.27.2<br>metrics-server v0.6.3<br>Multus v3.8.0<br>azure-arc-servers v1.0.0<br>CoreDNS v1.9.3<br>etcd v3.5.6-5<br>sriov-dp v3.10.0-60<br>Csi-nfs v4.6.0 | Azure Linux 2.0 | No breaking changes | |

+| 1.26.12 | 1 | Calico v3.27.3<br>metrics-server v0.7.1<br>Multus v4.0.0<br>azure-arc-servers v1.1.0<br>CoreDNS v1.9.4<br>etcd v3.5.13<br>sriov-dp v3.11.0-68<br>Csi-nfs v4.7.0<br>csi-volume v0.1.0 | Azure Linux 2.0 | No breaking changes | Beginning with this version bundle, volume orchestration connectivity is TLS encrypted and cluster nodes are Azure Arc-enabled |

+| 1.26.6 | 5 | Calico v3.27.3<br>metrics-server v0.7.1<br>Multus v4.0.0<br>azure-arc-servers v1.1.0<br>CoreDNS v1.9.4<br>etcd v3.5.13<br>sriov-dp v3.11.0-68<br>Csi-nfs v4.7.0<br>csi-volume v0.1.0 | Azure Linux 2.0 | No breaking changes | Beginning with this version bundle, volume orchestration connectivity is TLS encrypted |

+| 1.26.6 | 4 | Calico v3.27.2<br>metrics-server v0.6.3<br>Multus v3.8.0<br>azure-arc-servers v1.0.0<br>CoreDNS v1.9.3<br>etcd v3.5.6-5<br>sriov-dp v3.10.0-60<br>Csi-nfs v4.6.0 | Azure Linux 2.0 | No breaking changes | |

+| 1.25.11 | 5 | Calico v3.27.3<br>metrics-server v0.7.1<br>Multus v4.0.0<br>azure-arc-servers v1.1.0<br>CoreDNS v1.9.4<br>etcd v3.5.13<br>sriov-dp v3.11.0-68<br>Csi-nfs v4.7.0<br>csi-volume v0.1.0 | Azure Linux 2.0 | No breaking changes | Beginning with this version bundle, volume orchestration connectivity is TLS encrypted |

+| 1.25.11 | 4 | Calico v3.27.2<br>metrics-server v0.6.3<br>Multus v3.8.0<br>azure-arc-servers v1.0.0<br>CoreDNS v1.9.3<br>etcd v3.5.6-5<br>sriov-dp v3.10.0-60<br>Csi-nfs v4.6.0 | Azure Linux 2.0 | No breaking changes | |

+| 1.25.6 | 7 |Calico v3.27.3<br>metrics-server v0.7.1<br>Multus v4.0.0<br>azure-arc-servers v1.1.0<br>CoreDNS v1.9.4<br>etcd v3.5.13<br>sriov-dp v3.11.0-68<br>Csi-nfs v4.7.0<br>csi-volume v0.1.0 | Azure Linux 2.0 | No breaking changes | Beginning with this version bundle, volume orchestration connectivity is TLS encrypted |

+| 1.25.6 | 6 | Calico v3.27.2<br>metrics-server v0.6.3<br>Multus v3.8.0<br>azure-arc-servers v1.0.0<br>CoreDNS v1.9.3<br>etcd v3.5.6-5<br>sriov-dp v3.10.0-60<br>Csi-nfs v4.6.0 | Azure Linux 2.0 | No breaking changes | |

+This article describes how to troubleshoot a failed server hardware validation. Hardware validation (HWV) is run as part of cluster deploy action and a bare metal replace action. HWV validates a bare metal machine (BMM) by executing test cases against the baseboard management controller (BMC). The Azure Operator Nexus platform is deployed on Dell servers. Dell servers use the integrated Dell remote access controller (iDRAC) which is the equivalent of a BMC.

+1. Collect the following information:

+ - Subscription ID

+ - Cluster name

+ - Resource group

+2. Request access to the Cluster's Log Analytics Workspace (LAW).

+3. Access to BMC webui and/or jumpbox that allows running of racadm utility.

+4. Hardware validation results can be fetched with a query against the `HWVal_CL` table as per the following example

+* Memory/RAM Related Failure (memory_capacity_GB)

+ * Memory specs are defined in the SKU. Memory below threshold value indicates missing or failed Dual In-Line Memory Module (DIMM). A failed DIMM would also be reflected in the `health_info` category. The following example shows a failed memory check.

+ * To check memory information in BMC webui:

+ `BMC` -> `System` -> `Memory`

+ * To check memory information with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD hwinventory | grep SysMemTotalSize

+ ```

+ * To troubleshoot a memory problem engage vendor.

+ * CPU specs are defined in the SKU. Failed `cpu_sockets` check indicates a failed CPU or CPU count mismatch. The following example shows a failed CPU check.

+ * To check CPU information in BMC webui:

+ `BMC` -> `System` -> `CPU`

+ * To check CPU information with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD hwinventory | grep PopulatedCPUSockets

+ ```

+ * To troubleshoot a CPU problem engage vendor.

+ * Failed `Model` check indicates that wrong server is racked in the slot or there's a cabling mismatch. The following example shows a failed model check.

+ * To check model information in BMC webui:

+ `BMC` -> `Dashboard` - Shows Model

+ * To check model information with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD getsysinfo | grep Model

+ ```

+ * To troubleshoot this problem, ensure that server is racked in the correct location, cabled accordingly, and that the correct IP is assigned.

+* Serial Number Check Failure (Serial_Number)

+ * The server's serial number, also referred as the service tag, is defined in the cluster. Failed `Serial_Number` check indicates a mismatch between the serial number in the cluster and the actual serial number of the machine. The following example shows a failed serial number check.

+ ```json

+ {

+ "field_name": "Serial_Number",

+ "comparison_result": "Fail",

+ "expected": "1234567",

+ "fetched": "7654321"

+ }

+ ```

+ * To check serial number information in BMC webui:

+ `BMC` -> `Dashboard` - Shows Service Tag

+ * To check serial number information with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD getsysinfo | grep "Service Tag"

+ ```

+ * To troubleshoot this problem, ensure that server is racked in the correct location, cabled accordingly, and that the correct IP is assigned.

+* iDRAC License Check Failure

+ * All iDRACs require a perpetual/production iDRAC datacenter or enterprise license. Trial licenses are valid for only 30 days. A failed `iDRAC License Check` indicates that the required iDRAC license is missing. The following examples show a failed iDRAC license check for a trial license and missing license respectively.

+ ```json

+ {

+ "field_name": "iDRAC License Check",

+ "comparison_result": "Fail",

+ "expected": "idrac9 x5 datacenter license or idrac9 x5 enterprise license - perpetual or production",

+ "fetched": "iDRAC9 x5 Datacenter Trial License - Trial"

+ }

+ ```

+ ```json

+ {

+ "field_name": "iDRAC License Check",

+ "comparison_result": "Fail",

+ "expected": "idrac9 x5 datacenter license or idrac9 x5 enterprise license - perpetual or production",

+ "fetched": ""

+ }

+ ```

+ * To troubleshoot this problem engage vendor to obtain the correct license. Apply the license using the iDRAC webui in the following location:

+ `BMC` -> `Configuration` -> `Licenses`

+* Firmware Version Checks

+ * Firmware version checks were introduced in release 3.9. The following example shows the expected log for release versions before 3.9.

+ ```json

+ {

+ "system_info": {

+ "system_info_result": "Pass",

+ "result_log": [

+ "Firmware validation not supported in release 3.8"

+ ]

+ },

+ }

+ ```

+ * Firmware versions are determined based on the `cluster version` value in the cluster object. The following example shows a failed check due to indeterminate cluster version. If this problem is encountered, verify the version in the cluster object.

+ ```json

+ {

+ "system_info": {

+ "system_info_result": "Fail",

+ "result_log": [

+ "Unable to determine firmware release"

+ ]

+ },

+ }

+ ```

+ * Drive specs are defined in the SKU. Mismatched capacity values indicate incorrect drives or drives inserted in to incorrect slots. Missing capacity and type fetched values indicate drives that are failed, missing, or inserted in to incorrect slots.

+ * To check disk information in BMC webui:

+ `BMC` -> `Storage` -> `Physical Disks`

+ * To check disk information with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD raid get pdisks -o -p State,Size

+ ```

+ * To troubleshoot, ensure that disks are inserted in the correct slots. If the problem persists engage vendor.

+* Network Interface Cards (NIC) Check Failure

+ * Dell server NIC specs are defined in the SKU. A mismatched link status indicates loose or faulty cabling or crossed cables. A mismatched model indicates incorrect NIC card is inserted in to slot. Missing link/model fetched values indicate NICs that are failed, missing, or inserted in to incorrect slots.

+ * To check NIC information in BMC webui:

+ `BMC` -> `System` -> `Network Devices`

+ * To check all NIC information with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD hwinventory NIC

+ ```

+ * To check a specific NIC with racadm provide the Fully Qualified Device Descriptor (FQDD):

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD hwinventory NIC.Embedded.1-1-1

+ ```

+ * To troubleshoot, ensure that servers are cabled correctly and that ports are linked up. Bounce port on the fabric. Perform flea drain. If the problem persists engage vendor.

+ * HWV reports L2 switch information for each of the server interfaces. The switch connection ID (switch interface MAC) and switch port connection ID (switch interface label) are informational.

+ "comparison_result": "Info",

+ "fetched": "c0:d6:82:23:0c:7d"

+ "comparison_result": "Info",

+ "fetched": "Ethernet10/1"

+* Cabling Checks for Bonded Interfaces

+ * Mismatched cabling is reported in the result_log. Cable check validates that that bonded NICs connect to switch ports with same Port ID. In the following example Peripheral Component Interconnect (PCI) 3/1 and 3/2 connect to "Ethernet1/1" and "Ethernet1/3" respectively on TOR, triggering a failure for HWV.

+ "Cabling problem detected on PCI Slot 3 - server NIC.Slot.3-1-1 connected to switch Ethernet1/1 - server NIC.Slot.3-2-1 connected to switch Ethernet1/3"

+ * To fix the issue insert cables in to the correct interfaces.

+* iDRAC (BMC) MAC Address Check Failure

+ * The iDRAC MAC address is defined in the cluster for each BMM. A failed `iDRAC_MAC` check indicates a mismatch between the iDRAC/BMC MAC in the cluster and the actual MAC address retrieved from the machine.

+ ```json

+ {

+ "field_name": "iDRAC_MAC",

+ "comparison_result": "Fail",

+ "expected": "aa:bb:cc:dd:ee:ff",

+ "fetched": "aa:bb:cc:dd:ee:gg"

+ }

+ ```

+ * To troubleshoot this problem, ensure that correct MAC address is defined in the cluster. If MAC is correct in the cluster object, attempt a flea drain. If problem persists ensure that server is racked in the correct location, cabled accordingly, and that the correct IP is assigned.

+* Preboot execution environment (PXE) MAC Address Check Failure

+ * The PXE MAC address is defined in the cluster for each BMM. A failed `PXE_MAC` check indicates a mismatch between the PXE MAC in the cluster and the actual MAC address retrieved from the machine.

+ ```json

+ {

+ "field_name": "NIC.Embedded.1-1_PXE_MAC",

+ "comparison_result": "Fail",

+ "expected": "aa:bb:cc:dd:ee:ff",

+ "fetched": "aa:bb:cc:dd:ee:gg"

+ }

+ ```

+ * To troubleshoot this problem, ensure that correct MAC address is defined in the cluster. If MAC is correct in the cluster object, attempt a flea drain. If problem persists ensure that server is racked in the correct location, cabled accordingly, and that the correct IP is assigned.

+ * Server health checks cover various hardware component sensors. A failed health sensor indicates a problem with the corresponding hardware component. The following examples indicate fan, drive, and CPU failures respectively.

+ * To check health information in BMC webui:

+ `BMC` -> `Dashboard` - Shows Health Information

+ * To check health information with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD getsensorinfo

+ ```

+ * To troubleshoot a server health failure engage vendor.

+* Health Check LifeCycle (LC) Log Failures

+ * Dell server health checks fail for recent Critical LC Log Alarms. The hardware validation plugin logs the alarm ID, name, and timestamp. Recent LC Log critical alarms indicate need for further investigation. The following example shows a failure for a critical backplane voltage alarm.

+ ```json

+ {

+ "field_name": "LCLog_Critical_Alarms",

+ "comparison_result": "Fail",

+ "expected": "No Critical Errors",

+ "fetched": "53539 2023-07-22T23:44:06-05:00 The system board BP1 PG voltage is outside of range."

+ }

+ ```

+ * Virtual disk errors typically indicate a RAID cleanup false positive condition and are logged due to the timing of raid cleanup and system power off pre HWV. The following example shows an LC log critical error on virtual disk 238. If multiple errors are encountered blocking deployment, delete cluster, wait two hours, then reattempt cluster deployment. If the failures aren't deployment blocking, wait two hours then run BMM replace.

+ "comparison_result": "Fail",

+ "fetched": "104473 2024-07-26T16:05:19-05:00 Virtual Disk 238 on RAID Controller in SL 3 has failed."

+ * To check LC logs in BMC webui:

+ `BMC` -> `Maintenance` -> `Lifecycle Log`

+ * To check LC log critical alarms with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD lclog view -s critical

+ ```

+ * If `Backplane Comm` critical errors are logged, perform flea drain. Engage vendor to troubleshoot any other LC log critical failures.

+ * Dell server health checks fail for failed server power-up or failed iDRAC reset. A failed server control action indicates an underlying hardware issue. The following example shows failed power on attempt.

+ "comparison_result": "Fail",

+ "fetched": "Failed"

+ * To power server on in BMC webui:

+ `BMC` -> `Dashboard` -> `Power On System`

+ * To power server on with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD serveraction powerup

+ ```

+ * To troubleshoot server power-on failure attempt a flea drain. If problem persists engage vendor.

+ * Dell server health checks warn when one power supply is missing or failed. Power supply "field_name" might be displayed as 0/PS0/Power Supply 0 and 1/PS1/Power Supply 1 for the first and second power supplies respectively. A failure of one power supply doesn't trigger an HWV device failure.

+ "comparison_result": "Warning",

+ "fetched": "UnavailableOffline-Critical"

+ "comparison_result": "Warning",

+ "fetched": "Enabled-Critical"

+ * To check power supplies in BMC webui:

+ `BMC` -> `System` -> `Power`

+ * To check power supplies with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD getsensorinfo | grep PS

+ ```

+ * Reseating the power supply might fix the problem. If alarms persist engage vendor.

+* Boot Device Name Check Considerations

+ "field_name": "boot_device_name",

+ "fetched": "NIC.PxeDevice.1-1"

+ "comparison_result": "Fail",

+ "fetched": "NIC.Embedded.1-2-1"

+ "comparison_result": "Fail",

+ "fetched": "Disabled"

+ }

+ ```

+ * To update the PXE device state and name in BMC webui, set the value then select `Apply` followed by `Apply And Reboot`:

+ `BMC` -> `Configuration` -> `BIOS Settings` -> `Network Settings` -> `PXE Device1` -> `Enabled`

+ `BMC` -> `Configuration` -> `BIOS Settings` -> `Network Settings` -> `PXE Device1 Settings` -> `Interface` -> `Embedded NIC 1 Port 1 Partition 1`

+

+ * To update the PXE device state and name with racadm run the following commands:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD set bios.NetworkSettings.PxeDev1EnDis Enabled

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD set bios.PxeDev1Settings.PxeDev1Interface NIC.Embedded.1-1-1

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD jobqueue create BIOS.Setup.1-1

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD serveraction powercycle

+ ```

+### Device login check

+* Device Login Check Considerations

+ * The `device_login` check fails if the iDRAC isn't accessible or if the hardware validation plugin isn't able to sign-in.

+ ```json

+ {

+ "device_login": "Fail"

+ * To set password in BMC webui:

+ `BMC` -> `iDRAC Settings` -> `Users` -> `Local Users` -> `Edit`

+ * To set password with racadm:

+ ```bash

+ racadm -r $BMC_IP -u $BMC_USER -p $CURRENT_PASSWORD set iDRAC.Users.2.Password $BMC_PWD

+ ```

+ * To troubleshoot, ping the iDRAC from a jumpbox with access to the BMC network. If iDRAC pings check that passwords match.

+### Special considerations

+* Servers Failing Multiple Health and Network Checks

+ * Raid deletion is performed during cluster deploy and cluster delete actions for all releases inclusive of 3.12.

+ * If we observe servers getting powered off during hardware validation with multiple failed health and network checks, we need to reattempt cluster deployment.

+ * If issues persist, raid deletion needs to be performed manually on `control` nodes in the cluster.

+ * To clear raid in BMC webui:

+ `BMC` -> `Storage` -> `Virtual Disks` -> `Action` -> `Delete` -> `Apply Now`

+ * To clear raid with racadm:

+ ```bash

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD raid deletevd:Disk.Virtual.239:RAID.SL.3-1

+ racadm --nocertwarn -r $IP -u $BMC_USR -p $BMC_PWD jobqueue create RAID.SL.3-1 --realtime

+ ```

+Azure maintains the largest compliance portfolio in the industry. For details, see [Microsoft Azure Compliance Offerings](/compliance/regulatory/offering-home). Each offering description provides an up to-date-scope statement and links to useful downloadable resources.

+## Release: Aug 2024

+* General availability of [Database Size Metrics](./concepts-monitoring.md) for Azure Database for PostgreSQL flexible server.

+>| Azure Cosmos DB (Microsoft.DocumentDB/databaseAccounts) | Cassandra | privatelink.cassandra.cosmos.azure.us | cassandra.cosmos.azure.us |

+>| Azure Cosmos DB (Microsoft.DocumentDB/databaseAccounts) | Gremlin | privatelink.gremlin.cosmos.azure.us | gremlin.cosmos.azure.us |

+>| Azure Cosmos DB (Microsoft.DocumentDB/databaseAccounts) | Table | privatelink.table.cosmos.azure.us | table.cosmos.azure.us |

+- **Failure**: automation rule did not run any action due to one of the following reasons:

+| **Could not add task: *\<TaskName>*.**<br>Incident already contains the maximum allowed number of tasks. | If this task is required, see if there are any tasks that can be removed or consolidated, then try again. |

+ - [SAP HANA Security Guide for SAP HANA Platform](https://help.sap.com/docs/SAP_HANA_PLATFORM/b3ee5778bc2e4a089d3299b82ec762a7/4f7cde1125084ea3b8206038530e96ce.html)

+2. Check your operating system Syslog files for any relevant HANA database events.

+3. Sign into your HANA database operating system as a user with sudo privileges.

+4. Install an agent on your machine and confirm that your machine is connected. For more information, see:

+5. Configure your agent to collect Syslog data. For more information, see:

+Use the following steps in both Microsoft Sentinel and your SAP HANA database to verify that your system is configured as expected.

+### Microsoft Sentinel

+In Microsoft Sentinel's **Logs** page, check to confirm that HANA database events are now shown in the ingested logs. For example, run the following query:

+### SAP HANA

+In your SAP HANA database, check your configured audit policies. For more information on the required SQL statements, see [SAP Note 3016478](https://me.sap.com/notes/3016478/E).

+## Add analytics rules for SAP HANA in Microsoft Sentinel

+Learn more about the Microsoft Sentinel Solution for SAP BTP:

+- [Deploy Microsoft Sentinel solution for SAP® applications](deploy-sap-btp-solution.md)

+- [Microsoft Sentinel Solution for SAP BTP: security content reference](sap-btp-security-content.md)

+- [HANA audit log is not generated in SYSLOG | SAP note](https://me.sap.com/notes/3305033/E)

+- [How to Redirect syslog Auditing for HANA to an alternate location | SAP note](https://me.sap.com/notes/2386609)

+In the Azure Storage documentation, *data protection* refers to strategies for protecting the storage account and data within it from being deleted or modified, or for restoring data after it has been deleted or modified. Azure Storage also offers options for *disaster recovery*, including multiple levels of redundancy to protect your data from service outages due to hardware problems or natural disasters. Customer-managed (unplanned) failover is another disaster recovery option that allows you to fail over to a secondary region if the primary region becomes unavailable. For more information about how your data is protected from service outages, see [Disaster recovery](#disaster-recovery).

+If your storage account is configured for geo-redundancy, you have the option to initiate an unplanned failover from the primary to the secondary region during a data center failure. For more information, see [Disaster recovery planning and failover](../common/storage-disaster-recovery-guidance.md#customer-managed-unplanned-failover).

+Customer-managed failover currently supports storage accounts with a hierarchical namespace enabled in preview status only. For more information, see [Disaster recovery planning and failover](../common/storage-disaster-recovery-guidance.md#plan-for-failover).

+- Customer-managed account failover is supported at the preview level in select regions. For more information, see [Azure storage disaster recovery planning and failover](../common/storage-disaster-recovery-guidance.md#hierarchical-namespace-hns).