+ <LocalizedString ElementType="ClaimType" ElementId="VerificationCode" StringId="DisplayName">Verification Code</LocalizedString>

+ <LocalizedString ElementType="ClaimType" ElementId="VerificationCode" StringId="UserHelpText">Verification code received in the email.</LocalizedString>

+ <LocalizedString ElementType="ClaimType" ElementId="VerificationCode" StringId="AdminHelpText">Verification code received in the email.</LocalizedString>

+Global deployments are available in the same Azure OpenAI resources as non-global deployment types but allow you to leverage Azure's global infrastructure to dynamically route traffic to the data center with best availability for each request. Global standard provides the highest default quota and eliminates the need to load balance across multiple resources.

+Customers with high consistent volume may experience greater latency variability. The threshold is set per model. See the [quotas page to learn more](./quota.md). For applications that require the lower latency variance at large workload usage, we recommend purchasing provisioned throughput.

+ Title: How to migrate to OpenAI JavaScript v4.x

+description: Learn about migrating to the latest release of the OpenAI JavaScript library with Azure OpenAI.

+# Migrating to the OpenAI JavaScript API library 4.x

+As of June 2024, we recommend migrating to the OpenAI JavaScript API library 4.x, the latest version of the official OpenAI JavaScript client library that supports the Azure OpenAI Service API version `2022-12-01` and later. This article helps you bring you up to speed on the changes specific to Azure OpenAI.

+## Authenticating the client

+There are several ways to authenticate API requests to Azure OpenAI. We highly recommend using Microsoft Entra ID tokens. See the [Azure Identity documentation](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/identity/identity/README.md) for more information.

+### Microsoft Entra ID

+There are several ways to authenticate with the Azure OpenAI service using Microsoft Entra ID tokens. The default way is to use the `DefaultAzureCredential` class from the `@azure/identity` package, but your application might be using a different credential class. For the purposes of this guide, we assume that you are using the `DefaultAzureCredential` class. A credential can be created as follows:

+```typescript

+import { DefaultAzureCredential } from "@azure/identity";

+const credential = new DefaultAzureCredential();

+```

+This object is then passed to the second argument of the `OpenAIClient` and `AssistantsClient` client constructors.

+In order to authenticate the `AzureOpenAI` client, however, we need to use the `getBearerTokenProvider` function from the `@azure/identity` package. This function creates a token provider that `AzureOpenAI` uses internally to obtain tokens for each request. The token provider is created as follows:

+```typescript

+import { DefaultAzureCredential, getBearerTokenProvider } from "@azure/identity";

+const credential = new DefaultAzureCredential();

+const scope = "https://cognitiveservices.azure.com/.default";

+const azureADTokenProvider = getBearerTokenProvider(credential, scope);

+```

+`azureADTokenProvider` is passed to the options object when creating the `AzureOpenAI` client.

+### (Highly Discouraged) API Key

+API keys are not recommended for production use because they are less secure than other authentication methods. However, if you are using an API key to authenticate `OpenAIClient` or `AssistantsClient`, an `AzureKeyCredential` object must be created as follows:

+```typescript

+import { AzureKeyCredential } from "@azure/openai";

+const apiKey = new AzureKeyCredential("your API key");

+```

+Authenticating `AzureOpenAI` with an API key involves setting the `AZURE_OPENAI_API_KEY` environment variable or setting the `apiKey` string property in the options object when creating the `AzureOpenAI` client.

+## Constructing the client

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+import { AzureOpenAI } from "openai";

+const deployment = "Your Azure OpenAI deployment";

+const apiVersion = "2024-04-01-preview";

+const options = { azureADTokenProvider, deployment, apiVersion }

+const client = new AzureOpenAI(options);

+```

+The endpoint of the Azure OpenAI resource can be specified by setting the `endpoint` option but it can also be loaded by the client from the environment variable `AZURE_OPENAI_ENDPOINT`. This is the recommended way to set the endpoint because it allows the client to be used in different environments without changing the code and also to protect the endpoint from being exposed in the code.

+The API version is required to be specified, this is necessary to ensure that existing code doesn't break between preview API versions. Refer to [API versioning documentation](../api-version-deprecation.md) to learn more about Azure OpenAI API versions. Additionally, the `deployment` property isn't required but it's recommended to be set. Once `deployment` is set, it's used as the default deployment for all operations that require it. If the client isn't created with the `deployment` option, the `model` property in the options object should be set with the deployment name. However, audio operations such as `audio.transcriptions.create` require the client to be created with the `deployment` option set to the deployment name.

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+import { OpenAIClient } from "@azure/openai";

+const endpoint = "Your Azure OpenAI resource endpoint";

+const client = new OpenAIClient(endpoint, credential);

+```

+If not set, the API version defaults to the last known one before the release of the client. The client is also not locked to a single model deployment, meaning that the deployment name has to be passed to each method that requires it.

+## API differences

+There are key differences between the `OpenAIClient` and `AssistantsClient` clients and the `AzureOpenAI` client:

+- Operations are represented as a flat list of methods in both `OpenAIClient` and `AssistantsClient`, for example `client.getChatCompletions`. In `AzureOpenAI`, operations are grouped in nested groups, for example `client.chat.completions.create`.

+- `OpenAIClient` and `AssistantsClient` rename many of the names used in the Azure OpenAI service API. For example, snake case is used in the API but camel case is used in the client. In `AzureOpenAI`, names are kept the same as in the Azure OpenAI service API.

+## Migration examples

+The following sections provide examples of how to migrate from `OpenAIClient` and `AssistantsClient` to `AzureOpenAI`.

+### Chat completions

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+const result = await client.chat.completions.create({ messages, model: '', max_tokens: 100 });

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+const result = await client.getChatCompletions(deploymentName, messages, { maxTokens: 100 });

+```

+Note the following:

+- The `getChatCompletions` method has been replaced with the `chat.completions.create` method.

+- The `messages` parameter is now passed in the options object with the `messages` property.

+- The `maxTokens` property has been renamed to `max_tokens` and the `deploymentName` parameter has been removed. Generally, the names of the properties in the `options` object are the same as in the Azure OpenAI service API, following the snake case convention instead of the camel case convention used in the `AssistantsClient`. This is true for all the properties across all requests and responses in the `AzureOpenAI` client.

+- The `deploymentName` parameter isn't needed if the client was created with the `deployment` option. If the client was not created with the `deployment` option, the `model` property in the option object should be set with the deployment name.

+### Streaming chat completions

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+const stream = await client.chat.completions.create({ model: '', messages, max_tokens: 100, stream: true });

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+const stream = await client.streamChatCompletions(deploymentName, messages, { maxTokens: 100 });

+```

+### Azure On Your Data

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+import "@azure/openai/types";

+const azureSearchEndpoint = "Your Azure Search resource endpoint";

+const azureSearchIndexName = "Your Azure Search index name";

+const result = await client.chat.completions.create({ model: '', messages, data_sources: [{

+ type: "azure_search",

+ parameters: {

+ endpoint: azureSearchEndpoint,

+ index_name: azureSearchIndexName,

+ authentication: {

+ type: "system_assigned_managed_identity",

+ }

+ }

+ }]

+});

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+const azureSearchEndpoint = "Your Azure Search resource endpoint";

+const azureSearchIndexName = "Your Azure Search index name";

+const result = await client.getChatCompletions(deploymentName, messages, { azureExtensionOptions: {

+ data_sources: [{

+ type: "azure_search",

+ endpoint: azureSearchEndpoint,

+ indexName: azureSearchIndexName,

+ authentication: {

+ type: "system_assigned_managed_identity",

+ }

+ }]

+ }

+});

+```

+- `"@azure/openai/types"` is imported which adds Azure-specific definitions (for example, `data_sources`) to the client types.

+- The `azureExtensionOptions` property has been replaced with the inner `data_sources` property.

+- The `parameters` property has been added to wrap the parameters of the extension, which mirrors the schema of the Azure OpenAI service API.

+- Camel case properties have been replaced with snake case properties.

+### Audio transcription

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+import { createReadStream } from "fs";

+const result = await client.audio.transcriptions.create({

+ model: '',

+ file: createReadStream(audioFilePath),

+});

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+import { readFile } from "fs/promises";

+const audioFilePath = "path/to/audio/file";

+const audio = await readFile(audioFilePath);

+const result = await client.getAudioTranscription(deploymentName, audio);

+```

+- The `getAudioTranscription` method has been replaced with the `audio.transcriptions.create` method.

+- The `AzureOpenAI` has to be constructed with the `deployment` option set to the deployment name in order to use audio operations such as `audio.transcriptions.create`.

+- The `model` property is required to be set in the options object but its value isn't used in the operation so feel free to set it to any value.

+- The `file` property accepts various types including `Buffer`, `fs.ReadaStream`, and `Blob` but in this example, a file is streamed from disk using `fs.createReadStream`.

+### Audio translation

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+import { createReadStream } from "fs";

+const result = await client.audio.translations.create({

+ model: '',

+ file: createReadStream(audioFilePath),

+});

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+import { readFile } from "fs/promises";

+const audioFilePath = "path/to/audio/file";

+const audio = await readFile(audioFilePath);

+const result = await client.getAudioTranslation(deploymentName, audio);

+```

+- The `getAudioTranslation` method has been replaced with the `audio.translations.create` method.

+- All other changes are the same as in the audio transcription example.

+### Assistants

+The following examples show how to migrate some of the `AssistantsClient` methods.

+#### Assistant creation

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+const options = ...;

+const assistantResponse = await assistantsClient.beta.assistants.create(

+ options

+);

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+const options = {

+ model: azureOpenAIDeployment,

+ name: "Math Tutor",

+ instructions:

+ "You are a personal math tutor. Write and run JavaScript code to answer math questions.",

+ tools: [{ type: "code_interpreter" }],

+};

+const assistantResponse = await assistantsClient.createAssistant(options);

+```

+- The `createAssistant` method has been replaced with the `beta.assistants.create` method

+#### Thread creation

+The following example shows how to migrate the `createThread` method call.

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+const assistantThread = await assistantsClient.beta.threads.create();

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+const assistantThread = await assistantsClient.createThread();

+```

+- The `createThread` method has been replaced with the `beta.threads.create` method

+#### Message creation

+The following example shows how to migrate the `createMessage` method call.

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+const threadResponse = await assistantsClient.beta.threads.messages.create(

+ assistantThread.id,

+ {

+ role,

+ content: message,

+ }

+);

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+const threadResponse = await assistantsClient.createMessage(

+ assistantThread.id,

+ role,

+ message

+);

+```

+- The `createMessage` method has been replaced with the `beta.threads.messages.create` method.

+- The message specification has been moved from a parameter list to an object.

+#### Runs

+To run an assistant on a thread, the `createRun` method is used to create a run, and then a loop is used to poll the run status until it is in a terminal state. The following example shows how to migrate the run creation and polling.

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+This code can be migrated and simplified by using the `createAndPoll` method, which creates a run and polls it until it is in a terminal state.

+```typescript

+const runResponse = await assistantsClient.beta.threads.runs.createAndPoll(

+ assistantThread.id,

+ {

+ assistant_id: assistantResponse.id,

+ },

+ { pollIntervalMs: 500 }

+);

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+let runResponse = await assistantsClient.createRun(assistantThread.id, {

+ assistantId: assistantResponse.id,

+});

+do {

+ await new Promise((r) => setTimeout(r, 500));

+ runResponse = await assistantsClient.getRun(

+ assistantThread.id,

+ runResponse.id

+ );

+} while (

+ runResponse.status === "queued" ||

+ runResponse.status === "in_progress"

+```

+- The `createRun` method has been replaced with the `beta.threads.runs.create` and `createAndPoll` methods.

+- The `createAndPoll` method is used to create a run and poll it until it is in a terminal state.

+#### Processing Run results

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+Pages can be looped through by using the `for await` loop.

+```typescript

+for await (const runMessageDatum of runMessages) {

+ for (const item of runMessageDatum.content) {

+ ...

+ }

+}

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+Without paging, results had to be accessed manually page by page using the `data` property of the response object. For instance, accessing the first page can be done as follows:

+```typescript

+for (const runMessageDatum of runMessages.data) {

+ for (const item of runMessageDatum.content) {

+ ...

+ }

+}

+```

+### Embeddings

+The following example shows how to migrate the `getEmbeddings` method call.

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+const embeddings = await client.embeddings.create({ input, model: '' });

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+const embeddings = await client.getEmbeddings(deploymentName, input);

+```

+- The `getEmbeddings` method has been replaced with the `embeddings.create` method.

+- The `input` parameter is now passed in the options object with the `input` property.

+- The `deploymentName` parameter has been removed. The `deploymentName` parameter isn't needed if the client was created with the `deployment` option. If the client was not created with the `deployment` option, the `model` property in the option object should be set with the deployment name.

+### Image generation

+The following example shows how to migrate the `getImages` method call.

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+ const results = await client.images.generate({ prompt, model: '', n, size });

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+const results = await client.getImages(deploymentName, prompt, { n, size });

+```

+- The `getImages` method has been replaced with the `images.generate` method.

+- The `prompt` parameter is now passed in the options object with the `prompt` property.

+- The `deploymentName` parameter has been removed. The `deploymentName` parameter isn't needed if the client was created with the `deployment` option. If the client was not created with the `deployment` option, the `model` property in the option object should be set with the deployment name.

+### Content filter

+Content filter results are part of the chat completions response types in `OpenAIClient`. However, `AzureOpenAI` does not have a direct equivalent to the `contentFilterResults` property in the `ChatCompletion.Choice` interface. The content filter results can be accessed by importing `"@azure/openai/types"` and accessing the `content_filter_results` property. The following example shows how to access the content filter results.

+# [OpenAI JavaScript (new)](#tab/javascript-new)

+```typescript

+import "@azure/openai/types";

+const result = await client.chat.completions.create({ model: '', messages });

+for (const choice of results.choices) {

+ const filterResults = choice.content_filter_results;

+ if (!filterResults) {

+ console.log("No content filter is found");

+ return;

+ }

+ if (filterResults.error) {

+ console.log(

+ `Content filter ran into the error ${filterResults.error.code}: ${filterResults.error.message}`);

+ }

+ const { hate, sexual, self_harm, violence } = filterResults;

+ ...

+}

+```

+# [Azure OpenAI JavaScript (previous)](#tab/javascript-old)

+```typescript

+const results = await client.getChatCompletions(deploymentName, messages);

+for (const choice of results.choices) {

+ if (!choice.contentFilterResults) {

+ console.log("No content filter is found");

+ return;

+ }

+ if (choice.contentFilterResults.error) {

+ console.log(

+ `Content filter ran into the error ${choice.contentFilterResults.error.code}: ${choice.contentFilterResults.error.message}`);

+ }

+ const { hate, sexual, selfHarm, violence } = choice.contentFilterResults;

+ ...

+}

+```

+- Camel case properties have been replaced with snake case properties.

+- `"@azure/openai/types"` is imported which adds Azure-specific definitions (for example, content_filter_results) to the client types, see the [Azure types](#azure-types) section for more information.

+## Comparing Types

+The following table explores several type names from `@azure/openai` and shows their nearest `openai` equivalent. The names differences illustrate several of the above-mentioned changes. This table provides an overview, and more detail and code samples are provided in the following sections.

+| Old Type Name | Nearest New Type | Symbol Type | Change description |

+| - | - | -- | -- |

+| `OpenAIClient` | `AzureOpenAI` | Class | This class replaces the former and has no methods in common with it. See the section on `AzureOpenAI` below. |

+| `AudioResult` | `Transcription`/`Transcription` | Interface | Depending on the calling operation, the two interfaces replace the former one |

+| `AudioResultFormat` | inline union type of the `response_format` property | Alias | It doesn't exist |

+| `AudioResultSimpleJson ` | `Transcription`/`Transcription` | Interface | Depending on the calling operation, the two interfaces replace the former one |

+| `AudioResultVerboseJson ` | N/A | Interface | |

+| `AudioSegment ` | N/A | Interface | |

+| `AudioTranscriptionTask ` | N/A | Alias | |

+| `AzureChatEnhancementConfiguration`, `AzureChatEnhancements`, `AzureChatExtensionConfiguration`, `AzureChatExtensionConfigurationUnion`, `AzureChatExtensionDataSourceResponseCitation`, `AzureChatExtensionsMessageContext`, `AzureChatExtensionType`, `AzureChatGroundingEnhancementConfiguration`, `AzureChatOCREnhancementConfiguration`, `AzureCosmosDBChatExtensionConfiguration`, `AzureCosmosDBFieldMappingOptions`, `AzureExtensionsOptions`, `AzureGroundingEnhancement`, `AzureGroundingEnhancementCoordinatePoint`, `AzureGroundingEnhancementLine`, `AzureGroundingEnhancementLineSpan`, `AzureMachineLearningIndexChatExtensionConfiguration`, `AzureSearchChatExtensionConfiguration`, `AzureSearchIndexFieldMappingOptions`, `AzureSearchQueryType`, `ContentFilterBlocklistIdResult`, `ContentFilterCitedDetectionResult`, `ContentFilterDetectionResult`, `ContentFilterErrorResults`, `ContentFilterResult`, `ContentFilterResultDetailsForPrompt`, `ContentFilterResultsForChoice`, `ContentFilterSeverity`, `ContentFilterResultsForPrompt`, `ContentFilterSuccessResultDetailsForPrompt`, `ContentFilterSuccessResultsForChoice`, `ElasticsearchChatExtensionConfiguration`, `ElasticsearchIndexFieldMappingOptions`, `ElasticsearchQueryType`, `ImageGenerationContentFilterResults`, `ImageGenerationPromptFilterResults`, `OnYourDataAccessTokenAuthenticationOptions`, `OnYourDataApiKeyAuthenticationOptions`, `OnYourDataAuthenticationOptions`, `OnYourDataAuthenticationOptionsUnion`, `OnYourDataConnectionStringAuthenticationOptions`, `OnYourDataDeploymentNameVectorizationSource`, `OnYourDataEncodedApiKeyAuthenticationOptions`, `OnYourDataEndpointVectorizationSource`, `OnYourDataKeyAndKeyIdAuthenticationOptions`, `OnYourDataModelIdVectorizationSource`, `OnYourDataSystemAssignedManagedIdentityAuthenticationOptions`, `OnYourDataUserAssignedManagedIdentityAuthenticationOptions`, `OnYourDataVectorizationSource`, `OnYourDataVectorizationSourceType`, `OnYourDataVectorizationSourceUnion`, `PineconeChatExtensionConfiguration`, `PineconeFieldMappingOptions` | N/A | Interfaces and Aliases | See the Azure types section below |

+| `AzureKeyCredential` | N/A | Class | The API key can be provided as a string value |

+| `ChatChoice` | `ChatCompletion.Choice` | Interface | |

+| `ChatChoiceLogProbabilityInfo` | `Logprobs` | Interface | |

+| `ChatCompletions` | `ChatCompletion` and `ChatCompletionChunk` | Interface | |

+| `ChatCompletionsFunctionToolCall` | `ChatCompletionMessageToolCall` | Interface | |

+| `ChatRequestFunctionMessage` | `ChatCompletionFunctionMessageParam` | Interface | |

+| `ChatRequestMessage` | `ChatCompletionMessageParam` | Interface | |

+| `ChatRequestMessageUnion` | `ChatCompletionMessageParam` | |

+| `ChatRequestSystemMessage` | `ChatCompletionSystemMessageParam` | Interface | |

+| `ChatRequestToolMessage` | `ChatCompletionToolMessageParam` | Interface | |

+| `ChatRequestUserMessage` | `ChatCompletionUserMessageParam` | Interface | |

+| `ChatResponseMessage` | `Delta` / `ChatCompletionMessage` | Interface | |

+| `ChatRole` | N/A | Alias | |

+| `ChatTokenLogProbabilityInfo` | `TopLogprob` | Interface | |

+| `ChatTokenLogProbabilityResult` | `ChatCompletionTokenLogprob` | Interface | |

+| `Choice` | `Choice` | Interface | |

+| `Completions` | `Completion` | Interface | |

+| `CompletionsFinishReason` | N/A | Alias | |

+| `CompletionsLogProbabilityModel` | `Logprobs` | Interface | |

+| `CompletionsUsage` | `CompletionUsage` | Interface | |

+| `EmbeddingItem` | `Embedding` | Interface | |

+| `Embeddings` | `CreateEmbeddingResponse` | Interface | |

+| `EmbeddingsUsage` | `CreateEmbeddingResponse.Usage` | Interface | |

+| `EventStream` | `Stream` | Interface | |

+| `FunctionCall` | `FunctionCall` | Interface | |

+| `FunctionCallPreset` | N/A | Alias | |

+| `FunctionDefinition` | `Function` | Interface | |

+| `FunctionName` | N/A | Alias | |

+| `GetAudioTranscriptionOptions` | `TranscriptionCreateParams` | Interface | |

+| `GetAudioTranslationOptions` | `TranslationCreateParams` | Interface | |

+| `GetChatCompletionsOptions` | `ChatCompletionCreateParamsNonStreaming` and `ChatCompletionCreateParamsStreaming` | Interface | |

+| `GetCompletionsOptions` | `CompletionCreateParams` | Interface | |

+| `GetEmbeddingsOptions` | `EmbeddingCreateParams` | Interface | |

+| `GetImagesOptions` | `ImageGenerateParams` | Interface | |

+| `ImageGenerationData` | `Image` | Interface | |

+| `ImageGenerationQuality` | N/A | Alias | |

+| `ImageGenerationResponseFormat` | N/A | Alias | |

+| `ImageGenerations` | `ImagesResponse` | Interface | |

+| `ImageGenerationStyle` | N/A | Alias | |

+| `ImageSize` | N/A | Alias | |

+| `MaxTokensFinishDetails` | N/A | Interface | |

+| `OpenAIClientOptions` | `AzureClientOptions` | Interface | |

+| `OpenAIError` | `OpenAIError` | Interface | |

+| `OpenAIKeyCredential` | N/A | Class | |

+| `StopFinishDetails` | N/A | Interface | |

+## Azure types

+`AzureOpenAI` connects to the Azure OpenAI service and can call all the operations available in the service. However, the types of the requests and responses are inherited from the `OpenAI` and are not yet updated to reflect the additional features supported exclusively by the Azure OpenAI service. TypeScript users will need to import `"@azure/openai/types"` from `@azure/openai@2.0.0-beta.1` which will merge Azure-specific definitions into existing types. Examples in [the Migration examples](#migration-examples) section show how to do this.

+## Next steps

+- [Azure OpenAI Assistants](../concepts/assistants.md)

+|Enterprise agreement | 30 M | 180 K |

+# Customize text to speech avatar gestures with SSML

+# Batch synthesis properties for text to speech avatar

+# How to use batch synthesis for text to speech avatar

+The batch synthesis API for text to speech avatar allows for the asynchronous synthesis of text into a talking avatar as a video file. Publishers and video content platforms can utilize this API to create avatar video content in a batch. That approach can be suitable for various use cases such as training materials, presentations, or advertisements.

+| [Create batch synthesis](#create-a-batch-synthesis-request) | PUT | avatar/batchsyntheses/{SynthesisId}?api-version=2024-08-01 |

+| [Get batch synthesis](#get-batch-synthesis) | GET | avatar/batchsyntheses/{SynthesisId}?api-version=2024-08-01 |

+| [List batch synthesis](#list-batch-synthesis) | GET | avatar/batchsyntheses/?api-version=2024-08-01 |

+| [Delete batch synthesis](#delete-batch-synthesis) | DELETE | avatar/batchsyntheses/{SynthesisId}?api-version=2024-08-01 |

+}' "https://YourSpeechRegion.api.cognitive.microsoft.com/avatar/batchsyntheses/my-job-01?api-version=2024-08-01"

+curl -v -X GET "https://YourSpeechRegion.api.cognitive.microsoft.com/avatar/batchsyntheses/YourSynthesisId?api-version=2024-08-01" -H "Ocp-Apim-Subscription-Key: YourSpeechKey"

+curl -v -X GET "https://YourSpeechRegion.api.cognitive.microsoft.com/avatar/batchsyntheses?skip=0&maxpagesize=2&api-version=2024-08-01" -H "Ocp-Apim-Subscription-Key: YourSpeechKey"

+ "nextLink": "https://YourSpeechRegion.api.cognitive.microsoft.com/avatar/batchsyntheses/?api-version=2024-08-01&skip=2&maxpagesize=2"

+curl -v -X DELETE "https://YourSpeechRegion.api.cognitive.microsoft.com/avatar/batchsyntheses/YourSynthesisId?api-version=2024-08-01" -H "Ocp-Apim-Subscription-Key: YourSpeechKey"

+# How to create a custom text to speech avatar

+# How to record video samples for custom text to speech avatar

+ Title: Real-time synthesis for text to speech avatar - Speech service

+# How to do real-time synthesis for text to speech avatar

+In this how-to guide, you learn how to use text to speech avatar with real-time synthesis. The synthetic avatar video will be generated in almost real time after the system receives the text input.

+# What is custom text to speech avatar?

+1. **Train the avatar model:** We'll start training the custom text to speech model after verifying the consent statement of the avatar talent. This step is currently manually done by Microsoft. You'll be notified after the model is successfully trained.

+# Text to speech avatar overview

+| | Batch synthesis | Real-time synthesis |

+- Phi-3 family of models

+### Phi-3 family models

+The following Phi-3 family models are supported in Azure AI Studio for fine-tuning:

+- `Phi-3-mini-4k-instruct`

+- `Phi-3-mini-128k-instruct`

+- `Phi-3-medium-4k-instruct`

+- `Phi-3-medium-128k-instruct`

+Fine-tuning of Phi-3 models is currently supported in projects located in East US2.

+Fine-tuning of Llama 2 models is currently supported in projects located in West US3.

+- `Meta-Llama-3.1-8b-Instruct`

+Fine-tuning of Llama 3.1 models is currently supported in projects located in West US3.

+- [Learn how to fine-tune a Phi-3 model in Azure AI Studio](../how-to/fine-tune-phi-3.md)

+- [How to deploy Phi-3 family of small language models with Azure AI Studio](../how-to/deploy-models-phi-3.md)

+ Title: How to deploy Cohere Rerank models as serverless APIs

+description: Learn to deploy and use Cohere Rerank models with Azure AI Studio.

+# How to deploy Cohere Rerank models with Azure AI Studio

+In this article, you learn about the Cohere Rerank models, how to use Azure AI Studio to deploy them as serverless APIs with pay-as-you-go token-based billing, and how to work with the deployed models.

+## Cohere Rerank models

+Cohere offers two Rerank models in [Azure AI Studio](https://ai.azure.com). These models are available in the model catalog for deployment as serverless APIs:

+* Cohere Rerank 3 - English

+* Cohere Rerank 3 - Multilingual

+You can browse the Cohere family of models in the [Model Catalog](model-catalog.md) by filtering on the Cohere collection.

+### Cohere Rerank 3 - English

+Cohere Rerank English is a reranking model used for semantic search and retrieval-augmented generation (RAG). Rerank enables you to significantly improve search quality by augmenting traditional keyword-based search systems with a semantic-based reranking system that can contextualize the meaning of a user's query beyond keyword relevance. Cohere's Rerank delivers higher quality results than embedding-based search, lexical search, and even hybrid search, and it requires only adding a single line of code into your application.

+Use Rerank as a ranker after initial retrieval. In other words, after an initial search system finds the top 100 most relevant documents for a larger corpus of documents.

+Rerank supports JSON objects as documents where users can specify, at query time, the fields (keys) to use for semantic search. Some other attributes of Rerank include:

+* Context window of the model is 4,096 tokens

+* The max query length is 2,048 tokens

+Rerank English works well for code retrieval, semi-structured data retrieval, and long context.

+### Cohere Rerank 3 - Multilingual

+Cohere Rerank Multilingual is a reranking model used for semantic search and retrieval-augmented generation (RAG). Rerank Multilingual supports more than 100 languages and can be used to search within a language (for example, to search with a French query on French documents) and across languages (for example, to search with an English query on Chinese documents). Rerank enables you to significantly improve search quality by augmenting traditional keyword-based search systems with a semantic-based reranking system that can contextualize the meaning of a user's query beyond keyword relevance. Cohere's Rerank delivers higher quality results than embedding-based search, lexical search, and even hybrid search, and it requires only adding a single line of code into your application.

+Use Rerank as a ranker after initial retrieval. In other words, after an initial search system finds the top 100 most relevant documents for a larger corpus of documents.

+Rerank supports JSON objects as documents where users can specify, at query time, the fields (keys) to use for semantic search. Some other attributes of Rerank Multilingual include:

+* Context window of the model is 4,096 tokens

+* The max query length is 2,048 tokens

+Rerank multilingual performs well on multilingual benchmarks such as Miracl.

+## Deploy Cohere Rerank models as serverless APIs

+Certain models in the model catalog can be deployed as a serverless API with pay-as-you-go billing. This kind of deployment provides a way to consume models as an API without hosting them on your subscription, while keeping the enterprise security and compliance that organizations need. This deployment option doesn't require quota from your subscription.

+You can deploy the previously mentioned Cohere models as a service with pay-as-you-go billing. Cohere offers these models through the Microsoft Azure Marketplace and can change or update the terms of use and pricing of these models.

+### Prerequisites

+- An Azure subscription with a valid payment method. Free or trial Azure subscriptions won't work. If you don't have an Azure subscription, create a [paid Azure account](https://azure.microsoft.com/pricing/purchase-options/pay-as-you-go) to begin.

+- An [AI Studio hub](../how-to/create-azure-ai-resource.md). The serverless API model deployment offering for Cohere Rerank is only available with hubs created in these regions:

+ * East US

+ * East US 2

+ * North Central US

+ * South Central US

+ * West US

+ * West US 3

+ * Sweden Central

+

+ For a list of regions that are available for each of the models supporting serverless API endpoint deployments, see [Region availability for models in serverless API endpoints](deploy-models-serverless-availability.md).

+- An [Azure AI Studio project](../how-to/create-projects.md).

+- Azure role-based access controls are used to grant access to operations in Azure AI Studio. To perform the steps in this article, your user account must be assigned the __Azure AI Developer role__ on the resource group. For more information on permissions, see [Role-based access control in Azure AI Studio](../concepts/rbac-ai-studio.md).

+### Create a new deployment

+The following steps demonstrate the deployment of Cohere Rerank 3 - English, but you can use the same steps to deploy Cohere Rerank 3 - Multilingual by replacing the model name.

+To create a deployment:

+1. Sign in to [Azure AI Studio](https://ai.azure.com).

+1. Select **Model catalog** from the left sidebar.

+1. Search for *Cohere*.

+1. Select **cohere-rerank-3-english** to open the Model Details page.

+1. Select **Deploy** to open a serverless API deployment window for the model.

+1. Alternatively, you can initiate a deployment by starting from your project in AI Studio.

+ 1. From the left sidebar of your project, select **Components** > **Deployments**.

+ 1. Select **+ Create deployment**.

+ 1. Search for and select **Cohere-rerank-3-english**. to open the Model Details page.

+ 1. Select **Confirm** to open a serverless API deployment window for the model.

+1. Select the project in which you want to deploy your model.

+1. In the deployment wizard, select the link to **Azure Marketplace Terms** to learn more about the terms of use.

+1. Select the **Pricing and terms** tab to learn about pricing for the selected model.

+1. Select the **Subscribe and Deploy** button. If this is your first time deploying the model in the project, you have to subscribe your project for the particular offering. This step requires that your account has the **Azure AI Developer role** permissions on the resource group, as listed in the prerequisites. Each project has its own subscription to the particular Azure Marketplace offering of the model, which allows you to control and monitor spending. Currently, you can have only one deployment for each model within a project.

+1. Once you subscribe the project for the particular Azure Marketplace offering, subsequent deployments of the _same_ offering in the _same_ project don't require subscribing again. If this scenario applies to you, there's a **Continue to deploy** option to select.

+1. Give the deployment a name. This name becomes part of the deployment API URL. This URL must be unique in each Azure region.

+1. Select **Deploy**. Wait until the deployment is ready and you're redirected to the Deployments page.

+1. On the Deployments page, select the deployment, and note the endpoint's **Target** URL and the Secret **Key**. For more information on using the APIs, see the [reference](#rerank-api-reference-for-cohere-rerank-models-deployed-as-a-service) section.

+1. You can always find the endpoint's details, URL, and access keys by navigating to your **Project overview** page. Then, from the left sidebar of your project, select **Components** > **Deployments**.

+To learn about billing for the Cohere models deployed as a serverless API with pay-as-you-go token-based billing, see [Cost and quota considerations for Cohere models deployed as a service](#cost-and-quota-considerations-for-models-deployed-as-a-service).

+### Consume the Cohere Rerank models as a service

+Cohere Rerank models deployed as serverless APIs can be consumed using the Rerank API.

+1. From your **Project overview** page, go to the left sidebar and select **Components** > **Deployments**.

+1. Find and select the deployment you created.

+1. Copy the **Target** URL and the **Key** value.

+1. Cohere currently exposes `v1/rerank` for inference with the Rerank 3 - English and Rerank 3 - Multilingual models schema. For more information on using the APIs, see the [reference](#rerank-api-reference-for-cohere-rerank-models-deployed-as-a-service) section.

+## Rerank API reference for Cohere Rerank models deployed as a service

+Cohere Rerank 3 - English and Rerank 3 - Multilingual accept the native Cohere Rerank API on `v1/rerank`. This section contains details about the Cohere Rerank API.

+#### v1/rerank request

+```json

+ POST /v1/rerank HTTP/1.1

+ Host: <DEPLOYMENT_URI>

+ Authorization: Bearer <TOKEN>

+ Content-type: application/json

+```

+#### v1/rerank request schema

+Cohere Rerank 3 - English and Rerank 3 - Multilingual accept the following parameters for a `v1/rerank` API call:

+| Property | Type | Default | Description |

+| | | | |

+|`query` |`string` |Required |The search query. |

+|`documents` |`array` |None |A list of document objects or strings to rerank. |

+|`top_n` |`integer` |Length of `documents` |The number of most relevant documents or indices to return. |

+|`return_documents` |`boolean` |`FALSE` |If `FALSE`, returns results without the doc text - the API returns a list of {`index`, `relevance_score`} where index is inferred from the list passed into the request. </br>If `TRUE`, returns results with the doc text passed in - the API returns an ordered list of {`index`, `text`, `relevance_score`} where index + text refers to the list passed into the request. |

+|`max_chunks_per_doc` |`integer` |None |The maximum number of chunks to produce internally from a document.|

+|`rank_fields` |`array of strings` |None |If a JSON object is provided, you can specify which keys you would like to consider for reranking. The model reranks based on the order of the fields passed in (for example, `rank_fields=['title','author','text']` reranks, using the values in `title`, `author`, and `text` in that sequence. If the length of title, author, and text exceeds the context length of the model, the chunking won't reconsider earlier fields).<br> If not provided, the model uses the default text field for ranking. |

+#### v1/rerank response schema

+Response fields are fully documented on [Cohere's Rerank API reference](https://docs.cohere.com/reference/rerank). The response payload is a dictionary with the following fields:

+| Key | Type | Description |

+| | | |

+| `id` | `string` |An identifier for the response. |

+| `results` | `array of objects`|An ordered list of ranked documents, where each document is described by an object that includes `index` and `relevance_score` and, optionally, `text`. |

+| `meta` | `array of objects` | An optional meta object containing a list of warning strings. |

+<br>

+The `results` object is a dictionary with the following fields:

+| Key | Type | Description |

+| | | |

+| `document` | `object` |The document objects or strings that were reranked. |

+| `index` | `integer` |The `index` in the original list of documents to which the ranked document belongs. For example, if the first value in the `results` object has an index value of 3, it means in the list of documents passed in, the document at `index=3` had the highest relevance.|

+| `relevance_score` | `float` |Relevance scores are normalized to be in the range `[0, 1]`. Scores close to one indicate a high relevance to the query, and scores close to zero indicate low relevance. A score of `0.9` _doesn't_ necessarily mean that a document is twice as relevant as another with a score of `0.45`. |

+## Examples

+#### Request example

+```json

+ {

+ "query": "What is the capital of the United States?",

+ "rank_fields": ["Title", "Content"],

+ "documents": [

+ {"Title": "Facts about Carson City", "Content": "Carson City is the capital city of the American state of Nevada. "},

+ {"Title": "North Dakota", "Content" : "North Dakota is a state in the United States. 672,591 people lived in North Dakota in the year 2010. The capital and seat of government is Bismarck."},

+ {"Title": "Micronesia", "Content" : "Micronesia, officially the Federated States of Micronesia, is an island nation in the Pacific Ocean, northeast of Papua New Guinea. The country is a sovereign state in free association with the United States. The capital city of Federated States of Micronesia is Palikir."}

+ ],

+ "top_n": 3

+ }

+```

+#### Response example

+```json

+ {

+ "id": "571e6744-3074-457f-8935-08646a3352fb",

+ "results": [

+ {

+ "document": {

+ "Content": "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. The President of the USA and many major national government offices are in the territory. This makes it the political center of the United States of America.",

+ "Title": "Details about Washington D.C"

+ },

+ "index": 0,

+ "relevance_score": 0.98347044

+ },

+ {

+ "document": {

+ "Content": "Carson City is the capital city of the American state of Nevada. ",

+ "Title": "Facts about Carson City"

+ },

+ "index": 1,

+ "relevance_score": 0.07172112

+ },

+ {

+ "document": {

+ "Content": "Micronesia, officially the Federated States of Micronesia, is an island nation in the Pacific Ocean, northeast of Papua New Guinea. The country is a sovereign state in free association with the United States. The capital city of Federated States of Micronesia is Palikir.",

+ "Title": "Micronesia"

+ },

+ "index": 3,

+ "relevance_score": 0.05281402

+ },

+ {

+ "document": {

+ "Content": "North Dakota is a state in the United States. 672,591 people lived in North Dakota in the year 2010. The capital and seat of government is Bismarck.",

+ "Title": "North Dakota"

+ },

+ "index": 2,

+ "relevance_score": 0.03138043

+ }

+ ]

+ }

+```

+#### More inference examples

+| Package | Sample Notebook |

+|||

+|CLI using CURL and Python web requests| [cohere-rerank.ipynb](https://aka.ms/samples/cohere-rerank/webrequests)|

+|LangChain|[langchain.ipynb](https://aka.ms/samples/cohere-rerank/langchain)|

+|Cohere SDK|[cohere-sdk.ipynb](https://aka.ms/samples/cohere-rerank/cohere-python-sdk)|

+## Cost and quota considerations for models deployed as a service

+Quota is managed per deployment. Each deployment has a rate limit of 200,000 tokens per minute and 1,000 API requests per minute. However, we currently limit one deployment per model per project. Contact Microsoft Azure Support if the current rate limits aren't sufficient for your scenarios.

+Cohere models deployed as serverless APIs with pay-as-you-go billing are offered by Cohere through the Azure Marketplace and integrated with Azure AI Studio for use. You can find the Azure Marketplace pricing when deploying the model.

+Each time a project subscribes to a given offer from the Azure Marketplace, a new resource is created to track the costs associated with its consumption. The same resource is used to track costs associated with inference; however, multiple meters are available to track each scenario independently.

+For more information on how to track costs, see [monitor costs for models offered throughout the Azure Marketplace](./costs-plan-manage.md#monitor-costs-for-models-offered-through-the-azure-marketplace).

+## Related content

+- [What is Azure AI Studio?](../what-is-ai-studio.md)

+- [Azure AI FAQ article](../faq.yml)

+- [Region availability for models in serverless API endpoints](deploy-models-serverless-availability.md)

+* [Meta-Llama-3.1-405B-Instruct (preview)](https://aka.ms/aistudio/landing/meta-llama-3.1-405B-instruct)

+* [Meta-Llama-3.1-70B-Instruct (preview)](https://aka.ms/aistudio/landing/meta-llama-3.1-70B-instruct)

+* [Meta Llama-3.1-8B-Instruct (preview)](https://aka.ms/aistudio/landing/meta-llama-3.1-8B-instruct)

+ Title: Fine-tune Phi-3 models in Azure AI Studio

+description: This article introduces fine-tuning Phi-3 models in Azure AI Studio.

+# Fine-tune Phi-3 models in Azure AI Studio

+Azure AI Studio lets you tailor large language models to your personal datasets by using a process known as fine-tuning. Fine-tuning provides significant value by enabling customization and optimization for specific tasks and applications. It leads to improved performance, cost efficiency, reduced latency, and tailored outputs.

+In this article, you learn how to fine-tune Phi-3 family of small language models (SLMs) in Azure AI Studio as a service with pay-as you go billing.

+The Phi-3 family of SLMs is a collection of instruction-tuned generative text models. Phi-3 models are the most capable and cost-effective small language models (SLMs) available, outperforming models of the same size and next size up across various language, reasoning, coding, and math benchmarks.

+## [Phi-3-mini](#tab/phi-3-mini)

+Phi-3 Mini is a 3.8B parameters, lightweight, state-of-the-art open model built upon datasets used for Phi-2 - synthetic data and filtered websites - with a focus on high-quality, reasoning dense data. The model belongs to the Phi-3 model family, and the Mini version comes in two variants 4K and 128K which is the context length (in tokens) it can support.

+- [Phi-3-mini-4k-Instruct](https://ai.azure.com/explore/models/Phi-3-mini-4k-instruct/version/4/registry/azureml)

+- [Phi-3-mini-128k-Instruct](https://ai.azure.com/explore/models/Phi-3-mini-128k-instruct/version/4/registry/azureml)

+The model underwent a rigorous enhancement process, incorporating both supervised fine-tuning and direct preference optimization to ensure precise instruction adherence and robust safety measures. When assessed against benchmarks testing common sense, language understanding, math, code, long context and logical reasoning, Phi-3 Mini-4K-Instruct and Phi-3 Mini-128K-Instruct showcased a robust and state-of-the-art performance among models with less than 13 billion parameters.

+## [Phi-3-medium](#tab/phi-3-medium)

+Phi-3 Medium is a 14B parameters, lightweight, state-of-the-art open model. Phi-3-Medium was trained with Phi-3 datasets that include both synthetic data and the filtered, publicly available websites data, with a focus on high quality and reasoning-dense properties.

+The model belongs to the Phi-3 model family, and the Medium version comes in two variants, 4K and 128K, which denote the context length (in tokens) that each model variant can support.

+- Phi-3-medium-4k-Instruct

+- Phi-3-medium-128k-Instruct

+The model underwent a rigorous enhancement process, incorporating both supervised fine-tuning and direct preference optimization to ensure precise instruction adherence and robust safety measures. When assessed against benchmarks that test common sense, language understanding, math, code, long context and logical reasoning, Phi-3-Medium-4k-Instruct and Phi-3-Medium-128k-Instruct showcased a robust and state-of-the-art performance among models with less than 13 billion parameters.

+## [Phi-3-mini](#tab/phi-3-mini)

+The following models are available in Azure AI studio for Phi 3 when fine-tuning as a service with pay-as-you-go:

+- `Phi-3-mini-4k-instruct` (preview)

+- `Phi-3-mini-128k-instruct` (preview)

+Fine-tuning of Phi-3 models is currently supported in projects located in East US 2.

+## [Phi-3-medium](#tab/phi-3-medium)

+The following models are available in Azure AI studio for Phi 3 when fine-tuning as a service with pay-as-you-go:

+- `Phi-3-medium-4k-instruct` (preview)

+- `Phi-3-medium-128k-instruct` (preview)

+Fine-tuning of Phi-3 models is currently supported in projects located in East US 2.

+### Prerequisites

+- An Azure subscription. If you don't have an Azure subscription, create a [paid Azure account](https://azure.microsoft.com/pricing/purchase-options/pay-as-you-go) to begin.

+- An [AI Studio hub](../how-to/create-azure-ai-resource.md).

+ > [!IMPORTANT]

+ > For Phi-3 family models, the pay-as-you-go model fine-tune offering is only available with hubs created in **East US 2** regions.

+- An [AI Studio project](../how-to/create-projects.md).

+- Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure AI Studio. To perform the steps in this article, your user account must be assigned the __Azure AI Developer role__ on the resource group.

+ For more information on permissions, see [Role-based access control in Azure AI Studio](../concepts/rbac-ai-studio.md).

+### Subscription provider registration

+Verify the subscription is registered to the `Microsoft.Network` resource provider.

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. Select **Subscriptions** from the left menu.

+1. Select the subscription you want to use.

+1. Select **AI project settings** > **Resource providers** from the left menu.

+1. Confirm that **Microsoft.Network** is in the list of resource providers. Otherwise add it.

+### Data preparation

+Prepare your training and validation data to fine-tune your model. Your training data and validation data sets consist of input and output examples for how you would like the model to perform.

+Make sure all your training examples follow the expected format for inference. To fine-tune models effectively, ensure a balanced and diverse dataset.

+This involves maintaining data balance, including various scenarios, and periodically refining training data to align with real-world expectations, ultimately leading to more accurate and balanced model responses.

+Different model types require a different format of training data.

+### Chat completion

+The training and validation data you use **must** be formatted as a JSON Lines (JSONL) document. For `Phi-3-mini-128k-instruct` the fine-tuning dataset must be formatted in the conversational format that is used by the Chat completions API.

+### Example file format

+```json

+ {"messages": [{"role": "system", "content": "You are an Xbox customer support agent whose primary goal is to help users with issues they are experiencing with their Xbox devices. You are friendly and concise. You only provide factual answers to queries, and do not provide answers that are not related to Xbox."}, {"role": "user", "content": "Is Xbox better than PlayStation?"}, {"role": "assistant", "content": "I apologize, but I cannot provide personal opinions. My primary job is to assist you with any issues related to your Xbox device. Do you have any Xbox-related issues that need addressing?"}]}

+ {"messages": [{"role": "system", "content": "You are an Xbox customer support agent whose primary goal is to help users with issues they are experiencing with their Xbox devices. You are friendly and concise. You only provide factual answers to queries, and do not provide answers that are not related to Xbox."}, {"role": "user", "content": "Can I play PC games on my Xbox?"}, {"role": "assistant", "content": "While Xbox consoles don't natively support PC games, they can play an array of Xbox games, many of which are also available on PC. Can I assist you with finding games compatible with your Xbox device?"}]}

+ {"messages": [{"role": "system", "content": "You are an Xbox customer support agent whose primary goal is to help users with issues they are experiencing with their Xbox devices. You are friendly and concise. You only provide factual answers to queries, and do not provide answers that are not related to Xbox."}, {"role": "user", "content": "I'm having trouble connecting my Xbox to the Wi-Fi."}, {"role": "assistant", "content": "No worries, let's go through the network settings on your Xbox. Can you please tell me what happens when you try to connect it to the Wi-Fi?"}]}

+```

+The supported file type is JSON Lines. Files are uploaded to the default datastore and made available in your project.

+## Fine-tune a Phi-3 model

+# [Phi-3-mini](#tab/phi-3-mini)

+To fine-tune a Phi-3 model:

+1. Sign in to [Azure AI Studio](https://ai.azure.com).

+1. Choose the model you want to fine-tune from the Azure AI Studio [model catalog](https://ai.azure.com/explore/models).

+1. On the model's **Details** page, select **fine-tune**.

+1. Select the project in which you want to fine-tune your models. To use the pay-as-you-go model fine-tune offering, your workspace must belong to the **East US 2** region.

+1. On the fine-tune wizard, select the link to **Azure AI studio Terms** to learn more about the terms of use. You can also select the **Azure AI studio offer details** tab to learn about pricing for the selected model.

+1. If this is your first time fine-tuning the model in the project, you have to subscribe your project for the particular offering (for example, Phi-3-mini-128k-instruct) from Azure AI studio. This step requires that your account has the Azure subscription permissions and resource group permissions listed in the prerequisites. Each project has its own subscription to the particular Azure AI studio offering, which allows you to control and monitor spending. Select **Subscribe and fine-tune**.

+ > [!NOTE]

+ > Subscribing a project to a particular Azure AI studio offering (in this case, Phi-3-mini-128k-instruct) requires that your account has **Contributor** or **Owner** access at the subscription level where the project is created. Alternatively, your user account can be assigned a custom role that has the Azure subscription permissions and resource group permissions listed in the [prerequisites](#prerequisites).

+1. Once you sign up the project for the particular Azure AI studio offering, subsequent fine-tuning of the _same_ offering in the _same_ project don't require subscribing again. Therefore, you don't need to have the subscription-level permissions for subsequent fine-tune jobs. If this scenario applies to you, select **Continue to fine-tune**.

+1. Enter a name for your fine-tuned model and the optional tags and description.

+1. Select training data to fine-tune your model. See [data preparation](#data-preparation) for more information.

+ > [!NOTE]

+ > If the you have your training/validation files in a credential less datastore, you will need to allow workspace managed identity access to your datastore in order to proceed with MaaS fine-tuning with a credential less storage. On the "Datastore" page, after clicking "Update authentication" > Select the following option:

+

+ 

+ Make sure all your training examples follow the expected format for inference. To fine-tune models effectively, ensure a balanced and diverse dataset. This involves maintaining data balance, including various scenarios, and periodically refining training data to align with real-world expectations, ultimately leading to more accurate and balanced model responses.

+ - The batch size to use for training. When set to -1, batch_size is calculated as 0.2% of examples in training set and the max is 256.

+ - The fine-tuning learning rate is the original learning rate used for pretraining multiplied by this multiplier. We recommend experimenting with values between 0.5 and 2. Empirically, we've found that larger learning rates often perform better with larger batch sizes. Must be between 0.0 and 5.0.

+ - Number of training epochs. An epoch refers to one full cycle through the data set.

+1. Task parameters are an optional step and an advanced option- Tuning hyperparameter is essential for optimizing large language models (LLMs) in real-world applications. It allows for improved performance and efficient resource usage. Users can choose to keep he default settings or advanced users can customize parameters like epochs or learning rate.

+1. Review your selections and proceed to train your model.

+Once your model is fine-tuned, you can deploy the model and can use it in your own application, in the playground, or in prompt flow. For more information, see [How to deploy Phi-3 family of large language models with Azure AI Studio](./deploy-models-phi-3.md).

+# [Phi-3-medium](#tab/phi-3-medium)

+To fine-tune a Phi-3 model:

+1. Sign in to [Azure AI Studio](https://ai.azure.com).

+1. Choose the model you want to fine-tune from the Azure AI Studio [model catalog](https://ai.azure.com/explore/models).

+1. On the model's **Details** page, select **fine-tune**.

+1. Select the project in which you want to fine-tune your models. To use the pay-as-you-go model fine-tune offering, your workspace must belong to the **East US 2** region.

+1. On the fine-tune wizard, select the link to **Azure AI studio Terms** to learn more about the terms of use. You can also select the **Azure AI studio offer details** tab to learn about pricing for the selected model.

+1. If this is your first time fine-tuning the model in the project, you have to subscribe your project for the particular offering (for example, Phi-3-medium-128k-instruct) from Azure AI studio. This step requires that your account has the Azure subscription permissions and resource group permissions listed in the prerequisites. Each project has its own subscription to the particular Azure AI studio offering, which allows you to control and monitor spending. Select **Subscribe and fine-tune**.

+ > [!NOTE]

+ > Subscribing a project to a particular Azure AI studio offering (in this case, Phi-3-mini-128k-instruct) requires that your account has **Contributor** or **Owner** access at the subscription level where the project is created. Alternatively, your user account can be assigned a custom role that has the Azure subscription permissions and resource group permissions listed in the [prerequisites](#prerequisites).

+1. Once you sign up the project for the particular Azure AI studio offering, subsequent fine-tuning of the _same_ offering in the _same_ project don't require subscribing again. Therefore, you don't need to have the subscription-level permissions for subsequent fine-tune jobs. If this scenario applies to you, select **Continue to fine-tune**.

+1. Enter a name for your fine-tuned model and the optional tags and description.

+1. Select training data to fine-tune your model. See [data preparation](#data-preparation) for more information.

+ > [!NOTE]

+ > If you have your training/validation files in a credential less datastore, you will need to allow workspace managed identity access to your datastore in order to proceed with MaaS finetuning with a credential less storage. On the "Datastore" page, after clicking "Update authentication" > Select the following option:

+

+ 

+ Make sure all your training examples follow the expected format for inference. To fine-tune models effectively, ensure a balanced and diverse dataset. This involves maintaining data balance, including various scenarios, and periodically refining training data to align with real-world expectations, ultimately leading to more accurate and balanced model responses.

+ - The batch size to use for training. When set to -1, batch_size is calculated as 0.2% of examples in training set and the max is 256.

+ - The fine-tuning learning rate is the original learning rate used for pretraining multiplied by this multiplier. We recommend experimenting with values between 0.5 and 2. Empirically, we've found that larger learning rates often perform better with larger batch sizes. Must be between 0.0 and 5.0.

+ - Number of training epochs. An epoch refers to one full cycle through the data set.

+1. Task parameters are an optional step and an advanced option- Tuning hyperparameter is essential for optimizing large language models (LLMs) in real-world applications. It allows for improved performance and efficient resource usage. Users can choose to keep the default settings or advanced users can customize parameters like epochs or learning rate.

+1. Review your selections and proceed to train your model.

+Once your model is fine-tuned, you can deploy the model and can use it in your own application, in the playground, or in prompt flow. For more information, see [How to deploy Phi-3 family of large language models with Azure AI Studio](./deploy-models-phi-3.md).

+## Cleaning up your fine-tuned models

+You can delete a fine-tuned model from the fine-tuning model list in [Azure AI Studio](https://ai.azure.com) or from the model details page. Select the fine-tuned model to delete from the Fine-tuning page, and then select the Delete button to delete the fine-tuned model.

+>[!NOTE]

+> You can't delete a custom model if it has an existing deployment. You must first delete your model deployment before you can delete your custom model.

+## Cost and quotas

+### Cost and quota considerations for Phi-3 models fine-tuned as a service

+Phi models fine-tuned as a service are offered by Microsoft and integrated with Azure AI Studio for use. You can find the pricing when [deploying](./deploy-models-phi-3.md) or fine-tuning the models under the Pricing and terms tab on deployment wizard.

+## Content filtering

+Models deployed as a service with pay-as-you-go are protected by Azure AI Content Safety. When deployed to real-time endpoints, you can opt out of this capability. With Azure AI content safety enabled, both the prompt and completion pass through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions. Learn more about [Azure AI Content Safety](../concepts/content-filtering.md).

+## Next steps

+- [What is Azure AI Studio?](../what-is-ai-studio.md)

+- [Learn more about deploying Phi-3 models](./deploy-models-phi-3.md)

+- [Azure AI FAQ article](../faq.yml)

+ Title: LLMOps with prompt flow and Azure DevOps in Azure AI Studio

+description: Learn how to set up a LLMOps environment and pipeline on Azure DevOps for prompt flow project using Azure AI Studio.

+ - cli-v2

+ - sdk-v2

+ - ignite-2024

+ - build-2024

+# Streamlining LLMOps with Prompt Flow and Azure DevOps: A Comprehensive Approach

+Large Language Operations, or LLMOps, is the cornerstone of efficient prompt engineering and LLM-infused application development and deployment. As the demand for LLM-infused applications continues to soar, organizations find themselves in need of a cohesive and streamlined process to manage their end-to-end lifecycle.

+Azure AI Studio allows you to integrate with GitHub to automate the LLM-infused application development lifecycle with prompt flow.

+Azure AI Studio Prompt Flow provides a streamlined and structured approach to developing LLM-infused applications. Its well-defined process and lifecycle guides you through the process of building, testing, optimizing, and deploying flows, culminating in the creation of fully functional LLM-infused solutions.

+## LLMOps Prompt Flow Features

+LLMOps with prompt flow is a "LLMOps template and guidance" to help you build LLM-infused apps using prompt flow. It provides the following features:

+**Core Features**:

+- This template can be used for both **Azure AI Studio and Azure Machine Learning**.

+- It can be used for both **AZURE and LOCAL execution**.

+- It supports all types of flow - **Python Class flows, Function flows and YAML flows**.

+- It supports **Github, Azure DevOps and Jenkins CI/CD orchestration**.

+- It supports pure **python based Evaluation** as well using promptflow-evals package.

+- It supports INNER-LOOP Experimentation and Evaluation.

+- It supports OUTER-LOOP Deployment and Inferencing.

+- It supports **Centralized Code Hosting** for multiple flows based on prompt flow, providing a single repository for all your flows. Think of this platform as a single repository where all your prompt flow code resides. It's like a library for your flows, making it easy to find, access, and collaborate on different projects.

+- Each flow enjoys its own **Lifecycle Management**, allowing for smooth transitions from local experimentation to production deployment.

+ :::image type="content" source="../media/prompt-flow/llmops/workflow.png" alt-text="Screenshot of workflow." lightbox = "../media/prompt-flow/llmops/workflow.png":::

+- Experiment with multiple **Variant and Hyperparameter Experimentation**, evaluating flow variants with ease. Variants and hyperparameters are like ingredients in a recipe. This platform allows you to experiment with different combinations of variants across multiple nodes in a flow.

+- The repo supports deployment of flows to **Azure App Services, Kubernetes, Azure Managed computes** driven through configuration ensuring that your flows can scale as needed. It also generates **Docker images** infused with Flow compute session and your flows for deployment to **any target platform and Operating system** supporting Docker.

+ :::image type="content" source="../media/prompt-flow/llmops/endpoints.png" alt-text="Screenshot of endpoints." lightbox = "../media/prompt-flow/llmops/endpoints.png":::

+- Seamlessly implement **A/B Deployment**, enabling you to compare different flow versions effortlessly. As in traditional A/B testing for websites, this platform facilitates A/B deployment for prompt flow. This means you can effortlessly compare different versions of a flow in a real-world setting to determine which performs best.

+ :::image type="content" source="../media/prompt-flow/llmops/a-b-deployments.png" alt-text="Screenshot of deployments." lightbox = "../media/prompt-flow/llmops/a-b-deployments.png":::

+- Accommodates **Many-to-many dataset/flow relationships** for each standard and evaluation flow, ensuring versatility in flow test and evaluation. The platform is designed to accommodate multiple datasets for each flow.

+- It supports **Conditional Data and Model registration** by creating a new version for dataset in Azure AI Studio Data Asset and flows in model registry only when there's a change in them, not otherwise.

+- Generates **Comprehensive Reporting** for each **variant configuration**, allowing you to make informed decisions. Provides detailed Metric collection, experiment, and variant bulk runs for all runs and experiments, enabling data-driven decisions in csv as well as HTML files.

+ :::image type="content" source="../media/prompt-flow/llmops/variants.png" alt-text="Screenshot of flow variants report." lightbox = "../media/prompt-flow/llmops/variants.png":::

+Other features for customization:

+- Offers **BYOF** (bring-your-own-flows). A **complete platform** for developing multiple use-cases related to LLM-infused applications.

+- Offers **configuration based development**. No need to write extensive boiler-plate code.

+- Provides execution of both **prompt experimentation and evaluation** locally as well on cloud.

+- Endpoint testing within pipeline after deployment to check its availability and readiness.

+- Provides optional Human-in-loop to validate prompt metrics before deployment.

+LLMOps with prompt flow provides capabilities for both simple and complex LLM-infused apps. It's customizable to the needs of the application.

+## LLMOps Stages

+The lifecycle comprises four distinct stages:

+1. **Initialization:** Clearly define the business objective, gather relevant data samples, establish a basic prompt structure, and craft a flow that enhances its capabilities.

+2. **Experimentation:** Apply the flow to sample data, assess the prompt's performance, and refine the flow as needed. Continuously iterate until satisfied with the results.

+3. **Evaluation & Refinement:** Benchmark the flow's performance using a larger dataset, evaluate the prompt's effectiveness, and make refinements accordingly. Progress to the next stage if the results meet the desired standards.

+4. **Deployment:** Optimize the flow for efficiency and effectiveness, deploy it in a production environment including A/B deployment, monitor its performance, gather user feedback, and use this information to further enhance the flow.

+By adhering to this structured methodology, prompt flow empowers you to confidently develop, rigorously test, fine-tune, and deploy flows, leading to the creation of robust and sophisticated AI applications.

+LLMOps prompt flow template formalizes this structured methodology using code-first approach and helps you build LLM-infused apps using prompt flow using tools and process relevant to prompt flow. It offers a range of features including Centralized Code Hosting, Lifecycle Management, Variant and Hyperparameter Experimentation, A/B Deployment, reporting for all runs and experiments and more.

+The repository for this article is available at [LLMOps with Prompt flow template](https://github.com/microsoft/llmops-promptflow-template)

+## LLMOps process Flow

+1. The prompt engineer/data scientist opens a feature branch where they work on the specific task or feature. The prompt engineer/data scientist iterates on the flow using prompt flow for Microsoft Visual Studio Code, periodically committing changes and pushing those changes to the feature branch.

+2. Once local development and experimentation are completed, the prompt engineer/data scientist opens a pull request from the Feature branch to the Main branch. The pull request (PR) triggers a PR pipeline. This pipeline runs fast quality checks that should include:

+ - Execution of experimentation flows

+ - Execution of configured unit tests

+ - Compilation of the codebase

+ - Static code analysis

+3. The pipeline can contain a step that requires at least one team member to manually approve the PR before merging. The approver can't be the committer and they mush have prompt flow expertise and familiarity with the project requirements. If the PR isn't approved, the merge is blocked. If the PR is approved, or there's no approval step, the feature branch is merged into the Main branch.

+4. The merge to Main triggers the build and release process for the Development environment. Specifically:

+ a. The CI pipeline is triggered from the merge to Main. The CI pipeline performs all the steps done in the PR pipeline, and the following steps:

+ 1. Experimentation flow

+ 2. Evaluation flow

+ 3. Registers the flows in the AI Studio Registry when changes are detected

+ b. The CD pipeline is triggered after the completion of the CI pipeline. This flow performs the following steps:

+ 1. Deploys the flow from the AI Studio registry to a AI Studio deployment

+ 2. Runs integration tests that target the online endpoint

+ 3. Runs smoke tests that target the online endpoint

+5. An approval process is built into the release promotion process ΓÇô upon approval, the CI & CD processes described in steps 4.a. & 4.b. are repeated, targeting the Test environment. Steps 4.a. and 4.b. are the same, except that user acceptance tests are run after the smoke tests in the Test environment.

+6. The CI & CD processes described in steps 4.a. & 4.b. are run to promote the release to the Production environment after the Test environment is verified and approved.

+7. After release into a live environment, the operational tasks of monitoring performance metrics and evaluating the deployed language models take place. This includes but isn't limited to:

+ - Detecting data drifts

+ - Observing the infrastructure

+ - Managing costs

+ - Communicating the model's performance to stakeholders

+From here on, you can learn **LLMOps with prompt flow** by following the end-to-end samples we provided, which help you build LLM-infused applications using prompt flow and Azure DevOps. Its primary objective is to provide assistance in the development of such applications, leveraging the capabilities of prompt flow and LLMOps.

+## Prerequisites

+- An Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the [Azure AI Studio](https://azure.microsoft.com/free/).

+- An Azure AI Studio Hub and Project.

+- Git running on your local machine.

+- An [organization](/azure/devops/organizations/accounts/create-organization) in Azure DevOps. Organization in Azure DevOps helps to collaborate, Plan and track your work and code defects, issues and Set up continuous integration and deployment.

+- GitHub as the source control repository.

+> [!NOTE]

+>

+>Git version 2.27 or newer is required. For more information on installing the Git command, see https://git-scm.com/downloads and select your operating system

+> [!IMPORTANT]

+>The CLI commands in this article were tested using Bash. If you use a different shell, you may encounter errors.

+## Set up prompt flow

+Prompt flow uses connections resource to connect to endpoints like Azure OpenAI, OpenAI or Azure AI Search and uses compute session for the execution of the flows. These resources should be created before executing the flows in prompt flow.

+### Set up connections for prompt flow

+Connections can be created through **prompt flow portal UI** or using the **REST API**. Please follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/Azure_devops_how_to_setup.md#setup-connections-for-prompt-flow) to create connections for prompt flow.

+> [!NOTE]

+>

+> The sample flows use 'aoai' connection and connection named 'aoai' should be created to execute them.

+## Set up Azure Service Principal

+An **Azure Service Principal** is a security identity that applications, services, and automation tools use to access Azure resources. It represents an application or service that needs to authenticate with Azure and access resources on your behalf. Please follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/Azure_devops_how_to_setup.md#create-azure-service-principal) to create Service Principal in Azure.

+This Service Principal is later used to configure Azure DevOps Service connection and Azure DevOps to authenticate and connect to Azure Services. The jobs executed in Prompt Flow for both `experiment and evaluation runs` are under the identity of this Service Principal.

+> [!TIP]

+>

+>The set up provides `owner` permissions to the Service Principal.

+> * This is because the CD Pipeline automatically provides access to the newly provisioned AzureAI Studio Endpoint access to Azure AI Studio for reading connections information.

+> * It also adds it to Azure AI Studio associated key vault policy with `get` and `list` secret permissions.

+>

+>The owner permission can be changed to `contributor` level permissions by changing pipeline YAML code and removing the step related to permissions.

+## Set up Azure DevOps

+There are multiple steps that should be undertaken for setting up LLMOps process using Azure DevOps.

+### Create new Azure DevOps project

+Please follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/Azure_devops_how_to_setup.md#create-new-azure-devops-project) to create a new Azure DevOps project using **Azure DevOps UI**.

+### Set up authentication between Azure DevOps and Azure

+Please follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/Azure_devops_how_to_setup.md#set-up-authentication-with-azure-and-azure-devops) to use the earlier created [Service Principal](#set-up-azure-service-principal) and set up authentication between Azure DevOps and Azure Services.

+This step configures a new Azure DevOps Service Connection that stores the Service Principal information. The pipelines in the project can read the connection information using the connection name. This helps to configure Azure DevOps pipeline steps to connect to Azure automatically.

+### Create an Azure DevOps Variable Group

+Please follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/Azure_devops_how_to_setup.md#create-an-azure-devops-variable-group) to create a new Variable group and add a variable related to the Azure DevOps Service Connection.

+The Service principal name is available automatically as environment variable to the pipelines.

+### Configure Azure DevOps repository and pipelines

+This repo uses two branches - `main` and `development` for code promotions and execution of pipelines in lieu of changes to code in them. Please follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/Azure_devops_how_to_setup.md#configure-azure-devops-local-and-remote-repository) to set up your own local as well as remote repository to use code from this repository.

+The steps involve cloning both the `main` and `development branches` from the repository and associating the code to refer to the new Azure DevOps repository. Apart from code migration, pipelines - both PR and dev pipelines are configured such that they are executed automatically based on PR creation and merge triggers.

+The branch policy for development branch should also be configured to execute PR pipeline for any PR raised on development branch from a feature branch. The 'dev' pipeline is executed when the PR is merged to the development branch. The 'dev' pipeline consists of both CI and CD phases.

+There is also **human in the loop** implemented within the pipelines. After the CI phase in `dev` pipeline is executed, the CD phase follows after manual approval. The approval should happen from Azure DevOps pipeline build execution UI. The default time-out is 60 minutes after which the pipeline will be rejected and CD phase will not execute. Manually approving the execution will lead to execution of the CD steps of the pipeline. The manual approval is configured to send notifications to 'replace@youremail.com'. It should be replaced with an appropriate email ID.

+## Test the pipelines

+Please follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/Azure_devops_how_to_setup.md#test-the-pipelines) mentioned at to test the pipelines.

+The steps are:

+1. Raise a PR(Pull Request) from a feature branch to development branch.

+2. The PR pipeline should execute automatically as result of branch policy configuration.

+3. The PR is then merged to the development branch.

+4. The associated 'dev' pipeline is executed. It results in full CI and CD execution and result in provisioning or updating of existing Azure AI Studio Deployment.

+The test outputs should be similar to ones shown at [here](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/Azure_devops_how_to_setup.md#example-prompt-run-evaluation-and-deployment-scenario).

+## Local execution

+To harness the capabilities of the **local execution**, follow these installation steps:

+1. **Clone the Repository**: Begin by cloning the template's repository from its [GitHub repository](https://github.com/microsoft/llmops-promptflow-template.git).

+```bash

+git clone https://github.com/microsoft/llmops-promptflow-template.git

+```

+2. **Set up env file**: create .env file at top folder level and provide information for items mentioned. Some samples are shown next

+```bash

+SUBSCRIPTION_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

+AZURE_OPENAI_API_KEY=xxxxxxxxxxxxx

+AZURE_OPENAI_ENDPOINT=https://xxxxxxx

+MODEL_CONFIG_AZURE_ENDPOINT=https://xxxxxxx

+MODEL_CONFIG_API_KEY=xxxxxxxxxxx

+MAX_TOTAL_TOKEN=4096

+AOAI_API_KEY=xxxxxxxxxx

+AOAI_API_BASE=https://xxxxxxxxx

+```

+3. Prepare the local conda or virtual environment to install the dependencies.

+```bash

+python -m pip install -r ./.github/requirements/execute_job_requirements.txt

+```

+4. Change the value of `EXECUTION_TYPE` to `LOCAL` in `config.py` file located within `llmops/` directory.

+```python

+EXECUTION_TYPE = "LOCAL"

+```

+```bash

+python -m llmops.common.prompt_pipeline --subscription_id xxxx --base_path math_coding --env_name dev --output_file run_id.txt --build_id 100

+```

+Evaluations can be run using the `prompt_eval.py` python script locally.

+```bash

+python -m llmops.common.prompt_eval --run_id run_id.txt --subscription_id xxxxx --base_path math_coding --env_name dev --build_id 100

+```

+5. Bring or write your flows into the template based on documentation [here](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/how_to_onboard_new_flows.md).

+## Next steps

+* [LLMOps with Prompt flow template](https://github.com/microsoft/llmops-promptflow-template/) on GitHub

+* [LLMOps with Prompt flow template documentation](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/Azure_devops_how_to_setup.md) on GitHub

+* [FAQS for LLMOps with Prompt flow template](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/faqs.md)

+* [Prompt flow open source repository](https://github.com/microsoft/promptflow)

+* [Install and set up Python SDK v2](/python/api/overview/azure/ai-ml-readme)

+ Title: LLMOps with prompt flow and GitHub in Azure AI Studio

+description: Learn how to set up a LLMOps environment and workflows in Github for prompt flow project using Azure AI Studio.

+ - cli-v2

+ - sdk-v2

+ - ignite-2024

+ - build-2024

+# Elevating LLMOps with Prompt Flow and GitHub: A Unified Strategy for AI Workflows

+Large Language Operations, or LLMOps, is the cornerstone of efficient prompt engineering and LLM-infused application development and deployment. As the demand for LLM-infused applications continues to soar, organizations find themselves in need of a cohesive and streamlined process to manage their end-to-end lifecycle.

+Azure AI Studio allows you to integrate with GitHub to automate the LLM-infused application development lifecycle with prompt flow.

+Azure AI Studio Prompt Flow provides a streamlined and structured approach to developing LLM-infused applications. Its well-defined process and lifecycle guides you through the process of building, testing, optimizing, and deploying flows, culminating in the creation of fully functional LLM-infused solutions.

+## LLMOps Prompt Flow Features

+LLMOps with prompt flow is a "LLMOps template and guidance" to help you build LLM-infused apps using prompt flow. It provides the following features:

+- This template can be used for both **Azure AI Studio and Azure Machine Learning**.

+- It can be used for both **AZURE and LOCAL execution**.

+- It supports all types of flow - **Python Class flows, Function flows and YAML flows**.

+- It supports **Github, Azure DevOps and Jenkins CI/CD orchestration**.

+- It supports pure **python based Evaluation** as well using promptflow-evals package.

+- It supports INNER-LOOP Experimentation and Evaluation.

+- It supports OUTER-LOOP Deployment and Inferencing.

+- It supports **Centralized Code Hosting** for multiple flows based on prompt flow, providing a single repository for all your flows. Think of this platform as a single repository where all your prompt flow code resides. It's like a library for your flows, making it easy to find, access, and collaborate on different projects.

+- Each flow enjoys its own **Lifecycle Management**, allowing for smooth transitions from local experimentation to production deployment.

+ :::image type="content" source="../media/prompt-flow/llmops/workflow.png" alt-text="Screenshot of workflow." lightbox = "../media/prompt-flow/llmops/workflow.png":::

+- Experiment with multiple **Variant and Hyperparameter Experimentation**, evaluating flow variants with ease. Variants and hyperparameters are like ingredients in a recipe. This platform allows you to experiment with different combinations of variants across multiple nodes in a flow.

+- The repo supports deployment of flows to **Azure App Services, Kubernetes, Azure Managed computes** driven through configuration ensuring that your flows can scale as needed. It also generates **Docker images** infused with Flow compute session and your flows for deployment to **any target platform and Operating system** supporting Docker.

+ :::image type="content" source="../media/prompt-flow/llmops/endpoints.png" alt-text="Screenshot of endpoints." lightbox = "../media/prompt-flow/llmops/endpoints.png":::

+- Seamlessly implement **A/B Deployment**, enabling you to compare different flow versions effortlessly. As in traditional A/B testing for websites, this platform facilitates A/B deployment for prompt flow. This means you can effortlessly compare different versions of a flow in a real-world setting to determine which performs best.

+ :::image type="content" source="../media/prompt-flow/llmops/a-b-deployments.png" alt-text="Screenshot of deployments." lightbox = "../media/prompt-flow/llmops/a-b-deployments.png":::

+- Accommodates **Many-to-many dataset/flow relationships** for each standard and evaluation flow, ensuring versatility in flow test and evaluation. The platform is designed to accommodate multiple datasets for each flow.

+- It supports **Conditional Data and Model registration** by creating a new version for dataset in Azure AI Studio Data Asset and flows in model registry only when there's a change in them, not otherwise.

+- Generates **Comprehensive Reporting** for each **variant configuration**, allowing you to make informed decisions. Provides detailed Metric collection, experiment, and variant bulk runs for all runs and experiments, enabling data-driven decisions in csv as well as HTML files.

+ :::image type="content" source="../media/prompt-flow/llmops/variants.png" alt-text="Screenshot of flow variants report." lightbox = "../media/prompt-flow/llmops/variants.png":::

+Other features for customization:

+- Offers **BYOF** (bring-your-own-flows). A **complete platform** for developing multiple use-cases related to LLM-infused applications.

+- Offers **configuration based development**. No need to write extensive boiler-plate code.

+- Provides execution of both **prompt experimentation and evaluation** locally as well on cloud.

+- Endpoint testing within workflow after deployment to check its availability and readiness.

+- Provides optional Human-in-loop to validate prompt metrics before deployment.

+LLMOps with prompt flow provides capabilities for both simple as well as complex LLM-infused apps. It's customizable to the needs of the application.

+## LLMOps Stages

+The lifecycle comprises four distinct stages:

+1. **Initialization:** Clearly define the business objective, gather relevant data samples, establish a basic prompt structure, and craft a flow that enhances its capabilities.

+2. **Experimentation:** Apply the flow to sample data, assess the prompt's performance, and refine the flow as needed. Continuously iterate until satisfied with the results.

+3. **Evaluation & Refinement:** Benchmark the flow's performance using a larger dataset, evaluate the prompt's effectiveness, and make refinements accordingly. Progress to the next stage if the results meet the desired standards.

+4. **Deployment:** Optimize the flow for efficiency and effectiveness, deploy it in a production environment including A/B deployment, monitor its performance, gather user feedback, and use this information to further enhance the flow.

+By adhering to this structured methodology, Prompt Flow empowers you to confidently develop, rigorously test, fine-tune, and deploy flows, leading to the creation of robust and sophisticated AI applications.

+LLMOps Prompt Flow template formalizes this structured methodology using code-first approach and helps you build LLM-infused apps using Prompt Flow using tools and process relevant to Prompt Flow. It offers a range of features including Centralized Code Hosting, Lifecycle Management, Variant and Hyperparameter Experimentation, A/B Deployment, reporting for all runs and experiments and more.

+The repository for this article is available at [LLMOps with Prompt flow template.](https://github.com/microsoft/llmops-promptflow-template)

+## LLMOps process Flow

+1. The prompt engineer/data scientist opens a feature branch where they work on the specific task or feature. The prompt engineer/data scientist iterates on the flow using prompt flow for Microsoft Visual Studio Code, periodically committing changes and pushing those changes to the feature branch.

+2. Once local development and experimentation are completed, the prompt engineer/data scientist opens a pull request from the Feature branch to the Main branch. The pull request (PR) triggers a PR workflow. This workflow runs fast quality checks that should include:

+ 1. Execution of experimentation flows

+ 2. Execution of configured unit tests

+ 3. Compilation of the codebase

+ 4. Static code analysis

+3. The workflow can contain a step that requires at least one team member to manually approve the PR before merging. The approver can't be the committer and they mush have prompt flow expertise and familiarity with the project requirements. If the PR isn't approved, the merge is blocked. If the PR is approved, or there's no approval step, the feature branch is merged into the Main branch.

+4. The merge to Main triggers the build and release process for the Development environment. Specifically:

+ a. The CI workflow is triggered from the merge to Main. The CI workflow performs all the steps done in the PR workflow, and the following steps:

+ 1. Experimentation flow

+ 2. Evaluation flow

+ 3. Registers the flows in the AI Studio Registry when changes are detected

+ b. The CD workflow is triggered after the completion of the CI workflow. This flow performs the following steps:

+ 1. Deploys the flow from the Machine Learning registry to a AI Studio Deployment

+ 2. Runs integration tests that target the online endpoint

+ 3. Runs smoke tests that target the online endpoint

+5. An approval process is built into the release promotion process ΓÇô upon approval, the CI & CD processes described in steps 4.a. & 4.b. are repeated, targeting the Test environment. Steps a. and b. are the same, except that user acceptance tests are run after the smoke tests in the Test environment.

+6. The CI & CD processes described in steps 4.a. & 4.b. are run to promote the release to the Production environment after the Test environment is verified and approved.

+7. After release into a live environment, the operational tasks of monitoring performance metrics and evaluating the deployed language models take place. This includes but isn't limited to:

+ - Detecting data drifts

+ - Observing the infrastructure

+ - Managing costs

+ - Communicating the model's performance to stakeholders

+From here on, you can learn **LLMOps with prompt flow** by following the end-to-end samples we provided, which help you build LLM-infused applications using prompt flow and GitHub. Its primary objective is to provide assistance in the development of such applications, using the capabilities of prompt flow and LLMOps.

+## Prerequisites

+- An Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the [Azure AI Studio](https://azure.microsoft.com/free/).

+- An Azure AI Studio Hub and Project.

+- Git running on your local machine.

+- GitHub as the source control repository.

+> [!NOTE]

+>

+>Git version 2.27 or newer is required. For more information on installing the Git command, see https://git-scm.com/downloads and select your operating system

+> [!IMPORTANT]

+>The CLI commands in this article were tested using Bash. If you use a different shell, you may encounter errors.

+## Set up Prompt Flow

+Prompt Flow uses connections resource to connect to endpoints like Azure OpenAI, OpenAI, or Azure AI Search and uses compute session for the execution of the flows. These resources should be created before executing the flows in Prompt Flow.

+### Set up connections for prompt flow

+Connections can be created through **prompt flow portal UI** or using the **REST API**. Follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/Azure_devops_how_to_setup.md#setup-connections-for-prompt-flow) to create connections for prompt flow.

+> [!NOTE]

+>

+> The sample flows use 'aoai' connection and connection named 'aoai' should be created to execute them.

+## Set up GitHub Repository

+There are multiple steps that should be undertaken for setting up LLMOps process using GitHub Repository.

+### Fork and configure the repo

+Follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/github_workflows_how_to_setup.md#set-up-github-repo) to create a forked repo in your GitHub organization. This repo uses two branches - `main` and `development` for code promotions and execution of workflows in lieu of changes to code in them.

+### Set up authentication between GitHub and Azure

+Follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/github_workflows_how_to_setup.md#set-up-authentication-with-azure-and-github) to use the earlier created Service Principal and set up authentication between GitHub repository and Azure Services.

+This step configures a GitHub Secret that stores the Service Principal information. The workflows in the repository can read the connection information using the secret name. This helps to configure GitHub workflow steps to connect to Azure automatically.

+### Cloning the repo

+Follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/github_workflows_how_to_setup.md#cloning-the-repo) to create a new local repository.

+This helps us create a new feature branch from development branch and incorporate changes.

+## Test the workflows

+Follow the [guidelines](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/github_workflows_how_to_setup.md#cloning-the-repos) to test the workflows. The steps are

+1. Raise a PR(Pull Request) from a feature branch to development branch.

+2. The PR workflow should execute automatically as result of branch policy configuration.

+3. The PR is then merged to the development branch.

+4. The associated 'dev' workflow is executed. This results in full CI and CD execution and result in provisioning or updating of existing Azure AI Studio Deployment.

+The test outputs should be similar to ones shown at [here](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/github_workflows_how_to_setup.md#example-prompt-run-evaluation-and-deployment-scenario).

+## Local execution

+To harness the capabilities of the **local execution**, follow these installation steps:

+1. **Clone the Repository**: Begin by cloning the template's repository from its [GitHub repository](https://github.com/microsoft/llmops-promptflow-template.git).

+```bash

+git clone https://github.com/microsoft/llmops-promptflow-template.git

+```

+2. **Set up env file**: create .env file at top folder level and provide information for items mentioned. Some samples are shown next

+```bash

+SUBSCRIPTION_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

+AZURE_OPENAI_API_KEY=xxxxxxxxxxxxx

+AZURE_OPENAI_ENDPOINT=https://xxxxxxx

+MODEL_CONFIG_AZURE_ENDPOINT=https://xxxxxxx

+MODEL_CONFIG_API_KEY=xxxxxxxxxxx

+MAX_TOTAL_TOKEN=4096

+AOAI_API_KEY=xxxxxxxxxx

+AOAI_API_BASE=https://xxxxxxxxx

+```

+3. Prepare the local conda or virtual environment to install the dependencies.

+```bash

+python -m pip install -r ./.github/requirements/execute_job_requirements.txt

+```

+4. Change the value of `EXECUTION_TYPE` to `LOCAL` in `config.py` file located within `llmops/` directory.

+```python

+EXECUTION_TYPE = "LOCAL"

+```

+```bash

+python -m llmops.common.prompt_pipeline --subscription_id xxxx --base_path math_coding --env_name dev --output_file run_id.txt --build_id 100

+```

+Evaluations can be run using the `prompt_eval.py` python script locally.

+```bash

+python -m llmops.common.prompt_eval --run_id run_id.txt --subscription_id xxxxx --base_path math_coding --env_name dev --build_id 100

+```

+5. Bring or write your flows into the template based on documentation [here](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/how_to_onboard_new_flows.md).

+## Next steps

+* [LLMOps with Prompt flow template](https://github.com/microsoft/llmops-promptflow-template/) on GitHub

+* [LLMOps with Prompt flow template documentation](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/github_workflows_how_to_setup.md) on GitHub

+* [FAQS for LLMOps with Prompt flow template](https://github.com/microsoft/llmops-promptflow-template/blob/main/docs/faqs.md)

+* [Prompt flow open source repository](https://github.com/microsoft/promptflow)

+* [Install and set up Python SDK v2](/python/api/overview/azure/ai-ml-readme)

+Cohere family models | Not available | Cohere-command-r-plus <br> Cohere-command-r <br> Cohere-embed-v3-english <br> Cohere-embed-v3-multilingual <br> Cohere-rerank-3-english <br> Cohere-rerank-3-multilingual

+Cohere-rerank-3-english <br> Cohere-rerank-3-multilingual | [Microsoft Managed Countries](/partner-center/marketplace/tax-details-marketplace#microsoft-managed-countriesregions) | East US, East US 2, North Central US, South Central US, Sweden Central, West US, West US 3 | Not available

+ --sku automatic

+## Revert to original values

+If you want to restore the buffer size to its default value of *0.2 MB*, you can update the `linuxosconfig.json` file with the following values:

+```json

+{

+ "sysctls": {

+ "netCoreRmemMax": 212992,

+ "netCoreRmemDefault": 212992

+ }

+}

+```

+* It creates an API version named according to the `version` property in the API definition (or *1-0-0* by default), and an API definition named according to the specification format (for example, *openapi*).

+* [Register APIs in your API center using GitHub Actions](register-apis-github-actions.md)

+ Title: Register APIs using GitHub Actions - Azure API Center

+description: Learn how to automate the registration of APIs in your API center using a CI/CD workflow based on GitHub Actions.

+# Customer intent: As an API developer, I want to automate the registration of APIs in my API center using a CI/CD workflow based on GitHub Actions.

+# Register APIs in your API center using GitHub Actions

+This article shows how to set up a basic [GitHub Actions](https://docs.github.com/en/actions/learn-github-actions) workflow to register an API in your organization's [API center](overview.md) when an API specification file is added to a GitHub repository.

+Using a GitHub Actions workflow to register APIs in your API center provides a consistent and repeatable CI/CD process for every new or updated API. The workflow can be extended to include steps such as adding metadata to the API registration.

+The following diagram shows how API registration in your API center can be automated using a GitHub Actions workflow.

+1. Set up a GitHub Actions workflow in your repository that triggers when a pull request that adds an API definition file is merged.

+1. Create a branch from the main branch in your GitHub repository.

+1. Add an API definition file, commit the changes, and push them to the new branch.

+1. Create a pull request to merge the new branch into the main branch.

+1. Merge the pull request.

+1. The merge triggers a GitHub Actions workflow that registers the API in your API center.

+## Prerequisites

+* An API center in your Azure subscription. If you haven't created one already, see [Quickstart: Create your API center](set-up-api-center.md).

+* Permissions to create a service principal in the Microsoft Entra ID tenant

+* A GitHub account and a GitHub repo in which you can configure secrets and GitHub Actions workflows

+* For Azure CLI:

+ [!INCLUDE [include](~/reusable-content/azure-cli/azure-cli-prepare-your-environment-no-header.md)]

+ [!INCLUDE [install-apic-extension](includes/install-apic-extension.md)]

+ > [!NOTE]

+ > Azure CLI command examples in this article can run in PowerShell or a bash shell. Where needed because of different variable syntax, separate command examples are provided for the two shells.

+## Set up a GitHub Actions workflow

+In this section, you set up the GitHub Actions workflow for this scenario:

+* Create a service principal to use for Azure credentials in the workflow.

+* Add the credentials as a secret in your GitHub repository.

+* Configure a GitHub Actions workflow that triggers when a pull request that adds an API definition file is merged. The workflow YAML file includes a step that uses the Azure CLI to register the API in your API center from the definition file.

+### Set up a service principal secret

+In the following steps, create a Microsoft Entra ID service principal, which will be used to add credentials to the workflow to authenticate with Azure.

+> [!NOTE]

+> Configuring a service principal is shown for demonstration purposes. The recommended way to authenticate with Azure for GitHub Actions is with OpenID Connect, an authentication method that uses short-lived tokens. Setting up OpenID Connect with GitHub Actions is more complex but offers hardened security. [Learn more](../app-service/deploy-github-actions.md?tabs=openid%2Caspnetcore#1-generate-deployment-credentials)

+Create a service principal using the [az ad sp create-for-rbac](/cli/azure/ad#az-ad-sp-create-for-rbac) command. The following example first uses the [az apic show](/cli/azure/apic#az-apic-show) command to retrieve the resource ID of the API center. The service principal is then created with the Contributor role for the API center.

+#### [Bash](#tab/bash)

+```azurecli

+#! /bin/bash

+apiCenter=<api-center-name>

+resourceGroup=<resource-group-name>

+spName=<service-principal-name>

+apicResourceId=$(az apic show --name $apiCenter --resource-group $resourceGroup --query "id" --output tsv)

+az ad sp create-for-rbac --name $spName --role Contributor --scopes $apicResourceId --json-auth

+```

+#### [PowerShell](#tab/powershell)

+```azurecli

+# PowerShell syntax

+$apiCenter = "<api-center-name>"

+$resourceGroup = "<resource-group-name>"

+$spName = "<service-principal-name>"

+$apicResourceId = $(az apic show --name $apiCenter --resource-group $resourceGroup --query "id" --output tsv)

+az ad sp create-for-rbac --name $spName --role Contributor --scopes $apicResourceId --json-auth

+```

+Copy the JSON output, which should look similar to the following:

+```json

+{

+ "clientId": "<GUID>",

+ "clientSecret": "<GUID>",

+ "subscriptionId": "<GUID>",

+ "tenantId": "<GUID>",

+ "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",

+ "resourceManagerEndpointUrl": "https://management.azure.com/",

+ [...other endpoints...]

+}

+```

+### Add the service principal as a GitHub secret

+1. In [GitHub](https://github.com/), browse your repository. Select **Settings**.

+1. Under **Security**, select **Secrets and variables** > **Actions**

+1. Select **New repository secret**.

+1. Paste the entire JSON output from the Azure CLI command into the secret's value field. Name the secret `AZURE_CREDENTIALS`. Select **Add secret**.

+ The secret is listed under **Repository secrets**.

+ :::image type="content" source="media/register-apis-github-actions/repository-secrets-github-small.png" alt-text="Screenshot of secrets for Actions in a GitHub repository." lightbox="media/register-apis-github-actions/repository-secrets-github.png":::

+When you configure the GitHub workflow file later, you use the secret for the input `creds` of the [Azure/login](https://github.com/marketplace/actions/azure-login) action. For example:

+```yaml

+- uses: azure/login@v1

+ with:

+ creds: ${{ secrets.AZURE_CREDENTIALS }}

+```

+### Add the workflow file to your GitHub repository

+A GitHub Actions workflow is represented by a YAML (.yml) definition file. This definition contains the various steps and parameters that make up the workflow. [Learn more](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)

+The following is a basic workflow file for this example that you can use or modify.

+In this example:

+* The workflow is triggered when a pull request that adds a JSON definition in the `APIs` path is closed on the main branch.

+* The location of the definition is extracted from the pull request using a GitHub script, which is authenticated with the default GitHub token.

+* The Azure credentials saved in your repo are used to sign into Azure.

+* The [az apic register](/cli/azure/apic/api#az-apic-api-register) command registers the API in the API center specified in the environment variables.

+To configure the workflow file:

+1. Copy and save the file under a name such as `register-api.yml`.

+1. Update the values for the environment variables to match your API center in Azure.

+1. Confirm or update the name of the repository folder (`APIs`) where you'll add the API definition file.

+1. Add this workflow file in the `/.github/workflows/` path in your GitHub repository.

+> [!TIP]

+> Using the [Visual Studio Code extension](use-vscode-extension.md) for Azure API Center, you can generate a starting workflow file by running an extension command. In the Command Palette, select **Azure API Center: Register APIs**. Select **CI/CD** > **GitHub**. You can then modify the file for your scenario.

+```yml

+name: Register API Definition to Azure API Center

+on:

+ pull_request:

+ types: [closed]

+ branches:

+ - main

+ paths:

+ - "APIs/**/*.json"

+permissions:

+ contents: read

+ pull-requests: read

+env:

+ # set this to your Azure API Center resource group name

+ RESOURCE_GROUP: <YOUR_RESOURCE_GROUP>

+ # set this to your Azure API Center service name

+ SERVICE_NAME: <YOUR_API_CENTER>

+jobs:

+ register:

+ runs-on: ubuntu-latest

+ environment: production

+ steps:

+ - uses: actions/checkout@v2

+

+ - name: Get specification file path in the PR

+ id: get-file-location

+ uses: actions/github-script@v5

+ with:

+ github-token: ${{ secrets.GITHUB_TOKEN }}

+ script: |

+ const pull_number = context.payload.pull_request.number;

+ const owner = context.repo.owner;

+ const repo = context.repo.repo;

+ const files = await github.rest.pulls.listFiles({

+ owner,

+ repo,

+ pull_number

+ });

+ if (files.data.length === 1) {

+ const filename = files.data[0].filename;

+ core.exportVariable('API_FILE_LOCATION', hfilename);

+ console.log(`API_FILE_LOCATION: ${{ env.API_FILE_LOCATION }}`);

+ }

+ else {

+ console.log('The PR does not add exactly one specification file.');

+ }

+ - name: Azure login

+ uses: azure/login@v1

+ with:

+ creds: ${{ secrets.AZURE_CREDENTIALS }}

+

+ - name: Register to API Center

+ uses: azure/CLI@v2

+ with:

+ azcliversion: latest

+ inlineScript: |

+ az apic api register -g ${{ env.RESOURCE_GROUP }} -n ${{ env.SERVICE_NAME }} --api-location ${{ env.API_FILE_LOCATION }}

+```

+## Add API definition file to the repository

+Test the workflow by adding an API definition file to the repository. Follow these high-level steps, which are typical of a development workflow. For details on working with GitHub branches, see the [GitHub documentation](https://docs.github.com/en/github/collaborating-with-pull-requests/getting-started/about-collaborative-development-models).

+1. Create a new working branch from the main branch in your repository.

+1. Add the API definition file to the repository in the `APIs` path. For example, `APIs/catfacts-api/07-15-2024.json`.

+1. Commit the changes and push them to the working branch.

+1. Create a pull request to merge the working branch into the main branch.

+1. After review, merge the pull request. The merge triggers the GitHub Actions workflow that registers the API in your API center.

+ :::image type="content" source="media/register-apis-github-actions/workflow-action.png" alt-text="Screenshot showing successful workflow run in GitHub.":::

+## Verify the API registration

+Verify that the API is registered in your API center.

+1. In the [Azure portal](https://portal.azure.com), navigate to your API center.

+1. In the left menu, under **Assets**, select **APIs**.

+1. Verify that the newly registered API appears in the list of APIs.

+## Add a new API version

+To add a new version to an existing API in your API center, follow the preceding steps, with a slight modification:

+1. Change to the same working branch in your repo, or create a new working branch.

+1. Add a new API definition file to the repository in the `APIs` path, in the folder for an existing API. For example, if you previously added a Cat Facts API definition, add a new version such as `APIs/catfacts-api/07-22-2024.json`.

+1. Commit the changes and push them to the working branch.

+1. Create a pull request to merge the working branch into the main branch.

+1. After review, merge the pull request. The merge triggers the GitHub Actions workflow that registers the new API version in your API center.

+1. In the Azure portal, navigate to your API center and confirm that the new version is registered.

+## Extend the scenario

+You can extend the GitHub Actions workflow to include other steps, such as adding metadata for the API registration. For example:

+1. Using the [metadata schema](metadata.md) in your API center, create a metadata JSON file to apply metadata values to your API registration.

+ For example, if the metadata schema includes properties such as `approver`, `team`, and `cost center`, a metadata JSON file might look like this:

+ ```json

+ {

+ "approver": "diego@contoso.com",

+ "team": "Store API dev team",

+ "costCenter": "12345"

+ }

+ ```

+1. Upload a metadata JSON file in the folder for each API in the repository.

+1. Add a workflow step to apply the metadata to the API registration using the [az apic api update](/cli/azure/apic/api#az-apic-api-update) command. In the following example, the API ID and metadata file are passed in environment variables, which would be set elsewhere in the workflow file.

+ ```yml

+ [...]

+ - name: Apply metadata to API in API Center

+ uses: azure/CLI@v2

+ with:

+ azcliversion: latest

+ inlineScript: |

+ az apic api update -g ${{ env.RESOURCE_GROUP }} -n ${{ env.SERVICE_NAME }} --api-id {{ env.API_ID }} --custom-properties {{ env.METADATA_FILE }}

+ ```

+## Related content

+* [Using secrets in GitHub Actions](https://docs.github.com/en/actions/reference/encrypted-secrets)

+* [Creating configuration variables for a repository](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)

+If you're currently using App Service Environment v1 or v2, you have the opportunity to migrate your workloads to [App Service Environment v3](overview.md). App Service Environment v3 has [advantages and feature differences](overview.md#feature-differences) that provide enhanced support for your workloads and can reduce overall costs. Consider using [the automated migration features](upgrade-to-asev3.md) if your environment meets the criteria described in the [migration path decision tree](upgrade-to-asev3.md#migration-path-decision-tree).

+> [!IMPORTANT]

+> If your migration includes a custom domain suffix, the default host name behavior for App Service Environment v3 is different than for App Service Environment v2. For App Service Environment v3, the default host name always uses the default domain suffix and is in the form *APP-NAME.ASE-NAME.appserviceenvironment.net*. Review all your dependent resources, such as App Gateway, that use the host names of your apps to ensure they're updated to account for this behavior. For more information on App Service Environment feature differences between the different versions, see [App Service Environment version comparison](version-comparison.md).

+>

+### AMD SEV-SNP attestation on Confidential VMs

+Azure [Confidential VM](../confidential-computing/confidential-vm-overview.md) (CVM) is based on [AMD processors with SEV-SNP technology](../confidential-computing/virtual-machine-options.md). CVM offers VM OS disk encryption option with platform-managed keys or customer-managed keys and binds the disk encryption keys to the virtual machine's TPM. When a CVM boots up, SNP report containing the guest VM firmware measurements will be sent to Azure Attestation. The service validates the measurements and issues an attestation token that is used to release keys from [Managed-HSM](../key-vault/managed-hsm/overview.md) or [Azure Key Vault](../key-vault/general/basic-concepts.md). These keys are used to decrypt the vTPM state of the guest VM, unlock the OS disk and start the CVM. The attestation and key release process is performed automatically on each CVM boot, and the process ensures the CVM boots up only upon successful attestation of the hardware.

+### AMD SEV-SNP attestation on Confidential Containers

+Azure [Confidential Containers](../confidential-computing/confidential-containers.md) is based on [AMD processors with SEV-SNP technology](../confidential-computing/virtual-machine-options.md). Confidential containers, hosted on [Azure Container Instances](../container-instances/container-instances-confidential-overview.md) and on [Azure Kubernetes Service (in preview)](../aks/deploy-confidential-containers-default-policy.md) offer the ability to run groups of containers in an SEV-SNP protected trusted execution environment which isolates that group of containers from the container management control plane and other running containers. Attestation in confidential containers involves fetching the AMD hardware attestation report directly from the processor. This can be accomplished with our [SKR sidecar container](https://github.com/microsoft/confidential-sidecar-containers/tree/main/cmd/skr) or compiled directly into your application logic. The hardware report can then be exchanged with Azure Attestation and [managed-HSM](../key-vault/managed-hsm/overview.md) or Premium [Azure Key Vault (AKV)](../key-vault/general/basic-concepts.md) to retrieve secrets. You can also provide the hardware report to your own key vault system as desired.

+### Trusted Launch attestation

+Azure customers can [prevent bootkit and rootkit infections](https://www.youtube.com/watch?v=CQqu_rTSi0Q) by enabling [trusted launch](../virtual-machines/trusted-launch.md) for their virtual machines (VMs). When the VM is Secure Boot and vTPM enabled with guest attestation extension installed, vTPM measurements get submitted to Azure Attestation periodically for monitoring boot integrity. An attestation failure indicates potential malware, which is surfaced to customers via Microsoft Defender for Cloud, through Alerts and Recommendations.

+### SGX enclave attestation

+[Intel® Software Guard Extensions](https://www.intel.com/content/www/us/en/architecture-and-technology/software-guard-extensions.html) (SGX) refers to hardware-grade isolation, which is supported on certain Intel CPU models. SGX enables code to run in sanitized compartments known as SGX enclaves. Access and memory permissions are then managed by hardware to ensure a minimal attack surface with proper isolation.

+Client applications can be designed to take advantage of SGX enclaves by delegating security-sensitive tasks to take place inside those enclaves. Such applications can then make use of Azure Attestation to routinely establish trust in the enclave and its ability to access sensitive data.

+Intel® Xeon® Scalable processors only support [ECDSA-based attestation solutions](https://software.intel.com/content/www/us/en/develop/topics/software-guard-extensions/attestation-services.html#Elliptic%20Curve%20Digital%20Signature%20Algorithm%20(ECDSA)%20Attestation) for remotely attesting SGX enclaves. Utilizing ECDSA based attestation model, Azure Attestation supports validation of Intel® Xeon® E3 processors and Intel® Xeon® Scalable processor-based server platforms.

+> [!NOTE]

+> To perform attestation of Intel® Xeon® Scalable processor-based server platforms using Azure Attestation, users are expected to install [Azure DCAP version 1.10.0](https://github.com/microsoft/Azure-DCAP-Client) or higher.

+### Open Enclave attestation

+[Open Enclave](https://openenclave.io/sdk/) (OE) is a collection of libraries targeted at creating a single unified enclaving abstraction for developers to build TEE-based applications. It offers a universal secure app model that minimizes platform specificities. Microsoft views it as an essential stepping-stone toward democratizing hardware-based enclave technologies such as SGX and increasing their uptake on Azure.

+OE standardizes specific requirements for verification of an enclave evidence. This qualifies OE as a highly fitting attestation consumer of Azure Attestation.

+- Check the OS is supported, and the prerequisites have been met. See [Prerequisites](../extension-based-hybrid-runbook-worker-install.md#prerequisites).

+ - For Windows, you can find the settings file here:

+ > [!TIP]

+ > Replace `*` in the below path with the specific version that is installed if you know it.

+ ```

+ C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\*\RuntimeSettings

+ ```

+ - For Linux, you can find the settings file here:

+ ```

+ /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux/

+ ```

+ - For Windows, you can find the troubleshooter here:

+ > [!TIP]

+ > Replace `*` in the below path with the specific version that is installed if you know it.

+ ```

+ C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\*\bin\troubleshooter\TroubleShootWindowsExtension.ps1

+ ```

+ - For Linux, you can find the troubleshooter here:

+ > [!TIP]

+ > Replace `*` in the below path with the specific version that is installed if you know it.

+ ```

+ /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-*/Troubleshooter/LinuxTroubleshooter.py

+ ```

+ - For Windows, check theΓÇ»`Hybrid Worker Service` (***HybridWorkerService***) service.

+ - For Linux, check the `hwd` service.

+ - For Windows, run the log collector tool located here:

+ > [!TIP]

+ > Replace `*` in the below path with the specific version that is installed if you know it.

+ ```

+ C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\*\bin\troubleshooter\PullLogs.ps1

+ ```

+ Logs will be located here:

+ ```

+ C:\HybridWorkerExtensionLogs

+ ```

+ - For Linux: Logs are in the following folders:

+ ```

+ /var/log/azure/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux

+ ```

+ and

+ ```

+ /home/hweautomation

+ ```

+1. Navigate to the folder:

+ > [!TIP]

+ > Replace `*` in the below path with the specific version that is installed if you know it.

+ ```

+ C:\Program Files\Microsoft Monitoring Agent\Agent\AzureAutomation\*\HybridAgent

+ ```

+1. Edit the file with the name `Orchestrator.Sandbox.exe.config`

+1. Add the following lines inside the `<assemblyBinding>` tag:

+| `C:\ProgramData\AzureConnectedMachineAgent\Tokens` | Read |

+| `C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows` | Read and Execute |

+Job fails to start on a Hybrid Worker, and you see the following error:

+- The machines don't exist anymore.

+1. Open PowerShell console.

+1. **Remove the registry key**, if present: `HKLM:\Software\Microsoft\Azure\HybridWorker`

+ 1. PowerShell code to remove the registry key along with any subkeys and values under it.:

+

+ ```powershell

+ Get-Item HKLM:\Software\Microsoft\Azure\HybridWorker | Remove-Item -Recurse

+ ```

+

+1. **Remove the registry key**, if present: `HKLM:\Software\Microsoft\HybridRunbookWorkerV2`

+ 1. PowerShell code to remove the registry key along with any subkeys and values under it.:

+

+ ```powershell

+ Get-Item HKLM:\Software\Microsoft\HybridRunbookWorkerV2 | Remove-Item -Recurse

+ ```

+1. Navigate to the Hybrid Worker extension installation folder:

+ > [!TIP]

+ > Replace `*` in the below command with the specific version that is installed if you know it.

+ ```powershell

+ cd "C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\*"

+ ```

+1. **Install** the Hybrid Worker extension:

+ ```powershell

+ .\bin\install.ps1

+ ```

+1. **Enable** the Hybrid Worker extension:

+ ```powershell

+ .\bin\enable.ps1

+ ```

+1. Open PowerShell console.

+1. Navigate to the Hybrid Worker extension installation folder:

+ > [!TIP]

+ > Replace `*` in the below command with the specific version that is installed if you know it.

+ ```powershell

+ cd "C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\*"

+ ```

+1. **Disable** the Hybrid Worker extension:

+ ```powershell

+ .\bin\disable.cmd

+ ```

+1. **Uninstall** the Hybrid Worker extension:

+ ```powershell

+ .\bin\uninstall.ps1

+ ```

+1. **Remove registry key**, if present: `HKLM:\Software\Microsoft\Azure\HybridWorker`

+ 1. PowerShell code to remove the registry key along with any subkeys and values under it.:

+

+ ```powershell

+ Get-Item HKLM:\Software\Microsoft\Azure\HybridWorker | Remove-Item -Recurse

+ ```

+

+1. **Remove the registry key**, if present: `HKLM:\Software\Microsoft\HybridRunbookWorkerV2`

+ 1. PowerShell code to remove the registry key along with any subkeys and values under it.:

+

+ ```powershell

+ Get-Item HKLM:\Software\Microsoft\HybridRunbookWorkerV2 | Remove-Item -Recurse

+ ```

+1. **Delete** the `state` folder:

+ ```bash

+ rm -r /home/hweautomation/state

+ ```

+1. Navigate to the Hybrid Worker extension installation folder:

+ > [!TIP]

+ > Replace `*` in the below command with the specific version that is installed if you know it.

+ ```bash

+ cd /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-*/

+ ```

+

+1. **Delete** the mrseq file:

+ ```bash

+ rm mrseq

+ ```

+1. **Install** the Hybrid Worker Extension:

+ ```bash

+ ./extension_shim.shΓÇ»-cΓÇ»./HWExtensionHandlers.pyΓÇ»-i

+ ```

+1. **Enable** the Hybrid Worker extension:

+ ```bash

+ ./extension_shim.shΓÇ»-cΓÇ»./HWExtensionHandlers.pyΓÇ»-e

+ ```

+1. Navigate to the Hybrid Worker Extension installation folder:

+ > [!TIP]

+ > Replace `*` in the below command with the specific version that is installed if you know it.

+ ```bash

+ cd /var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-*/

+ ```

+1. **Disable** the Hybrid Worker extension:

+ ```bash

+ ./extension_shim.shΓÇ»-cΓÇ»./HWExtensionHandlers.pyΓÇ»-d

+ ```

+1. **Uninstall** the Hybrid Worker extension:

+ ```bash

+ ./extension_shim.shΓÇ»-cΓÇ»./HWExtensionHandlers.pyΓÇ»-u

+ ```

+```

+Connect-AzAccount : No certificate was found in the certificate store with thumbprint 0000000000000000000000000000000000000000

+At line:3 char:1

+ + CategoryInfo : CloseError: (:) [Connect-AzAccount],ArgumentException

+ + FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.ConnectAzAccountCommand

+```

+Place this file in the same folder as the executable file `OrchestratorSandbox.exe`. For example:

+> [!TIP]

+> Replace `*` in the below path with the specific version that is installed if you know it.

+```

+%ProgramFiles%\Microsoft Monitoring Agent\Agent\AzureAutomation\*\HybridAgent

+```

+SSH access to Arc-enabled servers is currently supported in all regions supported by Arc-Enabled Servers.

+## Prerequisites

+1. The disaster recovery script must be run from the same folder where the config (.yaml) files are present. The config files are present on the machine used to run the script to deploy Arc resource bridge.

+1. The machine being used to run the script must have bidirectional connectivity to the Arc resource bridge VM on port 6443 (Kubernetes API server) and 22 (SSH), and outbound connectivity to the Arc resource bridge VM on port 443 (HTTPS).

+> [!IMPORTANT]

+> The Flex Consumption plan currently doesn't support subnets with names that contain underscore (`_`) characters.

+> Targeting .NET 8 with the in-process model is not yet enabled for apps in sovereign clouds. Updates will be communicated on [this tracking thread on GitHub](https://github.com/Azure/azure-functions-host/issues/9951).

+ 3. Check if you see any errors in core agent logs located at `/var/opt/microsoft/azuremonitoragent/log/mdsd.*` on your machine

+2. Check that the file `C:\Resources\Directory\AMADataStore\mcs\mcsconfig.lkg.xml` exists.

+2. Check that the file `C:\Resources\Directory\AMADataStore\mcs\mcsconfig.lkg.xml` exists.

+ - IMDS service isn't running/accessible from the virtual machine. [Check if you can access IMDS from the machine](../../virtual-machines/windows/instance-metadata-service.md?tabs=windows).

+ - AMA can't access IMDS. Check if you see IMDS errors in `C:\WindowsAzure\Resources\AMADataStore.<virtual-machine-name>\Tables\MAEventTable.tsf` file.

+2. Check that the file `C:\WindowsAzure\Resources\AMADataStore.<virtual-machine-name>\mcs\mcsconfig.lkg.xml` exists.

+2. Check that the file `C:\WindowsAzure\Resources\AMADataStore.<virtual-machine-name>\mcs\mcsconfig.lkg.xml` exists.

+| 3 | mail |

+| 5 | auth |

+| 6 | syslog |

+| 7 | lpr |

+| 8 | news |

+| 9 | uucp |

+| 10 | ftp |

+| 11 | ntp |

+| 12 | audit |

+| 13 | alert |

+| 14 | clock |

+| 15 | local0 |

+| 16 | local1 |

+| 17 | local2 |

+| 18 | local3 |

+| 19 | local4 |

+| 20 | local5 |

+| 21 | local6 |

+| 22 | local7 |

+ Title: Best practices for scaling Azure Monitor Workspaces with Azure Monitor managed service for Prometheus

+description: Learn best practices for organizing your Azure Monitor Workspaces to meet your scale and growing volume of data ingestion

+# customer intent: As an azure administrator I want to understand the best practices for scaling Azure Monitor Workspaces to meet a growing volume of data ingestion

+# Best practices for scaling Azure Monitor Workspaces with Azure Monitor managed service for Prometheus

+Azure Monitor managed service for Prometheus allows you to collect and analyze metrics at scale. Prometheus metrics are stored in Azure Monitor Workspaces. The workspace supports analysis tools like Azure Managed Grafana, Azure Monitor metrics explorer with PromQL, and open source tools such as PromQL and Grafana.

+This article provides best practices for organizing your Azure Monitor Workspaces to meet your scale and growing volume of data ingestion.

+## Topology design criteria

+A single Azure Monitor workspace can be sufficient for many use cases. Some organizations create multiple workspaces to better meet their requirements. As you identify the right criteria to create additional workspaces, create the lowest number of workspaces that match your requirements, while optimizing for minimal administrative management overhead.

+The following are scenarios that require splitting an Azure Monitor workspace into multiple workspaces:

+| Scenario | Best practice |

+|||

+|Sovereign clouds.| When working with more than one sovereign cloud, create an Azure Monitor workspace in each cloud.|

+| Compliance or regulatory requirements.| If you're subject to regulations that mandate the storage of data in specific regions, create an Azure Monitor workspace per region as per your requirements. |

+| Regional scaling. | When you're managing metrics for regionally diverse organizations such as large services or financial institutions with regional accounts, create an Azure Monitor workspace per region.

+| Azure tenants.| For multiple Azure tenants, create an Azure Monitor workspace in each tenant. Querying data across tenants isn't supported.

+| Deployment environments. | Create a separate workspace for each of your deployment environments to maintain discrete metrics for development, test, preproduction, and production environments.|

+| Service limits and quotas. | Azure Monitor workspaces have default ingestion limits, which can be increased by opening a support ticket. If you're approaching the limit, or estimate that you'll exceed the ingestion limit, consider requesting an increase, or splitting your workspace into two or more workspaces.|

+## Service limits and quotas

+Azure Monitor workspaces have default quotas and limitations for metrics of 1 million event ingested per minute. As your usage grows and you need to ingest more metrics, you can request an increase. If your capacity requirements are exceptionally large and your data ingestion needs are exceeding the limits of a single Azure Monitor workspace, consider creating multiple Azure Monitor workspaces. Use the Azure monitor workspace platform metrics to monitor utilization and limits. For more information on limits and quotas, see [Azure Monitor service limits and quotas](/azure/azure-monitor/service-limits#prometheus-metrics).

+Consider the following best practices for managing Azure Monitor workspace limits and quotas:

+| Best practice | Description |

+|||

+| Monitor and create an alert on ingestion limits and utilization.| In the Azure portal, navigate to your Azure Monitor Workspace. Go to Metrics and verify that the metrics Active Time Series % Utilization and Events Per Minute Ingested % Utilization are below 100%. Set an Azure Monitor Alert to monitor the utilization and fire when the utilization is greater than 80% of the limit. For more information on monitoring utilization and limits, see [How can I monitor the service limits and quotas](/azure/azure-monitor/essentials/prometheus-metrics-overview#how-can-i-monitor-the-service-limits-and-quota).|

+|Request for a limit increase when the utilization exceeds 80% of the current limit.|As your Azure usage grows, the volume of data ingested is likely to increase. We recommend that you request an increase in limits if your data ingestion is exceeding or close to 80% of the ingestion limit. To request a limit increase, open a support ticket. To open a support ticket, see [Create an Azure support request](/azure/azure-supportability/how-to-create-azure-support-request).|

+|Estimate your projected scale.|As your usage grows and you ingest more metrics into your workspace, make an estimate of the projected scale and rate of growth. Based on your projections, request an increase in the limit.

+|Ingestion with Remote-write using the Azure monitor side-car container. |If you're using the Azure monitor side-car container and remote-write to ingest metrics into an Azure Monitor workspace, consider the following limits: <li>The side-car container can process up to 150,000 unique time series.</li><li> The container might throw errors serving requests over 150,000 due to the high number of concurrent connections. Mitigate this issue by increasing the remote batch size from the 500 default, to 1,000. Changing the remote batch size reduces the number of open connections.</li>|

+|DCR/DCE limits. |Limits apply to the data collection rules (DCR) and data collection endpoints (DCE) that send Prometheus metrics to your Azure Monitor workspace. For information on these limits, see [Prometheus Service limits](/azure/azure-monitor/service-limits#prometheus-metrics). These limits can't be increased. <p> Consider creating additional DCRs and DCEs to distribute the ingestion load across multiple endpoints. This approach helps optimize performance and ensures efficient data handling. For more information about creating DCRs and DCEs, see [How to create custom Data collection endpoint(DCE) and custom Data collection rule(DCR) for an existing Azure monitor workspace to ingest Prometheus metrics](https://github.com/Azure/prometheus-collector/tree/main/Azure-ARM-templates/Prometheus-RemoteWrite-DCR-artifacts)|

+## Optimizing performance for high volumes of data

+### Ingestion

+To optimize ingestion, consider the following best practices:

+| Best practice | Description |

+|||

+| Identify High cardinality Metrics. | Identify metrics that have a high cardinality, or metrics that are generating many time series. Once you identify high-cardinality metrics, optimize them to reduce the number of time series by dropping unnecessary labels.|

+| Use Prometheus config to optimize ingestion. | Azure Managed Prometheus provides Configmaps, which have settings that can be configured and used to optimize ingestion. For more information, see [ama-metrics-settings-configmap](https://aka.ms/azureprometheus-addon-settings-configmap) and [ama-metrics-prometheus-config-configmap](https://github.com/Azure/prometheus-collector/blob/main/otelcollector/configmaps/ama-metrics-prometheus-config-configmap.yaml). These configurations follow the same format as the Prometheus configuration file.<br> For information on customizing collection, see [Customize scraping of Prometheus metrics in Azure Monitor managed service for Prometheus](/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration).<p> For example, consider the following: <li> **Tune Scrape Intervals**.</li> The default scrape frequency is 30 seconds, which can be changed per default target using the configmap. To balance the trade-off between data granularity and resource usage, adjust the `scrape_interval` and `scrape_timeout` based on the criticality of metrics. <li> **Drop unnecessary labels for high cardinality metrics**.</li> For high cardinality metrics, identify labels that aren't necessary and drop them to reduce the number of time series. Use the `metric_relabel_configs` to drop specific labels from ingestion. For more information, see [Prometheus Configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config).|

+Use the configmap, change the settings as required, and apply the configmap to the kube-system namespace for your cluster. If you're using remote-writing into and Azure Monitor workspace, apply the customizations during ingestion directly in your Prometheus configuration.

+### Queries

+To optimize queries, consider the following best practices:

+#### Use Recording rules to optimize query performance

+Prometheus recording rules are used to precompute frequently used, or computationally expensive queries, making them more efficient and faster to query. Recording rules are especially useful for high volume metrics where querying raw data can be slow and resource-intensive. For more information, see [Recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules). Azure Managed Prometheus provides a managed and scalable way to create and update recording rules with the help of [Azure Managed Prometheus Rule Groups.](/azure/azure-monitor/essentials/prometheus-rule-groups#rule-types)

+Once the rule groups are created, Azure Managed Prometheus automatically loads and starts evaluating them. Query rule groups from the Azure Monitor workspace like other Prometheus metrics.

+Recording rules have the following benefits:

+- **Improve query performance**

+ Recording rules can be used to precompute complex queries, making them faster to query later. Precomputing complex queries reduces the load on Prometheus when these metrics are queried.

+- **Efficiency and Reduced query time**

+ Recording rules precompute the query results, reducing the time taken to query the data. This is especially useful for dashboards with multiple panels or high cardinality metrics.

+- **Simplicity**

+ Recording rules simplify queries in Grafana or other visualization tools, as they can reference precomputed metrics.

+

+The following example shows a recording rule as defined in Azure Managed Prometheus rule group:

+``` yaml

+"record": "job:request_duration_seconds:avg ",

+"expression": "avg(rate(request_duration_seconds_sum[5m])) by (job)",

+"labels": { "workload_type": "job"

+ },

+"enabled": true

+```

+For more complex metrics, create recording rules that aggregate multiple metrics or perform more advanced calculations. In the following example, `instance:node_cpu_utilisation:rate5m` computes the cpu utilization when the cpu isn't idle

+```yaml

+"record": "instance:node_cpu_utilisation:rate5m",

+ "expression": "1 - avg without (cpu) (sum without (mode)(rate(node_cpu_seconds_total{job=\"node\", mode=~\"idle|iowait|steal\"}[5m])))",

+"labels": {

+ "workload_type": "job"

+ },

+"enabled": true

+```

+Consider the following best practices for optimizing recording rules:

+| Best practice | Description |

+|||

+| Identify High Volume Metrics. | Focus on metrics that are queried frequently and have a high cardinality. |

+| Optimize Rule Evaluation Interval. | To balance between data freshness and computational load, adjust the evaluation interval of your recording rules. |

+| Monitor Performance. | Monitor query performance and adjust recording rules as necessary. |

+| Optimize rules by limiting scope.|To make recording rules faster, limit them in scope to a specific cluster. For more information, see [Limiting rules to a specific cluster](/azure/azure-monitor/essentials/prometheus-rule-groups#limiting-rules-to-a-specific-cluster).|

+#### Using filters in queries

+Optimizing Prometheus queries using filters involves refining the queries to return only the necessary data, reducing the amount of data processed and improving performance. The following are some common techniques to refine Prometheus queries.

+| Best practice | Description |

+|||

+| Use label filters.|Label filters help to narrow down the data to only what you need. Prometheus allows filtering by using `{label_name="label_value"}` syntax. If you have a large number of metrics across multiple clusters, an easy way to limit time series is to use the `cluster` filter. <p> For example, instead of querying `container_cpu_usage_seconds_total`, filter by cluster `container_cpu_usage_seconds_total{cluster="cluster1"}`.|

+| Apply time range selectors.|Using specific time ranges can significantly reduce the amount of data queried.<p> For example, instead of querying all data points for the last seven days `http_requests_total{job="myapp"}`, query for the last hour using `http_requests_total{job="myapp"}[1h]`.|

+| Use aggregation and grouping.| Aggregation functions can be used to summarize data, which can be more efficient than processing raw data points. When aggregating data, use `by` to group by specific labels, or `without` to exclude specific labels.<p> For example, sum requests grouped by job: `sum(rate(http_requests_total[5m])) by (job)`.|

+|Filter early in the query.| To limit the dataset from the start, apply filters as early as possible in your query.<p> For example, instead of `sum(rate(http_requests_total[5m])) by (job)`, filter first, then aggregate as follows: `sum(rate(http_requests_total{job="myapp"}[5m])) by (job)`.|

+| Avoid regex where possible.| Regex filters can be powerful but are also computationally expensive. Use exact matches whenever possible.<p> For example, instead of `http_requests_total{job=~"myapp.*"}`, use `http_requests_total{job="myapp"}`.|

+| Use offset for historical data.| If you're comparing current data with historical data, use the `offset` modifier.<p> For example, to compare current requests against requests from 24 hours ago, use `rate(http_requests_total[5m]) - rate(http_requests_total[5m] offset 24h)`.|

+| Limit data points in charts.| When creating charts, limit the number of data points to improve rendering performance. Use the step parameter to control the resolution.<p> For example, in Grafana: Set a higher step value to reduce data points:`http_requests_total{job="myapp"}[1h:10s]`.|

+#### Parallel queries

+Running a high number of parallel queries in Prometheus can lead to performance bottlenecks and can affect the stability of your Prometheus server. To handle a large volume of parallel queries efficiently, follow the best practices below:

+| Best practice | Description |

+|||

+| Query Load Distribution.| Distribute the query load by spreading the queries across different time intervals or Prometheus instances.|

+|Staggered Queries.| Schedule queries to run at different intervals to avoid peaks of simultaneous query executions.|

+If you're still seeing issues with running many parallel queries, create a support ticket to request an increase in the query limits.

+### Alerts and recording rules

+#### Optimizing alerts and recording rules for high scale

+Prometheus alerts and recording rules can be defined as Prometheus rule groups. One rule group can contain up to 20 alerts or recording rules. Create up to 500 rule groups for each workspace to accommodate the number of alerts/rules required. To raise this limit, open a support ticket

+When defining the recording rules, take into account the evaluation interval to optimize the number of time series per rule and the performance of rule evaluations. Evaluation intervals can be between 1 minute and 24 hours. The default is 1 minute.

+### Use Resource Health to view queries from recording rule status

+Set up Resource Health to view the health of your Prometheus rule group in the portal. Resource Health allows you to detect problems in your recording rules, such as incorrect configuration, or query throttling problems. For more information on setting up Resource Health, see [View the resource health states of your Prometheus rule groups](/azure/azure-monitor/essentials/prometheus-rule-groups#view-the-resource-health-states-of-your-prometheus-rule-groups)

+## <a name="transition"></a> Transition an Azure NetApp Files volume to customer-managed keys (preview)

+Azure NetApp Files supports the ability to move existing volumes using platform-managed keys to customer-managed keys. Once you complete the migration, you can't revert to platform-managed keys.

+### Register the feature

+Encryption key transition for Azure NetApp Files is currently in preview. Before using this feature for the first time, you need to register it.

+1. Register the feature:

+ ```azurepowershell-interactive

+ Register-AzProviderFeature -ProviderNamespace Microsoft.NetApp -FeatureName ANFMigratePmkToCmk

+ ```

+2. Check the status of the feature registration:

+ ```azurepowershell-interactive

+ Get-AzProviderFeature -ProviderNamespace Microsoft.NetApp -FeatureName ANFMigratePmkToCmk

+ ```

+ > [!NOTE]

+ > The **RegistrationState** may be in the `Registering` state for up to 60 minutes before changing to `Registered`. Wait until the status is **Registered** before continuing.

+You can also use [Azure CLI commands](/cli/azure/feature) `az feature register` and `az feature show` to register the feature and display the registration status.

+### Transition volumes

+>[!NOTE]

+>When you transition volumes to use customer-managed keys, you must perform the transition for every virtual network where your Azure NetApp Files account has volumes.

+1. Ensure you [configured your Azure NetApp Files account to use customer-managed keys](#configure-a-netapp-account-to-use-customer-managed-keys).

+1. In the Azure portal, navigate to **Encryption**.

+1. Select the **CMK Migration** tab.

+1. From the drop-down menu, select the virtual network and key vault private endpoint you want to use.

+1. Azure generates a list of volumes to be encrypted by your customer-managed key.

+1. Select **Confirm** to initiate the migration.

+Azure NetApp Files supports the ability to move existing volumes using platform-managed keys to customer-managed keys. Once you complete the transition, you cannot revert back to platform-managed keys. For additional information, see [Transition an Azure NetApp Files volume to customer-managed keys](configure-customer-managed-keys.md#transition).

+* [Transition a volume to customer-managed keys](configure-customer-managed-keys.md#transition) (Preview)

+ Azure NetApp Files now supports the ability to transition an existing volume to use customer-managed keys for volume encryption.

+## Microsoft.App

+* functions

+* logicApps

+* edgeDevices

+## Microsoft.Edge

+* connectivityStatuses

+* Sites

+* updates

+## Microsoft.EdgeMarketplace

+* offers

+* publishers

+* plugins

+* simplifiedSolutions

+## Microsoft.HybridCompute

+* networkConfigurations

+* settings

+## Microsoft.HybridContainerService

+* kubernetesVersions

+* provisionedClusterInstances

+## Microsoft.KubernetesRuntime

+* bgpPeers

+* loadBalancers

+* services

+* storageClasses

+## Microsoft.LoadTestService

+* loadTestMappings

+* loadTestProfileMappings

+* scheduledevents

+## Microsoft.Marketplace

+* products

+## Microsoft.Monitor

+* investigations

+* groupQuotas

+## Microsoft.ResourceNotifications

+* eventGridFilters

+* pricings

+* trustedIps

+* businessApplicationAgents

+* contenttranslators

+* enrichmentWidgets

+When calling the [Unique Tags API](/rest/api/resources/tags/list) there is a limit to the size of each API response page that is returned. A tag that has a large set of unique values will require the API to fetch the next page to retrieve the remaining set of values. When this happens the tag key is shown again to indicate that the values are still under this key.

+ Title: Use a chaos experiment template to induce an outage on a Microsoft Entra ID instance

+description: Use the Azure portal to create an experiment from the Microsoft Entra ID outage experiment template.

+# Use a chaos experiment template to induce an outage on a Microsoft Entra ID instance

+You can use a chaos experiment to verify that your application is resilient to failures by causing those failures in a controlled environment. In this article, you induce an outage on a Microsoft Entra ID resource using a pre-populated experiment template and Azure Chaos Studio.

+1. Select **Microsoft Entra ID Outage**.

+Now that you've run a Microsoft Entra ID outage template experiment, you're ready to:

+> Effective **April 19, 2024**, All UK alpha sender IDs now require a [registration application](../../quickstarts/sms/enable-alphanumeric-sender-id.md#enable-preregistered-alphanumeric-sender-id) approval.

+- Ultra disk capability is supported on confidential VMs

+ Navigate to the `/allRegistrationStatus` route to view all applications registered with the Eureka Server for Spring.

+ Title: Monitoring data reference for Azure Container Registry

+description: This article contains important reference material you need when you monitor Azure Container Registry.

+# Azure Container Registry monitoring data reference

+See [Monitor Azure Container Registry](monitor-container-registry.md) for details on the data you can collect for Azure Container Registry and how to use it.

+### Supported metrics for Microsoft.ContainerRegistry/registries

+The following table lists the metrics available for the Microsoft.ContainerRegistry/registries resource type.

+> [!NOTE]

+> Because of layer sharing, registry **Storage used** might be less than the sum of storage for individual repositories. When you [delete](container-registry-delete.md) a repository or tag, you recover only the storage used by manifest files and the unique layers referenced.

+- **Geolocation**. The Azure region for a registry or [geo-replica](container-registry-geo-replication.md).

+### Supported resource logs for Microsoft.ContainerRegistry/registries

+For a reference of all Azure Monitor Logs and Log Analytics tables, see the [Azure Monitor Log Table Reference](/azure/azure-monitor/reference/tables/tables-resourcetype).

+### Container Registry Microsoft.ContainerRegistry/registries

+- [AzureActivity](/azure/azure-monitor/reference/tables/azureactivity#columns). Entries from the Azure Activity log that provide insight into any subscription-level or management group level events that occurred in Azure.

+- [AzureMetrics](/azure/azure-monitor/reference/tables/azuremetrics#columns). Metric data emitted by Azure services that measure their health and performance.

+- [ContainerRegistryLoginEvents](/azure/azure-monitor/reference/tables/containerregistryloginevents#columns). Registry authentication events and status, including the incoming identity and IP address.

+- [ContainerRegistryRepositoryEvents](/azure/azure-monitor/reference/tables/containerregistryrepositoryevents#columns). Operations on images and other artifacts in registry repositories. The following operations are logged: push, pull, untag, delete (including repository delete), purge tag, and purge manifest.

+ Purge events are logged only if a registry [retention policy](container-registry-retention-policy.md) is configured.

+- [Microsoft.ContainerRegistry resource provider operations](/azure/role-based-access-control/resource-provider-operations#microsoftcontainerregistry)

+The following table lists operations related to Azure Container Registry that can be created in the Activity log. This list isn't exhaustive.

+| Operation | Description |

+|:|:|

+| Create or Update Container Registry | Create a container registry or update a registry property |

+| Delete Container Registry | Delete a container registry |

+| List Container Registry Login Credentials | Show credentials for registry's admin account |

+| Import Image | Import an image or other artifact to a registry |

+| Create Role Assignment | Assign an identity a Role-based access control (RBAC) role to access a resource |

+## Related content

+- See [Monitor Azure Container Registry](monitor-container-registry.md) for a description of monitoring Container Registry.

+- See [Monitor Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource) for details on monitoring Azure resources.

+ Title: Monitor Azure Container Registry

+description: Start here to learn how you can use the features of Azure Monitor to analyze and alert data in Azure Container Registry.

+# Monitor Azure Container Registry

+This article describes the monitoring data generated by Azure Container Registry and how you can use the features of Azure Monitor to analyze and alert on this data.

+## Monitor overview

+The **Overview** page in the Azure portal for each registry includes a brief view of recent resource usage and activity, such as push and pull operations. This high-level information is useful, but only a small amount of data is shown there.

+For more information about the resource types for Container Registry, see [Azure Container Registry monitoring data reference](monitor-container-registry-reference.md).

+## Monitoring data

+Azure Container Registry collects the same kinds of monitoring data as other Azure resources that are described in [Monitoring data from Azure resources](../azure-monitor/essentials/monitor-azure-resource.md#monitoring-data).

+See [Monitoring Azure Container Registry data reference](monitor-service-reference.md) for detailed information on the metrics and logs created by Azure Container Registry.

+## Collection and routing

+Platform metrics and the Activity log are collected and stored automatically, but can be routed to other locations by using a diagnostic setting.

+Resource Logs aren't collected and stored until you create a diagnostic setting and route them to one or more locations.

+See [Create diagnostic setting to collect platform logs and metrics in Azure](../azure-monitor/essentials/diagnostic-settings.md) for the detailed process for creating a diagnostic setting using the Azure portal, CLI, or PowerShell. When you create a diagnostic setting, you specify which categories of logs to collect. The categories for Azure Container Registry are listed in [Azure Container Registry monitoring data reference](monitor-service-reference.md#resource-logs).

+> [!TIP]

+> You can also create registry diagnostic settings by navigating to your registry in the portal. In the menu, select **Diagnostic settings** under **Monitoring**.

+The metrics and logs you can collect are discussed in the following sections.

+For a list of available metrics for Container Registry, see [Azure Container Registry monitoring data reference](monitor-container-registry-reference.md#metrics).

+## Analyzing metrics

+You can analyze metrics for an Azure container registry with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Analyze metrics with Azure Monitor metrics explorer](../azure-monitor/essentials/analyze-metrics.md) for details on using this tool.

+> [!TIP]

+> You can also go to the metrics explorer by navigating to your registry in the portal. In the menu, select **Metrics** under **Monitoring**.

+For a list of the platform metrics collected for Azure Container Registry, see [Monitoring Azure Container Registry data reference metrics](monitor-service-reference.md#metrics).

+### Azure CLI

+The following Azure CLI commands can be used to get information about the Azure Container Registry metrics.

+- [az monitor metrics list-definitions](/cli/azure/monitor/metrics#az-monitor-metrics-list-definitions) - List metric definitions and dimensions

+- [az monitor metrics list](/cli/azure/monitor/metrics#az-monitor-metrics-list) - Retrieve metric values

+### REST API

+You can use the Azure Monitor REST API to get information programmatically about the Azure Container Registry metrics.

+- [List metric definitions and dimensions](/rest/api/monitor/metricdefinitions/list)

+- [Retrieve metric values](/rest/api/monitor/metrics/list)

+## Analyzing logs

+Data in Azure Monitor Logs is stored in tables where each table has its own set of unique properties.

+All resource logs in Azure Monitor have the same fields followed by service-specific fields. The common schema is outlined in [Azure Monitor resource log schema](../azure-monitor/essentials/resource-logs-schema.md#top-level-common-schema). The schema for Azure Container Registry resource logs is found in the [Azure Container Registry Data Reference](monitor-service-reference.md#schemas).

+The [Activity log](../azure-monitor/essentials/activity-log.md) is a platform log in Azure that provides insight into subscription-level events. You can view it independently or route it to Azure Monitor Logs, where you can do much more complex queries using Log Analytics.

+For a list of the types of resource logs collected for Azure Container Registry, see [Monitoring Azure Container Registry data reference](monitor-service-reference.md#resource-logs).

+For a list of the tables used by Azure Monitor Logs and queryable by Log Analytics, see [Monitoring Azure Container Reference data reference](monitor-service-reference.md#azure-monitor-logs-tables).

+For the available resource log categories, their associated Log Analytics tables, and the log schemas for Container Registry, see [Azure Container Registry monitoring data reference](monitor-container-registry-reference.md#resource-logs).

+For example, the following query retrieves the most recent 24 hours of data from the **ContainerRegistryRepositoryEvents** table:

+```Kusto

+ContainerRegistryRepositoryEvents

+| where TimeGenerated > ago(1d)

+```

+The following image shows sample output:

+Following are queries that you can use to help you monitor your registry resource.

+Error events from the last hour:

+```Kusto

+union Event, Syslog // Event table stores Windows event records, Syslog stores Linux records

+| where TimeGenerated > ago(1h)

+| where EventLevelName == "Error" // EventLevelName is used in the Event (Windows) records

+ or SeverityLevel== "err" // SeverityLevel is used in Syslog (Linux) records

+```

+100 most recent registry events:

+```Kusto

+ContainerRegistryRepositoryEvents

+| union ContainerRegistryLoginEvents

+| top 100 by TimeGenerated

+| project TimeGenerated, LoginServer, OperationName, Identity, Repository, DurationMs, Region , ResultType

+```

+Identity of user or object that deleted repository:

+```Kusto

+ContainerRegistryRepositoryEvents

+| where OperationName contains "Delete"

+| project LoginServer, OperationName, Repository, Identity, CallerIpAddress

+```

+Identity of user or object that deleted tag:

+```Kusto

+ContainerRegistryRepositoryEvents

+| where OperationName contains "Untag"

+| project LoginServer, OperationName, Repository, Tag, Identity, CallerIpAddress

+```

+Repository-level operation failures:

+```kusto

+ContainerRegistryRepositoryEvents

+| where ResultDescription contains "40"

+| project TimeGenerated, OperationName, Repository, Tag, ResultDescription

+```

+Registry authentication failures:

+```kusto

+ContainerRegistryLoginEvents

+| where ResultDescription != "200"

+| project TimeGenerated, Identity, CallerIpAddress, ResultDescription

+```

+### Azure Container Registry alert rules

+The following table lists some suggested alert rules for Container Registry. These alerts are just examples. You can set alerts for any metric, log entry, or activity log entry listed in the [Azure Container Registry monitoring data reference](monitor-container-registry-reference.md).

+| Alert type | Condition | Description |

+|:|:|:|

+| metric | Signal: Storage used<br/>Operator: Greater than<br/>Aggregation type: Average<br/>Threshold value: 5 GB| Alerts if the registry storage used exceeds a specified value.|

+### Example: Send email alert when registry storage used exceeds a value

+1. In the Azure portal, navigate to your registry.

+1. Select **Metrics** under **Monitoring**.

+1. In the metrics explorer, in **Metric**, select **Storage used**.

+1. Select **New alert rule**.

+1. In **Scope**, confirm the registry resource for which you want to create an alert rule.

+1. In **Condition**, select **Add condition**.

+ 1. In **Signal name**, select **Storage used**.

+ 1. In **Chart period**, select **Over the last 24 hours**.

+ 1. In **Alert logic**, in **Threshold value**, select a value such as *5*. In **Unit**, select a value such as *GB*.

+ 1. Accept default values for the remaining settings, and select **Done**.

+1. In **Actions**, select **Add action groups** > **+ Create action group**.

+ 1. Enter details of the action group.

+ 1. On the **Notifications** tab, select **Email/SMS message/Push/Voice** and enter a recipient such as *admin@contoso.com*. Select **Review + create**.

+1. Enter a name and description of the alert rule, and select the severity level.

+1. Select **Create alert rule**.

+## Related content

+- See [Azure Container Registry monitoring data reference](monitor-container-registry-reference.md) for a reference of the metrics, logs, and other important values created for Container Registry.

+- See [Monitoring Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource) for general details on monitoring Azure resources.

+> [!TIP]

+> If youΓÇÖre unsure whether shutting down a VM will cause a problem for others, you can first restrict access to the VM. Consider [Configuring role assignments for the VM](/entr) to assess the VMΓÇÖs usage and determine if limiting access caused problems for others.

+> - If you have a disabled Access to Azure Active Directory subscription, the subscription gets automatically deleted after 90 days of cancelling. No action is needed to manually delete it.

+### Handling refresh tokens for the linked service

+When you use the QuickBooks Online connector in a linked service, it's important to manage OAuth 2.0 refresh tokens from QuickBooks correctly. The linked service uses a refresh token to obtain new access tokens. However, QuickBooks Online periodically updates the refresh token, invalidating the previous one. The linked service does not automatically update the refresh token in Azure Key Vault, so you need to manage updating the refresh token to ensure uninterrupted connectivity. Otherwise you might encounter authentication failures once the refresh token expires.

+You can manually update the refresh token in Azure Key Vault based on QuickBooks Online's refresh token expiry policy. But another approach is to automate updates with a scheduled task or [Azure Function](/samples/azure/azure-quickstart-templates/functions-keyvault-secret-rotation) that checks for a new refresh token and updates it in Azure Key Vault.

+| EndUserId | string | Acts as a unique identifier for the end user within the generative AI application. If Microsoft Entra ID authorization is used to authenticate end-users in the generative AI application, this should be a Microsoft Entra ID (previously known as Azure Active Directory) user object ID, otherwise this can be a GUID or some other identifier that uniquely identify the user. | Yes | 1234a123-12a3-1234-1ab2-a1b2c34d56e |

+| EndUserIdType | string | Specifies the type of end user identifier. It should be set to Microsoft Entra ID when using Microsoft Entra (previously known as Azure Active Directory) user object ID. | Yes, unless EndUserId is passed, in that case this must be set to proper value. | Microsoft Entra ID, Google, Other |

+> [!NOTE]

+> In the following tables, items marked with an asterisk (`*`) represent third-party tools.

+| Source | Target | Discover /<br />Inventory | Target and SKU<br />recommendation | TCO/ROI and<br />Business case |

+| SQL Server | Azure SQL Database | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/)<br />[Cloudamize*](https://cloudamize.com/) | [Azure SQL Migration extension](migration-using-azure-data-studio.md)<br />[DMA](/sql/dma/dma-overview)<br />[Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/)<br />[Cloudamize*](https://cloudamize.com/) | [TCO Calculator](https://azure.microsoft.com/pricing/tco/calculator/) |

+| SQL Server | Azure SQL Managed Instance | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/)<br />[Cloudamize*](https://cloudamize.com/) | [Azure SQL Migration extension](migration-using-azure-data-studio.md)<br />[DMA](/sql/dma/dma-overview)<br />[Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/)<br />[Cloudamize*](https://cloudamize.com/) | [TCO Calculator](https://azure.microsoft.com/pricing/tco/calculator/) |

+| SQL Server | SQL Server on Azure VM | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/)<br />[Cloudamize*](https://cloudamize.com/) | [Azure SQL Migration extension](migration-using-azure-data-studio.md)<br />[DMA](/sql/dma/dma-overview)<br />[Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/)<br />[Cloudamize*](https://cloudamize.com/) | [TCO Calculator](https://azure.microsoft.com/pricing/tco/calculator/) |

+| SQL Server | Azure Synapse Analytics | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/)<br />[Cloudamize*](https://cloudamize.com/) | | [TCO Calculator](https://azure.microsoft.com/pricing/tco/calculator/) |

+| Amazon RDS for SQL Server | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | | [Azure SQL Migration extension](migration-using-azure-data-studio.md)<br />[DMA](/sql/dma/dma-overview) | [TCO Calculator](https://azure.microsoft.com/pricing/tco/calculator/) |

+| Oracle | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/) | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[MigVisor*](https://solutionshub.epam.com/solution/migvisor-by-epam) | |

+| Oracle | Azure Synapse Analytics | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/) | [SSMA](/sql/ssma/sql-server-migration-assistant) | |

+| Oracle | Azure Database for PostgreSQL -<br />Single server | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/) | | |

+| MongoDB | Azure Cosmos DB | [Cloudamize*](https://cloudamize.com/) | [Cloudamize*](https://cloudamize.com/) | |

+| Cassandra | Azure Cosmos DB | | | |

+| MySQL | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/) | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/) | [TCO Calculator](https://azure.microsoft.com/pricing/tco/calculator/) |

+| MySQL | Azure Database for MySQL | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/) | | [TCO Calculator](https://azure.microsoft.com/pricing/tco/calculator/) |

+| Amazon RDS for MySQL | Azure Database for MySQL | | | [TCO Calculator](https://azure.microsoft.com/pricing/tco/calculator/) |

+| PostgreSQL | Azure Database for PostgreSQL -<br />Single server | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/) | | [TCO Calculator](https://azure.microsoft.com/pricing/tco/calculator/) |

+| Amazon RDS for PostgreSQL | Azure Database for PostgreSQL -<br />Single server | | | [TCO Calculator](https://azure.microsoft.com/pricing/tco/calculator/) |

+| DB2 | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/) | [SSMA](/sql/ssma/sql-server-migration-assistant) | |

+| Access | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/) | [SSMA](/sql/ssma/sql-server-migration-assistant) | |

+| Sybase - SAP ASE | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [Azure Migrate](https://azure.microsoft.com/services/azure-migrate/) | [SSMA](/sql/ssma/sql-server-migration-assistant) | |

+| Sybase - SAP IQ | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | | | |

+## Premigration phase

+| Source | Target | App Data Access<br />Layer Assessment | Database<br />Assessment | Performance<br />Assessment |

+| SQL Server | Azure SQL Database | [DAMT](https://marketplace.visualstudio.com/items?itemName=ms-databasemigration.data-access-migration-toolkit) / [DMA](/sql/dm)<br />[DMA](/sql/dma/dma-overview)<br />[Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/)<br />[Cloudamize*](https://cloudamize.com/) | [DEA](https://www.microsoft.com/download/details.aspx?id=54090)<br />[Cloudamize*](https://cloudamize.com/) |

+| SQL Server | Azure SQL Managed Instance | [DAMT](https://marketplace.visualstudio.com/items?itemName=ms-databasemigration.data-access-migration-toolkit) / [DMA](/sql/dm)<br />[DMA](/sql/dma/dma-overview)<br />[Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/)<br />[Cloudamize*](https://cloudamize.com/) | [DEA](https://www.microsoft.com/download/details.aspx?id=54090)<br />[Cloudamize*](https://cloudamize.com/) |

+| SQL Server | SQL Server on Azure VM | [DAMT](https://marketplace.visualstudio.com/items?itemName=ms-databasemigration.data-access-migration-toolkit) / [DMA](/sql/dm)<br />[DMA](/sql/dma/dma-overview)<br />[Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/)<br />[Cloudamize*](https://cloudamize.com/) | [DEA](https://www.microsoft.com/download/details.aspx?id=54090)<br />[Cloudamize*](https://cloudamize.com/) |

+| SQL Server | Azure Synapse Analytics | [DAMT](https://marketplace.visualstudio.com/items?itemName=ms-databasemigration.data-access-migration-toolkit) | | |

+| Amazon RDS for SQL | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [DAMT](https://marketplace.visualstudio.com/items?itemName=ms-databasemigration.data-access-migration-toolkit) / [DMA](/sql/dm)<br />[DMA](/sql/dma/dma-overview) | [DEA](https://www.microsoft.com/download/details.aspx?id=54090) |

+| Oracle | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [DAMT](https://marketplace.visualstudio.com/items?itemName=ms-databasemigration.data-access-migration-toolkit) / [SSMA](/sql/ssma/sql-server-migration-assistant) | [SSMA](/sql/ssma/sql-server-migration-assistant) | |

+| Oracle | Azure Synapse Analytics | [DAMT](https://marketplace.visualstudio.com/items?itemName=ms-databasemigration.data-access-migration-toolkit) / [SSMA](/sql/ssma/sql-server-migration-assistant) | [SSMA](/sql/ssma/sql-server-migration-assistant) | |

+| Oracle | Azure Database for PostgreSQL -<br />Single server | | [Ora2Pg*](https://ora2pg.darold.net/start.html) | |

+| Oracle | Azure Database for PostgreSQL -<br />Flexible server | | [Ora2Pg*](https://ora2pg.darold.net/start.html) | |

+| MongoDB | Azure Cosmos DB | | [Cloudamize*](https://cloudamize.com/) | [Cloudamize*](https://cloudamize.com/) |

+| Cassandra | Azure Cosmos DB | | | |

+| MySQL | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [DAMT](https://marketplace.visualstudio.com/items?itemName=ms-databasemigration.data-access-migration-toolkit) / [SSMA](/sql/ssma/sql-server-migration-assistant) | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/) | |

+| MySQL | Azure Database for MySQL | | | |

+| Amazon RDS for MySQL | Azure Database for MySQL | | | |

+| PostgreSQL | Azure Database for PostgreSQL -<br />Single server | | | |

+| PostgreSQL | Azure Database for PostgreSQL -<br />Flexible server | | | |

+| Amazon RDS for PostgreSQL | Azure Database for PostgreSQL -<br />Single server | | | |

+| DB2 | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [DAMT](https://marketplace.visualstudio.com/items?itemName=ms-databasemigration.data-access-migration-toolkit) / [SSMA](/sql/ssma/sql-server-migration-assistant) | [SSMA](/sql/ssma/sql-server-migration-assistant) | |

+| Access | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | | [SSMA](/sql/ssma/sql-server-migration-assistant) | |

+| Sybase - SAP ASE | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [DAMT](https://marketplace.visualstudio.com/items?itemName=ms-databasemigration.data-access-migration-toolkit) / [SSMA](/sql/ssma/sql-server-migration-assistant) | [SSMA](/sql/ssma/sql-server-migration-assistant) | |

+| Sybase - SAP IQ | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | | | |

+| Source | Target | Schema | Data<br />(Offline) | Data<br />(Online) |

+| SQL Server | Azure SQL Database | [SQL Database Projects extension](/azure-data-studio/extensions/sql-database-project-extension)<br />[DMA](/sql/dm)<br />[DMA](/sql/dma/dma-overview)<br />[Cloudamize*](https://cloudamize.com/) | [Cloudamize*](https://cloudamize.com/)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| SQL Server | Azure SQL Managed Instance | [Azure SQL Migration extension](migration-using-azure-data-studio.md)<br />[Cloudamize*](https://cloudamize.com/) | [Azure SQL Migration extension](migration-using-azure-data-studio.md)<br />[Cloudamize*](https://cloudamize.com/) | [Azure SQL Migration extension](migration-using-azure-data-studio.md)<br />[Cloudamize*](https://cloudamize.com/)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| SQL Server | SQL Server on Azure VM | [Azure SQL Migration extension](migration-using-azure-data-studio.md)<br />[DMA](/sql/dm)<br />[Cloudamize*](https://cloudamize.com/)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| SQL Server | Azure Synapse Analytics | | | |

+| Amazon RDS for SQL Server | Azure SQL Database | [SQL Database Projects extension](/azure-data-studio/extensions/sql-database-project-extension)<br />[DMA](/sql/dm)<br />[DMA](/sql/dma/dma-overview) | [Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Amazon RDS for SQL | Azure SQL Managed Instance | [Azure SQL Migration extension](migration-using-azure-data-studio.md) | [Azure SQL Migration extension](migration-using-azure-data-studio.md) | [Azure SQL Migration extension](migration-using-azure-data-studio.md)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Amazon RDS for SQL Server | SQL Server on Azure VM | [Azure SQL Migration extension](migration-using-azure-data-studio.md)<br />[DMA](/sql/dm)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Oracle | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[SharePlex*](https://www.quest.com/products/shareplex/)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[SharePlex*](https://www.quest.com/products/shareplex/)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [SharePlex*](https://www.quest.com/products/shareplex/)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Oracle | Azure Synapse Analytics | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [SharePlex*](https://www.quest.com/products/shareplex/)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Oracle | Azure Database for PostgreSQL -<br />Single server | [Ora2Pg*](https://ora2pg.darold.net/start.html)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [Ora2Pg*](https://ora2pg.darold.net/start.html)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) |<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Oracle | Azure Database for PostgreSQL -<br />Flexible server | [Ora2Pg*](https://ora2pg.darold.net/start.html)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [Ora2Pg*](https://ora2pg.darold.net/start.html)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) |<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| MongoDB | Azure Cosmos DB | [DMS](https://azure.microsoft.com/services/database-migration/)<br />[Cloudamize*](https://cloudamize.com/)<br />[Imanis Data*](https://azuremarketplace.microsoft.com/marketplace/apps/talena-inc.talena-solution-template?tab=Overview) | [DMS](https://azure.microsoft.com/services/database-migration/)<br />[Cloudamize*](https://cloudamize.com/)<br />[Imanis Data*](https://azuremarketplace.microsoft.com/marketplace/apps/talena-inc.talena-solution-template?tab=Overview) | [DMS](https://azure.microsoft.com/services/database-migration/)<br />[Cloudamize*](https://cloudamize.com/)<br />[Imanis Data*](https://azuremarketplace.microsoft.com/marketplace/apps/talena-inc.talena-solution-template?tab=Overview)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Cassandra | Azure Cosmos DB | [Imanis Data*](https://azuremarketplace.microsoft.com/marketplace/apps/talena-inc.talena-solution-template?tab=Overview) | [Imanis Data*](https://azuremarketplace.microsoft.com/marketplace/apps/talena-inc.talena-solution-template?tab=Overview) | [Imanis Data*](https://azuremarketplace.microsoft.com/marketplace/apps/talena-inc.talena-solution-template?tab=Overview) |

+| MySQL | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| MySQL | Azure Database for MySQL | [MySQL dump*](https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html) | [DMS](https://azure.microsoft.com/services/database-migration/) | [MyDumper/MyLoader*](https://centminmod.com/mydumper.html) with [data-in replication](../mysql/concepts-data-in-replication.md)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Amazon RDS for MySQL | Azure Database for MySQL | [MySQL dump*](https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html) | [DMS](https://azure.microsoft.com/services/database-migration/) | [MyDumper/MyLoader*](https://centminmod.com/mydumper.html) with [data-in replication](../mysql/concepts-data-in-replication.md)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| PostgreSQL | Azure Database for PostgreSQL -<br />Single server | [PG dump*](https://www.postgresql.org/docs/current/app-pgdump.html) | [PG dump*](https://www.postgresql.org/docs/current/app-pgdump.html) | [DMS](https://azure.microsoft.com/services/database-migration/)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Amazon RDS for PostgreSQL | Azure Database for PostgreSQL -<br />Single server | [PG dump*](https://www.postgresql.org/docs/current/app-pgdump.html) | [PG dump*](https://www.postgresql.org/docs/current/app-pgdump.html) | [DMS](https://azure.microsoft.com/services/database-migration/)<br />[Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| DB2 | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Access | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [SSMA](/sql/ssma/sql-server-migration-assistant) | [SSMA](/sql/ssma/sql-server-migration-assistant) | [SSMA](/sql/ssma/sql-server-migration-assistant) |

+| Sybase - SAP ASE | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [SSMA](/sql/ssma/sql-server-migration-assistant)<br />[Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [Qlik*](https://www.qlik.com/us/products/qlik-replicate)<br />[Striim*](https://www.striim.com/partners/striim-and-microsoft-azure/) |

+| Sybase - SAP IQ | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | [Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | [Ispirer*](https://www.ispirer.com/blog/migration-to-the-microsoft-technology-stack) | |

+| SQL Server | Azure SQL Database | [Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/)<br />[Cloudamize*](https://cloudamize.com/) |

+| SQL Server | Azure SQL Managed Instance | [Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/)<br />[Cloudamize*](https://cloudamize.com/) |

+| SQL Server | SQL Server on Azure VM | [Cloud Atlas*](https://www.unifycloud.com/cloud-migration-tool/)<br />[Cloudamize*](https://cloudamize.com/) |

+| SQL Server | Azure Synapse Analytics | |

+| RDS SQL | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | |

+| Oracle | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | |

+| Oracle | Azure Synapse Analytics | |

+| Oracle | Azure Database for PostgreSQL -<br />Single server | |

+| MongoDB | Azure Cosmos DB | [Cloudamize*](https://cloudamize.com/) |

+| Cassandra | Azure Cosmos DB | |

+| MySQL | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | |

+| MySQL | Azure Database for MySQL | |

+| Amazon RDS for MySQL | Azure Database for MySQL | |

+| PostgreSQL | Azure Database for PostgreSQL -<br />Single server | |

+| PostgreSQL | Azure Database for PostgreSQL -<br />Flexible server | |

+| Amazon RDS for PostgreSQL | Azure Database for PostgreSQL -<br />Single server | |

+| DB2 | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | |

+| Access | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | |

+| Sybase - SAP ASE | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | |

+| Sybase - SAP IQ | Azure SQL Database, Azure SQL Managed Instance, SQL Server on Azure VM | |

+## Related content

+- [What is the Azure Database Migration Service](dms-overview.md)

+## Migrate from an existing Expressroute circuit to a Metro circuit

+If you want to migrate from an existing ExpressRoute circuit, create a new ExpressRoute Metro circuit. Then, follow the steps for [circuit migration](circuit-migration.md) to transition from the existing standard ExpressRoute circuit to the ExpressRoute Metro circuit.

+- Hosts

+- Email Contacts

+- WHOIS Organizations

+- Hosts

+- Email Contacts

+- ASNs

+- WHOIS Organizations

+If a Firewall Policy is inherited from a parent policy, Rule Collection Groups in the parent policy always takes precedence regardless of the priority of a child policy.

+So, to summarize:

+Parent policy always takes precedence.

+1. Rule collection groups are processed in priority order.

+1. Rule collections are processed in priority order.

+1. DNAT rules, then Network rules, then Application rules are processed.

+## Apply the cluster resource placement

+### [Azure CLI](#tab/azure-cli)

+

+### [Portal](#tab/azure-portal)

+1. On the Azure portal overview page for your Fleet resource, in the **Fleet Resources** section, select **Resource placements**.

+1. Select **Create**.

+1. Create a `ClusterResourcePlacement` resource to specify the placement rules for distributing the cluster resource overrides across the cluster infrastructure, as shown in the following example. Make sure you select the appropriate resource. Replace the default template with the YAML example below, and select **Add**.

+ :::image type="content" source="./media/cluster-resource-override/crp-create-inline.png" lightbox="./media/cluster-resource-override/crp-create.png" alt-text="A screenshot of the Azure portal page for creating a resource placement, showing the YAML template with placeholder values.":::

+ ```yaml

+ apiVersion: placement.kubernetes-fleet.io/v1beta1

+ kind: ClusterResourcePlacement

+ metadata:

+ name: crp

+ spec:

+ resourceSelectors:

+ - group: rbac.authorization.k8s.io

+ kind: ClusterRole

+ version: v1

+ name: secret-reader

+ policy:

+ placementType: PickAll

+ affinity:

+ clusterAffinity:

+ requiredDuringSchedulingIgnoredDuringExecution:

+ clusterSelectorTerms:

+ - labelSelector:

+ matchLabels:

+ env: prod

+ ```

+ This example distributes resources across all clusters labeled with `env: prod`. As the changes are implemented, the corresponding `ClusterResourceOverride` configurations will be applied to the designated clusters, triggered by the selection of matching cluster role resource, `secret-reader`.

+1. Verify that the cluster resource placement is created successfully.

+ :::image type="content" source="./media/cluster-resource-override/crp-success-inline.png" lightbox="./media/cluster-resource-override/crp-success.png" alt-text="A screenshot of the Azure portal page for cluster resource placements, showing a successfully created cluster resource placement.":::

+1. Verify the cluster resource placement applied to the selected resources by selecting the resource from the list and checking the status.

+This article discusses creating cluster resource placements, which can be done via Azure CLI or the Azure portal. For more, see [Propagate resources from a Fleet hub cluster to member clusters](./quickstart-resource-propagation.md).

+### [Azure CLI](#tab/azure-cli)

+If you no longer wish to use the `ClusterResourcePlacement` object, you can delete it using the `kubectl delete` command. The following example deletes the `ClusterResourcePlacement` object named `crp`:

+kubectl delete clusterresourceplacement crp

+### [Portal](#tab/azure-portal)

+If you no longer wish to use your cluster resource placement, you can delete it from the Azure portal:

+1. On the Azure portal overview page for your Fleet resource, in the **Fleet Resources** section, select **Resource placements**.

+1. Select the cluster resource placement objects you want to delete, then select **Delete**.

+1. In the **Delete** tab, verify the correct objects are chosen. Once you're ready, select **Confirm delete** and **Delete**.

+### [Azure CLI](#tab/azure-cli)

+### [Portal](#tab/azure-portal)

+1. Sign in to the Azure portal.

+1. On the Azure portal overview page for your Fleet resource, in the **Fleet Resources** section, select **Resource placements**.

+1. Select **Create**.

+1. Replace the placeholder values with the following YAML, and select **Add**.

+ :::image type="content" source="./media/quickstart-resource-propagation/create-crp-inline.png" lightbox="./media/quickstart-resource-propagation/create-crp.png" alt-text="A screenshot of the Azure portal page for creating a resource placement, showing the YAML template with placeholder values.":::

+ ```yml

+ apiVersion: placement.kubernetes-fleet.io/v1beta1

+ kind: ClusterResourcePlacement

+ metadata:

+ name: crp

+ spec:

+ resourceSelectors:

+ - group: ""

+ kind: Namespace

+ version: v1

+ name: my-namespace

+ policy:

+ placementType: PickAll

+ ```

+

+1. Verify that the cluster resource placement is created successfully.

+ :::image type="content" source="./media/quickstart-resource-propagation/crp-success-inline.png" lightbox="./media/quickstart-resource-propagation/crp-success.png" alt-text="A screenshot of the Azure portal page for cluster resource placements, showing a successfully created cluster resource placement.":::

+1. To see more details on an individual cluster resource placement, select it from the list.

+ :::image type="content" source="./media/quickstart-resource-propagation/crp-details-inline.png" lightbox="./media/quickstart-resource-propagation/crp-details.png" alt-text="A screenshot of the Azure portal overview page for an individual cluster resource placement, showing events and details.":::

+1. You can view additional details on the cluster resource placement's snapshots, bindings, works, and scheduling policy snapshots using the individual tabs. For example, select the **Cluster Resources Snapshots** tab.

+ :::image type="content" source="./media/quickstart-resource-propagation/crp-snapshot-inline.png" lightbox="./media/quickstart-resource-propagation/crp-snapshot.png" alt-text="A screenshot of the Azure portal page for a cluster resource placement, with the cluster resources snapshots tab selected.":::

+### [Azure CLI](#tab/azure-cli)

+### [Portal](#tab/azure-portal)

+If you no longer wish to use your cluster resource placement, you can delete it from the Azure portal:

+1. On the Azure portal overview page for your Fleet resource, in the **Fleet Resources** section, select **Resource placements**.

+1. Select the cluster resource placement objects you want to delete, then select **Delete**.

+1. In the **Delete** tab, verify the correct objects are chosen. Once you're ready, select **Confirm delete** and **Delete**.

+## Apply the cluster resource placement

+### [Azure CLI](#tab/azure-cli)

+### [Portal](#tab/azure-portal)

+1. On the Azure portal overview page for your Fleet resource, in the **Fleet Resources** section, select **Resource placements**.

+1. Select **Create**.

+1. Create a `ClusterResourcePlacement` resource to specify the placement rules for distributing the resource overrides across the cluster infrastructure, as shown in the following example. Make sure you select the appropriate namespaces. When you're ready, select **Add**.

+ ```yaml

+ apiVersion: placement.kubernetes-fleet.io/v1beta1

+ kind: ClusterResourcePlacement

+ metadata:

+ name: crp-example

+ spec:

+ resourceSelectors:

+ - group: ""

+ kind: Namespace

+ name: test-namespace

+ version: v1

+ policy:

+ placementType: PickAll

+ affinity:

+ clusterAffinity:

+ requiredDuringSchedulingIgnoredDuringExecution:

+ clusterSelectorTerms:

+ - labelSelector:

+ matchLabels:

+ env: prod

+ - labelSelector:

+ matchLabels:

+ env: test

+ ```

+ This example distributes resources within the `test-namespace` across all clusters labeled with `env:prod` and `env:test`. As the changes are implemented, the corresponding `ResourceOverride` configurations will be applied to the designated resources, triggered by the selection of matching deployment resource, `my-deployment`.

+ :::image type="content" source="./media/quickstart-resource-propagation/create-resource-propagation-inline.png" lightbox="./media/quickstart-resource-propagation/create-resource-propagation.png" alt-text="A screenshot of the Azure portal page for creating a resource placement, showing the YAML template with placeholder values.":::

+1. Verify that the cluster resource placement is created successfully.

+ :::image type="content" source="./media/quickstart-resource-propagation/overview-cluster-resource-inline.png" lightbox="./media/quickstart-resource-propagation/overview-cluster-resource.png" alt-text="A screenshot of the Azure portal page for cluster resource placements, showing a successfully created cluster resource placement.":::

+1. Verify the cluster resource placement applied to the selected resources by selecting the resource from the list and checking the status.

+Cohere family models | Not available | Cohere-command-r-plus <br> Cohere-command-r <br> Cohere-embed-v3-english <br> Cohere-embed-v3-multilingual <br> Cohere-rerank-3-english <br> Cohere-rerank-3-multilingual

+Cohere-rerank-3-english <br> Cohere-rerank-3-multilingual | [Microsoft Managed Countries](/partner-center/marketplace/tax-details-marketplace#microsoft-managed-countriesregions)| East US 2, Sweden Central, North Central US, South Central US, East US, West US 3, West US | Not available

+A machine learning project typically starts with exploratory data analysis (EDA), data-preprocessing (cleaning, feature engineering), and it includes building ML model prototypes to validate hypotheses. This *prototyping* project phase is highly interactive in nature, and it lends itself to development in a Jupyter notebook, or in an IDE with a *Python interactive console*. In this article, learn how to:

+> * Access data from an Azure Machine Learning Datastores URI as if it were a file system.

+> * Materialize data into Pandas using the `mltable` Python library.

+> * Materialize Azure Machine Learning data assets into Pandas using the `mltable` Python library.

+* An Azure Machine Learning workspace. For more information, visit [Manage Azure Machine Learning workspaces in the portal or with the Python SDK (v2)](how-to-manage-workspace.md).

+* An Azure Machine Learning Datastore. For more information, visit [Create datastores](how-to-datastore.md).

+> The guidance in this article describes data access during interactive development. It applies to any host that can run a Python session. This can include your local machine, a cloud VM, a GitHub Codespace, etc. We recommend use of an Azure Machine Learning compute instance - a fully managed and pre-configured cloud workstation. For more information, visit [Create an Azure Machine Learning compute instance](how-to-create-compute-instance.md).

+> Ensure you have the latest `azure-fsspec` and `mltable` python libraries installed in your Python environment:

+These Datastore URIs are a known implementation of the [Filesystem spec](https://filesystem-spec.readthedocs.io/en/latest/https://docsupdatetracker.net/index.html) (`fsspec`): a unified pythonic interface to local, remote, and embedded file systems and bytes storage. First, pip install the `azureml-fsspec` package and its dependency `azureml-dataprep` package. Then, you can use the Azure Machine Learning Datastore `fsspec` implementation.

+The Azure Machine Learning Datastore `fsspec` implementation automatically handles the credential/identity passthrough that the Azure Machine Learning datastore uses. You can avoid both account key exposure in your scripts, and extra sign-in procedures, on a compute instance.

+> To avoid remembering the datastore URI format, you can copy-and-paste the datastore URI from the Studio UI with these steps:

+If the folders you specify in `rpath` don't yet exist, we create the folders for you.

+- APPEND: if a file with the same name exists in the destination path, APPEND keeps the original file

+- FAIL_ON_FILE_CONFLICT: if a file with the same name exists in the destination path, FAIL_ON_FILE_CONFLICT throws an error

+- MERGE_WITH_OVERWRITE: if a file with the same name exists in the destination path, MERGE_WITH_OVERWRITE overwrites that existing file with the new file

+The Pandas `read_csv()` method doesn't support reading a folder of CSV files. To handle this, glob the csv paths, and concatenate them to a data frame with the Pandas `concat()` method. The next code sample shows how to achieve this concatenation with the Azure Machine Learning filesystem:

+In these scenarios, you only read the parquet files in the folder, and ignore the ETL process files. This code sample shows how glob patterns can read only parquet files in a folder:

+To access data from the `dbfs` resource, you need:

+- **Personal Access Token (PAT)**; for more information about PAT creation, visit [Authentication using Azure Databricks personal access tokens](/azure/databricks/dev-tools/api/latest/authentication)

+With these values, you must create an environment variable for the PAT token on your compute instance:

+You can then access data in Pandas, as shown in this example:

+You can then instantiate the dataset, as shown here:

+### Files, folders, and globs

+> To avoid remembering the datastore URI format, you can copy-and-paste the datastore URI from the Studio UI with these steps:

+> Pandas is not designed to handle large datasets. Pandas can only process data that can fit into the memory of the compute instance.

+Use the `azcopy` utility to download the data to the local SSD of your host (local machine, cloud VM, Azure Machine Learning Compute Instance, etc.), into the local filesystem. The `azcopy` utility, which is preinstalled on an Azure Machine Learning compute instance, handles the data download. If you **don't** use an Azure Machine Learning compute instance or a Data Science Virtual Machine (DSVM), you might need to install `azcopy`. For more information, visit [azcopy](../storage/common/storage-ref-azcopy.md).

+In this article, learn how to connect to data sources located outside of Azure, to make that data available to Azure Machine Learning services. Azure connections serve as key vault proxies, and interactions with connections are direct interactions with an Azure key vault. An Azure Machine Learning connection securely stores username and password data resources, as secrets, in a key vault. The key vault RBAC controls access to these data resources. For this data availability, Azure supports connections to these external sources:

+> An Azure Machine Learning connection securely stores the credentials passed during connection creation in the Workspace Azure Key Vault. A connection references the credentials from the key vault storage location for further use. You don't need to directly deal with the credentials after they are stored in the key vault. You have the option to store the credentials in the YAML file. A CLI command or SDK can override them. We recommend that you **avoid** credential storage in a YAML file, because a security breach could lead to a credential leak.

+> For a successful data import, please verify that you installed the latest **azure-ai-ml** package (version 1.5.0 or later) for SDK, and the ml extension (version 2.15.1 or later).

+You can use these connection types to connect to Git:

+- Python feed

+- Azure Container Registry

+- a connection that uses an API key

+These connections aren't data connections, but are used to connect to external services for use in your code.

+The following example creates a Git connection to a GitHub repo. A Personal Access Token (PAT) authenticates the connection:

+Create a connection to a Python feed with one of following YAML files. Be sure to update the appropriate values:

+The following example creates a Python feed connection. A Personal Access Token (PAT), or a user name and password, authenticates the connection:

+Create a connection to an Azure Container Registry with one of following YAML files. Be sure to update the appropriate values:

+The following example creates an Azure Container Registry connection. A managed identity authenticates this connection:

+If you use a data connection (Snowflake DB, Amazon S3, or Azure SQL DB), these articles offer more information:

+ Title: How to deploy Cohere Rerank models as serverless APIs

+description: Learn to deploy and use Cohere Rerank models with Azure Machine Learning studio.

+#This functionality is also available in Azure AI Studio: /azure/ai-studio/how-to/deploy-models-cohere.md

+# How to deploy Cohere Rerank models with Azure Machine Learning studio

+In this article, you learn about the Cohere Rerank models, how to use Azure Machine Learning studio to deploy them as serverless APIs with pay-as-you-go token-based billing, and how to work with the deployed models.

+## Cohere Rerank models

+Cohere offers two Rerank models in Azure Machine Learning studio. These models are available in the model catalog for deployment as serverless APIs:

+* Cohere Rerank 3 - English

+* Cohere Rerank 3 - Multilingual

+You can browse the Cohere family of models in the [Model Catalog](concept-model-catalog.md) by filtering on the Cohere collection.

+### Cohere Rerank 3 - English

+Cohere Rerank English is a reranking model used for semantic search and retrieval-augmented generation (RAG). Rerank enables you to significantly improve search quality by augmenting traditional keyword-based search systems with a semantic-based reranking system that can contextualize the meaning of a user's query beyond keyword relevance. Cohere's Rerank delivers higher quality results than embedding-based search, lexical search, and even hybrid search, and it requires only adding a single line of code into your application.

+Use Rerank as a ranker after initial retrieval. In other words, after an initial search system finds the top 100 most relevant documents for a larger corpus of documents.

+Rerank supports JSON objects as documents where users can specify, at query time, the fields (keys) to use for semantic search. Some other attributes of Rerank include:

+* Context window of the model is 4,096 tokens

+* The max query length is 2,048 tokens

+Rerank English works well for code retrieval, semi-structured data retrieval, and long context.

+### Cohere Rerank 3 - Multilingual

+Cohere Rerank Multilingual is a reranking model used for semantic search and retrieval-augmented generation (RAG). Rerank Multilingual supports more than 100 languages and can be used to search within a language (for example, to search with a French query on French documents) and across languages (for example, to search with an English query on Chinese documents). Rerank enables you to significantly improve search quality by augmenting traditional keyword-based search systems with a semantic-based reranking system that can contextualize the meaning of a user's query beyond keyword relevance. Cohere's Rerank delivers higher quality results than embedding-based search, lexical search, and even hybrid search, and it requires only adding a single line of code into your application.

+Use Rerank as a ranker after initial retrieval. In other words, after an initial search system finds the top 100 most relevant documents for a larger corpus of documents.

+Rerank supports JSON objects as documents where users can specify, at query time, the fields (keys) to use for semantic search. Some other attributes of Rerank Multilingual include:

+* Context window of the model is 4,096 tokens

+* The max query length is 2,048 tokens

+Rerank multilingual performs well on multilingual benchmarks such as Miracl.

+## Deploy Cohere Rerank models as serverless APIs

+Certain models in the model catalog can be deployed as a serverless API with pay-as-you-go billing. This kind of deployment provides a way to consume models as an API without hosting them on your subscription, while keeping the enterprise security and compliance that organizations need. This deployment option doesn't require quota from your subscription.

+You can deploy the previously mentioned Cohere models as a service with pay-as-you-go billing. Cohere offers these models through the Microsoft Azure Marketplace and can change or update the terms of use and pricing of these models.

+### Prerequisites

+- An Azure subscription with a valid payment method. Free or trial Azure subscriptions won't work. If you don't have an Azure subscription, create a [paid Azure account](https://azure.microsoft.com/pricing/purchase-options/pay-as-you-go) to begin.

+- An Azure Machine Learning workspace. If you don't have these, use the steps in the [Quickstart: Create workspace resources](quickstart-create-resources.md) article to create them. The serverless API model deployment offering for Cohere Rerank is only available with workspaces created in these regions:

+ * East US

+ * East US 2

+ * North Central US

+ * South Central US

+ * West US

+ * West US 3

+ * Sweden Central

+ For a list of regions that are available for each of the models supporting serverless API endpoint deployments, see [Region availability for models in serverless API endpoints](concept-endpoint-serverless-availability.md).

+- Azure role-based access controls (Azure RBAC) are used to grant access to operations. To perform the steps in this article, your user account must be assigned the __Azure AI Developer role__ on the Resource Group.

+ For more information on permissions, see [Manage access to an Azure Machine Learning workspace](how-to-assign-roles.md).

+### Create a new deployment

+The following steps demonstrate the deployment of Cohere Rerank 3 - English, but you can use the same steps to deploy Cohere Rerank 3 - Multilingual by replacing the model name.

+To create a deployment:

+1. Go to [Azure Machine Learning studio](https://ml.azure.com/home).

+1. Select the workspace in which you want to deploy your models.

+1. Choose the model you want to deploy from the [model catalog](https://ml.azure.com/model/catalog).

+ Alternatively, you can initiate deployment by going to your workspace and selecting **Endpoints** > **Serverless endpoints** > **Create**.

+1. On the model's overview page in the model catalog, select **Deploy**.

+1. In the deployment wizard, select the link to **Azure Marketplace Terms** to learn more about the terms of use.

+1. You can also select the **Marketplace offer details** tab to learn about pricing for the selected model.

+1. If this is your first time deploying the model in the workspace, you have to subscribe your workspace for the particular offering of the model. This step requires that your account has the **Azure AI Developer role** permissions on the Resource Group, as listed in the prerequisites. Each workspace has its own subscription to the particular Azure Marketplace offering, which allows you to control and monitor spending. Select **Subscribe and Deploy**. Currently you can have only one deployment for each model within a workspace.

+1. Once you subscribe the workspace for the particular Azure Marketplace offering, subsequent deployments of the _same_ offering in the _same_ workspace don't require subscribing again. If this scenario applies to you, there's a **Continue to deploy** option to select.

+1. Give the deployment a name. This name becomes part of the deployment API URL. This URL must be unique in each Azure region.

+1. Select **Deploy**. Wait until the deployment is finished and you're redirected to the serverless endpoints page.

+1. Select the endpoint to open its Details page.

+1. You can always find the endpoint's details, URL, and access keys by navigating to **Workspace** > **Endpoints** > **Serverless endpoints**.

+1. Take note of the **Target** URL and the **Secret Key**. For more information on using the APIs, see the [reference] (#rerank-api-reference-for-cohere-rerank-models-deployed-as-a-serverless-api) section.

+To learn about billing for models deployed with pay-as-you-go, see [Cost and quota considerations for Cohere models deployed as a service](#cost-and-quota-considerations-for-models-deployed-as-a-service).

+### Consume the Cohere Rerank models as a service

+Cohere Rerank models deployed as serverless APIs can be consumed using the Rerank API.

+1. In the **workspace**, select **Endpoints** > **Serverless endpoints**.

+1. Find and select the deployment you created.

+1. Copy the **Target** URL and the **Key** token values.

+1. Cohere currently exposes `v1/rerank` for inference with the Rerank 3 - English and Rerank 3 - Multilingual models schema. For more information on using the APIs, see the [reference](#rerank-api-reference-for-cohere-rerank-models-deployed-as-a-serverless-api) section.

+## Rerank API reference for Cohere Rerank models deployed as a serverless API

+Cohere Rerank 3 - English and Rerank 3 - Multilingual accept the native Cohere Rerank API on `v1/rerank`. This section contains details about the Cohere Rerank API.

+#### v1/rerank request

+```json

+ POST /v1/rerank HTTP/1.1

+ Host: <DEPLOYMENT_URI>

+ Authorization: Bearer <TOKEN>

+ Content-type: application/json

+```

+#### v1/rerank request schema

+Cohere Rerank 3 - English and Rerank 3 - Multilingual accept the following parameters for a `v1/rerank` API call:

+| Property | Type | Default | Description |

+| | | | |

+|`query` |`string` |Required |The search query |

+|`documents` |`array` |None |A list of document objects or strings to rerank. |

+|`top_n` |`integer` |Length of `documents` |The number of most relevant documents or indices to return. |

+|`return_documents` |`boolean` |`FALSE` |If `FALSE`, returns results without the doc text - the API returns a list of {`index`, `relevance_score`} where index is inferred from the list passed into the request. </br>If `TRUE`, returns results with the doc text passed in - the API returns an ordered list of {`index`, `text`, `relevance_score`} where index + text refers to the list passed into the request. |

+|`max_chunks_per_doc` |`integer` |None |The maximum number of chunks to produce internally from a document.|

+|`rank_fields` |`array of strings` |None |If a JSON object is provided, you can specify which keys you would like to consider for reranking. The model reranks based on the order of the fields passed in (for example, `rank_fields=['title','author','text']` reranks, using the values in `title`, `author`, and `text` in that sequence. If the length of title, author, and text exceeds the context length of the model, the chunking won't reconsider earlier fields).<br> If not provided, the model uses the default text field for ranking. |

+#### v1/rerank response schema

+Response fields are fully documented on [Cohere's Rerank API reference](https://docs.cohere.com/reference/rerank). The response payload is a dictionary with the following fields:

+| Key | Type | Description |

+| | | |

+| `id` | `string` |An identifier for the response. |

+| `results` | `array of objects`|An ordered list of ranked documents, where each document is described by an object that includes `index` and `relevance_score` and, optionally, `text`. |

+| `meta` | `array of objects` | An optional meta object containing a list of warning strings. |

+<br>

+The `results` object is a dictionary with the following fields:

+| Key | Type | Description |

+| | | |

+| `document` | `object` |The document objects or strings that were reranked. |

+| `index` | `integer` |The `index` in the original list of documents to which the ranked document belongs. For example, if the first value in the `results` object has an index value of 3, it means in the list of documents passed in, the document at `index=3` had the highest relevance.|

+| `relevance_score` | `float` |Relevance scores are normalized to be in the range `[0, 1]`. Scores close to one indicate a high relevance to the query, and scores close to zero indicate low relevance. A score of `0.9` _doesn't_ necessarily mean that a document is twice as relevant as another with a score of `0.45`. |

+## Examples

+#### Request example

+```json

+ {

+ "query": "What is the capital of the United States?",

+ "rank_fields": ["Title", "Content"],

+ "documents": [

+ {"Title": "Facts about Carson City", "Content": "Carson City is the capital city of the American state of Nevada. "},

+ {"Title": "North Dakota", "Content" : "North Dakota is a state in the United States. 672,591 people lived in North Dakota in the year 2010. The capital and seat of government is Bismarck."},

+ {"Title": "Micronesia", "Content" : "Micronesia, officially the Federated States of Micronesia, is an island nation in the Pacific Ocean, northeast of Papua New Guinea. The country is a sovereign state in free association with the United States. The capital city of Federated States of Micronesia is Palikir."}

+ ],

+ "top_n": 3

+ }

+```

+#### Response example

+```json

+ {

+ "id": "571e6744-3074-457f-8935-08646a3352fb",

+ "results": [

+ {

+ "document": {

+ "Content": "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. The President of the USA and many major national government offices are in the territory. This makes it the political center of the United States of America.",

+ "Title": "Details about Washington D.C"

+ },

+ "index": 0,

+ "relevance_score": 0.98347044

+ },

+ {

+ "document": {

+ "Content": "Carson City is the capital city of the American state of Nevada. ",

+ "Title": "Facts about Carson City"

+ },

+ "index": 1,

+ "relevance_score": 0.07172112

+ },

+ {

+ "document": {

+ "Content": "Micronesia, officially the Federated States of Micronesia, is an island nation in the Pacific Ocean, northeast of Papua New Guinea. The country is a sovereign state in free association with the United States. The capital city of Federated States of Micronesia is Palikir.",

+ "Title": "Micronesia"

+ },

+ "index": 3,

+ "relevance_score": 0.05281402

+ },

+ {

+ "document": {

+ "Content": "North Dakota is a state in the United States. 672,591 people lived in North Dakota in the year 2010. The capital and seat of government is Bismarck.",

+ "Title": "North Dakota"

+ },

+ "index": 2,

+ "relevance_score": 0.03138043

+ }

+ ]

+ }

+```

+#### More inference examples

+| Package | Sample Notebook |

+|||

+|CLI using CURL and Python web requests| [cohere-rerank.ipynb](https://aka.ms/samples/cohere-rerank/webrequests)|

+|LangChain|[langchain.ipynb](https://aka.ms/samples/cohere-rerank/langchain)|

+|Cohere SDK|[cohere-sdk.ipynb](https://aka.ms/samples/cohere-rerank/cohere-python-sdk)|

+## Cost and quota considerations for models deployed as a service

+Quota is managed per deployment. Each deployment has a rate limit of 200,000 tokens per minute and 1,000 API requests per minute. However, we currently limit one deployment per model per project. Contact Microsoft Azure Support if the current rate limits aren't sufficient for your scenarios.

+Cohere models deployed as serverless APIs with pay-as-you-go billing are offered by Cohere through the Azure Marketplace and integrated with Azure Machine Learning studio for use. You can find the Azure Marketplace pricing when deploying the model.

+Each time a workspace subscribes to a given model offering from Azure Marketplace, a new resource is created to track the costs associated with its consumption. The same resource is used to track costs associated with inference; however, multiple meters are available to track each scenario independently.

+For more information on how to track costs, see [Monitor costs for models offered through the Azure Marketplace](../ai-studio/how-to/costs-plan-manage.md#monitor-costs-for-models-offered-through-the-azure-marketplace).

+## Related content

+- [Model Catalog and Collections](concept-model-catalog.md)

+- [Deploy and score a machine learning model by using an online endpoint](how-to-deploy-online-endpoints.md)

+- [Plan and manage costs for Azure AI Studio](concept-plan-manage-cost.md)

+- [Region availability for models in serverless API endpoints](concept-endpoint-serverless-availability.md)

+You can use diagnostic settings to trigger email notifications. To learn how to create diagnostic settings, see [Create diagnostic settings in Azure Monitor](/azure/azure-monitor/essentials/create-diagnostic-settings).

+To learn how to create and manage log alerts using Azure Monitor, see [Create or edit a log search alert rule](/azure/azure-monitor/alerts/alerts-create-log-alert-rule).

+ - update-code2

+ - update-code2

+Large Language Models Operations, or **LLMOps**, has become the cornerstone of efficient prompt engineering and LLM-infused application development and deployment. As the demand for LLM-infused applications continues to soar, organizations find themselves in need of a cohesive and streamlined process to manage their end-to-end lifecycle.

+If you don't yet have a workspace, create one now:

+1. Select your workspace.

+1. On the top right, select **New**.

+1. Select **Compute instance** in the list.

+ :::image type="content" source="media/quickstart-create-resources/create-compute.png" alt-text="Screenshot shows create compute in the New list.":::

+1. Supply a name.

+1. Select **Review + Create**.

+| `name` | string | Name of the job. Must be unique across all jobs in the workspace. If omitted, Azure Machine Learning autogenerates a GUID for the name. | | |

+| `display_name` | string | Display name of the job in the studio UI. Can be nonunique within the workspace. If omitted, Azure Machine Learning autogenerates a human-readable adjective-noun identifier for the display name. | | |

+| `experiment_name` | string | Experiment name to organize the job under. Each job's run record is organized under the corresponding experiment in the studio's "Experiments" tab. If omitted, Azure Machine Learning defaults it to the name of the working directory where the job was created. | | |

+| `environment` | string or object | **Required (if not using `component` field).** The environment to use for the job. Can be either a reference to an existing versioned environment in the workspace or an inline environment specification. <br><br> To reference an existing environment, use the `azureml:<environment_name>:<environment_version>` syntax or `azureml:<environment_name>@latest` (to reference the latest version of an environment). <br><br> To define an environment inline, follow the [Environment schema](reference-yaml-environment.md#yaml-syntax). Exclude the `name` and `version` properties as they aren't supported for inline environments. | | |

+| `compute` | string | Name of the compute target to execute the job on. Can be either a reference to an existing compute in the workspace (using the `azureml:<compute_name>` syntax) or `local` to designate local execution. **Note:** jobs in pipeline didn't support `local` as `compute` | | `local` |

+| `resources.instance_type` | string | The instance type to use for the job. Applicable for jobs running on Azure Arc-enabled Kubernetes compute (where the compute target specified in the `compute` field is of `type: kubernentes`). If omitted, defaults to the default instance type for the Kubernetes cluster. For more information, see [Create and select Kubernetes instance types](how-to-attach-kubernetes-anywhere.md). | | |

+| `resources.shm_size` | string | The size of the docker container's shared memory block. Should be in the format of `<number><unit>` where number has to be greater than 0 and the unit can be one of `b` (bytes), `k` (kilobytes), `m` (megabytes), or `g` (gigabytes). | | `2g` |

+| `limits.timeout` | integer | The maximum time in seconds the job is allowed to run. When this limit is reached, the system cancels the job. | | |

+| `inputs.<input_name>` | number, integer, boolean, string, or object | One of a literal value (of type number, integer, boolean, or string) or an object containing a [job input data specification](#job-inputs). | | |

+| `outputs.<output_name>` | object | You can leave the object empty, in which case by default the output is of type `uri_folder` and Azure Machine Learning generates an output location for the output. Files to the output directory are written via read-write mount. If you want to specify a different mode for the output, provide an object containing the [job output specification](#job-outputs). | |

+| `identity` | object | The identity is used for data accessing. It can be [UserIdentityConfiguration](#useridentityconfiguration), [ManagedIdentityConfiguration](#managedidentityconfiguration), or None. If UserIdentityConfiguration, the identity of job submitter is used to access, input data and write result to output folder, otherwise, the managed identity of the compute target is used. | |

+| `path` | string | The path to the data to use as input. Can be specified in a few ways: <br><br> - A local path to the data source file or folder, for example, `path: ./iris.csv`. The data gets uploaded during job submission. <br><br> - A URI of a cloud path to the file or folder to use as the input. Supported URI types are `azureml`, `https`, `wasbs`, `abfss`, `adl`. See [Core yaml syntax](reference-yaml-core-syntax.md) for more information on how to use the `azureml://` URI format. <br><br> - An existing registered Azure Machine Learning data asset to use as the input. To reference a registered data asset, use the `azureml:<data_name>:<data_version>` syntax or `azureml:<data_name>@latest` (to reference the latest version of that data asset), for example, `path: azureml:cifar10-data:1` or `path: azureml:cifar10-data@latest`. | | |

+| `mode` | string | Mode of how the data should be delivered to the compute target. <br><br> For read-only mount (`ro_mount`), the data is consumed as a mount path. A folder is mounted as a folder and a file is mounted as a file. Azure Machine Learning resolves the input to the mount path. <br><br> For `download` mode, the data is downloaded to the compute target. Azure Machine Learning resolves the input to the downloaded path. <br><br> If you only want the URL of the storage location of the data artifacts rather than mounting or downloading the data itself, you can use the `direct` mode. This mode passes in the URL of the storage location as the job input. In this case, you're fully responsible for handling credentials to access the storage. <br><br> The `eval_mount` and `eval_download` modes are unique to MLTable, and either mounts the data as a path or downloads the data to the compute target. <br><br> For more information on modes, see [Access data in a job](how-to-read-write-data-v2.md?tabs=cli#modes) | `ro_mount`, `download`, `direct`, `eval_download`, `eval_mount` | `ro_mount` |

+| `type` | string | The type of job output. For the default `uri_folder` type, the output corresponds to a folder. | `uri_folder` , `mlflow_model`, `custom_model`| `uri_folder` |

+| `mode` | string | Mode of how output files get delivered to the destination storage. For read-write mount mode (`rw_mount`), the output directory is a mounted directory. For upload mode, the files written get uploaded at the end of the job. | `rw_mount`, `upload` | `rw_mount` |

+Examples are available in the [examples GitHub repository](https://github.com/Azure/azureml-examples/tree/main/cli/jobs). The following sections show some of the examples.

+ :::image type="content" source="./media/nsg-flow-logs-portal/create-nsg-flow-log-basics.png" alt-text="Screenshot of creating an NSG flow log in the Azure portal.":::

+## Change flow log settings

+You can change the settings of a flow log after you create it. For example, you can change the flow log version or disable traffic analytics.

+ :::image type="content" source="./media/nsg-flow-logs-portal/change-flow-log.png" alt-text="Screenshot that shows how to edit flow log's settings in the Azure portal where you can change some virtual network flow log settings." lightbox="./media/nsg-flow-logs-portal/change-flow-log.png":::

+1. Select **Save** to apply the changes or **Cancel** to exit without saving them.

+1. Select **Cancel** to close the settings page without making changes.

+> If traffic analytics is enabled for a flow log, it must be disabled before you can disable the flow log. To disable traffic analytics, see [Change a flow log](#change-flow-log-settings).

+## Change flow log settings

+ | Setting | Value |

+ | - | -- |

+ | **Storage account** | |

+ | Subscription | Change the Azure subscription of the storage account that you want to use. |

+ | Storage account | Change the storage account that you want to save the flow logs to. If you want to create a new storage account, select **Create a new storage account**. |

+ | Retention (days) | Change the retention time in the storage account. Enter *0* if you want to retain the flow logs data in the storage account forever (until you manually delete the data from the storage account). |

+ | **Traffic analytics** | |

+ | Enable traffic analytics | Enable or disable traffic analytics by checking or unchecking the checkbox. |

+ | Subscription | Change the Azure subscription of the Log Analytics workspace that you want to use. |

+ | Log analytics workspace | Change the Log Analytics workspace that you want to save the flow logs to (if traffic analytics is enabled). |

+ | Traffic logging interval | Change the processing interval of traffic analytics (if traffic analytics is enabled). Available options are: one hour and 10 minutes. The default processing interval is every one hour. For more information, see [Traffic Analytics](traffic-analytics.md). |

+For a list of elements that are included in diagnostic log strings, see [Azure Monitor Logs tables](monitor-notification-hubs-reference.md#azure-monitor-logs-tables).

+[Azure Backup](relocation-backup.md)| ✅ | ❌| ❌ |

+[Azure Firewall](./relocation-firewall.md)|❌ | ✅| ❌ |

+ Title: Relocate Azure Backup to another region

+description: This article offers guidance on relocating Azure Backup to another region.

+# Customer intent: As ant administrator, I want to relocate Azure Backup to another region.

+<!-- remove ../backup/azure-backup-move-vaults-across-regions.md?toc=/azure/operational-excellence/toc.json -->

+# Relocate Azure Backup to another region

+This article covers relocation guidance for [Azure Backup](../backup/backup-overview.md) across regions.

+Azure Backup doesnΓÇÖt support the relocation of backup data from one Recovery Services vault to another. In order to continue to protect your resources, you must register and back them up to a Recovery Services vault in the new region.

+After you relocate your resources to the new region, you can choose to either keep or delete the backup data in the Recovery Services vaults in the old region.

+>[!NOTE]

+>If you do choose to keep the backup data in the old region, you do incur backup charges.

+## Prerequisites

+- Copy internal resources or settings of Azure Resource Vault.

+ - Network firewall reconfiguration

+ - Alert Notification.

+ - Move workbook if configured

+ - Diagnostic settings reconfiguration

+- List all Recovery Service Vault dependent resources. The most common dependencies are:

+ - Azure Virtual Machine (VM)

+ - Public IP address

+ - Azure Virtual Network

+ - Azure Recovery Service Vault

+- Whether the VM is moved with the vault or not, you can always restore the VM from the retained backup history in the vault.

+- Copy the backup VM configuration metadata to validate once the relocation is complete.

+- Confirm that all services and features that are in use by source resource vault are supported in the target region.

+## Prepare

+Azure Backup currently doesnΓÇÖt support the movement of backup data from one Recovery Services vault to another across regions.

+Instead, you must redeploy the Recovery Service vault and reconfigure the backup for resources to a Recovery Service vault in the new region.

+**To prepare for redeployment and configuration:**

+1. Export a Resource Manager template. This template contains settings that describe your Recovery Vault.

+ 1. Sign in to the [Azure portal](https://portal.azure.com).

+ 2. Select **All resources** and then select your Recovery Vault resource.

+ 3. Select **Export template**.

+ 4. Choose **Download** in the **Export template** page.

+ 5. Locate the .zip file that you downloaded from the portal, and unzip that file to a folder of your choice.

+ This zip file contains the .json files that include the template and scripts to deploy the template.

+

+1. Validate all the associated resources detail in the downloaded template, such as private endpoint, backup policy, and security settings.

+1. Update the parameter of the Recovery Vault by changing the value properties under parameters, such as Recovery Vault name, replication type, sku, target location etc.

+## Redeploy

+[Create and reconfigure the Recovery Service vault](/azure/backup/backup-create-recovery-services-vault) in the target region.

+Make sure to reconfigure all associated settings that were captured from the source Recovery service vault:

+- (Optional) Private Endpoint - Follow the procedure to relocate a [virtual network]](/technical-delivery-playbook/azure-services/networking/virtual-network/) as described and create the Private Endpoint.

+- Network firewall reconfiguration

+- Alert Notification.

+- Move workbook if configured

+- Diagnostic settings reconfiguration

+## Backup resources

+In order to continue to protect your resources, you must register and back them up to a Recovery Services vault in the new region. This section shows you how to back up the following:

+- [Azure Virtual Machines](#back-up-azure-virtual-machine)

+- [Azure File Share](#back-up-azure-file-share)

+- [SQL Server/SAP HANA in Azure VM](#back-up-sql-serversap-hana-in-azure-vm)

+- [On-premises resources](#back-up-services-for-on-premises-resources)

+### Back up Azure Virtual Machine

+When an Azure Virtual Machine (VM) protected by a Recovery Services vault is moved from one region to another, it can no longer be backed up to the older vault. The backups in the old vault may start failing with the errors **BCMV2VMNotFound** or [**ResourceNotFound**](../backup/backup-azure-vms-troubleshoot.md#320001-resourcenotfoundcould-not-perform-the-operation-as-vm-no-longer-exists--400094-bcmv2vmnotfoundthe-virtual-machine-doesnt-exist--an-azure-virtual-machine-wasnt-found).

+You can also choose to write a customized script for bulk VM protection:

+```azurepowershell

+https://management.azure.com/Subscriptions/{subscriptionId}/resourceGroups/{vaultresourceGroupName}/providers/Microsoft.RecoveryServices/vaults/{vaultName}/backupFabrics/{fabricName}/protectionContainers/{containerName}/protectedItems/{protectedItemName}?api-version=2019-05-13

+```

+1. Prepare Azure Virtual Machines (VMs) for relocation:

+

+ 1. See the [prerequisites associated with VM relocation](../resource-mover/tutorial-move-region-virtual-machines.md#prerequisites) and ensure that the VM is eligible for relocation.

+ 1. [Select the VM on the **Backup Items** tab](../backup/backup-azure-delete-vault.md#delete-protected-items-in-the-cloud) of existing vaultΓÇÖs dashboard and select **Stop protection** followed by retain/delete data as per your requirement. When the backup data for a VM is stopped with retain data, the recovery points remain forever and donΓÇÖt adhere to any policy.

+ >[!Note]

+ >Retaining data in the older vault will incur backup charges. If you no longer wish to retain data to avoid billing, you need to delete the retained backup data using the [Delete data option](../backup/backup-azure-manage-vms.md#delete-backup-data).

+ 1. Ensure that the VMs are turned on. All VMsΓÇÖ disks that need to be available in the destination region are attached and initialized in the VMs.

+ 1. Ensure that VMs have the latest trusted root certificates, and an updated certificate revocation list (CRL). To do so:

+ - On Windows VMs, install the latest Windows updates.

+ - On Linux VMs, refer to distributor guidance to ensure that machines have the latest certificates and CRL.

+ 1. Allow outbound connectivity from VMs:

+ - If you're using a URL-based firewall proxy to control outbound connectivity, allow access to [these URLs](../resource-mover/support-matrix-move-region-azure-vm.md#url-access).

+ - If you're using network security group (NSG) rules to control outbound connectivity, create [these service tag rules](../resource-mover/support-matrix-move-region-azure-vm.md#nsg-rules).

+1. Redeploy Azure VMs by using [Azure Resource Mover](../resource-mover/tutorial-move-region-virtual-machines.md) to relocate your VM to the new region.

+### Back up Azure File Share

+1. [Back up Azure file shares with Azure CLI](../backup/backup-afs-cli.md).

+1. Satisfy the [prerequisites to relocate the Storage Account](../storage/common/storage-account-move.md?tabs=azure-portal#prerequisites).

+1. Export and modify a Resource Move template. For more information, see [Prepare Storage Account for region relocation](../storage/common/storage-account-move.md?tabs=azure-portal#prepare).

+1. [Relocate the Azure Storage account to another region](../storage/common/storage-account-move.md).

+1. When Azure File Share is copied across regions, its associated snapshots donΓÇÖt relocate along with it. To relocate the snapshots data to the new region, you need to relocate the individual files and directories of the snapshots to the Storage Account in the new region by using [AzCopy](../storage/common/storage-use-azcopy-files.md#copy-all-file-shares-directories-and-files-to-another-storage-account).

+1. Choose whether you want to retain or delete the snapshots (and the corresponding recovery points) of the original Azure File Share by selecting your file share on the [Backup Items tab](../backup/backup-azure-delete-vault.md#delete-protected-items-in-the-cloud) of the original vaultΓÇÖs dashboard. When the backup data for Azure File Share is stopped with retain data, the recovery points remain forever and donΓÇÖt adhere to any policy.

+>[!NOTE]

+>While configuring File Share, if the Recovery Service Vault isn't available, check to see whether it is associated with another Recovery Service vault.

+### Back up SQL Server/SAP HANA in Azure VM

+When you relocate a VM that runs SQL or SAP HANA servers, you will no longer be able to back up the SQL and SAP HANA databases in the vault of the earlier region.

+**To protect the SQL and SAP HANA servers that are running in the new region:**

+1. Before you relocate SQL Server/SAP HANA running in a VM to a new region, ensure the following prerequisites are met:

+

+ 1. See the [prerequisites associated with VM relocate](../resource-mover/tutorial-move-region-virtual-machines.md#prerequisites) and ensure that the VM is eligible for relocate.

+ 1. Select the VM on the [Backup Items tab](../backup/backup-azure-delete-vault.md#delete-protected-items-in-the-cloud) of the existing vaultΓÇÖs dashboard and select _the databases_ for which backup needs to be stopped. Select **Stop protection** followed by retain/delete data as per your requirement. When the backup data is stopped with retain data, the recovery points remain forever and donΓÇÖt adhere to any policy.

+ >[!Note]

+ >Retaining data in the older vault will incur backup charges. If you no longer wish to retain data to avoid billing, you need to delete the retained backup data using [Delete data option](../backup/backup-azure-manage-vms.md#delete-backup-data).

+ 1. Ensure that the VMs to be moved are turned on. All VMs disks that need to be available in the destination region are attached and initialized in the VMs.

+ 1. Ensure that VMs have the latest trusted root certificates, and an updated certificate revocation list (CRL). To do so:

+ - On Windows VMs, install the latest Windows updates.

+ - On Linux VMs, refer to the distributor guidance and ensure that machines have the latest certificates and CRL.

+ 1. Allow outbound connectivity from VMs:

+ - If you're using a URL-based firewall proxy to control outbound connectivity, allow access to [these URLs](../resource-mover/support-matrix-move-region-azure-vm.md#url-access).

+ - If you're using network security group (NSG) rules to control outbound connectivity, create [these service tag rules](../resource-mover/support-matrix-move-region-azure-vm.md#nsg-rules).

+

+1. Relocate your VM to the new region using [Azure Resource Mover](../resource-mover/tutorial-move-region-virtual-machines.md).

+### Back up services for on-premises resources

+1. To backup files, folders, and system state for VMs (Hyper-V & VMware) and other on-premises workloads, see [About the Microsoft Azure Recovery Services (MARS)](../backup/backup-azure-about-mars.md).

+1. Download vault credentials to register the server in the vault.

+ :::image type="content" source="media\relocation\backup\mars-agent-credential-download.png" alt-text="Screenshot showing how to download vault credentials to register the server in the vault.":::

+1. Reconfigure backup agent on on-premises virtual machine.

+ :::image type="content" source="media\relocation\backup\mars-register-to-target-rsv.png" alt-text="Screenshot showing how to reconfigure an on premise virtual machine.":::

+ Title: Relocation guidance for Azure Firewall

+description: Learn how to relocate Azure Firewall to a new region

+ - subject-relocation

+# Relocate Azure Firewall to another region

+This article shows you how to relocate an Azure Firewall that protects an Azure Virtual Network.

+## Prerequisites

+- We highly recommend that you use Premium SKU. If you are on Standard SKU, consider [migrating from an existing Standard SKU Azure Firewall to Premium SKU](/azure/firewall-manager/migrate-to-policy) before you being relocation.

+- The following information must be collected in order to properly plan and execute an Azure Firewall relocation:

+ - **Deployment model.** *Classic Firewall Rules* or *Firewall policy*.

+ - **Firewall policy name.** (If *Firewall policy* deployment model is used).

+ - **Diagnostic setting at the firewall instance level.** (If Log Analytics workspace is used).

+ - **TLS (Transport Layer Security) Inspection configuration.**: (If Azure Key Vault, Certificate and Managed Identity is used.)

+ - **Public IP control.** Assess that any external identity relying on Azure Firewall public IP remains fixed and trusted.

+- Azure Firewall Standard and Premium tiers have the following dependencies that you may need to be deploy in the target region:

+ - [Azure Virtual Network](./relocation-virtual-network.md)

+ - (If used) [Log Analytics Workspace](./relocation-log-analytics.md)

+

+- If you're using the TLS Inspection feature of Azure Firewall Premium tier, the following dependencies also need to be deployed in the target region:

+ - [Azure Key Vault](./relocation-key-vault.md)

+ - [Azure Managed Identity](./relocation-managed-identity.md)

+## Downtime

+To understand the possible downtimes involved, see [Cloud Adoption Framework for Azure: Select a relocation method](/azure/cloud-adoption-framework/relocate/select#select-a-relocation-method).

+

+## Prepare

+To prepare for relocation, you need to first export and modify the template from the source region. To view a sample ARM template for Azure Firewall, see [review the template](../firewall-manager/quick-firewall-policy.md#review-the-template).

+### Export template

+# [portal](#tab/azure-portal)

+1. Sign in to the [Azure portal](https://portal.azure.com).

+2. Select **All resources** and then select your Azure Firewall resource.

+3. On the **Azure Firewall** page, select **Export template** under **Automation** in the left menu.

+4. Choose **Download** in the **Export template** page.

+5. Locate the .zip file that you downloaded from the portal, and unzip that file to a folder of your choice.

+ This zip file contains the .json files that include the template and scripts to deploy the template.

+# [PowerShell](#tab/azure-powershell)

+1. Sign in to your Azure subscription with the `Connect-AzAccount` command and follow the on-screen directions:

+```azurecli

+Connect-AzAccount

+```

+2. If your identity is associated with more than one subscription, then set your active subscription to subscription of the Azure Firewall resource that you want to move.

+```azurepowershell

+$context = Get-AzSubscription -SubscriptionId <subscription-id>

+Set-AzContext $context

+```

+3. Export the template of your source Azure Firewall resource by running the following commands:

+```azurepowershell

+$resource = Get-AzResource `

+ -ResourceGroupName <resource-group-name> `

+ -ResourceName <resource-name> `

+ -ResourceType <resource-type>

+Export-AzResourceGroup `

+ -ResourceGroupName <resource-group-name> `

+ -Resource $resource.ResourceId

+```

+4. Locate the `template.json` in your current directory.

+### Modify template

+In this section, you learn how to modify the template that you generated in the previous section.

+If you're running classic firewall rules without Firewall policy, migrate to Firewall policy before preceding with the steps in this section. To learn how to migrate from classic firewall rules to Firewall policy, see [Migrate Azure Firewall configuration to Azure Firewall policy using PowerShell](/azure/firewall-manager/migrate-to-policy).

+# [Azure portal](#tab/azure-portal)

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. If you're using Premium SKU with TLS Inspection enabled,

+ 1. [Relocate the key vault](./relocation-key-vault.md) that's used for TLS inspection into the new target region. Then, follow [the procedures](../application-gateway/key-vault-certs.md) to move certificates or generate new certificates for TLS inspection into the new key vault in the target region.

+ 1. [Relocate managed identity](./relocation-managed-identity.md) into the new target region. Reassign the corresponding roles for the key vault in the target region and subscription.

+1. In the Azure portal, select **Create a resource**.

+1. In **Search the Marketplace**, type `template deployment`, and then press **Enter**.

+1. Select **Template deployment** and the select **Create**.

+1. Select **Build your own template in the editor**.

+1. Select **Load file**, and then follow the instructions to load the `template.json` file that you downloaded in the previous section

+1. In the `template.json` file, replace:

+ - `firewallName` with the default value of your Azure Firewall name.

+ - `azureFirewallPublicIpId` with the ID of your public IP address in the target region.

+ - `virtualNetworkName` with the name of the virtual network in the target region.

+ - `firewallPolicy.id` with your policy ID.

+1. [Create a new firewall policy](/azure/firewall-manager/create-policy-powershell) using the configuration of the source region and reflect changes introduced by the new target region (IP Address Ranges, Public IP, Rule Collections).

+1. If you're using Premium SKU and you want to enable TLS Inspection, update the newly created firewall policy and enable TLS inspection by following [the instructions here](https://techcommunity.microsoft.com/t5/azure-network-security-blog/building-a-poc-for-tls-inspection-in-azure-firewall/ba-p/3676723).

+1. Review and update the configuration for the topics below to reflect the changes required for the target region.

+ - **IP Groups.** To include IP addresses from the target region, if different from the source, *IP Groups* should be reviewed. The IP addresses included in the groups must be modified.

+ - **Zones.** Configure the [availability Zones (AZ)](../reliability/availability-zones-overview.md) in the target region.

+ - **Forced Tunneling.** [Ensure that you've relcoated the virtual network](./relocation-virtual-network.md) and that the firewall *Management Subnet* is present before the Azure Firewall is relocated. Update the IP Address in the target region of the Network Virtual Appliance (NVA) to which the Azure Firewall should redirect the traffic, in the User Defined Route (UDR).

+ - **DNS.** Review IP Addresses for your custom custom *DNS Servers* to reflect your target region. If the *DNS Proxy* feature is enabled, be sure to configure your virtual network DNS server settings and set the Azure FirewallΓÇÖs private IP address as a *Custom DNS server*.

+ - **Private IP ranges (SNAT).** - If custom ranges are defined for SNAT, it's recommended that you review and eventually adjust to include the target region address space.

+ - **Tags.** - Verify and eventually update any tag that may reflect or refer to the new firewall location.

+ - **Diagnostic Settings.** When recreating the Azure Firewall in the target region, be sure to review the *Diagnostic Setting* adn configure it to reflect the target region (Log Analytics workspace, storage account, Event Hub, or 3rd-party partner solution).

+1. Edit the `location` property in the `template.json` file to the target region (The following example sets the target region to `centralus`.):

+```json

+ "resources": [

+ {

+ "type": "Microsoft.Network/azureFirewalls",

+ "apiVersion": "2023-09-01",

+ "name": "[parameters('azureFirewalls_fw_name')]",

+ "location": "centralus",}]

+```

+To find the location code for your target region, see [Data residency in Azure](https://azure.microsoft.com/explore/global-infrastructure/data-residency/#overview).

+1. Save the `template.json` file.

+# [PowerShell](#tab/azure-powershell)

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. If you're using Premium SKU with TLS Inspection enabled,

+ 1. [Relocate the key vault](./relocation-key-vault.md) used for TLS inspection into the new target region and follow the procedures to move certificates or generate new certificates for TLS inspection in the new key vault in the target region.

+ 1. [Relocate managed identity](./relocation-managed-identity.md) into the new target region and reassign the corresponding roles for the key vault in the target region and subscription.

+1. In the `template.json` file, replace:

+ - `firewallName` with the default value of your Azure Firewall name.

+ - `azureFirewallPublicIpId` with the ID of your public IP address in the target region.

+ - `virtualNetworkName` with the name of the virtual network in the target region.

+ - `firewallPolicy.id` with your policy ID.

+1. [Create a new firewall policy](/azure/firewall-manager/create-policy-powershell) using the configuration of the source region and reflect changes introduced by the new target region (IP Address Ranges, Public IP, Rule Collections).

+1. Review and update the configuration for the topics below to reflect the changes required for the target region.

+ - **IP Groups.** To include IP addresses from the target region, if different from the source, *IP Groups* should be reviewed. The IP addresses included in the groups must be modified.

+ - **Zones.** Configure the [availability Zones (AZ)](../reliability/availability-zones-overview.md) in the target region.

+ - **Forced Tunneling.** [Ensure that you've relcoated the virtual network](./relocation-virtual-network.md) and that the firewall *Management Subnet* is present before the Azure Firewall is relocated. Update the IP Address in the target region of the Network Virtual Appliance (NVA) to which the Azure Firewall should redirect the traffic, in the User Defined Route (UDR).

+ - **DNS.** Review IP Addresses for your custom custom *DNS Servers* to reflect your target region. If the *DNS Proxy* feature is enabled, be sure to configure your virtual network DNS server settings and set the Azure FirewallΓÇÖs private IP address as a *Custom DNS server*.

+ - **Private IP ranges (SNAT).** - If custom ranges are defined for SNAT, it's recommended that you review and eventually adjust to include the target region address space.

+ - **Tags.** - Verify and eventually update any tag that may reflect or refer to the new firewall location.

+ - **Diagnostic Settings.** When recreating the Azure Firewall in the target region, be sure to review the *Diagnostic Setting* adn configure it to reflect the target region (Log Analytics workspace, storage account, Event Hub, or 3rd-party partner solution).

+1. Edit the `location` property in the `template.json` file to the target region (The following example sets the target region to `centralus`.):

+```json

+ "resources": [

+ {

+ "type": "Microsoft.Network/azureFirewalls",

+ "apiVersion": "2023-09-01",

+ "name": "[parameters('azureFirewalls_fw_name')]",

+ "location": "centralus",}]

+```

+To find the location code for your target region, see [Data residency in Azure](https://azure.microsoft.com/explore/global-infrastructure/data-residency/#overview).

+## Redeploy

+Deploy the template to create a new Azure Firewall in the target region.

+# [Azure portal](#tab/azure-portal)

+1. Enter or select the property values:

+

+ - Subscription: Select an Azure subscription.

+

+ - Resource group: Select Create new and give the resource group a name.

+

+ - Location: Select an Azure location.

+1. The Azure Firewall is now deployed with the adopted configuration to reflect the needed changes in the target region.

+1. Verify configuration and functionality.

+# [PowerShell](#tab/azure-powershell)

+1. Obtain the subscription ID where you want to deploy the target public IP by running the following command:

+```azurepowershell

+Get-AzSubscription

+```

+2. Run the following commands to deploy your template:

+```azurepowershell

+$resourceGroupName = Read-Host -Prompt "Enter the Resource Group name"

+$location = Read-Host -Prompt "Enter the location (i.e. eastus)"

+New-AzResourceGroup -Name $resourceGroupName -Location "$location"

+New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateUri "<name of your local template file>"

+1. The Azure Firewall is now deployed with the adopted configuration to reflect the needed changes in the target region.

+1. Verify configuration and functionality.

+```

+## Related content

+- [Move resources to a new resource group or subscription](../azure-resource-manager/management/move-resource-group-and-subscription.md)

+- [Move Azure VMs to another region](../site-recovery/azure-to-azure-tutorial-migrate.md)

+description: Detail the cadence and process Nexus uses to release new runtime versions to customers.

+Operator Nexus releases a platform cluster runtime version with three minor versions per year and monthly patch versions in between.

+Platform Cluster patch releases are scheduled monthly to provide customers with an updated version of Azure Linux, starting in Nexus release 2408.1. These patch releases are applied to the latest minor release.

+The contents of these releases are primarily scoped to updating current versions of the HostOS and Kubernetes patch releases. Operator Nexus will also release patch platform cluster runtime releases addressing critical functional or high severity security issues. These patch runtime releases are released alongside a new management bundle release to enable the deployment of this new runtime.

+The patch releases are optional and not required to be applied. Operator Nexus certifies the various supported runtime releases to ensure there's a path to upgrade, regardless of the patch runtime release, to the next minor runtime version.

+When a customer is on a release that becomes out of support, Microsoft attempts to mitigate the customer tickets but it may not be possible to address. When a runtime minor release drops support, it will no longer be an option to deploy to a new instance.

+- The cluster continues to run; however, normal operations may start to degrade as older versions of software aren't validated

+- Support tickets raised will receive support, but the issues may not be able to be mitigated.

+- The n-3 release isn't available to customers to deploy a new instance.

+- An upgrade path is no longer supported, requiring customers to repave instances.

+- Platform Cluster runtime versions past support may continue to run but Microsoft doesn't guarantee all functionality to be compatible with the newest version of software in the Cluster Manager. An upgrade path is provided for customers on supported releases. Upgrading from an n-3 version or greater isn't supported and requires a redeployment of the site. Customers need to execute a platform cluster runtime upgrade before a site gets to n-3.

+- A requirement for the customer to update their platform cluster runtime within a year of the most recent platform cluster runtime upgrade or first deployment to ensure their Nexus instance can connect to Azure. After a year without any runtime upgrade, the Operator Nexus instance, losing connection to Azure, will require a new deployment

+ Select **Next: Azure OpenAI configuration** to create and configure Azure OpenAI connector that can be used within Elastic's AI Assistant.

+1. On **Azure OpenAI configuration**, specify the Azure OpenAI resource and the deployment that would be required to configure the connector. The details of the deployment (url, API keys etc.) are passed on to Elastic to prepare the connector to be used with Elastic's AI Assistant.

+ :::image type="content" source="media/create/configure-aoai-connector.png" alt-text="Screenshot of how to configure Azure OpenAI Connector.":::

+ >[!Note]

+ >Only deployments of text/chat completion models (like gpt4) are supported currently. Learn more about Elastic Connectors [here](https://www.elastic.co/guide/en/kibana/current/openai-action-type.html).

+ > [Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/elastic.ec-azure-pp?tab=Overview)

+## Configure Azure OpenAI Connector

+If not configured already while creating the resource, you can navigate to the **Azure OpenAI configuration** blade under the Elastic deployment configuration section. Click on **Add** to select the Azure OpenAI resource and a deployment of a text/chat completion model(like gpt4). This makes it seamless for you to have your connector ready without having to switch contexts between the AOAI resource(in Azure portal) and the Connectors page in Elastic portal, thus avoiding having to copy and paste urls and keys.

+Click on **Create**.

+Once the Connector is created, navigate to Kibana and search for Connectors under Stack Management. The newly created Azure OpenAI Connector should be visible there. This connector can be used within Elastic's Observability AI Assistant to help provide contextual responses to your natural language prompts on your observability data by invoking the Azure OpenAI deployment. Learn more about Elastic OpenAI Connectors [here](https://www.elastic.co/guide/en/kibana/current/openai-action-type.html).

+1. In the Microsoft admin center, go to **Identity** > **Applications** > **App registrations**.

+1. In the Microsoft Entra admin center, in **Identity** > **Applications** > **App registrations**, open your application by selecting its **Display name**. In the pane that opens, under **Overview**, copy the **Application ID URI**. If instead of the application ID URI you see **Add an Application ID URI**, select this option, then select **Add** and **Save**.

+1. Then select **Expose an API** in the app's left menu. Ensure the **Application ID URI** value is: `api://<Entra application ID>` where `Entra application ID` must be the same Microsoft Entra application ID.

+1. Go to the **App roles** menu of your app and select **Create app role**.

+1. Select or enter the following information in the pane that opens to create an *ExperimentationDataOwner* role. This role gives the app full access to execute all operations on the Split Experimentation resource.

+1. Create an *ExperimentationDataReader* role. This role gives the app read access on the Split Experimentation resource, but doesn't allow it to make any changes.

+1. Go to the **Overview** menu of your app and select the link under **Managed application in local directory**. This opens your app in the Microsoft admin center **Identity** > **Enterprise Application** menu.

+1. Open **Manage** > **Properties** on the left and select your preferred option for the **Assignment required** setting.

+1. Go back to the **Users and groups** menu and select **Add user/group**

+When Query Store is enabled, it saves data in aggregation windows of length determined by the `pg_qs.interval_length_minutes` server parameter (defaults to 15 minutes). For each window, it stores up to 500 distinct (with distinct userid, dbid, and queryid) queries per window. If during an interval the number of distinct queries reaches 500, the 5% with lower usage are deallocated to make room for more.

+3. Under **Source details**, for **Geo-redundant restore**, select the **Restore to paired region** checkbox.

+|Central India||:::image type="icon" source="media/yes-icon.svg":::||:::image type="icon" source="media/yes-icon.svg":::|

+- July 24, 2024: Release of SBD STONITH support using iSCSI target server or Azure shared disk in [Configuring Pacemaker on RHEL in Azure](./high-availability-guide-rhel-pacemaker.md).

+- July 19, 2024: Change in [Setting up Pacemaker on RHEL in Azure](./high-availability-guide-rhel-pacemaker.md) to add a statement around clusters spanning Virtual networks(VNets)/subnets.

+- June 19, 2024: Update the SAP high availability guides to lift the restriction of using floating IP on the NIC secondary IP address in load-balancing scenarios.

+8. **[A]** Disabling `systemd` services of the ASCS and ERS SAP instance. This step is only applicable, if SAP startup framework is managed by systemd as per SAP Note [3115048](https://me.sap.com/notes/3115048)

+ > [!NOTE]

+ > When managing SAP instances like SAP ASCS and SAP ERS using SLES cluster configuration, you would need to make additional modifications to integrate the cluster with the native systemd-based SAP start framework. This ensures that maintenance procedures do no compromise cluster stability. After installing or switching SAP startup framework to systemd-enabled setup as per SAP Note [3115048](https://me.sap.com/notes/3115048), you should disable the `systemd` services for the ASCS and ERS SAP instances.

+ ```bash

+ # Stop all ASCS and ERS instances using <sid>adm

+ sapcontrol -nr 10 -function Stop

+ sapcontrol -nr 10 -function StopService

+ sapcontrol -nr 12 -function Stop

+ sapcontrol -nr 12 -function StopService

+ # Execute below command on VM where you have performed ASCS instance installation for each SAP system (e.g. slesmsscl1)

+ sudo systemctl disable SAPNW2_10

+ sudo systemctl disable SAPNW3_20

+ # Execute below command on VM where you have performed ERS instance installation for each SAP system (e.g. slesmsscl2)

+ sudo systemctl disable SAPNW2_12

+ sudo systemctl disable SAPNW2_22

+ ```

+9. **[1]** Create the SAP cluster resources for the newly installed SAP system.

+9. **[A]** Disabling `systemd` services of the ASCS and ERS SAP instance. This step is only applicable, if SAP startup framework is managed by systemd as per SAP Note [3115048](https://me.sap.com/notes/3115048)

+ > [!NOTE]

+ > When managing SAP instances like SAP ASCS and SAP ERS using SLES cluster configuration, you would need to make additional modifications to integrate the cluster with the native systemd-based SAP start framework. This ensures that maintenance procedures do no compromise cluster stability. After installation or switching SAP startup framework to systemd-enabled setup as per SAP Note [3115048](https://me.sap.com/notes/3115048), you should disable the `systemd` services for the ASCS and ERS SAP instances.

+ # Stop ASCS and ERS instances using <sid>adm

+ sapcontrol -nr 00 -function Stop

+ sapcontrol -nr 00 -function StopService

+ sapcontrol -nr 01 -function Stop

+ sapcontrol -nr 01 -function StopService

+ # Execute below command on VM where you have performed ASCS instance installation (e.g. anftstsapcl1)

+ sudo systemctl disable SAPQAS_00

+ # Execute below command on VM where you have performed ERS instance installation (e.g. anftstsapcl2)

+ sudo systemctl disable SAPQAS_01

+ ```

+10. **[1]** Create the SAP cluster resources.

+ Depending on whether you are running an ENSA1 or ENSA2 system, select respective tab to define the resources. SAP introduced support for [ENSA2](https://help.sap.com/docs/ABAP_PLATFORM_NEW/cff8531bc1d9416d91bb6781e628d4e0/6d655c383abf4c129b0e5c8683e7ecd8.html), including replication, in SAP NetWeaver 7.52. Starting with ABAP Platform 1809, ENSA2 is installed by default. For ENSA2 support, see SAP Note [2630416](https://launchpad.support.sap.com/#/notes/2630416).

+ #### [ENSA1](#tab/ensa1)

+ ```bash

+ sudo crm configure property maintenance-mode="true"

+ # If using NFSv3

+ sudo crm configure primitive rsc_sap_QAS_ASCS00 SAPInstance \

+ # If using NFSv4.1

+ sudo crm configure primitive rsc_sap_QAS_ASCS00 SAPInstance \

+ # If using NFSv3

+ sudo crm configure primitive rsc_sap_QAS_ERS01 SAPInstance \

+ # If using NFSv4.1

+ sudo crm configure primitive rsc_sap_QAS_ERS01 SAPInstance \

+ sudo crm configure modgroup g-QAS_ASCS add rsc_sap_QAS_ASCS00

+ sudo crm configure modgroup g-QAS_ERS add rsc_sap_QAS_ERS01

+ sudo crm configure colocation col_sap_QAS_no_both -5000: g-QAS_ERS g-QAS_ASCS

+ sudo crm configure location loc_sap_QAS_failover_to_ers rsc_sap_QAS_ASCS00 rule 2000: runs_ers_QAS eq 1

+ sudo crm configure order ord_sap_QAS_first_start_ascs Optional: rsc_sap_QAS_ASCS00:start rsc_sap_QAS_ERS01:stop symmetrical=false

+ sudo crm_attribute --delete --name priority-fencing-delay

+ sudo crm node online anftstsapcl1

+ sudo crm configure property maintenance-mode="false"

+ ```

+ #### [ENSA2](#tab/ensa2)

+ > [!NOTE]

+ > If you have a two-node cluster running ENSA2, you have the option to configure priority-fencing-delay cluster property. This property introduces additional delay in fencing a node that has higher total resoure priority when a split-brain scenario occurs. For more information, see [SUSE Linux Enteprise Server high availability extension administration guide](https://documentation.suse.com/sle-ha/15-SP3/single-html/SLE-HA-administration/#pro-ha-storage-protect-fencing).

+ >

+ > The property priority-fencing-delay is only applicable for ENSA2 running on two-node cluster.

+ ```bash

+ sudo crm configure property maintenance-mode="true"

+ sudo crm configure property priority-fencing-delay=30

+ # If using NFSv3

+ sudo crm configure primitive rsc_sap_QAS_ASCS00 SAPInstance \

+ # If using NFSv4.1

+ sudo crm configure primitive rsc_sap_QAS_ASCS00 SAPInstance \

+ # If using NFSv3

+ sudo crm configure primitive rsc_sap_QAS_ERS01 SAPInstance \

+ # If using NFSv4.1

+ sudo crm configure primitive rsc_sap_QAS_ERS01 SAPInstance \

+ sudo crm configure modgroup g-QAS_ASCS add rsc_sap_QAS_ASCS00

+ sudo crm configure modgroup g-QAS_ERS add rsc_sap_QAS_ERS01

+ sudo crm configure colocation col_sap_QAS_no_both -5000: g-QAS_ERS g-QAS_ASCS

+ sudo crm configure order ord_sap_QAS_first_start_ascs Optional: rsc_sap_QAS_ASCS00:start rsc_sap_QAS_ERS01:stop symmetrical=false

+ sudo crm node online anftstsapcl1

+ sudo crm configure property maintenance-mode="false"

+ ```

+

+9. **[A]** Disabling `systemd` services of the ASCS and ERS SAP instance. This step is only applicable, if SAP startup framework is managed by systemd as per SAP Note [3115048](https://me.sap.com/notes/3115048)

+ > [!NOTE]

+ > When managing SAP instances like SAP ASCS and SAP ERS using SLES cluster configuration, you would need to make additional modifications to integrate the cluster with the native systemd-based SAP start framework. This ensures that maintenance procedures do no compromise cluster stability. After installation or switching SAP startup framework to systemd-enabled setup as per SAP Note [3115048](https://me.sap.com/notes/3115048), you should disable the `systemd` services for the ASCS and ERS SAP instances.

+ # Stop ASCS and ERS instances using <sid>adm

+ sapcontrol -nr 00 -function Stop

+ sapcontrol -nr 00 -function StopService

+ sapcontrol -nr 01 -function Stop

+ sapcontrol -nr 01 -function StopService

+ # Execute below command on VM where you have performed ASCS instance installation (e.g. sap-cl1)

+ sudo systemctl disable SAPNW1_00

+ # Execute below command on VM where you have performed ERS instance installation (e.g. sap-cl2)

+ sudo systemctl disable SAPNW1_01

+ ```

+10. **[1]** Create the SAP cluster resources

+ Depending on whether you are running an ENSA1 or ENSA2 system, select respective tab to define the resources. SAP introduced support for [ENSA2](https://help.sap.com/docs/ABAP_PLATFORM_NEW/cff8531bc1d9416d91bb6781e628d4e0/6d655c383abf4c129b0e5c8683e7ecd8.html), including replication, in SAP NetWeaver 7.52. Starting with ABAP Platform 1809, ENSA2 is installed by default. For ENSA2 support, see SAP Note [2630416](https://launchpad.support.sap.com/#/notes/2630416).

+ #### [ENSA1](#tab/ensa1)

+ ```bash

+ sudo crm configure property maintenance-mode="true"

+ sudo crm configure primitive rsc_sap_NW1_ASCS00 SAPInstance \

+ sudo crm configure primitive rsc_sap_NW1_ERS01 SAPInstance \

+ sudo crm configure modgroup g-NW1_ASCS add rsc_sap_NW1_ASCS00

+ sudo crm configure modgroup g-NW1_ERS add rsc_sap_NW1_ERS01

+ sudo crm configure colocation col_sap_NW1_no_both -5000: g-NW1_ERS g-NW1_ASCS

+ sudo crm configure location loc_sap_NW1_failover_to_ers rsc_sap_NW1_ASCS00 rule 2000: runs_ers_NW1 eq 1

+ sudo crm configure order ord_sap_NW1_first_start_ascs Optional: rsc_sap_NW1_ASCS00:start rsc_sap_NW1_ERS01:stop symmetrical=false

+ sudo crm_attribute --delete --name priority-fencing-delay

+ sudo crm node online sap-cl1

+ sudo crm configure property maintenance-mode="false"

+ ```

+ #### [ENSA2](#tab/ensa2)

+ > [!NOTE]

+ > If you have a two-node cluster running ENSA2, you have the option to configure priority-fencing-delay cluster property. This property introduces additional delay in fencing a node that has higher total resoure priority when a split-brain scenario occurs. For more information, see [SUSE Linux Enteprise Server high availability extension administration guide](https://documentation.suse.com/sle-ha/15-SP3/single-html/SLE-HA-administration/#pro-ha-storage-protect-fencing).

+ >

+ > The property priority-fencing-delay is only applicable for ENSA2 running on two-node cluster.

+ ```bash

+ sudo crm configure property maintenance-mode="true"

+ sudo crm configure property priority-fencing-delay=30

+ sudo crm configure primitive rsc_sap_NW1_ASCS00 SAPInstance \

+

+ sudo crm configure primitive rsc_sap_NW1_ERS01 SAPInstance \

+ sudo crm configure modgroup g-NW1_ASCS add rsc_sap_NW1_ASCS00

+ sudo crm configure modgroup g-NW1_ERS add rsc_sap_NW1_ERS01

+ sudo crm configure colocation col_sap_NW1_no_both -5000: g-NW1_ERS g-NW1_ASCS

+ sudo crm configure order ord_sap_NW1_first_start_ascs Optional: rsc_sap_NW1_ASCS00:start rsc_sap_NW1_ERS01:stop symmetrical=false

+ sudo crm node online sap-cl1

+ sudo crm configure property maintenance-mode="false"

+ ```

+

+9. **[A]** Disabling `systemd` services of the ASCS and ERS SAP instance. This step is only applicable, if SAP startup framework is managed by systemd as per SAP Note [3115048](https://me.sap.com/notes/3115048)

+ > [!NOTE]

+ > When managing SAP instances like SAP ASCS and SAP ERS using SLES cluster configuration, you would need to make additional modifications to integrate the cluster with the native systemd-based SAP start framework. This ensures that maintenance procedures do no compromise cluster stability. After installation or switching SAP startup framework to systemd-enabled setup as per SAP Note [3115048](https://me.sap.com/notes/3115048), you should disable the `systemd` services for the ASCS and ERS SAP instances.

+ ```bash

+ # Stop ASCS and ERS instances using <sid>adm

+ sapcontrol -nr 00 -function Stop

+ sapcontrol -nr 00 -function StopService

+ sapcontrol -nr 01 -function Stop

+ sapcontrol -nr 01 -function StopService

+ # Execute below command on VM where you have performed ASCS instance installation (e.g. sap-cl1)

+ sudo systemctl disable SAPNW1_00

+ # Execute below command on VM where you have performed ERS instance installation (e.g. sap-cl2)

+ sudo systemctl disable SAPNW1_01

+ ```

+10. **[A]** Enable `sapping` and `sappong`. The `sapping` agent runs before `sapinit` to hide the `/usr/sap/sapservices` file. The `sappong` agent runs after `sapinit` to unhide the `sapservices` file during VM boot. `SAPStartSrv` isn't started automatically for an SAP instance at boot time, because the Pacemaker cluster manages it.

+11. **[1]** Create `SAPStartSrv` resource for ASCS and ERS by creating a file and then load the file.

+12. **[1]** Create the SAP cluster resources.

+1. **[A]** Disabling `systemd` services of the ASCS and ERS SAP instance. This step is only applicable, if SAP startup framework is managed by systemd as per SAP Note [3115048](https://me.sap.com/notes/3115048)

+ > [!NOTE]

+ > When managing SAP instances like SAP ASCS and SAP ERS using SLES cluster configuration, you would need to make additional modifications to integrate the cluster with the native systemd-based SAP start framework. This ensures that maintenance procedures do no compromise cluster stability. After installation or switching SAP startup framework to systemd-enabled setup as per SAP Note [3115048](https://me.sap.com/notes/3115048), you should disable the `systemd` services for the ASCS and ERS SAP instances.

+ ```bash

+ # Stop ASCS and ERS instances using <sid>adm

+ sapcontrol -nr 00 -function Stop

+ sapcontrol -nr 00 -function StopService

+ sapcontrol -nr 01 -function Stop

+ sapcontrol -nr 01 -function StopService

+ # Execute below command on VM where you have performed ASCS instance installation (e.g. nw1-cl-0)

+ sudo systemctl disable SAPNW1_00

+ # Execute below command on VM where you have performed ERS instance installation (e.g. nw1-cl-1)

+ sudo systemctl disable SAPNW1_01

+ ```

+

+[Azure NetApp Files](../../azure-netapp-files/azure-netapp-files-introduction.md) is an Azure native, first-party, enterprise-class, high-performance file storage service certified for use with SAP HANA. It provides _Volumes as a service_ for which you create NetApp accounts, capacity pools, and volumes. With Azure NetApp Files, you select service and performance levels and manage data protection to create and manage high-performance, highly available, and scalable file shares by using the same protocols and tools that you're familiar with and rely on on-premises.

+The following types of SAP workload are supported on Azure NetApp Files volumes:

+- SAP DBMS workload

+- SAPMNT share

+- Global transport directory

+Azure NetApp Files is available in three service levels, each with their own throughput and pricing specifications. Which one is right for your deployment depends on the size of the deployment. Customized sizing recommendations are available in the [SAP on Azure NetApp Files TCO Estimator](https://aka.ms/anfsapcalc).

+For information about service levels, see [Service levels for Azure NetApp Files](../../azure-netapp-files/azure-netapp-files-service-levels.md).

+### Deploying volumes

+For optimal results, use [Application volume group for SAP HANA](../../azure-netapp-files/application-volume-group-introduction.md) to deploy the volumes. Application volume group places volumes in optimal locations in the Azure infrastructure using affinity and anti-affinity rules to reduce contention and to allow for the best throughput and lowest latency.

+> Capacity pools are a basic provisioning unit for Azure NetApp Files. Capacity pools are offered beginning at 1 TiB in size; you can expand a capacity pool in 1-TiB increments. Capacity pools are the parent unit for volumes; the smallest volume size is 100 GiB. For pricing, see [Azure NetApp Files Pricing](https://azure.microsoft.com/pricing/details/netapp/)

+Azure NetApp Files is supported for several SAP workload scenarios:

+- SAP HANA deployments using NFS shares for /han)

+- IBM Db2 in Suse or Red Hat Linux-based Azure VM

+- SAP on Oracle deployments in Oracle Linux guest OS using [dNFS](https://docs.oracle.com/en/database/oracle/oracle-database/19/ntdbi/creating-an-oracle-database-on-direct-nfs.html#GUID-2A0CCBAB-9335-45A8-B8E3-7E8C4B889DEA) for Oracle data and redo log volumes. Some more details can be found in the article [Azure Virtual Machines Oracle DBMS deployment for SAP workload](./dbms-guide-oracle.md)

+- SAP on ASE in Suse or Red Hat Linux guest OS

+- AP on MAXDB in Suse or Red Hat Linux guest OS

+- SAP on Microsoft SQL Server with SMB volumes

+> For DBMS workloads on Linux, use NFS-based volumes on Azure NetApp Files.

+### Decoupling throughput from volume size

+You can create a manual QoS capacity pool for this scenario and allocate throughput independently of the volume sizes. The total capacity required is 40 TiB, and the total throughput budget is 2500 MiB/s. A capacity pool in the Premium service level (64 MiB/s per allocated TiB) accommodates both performance and capacity requirements (40 MiB * 64 iB/s/TiB = 2560 MiB).

+Linear performance scaling would require considerable overprovisioning of the log volume to achieve the throughput requirement. To achieve the 2000 MiB/s throughput for the log volume, you'd need to deploy a capacity pool in the Ultra tier (128 MiB/s per allocated TiB) of 16 TiB, resulting in an overprovisioning and therefore a wasted capacity of 15 TiB.

+The capability matrix for SAP workload on Azure NetApp Files looks like:

+| OS base VHD | Use managed disk | - |

+| Data disk | Suitable | SAP HANA, Oracle on Oracle Linux, Db2 and SAP ASE on SLES/RHEL, MAXDB, SQL Server |

+| SAP global transport directory | Yes | SMB (Windows only) and NFS (Linux only) |

+| SAP sapmnt | Suitable |SMB (Windows only) or NFS (Linux only) |

+| Backup storage | Suitable | Use snapshots and/or Azure NetApp Files backup; log backup for HANA can also be used as file based backup destination |

+| Shares/shared disk | Yes | SMB, NFS |

+| Resiliency | LRS and GRS | [GRS with cross-region replication](../../azure-netapp-files/cross-region-replication-introduction.md); [ZRS with cross-zone replication](../../azure-netapp-files/cross-zone-replication-introduction.md) |

+| IOPS linear to capacity | Linear with auto QoS; independently configurable with Manual QoS | Three [service levels](../../azure-netapp-files/azure-netapp-files-service-levels.md) available |

+| Throughput SLA | Yes | Sizing recommendations are available in the SAP on the [Azure NetApp Files TCO Estimator](https://aka.ms/anfsapcalc) |

+| Throughput linear to capacity | Linear with auto QoS; independently configurable with Manual QoS | Three [service levels](../../azure-netapp-files/azure-netapp-files-service-levels.md) available |

+| HANA certified | [Yes](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN/#/solutions?filters=v:deCertified;ve:24&sort=Latest%20Certification&sortDesc=true) | - |

+| Disk snapshots possible | Yes | See [How Azure NetApp Files snapshots work](../../azure-netapp-files/snapshots-introduction.md) |

+| Application consistent snapshot and backup orchestration | No | Use [AzAcSnap](../../azure-netapp-files/azacsnap-introduction.md) or [SnapCenter](https://docs.netapp.com/us-en/snapcenter/concept/concept_snapcenter_overview.html) |

+| Costs | Use TCO estimation tools | Use the [SAP on Azure NetApp Files TCO Estimator](https://aka.ms/anfsapcalc) and enter the size of the landscape |

+- Capability to perform application consistent [snapshots](../..//azure-netapp-files/snapshots-introduction.md) of volume using [AzAcSnap](../../azure-netapp-files/azacsnap-introduction.md)

+- Cloning of Azure NetApp Files [volumes from snapshots](../../azure-netapp-files/snapshots-restore-new-volume.md) for testing and development

+- Restoring [volumes from from snapshots (snap-revert)](../../azure-netapp-files/snapshots-revert-volume.md) for rapid restores from corruptions and errors

+> When deploying Azure NetApp Files volumes take note of the zone in which the virtual machines are or will be deployed. Ensure you select the same zone. This functionality is documented in the article [Manage availability zone volume placement for Azure NetApp Files](../../azure-netapp-files/manage-availability-zone-volume-placement.md). Application volume group for SAP HANA uses the same functionality to deploy the volumes in the closest possible proximity to the application VMs.

+The motivation to have this type of Availability Zone alignment is the reduction of risk surface by having the NFS shares in the same availability zone as the application VMs.

+* Deploy Azure NetApp Files volumes for your SAP HANA deployment using [application volume group for SAP HANA](../../azure-netapp-files/application-volume-group-introduction.md). The advantage of Application Volume Group is that data volumes are deployed over multiple storage endpoints, reducing network contention and improving performance.

+**Summary**: Azure NetApp Files is a certified low latency storage solution for SAP HANA. The service provides volumes carved out of one or more capacity pools. Capacity pools are available in three service levels which define the total capacity and throughput allocated. The volumes can be resized, and allocated throughput can be adjusted without service interruption to cater for changing requirements and to control cost. The service provides functionality to replicate volumes to other regions or zones for disaster recovery and business continuance purposes.

+| `authResourceId` | (Optional) A string that if set, indicates that this skill should use a system managed identity on the connection to the function or app hosting the code. This property takes an application (client) ID or app's registration in Microsoft Entra ID, in any of these formats: `api://<appId>`, `<appId>/.default`, `api://<appId>/.default`. This value is used to scope the authentication token retrieved by the indexer, and is sent along with the custom Web skill API request to the function or app. Setting this property requires that your search service is [configured for managed identity](search-howto-managed-identities-data-sources.md) and your Azure function app is [configured for a Microsoft Entra sign in](../app-service/configure-authentication-provider-aad.md). To use this parameter, call the API with `api-version=2023-10-01-Preview`. |

+| `authIdentity` | (Optional) A user-managed identity used by the search service for connecting to the function or app hosting the code. You can use either a [system or user managed identity](search-howto-managed-identities-data-sources.md). To use a system manged identity, leave `authIdentity` blank. |

+[Skill reference documentation](cognitive-search-predefined-skills.md) for each skill describes the inputs it can consume. Each input has a "name" that identifies a specific input, and a "source" that specifies the location of the data in the enriched document. The following example is from the Entity Recognition skill:

+| `resourceUri` | The URI of the model provider, in this case, an Azure OpenAI resource. This parameter only supports URLs with domain `openai.azure.com`, such as `https://<resourcename>.openai.azure.com`. If the Azure OpenAI endpoint has a URL with domain `cognitiveservices.azure.com`, like `https://<resourcename>.cognitiveservices.azure.com`, a [custom subdomain](/azure/ai-services/openai/how-to/use-your-data-securely#enabled-custom-subdomain) with `openai.azure.com` must be created first for the Azure OpenAI resource and use `https://<resourcename>.openai.azure.com` instead. |

+1. Sign in to the [Azure portal](https://portal.azure.com).

+ 1. On the left menu, select **Access control (IAM)**.

+ 1. On Azure OpenAI, select **Access control (IAM)** to assign yourself to a role. The code for this quickstart runs locally. Requests to Azure OpenAI originate from your system:

+ Output should look similar to the following example. Results that are returned directly from the search engine consist of fields and their verbatim values, along with metadata like a search score and a semantic ranking score and caption if you use semantic ranking.

+ ```

+ "@search.score": 5.600783,

+ "@search.rerankerScore": 2.4191176891326904,

+ "@search.captions": [

+ {

+ "text": "Contoso Ocean Motel. Budget. pool\r\nair conditioning\r\nbar. Oceanfront hotel overlooking the beach features rooms with a private balcony and 2 indoor and outdoor pools. Various shops and art entertainment are on the boardwalk, just steps away..",

+ "highlights": "Contoso Ocean Motel. Budget.<em> pool\r\nair conditioning\r\nbar. O</em>ceanfront hotel overlooking the beach features rooms with a private balcony and 2 indoor and outdoor pools. Various shops and art entertainment are on the boardwalk, just steps away."

+ }

+ ],

+ "HotelId": "41",

+ "HotelName": "Contoso Ocean Motel",

+ "Description": "Oceanfront hotel overlooking the beach features rooms with a private balcony and 2 indoor and outdoor pools. Various shops and art entertainment are on the boardwalk, just steps away.",

+ "Category": "Budget",

+ "Tags": [

+ "pool",

+ "air conditioning",

+ "bar"

+ ],

+ ```

+In the remaining sections, you set up API calls to Azure OpenAI and Azure AI Search. Get the service endpoints so that you can provide them as variables in your code.

+This section uses Visual Studio Code and Python to call the chat completion APIs on Azure OpenAI.

+1. Start Visual Studio Code and [open the .ipynb file](https://github.com/Azure-Samples/azure-search-python-samples/tree/main/Quickstart-RAG) or create a new Python file.

+| Arizona | ✅ | ✅ | |

+| Virginia | ✅ | ✅ | ✅ |

+| `resourceUri` | The URI of the model provider, in this case, an Azure OpenAI resource. This parameter only supports URLs with domain `openai.azure.com`, such as `https://<resourcename>.openai.azure.com`. If the Azure OpenAI endpoint has a URL with domain `cognitiveservices.azure.com`, like `https://<resourcename>.cognitiveservices.azure.com`, a [custom subdomain](/azure/ai-services/openai/how-to/use-your-data-securely#enabled-custom-subdomain) with `openai.azure.com` must be created first for the Azure OpenAI resource and use `https://<resourcename>.openai.azure.com` instead. |

+- **If the source device enables configuration of the target facility**: On each source machine that sends logs to the log forwarder in CEF format, edit the Syslog configuration file to remove the facilities used to send CEF messages. This way, the facilities sent in CEF aren't also be sent in Syslog. Make sure that each DCR you configure uses the relevant facility for CEF or Syslog respectively.

+- **If changing the facility for the source appliance isn't applicable**: After you create the DCR, add ingestion time transformation to filter out CEF messages from the Syslog stream to avoid duplication. See [Tutorial: Edit a data collection rule (DCR)](../azure-monitor/essentials/data-collection-rule-edit.md). Add KQL transformation similar to the following example:

+ ```json

+ "transformKql": " source\n | where ProcessName !contains \"CEF\"\n"

+

+ Title: Deploying Microsoft Sentinel side-by-side to an existing SIEM.

+# Deploying Microsoft Sentinel side-by-side to an existing SIEM

+This article describes the approach and methods to consider when deploying Microsoft Sentinel in a side-by-side configuration together with your existing SIEM.

+## Side-by-side approach

+Use a side-by-side architecture either as a short-term, transitional phase that leads to a cloud-hosted SIEM, or as a medium- to long-term operational model, depending on the SIEM needs of your organization.

+For example, while the recommended architecture is to use a side-by-side architecture just long enough to complete a migration to Microsoft Sentinel, your organization might want to stay with your side-by-side configuration for longer, such as if you aren't ready to move away from your legacy SIEM. Typically, organizations who use a long-term, side-by-side configuration use Microsoft Sentinel to analyze only their cloud data. Many organizations avoid running multiple on-premises analytics solutions because of cost and complexity.

+Microsoft Sentinel provides [pay-as-you-go pricing](billing.md) and flexible infrastructure, giving SOC teams time to adapt to the change. Deploy and test your content at a pace that works best for your organization, and learn about how to [fully migrate to Microsoft Sentinel](migration.md).

+The following table describes the pros and cons of using a side-by-side architecture for a relatively short period of time.

+The following table describes the pros and cons of using a side-by-side architecture for a relatively medium or longer period of time.

+## Side-by-side method

+Determine how you'll configure and use Microsoft Sentinel side-by-side with your legacy SIEM.

+### Method 1: Send alerts from a legacy SIEM to Microsoft Sentinel (Recommended)

+For example, forward alerts using [Logstash](connect-logstash-data-connection-rules.md), [APIs](/rest/api/securityinsights/), or [Syslog](connect-cef-syslog-ama.md), and store them in [JSON](https://techcommunity.microsoft.com/t5/azure-sentinel/tip-easily-use-json-fields-in-sentinel/ba-p/768747) format in your Microsoft Sentinel Log Analytics workspace.

+### Method 2: Send alerts and enriched incidents from Microsoft Sentinel to a legacy SIEM

+## Streamline processes by using automation

+- [Automation in Microsoft Sentinel: Security orchestration, automation, and response (SOAR)](automation/automation.md)

+## Related content

+Consider increasing your threat protection by using Microsoft Sentinel alongside [Microsoft Defender XDR](./microsoft-365-defender-sentinel-integration.md) and [Microsoft Defender for Cloud](../security-center/azure-defender.md) for [integrated threat protection](https://www.microsoft.com/security/business/threat-protection). Benefit from the breadth of visibility that Microsoft Sentinel delivers, while diving deeper into detailed threat analysis.

+- [SOC optimizations now generally available](#soc-optimizations-now-generally-available)

+### SOC optimizations now generally available

+The SOC optimization experience in both the Azure and Defender portals is now generally available for all Microsoft Sentinel customers, including both data value and threat-based recommendations.

+- **Use data value recommendations** to improve your data usage of ingested billable logs, gain visibility to underused logs, and discover the right detections for those logs or the right adjustments to your log tier or ingestion.

+- **Use threat-based recommendations** to help identify gaps in coverage against specific attacks based on Microsoft research and mitigate them by ingesting the recommended logs and adding recommended detections.

+The [`recommendations`](soc-optimization/soc-optimization-api.md) API is still in Preview.

+For more information, see:

+- [Optimize your security operations](soc-optimization/soc-optimization-access.md)

+- [SOC optimization reference of recommendations](soc-optimization/soc-optimization-reference.md)

+description: This article provides a high-level overview of advanced features in Azure Service Bus such as sessions, scheduled delivery, autodelete on idle, etc.

+#customer intent: as a developer of messaging applications, I want to know what features are supported by Azure Service Bus to make informed decisions.

+A queue or subscription client can defer retrieval of a received message until a later time. The message might have been posted out of an expected order and the client wants to wait until it receives another message. Deferred messages remain in the queue or subscription and must be reactivated explicitly using their service-assigned sequence number. For more information, see [Message deferral](message-deferral.md).

+Autodelete on idle enables you to specify an idle interval after which a queue or topic subscription is automatically deleted. The interval is reset when a message is added to or removed from the subscription. The minimum duration is 5 minutes. For an overview on what is considered as idleness for entities, see [Idleness](message-expiration.md#idleness).

+## Batch deletion of Messages

+Azure Service Bus supports deletion of messages in batches. It's useful in scenarios when messages within queues or subscriptions have become expired, or no longer relevant, necessitating a cleanup. For more information, see [Batch delete](batch-delete.md).

+When an Azure region experiences downtime, the disaster recovery feature enables message processing to continue operating in a different region or data center. The feature keeps a structural mirror of a namespace available in the secondary region and allows the namespace identity to switch to the secondary namespace. Already posted messages remain in the former primary namespace for recovery once the availability episode subsides. For more information, see [Azure Service Bus Geo-disaster recovery](service-bus-geo-dr.md). This feature replicates only metadata (entities, configuration, properties) of Service Bus entities, not the data in them.

+## Geo replication

+The Service Bus Geo-Replication feature is one of the options to [insulate Azure Service Bus applications against outages and disasters](service-bus-outages-disasters.md), providing replication of both metadata (entities, configuration, properties) and data (message data and message property / state changes).

+Service Bus supports standard [Advanced Message Queuing Protocol (AMQP) 1.0](service-bus-amqp-overview.md) and [HTTP or REST](/rest/api/servicebus/) protocols and their respective security facilities, including transport-level security (TLS). Clients can be authorized for access using [Shared Access Signature](service-bus-sas.md) or [Microsoft Entra ID](service-bus-authentication-and-authorization.md) role-based security.

+## Related content

+#customer intent: As a developer or IT adminstrator, I want to know how to disable shared access key authentication and use only the Microsoft Entra ID authentication for higher security.

+There are two ways to authenticate to Azure Service Bus resources:

+- Microsoft Entra ID

+- Shared Access Signatures (SAS)

+Microsoft Entra ID provides superior security and ease of use over shared access signatures (SAS). With Microsoft Entra ID, thereΓÇÖs no need to store the tokens in your code and risk potential security vulnerabilities. We recommend that you use Microsoft Entra ID with your Azure Service Bus applications when possible.

+This article explains how to disable SAS key authentication (or local authentication) and use only Microsoft Entra ID for authentication.

+ :::image type="content" source="./media/disable-local-authentication/portal-overview-enabled.png" alt-text="Screenshot that shows the Overview page of a Service Bus namespace with Local Authentication set to Enabled." lightbox="./media/disable-local-authentication/portal-overview-enabled.png":::

+ :::image type="content" source="./media/disable-local-authentication/select-disabled.png" alt-text="Screenshot that shows the selection of Disabled option on the Local Authentication page.":::

+You can assign the [disable local auth](https://portal.azure.com/#blade/Microsoft_Azure_Policy/PolicyDetailBlade/definitionId/%2Fproviders%2FMicrosoft.Authorization%2FpolicyDefinitions%2Fcfb11c26-f069-4c14-8e36-56c394dae5af) Azure policy to an Azure subscription or a resource group to enforce disabling of local authentication for all Service Bus namespaces in the subscription or the resource group.

+## Related content

+> [!NOTE]

+> Messages that have expired may not be immediately removed by the broker. The broker may opt to lazily expire these messages, based on whether the entity is in active use at the time a message expires. Consequently, customers might observe an incorrect message count when using message expiration, and may even see these messages during a peek operation. However, when receiving messages, the expired message will not be included.

+ Title: Azure Service Bus message sequencing and timestamps

+#customer intent: As an architect or a developer, I want to know how the messages are sequenced (stamped with sequence number) and time-stamped in a queue or a topic in Azure Service Bus.

+The `SequenceNumber` value is a unique 64-bit integer assigned to a message as it is accepted and stored by the broker and functions as its internal identifier. For partitioned entities, the topmost 16 bits reflect the partition identifier. Sequence numbers roll over to zero when the 64-bit or 48-bit (excluding 16 bits for the partition identifier) range is exhausted.

+The sequence number can be trusted as a unique identifier since it's assigned by a central and neutral authority and not by clients. It also represents the true order of arrival, and is more precise than a time stamp as an order criterion, because time stamps might not have a high enough resolution at extreme message rates and might be subject to (however minimal) clock skew in situations where the broker ownership transitions between nodes.

+The time-stamping capability acts as a neutral and trustworthy authority that accurately captures the UTC time of arrival of a message, reflected in the `EnqueuedTimeUtc` property. The value is useful if a business scenario depends on deadlines, such as whether a work item was submitted on a certain date before midnight, but the processing is far behind the queue backlog.

+- Use the schedule message API, pass both the normal message and the scheduled time. The API returns the scheduled message's `SequenceNumber`, which you can later use to cancel the scheduled message if needed.

+The `SequenceNumber` for a scheduled message is only valid while the message is in the scheduled state. As the message transitions to the active state, the message is appended to the queue as if it had been enqueued at the current instant, which includes assigning a new `SequenceNumber`.

+> - Message enqueuing time doesn't mean that the message will be sent at the same time. It will get enqueued, but the actual sending time depends on the queue's workload and its state.

+> - Due to performance considerations, the activation and cancellation of scheduled messages are independent operations without mutual locking. If a message is in the process of being activated and is simultaneously cancelled, the activation process will not be reversed and the message will still be activated. Moreover, this can potentially lead to a negative count of scheduled messages. To minimize this race condition, we recommend that you avoid scheduling activation and cancellation operations in close succession.

+It's common to see longer-running business workflows that have an explicit time component to them, like 5-minute timeouts for 2-factor authentication, hour-long timeouts for users confirming their email address, and multi-day, week, or month long time components in domains like banking and insurance.

+## Related content

+The AMQP 1.0 protocol is designed to be extensible, enabling further specifications to enhance its capabilities. The three extension specifications discussed in this document illustrate it. For communication over existing HTTPS/WebSockets infrastructure, configuring the native AMQP TCP ports might be difficult. A binding specification defines how to layer AMQP over WebSockets. For interacting with the messaging infrastructure in a request/response fashion for management purposes or to provide advanced functionality, the AMQP management specification defines the required basic interaction primitives. For federated authorization model integration, the AMQP claims-based-security specification defines how to associate and renew authorization tokens associated with links.

+The most authoritative source to learn about how AMQP works is the [AMQP 1.0 specification](http://docs.oasis-open.org/amqp/core/v1.0/amqp-core-overview-v1.0.html), but the specification was written to precisely guide implementation and not to teach the protocol. This section focuses on introducing as much terminology as needed for describing how Service Bus uses AMQP 1.0. For a more comprehensive introduction to AMQP and a broader discussion of AMQP 1.0, review [this video course][this video course].

+The network connection is thus anchored on the container. It's initiated by the container in the client role making an outbound TCP socket connection to a container in the receiver role, which listens for and accepts inbound TCP connections. The connection handshake includes negotiating the protocol version, declaring or negotiating the use of Transport Level Security (TLS)/Secure Sockets Layer (SSL), and an authentication/authorization handshake at the connection scope that is based on SASL.

+On a link, transfers can only happen when the sender has enough *link credit*. Link credit is a counter set by the receiver using the *flow* performative, which is scoped to a link. When the sender is assigned link credit, it attempts to use up that credit by delivering messages. Each message delivery decrements the remaining link credit by 1. When the link credit is used, deliveries stop.

+#### Header

+#### Properties

+| `message-id` |Application-defined, free-form identifier for this message. Used for duplicate detection. |[MessageId](/dotnet/api/azure.messaging.servicebus.servicebusmessage.messageid) |

+| `user-id` |Application-defined user identifier, not interpreted by Service Bus. |Not accessible through the Service Bus API. |

+| `to` |Application-defined destination identifier, not interpreted by Service Bus. |[To](/dotnet/api/azure.messaging.servicebus.servicebusmessage.to) |

+| `subject` |Application-defined message purpose identifier, not interpreted by Service Bus. |[Subject](/dotnet/api/azure.messaging.servicebus.servicebusmessage.subject) |

+| `reply-to` |Application-defined reply-path indicator, not interpreted by Service Bus. |[ReplyTo](/dotnet/api/azure.messaging.servicebus.servicebusmessage.replyto) |

+| `correlation-id` |Application-defined correlation identifier, not interpreted by Service Bus. |[CorrelationId](/dotnet/api/azure.messaging.servicebus.servicebusmessage.correlationid) |

+| `content-type` |Application-defined content-type indicator for the body, not interpreted by Service Bus. |[ContentType](/dotnet/api/azure.messaging.servicebus.servicebusmessage.contenttype) |

+| `content-encoding` |Application-defined content-encoding indicator for the body, not interpreted by Service Bus. |Not accessible through the Service Bus API. |

+| `absolute-expiry-time` |Declares at which absolute instant the message expires. Ignored on input (header TTL is observed), authoritative on output. |Not accessible through the Service Bus API. |

+| `creation-time` |Declares at which time the message was created. Not used by Service Bus |Not accessible through the Service Bus API. |

+| `group-id` |Application-defined identifier for a related set of messages. Used for Service Bus sessions. |[SessionId](/dotnet/api/azure.messaging.servicebus.servicebusmessage.sessionid) |

+| `group-sequence` |Counter identifying the relative sequence number of the message inside a session. Ignored by Service Bus. |Not accessible through the Service Bus API. |

+| `reply-to-group-id` |- |[ReplyToSessionId](/dotnet/api/azure.messaging.servicebus.servicebusmessage.replytosessionid) |

+| `x-opt-scheduled-enqueue-time` | Declares at which time the message should appear on the entity |[ScheduledEnqueueTime](/dotnet/api/azure.messaging.servicebus.servicebusreceivedmessage.scheduledenqueuetime) |

+| `x-opt-partition-key` | Application-defined key that dictates which partition the message should land in. | [PartitionKey](/dotnet/api/azure.messaging.servicebus.servicebusreceivedmessage.partitionkey) |

+| `x-opt-via-partition-key` | Application-defined partition-key value when a transaction is to be used to send messages via a transfer queue. | [TransactionPartitionKey](/dotnet/api/azure.messaging.servicebus.servicebusreceivedmessage.transactionpartitionkey) |

+| `x-opt-enqueued-time` | Service-defined UTC time representing the actual time of enqueuing the message. Ignored on input. | [EnqueuedTime](/dotnet/api/azure.messaging.servicebus.servicebusreceivedmessage.enqueuedtime) |

+| `x-opt-sequence-number` | Service-defined unique number assigned to a message. | [SequenceNumber](/dotnet/api/azure.messaging.servicebus.servicebusreceivedmessage.sequencenumber) |

+| `x-opt-offset` | Service-defined enqueued sequence number of the message. | [EnqueuedSequenceNumber](/dotnet/api/azure.messaging.servicebus.servicebusreceivedmessage.enqueuedsequencenumber) |

+| `x-opt-locked-until` | Service-defined. The date and time until which the message will be locked in the queue/subscription. | [LockedUntil](/dotnet/api/azure.messaging.servicebus.servicebusreceivedmessage.lockeduntil) |

+| `x-opt-deadletter-source` | Service-Defined. If the message is received from dead letter queue, it represents the source of the original message. | [DeadLetterSource](/dotnet/api/azure.messaging.servicebus.servicebusreceivedmessage.deadlettersource) |

+To begin transactional work, the controller must obtain a `txn-id` from the coordinator. It does this by sending a `declare` type message. If the declaration is successful, the coordinator responds with a disposition outcome, which carries the assigned `txn-id`.

+The integration of Service Bus with [Virtual Network service endpoints][vnet-sep] enables secure access to messaging capabilities from workloads like virtual machines that are bound to virtual networks, with the network traffic path being secured on both ends.

+Once configured to be bound to at least one virtual network subnet service endpoint, the respective Service Bus namespace will no longer accept traffic from anywhere but the authorized virtual networks and, optionally, specific internet IP addresses. From the virtual network perspective, binding a Service Bus namespace to a service endpoint configures an isolated networking tunnel from the virtual network subnet to the messaging service.

+- Virtual Networks are supported only in [Premium tier](service-bus-premium-messaging.md) Service Bus namespaces. When using virtual network service endpoints with Service Bus, you shouldn't enable these endpoints in applications that mix standard and premium tier Service Bus namespaces. Because the standard tier doesn't support virtual networks. The endpoint is restricted to Premium tier namespaces only.

+## Advanced security scenarios enabled by virtual network integration

+## Binding Service Bus to virtual networks

+1. In the **Virtual Network** section of the page, select **+Add existing virtual network**. Select **+ Create new virtual network** if you want to create a new virtual network.

+ :::image type="content" source="./media/service-endpoints/select-subnet.png" alt-text="Screenshot showing the selection of virtual network and subnet.":::

+## Default action and public network access

+The default value of the `defaultAction` property was `Deny` for API version **2021-01-01-preview and earlier**. However, the deny rule isn't enforced unless you set IP filters or virtual network rules. That is, if you didn't have any IP filters or virtual network rules, it's treated as `Allow`.

+From API version **2021-06-01-preview onwards**, the default value of the `defaultAction` property is `Allow`, to accurately reflect the service-side enforcement. If the default action is set to `Deny`, IP filters and virtual network rules are enforced. If the default action is set to `Allow`, IP filters and virtual network rules aren't enforced. The service remembers the rules when you turn them off and then back on again.

+Azure portal always uses the latest API version to get and set properties. If you had previously configured your namespace using **2021-01-01-preview and earlier** with `defaultAction` set to `Deny`, and specified zero IP filters and virtual network rules, the portal would have previously checked **Selected Networks** on the **Networking** page of your namespace. Now, it checks the **All networks** option.

+> Confirm that your test endpoint ends with a slash (/) to ensure that the CSS file is loaded correctly. If your browser requires you to enter login credentials to view the page, use [URL decode](https://www.urldecoder.org/) to decode your test endpoint. URL decode returns a URL in the format `https://\<username>:\<password>@\<cluster-name>.test.azuremicroservices.io/demo/green`. Use this format to access your endpoint. If you want to disable basic authentication for your test endpoint, run the following Azure CLI command: `az spring app update --resource-group <resource-group-name> --service <Azure-Spring-Apps-instance-name> --name demo --disable-test-endpoint-auth true`

+1. You might receive a certificate warning during the sign-in process. Select **Yes** or **Continue** to create the connection.

+1. Copy the script from the Azure portal and paste it into **Notepad**, as in the following example.

+ :::image type="content" source="medilet-resize.png":::

+> Deprecation and disablement notification for Azure Synapse Runtime for Apache Spark 3.1

+> * Effective August 29, 2024, **disablement** of jobs running on Azure Synapse Runtime for Apache Spark 3.1 will be executed. **Immediately** migrate to higher runtime versions otherwise your jobs will stop executing.

+> * **All Spark jobs running on Azure Synapse Runtime for Apache Spark 3.1 will be disabled as of August 29, 2024.**

+ * End of Support for Azure Synapse Runtime for Apache Spark 3.1 announced January 26, 2023.

+ * Effective January 26, 2024, the Azure Synapse has stopped official support for Spark 3.1 Runtimes.

+ * Post January 26, 2024, we will not be addressing any support tickets related to Spark 3.1. There will be no release pipeline in place for bug or security fixes for Spark 3.1. Utilizing Spark 3.1 post the support cutoff date is undertaken at one's own risk. We strongly discourage its continued use due to potential security and functionality concerns.

+ * Recognizing that certain customers may need additional time to transition to a higher runtime version, we are temporarily extending the usage option for Spark 3.1, but we will not provide any official support for it.

+ * **We strongly advise proactively upgrading workloads to a more recent version of the runtime (e.g., [Azure Synapse Runtime for Apache Spark 3.4 (GA)](./apache-spark-34-runtime.md))**.

+- Source tables with collations that are unsupported by dedicated SQL pools, such as UTF-8 and certain Japanese collations, can't be replicated. See [supported collations in Synapse SQL pools](../sql/reference-collation-types.md).

+ * Additionally, Azure Synapse Link for SQL does not support some Thai language collations:

+ * `Thai100CaseInsensitiveAccentInsensitiveKanaSensitive`

+ * `Thai100CaseInsensitiveAccentSensitiveSupplementaryCharacters`

+ * `Thai100CaseSensitiveAccentInsensitiveKanaSensitive`

+ * `Thai100CaseSensitiveAccentInsensitiveKanaSensitiveWidthSensitiveSupplementaryCharacters`

+ * `Thai100CaseSensitiveAccentSensitiveKanaSensitive`

+ * `Thai100CaseSensitiveAccentSensitiveSupplementaryCharacters`

+ * `ThaiCaseSensitiveAccentInsensitiveWidthSensitive`

+- When Azure Synapse Link for SQL on Azure SQL Database or SQL Server 2022 is enabled, the aggressive log truncation feature of Accelerated Database Recovery (ADR) is automatically disabled. This is necessary because Azure Synapse Link for SQL accesses the database transaction log. This behavior is similar to changed data capture (CDC). Active transactions continue to hold the transaction log truncation until the transaction commits and Azure Synapse Link for SQL catches up, or transaction aborts. This might result in the transaction log filling up more than usual and should be monitored so that the transaction log does not fill.

+# [Windows IoT Enterprise on Arc enabled IaaS VMs (preview)](#tab/winio-arc)

+## July 2024

+### Support for Windows IoT Enterprise on Arc enabled IaaS VMs

+Public preview: Azure Update Manager now supports Windows IoT Enterprise on Arc enabled IaaS VMs. For more information, see [supported Windows IoT enterprise releases](https://learn.microsoft.com/azure/update-manager/support-matrix?tabs=winio-arc%2Cpublic%2Cthird-party-win#support-for-check-for-updatesone-time-updateperiodic-assessment-and-scheduled-patching).

+- When you configure client device redirection for the Remote Desktop app or Windows App on iOS and iPadOS, multifactor authentication (MFA) requests might get stuck in a loop. A common scenario of this issue happens when the Remote Desktop app or Windows App is being run on an Intune enrolled iPhone and the same iPhone is being used to receive MFA requests from the Microsoft Authenticator app when signing into the Remote Desktop app or Windows App. To work around this issue, use the Remote Desktop app or Windows App on a different device (such as an iPad) from the device being used to receive MFA requests (such as an iPhone).

+To configure the clipboard using Intune, follow these steps. This process creates an Intune [settings catalog](/mem/intune/configuration/settings-catalog) policy.

+1. Sign in to the [Microsoft Intune admin center](https://intune.microsoft.com/).

+1. Select **Devices** > **Manage devices** > **Configuration** > **Create** > **New policy**.

+1. Enter the following properties:

+ - **Platform**: Select **Windows 10 and later**.

+ - **Profile type**: Select **Settings catalog**.

+1. Select **Create**.

+1. In **Basics**, enter the following properties:

+ - **Name**: Enter a descriptive name for the profile. Name your profile so you can easily identify it later.

+ - **Description**: Enter a description for the profile. This setting is optional, but recommended.

+1. Select **Next**.

+1. In **Configuration settings**, select **Add settings**. Then:

+ 1. In the settings picker, expand **Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Device and Resource Redirection**.

+ 1. Select the following settings and make sure you select the settings with the correct scope. The `(User)` settings apply to the user scope. The other settings apply to the device scope. To determine which scope is correct for your scenario, go to [Settings catalog - Device scope vs. user scope settings](/mem/intune/configuration/settings-catalog#device-scope-vs-user-scope-settings):

+ - Restrict clipboard transfer from server to client

+ - Restrict clipboard transfer from client to server

+ **OR**

+ - Restrict clipboard transfer from server to client (User)

+ - Restrict clipboard transfer from client to server (User)

+ 1. Close the settings picker.

+1. Configure the settings:

+ - **Restrict clipboard transfer from server to client**: Select **Enabled**.

+ - **Restrict clipboard transfer from server to client**: Select the type of clipboard data you want to prevent or allow. Your options:

+ - Disable clipboard transfers from server to client

+ - Allow plain text

+ - Allow plain text and images

+ - Allow plain text, images, and Rich Text Format

+ - Allow plain text, images, Rich Text Format, and HTML

+ - **Restrict clipboard transfer from client to server**: Select **Enabled**.

+ - **Restrict clipboard transfer from client to server**: Select the type of clipboard data you want to prevent or allow. Your options:

+ - Disable clipboard transfers from server to client

+ - Allow plain text

+ - Allow plain text and images

+ - Allow plain text, images, and Rich Text Format

+ - Allow plain text, images, Rich Text Format, and HTML

+1. Select **Next**.

+1. At the **Scope tags** tab (optional), you can skip this step. For more information about scope tags in Intune, see [Use RBAC roles and scope tags for distributed IT](/mem/intune/fundamentals/scope-tags).

+ Select **Next**.

+1. For the **Assignments** tab, select the users, devices, or groups to receive the profile, then select **Next**. For more information on assigning profiles, see [Assign user and device profiles](/mem/intune/configuration/device-profile-assign).

+ Install-Module CimDiskImage

+The platform can return errors on VMs while performing Automatic Image Upgrade with Rolling Upgrade policy. The [Get Instance View](/rest/api/compute/virtual-machine-scale-sets/get-instance-view) of a VM contains the detailed error message to investigate and resolve an error. The [Rolling Upgrades - Get Latest](/rest/api/compute/virtual-machine-scale-sets/get) can provide more details on rolling upgrade configuration and status. The [Get OS Upgrade History](/rest/api/compute/virtual-machine-scale-sets/get-os-upgrade-history) provides details on the last image upgrade operation on the scale set. Below are the topmost errors that can result in Rolling Upgrades.

+# Convert the disk type of an Azure managed disk

+There are five disk types of Azure managed disks: Azure Ultra Disks, Premium SSD v2, premium SSD, Standard SSD, and Standard HDD. You can easily switch between Premium SSD, Standard SSD, and Standard HDD based on your performance needs. Premium SSD and Standard SSD are also available with [Zone-redundant storage](disks-redundancy.md#zone-redundant-storage-for-managed-disks). For most cases, you can't yet switch from or to an Ultra Disk or a Premium SSD v2, you must deploy a new one with a snapshot of an existing disk. However, you can sign up for a preview and then you can switch from existing disks to a Premium SSD v2. See [Migrate to Premium SSD v2 or Ultra Disk using snapshots](#migrate-to-premium-ssd-v2-or-ultra-disk-using-snapshots) for details.

+## Convert Premium SSD v2 disks (preview)

+As a public preview, you can switch existing disks to Premium SSD v2 disks the same way you do for other disk types. Use [this survey](https://aka.ms/SeamlessMigrationCustomerSurvey) to sign up for the preview. Premium SSD v2 disks have some limitations, see the [Premium SSD v2 limitations](disks-deploy-premium-v2.md#limitations) section of their article to learn more.

+The preview allowing direct switching to Premium SSD v2 disks has some additional limitations and regional restrictions:

+- You can't switch an OS disk to a Premium SSD v2 disk.

+- Existing disks can only be directly switched to 512 sector size Premium SSD v2 disks.

+- You can only perform 40 conversions at the same time per subscription per region.

+- If your existing disk is a shared disk, you must detach all VMs before changing to Premium SSD v2.

+- If your existing disk is using host caching, you must [set it to none](#disable-host-caching) before changing to Premium SSD v2.

+- If your existing disk is using bursting, you must [disable it](#disable-bursting) before changing to Premium SSD v2.

+- If your existing disk is using double encryption, you must [switch to one of the single encryption options](#disable-double-encryption) before changing to Premium SSD v2.

+- You can't directly switch from a Premium SSD v2 to another disk type. If you want to change a Premium SSD v2 to another disk type, you must migrate using [snapshots](#migrate-to-premium-ssd-v2-or-ultra-disk-using-snapshots).

+- You can't directly switch from Ultra Disks to Premium SSD v2 disks, you must migrate using [snapshots](#migrate-to-premium-ssd-v2-or-ultra-disk-using-snapshots).

+- If you're using the rest API, you must use an API version `2020-12-01` or newer for both the Compute Resource Provider and the Disk Resource Provider.

+This preview is currently only available in the following regions:

+- Central US

+- East US

+- East US 2

+- US West

+- West Europe

+- North Europe

+- West US 2

+- East Asia

+- Southeast Asia

+- Central India

+- France Central

+### Disable host caching

+If your disk is using host caching, you must disable it before converting to Premium SSD v2. You can use the following CLI script to identify your disk's LUN and disable host caching. Replace `yourResourceGroup` and `nameOfYourVM` with your own values, then run the script.

+```azurecli

+$myRG="yourResourceGroup"

+$myVM="nameOfYourVM"

+lun=$(az vm show -g $myRG -n $myVM --query "storageProfile.dataDisks[].lun")

+az vm update --resource-group $myRG --name $myVM --disk-caching $lun=None

+```

+### Disable bursting

+If your disk is using bursting, you must disable it before converting to Premium SSD v2. If you enabled bursting within 12 hours, you have to wait until the 13th hour or later to disable it.

+You can use the following command to disable disk bursting: `az disk update --name "yourDiskNameHere" --resource-group "yourRGNameHere" --enable-bursting false`

+### Disable double encryption

+If your disk is using double encryption, you must disable it before converting to Premium SSD v2. You can use the following command to change your disk from double encryption to encryption at rest with customer-managed keys:

+```azurecli

+az disk-encryption-set update --name "nameOfYourDiskEncryptionSetHere" --resource-group "yourRGNameHere" --key-url yourKeyURL --source-vault "yourKeyVaultName" --encryption-type EncryptionAtRestWithCustomerKey

+```

+rgName='yourResourceGroup'

+rgName='yourResourceGroup'

+## Migrate to Premium SSD v2 or Ultra Disk using snapshots

+Zone-redundant storage (ZRS) disks synchronously replicate data across three availability zones, which are separated groups of data centers in a region that have independent power, cooling, and networking infrastructure. With ZRS disks, your data is accessible even in the event of a zonal outage. Also, ZRS data disks allow you to [forcibly detach](/rest/api/compute/virtual-machines/attach-detach-data-disks?view=rest-compute-2024-03-01&tabs=HTTP#diskdetachoptiontypes) (preview) them from VMs experiencing issues. ZRS disks have limitations, see [Zone-redundant storage for managed disks](disks-redundancy.md#zone-redundant-storage-for-managed-disks) for details.

+A ZRS disk lets you recover from failures in availability zones. If a zone went down and your virtual machine (VM) wasn't affected, then your workloads continue running. But if your VM was affected by an outage and you want to recover before it's resolved, you can either take a snapshot or make a copy of your ZRS disks. Once you've created new disks, attach them to another VM. Alternatively, if you're using the [force detach](/rest/api/compute/virtual-machines/attach-detach-data-disks?view=rest-compute-2024-03-01&tabs=HTTP#diskdetachoptiontypes) (preview) feature, you can forcibly detach ZRS data disks from failed VMs and attach them to another VM.

+- To convert an LRS disk to ZRS, see [Convert a disk from LRS to ZRS](disks-migrate-lrs-zrs.md).

+ "enableAutomaticUpgrade": true,

+ "enableAutomaticUpgrade": true,

+ enableAutomaticUpgrade: true

+ enableAutomaticUpgrade: true

+### InternalError / InternalExecutionError / InternalOperationError / InternalDiskRestorePointError - An internal execution error occurred. Please retry later.

+**Error code**: InternalError / InternalExecutionError / InternalOperationError / InternalDiskRestorePointError

+Spot VMs can be stopped if Azure needs capacity for other pay-as-you-go workloads or when the price of the spot instance exceeds the maximum price that you have set. When creating an Azure Spot Virtual Machine, you can set the eviction policy to *Deallocate* (default) or *Delete*.

+You can opt in to receive in-VM notifications through [Azure Scheduled Events](./linux/scheduled-events.md). These are delivered on a best effort basis up to 30 seconds prior to the eviction.

+| Price for the VM has gone up and is now > the max price. | The VM gets evicted. Azure will attempt scheduled event delivery up to 30 seconds before actual eviction. |

+[Trusted launch](trusted-launch.md) is a way to enable foundational compute security on [Azure Generation 2 VMs](generation-2.md) and protects against advanced and persistent attack techniques like boot kits and rootkits. It does so by combining infrastructure technologies like Secure Boot, virtual Trusted Platform Module (vTPM), and boot integrity monitoring on your VM.

+Depends: libc6, lib32gcc-s1, libgssapi-krb5-2, libstdc++6, zlib1g, libicu72|libicu71|libicu70|libicu69|libicu68|libicu67|libicu66|libicu65|libicu63|libicu60|libicu57|libicu55|libicu52, libssl3|libssl1.1|libssl1.0.2|libssl1.

+sudo apt-get download lib32gcc-s1

+sudo apt-get download lib32gcc-s1

+sudo apt-get download lib32gcc-s1

+If the installer executable file doesn't support an uninstall parameter, you can sometimes look up the registry on a test machine to know where the uninstaller is located.

+The Azure CLI is used to create and manage Azure resources from the command line or in scripts. This quickstart shows you how to use the Azure CLI to deploy a virtual machine (VM) in Azure that runs Windows Server 2022. To see your VM in action, you then RDP to the VM and install the IIS web server.

+Support for EUS RHEL7 ended in June 30, 2028. For more information, see [Red Hat Enterprise Linux Extended Maintenance](https://access.redhat.com/support/policy/updates/errata/#Long_Support).

+- RHEL 7.9 EUS support ended June 30, 2028

+>Support for RHEL7 EUS ended in June 30, 2028. It is not recommended to switch to EUS repositories in RHEL7 anymore.

+ Title: Configure routing preference for a public IP address

+description: Learn how to create a public IP with an Internet traffic routing preference using the Azure portal, Azure PowerShell, or Azure CLI.

+# Configure routing preference for a public IP address

+This article shows you how to configure [routing preference](routing-preference-overview.md) via ISP network (**Internet** option) for a public IP address using the Azure portal, Azure PowerShell, or Azure CLI. After creating the public IP address, you can associate it with the following Azure resources for inbound and outbound traffic to the internet:

+## Prerequisites

+# [Azure portal](#tab/azureportal)

+# [Azure CLI](#tab/azurecli/)

+- This article requires version 2.0.49 or later of the Azure CLI. If using Azure Cloud Shell, the latest version is already installed.

+# [Azure PowerShell](#tab/azurepowershell/)

+If you choose to install and use PowerShell locally, this article requires the Azure PowerShell module version 6.9.0 or later. Run `Get-Module -ListAvailable Az` to find the installed version. If you need to upgrade, see [Install Azure PowerShell module](/powershell/azure/install-azure-powershell). If you are running PowerShell locally, you also need to run `Connect-AzAccount` to create a connection with Azure.

+# [Azure portal](#tab/azureportal)

+# [Azure CLI](#tab/azurecli/)

+Create a resource group with the [az group create](/cli/azure/group#az-group-create) command. The following example creates a resource group in the **East US** Azure region:

+```azurecli

+ az group create --name myResourceGroup --location eastus

+```

+## Create a public IP address

+Create a Public IP Address with routing preference of **Internet** type using command [az network public-ip create](/cli/azure/network/public-ip#az-network-public-ip-create), with the format as shown below.

+The following command creates a new public IP with **Internet** routing preference in the **East US** Azure region.

+```azurecli

+az network public-ip create \

+--name MyRoutingPrefIP \

+--resource-group MyResourceGroup \

+--location eastus \

+--ip-tags 'RoutingPreference=Internet' \

+--sku STANDARD \

+--allocation-method static \

+--version IPv4

+```

+> [!NOTE]

+> Currently, routing preference only supports IPV4 public IP addresses.

+You can associate the above created public IP address with a [Windows](../../virtual-machines/windows/overview.md?toc=%2fazure%2fvirtual-network%2ftoc.json) or [Linux](../../virtual-machines/linux/overview.md?toc=%2fazure%2fvirtual-network%2ftoc.json) virtual machine. Use the CLI section on the tutorial page: [Associate a public IP address to a virtual machine](./associate-public-ip-address-vm.md) to associate the Public IP to your VM. You can also associate the public IP address created above with an [Azure Load Balancer](../../load-balancer/load-balancer-overview.md), by assigning it to the load balancer **frontend** configuration. The public IP address serves as a load-balanced virtual IP address (VIP).

+# [Azure PowerShell](#tab/azurepowershell/)

+The following command creates a new public IP with a routing preference type as *Internet* in the *East US* Azure region:

+```azurepowershell

+$iptagtype="RoutingPreference"

+$tagName = "Internet"

+$ipTag = New-AzPublicIpTag -IpTagType $iptagtype -Tag $tagName

+# attach the tag

+$publicIp = New-AzPublicIpAddress `

+-Name "MyPublicIP" `

+-ResourceGroupName $rg.ResourceGroupName `

+-Location $rg.Location `

+-IpTag $ipTag `

+-AllocationMethod Static `

+-Sku Standard `

+-IpAddressVersion IPv4

+```

+You can associate the above created public IP address with a [Windows](../../virtual-machines/windows/overview.md?toc=%2fazure%2fvirtual-network%2ftoc.json) or [Linux](../../virtual-machines/linux/overview.md?toc=%2fazure%2fvirtual-network%2ftoc.json) virtual machine. Use the CLI section on the tutorial page: [Associate a public IP address to a virtual machine](./associate-public-ip-address-vm.md) to associate the Public IP to your VM. You can also associate the public IP address created above with an [Azure Load Balancer](../../load-balancer/load-balancer-overview.md), by assigning it to the load balancer **frontend** configuration. The public IP address serves as a load-balanced virtual IP address (VIP).

+- [Configure routing preference for a VM using the Azure CLI](./configure-routing-preference-virtual-machine-cli.md).

+- [Configure routing preference for a VM using the Azure PowerShell](./configure-routing-preference-virtual-machine-powershell.md).

+ Title: 'Configure VPN clients for P2S OpenVPN protocol connections: Microsoft Entra authentication: macOS'

+Make sure you have the following prerequisites before you proceed with the steps in this article:

+* Before you can connect and authenticate using Microsoft Entra ID, you must first configure your Microsoft Entra tenant. For more information, see [Configure a Microsoft Entra tenant](openvpn-azure-ad-tenant.md).

+ Title: 'Migrate manually registered Azure VPN client to Microsoft-registered for P2S Microsoft Entra ID authentication'

+# Migrate a manually registered Azure VPN Client to the Microsoft-registered client

+This article helps you migrate from a manually registered Azure VPN Client to the Microsoft-registered Azure VPN Client for point-to-site (P2S) Microsoft Entra ID authentication. The Microsoft-registered Azure VPN client uses a different Audience value. When you update an Audience value, you must make the change on both the P2S VPN gateway, and on any previously configured VPN clients.

+For more information about Audience values, see [About point-to-site VPN - Microsoft Entra ID authentication](point-to-site-about.md#entra-id). The examples in this article use the new Audience value for Azure Public.

+The following table shows the available supported Audience values.

+1. Change the **Audience** value. For this example, we changed the Audience value to the Azure Public value for the Microsoft-registered Azure VPN Client; **c632b3df-fb67-4d84-bdcf-b95ad541b5c8**.

+* If you've updated multiple values on the P2S gateway, or you want easily update the VPN clients by importing the new values, you can generate and download a new P2S VPN [client profile configuration package](#generate) and import it to each client.

+### <a name="manual"></a>Update an Azure VPN Client

+These steps help you update the Azure VPN Client manually, without using the profile configuration package.

+### <a name="generate"></a>Update using a profile configuration package

+Make sure you have the following prerequisites before you proceed with the steps in this article:

+ Title: 'Design highly available gateway connectivity'

+description: Learn about highly available configuration options for VPN Gateway.

+# Design highly available gateway connectivity for cross-premises and VNet-to-VNet connections

+This article helps you understand how to design highly available gateway connectivity for cross-premises and VNet-to-VNet connections.