-Learn more about [Azure Cosmos DB account - CosmosDBFreeTierOverage (Review the configuration of your Azure Cosmos DB free tier account)](../cosmos-db/understand-your-bill.md#azure-free-tier).

-Learn more about [Azure Cosmos DB account - CosmosDBAutoscaleRecommendations (Enable autoscale on your Azure Cosmos DB database or container)](../cosmos-db/provision-throughput-autoscale.md).

-Learn more about [Azure Cosmos DB account - CosmosDBMigrateToManualThroughputFromAutoscale (Configure manual throughput instead of autoscale on your Azure Cosmos DB database or container)](../cosmos-db/how-to-choose-offer.md).

-Learn more about [Azure Cosmos DB account - CosmosDBAttachments (Migrate Azure Cosmos DB attachments to Azure Blob Storage)](../cosmos-db/attachments.md#migrating-attachments-to-azure-blob-storage).

-Learn more about [Azure Cosmos DB account - CosmosDBMigrateToContinuousBackup (Improve resiliency by migrating your Azure Cosmos DB accounts to continuous backup)](../cosmos-db/continuous-backup-restore-introduction.md).

-Learn more about [Azure Cosmos DB account - CosmosDBOrderByHighRUCharge (Add composite indexes to your Azure Cosmos DB container)](../cosmos-db/index-policy.md#composite-indexes).

-Learn more about [Azure Cosmos DB account - CosmosDBDefaultIndexingWithManyPaths (Optimize your Azure Cosmos DB indexing policy to only index what's needed)](../cosmos-db/index-policy.md).

-Learn more about [Azure Cosmos DB account - CosmosDBUpgradeOldSDK (Upgrade your old Azure Cosmos DB SDK to the latest version)](../cosmos-db/index.yml).

-Learn more about [Azure Cosmos DB account - CosmosDBUpgradeOutdatedSDK (Upgrade your outdated Azure Cosmos DB SDK to the latest version)](../cosmos-db/index.yml).

-Learn more about [Azure Cosmos DB account - CosmosDBFixedCollections (Configure your Azure Cosmos DB containers with a partition key)](../cosmos-db/partitioning-overview.md#choose-partitionkey).

-Learn more about [Azure Cosmos DB account - CosmosDBSingleRegionProdAccounts (Add a second region to your production workloads on Azure Cosmos DB)](../cosmos-db/high-availability.md).

-Learn more about [Azure Cosmos DB account - CosmosDBKeyVaultWrap (Your Azure Cosmos DB account is unable to access its linked Azure Key Vault hosting your encryption key)](../cosmos-db/how-to-setup-cmk.md).

-There's a critical bug in version 2.6.13 and lower, of the Azure Cosmos DB Async Java SDK v2 causing errors when a Global logical sequence number (LSN) greater than the Max Integer value is reached. These service errors happen after a large volume of transactions occur in the lifetime of an Azure Cosmos DB container. Note: There's a critical hotfix for the Async Java SDK v2, however we still highly recommend you migrate to the [Java SDK v4](../cosmos-db/sql/sql-api-sdk-java-v4.md).

-Learn more about [Azure Cosmos DB account - CosmosDBMaxGlobalLSNReachedV2 (Upgrade to 2.6.14 version of the Async Java SDK v2 to avoid a critical issue or upgrade to Java SDK v4 as Async Java SDK v2 is being deprecated)](../cosmos-db/sql/sql-api-sdk-async-java.md).

-Learn more about [Azure Cosmos DB account - CosmosDBMaxGlobalLSNReachedV4 (Upgrade to the current recommended version of the Java SDK v4 to avoid a critical issue)](../cosmos-db/sql/sql-api-sdk-java-v4.md).

-> * The [Request Units](../../cosmos-db/request-units.md) for the Azure Cosmos DB automatically scale as needed.

-The following code snippets show you how to use an access token with the [Face SDK for C#](https://www.nuget.org/packages/Microsoft.Azure.CognitiveServices.Vision.Face).

-The following class uses an access token to create a **ServiceClientCredentials** object that can be used to authenticate a Face API client object. It automatically adds the access token as a header in every request that the Face client will make.

-public class LimitedAccessTokenWithApiKeyClientCredential : ServiceClientCredentials

- /// <summary>

- /// Creates a new instance of the LimitedAccessTokenWithApiKeyClientCredential class

- /// </summary>

- /// <param name="apiKey">API Key for the Face API or CognitiveService endpoint</param>

- /// <param name="limitedAccessToken">LimitedAccessToken to bypass the limited access program, requires ISV sponsership.</param>

- public LimitedAccessTokenWithApiKeyClientCredential(string apiKey, string limitedAccessToken)

- {

- this.ApiKey = apiKey;

- this.LimitedAccessToken = limitedAccessToken;

- private readonly string ApiKey;

- private readonly string LimitedAccesToken;

- /// <summary>

- /// Add the Basic Authentication Header to each outgoing request

- /// </summary>

- /// <param name="request">The outgoing request</param>

- /// <param name="cancellationToken">A token to cancel the operation</param>

- public override Task ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)

- {

- if (request == null)

- throw new ArgumentNullException("request");

- request.Headers.Add("Ocp-Apim-Subscription-Key", ApiKey);

- request.Headers.Add("LimitedAccessToken", $"Bearer {LimitedAccesToken}");

- return Task.FromResult<object>(null);

- }

-}

-static void Main(string[] args)

-{

- var faceClient = new FaceClient(new LimitedAccessTokenWithApiKeyClientCredential(apiKey: "<client-face-key>", limitedAccessToken: "<token>"));

- faceClient.Endpoint = "https://mytest-eastus2.cognitiveservices.azure.com";

- using (var stream = File.OpenRead("photo.jpg"))

- var result = faceClient.Face.DetectWithStreamAsync(stream, detectionModel: "Detection_03", recognitionModel: "Recognition_04", returnFaceId: true).Result;

- Console.WriteLine(JsonConvert.SerializeObject(result));

-See the following code example for the .NET client library.

-You can also specify a detection model when you add a face to an existing **FaceList** object. See the following code example for the .NET client library.

-Prompt Shields have been specifically trained and tested on the following languages: Chinese, English, French, German, Italian, Japanese, Portuguese. However, the feature can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.

- \"incidentName\": \"<text-incident-name>\",

- \"incidentDefinition\": \"string\"

-}'

- "IncidentSamples": [

- { "text": "<text-example-1>"},

- { "text": "<text-example-2>"},

- "text": "<test-text>",

- "incidentNames": [

- "<text-incident-name>"

- "IncidentSamples": [

- "image": {

- "content": "<base64-data>",

- "bloburl": "<your-blob-storage-url>.png"

- "image": {

- "url": "<your-blob-storage-url>/image.png",

- "incidentNames": [

- "<image-incident-name>"

- "IncidentSampleIds": [

- "<your-incident-sample-id>"

- "IncidentSampleIds": [

- "<your-incident-sample-id>"

-> Other Azure AI Content Safety models have been specifically trained and tested on the following languages: Chinese, English, French, German, Italian, Japanese, Portuguese. However, these features can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.

-* **[Concepts](concepts/harm-categories.md)** provide in-depth explanations of the service functionality and features.

-## Product types

-| Type | Functionality |

-| :-- | :- |

-| [Prompt Shields](/rest/api/contentsafety/text-operations/detect-text-jailbreak) (preview) | Scans text for the risk of a [User input attack](./concepts/jailbreak-detection.md) on a Large Language Model. [Quickstart](./quickstart-jailbreak.md) |

-| [Groundedness detection](/rest/api/contentsafety/text-groundedness-detection-operations/detect-groundedness-options) (preview) | Detects whether the text responses of large language models (LLMs) are grounded in the source materials provided by the users. [Quickstart](./quickstart-groundedness.md) |

-| [Protected material text detection](/rest/api/contentsafety/text-operations/detect-text-protected-material) (preview) | Scans AI-generated text for [known text content](./concepts/protected-material.md) (for example, song lyrics, articles, recipes, selected web content). [Quickstart](./quickstart-protected-material.md)|

-| Custom categories API (preview) | Lets you create and train your own [custom content categories](./concepts/custom-categories.md) and scan text for matches. [Quickstart](./quickstart-custom-categories.md) |

-| Custom categories (rapid) API (preview) | Lets you define [emerging harmful content patterns](./concepts/custom-categories.md) and scan text and images for matches. [How-to guide](./how-to/custom-categories-rapid.md) |

-| [Analyze text](/rest/api/contentsafety/text-operations/analyze-text) API | Scans text for sexual content, violence, hate, and self harm with multi-severity levels. |

-| [Analyze image](/rest/api/contentsafety/image-operations/analyze-image) API | Scans images for sexual content, violence, hate, and self harm with multi-severity levels. |

-Content Safety models have been specifically trained and tested in the following languages: English, German, Japanese, Spanish, French, Italian, Portuguese, and Chinese. However, the service can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.

-## Next steps

-Combine Groundedness detection with other LLM safety features like Prompt Shields.

-> [!div class="nextstepaction"]

-> [Prompt Shields quickstart](./quickstart-jailbreak.md)

-## Next steps

-Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.

-> [!div class="nextstepaction"]

-> [Content Safety Studio quickstart](./studio-quickstart.md)

-## Next steps

-Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.

-> [!div class="nextstepaction"]

-> [Content Safety Studio quickstart](./studio-quickstart.md)

-## Next steps

-Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.

-> [!div class="nextstepaction"]

-> [Content Safety Studio quickstart](./studio-quickstart.md)

-## Next steps

-Configure filters for each category and test on datasets using [Content Safety Studio](studio-quickstart.md), export the code and deploy.

-> [!div class="nextstepaction"]

-> [Content Safety Studio quickstart](./studio-quickstart.md)

-description: Use the custom neural document model to train a model to extract data from structured, semistructured, and unstructured documents.

-| Custom document | [Document Intelligence 3.1](/rest/api/aiservices/document-models/analyze-document?view=rest-aiservices-2023-07-31&preserve-view=true&tabs=HTTP)| [Document Intelligence SDK](quickstarts/get-started-sdks-rest-api.md?view=doc-intel-3.0.0&preserve-view=true)| [Document Intelligence Studio](https://formrecognizer.appliedai.azure.com/studio)

-```REST

-```REST

-```REST

-

-Starting with version `2024-07-31-preview`, you can train your custom neural model for longer durations than 30 minutes. Previous versions have been capped at 30 minutes per training instance, with a total of 20 free training instances per month. Now with `2024-07-31-preview`, you can receive **10 hours** of free model training, and train a model for as long as 10 hours. If you would like to train a model for longer than 10 hours, billing charges are calculated for model trainings that exceed 10 hours. You can choose to spend all of 10 free hours on a single build with a large set of data, or utilize it across multiple builds by adjusting the maximum duration value for the `build` operation by specifying `maxTrainingHours` as below:

-> [!NOTE]

-> For Document Intelligence versions `v3.1 (2023-07-31)` and `v3.0 (2022-08-31)`, custom neural model's paid training is not enabled. For the two older versions, you will get a maximum of 30 minutes training duration per model. If you would like to train more than 20 model instances, you can request for increase in the training limit.

-Each training hour is the amount of compute a single V100 GPU can perform in an hour. As each build takes different amount of time, billing is calculated for the actual time spent (excluding time in queue), with a minimum of 30 minutes per training job. The elapsed time is converted to V100 equivalent training hours and reported as part of the model.

-This billing structure enables you to train larger data sets for longer durations with flexibility in the training hours.

-For Document Intelligence versions `v3.1 (2023-07-31)` and `v3.0 (2022-08-31)`, you will get a maximum of 30 minutes training duration per model, and a maximum of 20 trainings for free per month. If you would like to train more than 20 model instances, you can request for increase in the training limit.

-If you are interested in training models for longer durations than 30 minutes, we support **paid training** for our newest version, `v4.0 (2024-07-31)`. Using the latest version, you can train your model for a longer duration to process larger documents.

-For Document Intelligence versions `v3.1 (2023-07-31)` and `v3.0 (2022-08-31)`, you will get a maximum of 30 minutes training duration per model, and a maximum of 20 trainings for free per month. If you would like to train more than 20 model instances, you can request for increase in the training limit.

-If you are interested in training models for longer durations than 30 minutes, we support **paid training** for our newest version, `v4.0 (2024-07-31)`. Using the latest version, you can train your model for a longer duration to process larger documents.

-* **Connection string**: The connection string to access your Azure Cosmos DB instance. This can be found in the Azure Cosmos DB resource in the Azure portal, in **Keys**. For more information, see [Secure access to data in Azure Cosmos DB](../../cosmos-db/secure-access-to-data.md).

-## Changes between 2024-4-01-preview and 2024-05-01-preview API specification

-If you're using the OpenAI Python client library or the REST API, you'll need to update your code directly to the latest preview API version.

-If you're using one of the Azure OpenAI SDKs for C#, Go, Java, or JavaScript you'll instead need to update to the latest version of the SDK. Each SDK release is hardcoded to work with specific versions of the Azure OpenAI API.

-To prepare for model retirements and version upgrades, we recommend that customers evaluate their applications with the new models and versions and evaluate their behavior. We also recommend that customers update their applications to use the new models and versions before the retirement date.

-For more information, see [How to upgrade to a new model or version](./model-versions.md).

-| Model | Version | Retirement date |

-| - | - | - |

-| `gpt-35-turbo` | 0301 | No earlier than October 1, 2024 |

-| `gpt-35-turbo`<br>`gpt-35-turbo-16k` | 0613 | November 1, 2024 |

-| `gpt-35-turbo` | 1106 | No earlier than Nov 17, 2024 |

-| `gpt-35-turbo` | 0125 | No earlier than Feb 22, 2025 |

-| `gpt-4`<br>`gpt-4-32k` | 0314 | **Deprecation:** October 1, 2024 <br> **Retirement:** June 6, 2025 |

-| `gpt-4`<br>`gpt-4-32k` | 0613 | **Deprecation:** October 1, 2024 <br> **Retirement:** June 6, 2025 |

-| `gpt-4` | 1106-preview | To be upgraded to `gpt-4` Version: `turbo-2024-04-09`, starting on November 15, 2024, or later **¹** |

-| `gpt-4` | 0125-preview |To be upgraded to `gpt-4` Version: `turbo-2024-04-09`, starting on November 15, 2024, or later **¹** |

-| `gpt-4` | vision-preview | To be upgraded to `gpt-4` Version: `turbo-2024-04-09`, starting on November 15, 2024, or later **¹** |

-| `gpt-3.5-turbo-instruct` | 0914 | No earlier than Sep 14, 2025 |

-| `text-embedding-ada-002` | 2 | No earlier than April 3, 2025 |

-| `text-embedding-ada-002` | 1 | No earlier than April 3, 2025 |

-| `text-embedding-3-small` | | No earlier than Feb 2, 2025 |

-| `text-embedding-3-large` | | No earlier than Feb 2, 2025 |

-| | | - | -- |

-For Assistants you need a combination of a supported model, and a supported region. Certain tools and capabilities require the latest models. The following models are available in the Assistants API, SDK, Azure AI Studio and Azure OpenAI Studio. The following table is for pay-as-you-go. For information on Provisioned Throughput Unit (PTU) availability, see [provisioned throughput](./provisioned-throughput.md). The listed models and regions can be used with both Assistants v1 and v2.

-Prompt transformation is a process in DALL-E 3 image generation that applies a safety and quality system message to your original prompt using a large language model (LLM) call before being sent to the model for image generation. This system message enriches your original prompt with the goal of generating more diverse and higher-quality images, while maintaining intent.

-Prompt transformation is applied to all Azure OpenAI DALL-E 3 requests by default. There may be scenarios in which your use case requires a lower level of enrichment. To generate images that use prompts that more closely resemble your original prompt, append this text to your prompt: `I NEED to test how the tool works with extremely simple prompts. DO NOT add any detail, just use it AS-IS:`. This ensures there is minimal prompt transformation. Evaluating your system behavior with and without this prompt helps you better understand the impact and value of prompt transformation.

-> [!NOTE]

-> Azure OpenAI Service does not offer configurability for prompt transformation at this time. To bypass prompt transformation, prepend the following to any request: `I NEED to test how the tool works with extremely simple prompts. DO NOT add any detail, just use it AS-IS:`.

->

-> While this addition will encourage the revised prompt to be more representative of your original prompt, the system may alter specific details.

-An embedding is a special format of data representation that machine learning models and algorithms can easily use. The embedding is an information dense representation of the semantic meaning of a piece of text. Each embedding is a vector of floating-point numbers, such that the distance between two embeddings in the vector space is correlated with semantic similarity between two inputs in the original format. For example, if two texts are similar, then their vector representations should also be similar. Embeddings power vector similarity search in Azure Databases such as [Azure Cosmos DB for MongoDB vCore](../../../cosmos-db/mongodb/vcore/vector-search.md) , [Azure SQL Database](/azure/azure-sql/database/ai-artificial-intelligence-intelligent-applications?view=azuresql&preserve-view=true#vector-search) or [Azure Database for PostgreSQL - Flexible Server](../../../postgresql/flexible-server/how-to-use-pgvector.md).

-* Store your embeddings and perform vector (similarity) search using [Azure Cosmos DB for MongoDB vCore](../../../cosmos-db/mongodb/vcore/vector-search.md), [Azure Cosmos DB for NoSQL](../../../cosmos-db/rag-data-openai.md) , [Azure SQL Database](/azure/azure-sql/database/ai-artificial-intelligence-intelligent-applications?view=azuresql&preserve-view=true#vector-search) or [Azure Database for PostgreSQL - Flexible Server](../../../postgresql/flexible-server/how-to-use-pgvector.md).

-An embedding is a special format of data representation that can be easily utilized by machine learning models and algorithms. The embedding is an information dense representation of the semantic meaning of a piece of text. Each embedding is a vector of floating point numbers, such that the distance between two embeddings in the vector space is correlated with semantic similarity between two inputs in the original format. For example, if two texts are similar, then their vector representations should also be similar. Embeddings power vector similarity search in Azure Databases such as [Azure Cosmos DB for MongoDB vCore](../../../cosmos-db/mongodb/vcore/vector-search.md) , [Azure SQL Database](/azure/azure-sql/database/ai-artificial-intelligence-intelligent-applications?view=azuresql&preserve-view=true#vector-search) or [Azure Database for PostgreSQL - Flexible Server](../../../postgresql/flexible-server/how-to-use-pgvector.md).

- * [Azure Cosmos DB for MongoDB vCore](../../../cosmos-db/mongodb/vcore/vector-search.md)

- * [Azure Cosmos DB for NoSQL](../../../cosmos-db/vector-search.md)

- * [Azure Cosmos DB for PostgreSQL](../../../cosmos-db/postgresql/howto-use-pgvector.md)

- * [Azure Database for PostgreSQL - Flexible Server](../../../postgresql/flexible-server/how-to-use-pgvector.md)

-| Models available | **GPT-4o**<br> **GPT-4 series (including GPT-4 Turbo with Vision)** <br>**GPT-3.5-Turbo series**<br> Embeddings series <br> Learn more in our [Models](./concepts/models.md) page.|

-| Fine-tuning | `GPT-4` (preview) <br>`GPT-3.5-Turbo` (0613) <br> `babbage-002` <br> `davinci-002`.|

- * [Azure Cosmos DB for MongoDB vCore](../../../cosmos-db/mongodb/vcore/vector-search.md)

- * [Azure Cosmos DB for NoSQL](../../../cosmos-db/vector-search.md)

- * [Azure Cosmos DB for PostgreSQL](../../../cosmos-db/postgresql/howto-use-pgvector.md)

-Here are some property options that you can use to configure a transcription when you call the [Transcriptions_Create](/rest/api/speechtotext/transcriptions/create) operation.

-Embedded speech isn't supported on RHEL/CentOS 7.

- <version>1.38.0</version>

- implementation 'com.microsoft.cognitiveservices.speech:client-sdk-embedded:1.38.0@aar'

- Environment.GetEnvironmentVariable("MODEL_KEY"));

- Environment.GetEnvironmentVariable("VOICE_KEY"));

- GetEnvironmentVariable("MODEL_KEY"));

- GetEnvironmentVariable("VOICE_KEY"));

- System.getenv("MODEL_KEY"));

- System.getenv("VOICE_KEY"));

-> [!CAUTION]

-> This article references CentOS, a Linux distribution that is End Of Life (EOL) status. Please consider your use and planning accordingly. For more information, see the [CentOS End Of Life guidance](~/articles/virtual-machines/workloads/centos/centos-end-of-life.md).

-description: Learn how to configure RHEL/CentOS 7 so that the Speech SDK can be used.

-# Configure RHEL/CentOS 7

-> [!CAUTION]

-> This article references CentOS, a Linux distribution that is End Of Life (EOL) status. Please consider your use and planning accordingly. For more information, see the [CentOS End Of Life guidance](~/articles/virtual-machines/workloads/centos/centos-end-of-life.md).

-To use the Speech SDK on Red Hat Enterprise Linux (RHEL) 7 x64 and CentOS 7 x64, update the C++ compiler (for C++ development) and the shared C++ runtime library on your system.

-## Install dependencies

-First install all general dependencies:

-```bash

-sudo rpm -Uvh https://packages.microsoft.com/config/rhel/7/packages-microsoft-prod.rpm

-# Install development tools and libraries

-sudo yum update -y

-sudo yum groupinstall -y "Development tools"

-sudo yum install -y alsa-lib dotnet-sdk-2.1 java-1.8.0-openjdk-devel openssl

-sudo yum install -y gstreamer1 gstreamer1-plugins-base gstreamer1-plugins-good gstreamer1-plugins-bad-free gstreamer1-plugins-ugly-free

-```

-## C/C++ compiler and runtime libraries

-Install the prerequisite packages with this command:

-```bash

-sudo yum install -y gmp-devel mpfr-devel libmpc-devel

-```

-Next update the compiler and runtime libraries:

-```bash

-# Build GCC 7.5.0 and runtimes and install them under /usr/local

-curl https://ftp.gnu.org/gnu/gcc/gcc-7.5.0/gcc-7.5.0.tar.gz -O

-tar -xf gcc-7.5.0.tar.gz

-mkdir gcc-7.5.0-build && cd gcc-7.5.0-build

-../gcc-7.5.0/configure --enable-languages=c,c++ --disable-bootstrap --disable-multilib --prefix=/usr/local

-make -j$(nproc)

-sudo make install-strip

-```

-If the updated compiler and libraries need to be deployed on several machines, you can copy them from under `/usr/local` to other machines. If only the runtime libraries are needed, then the files in `/usr/local/lib64` are enough.

-## Environment settings

-Run the following commands to complete the configuration:

-```bash

-# Add updated C/C++ runtimes to the library path

-# (this is required for any development/testing with Speech SDK)

-export LD_LIBRARY_PATH=/usr/local/lib64:$LD_LIBRARY_PATH

-# For C++ development only:

-# - add the updated compiler to PATH

-# (note, /usr/local/bin should be already first in PATH on vanilla systems)

-# - add Speech SDK libraries from the Linux tar package to LD_LIBRARY_PATH

-# (note, use the actual path to extracted files!)

-export PATH=/usr/local/bin:$PATH

-hash -r # reset cached paths in the current shell session just in case

-export LD_LIBRARY_PATH=/path/to/extracted/SpeechSDK-Linux-<version>/lib/centos7-x64:$LD_LIBRARY_PATH

-```

-> [!NOTE]

-> The Linux .tar package contains specific libraries for RHEL/CentOS 7. These are in `lib/centos7-x64` as shown in the environment setting example for `LD_LIBRARY_PATH` above. Speech SDK libraries in `lib/x64` are for all the other supported Linux x64 distributions (including RHEL/CentOS 8) and don't work on RHEL/CentOS 7.

-## Next steps

-> [!div class="nextstepaction"]

-> [About the Speech SDK](speech-sdk.md)

-1. On the **Create and Manage Projects** page, select **Upload file**.

-1. On the **Video file** page, upload your video file by dragging and dropping the video file or selecting the file manually.

- Ensure the video is in .mp4 format, less than 500 MB, and shorter than 60 minutes.

-1. Provide the **File name**, **Description**, and select **Voice type**, **Language of the video**, **Translate to** language.

- You can select **Prebuilt neural voice** or **Personal voice** for **Voice type**. For prebuilt neural voice, the system automatically selects the most suitable prebuilt voice by matching the speaker's voice in the video with prebuilt voices. For personal voice, the system provides the model with superior voice cloning similarity. To use personal voice, you need to apply for access. The application form will be available soon.

- :::image type="content" source="media/video-translation/upload-video-file.png" alt-text="Screenshot of uploading your video file on the video file page.":::

- When processing the video file, you can check the processing status on the project tab.

- Once the upload is complete, the project is created. You can then select the project to review detailed settings and make adjustments according to your preferences.

-If you make changes multiple times but haven't finished, you only need to save the changes you've made by selecting **Save**. After making all changes, select **Apply changes** to apply them to the video. You'll be charged only after you select **Apply changes**.

-Hubs are the primary top-level Azure resource for AI studio and provide a central way for a team to govern security, connectivity, and computing resources across playgrounds and projects. Once a hub is created, developers can create projects from it and access shared company resources without needing an IT administrator's repeated help.

-When you create a new hub, a set of dependent Azure resources are required to store data that you upload or get generated when working in AI studio. If not provided by you, and required, these resources are automatically created.

-description: In this article, you create an Azure AI hub, an AI project, an AI services resource, and more resources.

-> * Build an Azure AI hub

-> * Develop an AI project

-1. Once the extension is installed and authenticated to your Azure subscription, use the following command to create a new Azure AI project from an existing Azure AI hub:

-> Storage connections are not created directly with the project when your storage account has public network access set to disabled. These are created instead when a first user accesses AI studio over a private network connection. [Troubleshoot storage connections](troubleshoot-secure-connection-project.md#troubleshoot-missing-storage-connections)

-Models deployed in Azure Machine Learning and Azure AI studio in Serverless API endpoints support the [Azure AI Model Inference API](../reference/reference-model-inference-api.md) that exposes a common set of capabilities for foundational models and that can be used by developers to consume predictions from a diverse set of models in a uniform and consistent way.

-To set the PNA flag for the Azure AI hub:

-2. Search for the Resource group to which the hub belongs, and select your Azure AI hub from the resources listed for this Resource group.

-3. On the hub Overview page, use the left navigation pane to go to Settings > Networking.

-| [Contoso Chat Retail copilot with Azure AI Studio](https://github.com/Azure-Samples/contoso-chat) | [Azure AI Studio online endpoints](../../../machine-learning/concept-endpoints-online.md) | [Azure Cosmos DB](../../../cosmos-db/index-overview.md), [Azure Managed Identity](/entr), Bicep | A retailer conversation agent that can answer questions grounded in your product catalog and customer order history. This template uses a retrieval augmented generation architecture with cutting-edge models for chat completion, chat evaluation, and embeddings. Build, evaluate, and deploy, an end-to-end solution with a single command. |

-| [Contoso Chat Retail copilot with .NET and Semantic Kernel](https://github.com/Azure-Samples/contoso-chat-csharp-prompty) | [Azure Container Apps](../../../container-apps/overview.md) | [Azure Cosmos DB](../../../cosmos-db/index-overview.md), [Azure Monitor](../../../azure-monitor/overview.md), [Azure Managed Identity](/entr), [Semantic Kernel](/semantic-kernel/overview/?tabs=Csharp), Bicep | A retailer conversation agent that can answer questions grounded in your product catalog and customer order history. This template uses a retrieval augmented generation architecture with cutting-edge models for chat completion, chat evaluation, and embeddings. Build, evaluate, and deploy, an end-to-end solution with a single command. |

-After logging your custom evaluator to your AI project, you can view it in your [Evaluator library](../evaluate-generative-ai-app.md#view-and-manage-the-evaluators-in-the-evaluator-library) under Evaluation tab in AI studio.

-After logging your custom evaluator to your AI project, you can view it in your [Evaluator library](../evaluate-generative-ai-app.md#view-and-manage-the-evaluators-in-the-evaluator-library) under Evaluation tab in AI studio.

- # Optionally provide your AI Studio project information to track your evaluation results in your Azure AI studio project

- - OR you can use any another embedding model deployed in your AI studio project. In this example we use Cohere multi-lingual embedding. The instructions to deploy this model can be found [here](../deploy-models-cohere-embed.md).

-### Build an index locally using other embedding models deployed in your AI studio project

-To create an index that uses an embedding model deployed in your AI studio project, we configure the connection to the model using a `ConnectionConfig` as shown below. The `subscription`, `resource_group` and `workspace` refers to the project where the embedding model is installed. The `connection_name` refers to the connection name for the model, which can be found in the AI Studio project settings page.

-You can connect to embedding model deployed in your AI studio project (non Azure OpenAI models) using the serverless connection.

-Azure AI studio is a regional service and stores data both service-side and on a storage account in your subscription. If a regional disaster occurs, service data can't be recovered. But you can recover the data stored by the service on the storage account in your subscription given storage redundancy is enforced. Service-side stored data is mostly metadata (tags, asset names, descriptions). Stored on your storage account is typically non-metadata, for example, uploaded data.

- > For Meta Llama 3.1 models, the pay-as-you-go model fine-tune offering is only available with AI hubs created in **West US 3** regions.

- - On the Azure subscriptionΓÇöto subscribe the Azure AI project to the Azure Marketplace offering, once for each project, per offering:

- - On the Azure AI projectΓÇöto deploy endpoints (the Azure AI Developer role contains these permissions already):

-The following models are available in Azure AI studio for Phi 3 when fine-tuning as a service with pay-as-you-go:

-The following models are available in Azure AI studio for Phi 3 when fine-tuning as a service with pay-as-you-go:

-1. Select **AI project settings** > **Resource providers** from the left menu.

-1. On the fine-tune wizard, select the link to **Azure AI studio Terms** to learn more about the terms of use. You can also select the **Azure AI studio offer details** tab to learn about pricing for the selected model.

-1. If this is your first time fine-tuning the model in the project, you have to subscribe your project for the particular offering (for example, Phi-3-mini-128k-instruct) from Azure AI studio. This step requires that your account has the Azure subscription permissions and resource group permissions listed in the prerequisites. Each project has its own subscription to the particular Azure AI studio offering, which allows you to control and monitor spending. Select **Subscribe and fine-tune**.

- > Subscribing a project to a particular Azure AI studio offering (in this case, Phi-3-mini-128k-instruct) requires that your account has **Contributor** or **Owner** access at the subscription level where the project is created. Alternatively, your user account can be assigned a custom role that has the Azure subscription permissions and resource group permissions listed in the [prerequisites](#prerequisites).

-1. Once you sign up the project for the particular Azure AI studio offering, subsequent fine-tuning of the _same_ offering in the _same_ project don't require subscribing again. Therefore, you don't need to have the subscription-level permissions for subsequent fine-tune jobs. If this scenario applies to you, select **Continue to fine-tune**.

-1. On the fine-tune wizard, select the link to **Azure AI studio Terms** to learn more about the terms of use. You can also select the **Azure AI studio offer details** tab to learn about pricing for the selected model.

-1. If this is your first time fine-tuning the model in the project, you have to subscribe your project for the particular offering (for example, Phi-3-medium-128k-instruct) from Azure AI studio. This step requires that your account has the Azure subscription permissions and resource group permissions listed in the prerequisites. Each project has its own subscription to the particular Azure AI studio offering, which allows you to control and monitor spending. Select **Subscribe and fine-tune**.

- > Subscribing a project to a particular Azure AI studio offering (in this case, Phi-3-mini-128k-instruct) requires that your account has **Contributor** or **Owner** access at the subscription level where the project is created. Alternatively, your user account can be assigned a custom role that has the Azure subscription permissions and resource group permissions listed in the [prerequisites](#prerequisites).

-1. Once you sign up the project for the particular Azure AI studio offering, subsequent fine-tuning of the _same_ offering in the _same_ project don't require subscribing again. Therefore, you don't need to have the subscription-level permissions for subsequent fine-tune jobs. If this scenario applies to you, select **Continue to fine-tune**.

-* If you have an AI studio hub with MaaS deployments created before July 11, 2024, and you enable a private endpoint on this hub, the existing MaaS deployments won't follow the hub's networking configuration. For serverless API deployments in the hub to follow the hub's networking configuration, you need to create the deployments again.

-The `cosmosdb-data-source` resolver policy resolves data for an object type and field in a GraphQL schema by using a [Cosmos DB](../cosmos-db/introduction.md) data source. The schema must be imported to API Management as a GraphQL API.

-| [query-request](#query-request-attributes) | Specifies settings for a [query request](../cosmos-db/nosql/how-to-dotnet-query-items.md) to Cosmos DB container. | Configure one of `query-request`, `read-request`, `delete-request`, or `write-request` |

-| [read-request](#read-request-elements) | Specifies settings for a [read request](../cosmos-db/nosql/how-to-dotnet-read-item.md) to Cosmos DB container. | Configure one of `query-request`, `read-request`, `delete-request`, or `write-request` |

-| [partition-key](#partition-key-attributes) | A Cosmos DB [partition key](../cosmos-db/resource-model.md#azure-cosmos-db-containers) to route the query to the location in the container. | No |

-| [paging](#paging-elements) | Specifies settings to split query results into multiple [pages](../cosmos-db/nosql/query/pagination.md). | No |

-| [max-item-count](#max-item-count-attribute) | Specifies the [maximum number of items](../cosmos-db/nosql/query/pagination.md) returned by the query. Set to -1 if you don't want to place a limit on the number of results per query execution. | Yes |

-| [continuation-token](#continuation-token-attribute) | Specifies the [continuation token](../cosmos-db/nosql/query/pagination.md#continuation-tokens) to attach to the query to get the next set of results. | Yes |

-| consistency-level | String. Sets the Cosmos DB [consistency level](../cosmos-db/consistency-levels.md) of the delete request. | No | N/A |

-| pre-trigger | String. Identifier of a [pre-trigger](../cosmos-db/nosql/how-to-use-stored-procedures-triggers-udfs.md#how-to-run-pre-triggers) function that is registered in your Cosmos DB container. | No | N/A |

-| post-trigger | String. Identifier of a [post-trigger](../cosmos-db/nosql/how-to-use-stored-procedures-triggers-udfs.md#how-to-run-post-triggers) function that is registered in your Cosmos DB container. | No | N/A |

-| [etag](#etag-attribute) | Entity tag for the item in the container, used for [optimistic concurrency control](../cosmos-db/nosql/database-transactions-optimistic-concurrency.md#implementing-optimistic-concurrency-control-using-etag-and-http-headers). | No |

-| consistency-level | String. Sets the Cosmos DB [consistency level](../cosmos-db/consistency-levels.md) of the write request. | No | N/A |

-| indexing-directive | The [indexing policy](../cosmos-db/index-policy.md) that determines how the container's items should be indexed. | No | `default` |

-| pre-trigger | String. Identifier of a [pre-trigger](../cosmos-db/nosql/how-to-use-stored-procedures-triggers-udfs.md#how-to-run-pre-triggers) function that is registered in your Cosmos DB container. | No | N/A |

-| post-trigger | String. Identifier of a [post-trigger](../cosmos-db/nosql/how-to-use-stored-procedures-triggers-udfs.md#how-to-run-post-triggers) function that is registered in your Cosmos DB container. | No | N/A |

-| [etag](#etag-attribute) | Entity tag for the item in the container, used for [optimistic concurrency control](../cosmos-db/nosql/database-transactions-optimistic-concurrency.md#implementing-optimistic-concurrency-control-using-etag-and-http-headers). | No |

-The following examples show ways to construct Cosmos DB [parameterized queries](../cosmos-db/nosql/query/parameterized-queries.md) using policy expressions. Choose a method based on the form of your parameter input.

-This step is your opportunity to test and validate your new App Service Environment v3.

-| Linked database | Not backed up. | The following linked databases can be backed up: [SQL Database](/azure/azure-sql/database/), [Azure Database for MySQL](../mysql/index.yml), [Azure Database for PostgreSQL](../postgresql/index.yml), [MySQL in-app](https://azure.microsoft.com/blog/mysql-in-app-preview-app-service/). |

-* Backups of [TLS enabled Azure Database for MySQL](../mysql/concepts-ssl-connection-security.md) isn't supported. If a backup is configured, you get backup failures.

-* Backups of [TLS enabled Azure Database for PostgreSQL](../postgresql/concepts-ssl-connection-security.md) isn't supported. If a backup is configured, you get backup failures.

- To scale up the related resource, see the documentation for the specific resource type. For example, to scale up a single SQL Database, see [Scale single database resources in Azure SQL Database](/azure/azure-sql/database/single-database-scale). To scale up an Azure Database for MySQL resource, see [Scale MySQL resources](../mysql/concepts-pricing-tiers.md#scale-resources).

-> Azure Database for MySQL - Single Server is on the road to retirement by 16 September 2024. If your existing MySQL database is hosted on Azure Database for MySQL - Single Server, consider migrating to Azure Database for MySQL - Flexible Server using the following steps, or using [Azure Database Migration Service (DMS)](../mysql/single-server/whats-happening-to-mysql-single-server.md#migrate-from-single-server-to-flexible-server).

-> [Connect to Azure Database for PostgreSQL with Java](../postgresql/connect-java.md)

-[WordPress](https://www.wordpress.org) is an open source Content Management System (CMS) used by over 40% of the web to create websites, blogs, and other applications. WordPress can be run on a few different Azure

-In this quickstart, you'll learn how to create and deploy your first [WordPress](https://www.wordpress.org/) site to [Azure App Service on Linux](overview.md#app-service-on-linux) with [Azure Database for MySQL - Flexible Server](../mysql/flexible-server/index.yml) using the [WordPress Azure Marketplace item by App Service](https://azuremarketplace.microsoft.com/marketplace/apps/WordPress.WordPress?tab=Overview). This quickstart uses the **Standard** tier for your app and a **Burstable, B2s** tier for your database, and incurs a cost for your Azure Subscription. For pricing, visit [App Service pricing](https://azure.microsoft.com/pricing/details/app-service/linux/), [Azure Database for MySQL pricing](https://azure.microsoft.com/pricing/details/mysql/flexible-server/), [Content Delivery Network pricing](https://azure.microsoft.com/pricing/details/storage/blobs/), and [Azure Blob Storage pricing](https://azure.microsoft.com/pricing/details/storage/blobs/).

-> |`DATABASE_PASSWORD`|Database|-|-|Database password used to connect to the MySQL database. To change the MySQL database password, see [update admin password](../mysql/single-server/how-to-create-manage-server-portal.md#update-admin-password). Whenever the MySQL database password is changed, the Application Settings also need to be updated. |

-> This tutorial doesn't include guidance for [Azure Cosmos DB](../cosmos-db/index.yml), which supports Microsoft Entra authentication differently. For more information, see the Azure Cosmos DB documentation, such as [Use system-assigned managed identities to access Azure Cosmos DB data](../cosmos-db/managed-identity-based-authentication.yml).

-> For Azure Database for MySQL - Flexible Server, you must first [manually set up Microsoft Entra authentication](../mysql/flexible-server/how-to-azure-ad.md), which requires a separate user-assigned managed identity and specific Microsoft Graph permissions. This step can't be automated.

-1. Manually [set up Microsoft Entra authentication for Azure Database for MySQL - Flexible Server](../mysql/flexible-server/how-to-azure-ad.md).

-[Configure TLS connectivity in Azure Database for PostgreSQL - Single Server](../postgresql/concepts-ssl-connection-security.md)

-[Configure SSL connectivity in your application to securely connect to Azure Database for MySQL](../mysql/howto-configure-ssl.md)

-This tutorial shows how to build, configure, and deploy a secure [Quarkus](https://quarkus.io) application in Azure App Service that's connected to a PostgreSQL database (using [Azure Database for PostgreSQL](../postgresql/index.yml)). Azure App Service is a highly scalable, self-patching, web-hosting service that can easily deploy apps on Windows or Linux. When you're finished, you'll have a Quarkus app running on [Azure App Service on Linux](overview.md).

-When you are finished, you will have a [Spring Boot](https://spring.io/projects/spring-boot) application storing data in [Azure Cosmos DB](../cosmos-db/index.yml) running on [Azure App Service on Linux](overview.md).

-[Azure Cosmos DB](../cosmos-db/introduction.md) and

-[Azure App Service](overview.md) provides a highly scalable, self-patching web hosting service in Azure. It also provides a [managed identity](overview-managed-identity.md) for your app, which is a turn-key solution for securing access to [Azure Database for PostgreSQL](../postgresql/index.yml) and other Azure services. Managed identities in App Service make your app more secure by eliminating secrets from your app, such as credentials in the environment variables. In this tutorial, you learn how to:

-This tutorial shows how to build, configure, and deploy a secure Tomcat application in Azure App Service that connects to a MySQL database (using [Azure Database for MySQL](../mysql/index.yml)). Azure App Service is a highly scalable, self-patching, web-hosting service that can easily deploy apps on Windows or Linux. When you're finished, you'll have a Tomcat app running on [Azure App Service on Linux](overview.md).

-[Azure App Service](overview.md) provides a highly scalable, self-patching web hosting service using the Linux operating system. This tutorial shows how to create a secure Node.js app in Azure App Service that's connected to a [Azure Cosmos DB for MongoDB](../cosmos-db/mongodb/mongodb-introduction.md) database. When you're finished, you'll have an Express.js app running on Azure App Service on Linux.

- - **MYSQL_ATTR_SSL_CA**: Use */home/site/wwwroot/ssl/DigiCertGlobalRootCA.crt.pem* as the value. This app setting points to the path of the [TLS/SSL certificate you need to access the MySQL server](../mysql/flexible-server/how-to-connect-tls-ssl.md#download-the-public-ssl-certificate). It's included in the sample repository for convenience.

-In this tutorial, you deploy a data-driven Python web app (**[FastAPI](https://fastapi.tiangolo.com/)** ) to **[Azure App Service](./overview.md#app-service-on-linux)** with the **[Azure Database for PostgreSQL](../postgresql/index.yml)** relational database service. Azure App Service supports [Python](https://www.python.org/downloads/) in a Linux server environment.

-In this tutorial, you'll deploy a data-driven Python web app (**[Django](https://www.djangoproject.com/)** or **[Flask](https://flask.palletsprojects.com/)**) to **[Azure App Service](./overview.md#app-service-on-linux)** with the **[Azure Database for PostgreSQL](../postgresql/index.yml)** relational database service. Azure App Service supports [Python](https://www.python.org/downloads/) in a Linux server environment.

- * [Nodes and tables](../../postgresql/hyperscale/concepts-nodes.md)

- * [Determine application type](../../postgresql/hyperscale/howto-app-type.md)

- * [Choose a distribution column](../../postgresql/hyperscale/howto-choose-distribution-column.md)

- * [Table colocation](../../postgresql/hyperscale/concepts-colocation.md)

- * [Distribute and modify tables](../../postgresql/hyperscale/howto-modify-distributed-tables.md)

- * [Design a multi-tenant database](../../postgresql/hyperscale/tutorial-design-database-multi-tenant.md)*

- * [Design a real-time analytics dashboard](../../postgresql/hyperscale/tutorial-design-database-realtime.md)*

-Azure Arc resource bridge stores resource information in Azure Cosmos DB. As described in [Encryption at rest in Azure Cosmos DB](../../cosmos-db/database-encryption-at-rest.md), all the data is encrypted at rest.

-Azure Fluid Relay leverages the encryption-at-rest capability of [Azure Kubernetes Service](/azure/aks/enable-host-encryption), [Azure Cosmos DB](../../cosmos-db/database-encryption-at-rest.md) and [Azure Blob Storage](../../storage/common/storage-service-encryption.md). The service-to-service communication between Azure Fluid Relay and these resources is TLS encrypted and is enclosed in with the Azure Virtual Network boundary, protected from external interference by Network Security Rules.

-² The Cosmos DB trigger uses the [Azure Cosmos DB change feed](../cosmos-db/change-feed.md) and exposes change feed items as JSON-serializable types. The absence of SDK types is by-design for this scenario.

-This article shows you how to use Visual Studio Code to connect [Azure Cosmos DB](../cosmos-db/introduction.md) to the function you created in the previous quickstart article. The output binding that you add to this function writes data from the HTTP request to a JSON document stored in an Azure Cosmos DB container.

-Now, you create an Azure Cosmos DB account as a [serverless account type](../cosmos-db/serverless.md). This consumption-based mode makes Azure Cosmos DB a strong option for serverless workloads.

- |**Select an Azure Database Server**| Choose **Core (NoSQL)** to create a document database that you can query by using a SQL syntax or a Query Copilot ([Preview](../cosmos-db/nosql/query/how-to-enable-use-copilot.md)) converting natural language prompts to queries. [Learn more about the Azure Cosmos DB](../cosmos-db/introduction.md). |

- |**Select a capacity model**| Select **Serverless** to create an account in [serverless](../cosmos-db/serverless.md) mode.

- |**Enter the partition key for the collection**|Type `/id` as the [partition key](../cosmos-db/partitioning-overview.md).|

-> When the collection is [partitioned](../cosmos-db/partitioning-overview.md#logical-partitions), lookup operations must also specify the partition key value.

-> If you need to query by just the ID, it is recommended to use a look up, like the [previous examples](#http-trigger-look-up-id-from-query-string-c), as it will consume less [request units](../cosmos-db/request-units.md). Point read operations (GET) are [more efficient](../cosmos-db/optimize-cost-reads-writes.md) than queries by ID.

-> If you need to query by just the ID, it is recommended to use a look up, like the [previous examples](#http-trigger-look-up-id-from-query-stringpojo-parameter-java), as it will consume less [request units](../cosmos-db/request-units.md). Point read operations (GET) are [more efficient](../cosmos-db/optimize-cost-reads-writes.md) than queries by ID.

-The Azure Cosmos DB Trigger uses the [Azure Cosmos DB change feed](../cosmos-db/change-feed.md) to listen for inserts and updates across partitions. The change feed publishes new and updated items, not including updates from deletions.

-|**createLeaseContainerIfNotExists** | (Optional) When set to `true`, the leases container is automatically created when it doesn't already exist. The default value is `false`. When using Microsoft Entra identities if you set the value to `true`, creating containers isn't [an allowed operation](../cosmos-db/nosql/troubleshoot-forbidden.md#non-data-operations-are-not-allowed) and your Function won't start.|

-|**maxItemsPerInvocation**| (Optional) When set, this property sets the maximum number of items received per Function call. If operations in the monitored container are performed through stored procedures, [transaction scope](../cosmos-db/nosql/stored-procedures-triggers-udfs.md#transactions) is preserved when reading items from the change feed. As a result, the number of items received could be higher than the specified value so that the items changed by the same transaction are returned as part of one atomic batch. |

-This set of articles explains how to work with [Azure Cosmos DB](../cosmos-db/serverless-computing-database.md) bindings in Azure Functions 2.x and higher. Azure Functions supports trigger, input, and output bindings for Azure Cosmos DB.

-This version also changes the types that you can bind to, replacing the types from the v2 SDK `Microsoft.Azure.DocumentDB` with newer types from the v3 SDK [Microsoft.Azure.Cosmos](../cosmos-db/sql/sql-api-sdk-dotnet-standard.md). Learn more about how these new types are different and how to migrate to them from the [SDK migration guide](../cosmos-db/sql/migrate-dotnet-v3.md), [trigger](./functions-bindings-cosmosdb-v2-trigger.md), [input binding](./functions-bindings-cosmosdb-v2-input.md), and [output binding](./functions-bindings-cosmosdb-v2-output.md) examples.

-|**protocol**|`Https`|The connection protocol used by the function when connection to the Azure Cosmos DB service. Read [here for an explanation of both modes](../cosmos-db/performance-tips.md#networking). |

-This article explains how to work with [Azure Cosmos DB](../cosmos-db/serverless-computing-database.md) bindings in Azure Functions. Azure Functions supports trigger, input, and output bindings for Azure Cosmos DB.

-> Azure Cosmos DB bindings are only supported for use with the SQL API. For all other Azure Cosmos DB APIs, you should access the database from your function by using the static client for your API, including [Azure Cosmos DB for MongoDB](../cosmos-db/mongodb-introduction.md), [Azure Cosmos DB for Apache Cassandra](../cosmos-db/cassandra-introduction.md), [Azure Cosmos DB for Apache Gremlin](../cosmos-db/graph-introduction.md), and [Azure Cosmos DB for Table](../cosmos-db/table-introduction.md).

-The Azure Cosmos DB Trigger uses the [Azure Cosmos DB Change Feed](../cosmos-db/change-feed.md) to listen for inserts and updates across partitions. The change feed publishes inserts and updates, not deletions.

-|**collectionThroughput**|**CollectionThroughput**| When `CreateIfNotExists` is true, defines the [throughput](../cosmos-db/set-throughput.md) of the created collection.|

-* [Learn more about serverless database computing with Azure Cosmos DB](../cosmos-db/serverless-computing-database.md)

-Use the Azure Tables input binding to read a table in [Azure Cosmos DB for Table](../cosmos-db/table/introduction.md) or [Azure Table Storage](../storage/tables/table-storage-overview.md).

-For more information about how to use CloudTable, see [Get started with Azure Table storage](../cosmos-db/tutorial-develop-table-dotnet.md).

-Use an Azure Tables output binding to write entities to a table in [Azure Cosmos DB for Table](../cosmos-db/table/introduction.md) or [Azure Table Storage](../storage/tables/table-storage-overview.md).

-Azure Functions integrates with [Azure Tables](../cosmos-db/table/introduction.md) via [triggers and bindings](./functions-triggers-bindings.md). Integrating with Azure Tables allows you to build functions that read and write data using [Azure Cosmos DB for Table](../cosmos-db/table/introduction.md) and [Azure Table Storage](../storage/tables/table-storage-overview.md).

-Learn how to create a function in the Azure portal that is triggered when data is added to or changed in Azure Cosmos DB. To learn more about Azure Cosmos DB, see [Azure Cosmos DB: Serverless database computing using Azure Functions](../cosmos-db/serverless-computing-database.md).

- | **[Partition key](../cosmos-db/partitioning-overview.md)** | /category|A partition key that distributes data evenly to each partition. Selecting the correct partition key is important in creating a performant container. |

-|Protocol|Https|The connection protocol used by the function when connection to the Azure Cosmos DB service. Read [here for an explanation of both modes](../cosmos-db/performance-tips.md#networking)|

-| [Respond to database changes](./functions-scenarios.md#respond-to-database-changes)| Run custom logic when a document is created or updated in [Azure Cosmos DB](../cosmos-db/introduction.md). |

-|China East 2| 100 | 20 |

-|China North 2| 100 | 20 |

-|East US 2| 100 | 100 |

-|USGov Arizona| 100 | 20 |

-|USGov Texas| 100 | Not Available |

-|USGov Virginia| 100 | 20 |

-Azure Functions integrates well with [Azure Cosmos DB](../cosmos-db/introduction.md) for many [use cases](../cosmos-db/use-cases.md), including IoT, ecommerce, gaming, etc.

-For example, for [event sourcing](/azure/architecture/patterns/event-sourcing), the two services are integrated to power event-driven architectures using Azure Cosmos DB's [change feed](../cosmos-db/change-feed.md) functionality. The change feed provides downstream microservices the ability to reliably and incrementally read inserts and updates (for example, order events). This functionality can be used to provide a persistent event store as a message broker for state-changing events and drive order processing workflow between many microservices (which can be implemented as [serverless Azure Functions](https://azure.com/serverless)).

-To connect to Azure Cosmos DB, first [create an account, database, and container](../cosmos-db/nosql/quickstart-portal.md). Then you can connect your function code to Azure Cosmos DB using [trigger and bindings](functions-bindings-cosmosdb-v2.md), like this [example](functions-add-output-binding-cosmos-db-vs-code.md).

-The following tutorials use an Event Grid trigger to process files in a blob container:

-So much telemetry is generated and collected from cloud applications, IoT devices, and networking devices. Azure Functions can process that data in near real-time as the hot path, then store it in [Azure Cosmos DB](../cosmos-db/introduction.md) for use in an analytics dashboard.

-> Since Azure Cosmos DB is a partitioned workload, the target instance count for the database is capped by the number of physical partitions in your container. To learn more about Azure Cosmos DB scaling, see [physical partitions](../cosmos-db/nosql/change-feed-processor.md#dynamic-scaling) and [lease ownership](../cosmos-db/nosql/change-feed-processor.md#dynamic-scaling).

-[CosmosClient](/dotnet/api/microsoft.azure.cosmos.cosmosclient) connects to an Azure Cosmos DB instance. The Azure Cosmos DB documentation recommends that you [use a singleton Azure Cosmos DB client for the lifetime of your application](../cosmos-db/performance-tips-dotnet-sdk-v3-sql.md#sdk-usage). The following example shows one pattern for doing that in a function:

-[CosmosClient](/javascript/api/@azure/cosmos/cosmosclient) connects to an Azure Cosmos DB instance. The Azure Cosmos DB documentation recommends that you [use a singleton Azure Cosmos DB client for the lifetime of your application](../cosmos-db/performance-tips.md#sdk-usage). The following example shows one pattern for doing that in a function:

-The underlying SDK used by extension changed to use the Azure Cosmos DB V3 SDK, for cases where you were using SDK related types, please look at the [Azure Cosmos DB SDK V3 migration guide](../cosmos-db/nosql/migrate-dotnet-v3.md) for more information.

-|**CreateLeaseCollectionIfNotExists** |**CreateLeaseContainerIfNotExists** | (Optional) When set to `true`, the leases container is automatically created when it doesn't already exist. The default value is `false`. When using Microsoft Entra identities if you set the value to `true`, creating containers isn't [an allowed operation](../cosmos-db/nosql/troubleshoot-forbidden.md#non-data-operations-are-not-allowed) and your Function won't be able to start.|

-|**UseDefaultJsonSerialization** |*Removed* | This attribute is no longer needed as you can fully customize the serialization using built in support in the [Azure Cosmos DB version 3 .NET SDK](../cosmos-db/nosql/migrate-dotnet-v3.md#customize-serialization). |

-|**createLeaseCollectionIfNotExists** |**createLeaseContainerIfNotExists** | (Optional) When set to `true`, the leases container is automatically created when it doesn't already exist. The default value is `false`. When using Microsoft Entra identities if you set the value to `true`, creating containers isn't [an allowed operation](../cosmos-db/nosql/troubleshoot-forbidden.md#non-data-operations-are-not-allowed) and your Function won't be able to start.|

-The Azure Functions extension version 4 is built on top of the Azure Cosmos DB .NET SDK version 3, which removed support for the [`Document` class](../cosmos-db/nosql/migrate-dotnet-v3.md#major-name-changes-from-v2-sdk-to-v3-sdk). Instead of receiving a list of `Document` objects with each function invocation, which you must then deserialize into your own object type, you can now directly receive a list of objects of your own type.

-### [Azure Database for MySQL](../mysql/index.yml)

-### [Azure Database for PostgreSQL](../postgresql/index.yml)

-For Flexible Server availability in Azure Government regions, see [Azure Database for PostgreSQL ΓÇô Flexible Server](../postgresql/flexible-server/overview.md#azure-regions).

-| [Azure Cosmos DB](../../cosmos-db/index.yml) | &#x2705; | &#x2705; |

-| [Azure Database for MariaDB](../../mariadb/index.yml) | &#x2705; | &#x2705; |

-| [Azure Database for MySQL](../../mysql/index.yml) | &#x2705; | &#x2705; |

-| [Azure Database for PostgreSQL](../../postgresql/index.yml) | &#x2705; | &#x2705; |

-| [Database Migration Service](../../dms/index.yml) | &#x2705; | &#x2705; |

-| [Azure Cosmos DB](../../cosmos-db/index.yml) | &#x2705; | &#x2705; | &#x2705; | &#x2705; | &#x2705; |

-| [Azure Database for MariaDB](../../mariadb/index.yml) | &#x2705; | &#x2705; | &#x2705; | &#x2705; | |

-| [Azure Database for MySQL](../../mysql/index.yml) | &#x2705; | &#x2705; | &#x2705; | &#x2705; | |

-| [Azure Database for PostgreSQL](../../postgresql/index.yml) | &#x2705; | &#x2705; | &#x2705; | &#x2705; | |

-| [Database Migration Service](../../dms/index.yml) | &#x2705; | &#x2705; | &#x2705; | &#x2705; | |

-### [Azure Cosmos DB](../cosmos-db/index.yml)

-### [Azure Database for MySQL](../mysql/index.yml)

-### [Azure Database for PostgreSQL](../postgresql/index.yml)

-This article uses the [Postman](https://www.postman.com/) application to create the token request, but you can use a different API development environment.

-1. In the Postman app, select **New**.

-2. In the **Create New** window, select **HTTP Request**.

-3. Enter a **Request name** for the request, such as *POST Token Request*.

-4. Select the **POST** HTTP method.

-5. Enter the following URL to address bar (replace `{Tenant-ID}` with the Directory (Tenant) ID, the `{Client-ID}` with the Application (Client) ID, and `{Client-Secret}` with your client secret:

-6. Select **Send**

-7. You should see the following JSON response:

- "token_type": "Bearer",

- "expires_in": 86399,

- "ext_expires_in": 86399,

- "access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im5PbzNaRHJPRFhFSzFq..."

-You can run requests to Azure Maps APIs from most clients, like C#, Java, or JavaScript. [Postman](https://learning.postman.com/docs/sending-requests/generate-code-snippets) converts an API request into a basic client code snippet in almost any programming language or framework you choose. You can use this generated code snippet in your front-end applications.

-This article uses the [Postman] application to build REST calls, but you can choose any API development environment.

-* Cluster data based on travel time or distances. For example, your company has 50 employees, find all employees that live within 20 minute Drive Time from your office.

-This article uses the [Postman] application to build REST calls, but you can choose any API development environment.

-### Encode a URI to handle special characters

-[Search service]: /rest/api/maps/search?view=rest-maps-1.0&preserve-view=true

-[Search Fuzzy]: /rest/api/maps/search/getsearchfuzzy?view=rest-maps-1.0&preserve-view=true

-[subscription key]: quick-demo-map-app.md#get-the-subscription-key-for-your-account

-[Postman]: https://www.postman.com/downloads/

-[Search Address Reverse]: /rest/api/maps/search/getsearchaddressreverse?view=rest-maps-1.0&preserve-view=true

-[POI category search]: /rest/api/maps/search/getsearchpoicategory?view=rest-maps-1.0&preserve-view=true

-[Search Nearby]: /rest/api/maps/search/getsearchnearby?view=rest-maps-1.0&preserve-view=true

-[Azure Maps supported languages]: supported-languages.md

-[Search POIs inside the geometry]: /rest/api/maps/search/postsearchinsidegeometry?view=rest-maps-1.0&preserve-view=true

-> You can download the entire [rentalCarSimulation] C# project from GitHub as a single ZIP file by going to [the root of the sample] and selecting the green **<> Code** button, then **Download ZIP**.

-This tutorial uses the [Postman] application, but you can choose a different API development environment.

->[!IMPORTANT]

- >[!IMPORTANT]

->[!TIP]

-[Postman]: https://www.postman.com/

-### W3CIISLog Table for Internet Information Services (IIS)

-|LAW Field | Difference | Reason| Additional Information |

-|||||

-| Fileuri | No longer populated | not required for MMA parity | MMA doesn't collect this field. This field was only populated for IIS logs collected from Azure Cloud Services through the Azure Diagnostics Extension.|

-|LAW Field | Difference | Reason| Additional Information |

-|||||

-| UserName | MMA enriches the event with the username prior to sending the event for ingestion. AMA do not do the same enrichment. | The AMA enrichment is not yet implemented. | AMA principles dictate that the event data should remain unchanged by default. Adding and enriched field adds possible processing errors and additional cost for storage. In this case, the customer demand for the field is very high and work is underway to add the username. |

-## Known parity gaps that may impact your migration

-**Performance counters** is one of the data sources used in a [data collection rule (DCR)](../essentials/data-collection-rule-create-edit.md). Details for the creation of the DCR are provided in [Collect data with Azure Monitor Agent](./azure-monitor-agent-data-collection.md). This article provides additional details for the Windows events data source type.

-## Configure performance counters data source

-

-| Destination | Table / Namespace |

-|:|:|

-| Log Analytics workspace | [Perf](/azure/azure-monitor/reference/tables/perf) |

-| Azure Monitor Metrics | Windows: Virtual Machine Guest<br>Linux: azure.vm.linux.guestmetrics

-|[Azure Cosmos DB](https://www.nuget.org/packages/Microsoft.Azure.Cosmos) | Tracked automatically if HTTP/HTTPS is used. Tracing for operations in direct mode with TCP will also be captured automatically using preview package >= [3.33.0-preview](https://www.nuget.org/packages/Microsoft.Azure.Cosmos/3.33.0-preview). For more details visit the [documentation](../../cosmos-db/nosql/sdk-observability.md). |

-* Azure Cosmos DB is tracked automatically only if [HTTP/HTTPS](../../cosmos-db/performance-tips.md#networking) is used. TCP mode won't be automatically captured by Application Insights for SDK versions older than [`2.22.0-Beta1`](https://github.com/microsoft/ApplicationInsights-dotnet/blob/main/CHANGELOG.md#version-2220-beta1).

-| where InputStream == "Custom-MyTable_CL"

-| Azure Cosmos DB | [Azure Cosmos DB logging](../../cosmos-db/monitor-cosmos-db.md) |

-| Azure Database for MySQL | [Azure Database for MySQL diagnostic logs](../../mysql/concepts-server-logs.md#diagnostic-logs) |

-| Azure Database for PostgreSQL | [Azure Database for PostgreSQL logs](../../postgresql/concepts-server-logs.md#resource-logs) |

-| [Azure Cosmos DB Insights](../../cosmos-db/cosmosdb-insights-overview.md) | GA | [Yes](https://portal.azure.com/#blade/Microsoft_Azure_Monitoring/AzureMonitoringBrowseBlade/cosmosDBInsights) | Provides a view of the overall performance, failures, capacity, and operational health of all your Azure Cosmos DB resources in a unified interactive experience. |

-### Querying for data volumes excluding known free data types

-The following query will return the monthly data volume in GB, excluding all data types which are supposed to be free from data ingestion charges:

-```kusto

-let freeTables = dynamic([

-"AppAvailabilityResults","AppSystemEvents","ApplicationInsights","AzureActivity","AzureNetworkAnalyticsIPDetails_CL",

-"AzureNetworkAnalytics_CL","AzureTrafficAnalyticsInsights_CL","ComputerGroup","DefenderIoTRawEvent","Heartbeat",

-"MAApplication","MAApplicationHealth","MAApplicationHealthIssues","MAApplicationInstance","MAApplicationInstanceReadiness",

-"MAApplicationReadiness","MADeploymentPlan","MADevice","MADeviceNotEnrolled","MADeviceReadiness","MADriverInstanceReadiness",

-"MADriverReadiness","MAProposedPilotDevices","MAWindowsBuildInfo","MAWindowsCurrencyAssessment",

-"MAWindowsCurrencyAssessmentDailyCounts","MAWindowsDeploymentStatus","NTAIPDetails_CL","NTANetAnalytics_CL",

-"OfficeActivity","Operation","SecurityAlert","SecurityIncident","UCClient","UCClientReadinessStatus",

-"UCClientUpdateStatus","UCDOAggregatedStatus","UCDOStatus","UCDeviceAlert","UCServiceUpdateStatus","UCUpdateAlert",

-"Usage","WUDOAggregatedStatus","WUDOStatus","WaaSDeploymentStatus","WaaSInsiderStatus","WaaSUpdateStatus"]);

-Usage

-| where DataType !in (freeTables)

-| where TimeGenerated > ago(30d)

-| summarize MonthlyGB=sum(Quantity)/1000

-```

-To look for data which might not have IsBillable correctly set (and which could result in incorrect billing, or more specifically under-billing), use this query on your workspace:

-```kusto

-let freeTables = dynamic([

-"AppAvailabilityResults","AppSystemEvents","ApplicationInsights","AzureActivity","AzureNetworkAnalyticsIPDetails_CL",

-"AzureNetworkAnalytics_CL","AzureTrafficAnalyticsInsights_CL","ComputerGroup","DefenderIoTRawEvent","Heartbeat",

-"MAApplication","MAApplicationHealth","MAApplicationHealthIssues","MAApplicationInstance","MAApplicationInstanceReadiness",

-"MAApplicationReadiness","MADeploymentPlan","MADevice","MADeviceNotEnrolled","MADeviceReadiness","MADriverInstanceReadiness",

-"MADriverReadiness","MAProposedPilotDevices","MAWindowsBuildInfo","MAWindowsCurrencyAssessment",

-"MAWindowsCurrencyAssessmentDailyCounts","MAWindowsDeploymentStatus","NTAIPDetails_CL","NTANetAnalytics_CL",

-"OfficeActivity","Operation","SecurityAlert","SecurityIncident","UCClient","UCClientReadinessStatus",

-"UCClientUpdateStatus","UCDOAggregatedStatus","UCDOStatus","UCDeviceAlert","UCServiceUpdateStatus","UCUpdateAlert",

-"Usage","WUDOAggregatedStatus","WUDOStatus","WaaSDeploymentStatus","WaaSInsiderStatus","WaaSUpdateStatus"]);

-Usage

-| where DataType !in (freeTables)

-| where TimeGenerated > ago(30d)

-| where IsBillable == false

-| summarize MonthlyPotentialUnderbilledGB=sum(Quantity)/1000 by DataType

-```

-After an exception has occurred in your application and a snapshot has been created, you should have snapshots to view in the Azure portal within 5 to 10 minutes. To view snapshots, in the **Failure** pane, either:

-Select an operation or exception in the right pane to open the **End-to-End Transaction Details** pane, then select the exception event. If a snapshot is available for the given exception, an **Open Debug Snapshot** button appears on the right pane with details for the [exception](../app/asp-net-exceptions.md).

-Snapshots might include sensitive information. By default, you can only view snapshots if you've been assigned the `Application Insights Snapshot Debugger` role.

-## View Snapshots in Visual Studio 2017 Enterprise or above

-1. To open the `.diagsession` file, you need to have the Snapshot Debugger Visual Studio component installed. The Snapshot Debugger component is a required component of the ASP.NET workload in Visual Studio and can be selected from the Individual Component list in the Visual Studio installer. If you're using a version of Visual Studio before Visual Studio 2017 version 15.5, you'll need to install the extension from the [Visual Studio Marketplace](https://aka.ms/snapshotdebugger).

-* You must create a proximity placement group (PPG) and anchor it to your SAP HANA compute resources. Application volume group for SAP HANA needs this setup to search for an Azure NetApp Files resource that is close to the SAP HANA servers. For more information, see [Best practices about Proximity Placement Groups](#best-practices-about-proximity-placement-groups) and [Create a Proximity Placement Group using the Azure portal](../virtual-machines/windows/proximity-placement-groups-portal.md).

-## Best practices about proximity placement groups

- * You anchor the SAP HANA VMs using a PPG to guaranty lowest possible latency. The PPG enforces the creation of data, log, and shared volumes in the close proximity to the SAP HANA VMs. See [Best practices about Proximity Placement Groups](application-volume-group-considerations.md#best-practices-about-proximity-placement-groups) for details.

-Azure NetApp Files volumes are designed to be contained in a special purpose subnet called a [delegated subnet](../virtual-network/virtual-network-manage-subnet.md) within your Azure Virtual Network. Therefore, you can access the volumes directly from within Azure over VNet peering or from on-premises over a Virtual Network Gateway (ExpressRoute or VPN Gateway). The subnet is dedicated to Azure NetApp Files and there's no connectivity to the Internet.

-| Number of IPs in a VNet (including immediately peered VNets) accessing volumes in an Azure NetApp Files hosting VNet | [Same standard limits as VMs](../azure-resource-manager/management/azure-subscription-service-limits.md#azure-resource-manager-virtual-networking-limits) | 1000 |

-### UDRs and NSGs

-> [!div class="nextstepaction"]

-> [Storage hierarchy of Azure NetApp Files](azure-netapp-files-understand-storage-hierarchy.md)

-> [!div class="nextstepaction"]

-> [Service levels for Azure NetApp Files](azure-netapp-files-service-levels.md)

-> [!div class="nextstepaction"]

-> [Create an NFS volume](azure-netapp-files-create-volumes.md)

-> [!div class="nextstepaction"]

-> [Solution architectures using Azure NetApp Files](azure-netapp-files-solution-architectures.md)

-> [!div class="nextstepaction"]

-> [Application resilience FAQs for Azure NetApp Files](faq-application-resilience.md)

-| Number of IPs in a virtual network (including immediately peered VNets) accessing volumes in an Azure NetApp Files hosting VNet | <ul><li>**Basic**: 1000</li><li>**Standard**: [Same standard limits as VMs](../azure-resource-manager/management/azure-subscription-service-limits.md#azure-resource-manager-virtual-networking-limits)</li></ul> | No |

-SMB Multichannel is enabled by default in SMB shares. All SMB shares pre-dating existing SMB volumes have had the feature enabled, and all newly created volumes will also have the feature enabled at time of creation.

-Any SMB connection established before the feature enablement will need to be reset to take advantage of the SMB Multichannel functionality. To reset, you can disconnect and reconnect the SMB share.

-Windows has supported SMB Multichannel since Windows 2012 to enable best performance. See [Deploy SMB Multichannel](/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn610980(v%3Dws.11)) and [The basics of SMB Multichannel](/archive/blogs/josebda/the-basics-of-smb-multichannel-a-feature-of-windows-server-2012-and-smb-3-0) for details.

-With SMB Multichannel disabled on the client, pure 4 KiB read and write tests were performed using FIO and a 40 GiB working set. The SMB share was detached between each test, with increments of the SMB client connection count per RSS network interface settings of `1`,`4`,`8`,`16`, `set-SmbClientConfiguration -ConnectionCountPerRSSNetworkInterface <count>`. The tests show that the default setting of `4` is sufficient for I/O intensive workloads; incrementing to `8` and `16` had negligible effect.

-The command `netstat -na | findstr 445` proved that additional connections were established with increments from `1` to `4` to `8` and to `16`. Four CPU cores were fully utilized for SMB during each test, as confirmed by the perfmon `Per Processor Network Activity Cycles` statistic (not included in this article.)

-The Azure virtual machine does not affect SMB (nor NFS) storage I/O limits. As shown in the following chart, the D32ds instance type has a limited rate of 308,000 for cached storage IOPS and 51,200 for uncached storage IOPS. However, the graph above shows significantly more I/O over SMB.

-#### Sequential IO

-Tests similar to the random I/O tests described previously were performed with 64-KiB sequential I/O. Although the increases in client connection count per RSS network interface beyond 4ΓÇÖ had no noticeable effect on random I/O, the same does not apply to sequential I/O. As the following graph shows, each increase is associated with a corresponding increase in read throughput. Write throughput remained flat due to network bandwidth restrictions placed by Azure for each instance type/size.

-Azure places network rate limits on each virtual machine type/size. The rate limit is imposed on outbound traffic only. The number of NICs present on a virtual machine has no bearing on the total amount of bandwidth available to the machine. For example, the D32ds instance type has an imposed network limit of 16,000 Mbps (2,000 MiB/s). As the sequential graph above shows, the limit affects the outbound traffic (writes) but not multichannel reads.

-SMB Signing has a deleterious effect upon SMB performance. Among other potential causes of the performance degradation, the digital signing of each packet consumes additional client-side CPU as the perfmon output below shows. In this case, Core 0 appears responsible for SMB, including SMB Signing. A comparison with the non-multichannel sequential read throughput numbers in the previous section shows that SMB Signing reduces overall throughput from 875MiB/s to approximately 250MiB/s.

-To provide more detailed insight into workloads with read/write mixes, the following two charts show the performance of a single, Ultra service-level cloud volume of 50 TB with a 1-TB dataset and with SMB multichannel of 4. An optimal IODepth of 16 was used, and Flexible IO (FIO) parameters were used to ensure the full use of the network bandwidth (`numjobs=16`).

-After you have data traffic running in your volumes, you can monitor your adapters in Windows Performance Monitor. If you do not use all of these 16 virtual adapters, you might not be maximizing your network bandwidth capacity.

-SMB encryption is not required. As such, it is only enabled for a given share if the user requests that Azure NetApp Files enable it. Azure NetApp Files shares are never exposed to the internet. They are only accessible from within a given VNet, over VPN or express route, so Azure NetApp Files shares are inherently secure. The choice to enable SMB encryption is entirely up to the user. Be aware of the anticipated performance penalty before enabling this feature.

-For maximum performance, it is recommended that you configure [Accelerated Networking](../virtual-network/create-vm-accelerated-networking-powershell.md) on your virtual machines where possible. Keep the following considerations in mind:

-* The Azure portal enables Accelerated Networking by default for virtual machines supporting this feature. However, other deployment methods such as Ansible and similar configuration tools may not. Failure to enable Accelerated Networking can hobble the performance of a machine.

-* If Accelerated Networking is not enabled on the network interface of a virtual machine due to its lack of support for an instance type or size, it will remain disabled with larger instance types. You will need manual intervention in those cases.

-* There is no need to set accelerated networking for the NICs in the dedicated subnet of Azure NetApp Files. Accelerated networking is a capability that only applies to Azure virtual machines. Azure NetApp Files NICs are optimized by design.

-## RSS

-Azure NetApp Files supports receive-side-scaling (RSS).

-To see if your Azure virtual machine NICs support RSS, run the command

-

-You should not configure multiple NICs on your client for SMB. The SMB client will match the NIC count returned by the SMB server. Each storage volume is accessible from one and only one storage endpoint. That means that only one NIC will be used for any given SMB relationship.

-As the output of `Get-SmbClientNetworkInterace` below shows, the virtual machine has 2 network interfaces--15 and 12. As shown under the following command `Get-SmbMultichannelConnection`, even though there are two RSS-capable NICS, only interface 12 is used in connection with the SMB share; interface 15 is not in use.

-

- Backups on a destination volume are only supported for manually created snapshots. To take backups of a destination volume, create a snapshot on the source volume then wait for the snapshot to be replicated to the destination volume. From the destination volume, you can select the snapshot for backup, where you can select this snapshot for backup. Scheduled backups on a destination volume aren't supported.

-1. **Create AvSet and proximity placement group (PPG):** For production landscapes, you should create an AvSet that is manually pinned to a data center where Azure NetApp Files resources are available in proximity. The AvSet pinning ensures that VMs won't be moved on restart. The proximity placement group (PPG) needs to be assigned to the AvSet. With the help of application volume groups, the PPG can find the closest Azure NetApp Files hardware. For more information, see [Best practices about proximity placement groups](application-volume-group-considerations.md#best-practices-about-proximity-placement-groups).

-File and path length refers to the number of Unicode characters in a file path, including directories. This limit is a factor in the individual character lengths, which are determined by the size of the character in bytes. For instance, NFS and SMB allow path components of 255 bytes. The file encoding format of ASCII uses 8-bit encoding, meaning file path components (such as a file or folder name) in ASCII can be up to 255 characters since ASCII characters are 1 byte in size.

-When using Azure NetApp Files for dual-protocol access, the difference in how path lengths are handled in NFS and SMB protocols can create incompatibilities across file and folders. For instance, Windows SMB supports up to 32,767 characters in a path (provided the long path feature is enabled on the SMB client), but NFS support can exceed that amount. As such, if a path length is created in NFS that exceeds the support of SMB, clients are unable to access the data once the path length maximums have been reached. In those cases, either take care to consider the lower end limits of file path lengths across protocols when creating file and folder names (and folder path depth) or map SMB shares closer to the desired folder path to reduce the path length.

-In an Azure NetApp Files file sharing environment, file and folder names are represented by a series of characters that end users read and interpret. The way those characters are displayed depends on how the client sends and receives encoding of those characters. For instance, if a client is sending legacy [ASCII (American Standard Code for Information Interchange)](https://www.ascii-code.com/) encoding to the Azure NetApp Files volume when accessing it, then it's limited to displaying only characters that are supported in the ASCII format.

-For instance, Plane 1 (the [Supplementary Multilingual Planes (SMP)](https://unicodeplus.com/plane/1)) includes historic scripts like cuneiform and Egyptian hieroglyphs as well as some [Osage](https://en.wikipedia.org/wiki/Osage_script), [Warang Citi](https://en.wikipedia.org/wiki/Warang_Citi), [Adlam](https://en.wikipedia.org/wiki/Adlam_script), [Wancho](https://en.wikipedia.org/wiki/Wancho_language#Orthography) and [Toto](https://en.wikipedia.org/wiki/Toto_language#Writing_system). Plane 1 also includes some symbols and [emoticon](https://en.wikipedia.org/wiki/Emoticons_(Unicode_block)) characters.

-Remote connections to clients that mount Azure NetApp Files volumes (such as SSH connections to Linux clients to access NFS mounts) can be configured to send and receive specific volume language encodings. The language encoding sent to the client via the remote connection utility controls how character sets are created and viewed. As a result, a remote connection that uses a different language encoding than another remote connection (such as two different PuTTY windows) may show different results for characters when listing file and folder names in the Azure NetApp Files volume. In most cases, this won't create discrepancies (such as for Latin/English characters), but in the cases of special characters, such as emojis, results can vary.

-When using either CMD or [PowerShell](/powershell/scripting/dev-cross-plat/vscode/understanding-file-encoding), the character set view may depend on the font settings. These utilities have limited font choices by default. CMD uses Consolas as the default font.

-How NFS displays special characters depends on the version of NFS used, the client's locale settings, installed fonts, and the settings of the remote connection client in use. For instance, using Bastion to access an Ubuntu client may handle character displays differently than a PuTTY client set to a different locale on the same VM. The ensuing NFS examples rely on these locale settings for the Ubuntu VM:

-NFSv3 doesn't enforce UTF encoding on files and folders. In most cases, special character sets should have no issues. However, the connection client being used can affect how characters are sent and received. For instance, using Unicode characters outside of the BMP for a folder name in the Azure connection client Bastion can result in some unexpected behavior due to how the client encoding works.

-In some cases, a command may be allowed using a character outside of the [Basic Multilingual Plane (BMP)](https://en.wikipedia.org/wiki/Plane_%28Unicode%29#Basic_Multilingual_Plane), but it might not display the value after it's created.

-When a file or folder is created over an SMB share in Azure NetApp Files, the character set in use encode as UTF-16. As a result, clients using UTF-8 encoding (such as Linux-based NFS clients) may not be able to translate some character sets properly ΓÇô particularly characters that fall outside of the [Basic Multilingual Plane (BMP)](https://en.wikipedia.org/wiki/Plane_%28Unicode%29#Basic_Multilingual_Plane).

-| Microsoft.DBforMariaDB | [Azure Database for MariaDB](../../mariadb/index.yml) |

-| Microsoft.DBforMySQL | [Azure Database for MySQL](../../mysql/index.yml) |

-| Microsoft.DBforPostgreSQL | [Azure Database for PostgreSQL](../../postgresql/index.yml) |

-| Microsoft.DocumentDB | [Azure Cosmos DB](../../cosmos-db/index.yml) |

-| Microsoft.DataMigration | [Azure Database Migration Service](../../dms/index.yml) |

-For Azure Cosmos DB limits, see [Limits in Azure Cosmos DB](../../cosmos-db/concepts-limits.md).

-For Azure Database for MySQL limits, see [Limitations in Azure Database for MySQL](../../mysql/concepts-limits.md).

-For Azure Database for PostgreSQL limits, see [Limitations in Azure Database for PostgreSQL](../../postgresql/concepts-limits.md).

-> | servers | **Yes** | **Yes** | You can use a cross-region read replica to move an existing server. [Learn more](../../postgresql/howto-move-regions-portal.md).<br/><br/> If the service is provisioned with geo-redundant backup storage, you can use geo-restore to restore in other regions. [Learn more](../../mariadb/concepts-business-continuity.md#recovery-from-an-azure-regional-datacenter-outage).

-> | servers | **Yes** | **Yes** | You can use a cross-region read replica to move an existing server. [Learn more](../../mysql/howto-move-regions-portal.md).

-> | servers | **Yes** | **Yes** | You can use a cross-region read replica to move an existing server. [Learn more](../../postgresql/howto-move-regions-portal.md).

- > * Azure Domain Name System (DNS) zones don't support the use of spaces in the tag or a tag that starts with a number. Azure DNS tag names don't support special and unicode characters. The value can contain all characters.

-> [!div class="nextstepaction"]

-A client, be it a browser 💻, a mobile app 📱, or an IoT device 💡, uses a **Client Access URL** to connect and authenticate with your resource. This URL follows a pattern of `wss://<service_name>.webpubsub.azure.com/client/hubs/<hub_name>?access_token=<token>`. This article shows you several ways to get the Client Access URL.

- - Configure user ID

-## Invoke the Generate Client Token REST API

-

- 1. For the URI, enter `https://{Endpoint}/api/hubs/{hub}/:generateToken?api-version=2022-11-01`

-5. The Client Access URI is in the format of `wss://<endpoint>/client/hubs/<hub_name>?access_token=<token>`

-Azure Web PubSub Service makes it easy to build web applications where server and clients need to exchange data in real-time. Real-time data exchange is the bedrock of certain time-sensitive apps developers build and maintain. Developers have used the service in a variety of applications and industries, for example, in chat apps, real-time dashboards, multi-player games, online auctions, multi-user collaborative apps, location tracking, notifications, and more.

-When an app's usage is small, developers typically opt for a polling mechanism to provide real-time communication between server and clients - clients send repeated HTTP requests to server over a time interval. However, developers often report that while polling mechanism is straightforward to implement, it suffers three important drawbacks.

-With the recent surge in interest in AI, Web PubSub has become an invaluable tool to developers building AI-enabled applications for token streaming. The service is battle-tested to scale to tens of millions of concurrent connections and offers ultra-low latency.

-Any app scenario where updates at the data resource need to be delivered to other components across network can benefit from using Azure Web PubSub. As the name suggests, the service facilities the communication between a publisher and subscribers. A publisher is a component that publishes data updates. A subscriber is a component that subscribes to data updates.

-Azure Web PubSub service is used in a multitude of industries and app scenarios where data is time-sensitive. Here's a partial list of some common use cases.

-|Location tracking | Vehicle asset tracking, delivery status tracking, transportation status updates, ride-hailing apps |

-|Cross-platform push notifications | Social media, email, game status, travel alert |

-|Automation | Real-time trigger from upstream events |

-|Client pub/sub | A client sends messages to clients that are in a group arbitrarily defined by you without your server's involvement.|

-|Clients to server | Clients send messages to server at low latency. |

-You can use this solution independently or in addition to the [native backup solution offered by Azure PostgreSQL](../postgresql/concepts-backup.md) that offers retention up to 35 days. The native solution is suited for operational recoveries, such as when you want to recover from the latest backups. The Azure Backup solution helps you with your compliance needs and more granular and flexible backup/restore.

-|Deployments | [Azure Database for PostgreSQL - Single Server](../postgresql/overview.md#azure-database-for-postgresqlsingle-server) |

-This article explains how to back up [Azure PostgreSQL database](../postgresql/overview.md#azure-database-for-postgresqlsingle-server) using Azure CLI.

-This article explains how to back up [Azure PostgreSQL database](../postgresql/overview.md#azure-database-for-postgresqlsingle-server) using Azure PowerShell.

-Azure Database for PostgreSQL is a managed service that you use to run, manage, and scale highly available PostgreSQL databases in the cloud. This quickstart shows you how to back up Azure Database for PostgreSQL server running on an Azure VM to an Azure Backup Recovery Services vault. To create Azure Database for PostgreSQL server, see the [tutorial](../postgresql/tutorial-design-database-using-azure-portal.md).

-This article explains how to restore [Azure PostgreSQL databases](../postgresql/overview.md#azure-database-for-postgresqlsingle-server) to an Azure PostgreSQL server backed-up by Azure Backup.

-This article explains how to restore [Azure PostgreSQL databases](../postgresql/overview.md#azure-database-for-postgresqlsingle-server) to an Azure PostgreSQL server backed-up by Azure Backup.

-This article explains how to restore [Azure PostgreSQL databases](../postgresql/overview.md#azure-database-for-postgresqlsingle-server) to an Azure PostgreSQL server backed-up by Azure Backup.

-This section applies to the `Microsoft.DocumentDB/databaseAccounts` resource type. [Learn more about Cosmos DB](../cosmos-db/introduction.md).

-| Description | Causes an Azure Cosmos DB account with a single write region to fail over to a specified read region to simulate a [write region outage](../cosmos-db/high-availability.md). |

-A [storage service](../storage/common/storage-account-create.md) gives you access to Azure [blobs](../storage/blobs/storage-quickstart-blobs-python.md), [tables](../cosmos-db/table-storage-how-to-use-python.md), and [queues](/azure/storage/queues/storage-quickstart-queues-python?tabs=passwordless%2Croles-azure-portal%2Cenvironment-variable-windows%2Csign-in-azure-cli). To create a storage service, you need a name for the service (between 3 and 24 lowercase characters and unique within Azure). You also need a description, a label (up to 100 characters, automatically encoded to base64), and a location. The following example shows how to create a storage service by specifying a location:

-[Table Service]:../cosmos-db/table-storage-how-to-use-python.md

-is inactive for 20 minutes. Cloud Shell uses an Azure fileshare mounted to the `clouddrive` folder

-in your session. The fileshare contains the image file that contains your `$HOME` directory. Only

-files that you upload or create in the `clouddrive` folder are persisted across sessions. Any files

-created outside your `clouddrive` directory aren't persisted.

-Files stored in the `clouddrive` directory are visible in the Azure portal using Storage browser.

-However, any files created in the `$HOME` directory are stored in the image file and aren't visible

-in the portal.

-PowerShell users can use the `Azure:` drive to access Azure resources. The `Azure:` drive is created

-by a PowerShell provider that structures data as a file system drive. The `Azure:` drive is a

-virtual drive that doesn't allow you to create files.

-location is the `Azure:` drive, are saved to your `$HOME` directory.

-*Email status update operational logs* provide in-depth insights into message and recipient level delivery status updates on your sendmail API requests. These logs offer message-specific details, such as the time of delivery, as well as recipient-level details, such as email addresses and delivery status updates. By tracking these logs, you can ensure full visibility into your email delivery process, quickly identifying any issues that may arise and taking corrective action as necessary.

-| `RecipientId` | The email address for the targeted recipient. If this is a message-level event, the property will be empty. |

-| `DeliveryStatus` | The terminal status of the message. |

-```

-localization_priority: Normal

-For more information, see [Troubleshooting in Azure Communication Services](../troubleshooting-info.md).

-In this article, you learn how to use Azure Communication Services Calling SDK to retrieve Microsoft Teams Meeting audio conferencing details. This functionality allows users who are already connected to a Microsoft Teams Meeting to be able to get the conference ID and dial in phone number associated with the meeting. Teams audio conferencing feature returns a collection of all toll and toll-free numbers, with concomitant country names and city names, giving users control on what Teams meeting dial-in details to use.

-For example, a Standard workflow can use both managed connectors and built-in connectors for Azure Blob Storage, Azure Cosmos DB, Azure Event Hubs, Azure Service Bus, FTP, IBM DB2, IBM MQ, SFTP, and SQL Server. A Consumption workflow doesn't have the built-in versions. A Consumption workflow can use built-in connectors for Azure API Management, and Azure App Services, while a Standard workflow doesn't have these built-in connectors.

-| Azure API Management<br>Azure App Services <br>Azure Functions <br>Azure Logic Apps <br>Batch <br>Control <br>Data Operations <br>Date Time <br>Flat File <br>HTTP <br>Inline Code <br>Integration Account <br>Liquid <br>Request <br>Schedule <br>Variables <br>XML | AS2 (v2) <br>Azure AI Search* <br>Azure Automation* <br>Azure Blob Storage* <br>Azure Cosmos DB* <br>Azure Event Grid Publisher* <br>Azure Event Hubs* <br>Azure File Storage* <br>Azure Functions <br>Azure Key Vault* <br>Azure OpenAI* <br>Azure Queue Storage* <br>Azure Service Bus* <br>Azure Table Storage* <br>Batch Operations <br>Control <br>Data Mapper Operations <br>Data Operations <br>Date Time <br>EDIFACT <br>File System* <br>Flat File <br>FTP* <br>HTTP <br>IBM 3270* <br>IBM CICS* <br>IBM DB2* <br>IBM Host File* <br>IBM IMS* <br>IBM MQ* <br>Inline Code <br>Integration Account <br>JDBC* <br>Liquid Operations <br>Request <br>RosettaNet <br>SAP* <br>Schedule <br>SFTP* <br>SMTP* <br>SQL Server* <br>SWIFT <br>Variables <br>Workflow Operations <br>X12 <br>XML Operations |

- \

- \

- [**Schedule**][schedule-doc]

- \

- \

- [**Recurrence**][schedule-recurrence-doc]: Trigger a workflow based on the specified recurrence.

- \

- \

- [**Sliding Window**][schedule-sliding-window-doc]<br>(*Consumption workflow only*): <br>Trigger a workflow that needs to handle data in continuous chunks.

- \

- \

- [**Delay**][schedule-delay-doc]: Pause your workflow for the specified duration.

- \

- \

- [**Delay until**][schedule-delay-until-doc]: Pause your workflow until the specified date and time.

- \

- \

- [**HTTP**][http-doc]

- \

- \

- Call an HTTP or HTTPS endpoint by using either the HTTP trigger or action.

- \

- \

- You can also use these other built-in HTTP triggers and actions:

- \

- \

- [**Request**][http-request-doc]

- \

- \

- [**When a HTTP request is received**][http-request-doc]: Wait for a request from another workflow, app, or service. This trigger makes your workflow callable without having to be checked or polled on a schedule.

- \

- \

- [**Response**][http-request-doc]: Respond to a request received by the **When a HTTP request is received** trigger in the same workflow.

- \

- \

- [**Batch**][batch-doc]

- \

- \

- [**Batch messages**][batch-doc]: Trigger a workflow that processes messages in batches.

- \

- \

- [**Send messages to batch**][batch-doc]: Call an existing workflow that currently starts with a **Batch messages** trigger.

- \

- \

- [**File System**][file-system-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to a file system on your network machine to create and manage files.

- \

- \

- [**FTP**][ftp-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to an FTP or FTPS server in your Azure virtual network so that you can work with your files and folders.

- \

- \

- [**SFTP**][sftp-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to an SFTP server in your Azure virtual network so that you can work with your files and folders.

- \

- \

- [**SMTP**][smtp-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to an SMTP server so that you can send email.

- :::column-end:::

- :::column:::

- \

- \

- [**Azure AI Search**][azure-ai-search-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to AI Search so that you can perform document indexing and search operations in your workflow.

- \

- \

- [**Azure API Management**][azure-api-management-doc]<br>(*Consumption workflow only*)

- \

- \

- Call your own triggers and actions in APIs that you define, manage, and publish using [Azure API Management](../api-management/api-management-key-concepts.md). <br><br>**Note**: Not supported when using [Consumption tier for API Management](../api-management/api-management-features.md).

- :::column-end:::

- :::column:::

- [![Azure App Services icon][azure-app-services-icon]][azure-app-services-doc]

- \

- \

- [**Azure App Services**][azure-app-services-doc]<br>(*Consumption workflow only*)

- \

- \

- Call apps that you create and host on [Azure App Service](../app-service/overview.md), for example, API Apps and Web Apps.

- \

- \

- When Swagger is included, the triggers and actions defined by these apps appear like any other first-class triggers and actions in Azure Logic Apps.

- \

- \

- [**Azure Automation**][azure-automation-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to your Azure Automation accounts so you can create and manage Azure Automation jobs.

- \

- \

- [**Azure Blob Storage**][azure-blob-storage-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to your Azure Blob Storage account so you can create and manage blob content.

- \

- \

- [**Azure Cosmos DB**][azure-cosmos-db-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to Azure Cosmos DB so that you can access and manage Azure Cosmos DB documents.

- \

- \

- [**Azure Event Grid Publisher**][azure-event-grid-publisher-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to Azure Event Grid for event-based programming using pub-sub semantics.

- \

- \

- [**Azure Event Hubs**][azure-event-hubs-doc]<br>(*Standard workflow only*)

- \

- \

- Consume and publish events through an event hub. For example, get output from your workflow with Event Hubs, and then send that output to a real-time analytics provider.

- \

- \

- [**Azure File Storage**][azure-file-storage-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to your Azure Storage account so that you can create, update, and manage files.

- \

- \

- [**Azure Functions**][azure-functions-doc]

- \

- \

- Call [Azure-hosted functions](../azure-functions/functions-overview.md) to run your own *code snippets* (C# or Node.js) within your workflow.

- \

- \

- [**Azure Key Vault**][azure-key-vault-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to Azure Key Vault to store, access, and manage secrets.

- \

- \

- [**Azure Logic Apps**][nested-logic-app-doc]<br>(*Consumption workflow*) <br><br>-or-<br><br>**Workflow Operations**<br>(*Standard workflow*)

- \

- \

- Call other workflows that start with the Request trigger named **When a HTTP request is received**.

- \

- \

- [**Azure OpenAI**][azure-openai-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to Azure OpenAI to perform operations on large language models.

- \

- \

- [**Azure Service Bus**][azure-service-bus-doc]<br>(*Standard workflow only*)

- \

- \

- Manage asynchronous messages, queues, sessions, topics, and topic subscriptions.

- \

- \

- [**Azure Table Storage**][azure-table-storage-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to your Azure Storage account so that you can create, update, query, and manage tables.

- \

- \

- [**Azure Queue Storage**][azure-queue-storage-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to your Azure Storage account so that you can create, update, and manage queues.

- \

- \

- [**IBM 3270**][ibm-3270-doc]<br>(*Standard workflow only*)

- \

- \

- Call 3270 screen-driven apps on IBM mainframes from your workflow.

- \

- \

- [**IBM CICS**][ibm-cics-doc]<br>(*Standard workflow only*)

- \

- \

- Call CICS programs on IBM mainframes from your workflow.

- \

- \

- [**IBM DB2**][ibm-db2-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to IBM DB2 in the cloud or on-premises. Update a row, get a table, and more.

- \

- \

- [**IBM Host File**][ibm-host-file-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to IBM Host File and generate or parse contents.

- \

- \

- [**IBM IMS**][ibm-ims-doc]<br>(*Standard workflow only*)

- \

- \

- Call IMS programs on IBM mainframes from your workflow.

- \

- \

- [**IBM MQ**][ibm-mq-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to IBM MQ on-premises or in Azure to send and receive messages.

- \

- \

- [**JDBC**][jdbc-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to a relational database using JDBC drivers.

- \

- \

- [**SAP**][sap-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to SAP so you can send or receive messages and invoke actions.

- \

- \

- [**SQL Server**][sql-server-doc]<br>(*Standard workflow only*)

- \

- \

- Connect to your SQL Server on premises or an Azure SQL Database in the cloud so that you can manage records, run stored procedures, or perform queries.

- \

- \

- [**Azure Functions**][azure-functions-doc]

- \

- \

- Call [Azure-hosted functions](../azure-functions/functions-overview.md) to run your own *code snippets* (C# or Node.js) within your workflow.

- \

- \

- [**Inline Code**][inline-code-doc]

- \

- \

- [Add and run inline JavaScript code snippets](../logic-apps/logic-apps-add-run-inline-code.md) from your workflow.

- \

- \

- [**Local Function Operations**][local-function-doc]<br>(Standard workflow only)

- \

- \

- [Create and run .NET Framework code](../logic-apps/create-run-custom-code-functions.md) from your workflow.

- \

- \

- [**Condition**][condition-doc]

- \

- \

- Evaluate a condition and run different actions based on whether the condition is true or false.

- \

- \

- [**For Each**][for-each-doc]

- \

- \

- Perform the same actions on every item in an array.

- \

- \

- [**Scope**][scope-doc]

- \

- \

- Group actions into *scopes*, which get their own status after the actions in the scope finish running.

- \

- \

- [**Switch**][switch-doc]

- \

- \

- Group actions into *cases*, which are assigned unique values except for the default case. Run only that case whose assigned value matches the result from an expression, object, or token. If no matches exist, run the default case.

- \

- \

- [**Terminate**][terminate-doc]

- \

- \

- Stop an actively running workflow.

- \

- \

- [**Until**][until-doc]

- \

- \

- Repeat actions until the specified condition is true or some state has changed.

- \

- \

- [**Data Operations**][data-operations-doc]

- \

- \

- Perform operations with data.

- \

- \

- **Compose**: Create a single output from multiple inputs with various types.

- \

- \

- **Create CSV table**: Create a comma-separated-value (CSV) table from an array with JSON objects.

- \

- \

- **Create HTML table**: Create an HTML table from an array with JSON objects.

- \

- \

- **Filter array**: Create an array from items in another array that meet your criteria.

- \

- \

- **Join**: Create a string from all items in an array and separate those items with the specified delimiter.

- \

- \

- **Parse JSON**: Create user-friendly tokens from properties and their values in JSON content so that you can use those properties in your workflow.

- \

- \

- **Select**: Create an array with JSON objects by transforming items or values in another array and mapping those items to specified properties.

- \

- \

- **Date Time**

- \

- \

- Perform operations with timestamps.

- \

- \

- **Add to time**: Add the specified number of units to a timestamp.

- \

- \

- **Convert time zone**: Convert a timestamp from the source time zone to the target time zone.

- \

- \

- **Current time**: Return the current timestamp as a string.

- \

- \

- **Get future time**: Return the current timestamp plus the specified time units.

- \

- \

- **Get past time**: Return the current timestamp minus the specified time units.

- \

- \

- **Subtract from time**: Subtract a number of time units from a timestamp.

- \

- \

- [**Variables**][variables-doc]

- \

- \

- Perform operations with variables.

- \

- \

- **Append to array variable**: Insert a value as the last item in an array stored by a variable.

- \

- \

- **Append to string variable**: Insert a value as the last character in a string stored by a variable.

- \

- \

- **Decrement variable**: Decrease a variable by a constant value.

- \

- \

- **Increment variable**: Increase a variable by a constant value.

- \

- \

- **Initialize variable**: Create a variable and declare its data type and initial value.

- \

- \

- **Set variable**: Assign a different value to an existing variable.

- \

- \

- [**AS2 (v2)**][as2-doc]<br>(*Standard workflow only*)

- \

- \

- Encode and decode messages that use the AS2 protocol.

- \

- \

- [**EDIFACT**][edifact-doc]

- \

- \

- Encode and decode messages that use the EDIFACT protocol.

- \

- \

- [**Flat File**][flat-file-doc]

- \

- \

- Encode and decode XML messages between trading partners.

- \

- \

- [**Integration Account Artifact Lookup**][integration-account-doc]

- \

- \

- Get custom metadata for artifacts, such as trading partners, agreements, schemas, and so on, in your integration account.

- \

- \

- [**Liquid Operations**][liquid-transform-doc]

- \

- \

- Convert the following formats by using Liquid templates: <br><br>- JSON to JSON <br>- JSON to TEXT <br>- XML to JSON <br>- XML to TEXT

- \

- \

- [**RosettaNet**][rosettanet-doc]

- \

- \

- Encode and decode messages that use the RosettaNet protocol.

- \

- \

- [**SWIFT**][swift-doc]<br>(*Standard workflow only*)

- \

- \

- Encode and decode Society for Worldwide Interbank Financial Telecommuncation (SIWFT) transactions in flat-file XML message format.

- \

- \

- [**Transform XML**][xml-transform-doc]

- \

- \

- Convert the source XML format to another XML format.

- \

- \

- [**X12**][x12-doc]

- \

- \

- Encode and decode messages that use the X12 protocol.

- \

- \

- [**XML Validation**][xml-validate-doc]

- \

- \

- Validate XML documents against the specified schema.

-[azure-app-services-icon]: ./media/apis-list/azure-app-services.png

-[azure-app-services-doc]: ../logic-apps/logic-apps-custom-api-host-deploy-call.md "Integrate logic app workflows with App Service API Apps"

-If you're working with the **Logic App (Standard)** resource type, the built-in trigger called **When an item is created or modified (preview)** is available and is based on the [Azure Cosmos DB change feed pattern](../cosmos-db/sql/change-feed-design-patterns.md). This trigger is unavailable for the **Logic App (Consumption)** resource type.

- | **Create Lease Container** | No | **No** or **Yes** | If the lease container already exists in the specified database, select **No**. If you want the trigger to create this container, select **Yes**. If you select **Yes** and are using manual throughput dedicated for each container, make sure to open the **Add new parameter** list to select the **Lease Container Throughput** property. Enter the number of [request units (RUs)](../cosmos-db/request-units.md) that you want to provision for this container. |

- \

- \

- [**Azure Blob Storage**][azure-blob-storage-doc]

- \

- \

- Connect to your Azure Storage account so that you can create and manage blob content.

- \

- \

- [**Azure Event Hubs**][azure-event-hubs-doc]

- \

- \

- Consume and publish events through an event hub. For example, get output from your workflow with Event Hubs, and then send that output to a real-time analytics provider.

- \

- \

- [**Azure Queues**][azure-queues-doc]

- \

- \

- Connect to your Azure Storage account so that you can create and manage queues and messages.

- \

- \

- [**Azure Service Bus**][azure-service-bus-doc]

- \

- \

- Manage asynchronous messages, sessions, and topic subscriptions with the most commonly used connector in Logic Apps.

- \

- \

- [**Azure Table Storage**][azure-table-storage-doc]

- \

- \

- Connect to your Azure Storage account so that you can create, update, query, and manage tables.

- \

- \

- [**File System**][file-system-doc]

- \

- \

- Connect to your on-premises file share so that you can create and manage files.

- \

- \

- [**FTP**][ftp-doc]

- \

- \

- Connect to FTP servers you can access from the internet so that you can work with your files and folders.

- \

- \

- [**Office 365 Outlook**][office-365-outlook-doc]

- \

- \

- Connect to your work or school email account so that you can create and manage emails, tasks, calendar events and meetings, contacts, requests, and more.

- \

- \

- [**Salesforce**][salesforce-doc]

- \

- \

- Connect to your Salesforce account so that you can create and manage items such as records, jobs, objects, and more.

- \

- \

- [**SharePoint Online**][sharepoint-online-doc]

- \

- \

- Connect to SharePoint Online so that you can manage files, attachments, folders, and more.

- \

- \

- [**SFTP-SSH**][sftp-ssh-doc]

- \

- \

- Connect to SFTP servers that you can access from the internet by using SSH so that you can work with your files and folders.

- \

- \

- [**SQL Server**][sql-server-doc]

- \

- \

- Connect to your SQL Server on premises or an Azure SQL Database in the cloud so that you can manage records, run stored procedures, or perform queries.

- \

- \

- [**IBM 3270**][ibm-3270-doc]

- \

- \

- [**MQ**][ibm-mq-doc]

- \

- \

- [**SAP**][sap-connector-doc]

- \

- \

- [**Apache Impala**][apache-impala-doc]

- \

- \

- [**Biztalk Server**][biztalk-server-doc]

- \

- \

- [**File System**][file-system-doc]

- \

- \

- [**IBM DB2**][ibm-db2-doc]

- \

- \

- [**IBM Informix**][ibm-informix-doc]

- \

- \

- [**MySQL**][mysql-doc]

- \

- \

- [**Oracle DB**][oracle-db-doc]

- \

- \

- [**PostgreSQL**][postgre-sql-doc]

- \

- \

- [**SAP**][sap-connector-doc]

- \

- \

- [**SharePoint Server**][sharepoint-server-doc]

- \

- \

- [**SQL Server**][sql-server-doc]

- \

- \

- [**Teradata**][teradata-doc]

- \

- \

- [**AS2 Decode (v2)**][as2-doc]

- \

- \

- [**AS2 Encode (v2)**][as2-doc]

- \

- \

- [**AS2 decoding**][as2-doc]

- \

- \

- [**AS2 encoding**][as2-doc]

- \

- \

- [**EDIFACT decoding**][edifact-decode-doc]

- \

- \

- [**EDIFACT encoding**][edifact-encode-doc]

- \

- \

- [**X12 decoding**][x12-decode-doc]

- \

- \

- [**X12 encoding**][x12-encode-doc]

- \

- \

- [**AS2** ISE][as2-doc]

- \

- \

- [**Azure Automation** ISE][azure-automation-doc]

- \

- \

- [**Azure Blob Storage** ISE][azure-blob-storage-doc]

- \

- \

- [**Azure Cosmos DB** ISE][azure-cosmos-db-doc]

- \

- \

- [**Azure Event Hubs** ISE][azure-event-hubs-doc]

- \

- \

- [**Azure Event Grid** ISE][azure-event-grid-doc]

- \

- \

- [**Azure Files** ISE][azure-file-storage-doc]

- \

- \

- [**Azure Key Vault** ISE][azure-key-vault-doc]

- \

- \

- [**Azure Monitor Logs** ISE][azure-monitor-logs-doc]

- \

- \

- [**Azure Service Bus** ISE][azure-service-bus-doc]

- \

- \

- [**Azure Synapse Analytics** ISE][azure-sql-data-warehouse-doc]

- \

- \

- [**Azure Table Storage** ISE][azure-table-storage-doc]

- \

- \

- [**Azure Queues** ISE][azure-queues-doc]

- \

- \

- [**EDIFACT** ISE][edifact-doc]

- \

- \

- [**File System** ISE][file-system-doc]

- \

- \

- [**FTP** ISE][ftp-doc]

- \

- \

- [**IBM 3270** ISE][ibm-3270-doc]

- \

- \

- [**IBM DB2** ISE][ibm-db2-doc]

- \

- \

- [**IBM MQ** ISE][ibm-mq-doc]

- \

- \

- [**SAP** ISE][sap-connector-doc]

- \

- \

- [**SFTP-SSH** ISE][sftp-ssh-doc]

- \

- \

- [**SMTP** ISE][smtp-doc]

- \

- \

- [**SQL Server** ISE][sql-server-doc]

- \

- \

- [**X12** ISE][x12-doc]

- | Azure Location | The Azure location for your resources. [Make sure you select a location available for Azure PostgreSQL](../postgresql/flexible-server/overview.md#azure-regions). |

- | Azure Location | The Azure location for your resources. [Make sure you select a location available for Azure PostgreSQL](../postgresql/flexible-server/overview.md#azure-regions). |

- | Azure Location | The Azure location for your resources. [Make sure you select a location available for Azure PostgreSQL](../postgresql/flexible-server/overview.md#azure-regions). |

-[Azure Container Apps](overview.md) provides a [managed identity](managed-identity.md) for your app, which is a turn-key solution for securing access to [Azure Database for PostgreSQL](../postgresql/index.yml) and other Azure services. Managed identities in Container Apps make your app more secure by eliminating secrets from your app, such as credentials in the environment variables.

-This tutorial walks you through the process of building, configuring, deploying, and scaling Java container apps on Azure. At the end of this tutorial, you'll have a [Quarkus](https://quarkus.io) application storing data in a [PostgreSQL](../postgresql/index.yml) database with a managed identity running on [Container Apps](overview.md).

-This tutorial uses a sample Fruits list app with a web UI that calls a Quarkus REST API backed by [Azure Database for PostgreSQL](../postgresql/index.yml). The code for the app is available [on GitHub](https://github.com/quarkusio/quarkus-quickstarts/tree/main/hibernate-orm-panache-quickstart). To learn more about writing Java apps using Quarkus and PostgreSQL, see the [Quarkus Hibernate ORM with Panache Guide](https://quarkus.io/guides/hibernate-orm-panache) and the [Quarkus Datasource Guide](https://quarkus.io/guides/datasource).

-By default, Copilot in Azure is available to all users in a tenant. However, [Global Administrators](/entra/identity/role-based-access-control/permissions-reference#global-administrator) can choose to control access to Copilot in Azure for their organization. If you turn off access for your tenant, you can still grant access to specific Microsoft Entra users or groups.

-description: Use managed identity in Azure Cosmos DB to access Azure Key Vault.

-# Access Azure Key Vault from Azure Cosmos DB using a managed identity

-Azure Cosmos DB may need to read secret/key data from Azure Key Vault. For example, your Azure Cosmos DB may require a customer-managed key stored in Azure Key Vault. To do this, Azure Cosmos DB should be configured with a managed identity, and then an Azure Key Vault access policy should grant the managed identity access.

-## Prerequisites

-## Prerequisite check

-1. In a terminal or command window, store the names of your Azure Key Vault resource, Azure Cosmos DB account and resource group as shell variables named ``keyVaultName``, ``cosmosName``, and ``resourceGroupName``.

- ```azurecli-interactive

- # Variable for function app name

- keyVaultName="msdocs-keyvault"

-

- # Variable for Azure Cosmos DB account name

- cosmosName="msdocs-cosmos-app"

- # Variable for resource group name

- resourceGroupName="msdocs-cosmos-keyvault-identity"

- ```

- > [!NOTE]

- > These variables will be re-used in later steps. This example assumes your Azure Cosmos DB account name is ``msdocs-cosmos-app``, your key vault name is ``msdocs-keyvault`` and your resource group name is ``msdocs-cosmos-keyvault-identity``.

-## Create a system-assigned managed identity in Azure Cosmos DB

-First, create a system-assigned managed identity for the existing Azure Cosmos DB account.

-> [!IMPORTANT]

-> This how-to guide assumes that you are using a system-assigned managed identity. Many of the steps are similar when using a user-assigned managed identity.

-1. Run [``az cosmosdb identity assign``](/cli/azure/cosmosdb/identity#az-cosmosdb-identity-assign) to create a new system-assigned managed identity.

- ```azurecli-interactive

- az cosmosdb identity assign \

- --resource-group $resourceGroupName \

- --name $cosmosName

- ```

-1. Retrieve the metadata of the system-assigned managed identity using [``az cosmosdb identity show``](/cli/azure/cosmosdb/identity#az-cosmosdb-identity-show), filter to just return the ``principalId`` property using the **query** parameter, and store the result in a shell variable named ``principal``.

- ```azurecli-interactive

- principal=$(

- az cosmosdb identity show \

- --resource-group $resourceGroupName \

- --name $cosmosName \

- --query principalId \

- --output tsv

- )

- echo $principal

- ```

- > [!NOTE]

- > This variable will be re-used in a later step.

-## Create an Azure Key Vault access policy

-In this step, create an access policy in Azure Key Vault using the previously managed identity.

-1. Use the [``az keyvault set-policy``](/cli/azure/keyvault#az-keyvault-set-policy) command to create an access policy in Azure Key Vault that gives the Azure Cosmos DB managed identity permission to access Key Vault. Specifically, the policy will use the **key-permissions** parameters to grant permissions to ``get``, ``list``, and ``import`` keys.

- ```azurecli-interactive

- az keyvault set-policy \

- --name $keyVaultName \

- --object-id $principal \

- --key-permissions get list import

- ```

-## Next steps

-* To use customer-managed keys in Azure Key Vault with your Azure Cosmos DB account, see [configure customer-managed keys](how-to-setup-cmk.md#using-managed-identity)

-* To use Azure Key Vault to manage secrets, see [secure credentials](store-credentials-key-vault.md).

-description: Learn how to request access to Azure Cosmos DB previews

-# Access Azure Cosmos DB Preview Features

-## Steps to register for a preview feature from the portal

-Azure Cosmos DB offers several preview features that you can request access to. Here are the steps to request access to these preview features.

-1. Go to **Preview Features** area in your Azure subscription.

-2. Under **Type**, select "Microsoft.DocumentDBΓÇ¥.

-3. Click on the feature you would like access to in the list of available preview features.

-4. Click the **Register** button at the bottom of the page to join the preview.

-> [!TIP]

-> If your request is stuck in the **Pending** state for an abnormal amount of time, [create an Azure support request](../azure-portal/supportability/how-to-create-azure-support-request.md).

-## Next steps

-description: Try Azure Cosmos DB free with the Azure AI Advantage offer. Innovate with a full, integrated stack purpose-built for AI-powered applications.

- - ignite-2023

-# Try Azure Cosmos DB free with Azure AI Advantage

-Azure offers a full, integrated stack purpose-built for AI-powered applications. If you build your AI application stack on Azure using Azure Cosmos DB, your design can lead to solutions that get to market faster, experience lower latency, and have comprehensive built-in security.

-There are many benefits when using Azure Cosmos DB and Azure AI together:

-## The offer

-The Azure AI Advantage offer is for existing Azure AI and GitHub Copilot customers who want to use Azure Cosmos DB as part of their solution stack. With this offer, you get:

-If you decide that Azure Cosmos DB is right for you, you can receive up to 63% discount on [Azure Cosmos DB prices through Reserved Capacity](reserved-capacity.md).

-## Get started

-Get started with this offer by ensuring that you have the prerequisite services before applying.

-1. Make sure that you have an Azure account with an active subscription. If you don't already have an account, [create an account for free](https://azure.microsoft.com/free).

-1. Ensure that you previously used one of the qualifying services in your subscription:

- - Azure AI Services

- - Azure OpenAI Service

- - Azure Machine Learning

- - Azure AI Search

- - GitHub Copilot

-1. Create a new Azure Cosmos DB account using one of the following APIs:

- - API for NoSQL

- - API for MongoDB RU

- - API for Apache Cassandra

- - API for Apache Gremlin

- - API for Table

- > [!IMPORTANT]

- > The Azure Cosmos DB account must have been created within 30 days of registering for the offer.

-1. Register for the Azure AI Advantage offer: <https://aka.ms/AzureAIAdvantageSignupForm>

-1. The team reviews your registration and follows up via e-mail.

-## After the offer

-After 90 days, your Azure Cosmos DB account will continue to run at [standard pricing rates](https://azure.microsoft.com/pricing/details/cosmos-db/).

-## Related content

-description: Learn about key concepts for agents and step through the implementation of an AI agent memory system.

-# AI agent

-AI agents are designed to perform specific tasks, answer questions, and automate processes for users. These agents vary widely in complexity. They range from simple chatbots, to copilots, to advanced AI assistants in the form of digital or robotic systems that can run complex workflows autonomously.

-This article provides conceptual overviews and detailed implementation samples for AI agents.

-## What are AI agents?

-Unlike standalone large language models (LLMs) or rule-based software/hardware systems, AI agents have these common features:

-> [!NOTE]

-> The usage of the term *memory* in the context of AI agents is different from the concept of computer memory (like volatile, nonvolatile, and persistent memory).

-### Copilots

-Copilots are a type of AI agent. They work alongside users rather than operating independently. Unlike fully automated agents, copilots provide suggestions and recommendations to assist users in completing tasks.

-For instance, when a user is writing an email, a copilot might suggest phrases, sentences, or paragraphs. The user might also ask the copilot to find relevant information in other emails or files to support the suggestion (see [retrieval-augmented generation](vector-database.md#retrieval-augmented-generation)). The user can accept, reject, or edit the suggested passages.

-### Autonomous agents

-Autonomous agents can operate more independently. When you set up autonomous agents to assist with email composition, you could enable them to perform the following tasks:

-You can configure the agents to perform each of the preceding tasks with or without human approval.

-### Multi-agent systems

-A popular strategy for achieving performant autonomous agents is the use of multi-agent systems. In multi-agent systems, multiple autonomous agents, whether in digital or robotic form, interact or work together to achieve individual or collective goals. Agents in the system can operate independently and possess their own knowledge or information. Each agent might also have the capability to perceive its environment, make decisions, and execute actions based on its objectives.

-Multi-agent systems have these key characteristics:

-A multi-agent system provides the following advantages over a copilot or a single instance of LLM inference:

-## Implementation of AI agents

-### Reasoning and planning

-Complex reasoning and planning are the hallmark of advanced autonomous agents. Popular frameworks for autonomous agents incorporate one or more of the following methodologies (with links to arXiv archive pages) for reasoning and planning:

- Improve on chain of thought by having the model explicitly ask itself (and answer) follow-up questions before answering the initial question.

- Use LLMs to generate both reasoning traces and task-specific actions in an interleaved manner. Reasoning traces help the model induce, track, and update action plans, along with handling exceptions. Actions allow the model to connect with external sources, such as knowledge bases or environments, to gather additional information.

- Devise a plan to divide the entire task into smaller subtasks, and then carry out the subtasks according to the plan. This approach mitigates the calculation errors, missing-step errors, and semantic misunderstanding errors that are often present in zero-shot chain-of-thought prompting.

- Use *reflexion* agents that verbally reflect on task feedback signals. These agents maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials.

-### Frameworks

-Various frameworks and tools can facilitate the development and deployment of AI agents.

-For tool usage and perception that don't require sophisticated planning and memory, some popular LLM orchestrator frameworks are LangChain, LlamaIndex, Prompt Flow, and Semantic Kernel.

-For advanced and autonomous planning and execution workflows, [AutoGen](https://microsoft.github.io/autogen/) propelled the multi-agent wave that began in late 2022. OpenAI's [Assistants API](https://platform.openai.com/docs/assistants/overview) allows its users to create agents natively within the GPT ecosystem. [LangChain Agents](https://python.langchain.com/v0.1/docs/modules/agents/) and [LlamaIndex Agents](https://docs.llamaindex.ai/en/stable/use_cases/agents/) also emerged around the same time.

-> [!TIP]

-> The [implementation sample](#implementation-sample) later in this article shows how to build a simple multi-agent system by using one of the popular frameworks and a unified agent memory system.

-### AI agent memory system

-The prevalent practice for experimenting with AI-enhanced applications from 2022 through 2024 has been using standalone database management systems for various data workflows or types. For example, you can use an in-memory database for caching, a relational database for operational data (including tracing/activity logs and LLM conversation history), and a [pure vector database](vector-database.md#integrated-vector-database-vs-pure-vector-database) for embedding management.

-However, this practice of using a complex web of standalone databases can hurt an AI agent's performance. Integrating all these disparate databases into a cohesive, interoperable, and resilient memory system for AI agents is its own challenge.

-Also, many of the frequently used database services are not optimal for the speed and scalability that AI agent systems need. These databases' individual weaknesses are exacerbated in multi-agent systems.

-#### In-memory databases

-In-memory databases are excellent for speed but might struggle with the large-scale data persistence that AI agents need.

-#### Relational databases

-Relational databases are not ideal for the varied modalities and fluid schemas of data that agents handle. Relational databases require manual efforts and even downtime to manage provisioning, partitioning, and sharding.

-#### Pure vector databases

-Pure vector databases tend to be less effective for transactional operations, real-time updates, and distributed workloads. The popular pure vector databases nowadays typically offer:

-## Characteristics of a robust AI agent memory system

-Just as efficient database management systems are critical to the performance of software applications, it's critical to provide LLM-powered agents with relevant and useful information to guide their inference. Robust memory systems enable organizing and storing various kinds of information that the agents can retrieve at inference time.

-Currently, LLM-powered applications often use [retrieval-augmented generation](vector-database.md#retrieval-augmented-generation) that uses basic semantic search or vector search to retrieve passages or documents. [Vector search](vector-database.md#vector-search) can be useful for finding general information. But vector search might not capture the specific context, structure, or relationships that are relevant for a particular task or domain.

-For example, if the task is to write code, vector search might not be able to retrieve the syntax tree, file system layout, code summaries, or API signatures that are important for generating coherent and correct code. Similarly, if the task is to work with tabular data, vector search might not be able to retrieve the schema, the foreign keys, the stored procedures, or the reports that are useful for querying or analyzing the data.

-Weaving together a web of standalone in-memory, relational, and vector databases (as described [earlier](#ai-agent-memory-system)) is not an optimal solution for the varied data types. This approach might work for prototypical agent systems. However, it adds complexity and performance bottlenecks that can hamper the performance of advanced autonomous agents.

-A robust memory system should have the following characteristics.

-### Multimodal

-AI agent memory systems should provide collections that store metadata, relationships, entities, summaries, or other types of information that can be useful for various tasks and domains. These collections can be based on the structure and format of the data, such as documents, tables, or code. Or they can be based on the content and meaning of the data, such as concepts, associations, or procedural steps.

-Memory systems aren't just critical to AI agents. They're also important for the humans who develop, maintain, and use these agents.

-For example, humans might need to supervise agents' planning and execution workflows in near real time. While supervising, humans might interject with guidance or make in-line edits of agents' dialogues or monologues. Humans might also need to audit the reasoning and actions of agents to verify the validity of the final output.

-Human/agent interactions are likely in natural or programming languages, whereas agents "think," "learn," and "remember" through embeddings. This difference poses another requirement on memory systems' consistency across data modalities.

-### Operational

-Memory systems should provide memory banks that store information that's relevant for the interaction with the user and the environment. Such information might include chat history, user preferences, sensory data, decisions made, facts learned, or other operational data that's updated with high frequency and at high volumes.

-These memory banks can help the agents remember short-term and long-term information, avoid repeating or contradicting themselves, and maintain task coherence. These requirements must hold true even if the agents perform a multitude of unrelated tasks in succession. In advanced cases, agents might also test numerous branch plans that diverge or converge at different points.

-### Sharable but also separable

-At the macro level, memory systems should enable multiple AI agents to collaborate on a problem or process different aspects of the problem by providing shared memory that's accessible to all the agents. Shared memory can facilitate the exchange of information and the coordination of actions among the agents.

-At the same time, the memory system must allow agents to preserve their own persona and characteristics, such as their unique collections of prompts and memories.

-## Building a robust AI agent memory system

-The preceding characteristics require AI agent memory systems to be highly scalable and swift. Painstakingly weaving together disparate in-memory, relational, and vector databases (as described [earlier](#ai-agent-memory-system)) might work for early-stage AI-enabled applications. However, this approach adds complexity and performance bottlenecks that can hamper the performance of advanced autonomous agents.

-In place of all the standalone databases, Azure Cosmos DB can serve as a unified solution for AI agent memory systems. Its robustness successfully [enabled OpenAI's ChatGPT service](https://www.youtube.com/watch?v=6IIUtEFKJec&t) to scale dynamically with high reliability and low maintenance. Powered by an atom-record-sequence engine, it's the world's first globally distributed [NoSQL](distributed-nosql.md), [relational](distributed-relational.md), and [vector database](vector-database.md) service that offers a serverless mode. AI agents built on top of Azure Cosmos DB offer speed, scale, and simplicity.

-### Speed

-Azure Cosmos DB provides single-digit millisecond latency. This capability makes it suitable for processes that require rapid data access and management. These processes include caching (both traditional and [semantic caching](https://techcommunity.microsoft.com/t5/azure-architecture-blog/optimize-azure-openai-applications-with-semantic-caching/ba-p/4106867)), transactions, and operational workloads.

-Low latency is crucial for AI agents that need to perform complex reasoning, make real-time decisions, and provide immediate responses. In addition, the service's [use of the DiskANN algorithm](nosql/vector-search.md#enroll-in-the-vector-search-preview-feature) provides accurate and fast vector search with minimal memory consumption.

-### Scale

-Azure Cosmos DB is engineered for global distribution and horizontal scalability. It offers support for multiple-region I/O and multitenancy.

-The service helps ensure that memory systems can expand seamlessly and keep up with rapidly growing agents and associated data. The [availability guarantee in its service-level agreement (SLA)](https://www.microsoft.com/licensing/docs/view/Service-Level-Agreements-SLA-for-Online-Services) translates to less than 5 minutes of downtime per year. Pure vector database services, by contrast, come with 9 hours or more of downtime. This availability provides a solid foundation for mission-critical workloads. At the same time, the various service models in Azure Cosmos DB, like [Reserved Capacity](reserved-capacity.md) or Serverless, can help reduce financial costs.

-### Simplicity

-Azure Cosmos DB can simplify data management and architecture by integrating multiple database functionalities into a single, cohesive platform.

-Its integrated vector database capabilities can store, index, and query embeddings alongside the corresponding data in natural or programming languages. This capability enables greater data consistency, scale, and performance.

-Its flexibility supports the varied modalities and fluid schemas of the metadata, relationships, entities, summaries, chat history, user preferences, sensory data, decisions, facts learned, or other operational data involved in agent workflows. The database automatically indexes all data without requiring schema or index management, which helps AI agents perform complex queries quickly and efficiently.

-Azure Cosmos DB is fully managed, which eliminates the overhead of database administration tasks like scaling, patching, and backups. Without this overhead, developers can focus on building and optimizing AI agents without worrying about the underlying data infrastructure.

-### Advanced features

-Azure Cosmos DB incorporates advanced features such as change feed, which allows tracking and responding to changes in data in real time. This capability is useful for AI agents that need to react to new information promptly.

-Additionally, the built-in support for multi-master writes enables high availability and resilience to help ensure continuous operation of AI agents, even after regional failures.

-The five available [consistency levels](consistency-levels.md) (from strong to eventual) can also cater to various distributed workloads, depending on the scenario requirements.

-> [!TIP]

-> You can choose from two Azure Cosmos DB APIs to build your AI agent memory system:

->

-> - Azure Cosmos DB for NoSQL, which offers 99.999% availability guarantee and provides [three vector search algorithms](nosql/vector-search.md): IVF, HNSW, and DiskANN

-> - vCore-based Azure Cosmos DB for MongoDB, which offers 99.995% availability guarantee and provides [two vector search algorithms](mongodb/vcore/vector-search.md): IVF and HNSW (DiskANN is upcoming)

->

-> For information about the availability guarantees for these APIs, see the [service SLAs](https://www.microsoft.com/licensing/docs/view/Service-Level-Agreements-SLA-for-Online-Services).

-## Implementation sample

-This section explores the implementation of an autonomous agent to process traveler inquiries and bookings in a travel application for a cruise line.

-Chatbots are a long-standing concept, but AI agents are advancing beyond basic human conversation to carry out tasks based on natural language. These tasks traditionally required coded logic. The AI travel agent in this implementation sample uses the LangChain Agent framework for agent planning, tool usage, and perception.

-The AI travel agent's [unified memory system](#characteristics-of-a-robust-ai-agent-memory-system) uses the [vector database](vector-database.md) and document store capabilities of Azure Cosmos DB to address traveler inquiries and facilitate trip bookings. Using Azure Cosmos DB for this purpose helps ensure speed, scale, and simplicity, as described [earlier](#building-a-robust-ai-agent-memory-system).

-The sample agent operates within a Python FastAPI back end. It supports user interactions through a React JavaScript user interface.

-### Prerequisites

-### Download the project

-All of the code and sample datasets are available in [this GitHub repository](https://github.com/jonathanscholtes/Travel-AI-Agent-React-FastAPI-and-Cosmos-DB-Vector-Store). The repository includes these folders:

-### Load travel documents into Azure Cosmos DB

-The GitHub repository contains a Python project in the *loader* directory. It's intended for loading the sample travel documents into Azure Cosmos DB.

-#### Set up the environment

-Set up your Python virtual environment in the *loader* directory by running the following command:

-```python

- python -m venv venv

-```

-Activate your environment and install dependencies in the *loader* directory:

-```python

- venv\Scripts\activate

- python -m pip install -r requirements.txt

-```

-Create a file named *.env* in the *loader* directory, to store the following environment variables:

-```python

- OPENAI_API_KEY="<your OpenAI key>"

- MONGO_CONNECTION_STRING="mongodb+srv:<your connection string from Azure Cosmos DB>"

-```

-#### Load documents and vectors

-The Python file *main.py* serves as the central entry point for loading data into Azure Cosmos DB. This code processes the sample travel data from the GitHub repository, including information about ships and destinations. The code also generates travel itinerary packages for each ship and destination, so that travelers can book them by using the AI agent. The CosmosDBLoader tool is responsible for creating collections, vector embeddings, and indexes in the Azure Cosmos DB instance.

-Here are the contents of *main.py*:

-```python

-from cosmosdbloader import CosmosDBLoader

-from itinerarybuilder import ItineraryBuilder

-import json

-cosmosdb_loader = CosmosDBLoader(DB_Name='travel')

-#read in ship data

-with open('documents/ships.json') as file:

- ship_json = json.load(file)

-#read in destination data

-with open('documents/destinations.json') as file:

- destinations_json = json.load(file)

-builder = ItineraryBuilder(ship_json['ships'],destinations_json['destinations'])

-# Create five itinerary packages

-itinerary = builder.build(5)

-# Save itinerary packages to Cosmos DB

-cosmosdb_loader.load_data(itinerary,'itinerary')

-# Save destinations to Cosmos DB

-cosmosdb_loader.load_data(destinations_json['destinations'],'destinations')

-# Save ships to Cosmos DB, create vector store

-collection = cosmosdb_loader.load_vectors(ship_json['ships'],'ships')

-# Add text search index to ship name

-collection.create_index([('name', 'text')])

-```

-Load the documents, load the vectors, and create indexes by running the following command from the *loader* directory:

-```python

- python main.py

-```

-Here's the output of *main.py*:

-```markdown

-```

-### Build the AI travel agent by using Python FastAPI

-The AI travel agent is hosted in a back end API through Python FastAPI, which facilitates integration with the front-end user interface. The API project processes agent requests by [grounding](https://techcommunity.microsoft.com/t5/fasttrack-for-azure/grounding-llms/ba-p/3843857) the LLM prompts against the data layer, specifically the vectors and documents in Azure Cosmos DB.

-The agent makes use of various tools, particularly the Python functions provided at the API service layer. This article focuses on the code necessary for AI agents within the API code.

-The API project in the GitHub repository is structured as follows:

-### Set up the environment for the API

-We used Python version 3.11.4 for the development and testing of the API.

-Set up your Python virtual environment in the *api* directory:

-```python

- python -m venv venv

-```

-Activate your environment and install dependencies by using the *requirements* file in the *api* directory:

-```python

- venv\Scripts\activate

- python -m pip install -r requirements.txt

-```

-Create a file named *.env* in the *api* directory, to store your environment variables:

-```python

- OPENAI_API_KEY="<your Open AI key>"

- MONGO_CONNECTION_STRING="mongodb+srv:<your connection string from Azure Cosmos DB>"

-```

-Now that you've configured the environment and set up variables, run the following command from the *api* directory to initiate the server:

-```python

- python app.py

-```

-The FastAPI server starts on the localhost loopback 127.0.0.1 port 8000 by default. You can access the Swagger documents by using the following localhost address: `http://127.0.0.1:8000/docs`.

-### Use a session for the AI agent memory

-It's imperative for the travel agent to be able to reference previously provided information within the ongoing conversation. This ability is commonly known as *memory* in the context of LLMs.

-To achieve this objective, use the chat message history that's stored in the Azure Cosmos DB instance. The history for each chat session is stored through a session ID to ensure that only messages from the current conversation session are accessible. This necessity is the reason behind the existence of a `Get Session` method in the API. It's a placeholder method for managing web sessions to illustrate the use of chat message history.

-Select **Try it out** for `/session/`.

-```python

-{

- "session_id": "0505a645526f4d68a3603ef01efaab19"

-}

-```

-For the AI agent, you only need to simulate a session. The stubbed-out method merely returns a generated session ID for tracking message history. In a practical implementation, this session would be stored in Azure Cosmos DB and potentially in React `localStorage`.

-Here are the contents of *web/session.py*:

-```python

- @router.get("/")

- def get_session():

- return {'session_id':str(uuid.uuid4().hex)}

-```

-### Start a conversation with the AI travel agent

-Use the session ID that you obtained from the previous step to start a new dialogue with the AI agent, so you can validate its functionality. Conduct the test by submitting the following phrase: "I want to take a relaxing vacation."

-Select **Try it out** for `/agent/agent_chat`.

-Use this example parameter:

-```python

-{

- "input": "I want to take a relaxing vacation.",

- "session_id": "0505a645526f4d68a3603ef01efaab19"

-}

-```

-The initial execution results in a recommendation for the Tranquil Breeze Cruise and the Fantasy Seas Adventure Cruise, because the agent anticipates that they're the most relaxing cruises available through the vector search. These documents have the highest score for `similarity_search_with_score` called in the data layer of the API, `data.mongodb.travel.similarity_search()`.

-The similarity search scores appear as output from the API for debugging purposes. Here's the output after a call to `data.mongodb.travel.similarity_search()`:

-```markdown

-0.8394561085977978

-0.8086545112328692

-2

-```

-> [!TIP]

-> If documents are not being returned for vector search, modify the `similarity_search_with_score` limit or the score filter value as needed (`[doc for doc, score in docs if score >=.78]`) in `data.mongodb.travel.similarity_search()`.

-Calling `agent_chat` for the first time creates a new collection named `history` in Azure Cosmos DB to store the conversation by session. This call enables the agent to access the stored chat message history as needed. Subsequent executions of `agent_chat` with the same parameters produce varying results, because it draws from memory.

-### Walk through the AI agent

-When you're integrating the AI agent into the API, the web search components are responsible for initiating all requests. The web search components are followed by the search service, and finally the data components.

-In this specific case, you use a MongoDB data search that connects to Azure Cosmos DB. The layers facilitate the exchange of model components, with the AI agent and the AI agent tool code residing in the service layer. This approach enables the seamless interchangeability of data sources. It also extends the capabilities of the AI agent with additional, more intricate functionalities or tools.

-#### Service layer

-The service layer forms the cornerstone of core business logic. In this particular scenario, the service layer plays a crucial role as the repository for the LangChain Agent code. It facilitates the seamless integration of user prompts with Azure Cosmos DB data, conversation memory, and agent functions for the AI agent.

-The service layer employs a singleton pattern module for handling agent-related initializations in the *init.py* file. Here are the contents of *service/init.py*:

-```python

-from dotenv import load_dotenv

-from os import environ

-from langchain.globals import set_llm_cache

-from langchain_openai import ChatOpenAI

-from langchain_mongodb.chat_message_histories import MongoDBChatMessageHistory

-from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

-from langchain_core.runnables.history import RunnableWithMessageHistory

-from langchain.agents import AgentExecutor, create_openai_tools_agent

-from service import TravelAgentTools as agent_tools

-load_dotenv(override=False)

-chat : ChatOpenAI | None=None

-agent_with_chat_history : RunnableWithMessageHistory | None=None

-def LLM_init():

- global chat,agent_with_chat_history

- chat = ChatOpenAI(model_name="gpt-3.5-turbo-16k",temperature=0)

- tools = [agent_tools.vacation_lookup, agent_tools.itinerary_lookup, agent_tools.book_cruise ]

- prompt = ChatPromptTemplate.from_messages(

- [

- (

- "system",

- "You are a helpful and friendly travel assistant for a cruise company. Answer travel questions to the best of your ability providing only relevant information. In order to book a cruise you will need to capture the person's name.",

- ),

- MessagesPlaceholder(variable_name="chat_history"),

- ("user", "Answer should be embedded in html tags. {input}"),

- MessagesPlaceholder(variable_name="agent_scratchpad"),

- ]

- )

- #Answer should be embedded in HTML tags. Only answer questions related to cruise travel, If you can not answer respond with \"I am here to assist with your travel questions.\".

- agent = create_openai_tools_agent(chat, tools, prompt)

- agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

- agent_with_chat_history = RunnableWithMessageHistory(

- agent_executor,

- lambda session_id: MongoDBChatMessageHistory( database_name="travel",

- collection_name="history",

- connection_string=environ.get("MONGO_CONNECTION_STRING"),

- session_id=session_id),

- input_messages_key="input",

- history_messages_key="chat_history",

-)

-LLM_init()

-```

-The *init.py* file initiates the loading of environment variables from an *.env* file by using the `load_dotenv(override=False)` method. Then, a global variable named `agent_with_chat_history` is instantiated for the agent. This agent is intended for use by *TravelAgent.py*.

-The `LLM_init()` method is invoked during module initialization to configure the AI agent for conversation via the API web layer. The OpenAI `chat` object is instantiated through the GPT-3.5 model and incorporates specific parameters such as model name and temperature. The `chat` object, tools list, and prompt template are combined to generate `AgentExecutor`, which operates as the AI travel agent.

-The agent with history, `agent_with_chat_history`, is established through `RunnableWithMessageHistory` with chat history (`MongoDBChatMessageHistory`). This action enables it to maintain a complete conversation history via Azure Cosmos DB.

-#### Prompt

-The LLM prompt initially began with the simple statement "You are a helpful and friendly travel assistant for a cruise company." However, testing showed that you could obtain more consistent results by including the instruction "Answer travel questions to the best of your ability, providing only relevant information. To book a cruise, capturing the person's name is essential." The results appear in HTML format to enhance the visual appeal of the web interface.

-#### Agent tools

-[Tools](#what-are-ai-agents) are interfaces that an agent can use to interact with the world, often through function calling.

-When you're creating an agent, you must furnish it with a set of tools that it can use. The `@tool` decorator offers the most straightforward approach to defining a custom tool.

-By default, the decorator uses the function name as the tool name, although you can replace it by providing a string as the first argument. The decorator uses the function's docstring as the tool's description, so it requires the provisioning of a docstring.

-Here are the contents of *service/TravelAgentTools.py*:

-```python

-from langchain_core.tools import tool

-from langchain.docstore.document import Document

-from data.mongodb import travel

-from model.travel import Ship

-@tool

-def vacation_lookup(input:str) -> list[Document]:

- """find information on vacations and trips"""

- ships: list[Ship] = travel.similarity_search(input)

- content = ""

- for ship in ships:

- content += f" Cruise ship {ship.name} description: {ship.description} with amenities {'/n-'.join(ship.amenities)} "

- return content

-@tool

-def itinerary_lookup(ship_name:str) -> str:

- """find ship itinerary, cruise packages and destinations by ship name"""

- it = travel.itnerary_search(ship_name)

- results = ""

- for i in it:

- results += f" Cruise Package {i.Name} room prices: {'/n-'.join(i.Rooms)} schedule: {'/n-'.join(i.Schedule)}"

- return results

-@tool

-def book_cruise(package_name:str, passenger_name:str, room: str )-> str:

- """book cruise using package name and passenger name and room """

- print(f"Package: {package_name} passenger: {passenger_name} room: {room}")

- # LLM defaults empty name to John Doe

- if passenger_name == "John Doe":

- return "In order to book a cruise I need to know your name."

- else:

- if room == '':

- return "which room would you like to book"

- return "Cruise has been booked, ref number is 343242"

-```

-The *TravelAgentTools.py* file defines three tools:

-Specific instructions ("In order to book a cruise I need to know your name") might be necessary to ensure the capture of the passenger's name and room number for booking the cruise package, even though you included such instructions in the LLM prompt.

-#### AI agent

-The fundamental concept that underlies agents is to use a language model for selecting a sequence of actions to execute.

-Here are the contents of *service/TravelAgent.py*:

-```python

-from .init import agent_with_chat_history

-from model.prompt import PromptResponse

-import time

-from dotenv import load_dotenv

-load_dotenv(override=False)

-def agent_chat(input:str, session_id:str)->str:

- start_time = time.time()

- results=agent_with_chat_history.invoke(

- {"input": input},

- config={"configurable": {"session_id": session_id}},

- )

- return PromptResponse(text=results["output"],ResponseSeconds=(time.time() - start_time))

-```

-The *TravelAgent.py* file is straightforward, because `agent_with_chat_history` and its dependencies (tools, prompt, and LLM) are initialized and configured in the *init.py* file. This file calls the agent by using the input received from the user, along with the session ID for conversation memory. Afterward, `PromptResponse` (model/prompt) is returned with the agent's output and response time.

-## AI agent integration with the React user interface

-With the successful loading of the data and accessibility of the AI agent through the API, you can now complete the solution by establishing a web user interface (by using React) for your travel website. Using the capabilities of React helps illustrate the seamless integration of the AI agent into a travel site. This integration enhances the user experience with a conversational travel assistant for inquiries and bookings.

-### Set up the environment for React

-Install Node.js and the dependencies before testing the React interface.

-Run the following command from the *web* directory to perform a clean installation of project dependencies. The installation might take some time.

-```javascript

- npm ci

-```

-Next, create a file named *.env* within the *web* directory to facilitate the storage of environment variables. Include the following details in the newly created *.env* file:

-`REACT_APP_API_HOST=http://127.0.0.1:8000`

-Now, run the following command from the *web* directory to initiate the React web user interface:

-```javascript

- npm start

-```

-Running the previous command opens the React web application.

-### Walk through the React web interface

-The web project of the GitHub repository is a straightforward application to facilitate user interaction with the AI agent. The primary components required to converse with the agent are *TravelAgent.js* and *ChatLayout.js*. The *Main.js* file serves as the central module or user landing page.

-#### Main

-The main component serves as the central manager of the application. It acts as the designated entry point for routing. Within the render function, it produces JSX code to delineate the main page layout. This layout encompasses placeholder elements for the application, such as logos and links, a section that houses the travel agent component, and a footer that contains a sample disclaimer about the application's nature.

-Here are the contents of *main.js*:

-```javascript

- import React, { Component } from 'react'

-import { Stack, Link, Paper } from '@mui/material'

-import TravelAgent from './TripPlanning/TravelAgent'

-import './Main.css'

-class Main extends Component {

- constructor() {

- super()

- }

- render() {

- return (

- <div className="Main">

- <div className="Main-Header">

- <Stack direction="row" spacing={5}>

- <img src="/mainlogo.png" alt="Logo" height={'120px'} />

- <Link

- href="#"

- sx={{ color: 'white', fontWeight: 'bold', fontSize: 18 }}

- underline="hover"

- >

- Ships

- </Link>

- <Link

- href="#"

- sx={{ color: 'white', fontWeight: 'bold', fontSize: 18 }}

- underline="hover"

- >

- Destinations

- </Link>

- </Stack>

- </div>

- <div className="Main-Body">

- <div className="Main-Content">

- <Paper elevation={3} sx={{p:1}} >

- <Stack

- direction="row"

- justifyContent="space-evenly"

- alignItems="center"

- spacing={2}

- >

-

- <Link href="#">

- <img

- src={require('./images/destinations.png')} width={'400px'} />

- </Link>

- <TravelAgent ></TravelAgent>

- <Link href="#">

- <img

- src={require('./images/ships.png')} width={'400px'} />

- </Link>

-

- </Stack>

- </Paper>

- </div>

- </div>

- <div className="Main-Footer">

- <b>Disclaimer: Sample Application</b>

- <br />

- Please note that this sample application is provided for demonstration

- purposes only and should not be used in production environments

- without proper validation and testing.

- </div>

- </div>

- )

- }

-}

-export default Main

-```

-#### Travel agent

-The travel agent component has a straightforward purpose: capturing user inputs and displaying responses. It plays a key role in managing the integration with the back-end AI agent, primarily by capturing sessions and forwarding user prompts to the FastAPI service. The resulting responses are stored in an array for display, facilitated by the chat layout component.

-Here are the contents of *TripPlanning/TravelAgent.js*:

-```javascript

-import React, { useState, useEffect } from 'react'

-import { Button, Box, Link, Stack, TextField } from '@mui/material'

-import SendIcon from '@mui/icons-material/Send'

-import { Dialog, DialogContent } from '@mui/material'

-import ChatLayout from './ChatLayout'

-import './TravelAgent.css'

-export default function TravelAgent() {

- const [open, setOpen] = React.useState(false)

- const [session, setSession] = useState('')

- const [chatPrompt, setChatPrompt] = useState(

- 'I want to take a relaxing vacation.',

- )

- const [message, setMessage] = useState([

- {

- message: 'Hello, how can I assist you today?',

- direction: 'left',

- bg: '#E7FAEC',

- },

- ])

- const handlePrompt = (prompt) => {

- setChatPrompt('')

- setMessage((message) => [

- ...message,

- { message: prompt, direction: 'right', bg: '#E7F4FA' },

- ])

- console.log(session)

- fetch(process.env.REACT_APP_API_HOST + '/agent/agent_chat', {

- method: 'POST',

- headers: {

- 'Content-Type': 'application/json',

- },

- body: JSON.stringify({ input: prompt, session_id: session }),

- })

- .then((response) => response.json())

- .then((res) => {

- setMessage((message) => [

- ...message,

- { message: res.text, direction: 'left', bg: '#E7FAEC' },

- ])

- })

- }

- const handleSession = () => {

- fetch(process.env.REACT_APP_API_HOST + '/session/')

- .then((response) => response.json())

- .then((res) => {

- setSession(res.session_id)

- })

- }

- const handleClickOpen = () => {

- setOpen(true)

- }

- const handleClose = (value) => {

- setOpen(false)

- }

- useEffect(() => {

- if (session === '') handleSession()

- }, [])

- return (

- <Box>

- <Dialog onClose={handleClose} open={open} maxWidth="md" fullWidth="true">

- <DialogContent>

- <Stack>

- <Box sx={{ height: '500px' }}>

- <div className="AgentArea">

- <ChatLayout messages={message} />

- </div>

- </Box>

- <Stack direction="row" spacing={0}>

- <TextField

- sx={{ width: '80%' }}

- variant="outlined"

- label="Message"

- helperText="Chat with AI Travel Agent"

- defaultValue="I want to take a relaxing vacation."

- value={chatPrompt}

- onChange={(event) => setChatPrompt(event.target.value)}

- ></TextField>

- <Button

- variant="contained"

- endIcon={<SendIcon />}

- sx={{ mb: 3, ml: 3, mt: 1 }}

- onClick={(event) => handlePrompt(chatPrompt)}

- >

- Submit

- </Button>

- </Stack>

- </Stack>

- </DialogContent>

- </Dialog>

- <Link href="#" onClick={() => handleClickOpen()}>

- <img src={require('.././images/planvoyage.png')} width={'400px'} />

- </Link>

- </Box>

- )

-}

-```

-Select **Effortlessly plan your voyage** to open the travel assistant.

-#### Chat layout

-The chat layout component oversees the arrangement of the chat. It systematically processes the chat messages and implements the formatting specified in the `message` JSON object.

-Here are the contents of *TripPlanning/ChatLayout.py*:

-```javascript

-import React from 'react'

-import { Box, Stack } from '@mui/material'

-import parse from 'html-react-parser'

-import './ChatLayout.css'

-export default function ChatLayout(messages) {

- return (

- <Stack direction="column" spacing="1">

- {messages.messages.map((obj, i = 0) => (

- <div className="bubbleContainer" key={i}>

- <Box

- key={i++}

- className="bubble"

- sx={{ float: obj.direction, fontSize: '10pt', background: obj.bg }}

- >

- <div>{parse(obj.message)}</div>

- </Box>

- </div>

- ))}

- </Stack>

- )

-}

-```

-User prompts are on the right side and colored blue. Responses from the AI travel agent are on the left side and colored green. As the following image shows, the HTML-formatted responses are accounted for in the conversation.

-When your AI agent is ready to go into production, you can use semantic caching to improve query performance by 80% and to reduce LLM inference and API call costs. To implement semantic caching, see [this post on the Stochastic Coder blog](https://stochasticcoder.com/2024/03/22/improve-llm-performance-using-semantic-cache-with-cosmos-db/).

-## Related content

-description: Change data capture (CDC) in Azure Cosmos DB analytical store allows you to efficiently consume a continuous and incremental feed of changed data.

-# Change Data Capture in Azure Cosmos DB analytical store

-Change data capture (CDC) in [Azure Cosmos DB analytical store](analytical-store-introduction.md) allows you to efficiently consume a continuous and incremental feed of changed (inserted, updated, and deleted) data from analytical store. Seamlessly integrated with Azure Synapse and Azure Data Factory, it provides you with a scalable no-code experience for high data volume. As the change data capture feature is based on analytical store, it [doesn't consume provisioned RUs, doesn't affect your transactional workloads](analytical-store-introduction.md#decoupled-performance-for-analytical-workloads), provides lower latency, and has lower TCO.

-The change data capture feature in Azure Cosmos DB analytical store can write to various sinks using an Azure Synapse or Azure Data Factory data flow.

-For more information on supported sink types in a mapping data flow, see [data flow supported sink types](../data-factory/data-flow-sink.md#supported-sinks).

-In addition to providing incremental data feed from analytical store to diverse targets, change data capture supports the following capabilities:

-## Efficient incremental data capture with internally managed checkpoints

-Each change in Cosmos DB container appears exactly once in the CDC feed, and the checkpoints are managed internally for you. This helps to address the below disadvantages of the common pattern of using custom checkpoints based on the ΓÇ£_tsΓÇ¥ value:

- * The ΓÇ£_tsΓÇ¥ filter is applied against the data files which does not always guarantee minimal data scan. The internally managed GLSN based checkpoints in the new CDC capability ensure that the incremental data identification is done, just based on the metadata and so guarantees minimal data scanning in each stream.

-* The analytical store sync process does not guarantee ΓÇ£_tsΓÇ¥ based ordering which means that there could be cases where an incremental recordΓÇÖs ΓÇ£_tsΓÇ¥ is lesser than the last checkpointed ΓÇ£_tsΓÇ¥ and could be missed out in the incremental stream. The new CDC does not consider ΓÇ£_tsΓÇ¥ to identify the incremental records and thus guarantees that none of the incremental records are missed.

-## Features

-Change data capture in Azure Cosmos DB analytical store supports the following key features.

-### Capturing changes from the beginning

-When the `Start from beginning` option is selected, the initial load includes a full snapshot of container data in the first run, and changed or incremental data is captured in subsequent runs. This is limited by the `analytical TTL` property and documents TTL-removed from analytical store are not included in the change feed. Example: Imagine a container with `analytical TTL` set to 31536000 seconds, which is equivalent to 1 year. If you create a CDC process for this container, only documents newer than 1 year will be included in the initial load.

-### Capturing changes from a given timestamp

-When the `Start from timestamp` option is selected, the initial load processes the data from the given timestamp, and incremental or changed data is captured in subsequent runs. This process is also limited by the `analytical TTL` property.

-### Capturing changes from now

-When the `Start from timestamp` option is selected, all past operations of the container are not captured.

-### Capturing deletes, intermediate updates, and TTLs

-The change data capture feature for the analytical store captures deletes, intermediate updates, and TTL operations. The captured deletes and updates can be applied on Sinks that support delete and update operations. The {_rid} value uniquely identifies the records and so by specifying {_rid} as key column on the Sink side, the update and delete operations would be reflected on the Sink.

-Note that TTL operations are considered deletes. Check the [source settings](get-started-change-data-capture.md) section to check mode details and the support for intermediate updates and deletes in sinks.

-### Filter the change feed for a specific type of operation

-You can filter the change data capture feed for a specific type of operation. For example, you can selectively capture the insert and update operations only, thereby ignoring the user-delete and TTL-delete operations.

-### Applying filters, projections, and transformations on the Change feed via source query

-You can optionally use a source query to specify filter(s), projection(s), and transformation(s), which would all be pushed down to the columnar analytical store. Here's a sample source-query that would only capture incremental records with the filter `Category = 'Urban'`. This sample query projects only five fields and applies a simple transformation:

-```sql

-SELECT ProductId, Product, Segment, concat(Manufacturer, '-', Category) as ManufacturerCategory

-FROM c

-WHERE Category = 'Urban'

-```

-### Multiple CDC processes

-You can create multiple processes to consume CDC in analytical store. This approach brings flexibility to support different scenarios and requirements. While one process may have no data transformations and multiple sinks, another one can have data flattening and one sink. And they can run in parallel.

-### Throughput isolation, lower latency and lower TCO

-Operations on Cosmos DB analytical store don't consume the provisioned RUs and so don't affect your transactional workloads. Change data capture with analytical store also has lower latency and lower TCO. The lower latency is attributed to analytical store enabling better parallelism for data processing and reduces the overall TCO enabling you to drive cost efficiencies in these rapidly shifting economic conditions.

-## Scenarios

-Here are common scenarios where you could use change data capture and the analytical store.

-### Consuming incremental data from Cosmos DB

-You can use analytical store change data capture, if you're currently using or planning to use:

- - The analytical store has up to 2-min latency to sync transactional store data. You can schedule Data Flows in Azure Data Factory every minute.

- - If you need to stream without the above latency, we recommend using the change feed feature of the transactional store.

- - If you're using Azure Functions triggers or any other option with change feed and would like to capture deletes, incremental changes, apply transformations etc.; we recommend change data capture over analytical store.

-### Incremental feed to analytical platform of your choice

-Change data capture capability enables an end-to-end analytical solution providing you with the flexibility to use Azure Cosmos DB data with any of the supported sink types. For more information on supported sink types, see [data flow supported sink types](../data-factory/data-flow-sink.md#supported-sinks). Change data capture also enables you to bring Azure Cosmos DB data into a centralized data lake and join the data with data from other diverse sources. You can flatten the data, partition it, and apply more transformations either in Azure Synapse Analytics or Azure Data Factory.

-## Change data capture on Azure Cosmos DB for MongoDB containers

-The linked service interface for the API for MongoDB isn't available within Azure Data Factory data flows yet. You can use your API for MongoDB's account endpoint with the **Azure Cosmos DB for NoSQL** linked service interface as a work around until the Mongo linked service is directly supported.

-In the interface for a new NoSQL linked service, select **Enter Manually** to provide the Azure Cosmos DB account information. Here, use the account's NoSQL document endpoint (Example: `https://<account-name>.documents.azure.com:443/`) instead of the Mongo DB endpoint (Example: `mongodb://<account-name>.mongo.cosmos.azure.com:10255/`)

-## Next steps

-> [!div class="nextstepaction"]

-> [Get started with change data capture in the analytical store](get-started-change-data-capture.md)

-description: Learn about Azure Cosmos DB transactional (row-based) and analytical(column-based) store. Benefits of analytical store, performance impact for large-scale workloads, and auto sync of data from transactional store to analytical store.

-# What is Azure Cosmos DB analytical store?

- > [!IMPORTANT]

- > Mirroring in Microsoft Fabric is now available in preview for NoSql API. This feature provides all the capabilities of Azure Synapse Link with better analytical performance, ability to unify your data estate with Fabric OneLake and open access to your data in OneLake with Delta Parquet format. If you are considering Azure Synapse Link, we recommend that you try mirroring to assess overall fit for your organization. To get started with mirroring, click [here](/fabric/database/mirrored-database/azure-cosmos-db?context=/azure/cosmos-db/context/context).

-To get started with Azure Synapse Link, please visit [ΓÇ£Getting started with Azure Synapse LinkΓÇ¥](synapse-link.md)

-Azure Cosmos DB analytical store is a fully isolated column store for enabling large-scale analytics against operational data in your Azure Cosmos DB, without any impact to your transactional workloads.

-Azure Cosmos DB transactional store is schema-agnostic, and it allows you to iterate on your transactional applications without having to deal with schema or index management. In contrast to this, Azure Cosmos DB analytical store is schematized to optimize for analytical query performance. This article describes in detailed about analytical storage.

-## Challenges with large-scale analytics on operational data

-The multi-model operational data in an Azure Cosmos DB container is internally stored in an indexed row-based "transactional store". Row store format is designed to allow fast transactional reads and writes in the order-of-milliseconds response times, and operational queries. If your dataset grows large, complex analytical queries can be expensive in terms of provisioned throughput on the data stored in this format. High consumption of provisioned throughput in turn, impacts the performance of transactional workloads that are used by your real-time applications and services.

-Traditionally, to analyze large amounts of data, operational data is extracted from Azure Cosmos DB's transactional store and stored in a separate data layer. For example, the data is stored in a data warehouse or data lake in a suitable format. This data is later used for large-scale analytics and analyzed using compute engines such as the Apache Spark clusters. The separation of analytical from operational data results in delays for analysts that want to use the most recent data.

-The ETL pipelines also become complex when handling updates to the operational data when compared to handling only newly ingested operational data.

-## Column-oriented analytical store

-Azure Cosmos DB analytical store addresses the complexity and latency challenges that occur with the traditional ETL pipelines. Azure Cosmos DB analytical store can automatically sync your operational data into a separate column store. Column store format is suitable for large-scale analytical queries to be performed in an optimized manner, resulting in improving the latency of such queries.

-Using Azure Synapse Link, you can now build no-ETL HTAP solutions by directly linking to Azure Cosmos DB analytical store from Azure Synapse Analytics. It enables you to run near real-time large-scale analytics on your operational data.

-## Features of analytical store

-When you enable analytical store on an Azure Cosmos DB container, a new column-store is internally created based on the operational data in your container. This column store is persisted separately from the row-oriented transactional store for that container, in a storage account that is fully managed by Azure Cosmos DB, in an internal subscription. Customers don't need to spend time with storage administration. The inserts, updates, and deletes to your operational data are automatically synced to analytical store. You don't need the Change Feed or ETL to sync the data.

-## Column store for analytical workloads on operational data

-Analytical workloads typically involve aggregations and sequential scans of selected fields. The data analytical store is stored in a column-major order, allowing values of each field to be serialized together, where applicable. This format reduces the IOPS required to scan or compute statistics over specific fields. It dramatically improves the query response times for scans over large data sets.

-For example, if your operational tables are in the following format:

-The row store persists the above data in a serialized format, per row, on the disk. This format allows for faster transactional reads, writes, and operational queries, such as, "Return information about Product 1". However, as the dataset grows large and if you want to run complex analytical queries on the data it can be expensive. For example, if you want to get "the sales trends for a product under the category named 'Equipment' across different business units and months", you need to run a complex query. Large scans on this dataset can get expensive in terms of provisioned throughput and can also impact the performance of the transactional workloads powering your real-time applications and services.

-Analytical store, which is a column store, is better suited for such queries because it serializes similar fields of data together and reduces the disk IOPS.

-The following image shows transactional row store vs. analytical column store in Azure Cosmos DB:

-## Decoupled performance for analytical workloads

-There's no impact on the performance of your transactional workloads due to analytical queries, as the analytical store is separate from the transactional store. Analytical store doesn't need separate request units (RUs) to be allocated.

-## Auto-Sync

-Auto-Sync refers to the fully managed capability of Azure Cosmos DB where the inserts, updates, deletes to operational data are automatically synced from transactional store to analytical store in near real time. Auto-sync latency is usually within 2 minutes. In cases of shared throughput database with a large number of containers, auto-sync latency of individual containers could be higher and take up to 5 minutes.

-At the end of each execution of the automatic sync process, your transactional data will be immediately available for Azure Synapse Analytics runtimes:

-* Azure Synapse Analytics Spark pools can read all data, including the most recent updates, through Spark tables, which are updated automatically, or via the `spark.read` command, that always reads the last state of the data.

-* Azure Synapse Analytics SQL Serverless pools can read all data, including the most recent updates, through views, which are updated automatically, or via `SELECT` together with the `OPENROWSET` commands, which always reads the latest status of the data.

-> [!NOTE]

-> Your transactional data will be synchronized to analytical store even if your transactional time-to-live (TTL) is smaller than 2 minutes.

-> [!NOTE]

-> Please note that if you delete your container, analytical store is also deleted.

-## Scalability & elasticity

-Azure Cosmos DB transactional store uses horizontal partitioning to elastically scale the storage and throughput without any downtime. Horizontal partitioning in the transactional store provides scalability & elasticity in auto-sync to ensure data is synced to the analytical store in near real time. The data sync happens regardless of the transactional traffic throughput, whether it's 1000 operations/sec or 1 million operations/sec, and it doesn't impact the provisioned throughput in the transactional store.

-## <a id="analytical-schema"></a>Automatically handle schema updates

-Azure Cosmos DB transactional store is schema-agnostic, and it allows you to iterate on your transactional applications without having to deal with schema or index management. In contrast to this, Azure Cosmos DB analytical store is schematized to optimize for analytical query performance. With the auto-sync capability, Azure Cosmos DB manages the schema inference over the latest updates from the transactional store. It also manages the schema representation in the analytical store out-of-the-box which, includes handling nested data types.

-As your schema evolves, and new properties are added over time, the analytical store automatically presents a unionized schema across all historical schemas in the transactional store.

-> [!NOTE]

-> In the context of analytical store, we consider the following structures as property:

-> * JSON "elements" or "string-value pairs separated by a `:` ".

-> * JSON objects, delimited by `{` and `}`.

-> * JSON arrays, delimited by `[` and `]`.

-### Schema constraints

-The following constraints are applicable on the operational data in Azure Cosmos DB when you enable analytical store to automatically infer and represent the schema correctly:

-* You can have a maximum of 1000 properties across all nested levels in the document schema and a maximum nesting depth of 127.

- * Only the first 1000 properties are represented in the analytical store.

- * Only the first 127 nested levels are represented in the analytical store.

- * The first level of a JSON document is its `/` root level.

- * Properties in the first level of the document will be represented as columns.

-* Sample scenarios:

- * If your document's first level has 2000 properties, the sync process will represent the first 1000 of them.

- * If your documents have five levels with 200 properties in each one, the sync process will represent all properties.

- * If your documents have 10 levels with 400 properties in each one, the sync process will fully represent the two first levels and only half of the third level.

-* The hypothetical document below contains four properties and three levels.

- * The levels are `root`, `myArray`, and the nested structure within the `myArray`.

- * The properties are `id`, `myArray`, `myArray.nested1`, and `myArray.nested2`.

- * The analytical store representation will have two columns, `id`, and `myArray`. You can use Spark or T-SQL functions to also expose the nested structures as columns.

-```json

-{

- "id": "1",

- "myArray": [

- "string1",

- "string2",

- {

- "nested1": "abc",

- "nested2": "cde"

- }

- ]

-}

-```

-* While JSON documents (and Azure Cosmos DB collections/containers) are case-sensitive from the uniqueness perspective, analytical store isn't.

- * **In the same document:** Properties names in the same level should be unique when compared case-insensitively. For example, the following JSON document has "Name" and "name" in the same level. While it's a valid JSON document, it doesn't satisfy the uniqueness constraint and hence won't be fully represented in the analytical store. In this example, "Name" and "name" are the same when compared in a case-insensitive manner. Only `"Name": "fred"` will be represented in analytical store, because it's the first occurrence. And `"name": "john"` won't be represented at all.

-

-

- ```json

- {"id": 1, "Name": "fred", "name": "john"}

- ```

-

- * **In different documents:** Properties in the same level and with the same name, but in different cases, will be represented within the same column, using the name format of the first occurrence. For example, the following JSON documents have `"Name"` and `"name"` in the same level. Since the first document format is `"Name"`, this is what will be used to represent the property name in analytical store. In other words, the column name in analytical store will be `"Name"`. Both `"fred"` and `"john"` will be represented, in the `"Name"` column.

- ```json

- {"id": 1, "Name": "fred"}

- {"id": 2, "name": "john"}

- ```

-* The first document of the collection defines the initial analytical store schema.

- * Documents with more properties than the initial schema will generate new columns in analytical store.

- * Columns can't be removed.

- * The deletion of all documents in a collection doesn't reset the analytical store schema.

- * There's not schema versioning. The last version inferred from transactional store is what you'll see in analytical store.

-* Currently Azure Synapse Spark can't read properties that contain some special characters in their names, listed below. Azure Synapse SQL serverless isn't affected.

- * :

- * `

- * ,

- * ;

- * {}

- * ()

- * \n

- * \t

- * =

- * "

-> [!NOTE]

-> White spaces are also listed in the Spark error message returned when you reach this limitation. But we have added a special treatment for white spaces, please check out more details in the items below.

-

-* If you have properties names using the characters listed above, the alternatives are:

- * Change your data model in advance to avoid these characters.

- * Since currently we don't support schema reset, you can change your application to add a redundant property with a similar name, avoiding these characters.

- * Use Change Feed to create a materialized view of your container without these characters in properties names.

- * Use the `dropColumn` Spark option to ignore the affected columns and load all other columns into a DataFrame. The syntax is:

-```Python

-# Removing one column:

-df = spark.read\

- .format("cosmos.olap")\

- .option("spark.synapse.linkedService","<your-linked-service-name>")\

- .option("spark.synapse.container","<your-container-name>")\

- .option("spark.cosmos.dropColumn","FirstName,LastName")\

- .load()

-

-# Removing multiple columns:

-df = spark.read\

- .format("cosmos.olap")\

- .option("spark.synapse.linkedService","<your-linked-service-name>")\

- .option("spark.synapse.container","<your-container-name>")\

- .option("spark.cosmos.dropColumn","FirstName,LastName;StreetName,StreetNumber")\

- .option("spark.cosmos.dropMultiColumnSeparator", ";")\

- .load()

-```

-* Azure Synapse Spark now supports properties with white spaces in their names. For that, you need to use the `allowWhiteSpaceInFieldNames` Spark option to load the affected columns into a DataFrame, keeping the original name. The syntax is:

-```Python

-df = spark.read\

- .format("cosmos.olap")\

- .option("spark.synapse.linkedService","<your-linked-service-name>")\

- .option("spark.synapse.container","<your-container-name>")\

- .option("spark.cosmos.allowWhiteSpaceInFieldNames", "true")\

- .load()

-```

-* The following BSON datatypes aren't supported and won't be represented in analytical store:

- * Decimal128

- * Regular Expression

- * DB Pointer

- * JavaScript

- * Symbol

- * MinKey/MaxKey

-* When using DateTime strings that follow the ISO 8601 UTC standard, expect the following behavior:

- * Spark pools in Azure Synapse represent these columns as `string`.

- * SQL serverless pools in Azure Synapse represent these columns as `varchar(8000)`.

-* Properties with `UNIQUEIDENTIFIER (guid)` types are represented as `string` in analytical store and should be converted to `VARCHAR` in **SQL** or to `string` in **Spark** for correct visualization.

-* SQL serverless pools in Azure Synapse support result sets with up to 1000 columns, and exposing nested columns also counts towards that limit. It is a good practice to consider this information in your transactional data architecture and modeling.

-* If you rename a property, in one or many documents, it will be considered a new column. If you execute the same rename in all documents in the collection, all data will be migrated to the new column and the old column will be represented with `NULL` values.

-### Schema representation

-There are two methods of schema representation in the analytical store, valid for all containers in the database account. They have tradeoffs between the simplicity of query experience versus the convenience of a more inclusive columnar representation for polymorphic schemas.

-* Well-defined schema representation, default option for API for NoSQL and Gremlin accounts.

-* Full fidelity schema representation, default option for API for MongoDB accounts.

-#### Well-defined schema representation

-The well-defined schema representation creates a simple tabular representation of the schema-agnostic data in the transactional store. The well-defined schema representation has the following considerations:

-* The first document defines the base schema and properties must always have the same type across all documents. The only exceptions are:

- * From `NULL` to any other data type. The first non-null occurrence defines the column data type. Any document not following the first non-null datatype won't be represented in analytical store.

- * From `float` to `integer`. All documents are represented in analytical store.

- * From `integer` to `float`. All documents are represented in analytical store. However, to read this data with Azure Synapse SQL serverless pools, you must use a WITH clause to convert the column to `varchar`. And after this initial conversion, it's possible to convert it again to a number. Please check the example below, where **num** initial value was an integer and the second one was a float.

-```SQL

-SELECT CAST (num as float) as num

-FROM OPENROWSET(PROVIDER = 'CosmosDB',

- CONNECTION = '<your-connection',

- OBJECT = 'IntToFloat',

- SERVER_CREDENTIAL = 'your-credential'

-)

-WITH (num varchar(100)) AS [IntToFloat]

-```

- * Properties that don't follow the base schema data type won't be represented in analytical store. For example, consider the documents below: the first one defined the analytical store base schema. The second document, where `id` is `"2"`, **doesn't** have a well-defined schema since property `"code"` is a string and the first document has `"code"` as a number. In this case, the analytical store registers the data type of `"code"` as `integer` for lifetime of the container. The second document will still be included in analytical store, but its `"code"` property won't.

-

- * `{"id": "1", "code":123}`

- * `{"id": "2", "code": "123"}`

-

- > [!NOTE]

- > The condition above doesn't apply for `NULL` properties. For example, `{"a":123} and {"a":NULL}` is still well-defined.

-> [!NOTE]

- > The condition above doesn't change if you update `"code"` of document `"1"` to a string in your transactional store. In analytical store, `"code"` will be kept as `integer` since currently we don't support schema reset.

-* Array types must contain a single repeated type. For example, `{"a": ["str",12]}` isn't a well-defined schema because the array contains a mix of integer and string types.

-> [!NOTE]

-> If the Azure Cosmos DB analytical store follows the well-defined schema representation and the specification above is violated by certain items, those items won't be included in the analytical store.

-* Expect different behavior in regard to different types in well-defined schema:

- * Spark pools in Azure Synapse represent these values as `undefined`.

- * SQL serverless pools in Azure Synapse represent these values as `NULL`.

-* Expect different behavior in regard to explicit `NULL` values:

- * Spark pools in Azure Synapse read these values as `0` (zero), and as `undefined` as soon as the column has a non-null value.

- * SQL serverless pools in Azure Synapse read these values as `NULL`.

-

-* Expect different behavior in regard to missing columns:

- * Spark pools in Azure Synapse represent these columns as `undefined`.

- * SQL serverless pools in Azure Synapse represent these columns as `NULL`.

-##### Representation challenges workarounds

-It is possible that an old document, with an incorrect schema, was used to create your container's analytical store base schema. Based on all the rules presented above, you may be receiving `NULL` for certain properties when querying your analytical store using Azure Synapse Link. To delete or update the problematic documents won't help because base schema reset isn't currently supported. The possible solutions are:

- * To migrate the data to a new container, making sure that all documents have the correct schema.

- * To abandon the property with the wrong schema and add a new one with another name that has the correct schema in all documents. Example: You have billions of documents in the **Orders** container where the **status** property is a string. But the first document in that container has **status** defined with integer. So, one document will have **status** correctly represented and all other documents will have `NULL`. You can add the **status2** property to all documents and start to use it, instead of the original property.

-#### Full fidelity schema representation

-The full fidelity schema representation is designed to handle the full breadth of polymorphic schemas in the schema-agnostic operational data. In this schema representation, no items are dropped from the analytical store even if the well-defined schema constraints (that is no mixed data type fields nor mixed data type arrays) are violated.

-This is achieved by translating the leaf properties of the operational data into the analytical store as JSON `key-value` pairs, where the datatype is the `key` and the property content is the `value`. This JSON object representation allows queries without ambiguity, and you can individually analyze each datatype.

-In other words, in the full fidelity schema representation, each datatype of each property of each document will generate a `key-value`pair in a JSON object for that property. Each of them count as one of the 1000 maximum properties limit.

-For example, let's take the following sample document in the transactional store:

-```json

-{

-name: "John Doe",

-age: 32,

-profession: "Doctor",

-address: {

- streetNo: 15850,

- streetName: "NE 40th St.",

- zip: 98052

-},

-salary: 1000000

-}

-```

-The nested object `address` is a property in the root level of the document and will be represented as a column. Each leaf property in the `address` object will be represented as a JSON object: `{"object":{"streetNo":{"int32":15850},"streetName":{"string":"NE 40th St."},"zip":{"int32":98052}}}`.

-Unlike the well-defined schema representation, the full fidelity method allows variation in datatypes. If the next document in this collection of the example above has `streetNo` as a string, it will be represented in analytical store as `"streetNo":{"string":15850}`. In well-defined schema method, it wouldn't be represented.

-##### Datatypes map for full fidelity schema

-Here's a map of MongoDB data types and their representations in the analytical store in full fidelity schema representation. The map below isn't valid for NoSQL API accounts.

-|Original data type |Suffix |Example |

-||||

-| Double | ".float64" | 24.99|

-| Array | ".array" | ["a", "b"]|

-| Binary | ".binary" |0|

-| Boolean | ".bool" |True|

-| Int32 | ".int32" |123|

-| Int64 | ".int64" |255486129307|

-| NULL | ".NULL" | NULL|

-| String| ".string" | "ABC"|

-| Timestamp | ".timestamp" | Timestamp(0, 0)|

-| ObjectId |".objectId" | ObjectId("5f3f7b59330ec25c132623a2")|

-| Document |".object" | {"a": "a"}|

-* Expect different behavior in regard to explicit `NULL` values:

- * Spark pools in Azure Synapse will read these values as `0` (zero).

- * SQL serverless pools in Azure Synapse will read these values as `NULL`.

-

-* Expect different behavior in regard to missing columns:

- * Spark pools in Azure Synapse will represent these columns as `undefined`.

- * SQL serverless pools in Azure Synapse will represent these columns as `NULL`.

-* Expect different behavior in regard to `timestamp` values:

- * Spark pools in Azure Synapse will read these values as `TimestampType`, `DateType`, or `Float`. It depends on the range and how the timestamp was generated.

- * SQL Serverless pools in Azure Synapse will read these values as `DATETIME2`, ranging from `0001-01-01` through `9999-12-31`. Values beyond this range are not supported and will cause an execution failure for your queries. If this is your case, you can:

- * Remove the column from the query. To keep the representation, you can create a new property mirroring that column but within the supported range. And use it in your queries.

- * Use [Change Data Capture from analytical store](analytical-store-change-data-capture.md), at no RUs cost, to transform and load the data into a new format, within one of the supported sinks.

-

-##### Using full fidelity schema with Spark

-Spark will manage each datatype as a column when loading into a `DataFrame`. Let's assume a collection with the documents below.

-```json

-{

- "_id" : "1" ,

- "item" : "Pizza",

- "price" : 3.49,

- "rating" : 3,

- "timestamp" : 1604021952.6790195

-},

-{

- "_id" : "2" ,

- "item" : "Ice Cream",

- "price" : 1.59,

- "rating" : "4" ,

- "timestamp" : "2022-11-11 10:00 AM"

-}

-```

-While the first document has `rating` as a number and `timestamp` in utc format, the second document has `rating` and `timestamp` as strings. Assuming that this collection was loaded into `DataFrame` without any data transformation, the output of the `df.printSchema()` is:

-```JSON

-root

- |-- _rid: string (nullable = true)

- |-- _ts: long (nullable = true)

- |-- id: string (nullable = true)

- |-- _etag: string (nullable = true)

- |-- _id: struct (nullable = true)

- | |-- objectId: string (nullable = true)

- |-- item: struct (nullable = true)

- | |-- string: string (nullable = true)

- |-- price: struct (nullable = true)

- | |-- float64: double (nullable = true)

- |-- rating: struct (nullable = true)

- | |-- int32: integer (nullable = true)

- | |-- string: string (nullable = true)

- |-- timestamp: struct (nullable = true)

- | |-- float64: double (nullable = true)

- | |-- string: string (nullable = true)

- |-- _partitionKey: struct (nullable = true)

- | |-- string: string (nullable = true)

- ```

-In well-defined schema representation, both `rating` and `timestamp` of the second document wouldn't be represented. In full fidelity schema, you can use the following examples to individually access to each value of each datatype.

-In the example below, we can use `PySpark` to run an aggregation:

-```PySpark

-df.groupBy(df.item.string).sum().show()

-```

-In the example below, we can use `PySQL` to run another aggregation:

-```PySQL

-df.createOrReplaceTempView("Pizza")

-sql_results = spark.sql("SELECT sum(price.float64),count(*) FROM Pizza where timestamp.string is not null and item.string = 'Pizza'")

-sql_results.show()

-```

-##### Using full fidelity schema with SQL

-You can use the following syntax example, with the same documents of the Spark example above:

-```SQL

-SELECT rating,timestamp_string,timestamp_utc

-FROM OPENROWSET(PROVIDER = 'CosmosDB',

- CONNECTION = 'Account=<your-database-account-name';Database=<your-database-name>',

- OBJECT = '<your-collection-name>',

- SERVER_CREDENTIAL = '<your-synapse-sql-server-credential-name>')

-WITH (

-rating integer '$.rating.int32',

-timestamp varchar(50) '$.timestamp.string',

-timestamp_utc float '$.timestamp.float64'

-) as HTAP

-WHERE timestamp is not null or timestamp_utc is not null

-```

-You can implement transformations using `cast`, `convert` or any other T-SQL function to manipulate your data. You can also hide complex datatype structures by using views.

-```SQL

-create view MyView as

-SELECT MyRating=rating,MyTimestamp = convert(varchar(50),timestamp_utc)

-FROM OPENROWSET(PROVIDER = 'CosmosDB',

- CONNECTION = 'Account=<your-database-account-name';Database=<your-database-name>',

- OBJECT = '<your-collection-name>',

- SERVER_CREDENTIAL = '<your-synapse-sql-server-credential-name>')

-WITH (

-rating integer '$.rating.int32',

-timestamp_utc float '$.timestamp.float64'

-) as HTAP

-WHERE timestamp_utc is not null

-union all

-SELECT MyRating=convert(integer,rating_string),MyTimestamp = timestamp_string

-FROM OPENROWSET(PROVIDER = 'CosmosDB',

- CONNECTION = 'Account=<your-database-account-name';Database=<your-database-name>',

- OBJECT = '<your-collection-name>',

- SERVER_CREDENTIAL = '<your-synapse-sql-server-credential-name>')

-WITH (

-rating_string varchar(50) '$.rating.string',

-timestamp_string varchar(50) '$.timestamp.string'

-) as HTAP

-WHERE timestamp_string is not null

-```

-##### Working with MongoDB `_id` field

-MongoDB `_id` field is fundamental to every collection in MongoDB and originally has a hexadecimal representation. As you can see in the table above, full fidelity schema will preserve its characteristics, creating a challenge for its visualization in Azure Synapse Analytics. For correct visualization, you must convert the `_id` datatype as below:

-###### Working with MongoDB `_id` field in Spark

-The example below works on Spark 2.x and 3.x versions:

-```Scala

-val df = spark.read.format("cosmos.olap").option("spark.synapse.linkedService", "xxxx").option("spark.cosmos.container", "xxxx").load()

-val convertObjectId = udf((bytes: Array[Byte]) => {

- val builder = new StringBuilder

- for (b <- bytes) {

- builder.append(String.format("%02x", Byte.box(b)))

- }

- builder.toString

-}

- )

-val dfConverted = df.withColumn("objectId", col("_id.objectId")).withColumn("convertedObjectId", convertObjectId(col("_id.objectId"))).select("id", "objectId", "convertedObjectId")

-display(dfConverted)

-```

-###### Working with MongoDB `_id` field in SQL

-```SQL

-SELECT TOP 100 id=CAST(_id as VARBINARY(1000))

-FROM OPENROWSET('CosmosDB',

-                'Your-account;Database=your-database;Key=your-key',

-                HTAP) WITH (_id VARCHAR(1000)) as HTAP

-```

-##### Working with MongoDB `id` field

-The `id` property in MongoDB containers is automatically overridden with the Base64 representation of the "_id" property both in analytical store. The "id" field is intended for internal use by MongoDB applications. Currently, the only workaround is to rename the "id" property to something other than "id".

-#### Full fidelity schema for API for NoSQL or Gremlin accounts

-It's possible to use full fidelity Schema for API for NoSQL accounts, instead of the default option, by setting the schema type when enabling Synapse Link on an Azure Cosmos DB account for the first time. Here are the considerations about changing the default schema representation type:

-* Currently, if you enable Synapse Link in your NoSQL API account using the Azure portal, it will be enabled as well-defined schema.

-* Currently, if you want to use full fidelity schema with NoSQL or Gremlin API accounts, you have to set it at account level in the same CLI or PowerShell command that will enable Synapse Link at account level.

-* Currently Azure Cosmos DB for MongoDB isn't compatible with this possibility of changing the schema representation. All MongoDB accounts have full fidelity schema representation type.

-* Full Fidelity schema data types map mentioned above isn't valid for NoSQL API accounts that use JSON datatypes. As an example, `float` and `integer` values are represented as `num` in analytical store.

-* It's not possible to reset the schema representation type, from well-defined to full fidelity or vice-versa.

-* Currently, containers schemas in analytical store are defined when the container is created, even if Synapse Link has not been enabled in the database account.

- * Containers or graphs created before Synapse Link was enabled with full fidelity schema at account level will have well-defined schema.

- * Containers or graphs created after Synapse Link was enabled with full fidelity schema at account level will have full fidelity schema.

-

-The schema representation type decision must be made at the same time that Synapse Link is enabled on the account, using Azure CLI or PowerShell.

-

- With the Azure CLI:

- ```cli

- az cosmosdb create --name MyCosmosDBDatabaseAccount --resource-group MyResourceGroup --subscription MySubscription --analytical-storage-schema-type "FullFidelity" --enable-analytical-storage true

- ```

-

-> [!NOTE]

-> In the command above, replace `create` with `update` for existing accounts.

-

- With the PowerShell:

- ```PowerShell

- New-AzCosmosDBAccount -ResourceGroupName MyResourceGroup -Name MyCosmosDBDatabaseAccount -EnableAnalyticalStorage true -AnalyticalStorageSchemaType "FullFidelity"

- ```

-

-> [!NOTE]

-> In the command above, replace `New-AzCosmosDBAccount` with `Update-AzCosmosDBAccount` for existing accounts.

->

-## <a id="analytical-ttl"></a> Analytical Time-to-Live (TTL)

-Analytical TTL (ATTL) indicates how long data should be retained in your analytical store, for a container.

-Analytical store is enabled when ATTL is set with a value other than `NULL` and `0`. When enabled, inserts, updates, deletes to operational data are automatically synced from transactional store to analytical store, irrespective of the transactional TTL (TTTL) configuration. The retention of this transactional data in analytical store can be controlled at container level by the `AnalyticalStoreTimeToLiveInSeconds` property.

-The possible ATTL configurations are:

-* If the value is set to `0` or set to `NULL`: the analytical store is disabled and no data is replicated from transactional store to analytical store

-* If the value is set to `-1`: the analytical store retains all historical data, irrespective of the retention of the data in the transactional store. This setting indicates that the analytical store has infinite retention of your operational data

-* If the value is set to any positive integer `n` number: items will expire from the analytical store `n` seconds after their last modified time in the transactional store. This setting can be leveraged if you want to retain your operational data for a limited period of time in the analytical store, irrespective of the retention of the data in the transactional store

-Some points to consider:

-* After the analytical store is enabled with an ATTL value, it can be updated to a different valid value later.

-* While TTTL can be set at the container or item level, ATTL can only be set at the container level currently.

-* You can achieve longer retention of your operational data in the analytical store by setting ATTL >= TTTL at the container level.

-* The analytical store can be made to mirror the transactional store by setting ATTL = TTTL.

-* If you have ATTL bigger than TTTL, at some point in time you'll have data that only exists in analytical store. This data is read only.

-* Currently we don't delete any data from analytical store. If you set your ATTL to any positive integer, the data won't be included in your queries and you won't be billed for it. But if you change ATTL back to `-1`, all the data will show up again, you will start to be billed for all the data volume.

-How to enable analytical store on a container:

-* From the Azure portal, the ATTL option, when turned on, is set to the default value of -1. You can change this value to 'n' seconds, by navigating to container settings under Data Explorer.

-

-* From the Azure Management SDK, Azure Cosmos DB SDKs, PowerShell, or Azure CLI, the ATTL option can be enabled by setting it to either -1 or 'n' seconds.

-To learn more, see [how to configure analytical TTL on a container](configure-synapse-link.md).

-## Cost-effective analytics on historical data

-Data tiering refers to the separation of data between storage infrastructures optimized for different scenarios. Thereby improving the overall performance and cost-effectiveness of the end-to-end data stack. With analytical store, Azure Cosmos DB now supports automatic tiering of data from the transactional store to analytical store with different data layouts. With analytical store optimized in terms of storage cost compared to the transactional store, allows you to retain much longer horizons of operational data for historical analysis.

-After the analytical store is enabled, based on the data retention needs of the transactional workloads, you can configure `transactional TTL` property to have records automatically deleted from the transactional store after a certain time period. Similarly, the `analytical TTL` allows you to manage the lifecycle of data retained in the analytical store, independent from the transactional store. By enabling analytical store and configuring transactional and analytical `TTL` properties, you can seamlessly tier and define the data retention period for the two stores.

-> [!NOTE]

-> When `analytical TTL` is set to a value larger than `transactional TTL` value, your container will have data that only exists in analytical store. This data is read only and currently we don't support document level `TTL` in analytical store. If your container data may need an update or a delete at some point in time in the future, don't use `analytical TTL` bigger than `transactional TTL`. This capability is recommended for data that won't need updates or deletes in the future.

-> [!NOTE]

-> If your scenario doesn't demand physical deletes, you can adopt a logical delete/update approach. Insert in transactional store another version of the same document that only exists in analytical store but needs a logical delete/update. Maybe with a flag indicating that it's a delete or an update of an expired document. Both versions of the same document will co-exist in analytical store, and your application should only consider the last one.

-## Resilience

-Analytical store relies on Azure Storage and offers the following protection against physical failure:

- * By default, Azure Cosmos DB database accounts allocate analytical store in Locally Redundant Storage (LRS) accounts. LRS provides at least 99.999999999% (11 nines) durability of objects over a given year.

- * If any geo-region of the database account is configured for zone-redundancy, it is allocated in Zone-redundant Storage (ZRS) accounts. You need to enable Availability Zones on a region of their Azure Cosmos DB database account to have analytical data of that region stored in Zone-redundant Storage. ZRS offers durability for storage resources of at least 99.9999999999% (12 9's) over a given year.

-For more information about Azure Storage durability, see [this link.](/azure/storage/common/storage-redundancy)

-## Backup

-Although analytical store has built-in protection against physical failures, backup can be necessary for accidental deletes or updates in transactional store. In those cases, you can restore a container and use the restored container to backfill the data in the original container, or fully rebuild analytical store if necessary.

-> [!NOTE]

-> Currently analytical store isn't backed up, therefore it can't be restored. Your backup policy can't be planned relying on that.

-Synapse Link, and analytical store by consequence, has different compatibility levels with Azure Cosmos DB backup modes:

-* Periodic backup mode is fully compatible with Synapse Link and these 2 features can be used in the same database account.

-* Synapse Link for database accounts using continuous backup mode is GA.

-* Continuous backup mode for Synapse Link enabled accounts is in public preview. Currently, you can't migrate to continuous backup if you disabled Synapse Link on any of your collections in a Cosmos DB account.

-### Backup policies

-There are two possible backup policies and to understand how to use them, the following details about Azure Cosmos DB backups are very important:

- * The original container is restored without analytical store in both backup modes.

- * Azure Cosmos DB doesn't support containers overwrite from a restore.

-Now let's see how to use backup and restores from the analytical store perspective.

- #### Restoring a container with TTTL >= ATTL

-

- When `transactional TTL` is equal or bigger than `analytical TTL`, all data in analytical store still exists in transactional store. In case of a restore, you have two possible situations:

- * To use the restored container as a replacement for the original container. To rebuild analytical store, just enable Synapse Link at account level and container level.

- * To use the restored container as a data source to backfill or update the data in the original container. In this case, analytical store will automatically reflect the data operations.

-

- #### Restoring a container with TTTL < ATTL

-

-When `transactional TTL` is smaller than `analytical TTL`, some data only exists in analytical store and won't be in the restored container. Again, you have two possible situations:

- * To use the restored container as a replacement for the original container. In this case, when you enable Synapse Link at container level, only the data that was in transactional store will be included in the new analytical store. But please note that the analytical store of the original container remains available for queries as long as the original container exists. You may want to change your application to query both.

- * To use the restored container as a data source to backfill or update the data in the original container:

- * Analytical store will automatically reflect the data operations for the data that is in transactional store.

- * If you re-insert data that was previously removed from transactional store due to `transactional TTL`, this data will be duplicated in analytical store.

-Example:

- * Container `OnlineOrders` has TTTL set to one month and ATTL set for one year.

- * When you restore it to `OnlineOrdersNew` and turn on analytical store to rebuild it, there will be only one month of data in both transactional and analytical store.

- * Original container `OnlineOrders` isn't deleted and its analytical store is still available.

- * New data is only ingested into `OnlineOrdersNew`.

- * Analytical queries will do a UNION ALL from analytical stores while the original data is still relevant.

-If you want to delete the original container but don't want to lose its analytical store data, you can persist the analytical store of the original container in another Azure data service. Synapse Analytics has the capability to perform joins between data stored in different locations. An example: A Synapse Analytics query joins analytical store data with external tables located in Azure Blob Storage, Azure Data Lake Store, etc.

-It's important to note that the data in the analytical store has a different schema than what exists in the transactional store. While you can generate snapshots of your analytical store data, and export it to any Azure Data service, at no RUs costs, we can't guarantee the use of this snapshot to back feed the transactional store. This process isn't supported.

-## Global distribution

-If you have a globally distributed Azure Cosmos DB account, after you enable analytical store for a container, it will be available in all regions of that account. Any changes to operational data are globally replicated in all regions. You can run analytical queries effectively against the nearest regional copy of your data in Azure Cosmos DB.

-## Partitioning

-Analytical store partitioning is completely independent of partitioning in the transactional store. By default, data in analytical store isn't partitioned. If your analytical queries have frequently used filters, you have the option to partition based on these fields for better query performance. To learn more, see [introduction to custom partitioning](custom-partitioning-analytical-store.md) and [how to configure custom partitioning](configure-custom-partitioning.md).

-## Security

-* **Authentication with the analytical store** is the same as the transactional store for a given database. You can use primary, secondary, or read-only keys for authentication. You can leverage linked service in Synapse Studio to prevent pasting the Azure Cosmos DB keys in the Spark notebooks. For Azure Synapse SQL serverless, you can use SQL credentials to also prevent pasting the Azure Cosmos DB keys in the SQL notebooks. The Access to these Linked Services or to these SQL credentials are available to anyone who has access to the workspace. Please note that the Azure Cosmos DB read only key can also be used.

-* **Network isolation using private endpoints** - You can control network access to the data in the transactional and analytical stores independently. Network isolation is done using separate managed private endpoints for each store, within managed virtual networks in Azure Synapse workspaces. To learn more, see how to [Configure private endpoints for analytical store](analytical-store-private-endpoints.md) article.

-* **Data encryption at rest** - Your analytical store encryption is enabled by default.

-* **Data encryption with customer-managed keys** - You can seamlessly encrypt the data across transactional and analytical stores using the same customer-managed keys in an automatic and transparent manner. Azure Synapse Link only supports configuring customer-managed keys using your Azure Cosmos DB account's managed identity. You must configure your account's managed identity in your Azure Key Vault access policy before [enabling Azure Synapse Link](configure-synapse-link.md#enable-synapse-link) on your account. To learn more, see how to [Configure customer-managed keys using Azure Cosmos DB accounts' managed identities](how-to-setup-cmk.md#using-managed-identity) article.

-> [!NOTE]

-> If you change your database account from First Party to System or User Assigned Identy, and enable Azure Synapse Link in your database account, you won't be able to return to First Party identity since you can't disable Synapse Link from your database account.

-## Support for multiple Azure Synapse Analytics runtimes

-The analytical store is optimized to provide scalability, elasticity, and performance for analytical workloads without any dependency on the compute run-times. The storage technology is self-managed to optimize your analytics workloads without manual efforts.

-Data in Azure Cosmos DB analytical store can be queried simultaneously from the different analytics runtimes supported by Azure Synapse Analytics. Azure Synapse Analytics supports Apache Spark and serverless SQL pool with Azure Cosmos DB analytical store.

-> [!NOTE]

-> You can only read from analytical store using Azure Synapse Analytics runtimes. And the opposite is also true, Azure Synapse Analytics runtimes can only read from analytical store. Only the auto-sync process can change data in analytical store. You can write data back to Azure Cosmos DB transactional store using Azure Synapse Analytics Spark pool, using the built-in Azure Cosmos DB OLTP SDK.

-## <a id="analytical-store-pricing"></a> Pricing

-Analytical store follows a consumption-based pricing model where you're charged for:

-* Storage: the volume of the data retained in the analytical store every month including historical data as defined by analytical TTL.

-* Analytical write operations: the fully managed synchronization of operational data updates to the analytical store from the transactional store (auto-sync)

-* Analytical read operations: the read operations performed against the analytical store from Azure Synapse Analytics Spark pool and serverless SQL pool run times.

-Analytical store pricing is separate from the transaction store pricing model. There's no concept of provisioned RUs in the analytical store. See [Azure Cosmos DB pricing page](https://azure.microsoft.com/pricing/details/cosmos-db/) for full details on the pricing model for analytical store.

-Data in the analytics store can only be accessed through Azure Synapse Link, which is done in the Azure Synapse Analytics runtimes: Azure Synapse Apache Spark pools and Azure Synapse serverless SQL pools. See [Azure Synapse Analytics pricing page](https://azure.microsoft.com/pricing/details/synapse-analytics/) for full details on the pricing model to access data in analytical store.

-In order to get a high-level cost estimate to enable analytical store on an Azure Cosmos DB container, from the analytical store perspective, you can use the [Azure Cosmos DB Capacity planner](https://cosmos.azure.com/capacitycalculator/) and get an estimate of your analytical storage and write operations costs.

-Analytical store read operations estimates aren't included in the Azure Cosmos DB cost calculator since they are a function of your analytical workload. But as a high-level estimate, scan of 1 TB of data in analytical store typically results in 130,000 analytical read operations, and results in a cost of $0.065. As an example, if you use Azure Synapse serverless SQL pools to perform this scan of 1 TB, it will cost $5.00 according to [Azure Synapse Analytics pricing page](https://azure.microsoft.com/pricing/details/synapse-analytics/). The final total cost for this 1 TB scan would be $5.065.

-While the above estimate is for scanning 1TB of data in analytical store, applying filters reduces the volume of data scanned and this determines the exact number of analytical read operations given the consumption pricing model. A proof-of-concept around the analytical workload would provide a finer estimate of analytical read operations. This estimate doesn't include the cost of Azure Synapse Analytics.

-## Next steps

-To learn more, see the following docs:

-* [Azure Synapse Link for Azure Cosmos DB](synapse-link.md)

-* Check out the training module on how to [Design hybrid transactional and analytical processing using Azure Synapse Analytics](/training/modules/design-hybrid-transactional-analytical-processing-using-azure-synapse-analytics/)

-* [Get started with Azure Synapse Link for Azure Cosmos DB](configure-synapse-link.md)

-* [Frequently asked questions about Synapse Link for Azure Cosmos DB](synapse-link-frequently-asked-questions.yml)

-* [Azure Synapse Link for Azure Cosmos DB Use cases](synapse-link-use-cases.md)

-description: Learn how to set up managed private endpoints for Azure Cosmos DB analytical store to restrict network access.

-# Configure Azure Private Link for Azure Cosmos DB analytical store

-In this article, you will learn how to set up managed private endpoints for Azure Cosmos DB analytical store. If you are using the transactional store, see [Private endpoints for the transactional store](how-to-configure-private-endpoints.md) article. Using [managed private endpoints](../synapse-analytics/security/synapse-workspace-managed-private-endpoints.md), you can restrict network access of your Azure Cosmos DB analytical store, to a Managed Virtual Network associated with your Azure Synapse workspace. Managed private endpoints establish a private link to your analytical store.

-> [!NOTE]

-> If you are using Private DNS Zones for Azure Cosmos DB and wish to create a Synapse managed private endpoint to the analytical store sub-resource, you must first create a DNS zone for the analytical store (`privatelink.analytics.cosmos.azure.com`) linked to your Azure Cosmos DB's virtual network.

-## Enable a private endpoint for the analytical store

-### Set up Azure Synapse Analytics workspace with a managed virtual network and data-exfiltration

-[Create a workspace in Azure Synapse Analytics with data-exfiltration enabled.](../synapse-analytics/security/how-to-create-a-workspace-with-data-exfiltration-protection.md) With [data-exfiltration protection](../synapse-analytics/security/workspace-data-exfiltration-protection.md), you can ensure that malicious users cannot copy or transfer data from your Azure resources to locations outside your organizationΓÇÖs scope.

-The following access restrictions are applicable when data-exfiltration protection is turned on for an Azure Synapse Analytics workspace:

-* If you are using Azure Spark for Azure Synapse Analytics, access is only allowed to the approved managed private endpoints for Azure Cosmos DB analytical store.

-* If you are using Synapse serverless SQL pools, you can query any Azure Cosmos DB account using Azure Synapse Link. However, write requests that [create external tables as select (CETAS)](../synapse-analytics/sql/develop-tables-cetas.md) are only allowed to the approved manage private endpoints in the workspace virtual network.

-> [!NOTE]

-> You cannot change managed virtual network and data-exfiltration configuration after the workspace is created.

-### Add a managed private endpoint for Azure Cosmos DB analytical store

-1. Sign in to the [Azure portal](https://portal.azure.com).

-1. From the Azure portal, navigate to your Synapse Analytics workspace and open the **Overview** pane.

-1. Launch Synapse Studio by navigating to **Getting Started** pane and select **Open** under **Open Synapse Studio**.

-1. In the Synapse Studio, open the **Manage** tab.

-1. Navigate to **Managed private endpoints** and select **New**

- :::image type="content" source="./media/analytical-store-private-endpoints/create-new-private-endpoint.png" alt-text="Create a new private endpoint for analytical store." border="true":::

-1. Select **Azure Cosmos DB (API for NoSQL)** account type > **Continue**.

- :::image type="content" source="./media/analytical-store-private-endpoints/select-private-endpoint.png" alt-text="Select Azure Cosmos DB API for NoSQL to create a private endpoint." border="true":::

-1. Fill out the **New managed private endpoint** form with the following details:

- * **Name** - Name for your managed private endpoint. This name cannot be updated after it's created.

- * **Description** - Provide a friendly description to identify your private endpoint.

- * **Azure subscription** - Select an Azure Cosmos DB account from the list of available accounts in your Azure subscriptions.

- * **Azure Cosmos DB account name** - Select an existing Azure Cosmos DB account of type SQL or MongoDB.

- * **Target sub-resource** - Select one of the following options:

- **Analytical**: If you want to add the private endpoint for Azure Cosmos DB analytical store.

- **NoSQL** (or **MongoDB**): If you want to add OLTP or transactional account endpoint.

- > [!NOTE]

- > You can add both transactional store and analytical store private endpoints to the same Azure Cosmos DB account in an Azure Synapse Analytics workspace. If you only want to run analytical queries, you may only want to map the analytical private endpoint.

- :::image type="content" source="./media/analytical-store-private-endpoints/choose-analytical-private-endpoint.png" alt-text="Choose analytical for the target subresource." border="true":::

-1. After creating, go to the private endpoint name and select **Manage approvals in Azure portal**.

-1. Navigate to your Azure Cosmos DB account, select the private endpoint, and select **Approve**.

-1. Navigate back to Synapse Analytics workspace and click **Refresh** on the **Managed private endpoints** pane. Verify that private endpoint is in **Approved** state.

- :::image type="content" source="./media/analytical-store-private-endpoints/approved-private-endpoint.png" alt-text="Verify that the private endpoint is approved." border="true":::

-## Use Apache Spark for Azure Synapse Analytics

-If you created an Azure Synapse workspace with data-exfiltration protection turned on, the outbound access from Synapse Spark to Azure Cosmos DB accounts will be blocked, by default. Also, if the Azure Cosmos DB already has an existing private endpoint, Synapse Spark will be blocked from accessing it.

-To allow access to Azure Cosmos DB data:

-* If you are using Azure Synapse Link to query Azure Cosmos DB data, add a managed **analytical** private endpoint for the Azure Cosmos DB account.

-* If you are using batch writes/reads and/or streaming writes/reads to transactional store, add a managed *SQL* or *MongoDB* private endpoint for the Azure Cosmos DB account. In addition, you should also set *connectionMode* to *Gateway* as shown in the following code snippet:

- ```python

- # Write a Spark DataFrame into an Azure Cosmos DB container

- # To select a preferred lis of regions in a multi-region account, add .option("spark.cosmos.preferredRegions", "<Region1>, <Region2>")

-

- YOURDATAFRAME.write\

- .format("cosmos.oltp")\

- .option("spark.synapse.linkedService", "<your-Cosmos-DB-linked-service-name>")\

- .option("spark.cosmos.container","<your-Cosmos-DB-container-name>")\

- .option("spark.cosmos.write.upsertEnabled", "true")\

- .option("spark.cosmos.connection.mode", "Gateway")\

- .mode('append')\

- .save()

-

- ```

-## Using Synapse serverless SQL pools

-Synapse serverless SQL pools use multi-tenant capabilities that are not deployed into managed virtual network. If the Azure Cosmos DB account has an existing private endpoint, Synapse serverless SQL pool will be blocked from accessing the account, due to network isolation checks on the Azure Cosmos DB account.

-To configure network isolation for this account from a Synapse workspace:

-1. Allow the Synapse workspace to access the Azure Cosmos DB account by specifying `NetworkAclBypassResourceId` setting on the account.

- **Using PowerShell**

- ```powershell-interactive

- Update-AzCosmosDBAccount -Name MyCosmosDBDatabaseAccount -ResourceGroupName MyResourceGroup -NetworkAclBypass AzureServices -NetworkAclBypassResourceId "/subscriptions/subId/resourceGroups/rgName/providers/Microsoft.Synapse/workspaces/wsName"

- ```

- **Using Azure CLI**

- ```azurecli-interactive

- az cosmosdb update --name MyCosmosDBDatabaseAccount --resource-group MyResourceGroup --network-acl-bypass AzureServices --network-acl-bypass-resource-ids "/subscriptions/subId/resourceGroups/rgName/providers/Microsoft.Synapse/workspaces/wsName"

- ```

- > [!NOTE]

- > Azure Cosmos DB account and Azure Synapse Analytics workspace should be under same Microsoft Entra tenant.

-2. You can now access the account from serverless SQL pools, using T-SQL queries over Azure Synapse Link. However, to ensure network isolation for the data in analytical store, you must add an **analytical** managed private endpoint for this account. Otherwise, the data in the analytical store will not be blocked from public access.

-> [!IMPORTANT]

-> If you are using Azure Synapse Link and need network isolation for your data in analytical store, you must map the Azure Cosmos DB account into Synapse workspace using **Analytical** managed private endpoint.

-## Next steps

-* Get started with [querying analytical store with Azure Synapse Spark 3](../synapse-analytics/synapse-link/how-to-query-analytical-store-spark-3.md?toc=/azure/cosmos-db/toc.json&bc=/azure/cosmos-db/breadcrumb/toc.json)

-* Get started with [querying analytical store with Azure Synapse Spark 2](../synapse-analytics/synapse-link/how-to-query-analytical-store-spark.md?toc=/azure/cosmos-db/toc.json&bc=/azure/cosmos-db/breadcrumb/toc.json)

-* Get started with [querying analytical store with Azure Synapse serverless SQL pools](../synapse-analytics/sql/query-cosmos-db-analytical-store.md?toc=/azure/cosmos-db/toc.json&bc=/azure/cosmos-db/breadcrumb/toc.json)

-description: Review Azure Cosmos DB options to enable large-scale analytics and BI reporting on your operational data.

-# Analytics and Business Intelligence (BI) on your Azure Cosmos DB data

-Azure Cosmos DB offers various options to enable large-scale analytics and BI reporting on your operational data.

-To get meaningful insights on your Azure Cosmos DB data, you may need to query across multiple partitions, collections, or databases. In some cases, you might combine this data with other data sources in your organization such as Azure SQL Database, Azure Data Lake Storage Gen2 etc. You might also query with aggregate functions such as sum, count etc. Such queries need heavy computational power, which likely consumes more request units (RUs) and as a result, these queries might potentially affect your mission critical workload performance.

-To isolate transactional workloads from the performance impact of complex analytical queries, database data is ingested nightly to a central location using complex Extract-Transform-Load (ETL) pipelines. Such ETL-based analytics are complex, costly with delayed insights on business data.

-Azure Cosmos DB addresses these challenges by providing zero ETL, cost-effective analytics offerings.

-## Zero ETL, near real-time analytics on Azure Cosmos DB

-Azure Cosmos DB offers zero ETL, near real-time analytics on your data without affecting the performance of your transactional workloads or request units (RUs). These offerings remove the need for complex ETL pipelines, making your Azure Cosmos DB data seamlessly available to analytics engines. With reduced latency to insights, you can provide enhanced customer experience and react more quickly to changes in market conditions or business environment. Here are some sample [scenarios](synapse-link-use-cases.md) you can achieve with quick insights into your data.

-

- You can enable zero-ETL analytics and BI reporting on Azure Cosmos DB using the following options:

-* Mirroring your data into Microsoft Fabric

-* Enabling Azure Synapse Link to access data from Azure Synapse Analytics

-

-### Option 1: Mirroring your Azure Cosmos DB data into Microsoft Fabric

-Mirroring enables you to seamlessly bring your Azure Cosmos DB database data into Microsoft Fabric. With zero ETL, you can get quick, rich business insights on your Azure Cosmos DB data using FabricΓÇÖs built-in analytics, BI, and AI capabilities.

-Your Cosmos DB operational data is incrementally replicated into Fabric OneLake in near real-time. Data in OneLake is stored in open-source Delta Parquet format and made available to all analytical engines in Fabric. With open access, you can use it with various Azure services such as Azure Databricks, Azure HDInsight, and more. OneLake also helps unify your data estate for your analytical needs. Mirrored data can be joined with any other data in OneLake, such as Lakehouses, Warehouses or shortcuts. You can also join Azure Cosmos DB data with other mirrored database sources such as Azure SQL Database, Snowflake.

-You can query across Azure Cosmos DB collections or databases mirrored into OneLake.

-With Mirroring in Fabric, you don't need to piece together different services from multiple vendors. Instead, you can enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs.

-You can use T-SQL to run complex aggregate queries and Spark for data exploration. You can seamlessly access the data in notebooks, use data science to build machine learning models, and build Power BI reports using Direct Lake powered by rich Copilot integration.

-If you're looking for analytics on your operational data in Azure Cosmos DB, mirroring provides:

-* Zero ETL, cost-effective near real-time analytics on Azure Cosmos DB data without affecting your request unit (RU) consumption

-* Ease of bringing data across various sources into Fabric OneLake.

-* Improved query performance of SQL engine handling delta tables, with V-order optimizations

-* Improved cold start time for Spark engine with deep integration with ML/notebooks

-* One-click integration with Power BI with Direct Lake and Copilot

-* Richer app integration to access queries and views with GraphQL

-* Open access to and from other services such as Azure Databricks

-To get started with mirroring, visit ["Get started with mirroring tutorial"](/fabric/database/mirrored-database/azure-cosmos-db-tutorial?context=/azure/cosmos-db/context/context).

-### Option 2: Azure Synapse Link to access data from Azure Synapse Analytics

-Azure Synapse Link for Azure Cosmos DB creates a tight seamless integration between Azure Cosmos DB and Azure Synapse Analytics, enabling zero ETL, near real-time analytics on your operational data.

-Transactional data is seamlessly synced to Analytical store, which stores the data in columnar format optimized for analytics.

-Azure Synapse Analytics can access this data in Analytical store, without further movement, using Azure Synapse Link. Business analysts, data engineers, and data scientists can now use Synapse Spark or Synapse SQL interchangeably to run near real time business intelligence, analytics, and machine learning pipelines.

-The following image shows the Azure Synapse Link integration with Azure Cosmos DB and Azure Synapse Analytics:

- > [!IMPORTANT]

- > Mirroring in Microsoft Fabric is now available in preview for NoSql API. This feature provides all the capabilities of Azure Synapse Link with better analytical performance, ability to unify your data estate with Fabric OneLake and open access to your data in OneLake with Delta Parquet format. If you are considering Azure Synapse Link, we recommend that you try mirroring to assess overall fit for your organization. To get started with mirroring, click [here](/fabric/database/mirrored-database/azure-cosmos-db?context=/azure/cosmos-db/context/context).

-To get started with Azure Synapse Link, visit ["Getting started with Azure Synapse Link"](synapse-link.md).

-## Real-time analytics and BI on Azure Cosmos DB: Other options

-There are a few other options to enable real-time analytics on Azure Cosmos DB data:

-* Using [change feed](nosql/changefeed-ecommerce-solution.md)

-* Using [Spark connector directly on Azure Cosmos DB](nosql/tutorial-spark-connector.md)

-* Using Power BI connector directly on Azure Cosmos DB

-While these options are included for completeness and work well with single partition queries in real-time, these methods have the following challenges for analytical queries:

-* Performance impact on your workload:

- Analytical queries tend to be complex and consume significant compute capacity. When these queries are run against your Azure Cosmos DB data directly, you might experience performance degradation on your transactional queries.

-* Cost impact:

-

- When analytical queries are run directly against your database or collections, they increase the need for request units allocated, as analytical queries tend to be complex and need more computation power. Increased RU usage will likely lead to significant cost impact over time, if you run aggregate queries.

-Instead of these options, we recommend that you use Mirroring in Microsoft Fabric or Azure Synapse Link, which provide zero ETL analytics, without affecting transactional workload performance or request units.

-## Related content

-* [Mirroring Azure Cosmos DB overview](/fabric/database/mirrored-database/azure-cosmos-db?context=/azure/cosmos-db/context/context)

-* [Getting started with mirroring](/fabric/database/mirrored-database/azure-cosmos-db-tutorial?context=/azure/cosmos-db/context/context)

-* [Azure Synapse Link for Azure Cosmos DB](synapse-link.md)

-* [Working with Azure Synapse Link for Azure Cosmos DB](configure-synapse-link.md)

-description: Learn how real-time analytics is used in Supply chain analytics, forecasting, reporting, real-time personalization, and IOT predictive maintenance.

-# Azure Cosmos DB: No-ETL analytics use cases

-Azure Cosmos DB provides various analytics options for no-ETL, near real-time analytics over operational data. You can enable analytics on your Azure Cosmos DB data using following options:

-* Mirroring Azure Cosmos DB in Microsoft Fabric

-* Azure Synapse Link for Azure Cosmos DB

-To learn more about these options, see ["Analytics and BI on your Azure Cosmos DB data."](analytics-and-business-intelligence-overview.md)

-> [!IMPORTANT]

-> Mirroring Azure Cosmos DB in Microsoft Fabric is now available in preview for NoSql API. This feature provides all the capabilities of Azure Synapse Link with better analytical performance, ability to unify your data estate with Fabric OneLake and open access to your data in OneLake with Delta Parquet format. If you are considering Azure Synapse Link, we recommend that you try mirroring to assess overall fit for your organization. To get started with mirroring, click [here](/fabric/database/mirrored-database/azure-cosmos-db?context=/azure/cosmos-db/context/context).

-No-ETL, near real-time analytics can open up various possibilities for your businesses. Here are three sample scenarios:

-* Supply chain analytics, forecasting & reporting

-* Real-time personalization

-* Predictive maintenance, anomaly detection in IOT scenarios

-## Supply chain analytics, forecasting & reporting

-Research studies show that embedding big data analytics in supply chain operations leads to improvements in order-to-cycle delivery times and supply chain efficiency.

-Manufacturers are onboarding to cloud-native technologies to break out of constraints of legacy Enterprise Resource Planning (ERP) and Supply Chain Management (SCM) systems. With supply chains generating increasing volumes of operational data every minute (order, shipment, transaction data), manufacturers need an operational database. This operational database should scale to handle the data volumes as well as an analytical platform to get to a level of real-time contextual intelligence to stay ahead of the curve.

-The following architecture shows the power of using Azure Cosmos DB as the cloud-native operational database in supply chain analytics:

-Based on previous architecture, you can achieve the following use cases:

-* **Prepare & train predictive pipeline:** Generate insights over the operational data across the supply chain using machine learning translates. This way you can lower inventory, operations costs, and reduce the order-to-delivery times for customers.

- Mirroring and Synapse Link allow you to analyze the changing operational data in Azure Cosmos DB without any manual ETL processes. These offerings save you from additional cost, latency, and operational complexity. They enable data engineers and data scientists to build robust predictive pipelines:

- * Query operational data from Azure Cosmos DB by using native integration with Apache Spark pools in Microsoft Fabric or Azure Synapse Analytics. You can query the data in an interactive notebook or scheduled remote jobs without complex data engineering.

- * Build Machine Learning (ML) models with Spark ML algorithms and Azure Machine Learning (AML) integration in Microsoft Fabric or Azure Synapse Analytics.

- * Write back the results after model inference into Azure Cosmos DB for operational near-real-time scoring.

-* **Operational reporting:** Supply chain teams need flexible and custom reports over real-time, accurate operational data. These reports are required to obtain a snapshot view of supply chain effectiveness, profitability, and productivity. It allows data analysts and other key stakeholders to constantly reevaluate the business and identify areas to tweak to reduce operational costs.

- Mirroring and Synapse Link for Azure Cosmos DB enable rich business intelligence (BI)/reporting scenarios:

- * Query operational data from Azure Cosmos DB by using native integration with full expressiveness of T-SQL language.

- * Model and publish auto refreshing BI dashboards over Azure Cosmos DB through Power BI integrated in Microsoft Fabric or Azure Synapse Analytics.

-The following is some guidance for data integration for batch & streaming data into Azure Cosmos DB:

-* **Batch data integration & orchestration:** With supply chains getting more complex, supply chain data platforms need to integrate with variety of data sources and formats. Microsoft Fabric and Azure Synapse come built-in with the same data integration engine and experiences as Azure Data Factory. This integration allows data engineers to create rich data pipelines without a separate orchestration engine:

- * Move data from 85+ supported data sources to [Azure Cosmos DB with Azure Data Factory](../data-factory/connector-azure-cosmos-db.md).

- * Write code-free ETL pipelines to Azure Cosmos DB including [relational-to-hierarchical and hierarchical-to-hierarchical mappings with mapping data flows](../data-factory/how-to-sqldb-to-cosmosdb.md).

-* **Streaming data integration & processing:** With the growth of Industrial IoT (sensors tracking assets from 'floor-to-store', connected logistics fleets, etc.), there is an explosion of real-time data being generated in a streaming fashion that needs to be integrated with traditional slow moving data for generating insights. Azure Stream Analytics is a recommended service for streaming ETL and processing on Azure with a [wide range of scenarios](../stream-analytics/streaming-technologies.md). Azure Stream Analytics supports [Azure Cosmos DB as a native data sink](../stream-analytics/stream-analytics-documentdb-output.md).

-## Real-time personalization

-Retailers today must build secure and scalable e-commerce solutions that meet the demands of both customers and business. These e-commerce solutions need to engage customers through customized products and offers, process transactions quickly and securely, and focus on fulfillment and customer service. Azure Cosmos DB along with the latest Synapse Link for Azure Cosmos DB allows retailers to generate personalized recommendations for customers in real time. They use low-latency and tunable consistency settings for immediate insights as shown in the following architecture:

-* **Prepare & train predictive pipeline:** You can generate insights over the operational data across your business units or customer segments using Fabric or Synapse Spark and machine learning models. This translates to personalized delivery to target customer segments, predictive end-user experiences, and targeted marketing to fit your end-user requirements.

-)

-## IOT predictive maintenance

-Industrial IOT innovations have drastically reduced downtimes of machinery and increased overall efficiency across all fields of industry. One of such innovations is predictive maintenance analytics for machinery at the edge of the cloud.

-The following is an architecture using the cloud native HTAP capabilities in IoT predictive maintenance:

-* **Prepare & train predictive pipeline:** The historical operational data from IoT device sensors could be used to train predictive models such as anomaly detectors. These anomaly detectors are then deployed back to the edge for real-time monitoring. Such a virtuous loop allows for continuous retraining of the predictive models.

-* **Operational reporting:** With the growth of digital twin initiatives, companies are collecting vast amounts of operational data from large number of sensors to build a digital copy of each machine. This data powers BI needs to understand trends over historical data in addition to recent hot data.

-## Related content

-* [Mirroring Azure Cosmos DB overview](/fabric/database/mirrored-database/azure-cosmos-db?context=/azure/cosmos-db/context/context)

-* [Getting started with mirroring](/fabric/database/mirrored-database/azure-cosmos-db-tutorial?context=/azure/cosmos-db/context/context)

-

-* [Azure Synapse Link for Azure Cosmos DB](synapse-link.md)

-* [Working with Azure Synapse Link for Azure Cosmos DB](configure-synapse-link.md)

-description: This article presents an overview of Azure Cosmos DB Attachments.

-# Azure Cosmos DB Attachments

-Azure Cosmos DB attachments are special items that contain references to an associated metadata with an external blob or media file.

-Azure Cosmos DB supports two types of attachments:

-* **Unmanaged Attachments** are a wrapper around a URI reference to a blob that is stored in an external service (for example, Azure Storage, OneDrive, etc.). This approach is similar to storing a URI property in a standard Azure Cosmos DB item.

-* **Managed Attachments** are blobs managed and stored internally by Azure Cosmos DB and exposed via a system-generated mediaLink.

-> [!NOTE]

-> Attachments are a legacy feature. Their support is scoped to offer continued functionality if you are already using this feature.

->

-> Instead of using attachments, we recommend you to use Azure Blob Storage as a purpose-built blob storage service to store blob data. You can continue to store metadata related to blobs, along with reference URI links, in Azure Cosmos DB as item properties. Storing this data in Azure Cosmos DB provides the ability to query metadata and links to blobs stored in Azure Blob Storage.

->

-> Microsoft is committed to provide a minimum 36-month notice prior to fully deprecating attachments ΓÇô which will be announced at a further date.

-## Known limitations

-Azure Cosmos DBΓÇÖs managed attachments are distinct from its support for standard items ΓÇô for which it offers unlimited scalability, global distribution, and integration with other Azure services.

-> [!NOTE]

-> Azure Cosmos DB for MongoDB version 3.2 utilizes managed attachments for GridFS and these are subject to the same limitations.

->

-> We recommend developers using the MongoDB GridFS feature set to upgrade to Azure Cosmos DB for MongoDB version 3.6 or higher, which is decoupled from attachments and provides a better experience. Alternatively, developers using the MongoDB GridFS feature set should also consider using Azure Blob Storage - which is purpose-built for storing blob content and offers expanded functionality at lower cost compared to GridFS.

-## Migrating Attachments to Azure Blob Storage

-We recommend migrating Azure Cosmos DB attachments to Azure Blob Storage by following these steps:

-1. Copy attachments data from your source Azure Cosmos DB container to your target Azure Blob Storage container.

-2. Validate the uploaded blob data in the target Azure Blob Storage container.

-3. If applicable, add URI references to the blobs contained in Azure Blob Storage as string properties within your Azure Cosmos DB dataset.

-4. Refactor your application code to read and write blobs from the new Azure Blob Storage container.

-The following code sample shows how to copy attachments from Azure Cosmos DB to Azure Blob storage as part of a migration flow by using Azure Cosmos DB's .NET SDK v2 and Azure Blob Storage .NET SDK v12. Make sure to replace the `<placeholder values>` for the source Azure Cosmos DB account and target Azure Blob storage container.

-```csharp

-using System;

-using System.IO;

-using System.Threading.Tasks;

-using Microsoft.Azure.Documents;

-using Microsoft.Azure.Documents.Client;

-using Azure.Storage.Blobs;

-using Azure.Storage.Blobs.Models;

-namespace attachments

-{

- class Program

- {

- private static string cosmosAccount = "<Your_Azure_Cosmos_account_URI>";

- private static string cosmosKey = "<Your_Azure_Cosmos_account_PRIMARY_KEY>";

- private static string cosmosDatabaseName = "<Your_Azure_Cosmos_database>";

- private static string cosmosCollectionName = "<Your_Azure_Cosmos_collection>";

- private static string storageConnectionString = "<Your_Azure_Storage_connection_string>";

- private static string storageContainerName = "<Your_Azure_Storage_container_name>";

- private static DocumentClient cosmosClient = new DocumentClient(new Uri(cosmosAccount), cosmosKey);

- private static BlobServiceClient storageClient = new BlobServiceClient(storageConnectionString);

- private static BlobContainerClient storageContainerClient = storageClient.GetBlobContainerClient(storageContainerName);

- static void Main(string[] args)

- {

- CopyAttachmentsToBlobsAsync().Wait();

- }

- private async static Task CopyAttachmentsToBlobsAsync()

- {

- Console.WriteLine("Copying Azure Cosmos DB Attachments to Azure Blob Storage ...");

- int totalCount = 0;

- string docContinuation = null;

- // Iterate through each item (document in v2) in the Azure Cosmos DB container (collection in v2) to look for attachments.

- do

- {

- FeedResponse<dynamic> response = await cosmosClient.ReadDocumentFeedAsync(

- UriFactory.CreateDocumentCollectionUri(cosmosDatabaseName, cosmosCollectionName),

- new FeedOptions

- {

- MaxItemCount = -1,

- RequestContinuation = docContinuation

- });

- docContinuation = response.ResponseContinuation;

- foreach (Document document in response)

- {

- string attachmentContinuation = null;

- PartitionKey docPartitionKey = new PartitionKey(document.Id);

- // Iterate through each attachment within the item (if any).

- do

- {

- FeedResponse<Attachment> attachments = await cosmosClient.ReadAttachmentFeedAsync(

- document.SelfLink,

- new FeedOptions

- {

- PartitionKey = docPartitionKey,

- RequestContinuation = attachmentContinuation

- }

- );

- attachmentContinuation = attachments.ResponseContinuation;

- foreach (var attachment in attachments)

- {

- // Download the attachment in to local memory.

- MediaResponse content = await cosmosClient.ReadMediaAsync(attachment.MediaLink);

- byte[] buffer = new byte[content.ContentLength];

- await content.Media.ReadAsync(buffer, 0, buffer.Length);

- // Upload the locally buffered attachment to blob storage

- string blobId = String.Concat(document.Id, "-", attachment.Id);

- Azure.Response<BlobContentInfo> uploadedBob = await storageContainerClient.GetBlobClient(blobId).UploadAsync(

- new MemoryStream(buffer, writable: false),

- true

- );

- Console.WriteLine("Copied attachment ... Item Id: {0} , Attachment Id: {1}, Blob Id: {2}", document.Id, attachment.Id, blobId);

- totalCount++;

- // Clean up attachment from Azure Cosmos DB.

- // Warning: please verify you've succesfully migrated attachments to blog storage prior to cleaning up Azure Cosmos DB.

- // await cosmosClient.DeleteAttachmentAsync(

- // attachment.SelfLink,

- // new RequestOptions { PartitionKey = docPartitionKey }

- // );

- // Console.WriteLine("Cleaned up attachment ... Document Id: {0} , Attachment Id: {1}", document.Id, attachment.Id);

- }

- } while (!string.IsNullOrEmpty(attachmentContinuation));

- }

- }

- while (!string.IsNullOrEmpty(docContinuation));

- Console.WriteLine("Finished copying {0} attachments to blob storage", totalCount);

- }

- }

-}

-```

-## Next steps

-description: Learn how to audit the control plane operations such as add a region, update throughput, region failover, add a VNet etc. in Azure Cosmos DB

-# How to audit Azure Cosmos DB control plane operations

-Control Plane in Azure Cosmos DB is a RESTful service that enables you to perform a diverse set of operations on the Azure Cosmos DB account. It exposes a public resource model (for example: database, account) and various operations to the end users to perform actions on the resource model. The control plane operations include changes to the Azure Cosmos DB account or container. For example, operations such as create an Azure Cosmos DB account, add a region, update throughput, region failover, add a VNet etc. are some of the control plane operations. This article explains how to audit the control plane operations in Azure Cosmos DB. You can run the control plane operations on Azure Cosmos DB accounts by using Azure CLI, PowerShell or Azure portal, whereas for containers, use Azure CLI or PowerShell.

-The following are some example scenarios where auditing control plane operations is helpful:

-* You want to get an alert when the firewall rules for your Azure Cosmos DB account are modified. The alert is required to find unauthorized modifications to rules that govern the network security of your Azure Cosmos DB account and take quick action.

-* You want to get an alert if a new region is added or removed from your Azure Cosmos DB account. Adding or removing regions has implications on billing and data sovereignty requirements. This alert will help you detect an accidental addition or removal of region on your account.

-* You want to get more details from the diagnostic logs on what has changed. For example, a VNet was changed.

-## Disable key based metadata write access

-Before you audit the control plane operations in Azure Cosmos DB, disable the key-based metadata write access on your account. When key based metadata write access is disabled, clients connecting to the Azure Cosmos DB account through account keys are prevented from accessing the account. You can disable write access by setting the `disableKeyBasedMetadataWriteAccess` property to true. After you set this property, changes to any resource can happen from a user with the proper Azure role and credentials. To learn more on how to set this property, see the [Preventing changes from SDKs](role-based-access-control.md#prevent-sdk-changes) article.

-After the `disableKeyBasedMetadataWriteAccess` is turned on, if the SDK based clients run create or update operations, an error *"Operation 'POST' on resource 'ContainerNameorDatabaseName' is not allowed through Azure Cosmos DB endpoint* is returned. You have to turn on access to such operations for your account, or perform the create/update operations through Azure Resource Manager, Azure CLI or Azure PowerShell. To switch back, set the disableKeyBasedMetadataWriteAccess to **false** by using Azure CLI as described in the [Preventing changes from Azure Cosmos DB SDK](role-based-access-control.md#prevent-sdk-changes) article. Make sure to change the value of `disableKeyBasedMetadataWriteAccess` to false instead of true.

-Consider the following points when turning off the metadata write access:

-* Evaluate and ensure that your applications do not make metadata calls that change the above resources (For example, create collection, update throughput, …) by using the SDK or account keys.

-* When `disableKeyBasedMetadataWriteAccess` is set to true, the metadata operations issued by the SDK are blocked. Alternatively, you can use Azure portal, Azure CLI, Azure PowerShell, or Azure Resource Manager template deployments to perform these operations.

-## Enable diagnostic logs for control plane operations

-You can enable diagnostic logs for control plane operations by using the Azure portal. After enabling, the diagnostic logs will record the operation as a pair of start and complete events with relevant details. For example, the *RegionFailoverStart* and *RegionFailoverComplete* will complete the region failover event.

-Use the following steps to enable logging on control plane operations:

-1. Sign in to the [Azure portal](https://portal.azure.com) and navigate to your Azure Cosmos DB account.

-1. Open the **Diagnostic settings** pane, provide a **Name** for the logs to create.

-1. Select **ControlPlaneRequests** for log type and select the **Send to Log Analytics** option.

-1. Optionally, send the diagnostic logs to Azure Storage, Azure Event Hubs, Azure Monitor, or a third party.

-You can also store the logs in a storage account or stream to an event hub. This article shows how to send logs to log analytics and then query them. After you enable, it takes a few minutes for the diagnostic logs to take effect. All the control plane operations performed after that point can be tracked. The following screenshot shows how to enable control plane logs:

-## View the control plane operations

-After you turn on logging, use the following steps to track down operations for a specific account:

-1. Sign in to the [Azure portal](https://portal.azure.com).

-1. Open the **Monitor** tab from the left-hand navigation and then select the **Logs** pane. It opens a UI where you can easily run queries with that specific account in scope. Run the following query to view control plane logs:

- ```kusto

- AzureDiagnostics

- | where ResourceProvider=="MICROSOFT.DOCUMENTDB" and Category=="ControlPlaneRequests"

- | where TimeGenerated >= ago(1h)

- ```

- The following screenshots capture logs when a consistency level is changed for an Azure Cosmos DB account. The `activityId_g` value from results is different from the activity ID of an operation:

- :::image type="content" source="./media/audit-control-plane-logs/add-ip-filter-logs.png" alt-text="Control plane logs when a VNet is added":::

- The following screenshots capture logs when the keyspace or a table of a Cassandra account are created and when the throughput is updated. The control plane logs for create and update operations on the database and the container are logged separately as shown in the following screenshot:

- :::image type="content" source="./media/audit-control-plane-logs/throughput-update-logs.png" alt-text="Control plane logs when throughput is updated":::

-## Identify the identity associated to a specific operation

-If you want to debug further, you can identify a specific operation in the **Activity log** by using the `activityId_g` or by the timestamp of the operation. Timestamp is used for some Resource Manager clients where the activity ID is not explicitly passed. The Activity log gives details about the identity with which the operation was initiated. The following screenshot shows how to use the `activityId_g` to find the operations associated with it in the Activity log:

-## Control plane operations for Azure Cosmos DB account

-The following are the control plane operations available at the account level. Most of the operations are tracked at account level. These operations are available as metrics in Azure monitor:

-* Region added

-* Region removed

-* Account deleted

-* Region failed over

-* Account created

-* Virtual network deleted

-* Account network settings updated

-* Account replication settings updated

-* Account keys updated

-* Account backup settings updated

-* Account diagnostic settings updated

-## Control plane operations for database or containers

-The following are the control plane operations available at the database and container level. These operations are available as metrics in Azure monitor:

-* SQL Database Created

-* SQL Database Updated

-* SQL Database Throughput Updated

-* SQL Database Deleted

-* SQL Container Created

-* SQL Container Updated

-* SQL Container Throughput Updated

-* SQL Container Deleted

-* Cassandra Keyspace Created

-* Cassandra Keyspace Updated

-* Cassandra Keyspace Throughput Updated

-* Cassandra Keyspace Deleted

-* Cassandra Table Created

-* Cassandra Table Updated

-* Cassandra Table Throughput Updated

-* Cassandra Table Deleted

-* Gremlin Database Created

-* Gremlin Database Updated

-* Gremlin Database Throughput Updated

-* Gremlin Database Deleted

-* Gremlin Graph Created

-* Gremlin Graph Updated

-* Gremlin Graph Throughput Updated

-* Gremlin Graph Deleted

-* Mongo Database Created

-* Mongo Database Updated

-* Mongo Database Throughput Updated

-* Mongo Database Deleted

-* Mongo Collection Created

-* Mongo Collection Updated

-* Mongo Collection Throughput Updated

-* Mongo Collection Deleted

-* AzureTable Table Created

-* AzureTable Table Updated

-* AzureTable Table Throughput Updated

-* AzureTable Table Deleted

-## Diagnostic log operations

-The following are the operation names in diagnostic logs for different operations:

-* RegionAddStart, RegionAddComplete

-* RegionRemoveStart, RegionRemoveComplete

-* AccountDeleteStart, AccountDeleteComplete

-* RegionFailoverStart, RegionFailoverComplete

-* AccountCreateStart, AccountCreateComplete

-* AccountUpdateStart, AccountUpdateComplete

-* VirtualNetworkDeleteStart, VirtualNetworkDeleteComplete

-* DiagnosticLogUpdateStart, DiagnosticLogUpdateComplete

-For API-specific operations, the operation is named with the following format:

-* ApiKind + ApiKindResourceType + OperationType

-* ApiKind + ApiKindResourceType + "Throughput" + operationType

-**Example**

-* CassandraKeyspacesCreate

-* CassandraKeyspacesUpdate

-* CassandraKeyspacesThroughputUpdate

-* SqlContainersUpdate

-The *ResourceDetails* property contains the entire resource body as a request payload and it contains all the properties requested to update

-## Diagnostic log queries for control plane operations

-The following are some examples to get diagnostic logs for control plane operations:

-```kusto

-AzureDiagnostics 

-| where Category startswith "ControlPlane"

-| where OperationName contains "Update"

-| project httpstatusCode_s, statusCode_s, OperationName, resourceDetails_s, activityId_g

-```

-```kusto

-AzureDiagnostics 

-| where Category =="ControlPlaneRequests"

-| where TimeGenerated >= todatetime('2020-05-14T17:37:09.563Z')

-| project TimeGenerated, OperationName, apiKind_s, apiKindResourceType_s, operationType_s, resourceDetails_s

-```

-```kusto

-AzureDiagnostics

-| where Category == "ControlPlaneRequests"

-| where OperationName startswith "SqlContainersUpdate"

-```

-```kusto

-AzureDiagnostics

-| where Category == "ControlPlaneRequests"

-| where OperationName startswith "SqlContainersThroughputUpdate"

-```

-Query to get the activityId and the caller who initiated the container delete operation:

-```kusto

-(AzureDiagnostics

-| where Category == "ControlPlaneRequests"

-| where OperationName == "SqlContainersDelete"

-| where TimeGenerated >= todatetime('9/3/2020, 5:30:29.300 PM')

-| summarize by activityId_g )

-| join (

-AzureActivity

-| parse HTTPRequest with * "clientRequestId\": \"" activityId_g "\"" *

-| summarize by Caller, HTTPRequest, activityId_g)

-on activityId_g

-| project Caller, activityId_g

-```

-Query to get index or ttl updates. You can then compare the output of this query with an earlier update to see the change in index or ttl.

-```Kusto

-AzureDiagnostics

-| where Category =="ControlPlaneRequests"

-| where OperationName == "SqlContainersUpdate"

-| project resourceDetails_s

-```

-**output:**

-```json

-{id:skewed,indexingPolicy:{automatic:true,indexingMode:consistent,includedPaths:[{path:/*,indexes:[]}],excludedPaths:[{path:/_etag/?}],compositeIndexes:[],spatialIndexes:[]},partitionKey:{paths:[/pk],kind:Hash},defaultTtl:1000000,uniqueKeyPolicy:{uniqueKeys:[]},conflictResolutionPolicy:{mode:LastWriterWins,conflictResolutionPath:/_ts,conflictResolutionProcedure:}

-```

-## Next steps

-* [Prevent Azure Cosmos DB resources from being deleted or changed](resource-locks.md)

-* [Explore Azure Monitor for Azure Cosmos DB](insights-overview.md?toc=/azure/cosmos-db/toc.json&bc=/azure/cosmos-db/breadcrumb/toc.json)

-* [Monitor and debug with metrics in Azure Cosmos DB](use-metrics.md)

-description: This article provides details available to audit Azure Cosmos DB's point in time restore feature in continuous backup mode.

-# Audit the point-in-time restore action for continuous backup mode in Azure Cosmos DB

-Azure Cosmos DB provides you with a list of all point-in-time restores for continuous mode that were performed on an Azure Cosmos DB account using [activity logs](monitor.md#activity-log). Activity logs can be viewed for any Azure Cosmos DB account from the **Activity Logs** page in the Azure portal. The activity log shows all the operations that were triggered on the specific account. When a point-in-time restore is triggered, it shows up as `Restore Database Account` operation on the source account as well as the target account. The activity log for the source account can be used to audit restore events, and the activity logs on the target account can be used to get the updates about the progress of the restore.

-## Audit the restores that were triggered on a live database account

-When a restore is triggered on a source account, a log is emitted with the status *Started*. And when the restore succeeds or fails, a new log is emitted with the status *Succeeded* or *Failed* respectively.

-To get the list of just the restore operations that were triggered on a specific account, you can open the Activity Log of the source account, and search for **Restore database account** in the search bar with the required **Timespan** filter. The `UserPrincipalName` of the user that triggered the restore can be found from the `Event initiated by` column.

-The parameters of the restore request can be found by clicking on the event and selecting the JSON tab:

-## Audit the restores that were triggered on a deleted database account

-For the accounts that were already deleted, there would not be any database account page. Instead, the Activity Log in the subscription page can be used to get the restores that were triggered on a deleted account. Once the Activity Log page is opened, a new filter can be added to narrow down the results specific to the resource group the account existed in, or even using the database account name in the Resource filter. The Resource for the activity log is the database account on which the restore was triggered.

-The activity logs can also be accessed using Azure CLI or Azure PowerShell. For more information on activity logs, review [Azure Activity log - Azure Monitor](../azure-monitor/essentials/activity-log.md).

-## Track the progress of the restore operation

-Azure Cosmos DB allows you to track the progress of the restore using the activity logs of the restored database account. Once the restore is triggered, you will see a notification with the title **Restore Account**.

-The account status would be *Creating*, but it would have an Activity Log page. A new log event will appear after the restore of each collection. Note that there can be a delay of 5-10 minutes to see the log event after the actual restore of the collection is complete.

- ## Next steps

-

- * Learn more about [continuous backup](continuous-backup-restore-introduction.md) mode.

- * Provision an account with continuous backup by using the [Azure portal](provision-account-continuous-backup.md#provision-portal), [PowerShell](provision-account-continuous-backup.md#provision-powershell), the [Azure CLI](provision-account-continuous-backup.md#provision-cli), or [Azure Resource Manager](provision-account-continuous-backup.md#provision-arm-template).

- * [Manage permissions](continuous-backup-restore-permissions.md) required to restore data with continuous backup mode.

- * Learn about the [resource model of continuous backup mode](continuous-backup-restore-resource-model.md).

- * Explore the [Frequently asked questions for continuous mode](continuous-backup-restore-frequently-asked-questions.yml).

-description: Learn how to view customized performance, cost, security, and other recommendations for Azure Cosmos DB based on your workload patterns.

-# Automated recommendations for Azure Cosmos DB

-All the cloud services including Azure Cosmos DB get frequent updates with new features, capabilities, and improvements. ItΓÇÖs important for your application to keep up with the latest performance and security updates. The Azure portal offers customized recommendations that enable you to maximize the performance of your application. The Azure Cosmos DB's advisory engine continuously analyzes the usage history of your Azure Cosmos DB resources and provides recommendations based on your workload patterns. These recommendations correspond to areas like partitioning, indexing, network, security etc. These customized recommendations help you to improve the performance of your application.

-## View recommendations

-You can view recommendations for Azure Cosmos DB in the following ways:

- :::image type="content" source="./media/automated-recommendations/cosmos-db-pane-recommendations.png" alt-text="View recommendations from Azure Cosmos DB pane":::

- :::image type="content" source="./media/automated-recommendations/advisor-pane-recommendations.png" alt-text="View recommendations from Azure Advisor pane":::

-Not all recommendations shown in the Azure Cosmos DB pane are available in the Azure Advisor and vice versa. ThatΓÇÖs because based on the type of recommendation they fit in either the Azure Advisor pane, Azure Cosmos DB pane or both.

-Currently Azure Cosmos DB supports recommendations on the following areas. Each of these recommendations includes a link to the relevant section of the documentation, so itΓÇÖs easy for you to take the next steps.

-## SDK usage recommendations

-In this category, the advisor detects the usage of an old version of SDKs and recommends that you upgrade to a newer version to leverage the latest bug fixes and performance improvements. Currently the following SDK-specific recommendations are available:

-|Name |Description |

-|||

-| Old Spark connector | Detects the usage of old versions of the Spark connector and recommends upgrading. |

-| Old .NET SDK | Detects the usage of old versions of the .NET SDK and recommends upgrading. |

-| Old Java SDK | Detects the usage of old versions of the Java connector and recommends upgrading. |

-## Indexing recommendations

-In this category, the advisor detects the indexing mode, indexing policy, indexed paths and recommends changing if the current configuration impacts the query performance. Currently the following indexing-specific recommendations are available:

-|Name |Description |

-|||

-| Lazy indexing | Detects usage of lazy indexing mode and recommends using consistent indexing mode instead. The purpose of Azure Cosmos DBΓÇÖs lazy indexing mode is limited and can impact the freshness of query results in some situations so consistent indexing mode is recommended. |

-| Default indexing policy with many indexed paths | Detects containers running on default indexing with many indexed paths and recommends customizing the indexing policy.|

-| ORDER BY queries with high RU/s charge| Detects containers issuing ORDER BY queries with high RU/s charge and recommends exploring composite indexes for one container per account that issues the highest number of these queries in a 24 hour period.|

-| MongoDB 3.6 accounts with no index and high RU/s consumption| Detects Azure Cosmos DBΓÇÖs API for MongoDB with 3.6 version of containers issuing queries with high RU/s charge and recommends adding indexes.|

-## Cost optimization recommendations

-In this category, the advisor detects the RU/s usage and determines that you can optimize the price by making some changes to your resources or by leveraging a different pricing model. Currently the following cost optimization-specific recommendations are available:

-|Name |Description |

-|||

-| Reserved capacity | Detects your RU/s utilization and recommends reserved instances to users who can benefit from it. |

-| Inactive containers | Detects the containers that haven't been used for more than 30 days and recommends reducing the throughput for such containers or deleting them.|

-| New subscriptions with high throughput | Detects new subscriptions with accounts spending unusually high RU/s per day and provides them a notification. This notification is specifically to bring awareness to new customers that Azure Cosmos DB operates on provisioned throughput-based model and not consumption-based model. |

-| Enable autoscale | Detects if your databases and containers currently using manual throughput would see cost savings by enabling autoscale. |

-| Use manual throughput instead of autoscale | Detects if your databases and containers currently using autoscale throughput would see cost savings by switching to manual throughput. |

-## Migration recommendations

-In this category, the advisor detects that you are using legacy features recommends migrating so that you can leverage Azure Cosmos DBΓÇÖs massive scalability and other benefits. Currently the following migration-specific recommendations are available:

-|Name |Description |

-|||

-| Non-partitioned containers | Detects fixed-size containers that are approaching their max storage limit and recommends migrating them to partitioned containers.|

-## Query usage recommendations

-In this category, the advisor detects the query execution and identifies that the query performance can be tuned with some changes. Currently the following query usage recommendations are available:

-|Name |Description |

-|||

-| Queries with fixed page size | Detects queries issued with a fixed page size and recommends using -1 (no limit on the page size) instead of defining a specific value. This option reduces the number of network round trips required to retrieve all results. |

-## Next steps

-* [Tuning query performance in Azure Cosmos DB](nosql/query-metrics.md)

-* [Troubleshoot query issues](troubleshoot-query-performance.md) when using Azure Cosmos DB

-* Trying to do capacity planning for a migration to Azure Cosmos DB? You can use information about your existing database cluster for capacity planning.

- * If all you know is the number of vcores and servers in your existing database cluster, read about [estimating request units using vCores or vCPUs](convert-vcore-to-request-unit.md)

- * If you know typical request rates for your current database workload, read about [estimating request units using Azure Cosmos DB capacity planner](estimate-ru-with-capacity-planner.md)

-description: Configure autoscale in Azure Cosmos DB for uneven workload patterns by customizing autoscale for specific regions or partitions.

- - ignite-2023

-# CustomerIntent: As a database adminstrator, I want to fine tune autoscaler for specific regions or partitions so that I can balance an uneven workload.

-# Per-region and per-partition autoscale (preview)

-By default, Azure Cosmos DB autoscale scales workloads based on the most active region and partition. For nonuniform workloads that have different workload patterns across regions and partitions, this scaling can cause unnecessary scale-ups. With this improvement to autoscale, also known as "dynamic scaling," the per region and per partition autoscale feature now allows your workloadsΓÇÖ regions and partitions to scale independently based on usage.

-> [!IMPORTANT]

-> By default, this feature is only available for Azure Cosmos DB accounts created after **November 15, 2023**. For customers who can significantly benefit from dynamic scaling, Azure Cosmos DB is progressively enabling the feature in stages for existing accounts and providing GA support, ahead of broader GA. Customers in this cohort will be notified by email before the enablement. This update wonΓÇÖt impact your account(s) performance, availability, and won't cause downtime or data movement. Please contact your Microsoft representative for questions.

-This feature is recommended for autoscale workloads that are nonuniform across regions and partitions. This feature allows you to save costs if you often experience hot partitions and/or have multiple regions. When enabled, this feature applies to all autoscale resources in the account.

-## Use cases

- - By enabling autoscale per region and partition, you can now save on costs as the secondary region independently and automatically scales down while idle. The secondary regions also automatically scales-up as it becomes active and while handling write replication from the primary region.

- - These workloads often observe uneven distribution of requests across regions due to natural traffic growth and dips throughout the day. For example, a database might be active during business hours across globally distributed time zones.

-## Example

-For example, if we have a collection with **1000** RU/s and **2** partitions, each partition can go up to **500** RU/s. For one hour of activity, the utilization would look like this:

-| Region | Partition | Throughput | Utilization | Notes |

-| | | | | |

-| Write | P1 | <= 500 RU/s | 100% | 500 RU/s consisting of 50 RU/s used for write operations and 450 RU/s for read operations. |

-| Write | P2 | <= 200 RU/s | 40% | 200 RU/s consisting of all read operations. |

-| Read | P1 | <= 150 RU/s | 30% | 150 RU/s consisting of 50 RU/s used for writes replicated from the write region. 100 RU/s are used for read operations in this region. |

-| Read | P2 | <= 50 RU/s | 10% | |

-Because all partitions are scaled uniformly based on the hottest partition, both the write and read regions are scaled to 1000 RU/s, making the total RU/s as much as **2000 RU/s**.

-With per-partition or per-region scaling, you can optimize your throughput. The total consumption would be **900 RU/s** as each partition or region's throughput is scaled independently and measured per hour using the same scenario.

-## Get started

-This feature is available for new Azure Cosmos DB accounts. To enable this feature, follow these steps:

-1. Navigate to your Azure Cosmos DB account in the [Azure portal](https://portal.azure.com).

-1. Navigate to the **Features** page.

-1. Locate and enable the **Per Region and Per Partition Autoscale** feature.

- :::image type="content" source="media/autoscale-per-partition-region/enable-feature.png" lightbox="media/autoscale-per-partition-region/enable-feature.png" alt-text="Screenshot of the 'Per Region and Per Partition Autoscale' feature in the Azure portal.":::

- > [!IMPORTANT]

- > The feature is enabled at the account level, so all containers within the account will automatically have this capability applied. The feature is available for both shared throughput databases and containers with dedicated throughput. Provisioned throughput accounts must switch over to autoscale and then enable this feature, if interested.

-1. Use [Azure Monitor metrics](monitor-reference.md#supported-metrics-for-microsoftdocumentdbdatabaseaccounts) to analyze how the new autoscaling is applied across partitions and regions. Filter to your desired database account and container, then filter or split by the `PhysicalPartitionID` metric. This metric shows all partitions across their various regions.

- Then, use `NormalizedRUConsumption` to see which partitions and regions scale independently. You can use the `ProvisionedThroughput` metric to see what throughput value is emitted to our billing service.

-## Requirements/Limitations

-Accounts must be created after 11/15/2023 to enable this feature.

-description: Perform bulk operations in Azure Cosmos DB through bulk import and bulk update APIs offered by the bulk executor library.

-# Azure Cosmos DB bulk executor library overview

-Azure Cosmos DB is a fast, flexible, and globally distributed database service that elastically scales out to support:

-* Large read and write throughput, on the order of millions of operations per second.

-* Storing high volumes of transactional and operational data, on the order of hundreds of terabytes or even more, with predictable millisecond latency.

-The bulk executor library helps you use this massive throughput and storage. The bulk executor library allows you to perform bulk operations in Azure Cosmos DB through bulk import and bulk update APIs. You can read more about the features of bulk executor library in the following sections.

-> [!NOTE]

-> Currently, bulk executor library supports import and update operations. Azure Cosmos DB API supports this library for NoSQL and Gremlin accounts only.

-> [!IMPORTANT]

-> The bulk executor library is not currently supported on [serverless](serverless.md) accounts. On .NET, we recommend that you use the [bulk support](https://devblogs.microsoft.com/cosmosdb/introducing-bulk-support-in-the-net-sdk/) available in the V3 version of the SDK.

-## Key features of the bulk executor library

-* Using the bulk executor library significantly reduces the client-side compute resources needed to saturate the throughput allocated to a container. A single threaded application that writes data using the bulk import API achieves 10 times greater write throughput when compared to a multi-threaded application that writes data in parallel while it saturates the client machine's CPU.

-* The bulk executor library abstracts away the tedious tasks of writing application logic to handle rate limiting of request, request timeouts, and other transient exceptions. It efficiently handles them within the library.

-* It provides a simplified mechanism for applications to perform bulk operations to scale out. A single bulk executor instance that runs on an Azure virtual machine can consume greater than 500 K RU/s. You can achieve a higher throughput rate by adding more instances on individual client virtual machines.

-* The bulk executor library can bulk import more than a terabyte of data within an hour by using a scale-out architecture.

-* It can bulk update existing data in Azure Cosmos DB containers as patches.

-## How does the bulk executor operate?

-When a bulk operation to import or update documents is triggered with a batch of entities, they're initially shuffled into buckets that correspond to their Azure Cosmos DB partition key range. Within each bucket that corresponds to a partition key range, they're broken down into mini-batches.

-Each mini-batch acts as a payload that is committed on the server-side. The bulk executor library has built in optimizations for concurrent execution of the mini-batches both within and across partition key ranges.

-The following diagram illustrates how bulk executor batches data into different partition keys:

-The bulk executor library makes sure to maximally utilize the throughput allocated to a collection. It uses an [AIMD-style congestion control mechanism](https://tools.ietf.org/html/rfc5681) for each Azure Cosmos DB partition key range to efficiently handle rate limiting and timeouts.

-For more information about sample applications that consume the bulk executor library, see [Use the bulk executor .NET library to perform bulk operations in Azure Cosmos DB](nosql/bulk-executor-dotnet.md) and [Perform bulk operations on Azure Cosmos DB data](bulk-executor-java.md).

-For reference information, see [.NET bulk executor library](nosql/sdk-dotnet-bulk-executor-v2.md) and [Java bulk executor library](nosql/sdk-java-bulk-executor-v2.md).

-## Next steps

-

-* [Azure Cosmos DB Spark connector](./nosql/quickstart-spark.md)

-* [Azure Cosmos DB connector](../data-factory/connector-azure-cosmos-db.md)

-description: Use your database or container's idle throughput capacity to handle spikes of traffic with burst capacity in Azure Cosmos DB.

-# Burst capacity in Azure Cosmos DB

-Azure Cosmos DB burst capacity allows you to take advantage of your database or container's idle throughput capacity to handle spikes of traffic. With burst capacity, each physical partition can accumulate up to 5 minutes of idle capacity, which can be consumed at a rate up to 3000 RU/s. With burst capacity, requests that would have otherwise been rate limited can now be served with burst capacity while it's available.

-Burst capacity applies only to Azure Cosmos DB accounts using provisioned throughput (manual and autoscale) and doesn't apply to serverless containers. The feature is configured at the Azure Cosmos DB account level and automatically applies to all databases and containers in the account that have physical partitions with less than 3000 RU/s of provisioned throughput. Resources that have greater than or equal to 3000 RU/s per physical partition can't benefit from or use burst capacity.

-## How burst capacity works

-> [!NOTE]

-> The current implementation of burst capacity is subject to change in the future. Usage of burst capacity is subject to system resource availability and is **not guaranteed**. Azure Cosmos DB may also use burst capacity for background maintenance tasks. If your workload requires consistent throughput beyond what you have provisioned, it's recommended to provision your RU/s accordingly without relying on burst capacity. Before enabling burst capacity, it is also recommended to evaluate if your partition layout can be [merged](merge.md) to permanently give more RU/s per physical partition without relying on burst capacity.

-Let's take an example of a physical partition that has 100 RU/s of provisioned throughput and is idle for 5 minutes. With burst capacity, it can accumulate a maximum of 100 RU/s * 300 seconds = 30,000 RU of burst capacity. The capacity can be consumed at a maximum rate of 3000 RU/s, so if there's a sudden spike in request volume, the partition can burst up to 3000 RU/s for up 30,000 RU / 3000 RU/s = 10 seconds. Without burst capacity, any requests that are consumed beyond the provisioned 100 RU/s would have been rate limited (429).

-After the 10 seconds is over, the burst capacity has been used up. If the workload continues to exceed the provisioned 100 RU/s, any requests that are consumed beyond the provisioned 100 RU/s would now be rate limited (429). The maximum amount of burst capacity a physical partition can accumulate at any point in time is equal to 300 seconds * the provisioned RU/s of the physical partition.

-## Getting started

-To get started using burst capacity, navigate to the **Features** page in your Azure Cosmos DB account. Select and enable the **Burst Capacity** feature.

-Once you've enabled the feature, it takes 15-20 minutes to take effect.

-## Requirements

-To enable burst capacity, your Azure Cosmos DB account must meet all the following criteria:

-## Next steps

-description: Learn how to use Spring Data API for Apache Cassandra with Azure Cosmos DB for Apache Cassandra.

-# How to use Spring Data API for Apache Cassandra with Azure Cosmos DB for Apache Cassandra

-This article demonstrates creating a sample application that uses [Spring Data] to store and retrieve information using the [Azure Cosmos DB for Apache Cassandra](/azure/cosmos-db/cassandra-introduction).

-## Prerequisites

-The following prerequisites are required in order to complete the steps in this article:

-* An Azure subscription; if you don't already have an Azure subscription, you can activate your [MSDN subscriber benefits] or sign up for a [free Azure account].

-* A supported Java Development Kit (JDK). For more information about the JDKs available for use when developing on Azure, see [Java support on Azure and Azure Stack](/azure/developer/java/fundamentals/java-support-on-azure).

-* [Apache Maven](http://maven.apache.org/), version 3.0 or later.

-* [Curl](https://curl.haxx.se/) or similar HTTP utility to test functionality.

-* A [Git](https://git-scm.com/downloads) client.

-> [!NOTE]

-> The samples mentioned below implement custom extensions for a better experience when using Azure Cosmos DB for Apache Cassandra. They include custom retry and load balancing policies, as well as implementing recommended connection settings. For a more extensive exploration of how the custom policies are used, see Java samples for [version 3](https://github.com/Azure-Samples/azure-cosmos-cassandra-extensions-java-sample) and [version 4](https://github.com/Azure-Samples/azure-cosmos-cassandra-extensions-java-sample-v4).

-## Create an Azure Cosmos DB for Apache Cassandra account

-## Configure the sample application

-The following procedure configures the test application.

-1. Open a command shell and clone either of the following examples:

- For Java [version 3 driver](https://github.com/datastax/java-driver/tree/3.x) and corresponding Spring version:

- ```shell

- git clone https://github.com/Azure-Samples/spring-data-cassandra-on-azure-extension-v3.git

- ```

-

- For Java [version 4 driver](https://github.com/datastax/java-driver/tree/4.x) and corresponding Spring version:

- ```shell

- git clone https://github.com/Azure-Samples/spring-data-cassandra-on-azure-extension-v4.git

- ```

- > [!NOTE]

- > Although the usage described below is identical for both Java version 3 and version 4 samples above, the way in which they have been implemented in order to include custom retry and load balancing policies is different. We recommend reviewing the code to understand how to implement custom policies if you are making changes to an existing Spring Java application.

-1. Locate the *application.properties* file in the *resources* directory of the sample project, or create the file if it does not already exist.

-1. Open the *application.properties* file in a text editor, and add or configure the following lines in the file, and replace the sample values with the appropriate values from earlier:

- ```yaml

- spring.data.cassandra.contact-points=<Account Name>.cassandra.cosmos.azure.com

- spring.data.cassandra.port=10350

- spring.data.cassandra.username=<Account Name>

- spring.data.cassandra.password=********

- ```

- Where:

- | Parameter | Description |

- |||

- | `spring.data.cassandra.contact-points` | Specifies the **Contact Point** from earlier in this article. |

- | `spring.data.cassandra.port` | Specifies the **Port** from earlier in this article. |

- | `spring.data.cassandra.username` | Specifies your **Username** from earlier in this article. |

- | `spring.data.cassandra.password` | Specifies your **Primary Password** from earlier in this article. |

-1. Save and close the *application.properties* file.

-## Package and test the sample application

-Browse to the directory that contains the .pom file to build and test the application.

-1. Build the sample application with Maven; for example:

- ```shell

- mvn clean package

- ```

-1. Start the sample application; for example:

- ```shell

- java -jar target/spring-data-cassandra-on-azure-0.1.0-SNAPSHOT.jar

- ```

-1. Create new records using `curl` from a command prompt like the following examples:

- ```shell

- curl -s -d "{\"name\":\"dog\",\"species\":\"canine\"}" -H "Content-Type: application/json" -X POST http://localhost:8080/pets

- curl -s -d "{\"name\":\"cat\",\"species\":\"feline\"}" -H "Content-Type: application/json" -X POST http://localhost:8080/pets

- ```

- Your application should return values like the following:

- ```shell

- Added Pet{id=60fa8cb0-0423-11e9-9a70-39311962166b, name='dog', species='canine'}.

- Added Pet{id=72c1c9e0-0423-11e9-9a70-39311962166b, name='cat', species='feline'}.

- ```

-1. Retrieve all of the existing records using `curl` from a command prompt like the following examples:

- ```shell

- curl -s http://localhost:8080/pets

- ```

- Your application should return values like the following:

- ```json

- [{"id":"60fa8cb0-0423-11e9-9a70-39311962166b","name":"dog","species":"canine"},{"id":"72c1c9e0-0423-11e9-9a70-39311962166b","name":"cat","species":"feline"}]

- ```

-## Clean up resources

-## Next steps

-To learn more about Spring and Azure, continue to the Spring on Azure documentation center.

-> [!div class="nextstepaction"]

-> [Spring on Azure](../../index.yml)

-### Additional Resources

-For more information about using Azure with Java, see the [Azure for Java Developers] and the [Working with Azure DevOps and Java].

-[Azure for Java Developers]: ../index.yml

-[free Azure account]: https://azure.microsoft.com/pricing/free-trial/

-[Working with Azure DevOps and Java]: /azure/devops/

-[MSDN subscriber benefits]: https://azure.microsoft.com/pricing/member-offers/msdn-benefits-details/

-[Spring Boot]: http://projects.spring.io/spring-boot/

-[Spring Data]: https://spring.io/projects/spring-data

-[Spring Initializr]: https://start.spring.io/

-[Spring Framework]: https://spring.io/

-[COSMOSDB01]: media/access-data-spring-data-app/create-cosmos-db-01.png

-[COSMOSDB02]: media/access-data-spring-data-app/create-cosmos-db-02.png

-[COSMOSDB03]: media/access-data-spring-data-app/create-cosmos-db-03.png

-[COSMOSDB04]: media/access-data-spring-data-app/create-cosmos-db-04.png

-[COSMOSDB05]: media/access-data-spring-data-app/create-cosmos-db-05.png

-[COSMOSDB05-1]: media/access-data-spring-data-app/create-cosmos-db-05-1.png

-[COSMOSDB06]: media/access-data-spring-data-app/create-cosmos-db-06.png

-description: Learn best practices and ways to successfully use the Azure Cosmos DB for Apache Cassandra with Apache Cassandra applications.

-# How to adapt to Azure Cosmos DB for Apache Cassandra if you are coming from Apache Cassandra

-The Azure Cosmos DB for Apache Cassandra provides wire protocol compatibility with existing Cassandra SDKs and tools. You can run applications that are designed to connect to Apache Cassandra by using the API for Cassandra with minimal changes.

-When you use the API for Cassandra, it's important to be aware of differences between Apache Cassandra and Azure Cosmos DB. If you're familiar with native [Apache Cassandra](https://cassandra.apache.org/), this article can help you begin to use the Azure Cosmos DB for Apache Cassandra.

-## Feature support

-The API for Cassandra supports a large number of Apache Cassandra features. Some features aren't supported or they have limitations. Before you migrate, be sure that the [Azure Cosmos DB for Apache Cassandra features](support.md) you need are supported.

-## Replication

-When you plan for replication, it's important to look at both migration and consistency.

-Although you can communicate with the API for Cassandra through the Cassandra Query Language (CQL) binary protocol v4 wire protocol, Azure Cosmos DB implements its own internal replication protocol. You can't use the Cassandra gossip protocol for live migration or replication. For more information, see [Live-migrate from Apache Cassandra to the API for Cassandra by using dual writes](migrate-data-dual-write-proxy.md).

-For information about offline migration, see [Migrate data from Cassandra to an Azure Cosmos DB for Apache Cassandra account by using Azure Databricks](migrate-data-databricks.md).

-Although the approaches to replication consistency in Apache Cassandra and Azure Cosmos DB are similar, it's important to understand how they're different. A [mapping document](consistency-mapping.md) compares Apache Cassandra and Azure Cosmos DB approaches to replication consistency. However, we highly recommend that you specifically review [Azure Cosmos DB consistency settings](../consistency-levels.md) or watch a brief [video guide to understanding consistency settings in the Azure Cosmos DB platform](https://aka.ms/docs.consistency-levels).

-## Recommended client configurations

-When you use the API for Cassandra, you don't need to make substantial code changes to existing applications that run Apache Cassandra. We recommend some approaches and configuration settings for the API for Cassandra in Azure Cosmos DB. Review the blog post [API for Cassandra recommendations for Java](https://devblogs.microsoft.com/cosmosdb/cassandra-api-java/).

-## Code samples

-The API for Cassandra is designed to work with your existing application code. If you encounter connectivity-related errors, use the [quickstart samples](manage-data-java-v4-sdk.md) as a starting point to discover minor setup changes you might need to make in your existing code.

-We also have more in-depth samples for [Java v3](https://github.com/Azure-Samples/azure-cosmos-cassandra-extensions-java-sample) and [Java v4](https://github.com/Azure-Samples/azure-cosmos-cassandra-extensions-java-sample-v4) drivers. These code samples implement custom [extensions](https://github.com/Azure/azure-cosmos-cassandra-extensions/tree/release/java-driver-4/1.0.0), which in turn implement recommended client configurations.

-You also can use samples for [Java Spring Boot (v3 driver)](https://github.com/Azure-Samples/spring-data-cassandra-on-azure-extension-v3) and [Java Spring Boot (v4 driver)](https://github.com/Azure-Samples/spring-data-cassandra-on-azure-extension-v4.git).

-## Storage

-The API for Cassandra is backed by Azure Cosmos DB, which is a document-oriented NoSQL database engine. Azure Cosmos DB maintains metadata, which might result in a change in the amount of physical storage required for a specific workload.

-The difference in storage requirements between native Apache Cassandra and Azure Cosmos DB is most noticeable in small row sizes. In some cases, the difference might be offset because Azure Cosmos DB doesn't implement compaction or tombstones. This factor depends significantly on the workload. If you're uncertain about storage requirements, we recommend that you first create a proof of concept.

-## Multi-region deployments

-Native Apache Cassandra is a multi-master system by default. Apache Cassandra doesn't have an option for single-master with multi-region replication for reads only. The concept of application-level failover to another region for writes is redundant in Apache Cassandra. All nodes are independent, and there's no single point of failure. However, Azure Cosmos DB provides the out-of-box ability to configure either single-master or multi-master regions for writes.

-An advantage of having a single-master region for writes is avoiding cross-region conflict scenarios. It gives you the option to maintain strong consistency across multiple regions while maintaining a level of high availability.

-> [!NOTE]

-> Strong consistency across regions and a Recovery Point Objective (RPO) of zero isn't possible for native Apache Cassandra because all nodes are capable of serving writes. You can configure Azure Cosmos DB for strong consistency across regions in a *single write region* configuration. However, like with native Apache Cassandra, you can't configure an Azure Cosmos DB account that's configured with multiple write regions for strong consistency. A distributed system can't provide an RPO of zero *and* a Recovery Time Objective (RTO) of zero.

-For more information, see [Load balancing policy](https://devblogs.microsoft.com/cosmosdb/cassandra-api-java/#load-balancing-policy) in our [API for Cassandra recommendations for Java blog](https://devblogs.microsoft.com/cosmosdb/cassandra-api-java). Also, see [Failover scenarios](https://github.com/Azure-Samples/azure-cosmos-cassandra-extensions-java-sample-v4#failover-scenarios) in our official [code sample for the Cassandra Java v4 driver](https://github.com/Azure-Samples/azure-cosmos-cassandra-extensions-java-sample-v4).

-## Request units

-One of the major differences between running a native Apache Cassandra cluster and provisioning an Azure Cosmos DB account is how database capacity is provisioned. In traditional databases, capacity is expressed in terms of CPU cores, RAM, and IOPS. Azure Cosmos DB is a multi-tenant platform-as-a-service database. Capacity is expressed by using a single normalized metric called [request units](../request-units.md). Every request sent to the database has a request unit cost (RU cost). Each request can be profiled to determine its cost.

-The benefit of using request units as a metric is that database capacity can be provisioned deterministically for highly predictable performance and efficiency. After you profile the cost of each request, you can use request units to directly associate the number of requests sent to the database with the capacity you need to provision. The challenge with this way of provisioning capacity is that you need to have a solid understanding of the throughput characteristics of your workload.

-We highly recommend that you profile your requests. Use that information to help you estimate the number of request units you'll need to provision. Here are some articles that might help you make the estimate:

-## Capacity provisioning models

-In traditional database provisioning, a fixed capacity is provisioned up front to handle the anticipated throughput. Azure Cosmos DB offers a capacity-based model called [provisioned throughput](../set-throughput.md). As a multi-tenant service, Azure Cosmos DB also offers *consumption-based* models in [autoscale](../provision-throughput-autoscale.md) mode and [serverless](../serverless.md) mode. The extent to which a workload might benefit from either of these consumption-based provisioning models depends on the predictability of throughput for the workload.

-In general, steady-state workloads that have predictable throughput benefit most from provisioned throughput. Workloads that have large periods of dormancy benefit from serverless mode. Workloads that have a continuous level of minimal throughput, but with unpredictable spikes, benefit most from autoscale mode. We recommend that you review the following articles for a clear understanding of the best capacity model for your throughput needs:

-## Partitioning

-Partitioning in Azure Cosmos DB is similar to partitioning in Apache Cassandra. One of the main differences is that Azure Cosmos DB is more optimized for *horizontal scale*. In Azure Cosmos DB, limits are placed on the amount of *vertical throughput* capacity that's available in any physical partition. The effect of this optimization is most noticeable when an existing data model has significant throughput skew.

-Take steps to ensure that your partition key design results in a relatively uniform distribution of requests. For more information about how logical and physical partitioning work and limits on throughput capacity per partition, see [Partitioning in the Azure Cosmos DB for Apache Cassandra](partitioning.md).

-## Scaling

-In native Apache Cassandra, increasing capacity and scale involves adding new nodes to a cluster and ensuring that the nodes are properly added to the Cassandra ring. In Azure Cosmos DB, adding nodes is transparent and automatic. Scaling is a function of how many [request units](../request-units.md) are provisioned for your keyspace or table. Scaling in physical machines occurs when either physical storage or required throughput reaches limits allowed for a logical or a physical partition. For more information, see [Partitioning in the Azure Cosmos DB for Apache Cassandra](partitioning.md).

-## Rate limiting

-A challenge of provisioning [request units](../request-units.md), particularly if you're using [provisioned throughput](../set-throughput.md), is rate limiting. Azure Cosmos DB returns rate-limited errors if clients consume more resources than the amount you provisioned. The API for Cassandra in Azure Cosmos DB translates these exceptions to overloaded errors on the Cassandra native protocol. For information about how to avoid rate limiting in your application, see [Prevent rate-limiting errors for Azure Cosmos DB for Apache Cassandra operations](prevent-rate-limiting-errors.md).

-## Apache Spark connector

-Many Apache Cassandra users use the Apache Spark Cassandra connector to query their data for analytical and data movement needs. You can connect to the API for Cassandra the same way and by using the same connector. Before you connect to the API for Cassandra, review [Connect to the Azure Cosmos DB for Apache Cassandra from Spark](connect-spark-configuration.md). In particular, see the section [Optimize Spark connector throughput configuration](connect-spark-configuration.md#optimizing-spark-connector-throughput-configuration).

-## Troubleshoot common issues

-For solutions to common issues, see [Troubleshoot common issues in the Azure Cosmos DB for Apache Cassandra](troubleshoot-common-issues.md).

-## Next steps

-description: Learn how to use change feed in the Azure Cosmos DB for Apache Cassandra to get the changes made to your data.

-# Change feed in the Azure Cosmos DB for Apache Cassandra

-[Change feed](../change-feed.md) support in the Azure Cosmos DB for Apache Cassandra is available through the query predicates in the Cassandra Query Language (CQL). Using these predicate conditions, you can query the change feed API. Applications can get the changes made to a table using the primary key (also known as the partition key) as is required in CQL. You can then take further actions based on the results. Changes to the rows in the table are captured in the order of their modification time and the sort order per partition key.

-The following example shows how to get a change feed on all the rows in a API for Cassandra Keyspace table using .NET. The predicate COSMOS_CHANGEFEED_START_TIME() is used directly within CQL to query items in the change feed from a specified start time (in this case current datetime). You can download the full sample, for C# [here](https://github.com/azure-samples/azure-cosmos-db-cassandra-change-feed) and for Java [here](https://github.com/Azure-Samples/cosmos-changefeed-cassandra-java).

-In each iteration, the query resumes at the last point changes were read, using paging state. We can see a continuous stream of new changes to the table in the Keyspace. We will see changes to rows that are inserted, or updated. Watching for delete operations using change feed in API for Cassandra is currently not supported.

-> [!NOTE]

-> Reusing a token after dropping a collection and then recreating it with the same name results in an error.

-> We advise you to set the pageState to null when creating a new collection and reusing collection name.

-# [Java](#tab/java)

-```java

- Session cassandraSession = utils.getSession();

- try {

- DateTimeFormatter dtf = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");

- LocalDateTime now = LocalDateTime.now().minusHours(6).minusMinutes(30);

- String query="SELECT * FROM uprofile.user where COSMOS_CHANGEFEED_START_TIME()='"

- + dtf.format(now)+ "'";

-

- byte[] token=null;

- System.out.println(query);

- while(true)

- {

- SimpleStatement st=new SimpleStatement(query);

- st.setFetchSize(100);

- if(token!=null)

- st.setPagingStateUnsafe(token);

-

- ResultSet result=cassandraSession.execute(st) ;

- token=result.getExecutionInfo().getPagingState().toBytes();

-

- for(Row row:result)

- {

- System.out.println(row.getString("user_name"));

- }

- }

- } finally {

- utils.close();

- LOGGER.info("Please delete your table after verifying the presence of the data in portal or from CQL");

- }

-```

-# [C#](#tab/csharp)

-```C#

- //set initial start time for pulling the change feed

- DateTime timeBegin = DateTime.UtcNow;

- //initialise variable to store the continuation token

- byte[] pageState = null;

- while (true)

- {

- try

- {

- //Return the latest change for all rows in 'user' table

- IStatement changeFeedQueryStatement = new SimpleStatement(

- $"SELECT * FROM uprofile.user where COSMOS_CHANGEFEED_START_TIME() = '{timeBegin.ToString("yyyy-MM-ddTHH:mm:ss.fffZ", CultureInfo.InvariantCulture)}'");

- if (pageState != null)

- {

- changeFeedQueryStatement = changeFeedQueryStatement.SetPagingState(pageState);

- }

- Console.WriteLine("getting records from change feed at last page state....");

- RowSet rowSet = session.Execute(changeFeedQueryStatement);

- //store the continuation token here

- pageState = rowSet.PagingState;

- List<Row> rowList = rowSet.ToList();

- if (rowList.Count != 0)

- {

- for (int i = 0; i < rowList.Count; i++)

- {

- string value = rowList[i].GetValue<string>("user_name");

- int key = rowList[i].GetValue<int>("user_id");

- // do something with the data - e.g. compute, forward to another event, function, etc.

- // here, we just print the user name field

- Console.WriteLine("user_name: " + value);

- }

- }

- else

- {

- Console.WriteLine("zero documents read");

- }

- }

- catch (Exception e)

- {

- Console.WriteLine("Exception " + e);

- }

- }

-```

-In order to get the changes to a single row by primary key, you can add the primary key in the query. The following example shows how to track changes for the row where "user_id = 1"

-# [C#](#tab/csharp)

-```C#

- //Return the latest change for all row in 'user' table where user_id = 1

- IStatement changeFeedQueryStatement = new SimpleStatement(

- $"SELECT * FROM uprofile.user where user_id = 1 AND COSMOS_CHANGEFEED_START_TIME() = '{timeBegin.ToString("yyyy-MM-ddTHH:mm:ss.fffZ", CultureInfo.InvariantCulture)}'");

-```

-# [Java](#tab/java)

-```java

- String query="SELECT * FROM uprofile.user where user_id=1 and COSMOS_CHANGEFEED_START_TIME()='"

- + dtf.format(now)+ "'";

- SimpleStatement st=new SimpleStatement(query);

-```

-## Current limitations

-The following limitations are applicable when using change feed with API for Cassandra:

-* Inserts and updates are currently supported. Delete operation is not yet supported. As a workaround, you can add a soft marker on rows that are being deleted. For example, add a field in the row called "deleted" and set it to "true".

-* Last update is persisted as in core API for NoSQL and intermediate updates to the entity are not available.

-## Error handling

-The following error codes and messages are supported when using change feed in API for Cassandra:

-* **HTTP error code 429** - When the change feed is rate limited, it returns an empty page.

-## Next steps

-* [Manage Azure Cosmos DB for Apache Cassandra resources using Azure Resource Manager templates](templates-samples.md)

-description: Learn about the differences between Azure Managed Instance for Apache Cassandra and Azure Cosmos DB for Apache Cassandra. You also learn the benefits of each of these services and when to choose them.

-# Differences between Azure Managed Instance for Apache Cassandra and Azure Cosmos DB for Apache Cassandra

-In this article, you will learn the differences between [Azure Managed Instance for Apache Cassandra](../../managed-instance-apache-cassandr) Azure Cosmos DB for Apache Cassandra. This article provides recommendations on how to choose between the two services, or when to host your own Apache Cassandra environment.

-## Key differences

-Azure Managed Instance for Apache Cassandra is a fully managed service for pure open-source Apache Cassandra clusters. The service also allows configurations to be overridden, depending on the specific needs of each workload, allowing maximum flexibility and control where needed. It also provides the capability to scale out the capacity of existing on-premises or cloud self-hosted Apache Cassandra clusters. It scales out by adding managed Cassandra datacenters to the existing cluster ring.

-The RU-based [Azure Cosmos DB for Apache Cassandra](introduction.md) in Azure Cosmos DB is a compatibility layer over Microsoft's globally distributed cloud-native database service [Azure Cosmos DB](../index.yml).

-## How to choose?

-The following table shows the common scenarios, workload requirements, and aspirations where each of this deployment approaches fit:

-| |Self-hosted Apache Cassandra on-premises or in Azure | Azure Managed Instance for Apache Cassandra | Azure Cosmos DB for Apache Cassandra |

-|||||

-|**Deployment type**| You have a highly customized Apache Cassandra deployment with custom patches or snitches. | You have a standard open-source Apache Cassandra deployment without any custom code. | You are content with a platform that is not Apache Cassandra underneath but is compliant with all open-source client drivers at a [wire protocol](../cassandra-support.md) level. |

-|**Operational overhead**| You have existing Cassandra experts who can deploy, configure, and maintain your clusters. | You want to eliminate the operational overhead by using a fully managed Database-as-as-Service for open-source Apache Cassandra, but have the option of controling Cassandra-specific configurations such as replication and consistency when required. | You want to eliminate the operational overhead by using a fully managed Platform-as-as-service database in the cloud. |

-|**Production Support**| You handle live incidents and outages yourself, including contacting relevant infrastructure teams for compute, networking, storage, etc. | You want a first-party managed service experience that will act as a one-stop shop for supporting live incidents and outages. | You want a first-party managed service experience that will act as a one-stop shop for live incidents and outages. |

-|**Software Support**| You handle all patches, and ensure that software is upgraded before end of life.| You want a first-party managed service experience that will offer Cassandra software level support beyond end of live, automated patching, and turnkey upgrades for major versions | You want a first-party managed service experience where software level support is completely abstracted.|

-|**Operating system requirements**| You have a requirement to maintain custom or golden Virtual Machine operating system images. | You can use vanilla images but want to have control over the selection of SKUs, memory, disks, and IOPS. | You want capacity provisioning to be simplified and expressed as a single normalized metric, with a one-to-one relationship to throughput, such as [request units](../request-units.md) in Azure Cosmos DB. |

-|**Pricing model**| You want to use management software such as Datastax tooling and are happy with licensing costs. | You prefer pure open-source licensing and VM instance-based pricing. | You want to use cloud-native pricing, which includes [autoscale](scale-account-throughput.md#use-autoscale) and [serverless](../serverless.md) offers. |

-|**Analytics**| You want full control over the provisioning of analytical pipelines regardless of the overhead to build and maintain them. | You want to use cloud-based analytical services like Azure Databricks. | You want near real-time hybrid transactional analytics built into the platform with [Azure Synapse Link for Azure Cosmos DB](../synapse-link.md). |

-|**Workload pattern**| Your workload is fairly steady-state and you don't require scaling nodes in the cluster frequently. | Your workload is volatile and you need to be able to scale up or scale down nodes in a data center or add/remove data centers easily. | Your workload is often volatile and you need to be able to scale up or scale down quickly and at a significant volume. |

-|**SLAs**| You are happy with your processes for maintaining SLAs on consistency, throughput, availability, and disaster recovery. | You are happy with your processes for maintaining SLAs on consistency and throughput, but want an [SLA for availability](https://azure.microsoft.com/support/legal/sl#backup-and-restore). | You want [fully comprehensive SLAs](https://azure.microsoft.com/support/legal/sla/cosmos-db/v1_4/) on consistency, throughput, availability, and disaster recovery. |

-|**Replication and consistency**| You need to be able to configure the full array of [tunable consistency settings](https://cassandra.apache.org/doc/latest/cassandr)) |

-|**Data model**| You are migrating workloads which have a mixture of uniform distribution of data, and skewed data (with respect to both storage and throughput across partition keys) requiring flexibility on vertical scale of nodes. | You are migrating workloads which have a mixture of uniform distribution of data, and skewed data (with respect to both storage and throughput across partition keys) requiring flexibility on vertical scale of nodes. | You are building a new application, or your existing application has a relatively uniform distribution of data with respect to both storage and throughput across partition keys. |

-## Next steps

-* [Build a Java app to manage Azure Cosmos DB for Apache Cassandra data](manage-data-java-v4-sdk.md)

-* [Create an Azure Managed Instance for Apache Cassandra cluster in Azure portal](../../managed-instance-apache-cassandr)

-description: Azure CLI Samples for Azure Cosmos DB for Apache Cassandra

-# Azure CLI samples for Azure Cosmos DB for Apache Cassandra

-The following tables include links to sample Azure CLI scripts for the Azure Cosmos DB for Apache Cassandra and to sample Azure CLI scripts that apply to all Azure Cosmos DB APIs. Common samples are the same across all APIs.

-These samples require Azure CLI version 2.12.1 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI](/cli/azure/install-azure-cli). If using Azure Cloud Shell, the latest version is already installed.

-## API for Cassandra Samples

-|Task | Description |

-|||

-| [Create an Azure Cosmos DB account, keyspace and table](../scripts/cli/cassandr)| Creates an Azure Cosmos DB account, keyspace, and table for API for Cassandra. |

-| [Create a serverless Azure Cosmos DB account for API for Cassandra, keyspace and table](../scripts/cli/cassandr)| Creates a serverless Azure Cosmos DB account, keyspace, and table for API for Cassandra. |

-| [Create an Azure Cosmos DB account, keyspace and table with autoscale](../scripts/cli/cassandr)| Creates an Azure Cosmos DB account, keyspace, and table with autoscale for API for Cassandra. |

-| [Perform throughput operations](../scripts/cli/cassandr) | Read, update and migrate between autoscale and standard throughput on a keyspace and table.|

-| [Lock resources from deletion](../scripts/cli/cassandr)| Prevent resources from being deleted with resource locks.|

-|||

-## Common API Samples

-These samples apply to all Azure Cosmos DB APIs. These samples use a API for NoSQL account, but these operations are identical across all database APIs in Azure Cosmos DB.

-|Task | Description |

-|||

-| [Add or fail over regions](../scripts/cli/common/regions.md) | Add a region, change failover priority, trigger a manual failover.|

-| [Perform account key operations](../scripts/cli/common/keys.md) | List account keys, read-only keys, regenerate keys and list connection strings.|

-| [Secure with IP firewall](../scripts/cli/common/ipfirewall.md)| Create an Azure Cosmos DB account with IP firewall configured.|

-| [Secure new account with service endpoints](../scripts/cli/common/service-endpoints.md)| Create an Azure Cosmos DB account and secure with service-endpoints.|

-| [Secure existing account with service endpoints](../scripts/cli/common/service-endpoints-ignore-missing-vnet.md)| Update an Azure Cosmos DB account to secure with service-endpoints when the subnet is eventually configured.|

-| [Find existing free-tier account](../scripts/cli/common/free-tier.md)| Find whether there is an existing free-tier account in your subscription.|

-|||

-## Next steps

-Reference pages for all Azure Cosmos DB CLI commands are available in the [Azure CLI Reference](/cli/azure/cosmosdb).

-For Azure CLI samples for other APIs see:

-description: This article is the main page for Azure Cosmos DB for Apache Cassandra integration from Spark.

-# Connect to Azure Cosmos DB for Apache Cassandra from Spark

-This article is one among a series of articles on Azure Cosmos DB for Apache Cassandra integration from Spark. The articles cover connectivity, Data Definition Language(DDL) operations, basic Data Manipulation Language(DML) operations, and advanced Azure Cosmos DB for Apache Cassandra integration from Spark.

-## Prerequisites

-* [Provision an Azure Cosmos DB for Apache Cassandra account.](manage-data-dotnet.md#create-a-database-account)

-* Provision your choice of Spark environment [[Azure Databricks](/azure/databricks/scenarios/quickstart-create-databricks-workspace-portal) | [Azure HDInsight-Spark](../../hdinsight/spark/apache-spark-jupyter-spark-sql.md) | Others].

-## Dependencies for connectivity

-* **Spark connector for Cassandra:**

- Spark connector is used to connect to Azure Cosmos DB for Apache Cassandra. Identify and use the version of the connector located in [Maven central](https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector-assembly) that is compatible with the Spark and Scala versions of your Spark environment. We recommend an environment that supports Spark 3.2.1 or higher, and the spark connector available at maven coordinates `com.datastax.spark:spark-cassandra-connector-assembly_2.12:3.2.0`. If using Spark 2.x, we recommend an environment with Spark version 2.4.5, using spark connector at maven coordinates `com.datastax.spark:spark-cassandra-connector_2.11:2.4.3`.

-* **Azure Cosmos DB helper library for API for Cassandra:**

- If you're using a version Spark 2.x, then in addition to the Spark connector, you need another library called [azure-cosmos-cassandra-spark-helper]( https://search.maven.org/artifact/com.microsoft.azure.cosmosdb/azure-cosmos-cassandra-spark-helper/1.2.0/jar) with maven coordinates `com.microsoft.azure.cosmosdb:azure-cosmos-cassandra-spark-helper:1.2.0` from Azure Cosmos DB in order to handle [rate limiting](./scale-account-throughput.md#handling-rate-limiting-429-errors). This library contains custom connection factory and retry policy classes.

- The retry policy in Azure Cosmos DB is configured to handle HTTP status code 429("Request Rate Large") exceptions. The Azure Cosmos DB for Apache Cassandra translates these exceptions into overloaded errors on the Cassandra native protocol, and you can retry with back-offs. Because Azure Cosmos DB uses provisioned throughput model, request rate limiting exceptions occur when the ingress/egress rates increase. The retry policy protects your spark jobs against data spikes that momentarily exceed the throughput allocated for your container. If using the Spark 3.x connector, implementing this library isn't required.

- > [!NOTE]

- > The retry policy can protect your spark jobs against momentary spikes only. If you have not configured enough RUs required to run your workload, then the retry policy is not applicable and the retry policy class rethrows the exception.

-* **Azure Cosmos DB account connection details:** Your Azure API for Cassandra account name, account endpoint, and key.

-## Optimizing Spark connector throughput configuration

-Listed in the next section are all the relevant parameters for controlling throughput using the Spark Connector for Cassandra. In order to optimize parameters to maximize throughput for spark jobs, the `spark.cassandra.output.concurrent.writes`, `spark.cassandra.concurrent.reads`, and `spark.cassandra.input.reads_per_sec` configs needs to be correctly configured, in order to avoid too much throttling and back-off (which in turn can lead to lower throughput).

-The optimal value of these configurations depends on four factors:

-As an example, if we have five workers and a value of `spark.cassandra.output.concurrent.writes`= 1, and a value of `spark.cassandra.connection.remoteConnectionsPerExecutor` = 1, then we have five workers that are concurrently writing into the table, each with one thread. If it takes 10 ms to perform a single write, then we can send 100 requests (1000 milliseconds divided by 10) per second, per thread. With five workers, this would be 500 writes per second. At an average cost of five request units (RUs) per write, the target table would need a minimum 2500 request units provisioned (5 RUs x 500 writes per second).

-Increasing the number of executors can increase the number of threads in a given job, which can in turn increase throughput. However, the exact impact of this can be variable depending on the job, while controlling throughput with number of workers is more deterministic. You can also determine the exact cost of a given request by profiling it to get the Request Unit (RU) charge. This will help you to be more accurate when provisioning throughput for your table or keyspace. Have a look at our article [here](./find-request-unit-charge.md) to understand how to get request unit charges at a per request level.

-### Scaling throughput in the database

-The Cassandra Spark connector will saturate throughput in Azure Cosmos DB efficiently. As a result, even with effective retries, you'll need to ensure you have sufficient throughput (RUs) provisioned at the table or keyspace level to prevent rate limiting related errors. The minimum setting of 400 RUs in a given table or keyspace won't be sufficient. Even at minimum throughput configuration settings, the Spark connector can write at a rate corresponding to around **6000 request units** or more.

-If the RU setting required for data movement using Spark is higher than what is required for your steady state workload, you can easily scale throughput up and down systematically in Azure Cosmos DB to meet the needs of your workload for a given time period. Read our article on [elastic scale in API for Cassandra](scale-account-throughput.md) to understand the different options for scaling programmatically and dynamically.

-> [!NOTE]

-> The guidance above assumes a reasonably uniform distribution of data. If you have a significant skew in the data (that is, an inordinately large number of reads/writes to the same partition key value), then you might still experience bottlenecks, even if you have a large number of [request units](../request-units.md) provisioned in your table. Request units are divided equally among physical partitions, and heavy data skew can cause a bottleneck of requests to a single partition.

-

-## Spark connector throughput configuration parameters

-The following table lists Azure Cosmos DB for Apache Cassandra-specific throughput configuration parameters provided by the connector. For a detailed list of all configuration parameters, see [configuration reference](https://github.com/datastax/spark-cassandra-connector/blob/master/doc/reference.md) page of the Spark Cassandra Connector GitHub repository.

-| **Property Name** | **Default value** | **Description** |

-||||

-| spark.cassandra.output.batch.size.rows | 1 |Number of rows per single batch. Set this parameter to 1. This parameter is used to achieve higher throughput for heavy workloads. |

-| spark.cassandra.connection.connections_per_executor_max (Spark 2.x) spark.cassandra.connection.remoteConnectionsPerExecutor (Spark 3.x) | None | Maximum number of connections per node per executor. 10*n is equivalent to 10 connections per node in an n-node Cassandra cluster. So, if you require five connections per node per executor for a five node Cassandra cluster, then you should set this configuration to 25. Modify this value based on the degree of parallelism or the number of executors that your spark jobs are configured for. |

-| spark.cassandra.output.concurrent.writes | 100 | Defines the number of parallel writes that can occur per executor. Because you set "batch.size.rows" to 1, make sure to scale up this value accordingly. Modify this value based on the degree of parallelism or the throughput that you want to achieve for your workload. |

-| spark.cassandra.concurrent.reads | 512 | Defines the number of parallel reads that can occur per executor. Modify this value based on the degree of parallelism or the throughput that you want to achieve for your workload |

-| spark.cassandra.output.throughput_mb_per_sec | None | Defines the total write throughput per executor. This parameter can be used as an upper limit for your spark job throughput, and base it on the provisioned throughput of your Azure Cosmos DB container. |

-| spark.cassandra.input.reads_per_sec| None | Defines the total read throughput per executor. This parameter can be used as an upper limit for your spark job throughput, and base it on the provisioned throughput of your Azure Cosmos DB container. |

-| spark.cassandra.output.batch.grouping.buffer.size | 1000 | Defines the number of batches per single spark task that can be stored in memory before sending to API for Cassandra |

-| spark.cassandra.connection.keep_alive_ms | 60000 | Defines the period of time until which unused connections are available. |

-Adjust the throughput and degree of parallelism of these parameters based on the workload you expect for your spark jobs, and the throughput you've provisioned for your Azure Cosmos DB account.

-## <a id="connecting-to-azure-cosmos-db-cassandra-api-from-spark"></a>Connecting to Azure Cosmos DB for Apache Cassandra from Spark

-### cqlsh

-The following commands detail how to connect to Azure Cosmos DB for Apache Cassandra from cqlsh. This is useful for validation as you run through the samples in Spark.<br>

-**From Linux/Unix/Mac:**

-```bash

-export SSL_VERSION=TLSv1_2

-export SSL_VALIDATE=false

-cqlsh.py YOUR-COSMOSDB-ACCOUNT-NAME.cassandra.cosmosdb.azure.com 10350 -u YOUR-COSMOSDB-ACCOUNT-NAME -p YOUR-COSMOSDB-ACCOUNT-KEY --ssl

-```

-### 1. Azure Databricks

-The article below covers Azure Databricks cluster provisioning, cluster configuration for connecting to Azure Cosmos DB for Apache Cassandra, and several sample notebooks that cover DDL operations, DML operations and more.<BR>

-[Work with Azure Cosmos DB for Apache Cassandra from Azure Databricks](spark-databricks.md)<BR>

-

-### 2. Azure HDInsight-Spark

-The article below covers HDinsight-Spark service, provisioning, cluster configuration for connecting to Azure Cosmos DB for Apache Cassandra, and several sample notebooks that cover DDL operations, DML operations and more.<BR>

-[Work with Azure Cosmos DB for Apache Cassandra from Azure HDInsight-Spark](spark-hdinsight.md)

-

-### 3. Spark environment in general

-While the sections above were specific to Azure Spark-based PaaS services, this section covers any general Spark environment. Connector dependencies, imports, and Spark session configuration are detailed below. The "Next steps" section covers code samples for DDL operations, DML operations and more.

-#### Connector dependencies:

-1. Add the maven coordinates to get the [Cassandra connector for Spark](connect-spark-configuration.md#dependencies-for-connectivity)

-2. Add the maven coordinates for the [Azure Cosmos DB helper library](connect-spark-configuration.md#dependencies-for-connectivity) for API for Cassandra

-#### Imports:

-```scala

-import org.apache.spark.sql.cassandra._

-//Spark connector

-import com.datastax.spark.connector._

-import com.datastax.spark.connector.cql.CassandraConnector

-//CosmosDB library for multiple retry

-import com.microsoft.azure.cosmosdb.cassandra

-```

-#### Spark session configuration:

-```scala

- spark.cassandra.connection.host YOUR_ACCOUNT_NAME.cassandra.cosmosdb.azure.com

- spark.cassandra.connection.port 10350

- spark.cassandra.connection.ssl.enabled true

- spark.cassandra.auth.username YOUR_ACCOUNT_NAME

- spark.cassandra.auth.password YOUR_ACCOUNT_KEY

-// if using Spark 2.x

-// spark.cassandra.connection.factory com.microsoft.azure.cosmosdb.cassandra.CosmosDbConnectionFactory

-//Throughput-related...adjust as needed

- spark.cassandra.output.batch.size.rows 1

-// spark.cassandra.connection.connections_per_executor_max 10 // Spark 2.x

- spark.cassandra.connection.remoteConnectionsPerExecutor 10 // Spark 3.x

- spark.cassandra.output.concurrent.writes 1000

- spark.cassandra.concurrent.reads 512

- spark.cassandra.output.batch.grouping.buffer.size 1000

- spark.cassandra.connection.keep_alive_ms 600000000

-```

-## Next steps

-The following articles demonstrate Spark integration with Azure Cosmos DB for Apache Cassandra.

-

-* [DDL operations](spark-ddl-operations.md)

-* [Create/insert operations](spark-create-operations.md)

-* [Read operations](spark-read-operation.md)

-* [Upsert operations](spark-upsert-operations.md)

-* [Delete operations](spark-delete-operation.md)

-* [Aggregation operations](spark-aggregation-operations.md)

-* [Table copy operations](spark-table-copy-operations.md)

-description: Apache Cassandra and Azure Cosmos DB consistency levels.

-# Apache Cassandra and Azure Cosmos DB for Apache Cassandra consistency levels

-Unlike Azure Cosmos DB, Apache Cassandra doesn't natively provide precisely defined consistency guarantees. Instead, Apache Cassandra provides a write consistency level and a read consistency level, to enable the high availability, consistency, and latency tradeoffs. When using Azure Cosmos DB for Cassandra:

-## Multi-region writes vs single-region writes

-Apache Cassandra database is a multi-master system by default, and doesn't provide an out-of-box option for single-region writes with multi-region replication for reads. However, Azure Cosmos DB provides turnkey ability to have either single region, or [multi-region](../how-to-multi-master.md) write configurations. One of the advantages of being able to choose a single region write configuration across multiple regions is the avoidance of cross-region conflict scenarios, and the option of maintaining strong consistency across multiple regions.

-With single-region writes, you can maintain strong consistency, while still maintaining a level of high availability across regions with [service-managed failover](../high-availability.md#region-outages). In this configuration, you can still exploit data locality to reduce read latency by downgrading to eventual consistency on a per request basis. In addition to these capabilities, the Azure Cosmos DB platform also offers the option of [zone redundancy](/azure/architecture/reliability/architect) when selecting a region. Thus, unlike native Apache Cassandra, Azure Cosmos DB allows you to navigate the CAP Theorem [trade-off spectrum](../consistency-levels.md#rto) with more granularity.

-## Mapping consistency levels

-The Azure Cosmos DB platform provides a set of five well-defined, business use-case oriented consistency settings with respect to replication. The tradeoffs to these consistency settings are defined by the [CAP](https://en.wikipedia.org/wiki/CAP_theorem) and [PACLC](https://en.wikipedia.org/wiki/PACELC_theorem) theorems. As this approach differs significantly from Apache Cassandra, we would recommend that you take time to review and understand [Azure Cosmos DB consistency](../consistency-levels.md). Alternatively, you can review this short [video guide to understanding consistency settings](https://aka.ms/docs.consistency-levels) in the Azure Cosmos DB platform. The following table illustrates the possible mappings between Apache Cassandra and Azure Cosmos DB consistency levels when using API for Cassandra. This table shows configurations for single region, multi-region reads with single-region writes, and multi-region writes.

-### Mappings

-> [!NOTE]

-> These are not exact mappings. Rather, we have provided the closest analogues to Apache Cassandra, and disambiguated any qualitative differences in the rightmost column. As mentioned above, we recommend reviewing Azure Cosmos DB's [consistency settings](../consistency-levels.md).

-### `ALL`, `EACH_QUOROM`, `QUOROM`, `LOCAL_QUORUM`, or `THREE` write consistency in Apache Cassandra

-| Apache read consistency | Reading from | Closest Azure Cosmos DB consistency level to Apache Cassandra read/write settings |

-| | | |

-| `ALL` | Local region | `Strong` |

-| `EACH_QUOROM` | Local region | `Strong` |

-| `QUOROM` | Local region | `Strong` |

-| `LOCAL_QUORUM` | Local region | `Strong` |

-| `LOCAL_ONE` | Local region | `Eventual` |

-| `ONE` | Local region | `Eventual` |

-| `TWO` | Local region | `Strong` |

-| `THREE` | Local region | `Strong` |

-Unlike Apache and DSE Cassandra, Azure Cosmos DB durably commits a quorum write by default. At least three out of four (3/4) nodes commit the write to disk, and NOT just an in-memory commit log.

-### `ONE`, `LOCAL_ONE`, or `ANY` write consistency in Apache Cassandra

-| Apache read consistency | Reading from | Closest Azure Cosmos DB consistency level to Apache Cassandra read/write settings |

-| | | |

-| `ALL` | Local region | `Strong` |

-| `EACH_QUOROM` | Local region | `Eventual` |

-| `QUOROM` | Local region | `Eventual` |

-| `LOCAL_QUORUM` | Local region | `Eventual` |

-| `LOCAL_ONE` | Local region | `Eventual` |

-| `ONE` | Local region | `Eventual` |

-| `TWO` | Local region | `Eventual` |

-| `THREE` | Local region | `Eventual` |

-Azure Cosmos DB API for Cassandra always durably commits a quorum write by default, hence all read consistencies can be made use of.

-### `TWO` write consistency in Apache Cassandra

-| Apache read consistency | Reading from | Closest Azure Cosmos DB consistency level to Apache Cassandra read/write settings |

-| | | |

-| `ALL` | Local region | `Strong` |

-| `EACH_QUOROM` | Local region | `Strong` |

-| `QUOROM` | Local region | `Strong` |

-| `LOCAL_QUORUM` | Local region | `Strong` |

-| `LOCAL_ONE` | Local region | `Eventual` |

-| `ONE` | Local region | `Eventual` |

-| `TWO` | Local region | `Eventual` |

-| `THREE` | Local region | `Strong` |

-Azure Cosmos DB has no notion of write consistency to only two nodes, hence we treat this consistency similar to quorum for most cases. For read consistency `TWO`, this consistency is equivalent to write with `QUOROM` and read from `ONE`.

-### `Serial`, or `Local_Serial` write consistency in Apache Cassandra

-| Apache read consistency | Reading from | Closest Azure Cosmos DB consistency level to Apache Cassandra read/write settings |

-| | | |

-| `ALL` | Local region | `Strong` |

-| `EACH_QUOROM` | Local region | `Strong` |

-| `QUOROM` | Local region | `Strong` |

-| `LOCAL_QUORUM` | Local region | `Strong` |

-| `LOCAL_ONE` | Local region | `Eventual` |

-| `ONE` | Local region | `Eventual` |

-| `TWO` | Local region | `Strong` |

-| `THREE` | Local region | `Strong` |

-Serial only applies to lightweight transactions. Azure Cosmos DB follows a [durably committed algorithm](https://www.microsoft.com/research/publication/revisiting-paxos-algorithm/) by default, and hence `Serial` consistency is similar to quorum.

-### Other regions for single-region write

-Azure Cosmos DB facilitates five consistency settings, including strong, across multiple regions where single-region writes is configured. This facilitation occurs as long as regions are within 2,000 miles of each other.

-Azure Cosmos DB doesn't have an applicable mapping to Apache Cassandra as all nodes/regions are writes and a strong consistency guarantee isn't possible across all regions.

-### Other regions for multi-region write

-Azure Cosmos DB facilitates only four consistency settings; `eventual`, `consistent prefix`, `session`, and `bounded staleness` across multiple regions where multi-region write is configured.

-Apache Cassandra would only provide eventual consistency for reads across other regions regardless of settings.

-### Dynamic overrides supported

-| Azure Cosmos DB account setting | Override value in client request | Override effect |

-| | | |

-| `Strong` | `All` | No effect (remain as `strong`) |

-| `Strong` | `Quorum` | No effect (remain as `strong`) |

-| `Strong` | `LocalQuorum` | No effect (remain as `strong`) |

-| `Strong` | `Two` | No effect (remain as `strong`) |

-| `Strong` | `Three` | No effect (remain as `strong`) |

-| `Strong` | `Serial` | No effect (remain as `strong`) |

-| `Strong` | `LocalSerial` | No effect (remain as `strong`) |

-| `Strong` | `One` | Consistency changes to `Eventual` |

-| `Strong` | `LocalOne` | Consistency changes to `Eventual` |

-| `Strong` | `Any` | Not allowed (error) |

-| `Strong` | `EachQuorum` | Not allowed (error) |

-| `Bounded staleness`, `session`, or `consistent prefix` | `All` | Not allowed (error) |

-| `Bounded staleness`, `session`, or `consistent prefix` | `Quorum` | Not allowed (error) |

-| `Bounded staleness`, `session`, or `consistent prefix` | `LocalQuorum` | Not allowed (error) |

-| `Bounded staleness`, `session`, or `consistent prefix` | `Two` | Not allowed (error) |

-| `Bounded staleness`, `session`, or `consistent prefix` | `Three` | Not allowed (error) |

-| `Bounded staleness`, `session`, or `consistent prefix` | `Serial` | Not allowed (error) |

-| `Bounded staleness`, `session`, or `consistent prefix` | `LocalSerial` | Not allowed (error) |

-| `Bounded staleness`, `session`, or `consistent prefix` | `One` | Consistency changes to `Eventual` |

-| `Bounded staleness`, `session`, or `consistent prefix` | `LocalOne` | Consistency changes to `Eventual` |

-| `Bounded staleness`, `session`, or `consistent prefix` | `Any` | Not allowed (error) |

-| `Bounded staleness`, `session`, or `consistent prefix` | `EachQuorum` | Not allowed (error) |

-### Metrics

-If your Azure Cosmos DB account is configured with a consistency level other than the strong consistency, review the *Probabilistically Bounded Staleness (PBS)* metric. The metric captures the probability that your clients may get strong and consistent reads for your workloads. This metric is exposed in the Azure portal. To find more information about the PBS metric, see [Monitor Probabilistically Bounded Staleness (PBS) metric](../how-to-manage-consistency.md#monitor-probabilistically-bounded-staleness-pbs-metric).

-Probabilistically bounded staleness shows how eventual is your eventual consistency. This metric provides an insight into how often you can get a stronger consistency than the consistency level that you've currently configured on your Azure Cosmos DB account. In other words, you can see the probability (measured in milliseconds) of getting consistent reads for a combination of write and read regions.

-## Global strong consistency for write requests in Apache Cassandra

-Apache Cassandra, the setting of `EACH_QUORUM` or `QUORUM` gives a strong consistency. When a write request is sent to a region, `EACH_QUORUM` persists the data in a quorum number of nodes in each data center. This persistence requires every data center to be available for the write operation to succeed. `QUORUM` is slightly less restrictive, with a `QUORUM` number of nodes across all the data centers needed to persist the data prior to acknowledging the write to be successful.

-The following graphic illustrates a global strong consistency setting in Apache Cassandra between two regions 1 and 2. After data is written to region 1, the write needs to be persisted in a quorum number of nodes in both region 1, and region 2 before an acknowledgment is received by the application.

-## Global strong consistency for write requests in Azure Cosmos DB for Apache Cassandra

-In Azure Cosmos DB consistency is set at the account level. With `Strong` consistency in Azure Cosmos DB for Cassandra, data is replicated synchronously to the read regions for the account. The further apart the regions for the Azure Cosmos DB account are, the higher the latency of the consistent write operations.

-How the number of regions affects your read or write request:

-> [!NOTE]

-> If a globally strong consistency is required for all write operations, the consistency for Azure Cosmos DB for Cassandra account must be set to Strong. The consistency level for write operations cannot be overridden to a lower consistency level on a per request basis in Azure Cosmos DB.

-## Weaker consistency for write requests in Apache Cassandra

-A consistency level of `ANY`, `ONE`, `TWO`, `THREE`, `LOCAL_QUORUM`, `Serial` or `Local_Serial`? Consider a write request with `LOCAL_QUORUM` with an `RF` of `4` in a six-node datacenter. `Quorum = 4/2 + 1 = 3`.

-## Weaker consistency for write requests in Azure Cosmos DB for Apache Cassandra

-When a write request is sent with any of the consistency levels lower than `Strong`, a success response is returned as soon as the local region persists the write in at least three out of four replicas.

-## Global strong consistency for read requests in Apache Cassandra

-With a consistency of `EACH_QUORUM`, a consistent read can be achieved in Apache Cassandra. In, a multi-region setup for `EACH_QUORUM` if the quorum number of nodes isn't met in each region, then the read will be unsuccessful.

-## Global strong consistency for read requests in Azure Cosmos DB for Apache Cassandra

-The read request is served from two replicas in the specified region. Since the write already took care of persisting to a quorum number of regions (and all regions if every region was available), simply reading from two replicas in the specified region provides Strong consistency. This strong consistency requires `EACH_QUORUM` to be specified in the driver when issuing the read against a region for the Cosmos DB account along with Strong Consistency as the default consistency level for the account.

-## Local strong consistency in Apache Cassandra

-A read request with a consistency level of `TWO`, `THREE`, or `LOCAL_QUORUM` will give us strong consistency reading from local region. With a consistency level of `LOCAL_QUORUM`, you need a response from two nodes in the specified datacenter for a successful read.

-## Local strong consistency in Azure Cosmos DB for Apache Cassandra

-In Azure Cosmos DB for Cassandra, having a consistency level of `TWO`, `THREE` or `LOCAL_QUORUM` will give a local strong consistency for a read request. Since the write path guarantees replicating to a minimum of three out of four replicas, a read from two replicas in the specified region will guarantee a quorum read of the data in that region.

-## Eventual consistency in Apache Cassandra

-A consistency level of `LOCAL_ONE`, `One` and `ANY with LOCAL_ONE` will result in eventual consistency. This consistency is used in cases where the focus is on latency.

-## Eventual consistency in Azure Cosmos DB for Apache Cassandra?

-A consistency level of `LOCAL_ONE`, `ONE` or `Any` will give you eventual consistency. With eventual consistency, a read is served from just one of the replicas in the specified region.

-## Override consistency level for read operations in Azure Cosmos DB for Cassandra

-Previously, the consistency level for read requests could only be overridden to a lower consistency than the default set on the account. For instance, with the default consistency of Strong, read requests could be issued with Strong by default and overridden on a per request basis (if needed) to a consistency level weaker than Strong. However, read requests couldn't be issued with an overridden consistency level higher than the accountΓÇÖs default. An account with Eventual consistency couldn't receive read requests with a consistency level higher than Eventual (which in the Apache Cassandra drivers translate to `TWO`, `THREE`, `LOCAL_QUORUM` or `QUORUM`).

-Azure Cosmos DB for Cassandra now facilitates overriding the consistency on read requests to a value higher than the accountΓÇÖs default consistency. For instance, with the default consistency on the Cosmos DB account set to Eventual (Apache Cassandra equivalent of `One` or `ANY`), read requests can be overridden on a per request basis to `LOCAL_QUORUM`. This override ensures that a quorum number of replicas within the specified region are consulted prior to returning the result set, as required by `LOCAL_QUORUM`.

-This option also prevents the need to set a default consistency that is higher than `Eventual`, when it's only needed for read requests.

-## Next steps

-Learn more about global distribution and consistency levels for Azure Cosmos DB:

-description: This tutorial shows how to create a API for Cassandra account, add a database (also called a keyspace), and add a table to that account by using a Java application.

-#Customer intent: As a developer, I want to build a Java application to access and manage Azure Cosmos DB resources so that customers can store key/value data and utilize the global distribution, elastic scaling, multi-region writes, and other capabilities offered by Azure Cosmos DB.

-# Tutorial: Create a API for Cassandra account in Azure Cosmos DB by using a Java application to store key/value data

-As a developer, you might have applications that use key/value pairs. You can use a API for Cassandra account in Azure Cosmos DB to store the key/value data. This tutorial describes how to use a Java application to create a API for Cassandra account in Azure Cosmos DB, add a database (also called a keyspace), and add a table. The Java application uses the [Java driver](https://github.com/datastax/java-driver) to create a user database that contains details such as user ID, user name, and user city.

-This tutorial covers the following tasks:

-> [!div class="checklist"]

-> * Create a Cassandra database account

-> * Get the account connection string

-> * Create a Maven project and dependencies

-> * Add a database and a table

-> * Run the app

-## Prerequisites

-* If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio) before you begin.

-* Get the latest version of [Java Development Kit (JDK)](/java/azure/jdk/).

-* [Download](https://maven.apache.org/download.cgi) and [install](https://maven.apache.org/install.html) the [Maven](https://maven.apache.org/) binary archive.

- - On Ubuntu, you can run `apt-get install maven` to install Maven.

-## Create a database account

-## Get the connection details of your account

-Get the connection string information from the Azure portal, and copy it into the Java configuration file. The connection string enables your app to communicate with your hosted database.

-1. From the [Azure portal](https://portal.azure.com/), go to your Azure Cosmos DB account.

-2. Open the **Connection String** pane.

-3. Copy the **CONTACT POINT**, **PORT**, **USERNAME**, and **PRIMARY PASSWORD** values to use in the next steps.

-## Create the project and the dependencies

-The Java sample project that you use in this article is hosted in GitHub. You can run the steps in this doc or download the sample from the [azure-cosmos-db-cassandra-java-getting-started](https://github.com/Azure-Samples/azure-cosmos-db-cassandra-java-getting-started) repository.

-After you download the files, update the connection string information within the `java-examples\src\main\resources\config.properties` file and run it.

-```java

-cassandra_host=<FILLME_with_CONTACT POINT>

-cassandra_port = 10350

-cassandra_username=<FILLME_with_USERNAME>

-cassandra_password=<FILLME_with_PRIMARY PASSWORD>

-```

-Use the following steps to build the sample from scratch:

-1. From the terminal or command prompt, create a new Maven project called Cassandra-demo.

- ```bash

- mvn archetype:generate -DgroupId=com.azure.cosmosdb.cassandra -DartifactId=cassandra-demo -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

- ```

-

-2. Locate the `cassandra-demo` folder. Using a text editor, open the `pom.xml` file that was generated.

- Add the Cassandra dependencies and build plugins required by your project, as shown in the [pom.xml](https://github.com/Azure-Samples/azure-cosmos-db-cassandra-java-getting-started/blob/main/pom.xml) file.

-3. Under the `cassandra-demo\src\main` folder, create a new folder named `resources`. Under the resources folder, add the config.properties and log4j.properties files:

- - The [config.properties](https://github.com/Azure-Samples/azure-cosmos-db-cassandra-java-getting-started/blob/main/src/main/resources/config.properties) file stores the connection endpoint and key values of the API for Cassandra account.

-

- - The [log4j.properties](https://github.com/Azure-Samples/azure-cosmos-db-cassandra-java-getting-started/blob/main/src/main/resources/log4j.properties) file defines the level of logging required for interacting with the API for Cassandra.

-4. Browse to the `src/main/java/com/azure/cosmosdb/cassandra/` folder. Within the cassandra folder, create another folder named `utils`. The new folder stores the utility classes required to connect to the API for Cassandra account.

- Add the [CassandraUtils](https://github.com/Azure-Samples/azure-cosmos-db-cassandra-java-getting-started/blob/main/src/main/java/com/azure/cosmosdb/cassandra/util/CassandraUtils.java) class to create the cluster and to open and close Cassandra sessions. The cluster connects to the API for Cassandra account in Azure Cosmos DB and returns a session to access. Use the [Configurations](https://github.com/Azure-Samples/azure-cosmos-db-cassandra-java-getting-started/blob/main/src/main/java/com/azure/cosmosdb/cassandra/util/Configurations.java) class to read connection string information from the config.properties file.

-5. The Java sample creates a database with user information such as user name, user ID, and user city. You need to define get and set methods to access user details in the main function.

-

- Create a [User.java](https://github.com/Azure-Samples/azure-cosmos-db-cassandra-java-getting-started/blob/main/src/main/java/com/azure/cosmosdb/cassandra/examples/UserProfile.java) class under the `src/main/java/com/azure/cosmosdb/cassandra/` folder with get and set methods.

-## Add a database and a table

-This section describes how to add a database (keyspace) and a table, by using CQL.

-1. Under the `src\main\java\com\azure\cosmosdb\cassandra` folder, create a new folder named `repository`.

-2. Create the `UserRepository` Java class and add the following code to it:

- ```java

- package com.azure.cosmosdb.cassandra.repository;

- import java.util.List;

- import com.datastax.driver.core.BoundStatement;

- import com.datastax.driver.core.PreparedStatement;

- import com.datastax.driver.core.Row;

- import com.datastax.driver.core.Session;

- import org.slf4j.Logger;

- import org.slf4j.LoggerFactory;

-

- /**

- * Create a Cassandra session

- */

- public class UserRepository {

-

- private static final Logger LOGGER = LoggerFactory.getLogger(UserRepository.class);

- private Session session;

- public UserRepository(Session session) {

- this.session = session;

- }

-

- /**

- * Create keyspace uprofile in cassandra DB

- */

-

- public void createKeyspace() {

- final String query = "CREATE KEYSPACE IF NOT EXISTS uprofile WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 1 }";

- session.execute(query);

- LOGGER.info("Created keyspace 'uprofile'");

- }

-

- /**

- * Create user table in cassandra DB

- */

-

- public void createTable() {

- final String query = "CREATE TABLE IF NOT EXISTS uprofile.user (user_id int PRIMARY KEY, user_name text, user_bcity text)";

- session.execute(query);

- LOGGER.info("Created table 'user'");

- }

- }

- ```

-3. Locate the `src\main\java\com\azure\cosmosdb\cassandra` folder, and create a new subfolder named `examples`.

-4. Create the `UserProfile` Java class. This class contains the main method that calls the createKeyspace and createTable methods you defined earlier:

- ```java

- package com.azure.cosmosdb.cassandra.examples;

- import java.io.IOException;

- import com.azure.cosmosdb.cassandra.repository.UserRepository;

- import com.azure.cosmosdb.cassandra.util.CassandraUtils;

- import com.datastax.driver.core.PreparedStatement;

- import com.datastax.driver.core.Session;

- import org.slf4j.Logger;

- import org.slf4j.LoggerFactory;

-

- /**

- * Example class which will demonstrate following operations on Cassandra Database on CosmosDB

- * - Create Keyspace

- * - Create Table

- * - Insert Rows

- * - Select all data from a table

- * - Select a row from a table

- */

-

- public class UserProfile {

-

- private static final Logger LOGGER = LoggerFactory.getLogger(UserProfile.class);

- public static void main(String[] s) throws Exception {

- CassandraUtils utils = new CassandraUtils();

- Session cassandraSession = utils.getSession();

-

- try {

- UserRepository repository = new UserRepository(cassandraSession);

- //Create keyspace in cassandra database

- repository.createKeyspace();

- //Create table in cassandra database

- repository.createTable();

-

- } finally {

- utils.close();

- LOGGER.info("Please delete your table after verifying the presence of the data in portal or from CQL");

- }

- }

- }

- ```

-

-## Run the app

-1. Open a command prompt or terminal window. Paste the following code block.

- This code changes the directory (cd) to the folder path where you created the project. Then, it runs the `mvn clean install` command to generate the `cosmosdb-cassandra-examples.jar` file within the target folder. Finally, it runs the Java application.

- ```bash

- cd cassandra-demo

- mvn clean install

- java -cp target/cosmosdb-cassandra-examples.jar com.azure.cosmosdb.cassandra.examples.UserProfile

- ```

- The terminal window displays notifications that the keyspace and table are created.

-

-2. Now, in the Azure portal, open **Data Explorer** to confirm that the keyspace and table were created.

-## Next steps

-In this tutorial, you've learned how to create a API for Cassandra account in Azure Cosmos DB, a database, and a table by using a Java application. You can now proceed to the next article:

-> [!div class="nextstepaction"]

-> [load sample data to the API for Cassandra table](load-data-table.md).

-description: Learn how to query diagnostics logs for troubleshooting data stored in Azure Cosmos DB for Apache Cassandra.

-# Troubleshoot issues with advanced diagnostics queries with Azure Cosmos DB for Apache Cassandra

-In this article, we'll cover how to write more advanced queries to help troubleshoot issues with your Azure Cosmos DB for Cassandra account by using diagnostics logs sent to **resource-specific** tables.

-## Prerequisites

-> [!WARNING]

-> When creating a Diagnostic Setting for the API for Cassandra account, ensure that "DataPlaneRequests" remain unselected. In addition, for the Destination table, ensure "Resource specific" is chosen as it offers significant cost savings over "Azure diagnostics.

-> [!NOTE]

-> Note that enabling full text diagnostics, the queries returned will contain PII data.

-> This feature will not only log the skeleton of the query with obfuscated parameters but log the values of the parameters themselves.

-> This can help in diagnosing whether queries on a specific Primary Key (or set of Primary Keys) are consuming far more RUs than queries on other Primary Keys.

-## Log Analytics queries with different scenarios

-### RU consumption

- ```kusto

- CDBCassandraRequests

- | where DatabaseName=="azure_comos" and CollectionName=="user"

- | project TimeGenerated, RequestCharge, OperationName,

- requestType=split(split(PIICommandText,'"')[3], ' ')[0]

- | summarize max(RequestCharge) by bin(TimeGenerated, 10m), tostring(requestType), OperationName;

- ```

- ```kusto

- CDBPartitionKeyRUConsumption

- | where DatabaseName=="azure_comos" and CollectionName=="user"

- | summarize TotalRequestCharge=sum(todouble(RequestCharge)) by PartitionKey, PartitionKeyRangeId

- | order by TotalRequestCharge;

-

- CDBPartitionKeyRUConsumption

- | where DatabaseName=="azure_comos" and CollectionName=="user"

- | summarize TotalRequestCharge=sum(todouble(RequestCharge)) by OperationName, PartitionKey

- | order by TotalRequestCharge;

-

- CDBPartitionKeyRUConsumption

- | where DatabaseName=="azure_comos" and CollectionName=="user"

- | summarize TotalRequestCharge=sum(todouble(RequestCharge)) by bin(TimeGenerated, 1m), PartitionKey

- | render timechart;

- ```

- ```kusto

- CDBCassandraRequests

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | where TimeGenerated > ago(24h)

- | project ActivityId, DatabaseName, CollectionName, queryText=split(split(PIICommandText,'"')[3], ' ')[0], RequestCharge, TimeGenerated

- | order by RequestCharge desc;

- ```

- ```kusto

- // This query is looking at read operations

- CDBCassandraRequests

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | project ResponseLength, TimeGenerated, RequestCharge, cassandraOperationName=split(split(PIICommandText,'"')[3], ' ')[0]

- | where cassandraOperationName =="SELECT"

- | summarize maxResponseLength=max(ResponseLength), maxRU=max(RequestCharge) by bin(TimeGenerated, 10m), tostring(cassandraOperationName)

-

- // This query is looking at write operations

- CDBCassandraRequests

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | project ResponseLength, TimeGenerated, RequestCharge, cassandraOperationName=split(split(PIICommandText,'"')[3], ' ')[0]

- | where cassandraOperationName in ("CREATE", "UPDATE", "INSERT", "DELETE", "DROP")

- | summarize maxResponseLength=max(ResponseLength), maxRU=max(RequestCharge) by bin(TimeGenerated, 10m), tostring(cassandraOperationName)

-

- // Write operations over a time period.

- CDBCassandraRequests

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | project ResponseLength, TimeGenerated, RequestCharge, cassandraOperationName=split(split(PIICommandText,'"')[3], ' ')[0]

- | where cassandraOperationName in ("CREATE", "UPDATE", "INSERT", "DELETE", "DROP")

- | summarize maxResponseLength=max(ResponseLength), maxRU=max(RequestCharge) by bin(TimeGenerated, 10m), tostring(cassandraOperationName)

- | render timechart;

-

- // Read operations over a time period.

- CDBCassandraRequests

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | project ResponseLength, TimeGenerated, RequestCharge, cassandraOperationName=split(split(PIICommandText,'"')[3], ' ')[0]

- | where cassandraOperationName =="SELECT"

- | summarize maxResponseLength=max(ResponseLength), maxRU=max(RequestCharge) by bin(TimeGenerated, 10m), tostring(cassandraOperationName)

- | render timechart;

- ```

- ```kusto

- CDBPartitionKeyRUConsumption

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | where OperationName in ("Delete", "Read", "Upsert")

- | summarize totalRU=max(RequestCharge) by OperationName, PartitionKeyRangeId

- ```

- ```kusto

- CDBPartitionKeyRUConsumption

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | summarize totalRequestCharge=sum(RequestCharge) by PartitionKey, PartitionKeyRangeId;

- ```

- ```kusto

- CDBPartitionKeyStatistics

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | where TimeGenerated > now(-8h)

- | summarize StorageUsed = sum(SizeKb) by PartitionKey

- | order by StorageUsed desc

- ```

- ```kusto

- let storageUtilizationPerPartitionKey =

- CDBPartitionKeyStatistics

- | project AccountName=tolower(AccountName), PartitionKey, SizeKb;

- CDBCassandraRequests

- | project AccountName=tolower(AccountName),RequestCharge, ErrorCode, OperationName, ActivityId, DatabaseName, CollectionName, PIICommandText, RegionName

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | join kind=inner storageUtilizationPerPartitionKey on $left.AccountName==$right.AccountName

- | where ErrorCode != -1 //successful

- | project AccountName, PartitionKey,ErrorCode,RequestCharge,SizeKb, OperationName, ActivityId, DatabaseName, CollectionName, PIICommandText, RegionName;

- ```

-### Latency

- ```kusto

- CDBCassandraRequests

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | where ErrorCode in (4608, 4352) //Corresponding code in Cassandra

- | summarize max(DurationMs) by bin(TimeGenerated, 10m), ErrorCode

- | render timechart;

- ```

- ```kusto

- CDBCassandraRequests

- | where TimeGenerated > now(-6h)

- | DatabaseName=="azure_cosmos" and CollectionName=="user"

- | summarize max(DurationMs) by bin(TimeGenerated, 10m)

- | render timechart;

- ```

- ```kusto

- CDBCassandraRequests

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | project RequestLength, ResponseLength,

- RequestCharge, DurationMs, TimeGenerated, OperationName,

- query=split(split(PIICommandText,'"')[3], ' ')[0]

- | summarize max(DurationMs) by bin(TimeGenerated, 10m), RequestCharge, tostring(query),

- RequestLength, OperationName

- | order by RequestLength, RequestCharge;

- ```

-### Throttling

- ```kusto

- CDBCassandraRequests

- | where RetriedDueToRateLimiting != false and RateLimitingDelayMs > 0;

- ```

- ```kusto

- CDBCassandraRequests

- | where DatabaseName=="azure_cosmos" and CollectionName=="user"

- | where ErrorCode==4097 // Corresponding error code in Cassandra

- | project DatabaseName , CollectionName , CassandraCommands=split(split(PIICommandText,'"')[3], ' ')[0] , OperationName, TimeGenerated;

- ```

-## Next steps

-description: Learn about Azure Cosmos DB for Apache Cassandra extension driver and the recommended settings.

-# Azure Cosmos DB for Apache Cassandra driver extension

-Azure Cosmos DB offers a driver extension for DataStax Java Driver 3 and 4. These driver extensions provide developers with different features to help improve the performance and reliability of your application and optimize your workloads on Azure Cosmos DB.

-In this article, the focus will be on Java v4 of the DataStax Java Driver. The extension created can be implemented without any changes to your code but an update to the `pom.xml` and `application.conf` files. In this article, we share the default values for all configuration options set by the Azure Cosmos DB Cassandra extensions and in what cases you might wish to override them.

-## Recommended settings for Java SDK

-The following settings are specifically for Cassandra client driver Java version 4.

-### Authentication

-PlainTextAuthProvider is used by default. This is because the Azure Cosmos DB for Apache Cassandra requires authentication and uses plain text authentication.

-```java

- auth-provider {

- class = PlainTextAuthProvider

- }

-```

-### Connection

-Azure Cosmos DB load-balances requests against a large number of backend nodes. The default settings in the extension for local and remote node sizes work well in development, test, and low-volume production or staging environments. In high-volume environments, you should consider increasing these values to 50 or 100.

-```java

- connection {

- pool {

- local {

- size = 10

- }

- remote {

- size = 10

- }

- }

- }

-```

-### Token map

-The session token map is used internally by the driver to send requests to the optimal coordinator when token-aware routing is enabled. This is an effective optimization when you are connected to an Apache Cassandra instance. It is irrelevant and generates spurious error messages when you are connected to an Azure Cosmos DB Cassandra endpoint. Hence, we recommend disabling the session token map when you are connected to an Azure Cosmos DB for Apache Cassandra instance.

-```yml

- metadata {

- token-map {

- enabled = false

- }

- }

-```

-### Reconnection policy

-We recommend using the `ConstantReconnectionPolicy` for API for Cassandra, with a `base-delay` of 2 seconds.

-```java

- reconnection-policy {

- class = ConstantReconnectionPolicy

- base-delay = 2 second

- }

-```

-### Retry policy

-The default retry policy in the Java Driver does not handle the `OverLoadedException`. We have created a custom policy for API for Cassandra to help handle this exception.

-The parameters for the retry policy are defined within the [reference.conf](https://github.com/Azure/azure-cosmos-cassandra-extensions/blob/release/java-driver-4/1.1.2/driver-4/src/main/resources/reference.conf) of the Azure Cosmos DB extension.

-```java

- retry-policy {

- class = com.azure.cosmos.cassandra.CosmosRetryPolicy

- max-retries = 5

- fixed-backoff-time = 5000

- growing-backoff-time = 1000

- }

-```

-### Balancing policy and preferred regions

-The default load balancing policy in the v4 driver restricts application-level failover and specifying a single local datacenter for the `CqlSession`, object is required by the policy. This provides a good out-of-box experience for communicating with Azure Cosmos DB Cassandra instances. In addition to setting the load balancing policy, you can configure failover to specified regions in a multi-region-writes deployment, if there are regional outages using the `preferred-regions` parameter.

-```java

- load-balancing-policy {

- multi-region-writes=false

- preferred-regions=["Australia East","UK West"]

-}

-```

-### SSL connection and timeouts

-The `DefaultsslEngineFactory` is used by default. This is because Azure Cosmos DB Cassandra API requires SSL:

-```java

- ssl-engine-factory {

- class = DefaultSslEngineFactory

- }

-```

-A request timeout of 60 seconds provides a better out-of-box experience than the default value of 2 seconds. Adjust this value up or down based on workload and Azure Cosmos DB Cassandra throughput provisioning. The more throughput you provide, the lower you might set this value.

-``` java

- request {

- timeout = "60 seconds"

- }

-```

-## Next steps

-description: This article explains some common error codes in Azure Cosmos DB's API for Cassandra and how to troubleshoot using Log Analytics

-# Server diagnostics for Azure Cosmos DB for Apache Cassandra

-Log Analytics is a tool in the Azure portal that helps you run server diagnostics on your API for Cassandra account.

-## Prerequisites

-## Use Log Analytics

-After you've completed the log analytics setup, you can begin to explore your logs to gain more insights.

-### Explore Data Plane Operations

-Use the [CDBCassandraRequests table](/azure/azure-monitor/reference/tables/cdbcassandrarequests) to see data plane operations specifically for your API for Cassandra account. A sample query to see the topN(10) consuming request and get detailed information on each request made.

-```Kusto

-CDBCassandraRequests

-| where RequestCharge > 0

-| project DatabaseName, CollectionName, DurationMs, OperationName, ActivityId, ErrorCode, RequestCharge, PIICommandText

-| order by RequestCharge

-| take 10

-```

-For a list of error codes and their possible solutions, see [Error codes](../monitor-reference.md#error-codes-for-cassandra).

-### Troubleshoot Query Consumption

-The [CDBPartitionKeyRUConsumption table](/azure/azure-monitor/reference/tables/cdbpartitionkeyruconsumption) contains details on request unit (RU) consumption for logical keys in each region within each of their physical partitions.

-```Kusto

-CDBPartitionKeyRUConsumption

-| summarize sum(todouble(RequestCharge)) by PartitionKey, PartitionKeyRangeId

-| render columnchart

- ```

-### Explore Control Plane Operations

-The [CBDControlPlaneRequests table](/azure/azure-monitor/reference/tables/cdbcontrolplanerequests) contains details on control plane operations, specifically for API for Cassandra accounts.

-```Kusto

-CDBControlPlaneRequests

-| where TimeGenerated > now(-6h)

-| where ApiKind == "Cassandra"

-| where OperationName in ("Create", "Upsert", "Delete", "Execute")

-| summarize by OperationName

- ```

-## Next steps

-description: Learn how to find the request unit (RU) charge for Cassandra queries executed against an Azure Cosmos DB container. You can use the Azure portal, .NET and Java drivers to find the RU charge.

-# ms.devlang: csharp, java, golang

-# Find the request unit charge for operations executed in Azure Cosmos DB for Apache Cassandra

-Azure Cosmos DB supports many APIs, such as SQL, MongoDB, Cassandra, Gremlin, and Table. Each API has its own set of database operations. These operations range from simple point reads and writes to complex queries. Each database operation consumes system resources based on the complexity of the operation.

-The cost of all database operations is normalized by Azure Cosmos DB and is expressed by Request Units (or RUs, for short). Request charge is the request units consumed by all your database operations. You can think of RUs as a performance currency abstracting the system resources such as CPU, IOPS, and memory that are required to perform the database operations supported by Azure Cosmos DB. No matter which API you use to interact with your Azure Cosmos DB container, costs are always measured by RUs. Whether the database operation is a write, point read, or query, costs are always measured in RUs. To learn more, see the [request units and it's considerations](../request-units.md) article.

-This article presents the different ways you can find the [request unit](../request-units.md) (RU) consumption for any operation executed against a container in Azure Cosmos DB for Apache Cassandra. If you are using a different API, see [API for MongoDB](../mongodb/find-request-unit-charge.md), [API for NoSQL](../find-request-unit-charge.md), [API for Gremlin](../gremlin/find-request-unit-charge.md), and [API for Table](../table/find-request-unit-charge.md) articles to find the RU/s charge.

-When you perform operations against the Azure Cosmos DB for Apache Cassandra, the RU charge is returned in the incoming payload as a field named `RequestCharge`. You have multiple options for retrieving the RU charge.

-## Use a Cassandra Driver

-### [.NET Driver](#tab/dotnet-driver)

-When you use the [.NET SDK](https://www.nuget.org/packages/CassandraCSharpDriver/), you can retrieve the incoming payload under the `Info` property of a `RowSet` object:

-```csharp

-RowSet rowSet = session.Execute("SELECT table_name FROM system_schema.tables;");

-double requestCharge = BitConverter.ToDouble(rowSet.Info.IncomingPayload["RequestCharge"].Reverse().ToArray(), 0);

-```

-For more information, see [Quickstart: Build a Cassandra app by using the .NET SDK and Azure Cosmos DB](manage-data-dotnet.md).

-### [Java Driver](#tab/java-driver)

-When you use the [Java SDK](https://mvnrepository.com/artifact/com.datastax.cassandra/cassandra-driver-core), you can retrieve the incoming payload by calling the `getExecutionInfo()` method on a `ResultSet` object:

-```java

-ResultSet resultSet = session.execute("SELECT table_name FROM system_schema.tables;");

-Double requestCharge = resultSet.getExecutionInfo().getIncomingPayload().get("RequestCharge").getDouble();

-```

-For more information, see [Quickstart: Build a Cassandra app by using the Java SDK and Azure Cosmos DB](manage-data-java.md).

-### [GOCQL Driver](#tab/gocql-driver)

-When you use the [GOCQL driver](https://github.com/gocql/gocql), you can retrieve the incoming payload by calling the `GetCustomPayload()` method on a [`Iter`](https://pkg.go.dev/github.com/gocql/gocql#Iter) type:

-```go

-query := session.Query(fmt.Sprintf("SELECT * FROM <keyspace.table> where <value> = ?", keyspace, table)).Bind(<value>)

-iter := query.Iter()

-requestCharge := iter.GetCustomPayload()["RequestCharge"]

-requestChargeBits := binary.BigEndian.Uint64(requestCharge)

-requestChargeValue := math.Float64frombits(requestChargeBits)

-fmt.Printf("%v\n", requestChargeValue)

-```

-For more information, see [Quickstart: Build a Cassandra app by using GOCQL and Azure Cosmos DB](manage-data-go.md).

-## Next steps

-To learn about optimizing your RU consumption, see these articles:

-* [Request units and throughput in Azure Cosmos DB](../request-units.md)

-* [Optimize provisioned throughput cost in Azure Cosmos DB](../optimize-cost-throughput.md)

-* [Optimize query cost in Azure Cosmos DB](../optimize-cost-reads-writes.md)

-description: This article details how to run Glowroot in Azure Cosmos DB for Apache Cassandra.

-# Run Glowroot on Azure Cosmos DB for Apache Cassandra

-Glowroot is an application performance management tool used to optimize and monitor the performance of your applications. This article explains how you can now use Glowroot within Azure Cosmos DB for Apache Cassandra to monitor your application's performance.

-## Prerequisites and Setup

-* [Create an Azure Cosmos DB for Apache Cassandra account](manage-data-java.md#create-a-database-account).

-* [Install Java (version 8) for Windows](https://developers.redhat.com/products/openjdk/download)

-> [!NOTE]

-> Note that there are certain known incompatible build targets with newer versions. If you already have a newer version of Java, you can still download JDK8.

-> If you have newer Java installed in addition to JDK8: Set the %JAVA_HOME% variable in the local command prompt to target JDK8. This will only change Java version for the current session and leave global machine settings intact.

-* [Install maven](https://maven.apache.org/download.cgi)

- * Verify successful installation by running: `mvn --version`

-## Run Glowroot central collector with Azure Cosmos DB endpoint

-Once the endpoint configuration has been completed.

-1. [Download Glowroot central collector distribution](https://github.com/glowroot/glowroot)

-2. In the glowroot-central.properties file, populate the following properties from your Azure Cosmos DB for Apache Cassandra endpoint

- * cassandra.contactPoints

- * cassandra.username

- * cassandra.password

-3. Set properties `cassandra.ssl=true`, `cassandra.gcGraceSeconds=0`, and `cassandra.port=10350`.

-4. Ensure that the glowroot-central.properties is in the same folder as the glowroot-central.jar.

-5. Run `java -jar glowroot-central.jar` to begin running Glowroot.

-## FAQs

-Open a support ticket if you have issues running or testing Glowroot. Providing the subscription ID and account name where your Glowroot test will be running.

-## Next steps

-description: Learn how to create a container in Azure Cosmos DB for Apache Cassandra by using Azure portal, .NET, Java, Python, Node.js, and other SDKs.

-# Create a container in Azure Cosmos DB for Apache Cassandra

-This article explains the different ways to create a container in Azure Cosmos DB for Apache Cassandra. It shows how to create a container using Azure portal, Azure CLI, PowerShell, or supported SDKs. This article demonstrates how to create a container, specify the partition key, and provision throughput.

-This article explains the different ways to create a container in Azure Cosmos DB for Apache Cassandra. If you are using a different API, see [API for MongoDB](../mongodb/how-to-create-container.md), [API for Gremlin](../gremlin/how-to-create-container.md), [API for Table](../table/how-to-create-container.md), and [API for NoSQL](../how-to-create-container.md) articles to create the container.

-> [!NOTE]

-> When creating containers, make sure you donΓÇÖt create two containers with the same name but different casing. ThatΓÇÖs because some parts of the Azure platform are not case-sensitive, and this can result in confusion/collision of telemetry and actions on containers with such names.

-## <a id="portal-cassandra"></a>Create using Azure portal

-1. Sign in to the [Azure portal](https://portal.azure.com/).

-1. [Create a new Azure Cosmos DB account](manage-data-dotnet.md#create-a-database-account), or select an existing account.

-1. Open the **Data Explorer** pane, and select **New Table**. Next, provide the following details:

- * Indicate whether you are creating a new keyspace, or using an existing one.

- * Enter a table name.

- * Enter the properties and specify a primary key.

- * Enter a throughput to be provisioned (for example, 1000 RUs).

- * Select **OK**.

- :::image type="content" source="../media/how-to-create-container/partitioned-collection-create-cassandra.png" alt-text="Screenshot of API for Cassandra, Add Table dialog box":::

-> [!NOTE]

-> For API for Cassandra, the primary key is used as the partition key.

-## <a id="dotnet-cassandra"></a>Create using .NET SDK

-```csharp

-// Create a Cassandra table with a partition/primary key and provision 1000 RU/s throughput.

-session.Execute(CREATE TABLE myKeySpace.myTable(

- user_id int PRIMARY KEY,

- firstName text,

- lastName text) WITH cosmosdb_provisioned_throughput=1000);

-```

-If you encounter timeout exception when creating a collection, do a read operation to validate if the collection was created successfully. The read operation throws an exception until the collection create operation is successful. For the list of status codes supported by the create operation see the [HTTP Status Codes for Azure Cosmos DB](/rest/api/cosmos-db/http-status-codes-for-cosmosdb) article.

-## <a id="cli-mongodb"></a>Create using Azure CLI

-[Create a Cassandra table with Azure CLI](../scripts/cli/cassandr).

-## Create using PowerShell

-[Create a Cassandra table with PowerShell](../scripts/powershell/cassandr)

-## Next steps

-* [Partitioning in Azure Cosmos DB](../partitioning-overview.md)

-* [Request Units in Azure Cosmos DB](../request-units.md)

-* [Provision throughput on containers and databases](../set-throughput.md)

-* [Work with Azure Cosmos DB account](../resource-model.md)

-description: Learn how to provision container, database, and autoscale throughput in Azure Cosmos DB for Apache Cassandra resources. You will use Azure portal, CLI, PowerShell and various other SDKs.

-# Provision database, container or autoscale throughput on Azure Cosmos DB for Apache Cassandra resources

-This article explains how to provision throughput in Azure Cosmos DB for Apache Cassandra. You can provision standard(manual) or autoscale throughput on a container, or a database and share it among the containers within the database. You can provision throughput using Azure portal, Azure CLI, or Azure Cosmos DB SDKs.

-If you are using a different API, see [API for NoSQL](../how-to-provision-container-throughput.md), [API for MongoDB](../mongodb/how-to-provision-throughput.md), [API for Gremlin](../gremlin/how-to-provision-throughput.md) articles to provision the throughput.

-## <a id="portal-cassandra"></a> Azure portal

-1. Sign in to the [Azure portal](https://portal.azure.com/).

-1. [Create a new Azure Cosmos DB account](../mongodb/create-mongodb-dotnet.md#create-an-azure-cosmos-db-account), or select an existing Azure Cosmos DB account.

-1. Open the **Data Explorer** pane, and select **New Table**. Next, provide the following details:

- * Indicate whether you are creating a new keyspace or using an existing one. Select the **Provision database throughput** option if you want to provision throughput at the keyspace level.

- * Enter the table ID within the CQL command.

- * Enter a primary key value (for example, `/userrID`).

- * Enter a throughput that you want to provision (for example, 1000 RUs).

- * Select **OK**.

- :::image type="content" source="./media/how-to-provision-throughput/provision-table-throughput-portal-cassandra-api.png" alt-text="Screenshot of Data Explorer, when creating a new collection with database level throughput":::

-> [!Note]

-> If you are provisioning throughput on a container in an Azure Cosmos DB account configured with API for Cassandra, use `/myPrimaryKey` for the partition key path.

-## <a id="dotnet-cassandra"></a> .NET SDK

-### Provision throughput for a Cassandra table

-```csharp

-// Create a Cassandra table with a partition (primary) key and provision throughput of 400 RU/s

-session.Execute("CREATE TABLE myKeySpace.myTable(

- user_id int PRIMARY KEY,

- firstName text,

- lastName text) WITH cosmosdb_provisioned_throughput=400");

-```

-Similar commands can be issued through any CQL-compliant driver.

-### Alter or change throughput for a Cassandra table

-```csharp

-// Altering the throughput too can be done through code by issuing following command

-session.Execute("ALTER TABLE myKeySpace.myTable WITH cosmosdb_provisioned_throughput=5000");

-```

-Similar command can be executed through any CQL compliant driver.

-```csharp

-// Create a Cassandra keyspace and provision throughput of 400 RU/s

-session.Execute("CREATE KEYSPACE IF NOT EXISTS myKeySpace WITH cosmosdb_provisioned_throughput=400");

-```

-## Azure Resource Manager

-Azure Resource Manager templates can be used to provision autoscale throughput on database or container-level resources for all Azure Cosmos DB APIs. See [Azure Resource Manager templates for Azure Cosmos DB](templates-samples.md) for samples.

-## Azure CLI

-Azure CLI can be used to provision autoscale throughput on a database or container-level resources for all Azure Cosmos DB APIs. For samples see [Azure CLI Samples for Azure Cosmos DB](cli-samples.md).

-## Azure PowerShell

-Azure PowerShell can be used to provision autoscale throughput on a database or container-level resources for all Azure Cosmos DB APIs. For samples see [Azure PowerShell samples for Azure Cosmos DB](powershell-samples.md).

-## Next steps

-See the following articles to learn about throughput provisioning in Azure Cosmos DB:

-* [Request units and throughput in Azure Cosmos DB](../request-units.md)

-description: Use Azure Cosmos DB for Apache Cassandra to power existing and new applications with Cassandra drivers and CQL.

-# What is Azure Cosmos DB for Apache Cassandra?

-[Azure Cosmos DB](../introduction.md) is a fully managed NoSQL and relational database for modern app development.

-Azure Cosmos DB for Apache Cassandra can be used as the data store for apps written for [Apache Cassandra](https://cassandra.apache.org). This compatibility means that by using existing [Apache drivers](https://cassandra.apache.org/doc/latest/cassandra/getting_started/drivers.html?highlight=driver) compliant with CQLv4, your existing Cassandra application can now communicate with the API for Cassandra. In many cases, you can switch from using Apache Cassandra to using the API for Cassandra, by just changing a connection string. The API for Cassandra enables you to interact with data stored in Azure Cosmos DB using the Cassandra Query Language (CQL), Cassandra-based tools (like cqlsh) and Cassandra client drivers that you're already familiar with.

-> [!TIP]

-> Want to try the API for Cassandra with no commitment? Create an Azure Cosmos DB account using [Try Azure Cosmos DB](../try-free.md) for free.

-## API for Cassandra benefits

-The API for Cassandra has added benefits of being built on Azure Cosmos DB:

-## Azure Managed Instance for Apache Cassandra

-For some customers, adapting to API for Cassandra can be a challenge due to differences in behaviour and/or configuration, especially for lift-and-shift migrations. [Azure Managed Instance for Apache Cassandra](../../managed-instance-apache-cassandr) is a first-party Azure service for hosting and maintaining pure open-source Apache Cassandra clusters with 100% compatibility.

-## Next steps

-description: Learn how to ingest data from Kafka to Azure Cosmos DB for Apache Cassandra using DataStax Apache Kafka Connector

-# Ingest data from Apache Kafka into Azure Cosmos DB for Apache Cassandra using Kafka Connect

-Existing Cassandra applications can easily work with the [Azure Cosmos DB for Apache Cassandra](introduction.md) because of its [CQLv4 driver compatibility](https://cassandra.apache.org/doc/latest/cassandra/getting_started/drivers.html?highlight=driver). You leverage this capability to integrate with streaming platforms such as [Apache Kafka](https://kafka.apache.org/) and bring data into Azure Cosmos DB.

-Data in Apache Kafka (topics) is only useful when consumed by other applications or ingested into other systems. It's possible to build a solution using the [Kafka Producer/Consumer](https://kafka.apache.org/documentation/#api) APIs [using a language and client SDK of your choice](https://cwiki.apache.org/confluence/display/KAFKA/Clients). Kafka Connect provides an alternative solution. It's a platform to stream data between Apache Kafka and other systems in a scalable and reliable manner. Since Kafka Connect supports off the shelf connectors which includes Cassandra, you don't need to write custom code to integrate Kafka with Azure Cosmos DB for Apache Cassandra.

-In this article, we will be using the open-source [DataStax Apache Kafka connector](https://docs.datastax.com/en/kafka/doc/kafka/kafkaIntro.html), that works on top of Kafka Connect framework to ingest records from a Kafka topic into rows of one or more Cassandra tables. The example provides a reusable setup using Docker Compose. This is quite convenient since it enables you to bootstrap all the required components locally with a single command. These components include Kafka, Zookeeper, Kafka Connect worker, and the sample data generator application.

-Here is a breakdown of the components and their service definitions - you can refer to the complete `docker-compose` file [in the GitHub repo](https://github.com/Azure-Samples/cosmosdb-cassandra-kafka/blob/main/docker-compose.yaml).

-## Prerequisites

-* [Provision an Azure Cosmos DB for Apache Cassandra account](manage-data-dotnet.md#create-a-database-account)

-* [Use cqlsh for validation](support.md#cql-shell)

-* Install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install)

-## Create Keyspace, tables and start the integration pipeline

-Using the Azure portal, create the Cassandra Keyspace and the tables required for the demo application.

-> [!NOTE]

-> Use the same Keyspace and table names as below

-```sql

-CREATE KEYSPACE weather WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'datacenter1' : 1};

-CREATE TABLE weather.data_by_state (station_id text, temp int, state text, ts timestamp, PRIMARY KEY (state, ts)) WITH CLUSTERING ORDER BY (ts DESC) AND cosmosdb_cell_level_timestamp=true AND cosmosdb_cell_level_timestamp_tombstones=true AND cosmosdb_cell_level_timetolive=true;

-CREATE TABLE weather.data_by_station (station_id text, temp int, state text, ts timestamp, PRIMARY KEY (station_id, ts)) WITH CLUSTERING ORDER BY (ts DESC) AND cosmosdb_cell_level_timestamp=true AND cosmosdb_cell_level_timestamp_tombstones=true AND cosmosdb_cell_level_timetolive=true;

-```

-Clone the GitHub repo:

-```bash

-git clone https://github.com/Azure-Samples/cosmosdb-cassandra-kafka

-cd cosmosdb-cassandra-kafka

-```

-Start all the

-```shell

-docker-compose --project-name kafka-cosmos-cassandra up --build

-```

-> [!NOTE]

-> It might take a while to download and start the containers: this is just a one time process.

-To confirm whether all the containers have started:

-```shell

-docker-compose -p kafka-cosmos-cassandra ps

-```

-The data generator application will start pumping data into the `weather-data` topic in Kafka. You can also do quick sanity check to confirm. Peek into the Docker container running the Kafka connect worker:

-```bash

-docker exec -it kafka-cosmos-cassandra_cassandra-connector_1 bash

-```

-Once you drop into the container shell, just start the usual Kafka console consumer process and you should see weather data (in JSON format) flowing in.

-```bash

-cd ../bin

-./kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic weather-data

-```

-## Cassandra Sink connector setup

-Copy the JSON contents below to a file (you can name it `cassandra-sink-config.json`). You will need to update it as per your setup and the rest of this section will provide guidance around this topic.

-```json

-{

- "name": "kafka-cosmosdb-sink",

- "config": {

- "connector.class": "com.datastax.oss.kafka.sink.CassandraSinkConnector",

- "tasks.max": "1",

- "topics": "weather-data",

- "contactPoints": "<cosmos db account name>.cassandra.cosmos.azure.com",

- "port": 10350,

- "loadBalancing.localDc": "<cosmos db region e.g. Southeast Asia>",

- "auth.username": "<enter username for cosmosdb account>",

- "auth.password": "<enter password for cosmosdb account>",

- "ssl.hostnameValidation": true,

- "ssl.provider": "JDK",

- "ssl.keystore.path": "/etc/alternatives/jre/lib/security/cacerts/",

- "ssl.keystore.password": "changeit",

- "datastax-java-driver.advanced.connection.init-query-timeout": 5000,

- "maxConcurrentRequests": 500,

- "maxNumberOfRecordsInBatch": 32,

- "queryExecutionTimeout": 30,

- "connectionPoolLocalSize": 4,

- "topic.weather-data.weather.data_by_state.mapping": "station_id=value.stationid, temp=value.temp, state=value.state, ts=value.created",

- "topic.weather-data.weather.data_by_station.mapping": "station_id=value.stationid, temp=value.temp, state=value.state, ts=value.created",

- "key.converter": "org.apache.kafka.connect.storage.StringConverter",

- "value.converter": "org.apache.kafka.connect.json.JsonConverter",

- "value.converter.schemas.enable": false,

- "offset.flush.interval.ms": 10000

- }

-}

-```

-Here is a summary of the attributes:

-**Basic connectivity**

-**SSL configuration**

-Azure Cosmos DB enforces [secure connectivity over SSL](../database-security.md) and Kafka Connect connector supports SSL as well.

-**Generic parameters**

-### Install the connector

-Install the connector using the Kafka Connect REST endpoint:

-```shell

-curl -X POST -H "Content-Type: application/json" --data @cassandra-sink-config.json http://localhost:8083/connectors

-```

-To check the status:

-```

-curl http://localhost:8080/connectors/kafka-cosmosdb-sink/status

-```

-If all goes well, the connector should start weaving its magic. It should authenticate to Azure Cosmos DB and start ingesting data from the Kafka topic (`weather-data`) into Cassandra tables - `weather.data_by_state` and `weather.data_by_station`

-You can now query data in the tables. Head over to the Azure portal, bring up the hosted CQL Shell for your Azure Cosmos DB account.

-## Query data from Azure Cosmos DB

-Check the `data_by_state` and `data_by_station` tables. Here is some sample queries to get you started:

-```sql

-select * from weather.data_by_state where state = 'state-1';

-select * from weather.data_by_state where state IN ('state-1', 'state-2');