- - ignite-2023

-# Document Intelligence custom generative model

-The custom generative model combines the power of document understanding with Large Language Models (LLMs) and the rigor and schema from custom extraction capabilities. Custom generative extraction enables you to easily automate data extraction workflows for any type of document, with minimal labeling and greater accuracy and speed.

-## Custom generative model key features

-* **Grounded results**. Localize the data extracted in documents and ensure the response is generated from the content and enables human review workflows.

-* **High confidence scores**. Use confidence scores and quickly filter high quality extracted data for downstream processing and lower manual review time.

-* **Contract Lifecycle Management**. Build a generative model and extract the fields, clauses, and obligations from a wide array of contract types.  

-* **Loan & Mortgage Applications**. Automation of loan and mortgage application process enables banks, lenders, and government entities to quickly process loan and mortgage application.  

-* **Financial Services**. Analyze complex documents like financial reports and asset management reports.

-* **Expense management**. The custom generative model can extract expenses, receipts, and invoices with varying formats and templates.  

-The Custom generative model currently supports dynamic table with the `2024-07-31-preview` and the following fields:

-The `build custom model` operation supports custom _template_, _neural_ and _generative_ models, _see_ [Custom model build mode](concept-custom.md#build-mode):

-* **Custom generative models** can process complex documents in various formats, templates, and unstructured data.

-* **Custom neural models** support complex document processing and also support more variance in page for structured and semi-structured documents.

-The custom generative model `2024-07-31-preview` version supports the **en-us** locale. For more information on language support, *see* [Language support - custom models](language-support-custom.md).

-The custom generative model `2024-07-31-preview` version is only available in `North Central US`.  

-## Input requirements 

-* **Field Description**. Provide more contextual information in description to help clarify the field that needs to be extracted. Examples include location in the document, potential field labels it may be associated with, ways to differentiate with other terms that could be ambiguous.  

-* **Variation**. Custom generative models can generalize across different document templates of the same document type. As a best practice, create a single model for all variations of a document type. Ideally, include a visual template for each type, especially for ones that 

-* Grounding, especially for tabular fields, is challenging and might not be perfect in some cases.  

-The `build operation` to train model supports the ```buildMode``` property, to train a custom generative model, set the ```buildMode``` to ```generative```.

-See the [Getting Started Guide for the Ingestion Client](https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/master/samples/ingestion/ingestion-client/Setup/guide.md) on GitHub to learn how to set up and use the tool.

-When you create your data, you need to set the data type. AI Studio supports three data types:

-| `DiscQuotaInMb` | Defines the amount of disk space used by the Azure Monitor Agent log files and cache. | 1,000-50,000 (or 1-50 GB) |

-1. **Create a DCR via template deployment:**

- The following example changes the maximum amount of disk space used by AMA cache to 5 GB.

- > [!NOTE]

- > You can use the Get DataCollectionRule API to get the DCR payload you created with this template.

-

-1. **Associate DCR with your machine:**

- This can be done with a template or by using the [Create API](/rest/api/monitor/data-collection-rule-associations/create) with the following details:

-

- * **AssociationName:** agentSettings

- * **ResourceUri:** Full ARM ID of the VM

- * **api-version:** 2023-03-11 (Old API version is also fine)

- * **Body:**

- ```json

- {

- "properties": {

- "dataCollectionRuleId": ΓÇ£Full ARM ID for agent setting DCRΓÇ¥

- }

- }

- ```

- "parameters": {

- "appName": {

- "type": "string"

- },

- "pingURL": {

- "type": "string"

- },

- "pingText": {

- "type": "string",

- "defaultValue": ""

- },

- "actionGroupId": {

- "type": "string"

- },

- "location": {

- "type": "string"

-A connection string in Application Insights defines the target location for sending telemetry data, ensuring it reaches the appropriate resource for monitoring and analysis.

- // Create a new ASP.NET Core web application builder.

- // Build the ASP.NET Core web application.

- // Start the ASP.NET Core web application.

-1. Retrieve the virtual machine scale set resource ID and resource group of your virtual machine scale set. For example: /subscriptions/e954e48d-abcd-abcd-abcd-3e0353cb45ae/resourceGroups/patest2/providers/Microsoft.Compute/virtualMachineScaleSets/patest2

- "value": "/subscriptions/e954e48d-b252-b252-b252-3e0353cb45ae/resourceGroups/patest2/providers/Microsoft.Compute/virtualMachineScaleSets/patest2"

-[Reliability](/azure/well-architected/resiliency/overview) refers to the ability of a system to recover from failures and continue to function. Instead of trying to prevent failures altogether in the cloud, the goal is to minimize the effects of a single failing component. Use the following information to minimize failure of your Log Analytics workspaces and to protect the data they collect.

-[Azure Monitor managed service for Prometheus](prometheus-metrics-overview.md) allows you to collect and analyze metrics at scale using a [Prometheus](https://aka.ms/azureprometheus-promio)-compatible monitoring solution. The most common way to analyze and present Prometheus data is with a Grafana dashboard. This article explains how to configure Prometheus as a data source for [self-hosted Grafana](https://grafana.com/) using Microsoft Entra ID.

- :::image type="content" source="./media/prometheus-self-managed-grafana-azure-active-directory/app-registration-overview.png" alt-text="A screenshot showing the App registration overview page.":::

- :::image type="content" source="./media/prometheus-self-managed-grafana-azure-active-directory/add-a-client-secret.png" alt-text="A screenshot showing the Add client secret page.":::

- :::image type="content" source="./media/prometheus-self-managed-grafana-azure-active-directory/client-secret.png" alt-text="A screenshot showing the client secret page with generated secret value.":::

- :::image type="content" source="./media/prometheus-self-managed-grafana-azure-active-directory/add-role-assignment.png" alt-text="A screenshot showing the Add role assignment page":::

- :::image type="content" source="./media/prometheus-self-managed-grafana-azure-active-directory/select-members.png" alt-text="A screenshot showing the Add role assignment, select members page.":::

- :::image type="content" source="./media/prometheus-self-managed-grafana-azure-active-directory/configure-grafana.png" alt-text="A screenshot showing the Grafana settings page for adding a data source.":::

-Connect Call action can be used to connect to an ongoing call and take call actions on it. You can also use this action to connect and manage a Rooms call programmatically, like performing PSTN dial outs for Room using your service.

-To connect to a Rooms call, use RoomCallLocator which takes RoomId.

-For workflows that need direct access to resources in an Azure virtual network, you can create a dedicated [integration service environment (ISE)](../logic-apps/connect-virtual-network-vnet-isolated-environment-overview.md) where you can build, deploy, and run your workflows on dedicated resources. For more information about creating ISEs, see [Connect to Azure virtual networks from Azure Logic Apps](../logic-apps/connect-virtual-network-vnet-isolated-environment.md).

-When you work with Azure Logic Apps, you can set up an [*integration service environment (ISE)*](../logic-apps/connect-virtual-network-vnet-isolated-environment-overview.md) for hosting logic apps that need access to resources in an [Azure virtual network](../virtual-network/virtual-networks-overview.md). When you have multiple ISE instances that need access to other endpoints that have IP restrictions, deploy an [Azure Firewall](../firewall/overview.md) or a [network virtual appliance](../virtual-network/virtual-networks-overview.md#filter-network-traffic) into your virtual network and route outbound traffic through that firewall or network virtual appliance. You can then have all the ISE instances in your virtual network use a single, public, static, and predictable IP address to communicate with the destination systems that you want. That way, you don't have to set up additional firewall openings at your destination systems for each ISE.

- |||

- * [Azure Firewall FAQ](../firewall/firewall-faq.yml)

-## Next steps

-* [Connect to Azure virtual networks from Azure Logic Apps](../logic-apps/connect-virtual-network-vnet-isolated-environment.md)

-This table describes the ports that your ISE requires to be accessible and the purpose for those ports. To help reduce complexity when you set up security rules, the table uses [service tags](../virtual-network/service-tags-overview.md) that represent groups of IP address prefixes for a specific Azure service. Where noted, *internal ISE* and *external ISE* refer to the [access endpoint that's selected during ISE creation](connect-virtual-network-vnet-isolated-environment.md#create-environment). For more information, review [Endpoint access](connect-virtual-network-vnet-isolated-environment-overview.md#endpoint-access).

-When you create an ISE, Azure *injects* or deploys that ISE into your Azure virtual network. You can then use this ISE as the location for the logic apps and integration accounts that need access. For more information about creating an ISE, review [Connect to Azure virtual networks from Azure Logic Apps](connect-virtual-network-vnet-isolated-environment.md).

-In Visual Studio, if your logic app exists as a JSON (.json) file within an [Azure Resource Group project](../azure-resource-manager/templates/create-visual-studio-deployment-project.md) that you use to automate deployment, that logic app is set to a location type and a specific location. This location is either an Azure region or an existing [integration service environment (ISE)](connect-virtual-network-vnet-isolated-environment.md).

- | **Location** | **Same as Resource Group** | The location type and specific location for deploying your logic app resource. The location type is either an Azure region or an existing [integration service environment (ISE)](connect-virtual-network-vnet-isolated-environment.md). <p>For this quickstart, keep the location type set to **Region** and the location set to **Same as Resource Group**. <p>**Note**: After you create your resource group project, you can [change the location type and the location](manage-logic-apps-with-visual-studio.md#change-location), but different location type affects your logic app in various ways. |

-description: Learn about the architecture of Azure Operator Insights and how you can integrate with it to analyze date from your network.

-# Customer intent: As a systems architect at an operator, I want to understand the architecture of Azure Operator Insights so that I can integrate with it to analyze data from my network.

-# Architecture of Azure Operator Insights

-Azure Operator Insights is a fully managed service that enables the collection and analysis of massive quantities of network data gathered from complex multi-part or multi-vendor network functions. It delivers statistical, machine learning, and AI-based insights for operator-specific workloads to help operators understand the health of their networks and the quality of their subscribers' experiences in near real-time. For more information on the problem Azure Operator Insights solves, see [the general overview of Azure Operator Insights](overview.md).

-Azure Operator Insights deploys a Data Product resource to encapsulate a specific category or namespace of data. Azure Operator Insights enables a fourth generation data mesh architecture, which offers query-time federation to correlate and query across multiple Data Products.

-This following diagram shows the architecture of an Azure Operator Insights Data Product, and the surrounding services it interacts with.

- An Azure Operator Insights Data Product is in its own resource group. It deploys a managed resource group containing an Azure Key Vault instance that provides a shared access signature (SAS) token for ingestion storage. The SAS token is used for authentication when ingesting data. The options for ingesting data include Azure Operator Insights ingestion agents; Azure tools such as AzCopy, Azure Storage Explorer, and Azure Data Factory; and code-based mechanisms. The ingestion options can upload data from data sources such as Microsoft products and services, non-Microsoft products, and platforms. The data ingestion options can use the public internet, ExpressRoute, or Azure VPN Gateway. Data Products make data available over an ADLS consumption URL and a KQL consumption URL. Applications and services that can consume data include Azure Data Explorer (in dashboards and a follower database), Microsoft Power BI, Microsoft Fabric, Azure Machine Learning studio, Azure Databricks, Azure Logic Apps, Azure Storage Explorer, AzCopy, and non-Microsoft applications and services. The optional features and capabilities of Azure Operator Insights include Azure Monitor for logs and metrics, customer managed keys, Purview integration for data catalog, restricted IP addresses or private networking for data access, Microsoft Entra ID role-based access control for KQL consumption, and data retention and hot cache sizes.

-The rest of this article gives an overview of:

-## Deployment of Data Products

-You can deploy Azure Operator Insights Data Products with any standard Azure interface, including the Azure portal, Azure CLI, Azure PowerShell, or direct calls to the Azure Resource Manager (ARM) API. See [Create an Azure Operator Insights Data Product](data-product-create.md?tabs=azure-portal) for a quickstart guide to deploying with the Azure portal or the Azure CLI. When you deploy a Data Product, you can enable specific features such as integration with Microsoft Purview, customer-managed keys for data encryption, or restricted access to the Data Product. For more information on features you can enable at deployment, see [Data Product configuration options and controls](#data-product-configuration-options-and-controls).

-Each Azure Operator Insights Data Product is scoped for a given category or namespace of data. An example is the data from a single network function (NF) such as a voice SBC. Some Data Products might contain correlated data from multiple NFs, particularly if the NFs are from the same vendor, such as the UPF, SMF, and AMF from a mobile packet core vendor. Each Data Product appears as a single Azure resource in your resource group and subscription. You can deploy multiple Data Products, for different categories of data, for example different mobile packet core NFs from different vendors, or a mobile packet core plus a radio access network (RAN) Data Product.

-Microsoft publishes several Data Products; the following image shows some examples. Partners and operators can also design and publish Data Products using the Azure Operator Insights data product factory (preview). For more information on the data product factory, see the [overview of the data product factory](data-product-factory.md).

-Deploying an Azure Operator Insights Data Product creates the resource itself and a managed resource group in your subscription. The managed resource group contains an Azure Key Vault instance. The Key Vault instance contains a shared access signature (SAS) that you can use to authenticate when you upload files to the Data Product's ingestion storage URL.

-Once deployed, the Overview screen of the Azure Operator Insights Data Product resource shows essential information including:

-## Data sources

-Each Azure Operator Insights Data Product ingests data from a particular data source. The data source could be:

-## Data ingestion

-There are a range of options for ingesting data from the source into your Azure Operator Insights Data Product.

- - [AzCopy v10](/azure/storage/common/storage-use-azcopy-v10) ΓÇô AzCopy from Azure Storage is a robust, high throughput, and reliable ingestion mechanism across both low latency links and high latency links. With `azcopy sync`, you can use cron to automate ingestion from an on-premises virtual machine and achieve "free" ingestion into the Data Product (except for the cost of the on-premises virtual machine and networking).

- - [Azure Data Factory](/azure/data-factory/introduction) - See [Use Azure Data Factory to ingest data into an Azure Operator Insights Data Product](ingestion-with-data-factory.md).

-## Azure connectivity

-There are multiple ways to connect your on-premises private data centers where your network function data sources reside to the Azure cloud. For a general overview of the options, see [Connectivity to Azure - Cloud Adoption Framework](/azure/cloud-adoption-framework/ready/azure-best-practices/connectivity-to-azure). For telco-specific recommendations, see the [Network Analytics Landing Zone for Operators](https://github.com/microsoft/industry/blob/main/telco/solutions/observability/userGuide/readme.md).

-## Data consumption

-Azure Operator Insights Data Products offer two consumption URLs for accessing the data in the Data Product:

-There are multiple possible integrations that can be built on top of one or both of these consumption URLs.

-| | Supported with Data Product ADLS consumption URL | Supported with Data Product KQL consumption URL |

-||||

-| [**Azure Data Explorer dashboards**](/azure/data-explorer/azure-data-explorer-dashboards) | ❌ | ✅ |

-| [**Azure Data Explorer follower database**](/azure/data-explorer/follower) | ❌ | ✅ |

-| [**Power BI reports**](/power-bi/create-reports/) | ✅ | ✅ |

-| [**Microsoft Fabric**](/fabric/get-started/microsoft-fabric-overview) | ✅ | ✅ |

-| [**Azure Machine Learning Studio**](/azure/machine-learning/overview-what-is-azure-machine-learning) | ✅ | ❌ |

-| [**Azure Databricks**](/azure/databricks/introduction/) | ✅ | ✅ |

-| [**Azure Logic Apps**](/azure/logic-apps/logic-apps-overview) | ❌ | ✅ |

-| [**Azure Storage Explorer**](/azure/storage/storage-explorer/vs-azure-tools-storage-manage-with-storage-explorer) | ✅ | ❌ |

-| [**AzCopy**](/azure/storage/common/storage-use-azcopy-v10) | ✅ | ❌ |

-## Data Product configuration options and controls

-Azure Operator Insights Data Products have several configuration options that can be set when first deploying or modified after deployment.

-| | Description | When configurable | More information |

-| | | | |

-| **Integration with Microsoft Purview** | Enabling Purview integration during deployment causes the existence of the Data Product and its data type tables, schemas, and lineage to be published to Purview and visible to your organization in Purview's data catalog. | At deployment | [Use Microsoft Purview with an Azure Operator Insights Data Product](purview-setup.md) |

-| **Customer Managed Keys for Data Product storage** | Azure Operator Insights Data Products can secure your data using Microsoft Managed Keys or Customer Managed Keys. | At deployment | [Set up resources for CMK-based data encryption or Microsoft Purview](data-product-create.md#set-up-resources-for-cmk-based-data-encryption-or-microsoft-purview) |

-| **Connectivity for ingestion and ADLS consumption URLs** | Azure Operator Insights Data Products can be configured to allow public access from all networks or selected virtual networks and IP addresses. | At deployment. If you deploy with selected virtual networks and IP addresses, you can add or remove networks and IP addresses after deployment. |--|

-| **Connectivity for the KQL consumption URL** | Azure Operator Insights Data Products can be configured to allow public access from all networks or selected IP addresses. | At deployment. If you deploy with selected IP addresses, you can add or remove IP addresses after deployment. |--|

-| **Data retention and hot cache size** | Azure Operator Insights Data Products are initially deployed with default retention periods and KQL hot cache durations for each data type (group of data within a Data Product). You can set custom thresholds | After deployment | [Data types in Azure Operator Insights](concept-data-types.md) |

-| **Access control for ADLS consumption URL** | Access to the ADLS consumption URL is managed on an Azure Operator Insights Data Product by generating a SAS token after deployment. | After deployment |--|

-| **Access control for KQL consumption URL** | Access to the KQL consumption URL is granted by adding a principal (which can be an individual user, group, or managed identity) as a Reader or Restricted Reader. | After deployment | [Manage permissions to the KQL consumption URL](consumption-plane-configure-permissions.md) |

-## Monitoring

-After you deploy a Data Product, you can monitor it for healthy operation or troubleshooting purposes using metrics, resource logs, and activity logs. For more information, see [Monitoring Azure Operator Insights](monitor-operator-insights.md).

-## Next step

-> [!div class="nextstepaction"]

-> [Learn about business continuity and disaster recovery](business-continuity-disaster-recovery.md)

-description: This article helps you understand BCDR concepts Azure Operator Insights.

-# Business continuity and disaster recovery

-Disasters can be hardware failures, natural disasters, or software failures. The process of preparing for and recovering from a disaster is called disaster recovery (DR). This article discusses recommended practices to achieve business continuity and disaster recovery (BCDR) for Azure Operator Insights.

-BCDR strategies include availability zone redundancy and user-managed recovery.

-## Control plane

-The Azure Operator Insights control plane is resilient both to software errors and failure of an Availability Zone. The ability to create and manage Data Products isn't affected by these failure modes.

-The control plane isn't regionally redundant. During an outage in an Azure region, you can't create new Data Products in that region or access/manage existing ones. Once the region recovers from the outage, you can access and manage existing Data Products again.

-## Data plane

-Data Products are resilient to software or hardware failures. For example, if a software bug causes the service to crash, or a hardware failure causes the compute resources for enrichment queries to be lost, service automatically recovers. The only impact is a slight delay in newly ingested data becoming available in the Data Product's storage endpoint and in the KQL consumption URL.

-### Zone redundancy

-Data Products don't support zone redundancy. When an availability zone fails, the Data Product's ingestion, blob/DFS and KQL/SQL APIs are all unavailable, and dashboards don't work. Transformation of already-ingested data is paused. No previously ingested data is lost. Processing resumes when the availability zone recovers.

-What happens to data that was generated during the availability zone outage depends on the behavior of the ingestion agent:

-* If the ingestion agent buffers data and resends it when the availability zone recovers, data isn't lost. Azure Operator Insights might take some time to work through its transformation backlog.

-* Otherwise, data is lost.

-### Disaster recovery

-Azure Operator Insights has no innate region redundancy. Regional outages affect Data Products in the same way as [availability zone failures](#zone-redundancy). We have recommendations and features to support customers that want to be able to handle failure of an entire Azure region.

-#### User-managed redundancy

-For maximal redundancy, you can deploy Data Products in an active-active mode. Deploy a second Data Product in a backup Azure region of your choice, and configure your ingestion agents to fork data to both Data Products simultaneously. The backup Data Product is unaffected by the failure of the primary region. During a regional outage, look at dashboards that use the backup Data Product as the data source. This architecture doubles the cost of the solution.

-Alternatively, you could use an active-passive mode. Deploy a second Data Product in a backup Azure region, and configure your ingestion agents to send to the primary Data Product. During a regional outage, reconfigure your ingestion agents to send data to the backup Data Product during a region outage. This architecture gives full access to data created during the outage (starting from the time where you reconfigure the ingestion agents), but during the outage you don't have access to data ingested before that time. This architecture requires a small infrastructure charge for the second Data Product, but no additional data processing charges.

-description: Learn how to make and roll back configuration changes for Azure Operator Insights ingestion agents.

-#CustomerIntent: As a someone managing an agent that has already been set up, I want to update its configuration so that Data Products in Azure Operator Insights receive the correct data.

-# Change configuration for Azure Operator Insights ingestion agents

-The ingestion agent is a software package that is installed onto a Linux Virtual Machine (VM) owned and managed by you. You might need to change the agent configuration.

-In this article, you'll change your ingestion agent configuration and roll back a configuration change.

-## Prerequisites

-## Update agent configuration

-> [!WARNING]

-> Changing the configuration requires restarting the agent. For the MCC EDR source, a small number of EDRs being handled might be dropped.  It is not possible to gracefully restart without dropping any data. For safety, update agents one at a time, only updating the next when you are sure the previous was successful.

-> [!WARNING]

-> If you change the pipeline ID for an SFTP pull source, the agent treats it as a new source and might upload duplicate files with the new pipeline ID. To avoid this, add the `exclude_before_time` parameter to the file source configuration. For example, if you configure `exclude_before_time: "2024-01-01T00:00:00-00:00"` then any files last modified before midnight on January 1, 2024 UTC will be ignored by the agent.

-If you need to change the agent's configuration, carry out the following steps.

-1. Save a copy of the existing */etc/az-aoi-ingestion/config.yaml* configuration file.

-1. Edit the configuration file to change the config values.

-1. Restart the agent.

- ```

- sudo systemctl restart az-aoi-ingestion.service

- ```

-## Roll back configuration changes

-If a configuration change fails:

-1. Copy the backed-up configuration file from before the change to the */etc/az-aoi-ingestion/config.yaml* file.

-1. Restart the agent.

- ```

- sudo systemctl restart az-aoi-ingestion.service

- ```

-## Related content

-Learn how to:

-description: This article helps you understand how data quality and quality monitoring work in Azure Operator Insights.

-# Data quality and quality monitoring

-Every Data Product working on Azure Operator Insights platform has built-in support for data quality monitoring. Data quality is crucial because it ensures accurate, reliable, and trustworthy information for decision-making. It prevents costly mistakes, builds credibility with customers and regulators, and enables personalized experiences.

-Azure Operator Insights platform monitors data quality when data is ingested into Data Product input storage (the Data Product Input block in the following image) and after data is processed and made available to customers (the Data Product Compute block in the following image).

- Diagram of the Azure Operator Insights architecture. It shows ingestion by ingestion agents from on-premises data sources, processing in a Data Product, and analysis and use in Logic Apps and Power BI.

-## Quality dimensions

-Data quality dimensions are the various aspects or characteristics that define the quality of data. Azure Operator Insights support the following dimensions:

-## Metrics

-All data quality dimensions are covered by quality metrics produced by Azure Operator Insights platform. There are two types of the quality metrics:

-The basic quality metrics produced by the platform are available in the following table.

-| **Metric** | **Dimension** | **Data Source** |

-|-||--|

-| Number of ingested rows | Timeliness | Ingested |

-| Number of rows containing null for required columns | Completeness | Ingested |

-| Number of rows failed validation against schema | Validity | Ingested |

-| Number of filtered out rows | Completeness | Ingested |

-| Number of processed rows | Timeliness | Processed |

-| Number of incomplete rows, which don't contain required data | Completeness | Processed |

-| Number of duplicated rows | Uniqueness | Processed |

-| Percentiles for overall lag between record generation and available for querying | Timeliness | Processed |

-| Percentiles for lag between record generation and ingested into input storage | Timeliness | Processed |

-| Percentiles for lag between data ingested and processed | Timeliness | Processed |

-| Percentiles for lag between data processed and available for querying | Timeliness | Processed |

-| Ages for materialized views | Timeliness | Processed |

-The custom data quality metrics are implemented on per Data Product basis. These metrics cover the accuracy and consistency dimensions. Data Product documentation contains description for the custom quality metrics available.

-## Monitoring

-All Azure Operator Insight Data Products are deployed with a dashboard showing quality metrics. You can use the dashboard to monitor quality of their data.

-All data quality metrics are saved to the Data Product ADX tables. For exploration of the data quality metrics, you can use the standard Data Product KQL endpoint and then extend the dashboard if necessary.

-description: This article provides an overview of the data types used by Azure Operator Insights Data Products.

-#CustomerIntent: As a Data Product user, I want to understand the concept of Data Types so that I can use Data Product(s) effectively.

-# Data types in Azure Operator Insights

-A Data Product ingests data from one or more sources, digests and enriches this data, and presents this data to provide domain-specific insights and to support further data analysis.

-A data type is used to refer to an individual data source. Data types can be from outside the Data Product, such as from a network element. Data types can also be created within the Data Product itself by aggregating or enriching information from other data types.

-Data Product operators can choose the data retention period for each data type.

-## Data type contents

-Each data type contains data from a specific source. The primary source for a data type might be a network element within the subject domain. Some data types are derived by aggregating or enriching information from other data types. For a description of the data types available in a given Data Product, refer to the documentation for that Data Product.

-## Data type settings

-Data types are presented as child resources of the Data Product within the Azure portal as shown in the Data Types page. Relevant settings can be controlled independently for each individual data type.

-Data Product operators can configure different data retention periods for each data type as shown in the Data Retention page. For example, data types containing personal data are typically configured with a shorter retention period to comply with privacy legislation.

- :::image type="content" source="media/concept-data-types/data-types-data-retention.png" alt-text="Screenshot of Data Types Data Retention portal page.":::

-description: This article outlines how data is stored and visualized in Azure Operator Insights Data Products.

-#CustomerIntent: As a Data Product user, I want understand data visualization in the Data Product so that I can access my data.

-# Data visualization in Data Products overview

-The Azure Operator Insights Data Product is an Azure service that handles processing and enrichment of data. A set of dashboards is deployed with the Data Product, but users can also query and visualize the data.

-## Data Explorer

-Enriched and processed data is stored in the Data Product and is made available for querying with the Consumption URL, which you can connect to in the [Azure Data Explorer web UI](https://dataexplorer.azure.com/). Permissions are governed by role-based access control.

-The Data Product exposes a database, which contains a set of tables and materialized views. You can query this data in the Data Explorer GUI using [Kusto Query Language](/azure/data-explorer/kusto/query/).

-## Enrichment and aggregation

-The Data Product enriches the raw data by combining data from different tables together. This enriched data is then aggregated in materialized views that summarize the data over various dimensions.

-The data is enriched and aggregated after it has been ingested into the raw tables. As a result, there is a slight delay between the arrival of the raw data and the arrival of the enriched data.

-The Data Product has metrics that monitor the quality of the raw and enriched data. For more information, see [Data quality and data monitoring](concept-data-quality-monitoring.md).

-## Visualizations

-Dashboards are deployed with the Data Product. These dashboards include a set of visualizations organized according to different KPIs in the data, which can be filtered on a range of dimensions. For example, visualizations provided in the Mobile Content Cloud (MCC) Data Product include upload/download speeds and data volumes.

-For information on accessing and using the built-in dashboards, see [Use Data Product dashboards](dashboards-use.md).

-You can also create your own visualizations, either by using the KQL [render](/azure/data-explorer/kusto/query/renderoperator?pivots=azuredataexplorer) operator in the [Azure Data Explorer web UI](https://dataexplorer.azure.com/) or by creating dashboards following the guidance in [Visualize data with Azure Data Explorer dashboards](/azure/data-explorer/azure-data-explorer-dashboards).

-## Querying

-On top of the dashboards provided as part of the Data Product, the data can be directly queried in the Azure Data Explorer web UI. See [Query data in the Data Product](data-query.md) for information on accessing and querying the data.

-## Related content

-description: This article gives an overview of the Azure Operator Insights Data Products provided to monitor the Quality of Experience for the Affirmed Mobile Content Cloud (MCC).

-#CustomerIntent: As an MCC operator, I want to understand the capabilities of the relevant Quality of Experience Data Product, in order to provide insights to my network.

-# Quality of Experience - Affirmed MCC Data Product overview

-The *Quality of Experience - Affirmed MCC* Data Products support data analysis and insight for operators of the Affirmed Networks Mobile Content Cloud (MCC). They ingest Event Data Records (EDRs) from MCC network elements, and then digest and enrich this data to provide a range of visualizations for the operator. Operator data scientists have access to the underlying enriched data to support further data analysis.

-## Background

-The Affirmed Networks Mobile Content Cloud (MCC) is a virtualized Evolved Packet Core (vEPC) that can provide the following functionality.

-The data produced by the MCC varies according to the functionality. This variation affects the enrichments and visualizations that are relevant. Azure Operator Insights provides the following Quality of Experience Data Products to support specific MCC functions.

-## Data types

-The following data types are provided for all Quality of Experience - Affirmed MCC Data Products.

- - `Status`

- - `Session`

- - `Bearer`

- - `Flow`

- - `HTTP`

- - `RTT`

- - `MME CRR`

- - `SGSN CRR`

-

- > [!Note]

- > Both kinds of `CRR` records are stored in the `all_mme_sgsn_events` table.

- - `agg-enrichment-5m`: contains enriched Event Data Records aggregated over five-minute intervals.

- - `agg-enrichment-1h`: contains enriched Event Data Records aggregated over one-hour intervals.

- - `agg-enrichment-1d`: contains enriched Event Data Records aggregated over one-day intervals.

- - `enriched-flow-dcount`: contains precomputed counts used to report the unique IMSIs, MCCs, and Applications over time.

- - `agg-location-5m`: contains enriched location data aggregated over five-minute intervals.

- - `agg-location-1h`: contains enriched location data aggregated over one-hour intervals.

- - `agg-location-1d`: contains enriched location data aggregated over one-day intervals.

- - `enriched-loc-dcount`: contains precomputed counts used to report location data over time.

-## Setup

-To use the Quality of Experience - Affirmed MCC Data Product:

- - Use the information in [Required ingestion configuration](#required-ingestion-configuration) when you're setting up ingestion.

- - We recommend the Azure Operator Insights ingestion agent for the `edr` data type. To ingest the `device` and `edr-validation` data types, you can use a separate instance of the ingestion agent, or set up your own ingestion method.

- - If you're using the Azure Operator Insights ingestion agent, also meet the requirements in [Requirements for the Azure Operator Insights ingestion agent](#requirements-for-the-azure-operator-insights-ingestion-agent).

-### Required ingestion configuration

-Use the information in this section to configure your ingestion method. Refer to the documentation for your chosen method to determine how to supply these values.

-| Data type | Required container name | Requirements for data |

-||||

-| `edr` | `edr` | MCC EDR data. |

-| `device` | `device` | Device reference data. |

-| `edr-validation` | `edr-validation` | PM Stat data for `EDR_HTTP_STATS`, `EDR_FLOW_STATS`, and `EDR_SESSION_STATS` datasets. File name prefixes must match the name of the dataset. |

-### Requirements for the Azure Operator Insights ingestion agent

-Use the VM requirements to set up one or more VMs for the ingestion agent. Use the example configuration to configure the ingestion agent to upload data to the Data Product, as part of following [Install the Azure Operator Insights ingestion agent and configure it to upload data](set-up-ingestion-agent.md).

-# [EDR ingestion](#tab/edr-ingestion)

-#### VM requirements

-Each agent instance must run on its own Linux VM. The number of VMs needed depends on the scale and redundancy characteristics of your deployment. This recommended specification can achieve 1.5-Gbps throughput on a standard D4s_v3 Azure VM. For any other VM spec, we recommend that you measure throughput at the network design stage.

-Latency on the MCC to agent connection can negatively affect throughput. Latency should usually be low if the MCC and agent are colocated or the agent runs in an Azure region close to the MCC.

-Talk to the Affirmed Support Team to determine your requirements.

-Each VM running the agent must meet the following minimum specifications for EDR ingestion.

-| Resource | Requirements |

-|-||

-| OS | Red Hat Enterprise Linux 8.6 or later, or Oracle Linux 8.8 or later |

-| vCPUs | 4 |

-| Memory | 32 GB |

-| Disk | 64 GB |

-| Network | Connectivity from MCCs and to Azure |

-| Software | systemd, logrotate, and zip installed |

-| Other | SSH or alternative access to run shell commands |

-| DNS | (Preferable) Ability to resolve Microsoft hostnames. If not, you need to perform extra configuration when you set up the agent (described in [Map Microsoft hostnames to IP addresses for ingestion agents that can't resolve public hostnames](map-hostnames-ip-addresses.md).) |

-#### Deploying multiple VMs for fault tolerance

-The ingestion agent is designed to be highly reliable and resilient to low levels of network disruption. If an unexpected error occurs, the agent restarts and provides service again as soon as it's running.

-The agent doesn't buffer data, so if a persistent error or extended connectivity problems occur, EDRs are dropped.

-For extra fault tolerance, you can deploy multiple instances of the ingestion agent and configure the MCC to switch to a different instance if the original instance becomes unresponsive, or to share EDR traffic across a pool of agents. For more information, see the [Affirmed Networks Active Intelligent vProbe System Administration Guide](https://manuals.metaswitch.com/vProbe/latest/vProbe_System_Admin/Content/02%20AI-vProbe%20Configuration/Generating_SESSION__BEARER__FLOW__and_HTTP_Transac.htm) (only available to customers with Affirmed support) or speak to the Affirmed Networks Support Team.

-# [Performance management and device data ingestion](#tab/pm-stat-or-device-data-ingestion)

-#### Performance management ingestion via an SFTP server

-If you're using the Azure Operator Insights ingestion agent to ingest performance management stats files for the `edr-validation` data type:

-|Information | Configuration setting for Azure Operator Ingestion agent | Recommended value |

-| | | |

-| [Settling time](ingestion-agent-overview.md#processing-files) | `source.sftp_pull.filtering.settling_time` | `60s` (upload files that haven't been modified in the last 60 seconds) |

-| Schedule for checking for new files | `source.sftp_pull.scheduling.cron` | `0 */5 * * * * *` (every 5 minutes) |

-#### Device data ingestion via an SFTP server

-If the device data is stored on an SFTP server, you can ingest device data by configuring an extra `sftp_pull` ingestion pipeline on the same ingestion agent instance that you're using for PM stat ingestion. You can choose your own value for `source.sftp_pull.scheduling.cron` for the device data pipeline, depending on how frequently you want the ingestion pipeline to check for new device data files.

-> [!TIP]

-> For more information about all the configuration options for the ingestion agent, see [Configuration reference for Azure Operator Insights ingestion agent](ingestion-agent-configuration-reference.md).

-#### VM requirements

-Each agent instance running SFTP pull pipelines must run on a separate Linux VM to any agent instance used for EDR ingestion. The number of VMs needed depends on the scale and redundancy characteristics of your deployment.

-As a guide, this table documents the throughput that the recommended specification on a standard D4s_v3 Azure VM can achieve.

-| File count | File size (KiB) | Time (seconds) | Throughput (Mbps) |

-||--|-|-|

-| 64 | 16,384 | 6 | 1,350 |

-| 1,024 | 1,024 | 10 | 910 |

-| 16,384 | 64 | 80 | 100 |

-| 65,536 | 16 | 300 | 25 |

-Each Linux VM running the agent must meet the following minimum specifications for SFTP pull ingestion.

-| Resource | Requirements |

-|-||

-| OS | Red Hat Enterprise Linux 8.6 or later, or Oracle Linux 8.8 or later |

-| vCPUs | Minimum 4, recommended 8 |

-| Memory | Minimum 32 GB |

-| Disk | 30 GB |

-| Network | Connectivity to the SFTP server and to Azure |

-| Software | systemd, logrotate, and zip installed |

-| Other | SSH or alternative access to run shell commands |

-| DNS | (Preferable) Ability to resolve Microsoft hostnames. If not, you need to perform extra configuration when you set up the agent (described in [Map Microsoft hostnames to IP addresses for ingestion agents that can't resolve public hostnames](map-hostnames-ip-addresses.md).) |

-### Configuration for Affirmed MCCs

-After installing and configuring your ingestion agents, configure the MCCs to send EDRs to them.

-Follow the steps in "Generating SESSION, BEARER, FLOW, and HTTP Transaction EDRs" in the [Affirmed Networks Active Intelligent vProbe System Administration Guide](https://manuals.metaswitch.com/vProbe/latest) (only available to customers with Affirmed support), making the following changes:

- - `port`: 36001

- - `encoding`: protobuf

- - `keep-alive`: 2 seconds

-### Configuration for Affirmed EMS

-If you're using the `edr-validation` data type, configure the EMS to export the relevant performance management statistics to a remote server. If you're using the Azure Operator Insights ingestion agent to ingest performance management statistics, the remote server must be an [SFTP server](set-up-ingestion-agent.md#prepare-the-sftp-server), otherwise the remote server needs to be accessible by your ingestion method.

-1. Obtain the IP address, user, and password of the remote server.

-1. Configure the transfer of EMS statistics to a remote server

- - Use the instructions in [Copying Performance Management Statistics Files to Destination Server](https://manuals.metaswitch.com/MCC/13.1/Acuitas_Users_RevB/Content/Appendix%20Interfacing%20with%20Northbound%20Interfaces/Exported_Performance_Management_Data.htm#northbound_2817469247_308739) in the _Acuitas User's Guide_.

- - For `edr-validation`, you only need to export three CSV files. List these file names in the `opt/Affirmed/NMS/conf/pm/mcc.files.txt` file on the EMS:

- - `EDR_HTTP_STATS`

- - `EDR_FLOW_STATS`

- - `EDR_SESSION_STATS`

-> [!IMPORTANT]

-> Increase the frequency of the cron job by reducing the `timeInterval` argument from `15` (default) to `5` minutes.

-## Related content

-> [!NOTE]

-> Affirmed Networks login credentials are required to access the MCC product documentation.

-description: This article gives an overview of the Monitoring - Affirmed MCC Data Product provided by Azure Operator Insights.

-#CustomerIntent: As an MCC operator, I want to understand the capabilities of the Monitoring - Affirmed MCC Data Product so that I can use it to provide insights to my network.

-# Monitoring - Affirmed MCC Data Product overview

-The Monitoring - Affirmed MCC Data Product supports data analysis and insight for operators of the Affirmed Networks Mobile Content Cloud (MCC). It ingests performance management data (performance statistics) from MCC network elements. It then digests and enriches this data to provide visualizations for the operator and to provide access to the underlying enriched data for operator data scientists.

-## Background

-The Affirmed Networks Mobile Content Cloud (MCC) is a virtualized Evolved Packet Core (vEPC) that can provide the following functionality.

-The Monitoring - Affirmed MCC Data Product supports all of the MCC variants described.

-## Data types

-The following data type is provided as part of the Monitoring - Affirmed MCC Data Product.

-## Setup

-To use the Monitoring - Affirmed MCC Data Product:

-1. Deploy the Data Product by following [Create an Azure Operator Insights Data Product](data-product-create.md).

-1. Configure your network to produce performance management data, as described in [Required network configuration](#required-network-configuration).

-1. Set up ingestion (data upload) from your network. For example, you could use the [Azure Operator Insights ingestion agent](ingestion-agent-overview.md) or [connect Azure Data Factory](ingestion-with-data-factory.md) to your Data Product.

- - Use the information in [Required ingestion configuration](#required-ingestion-configuration) when you're setting up ingestion.

- - If you're using the Azure Operator Insights ingestion agent, also meet the requirements in [Requirements for the Azure Operator Insights ingestion agent](#requirements-for-the-azure-operator-insights-ingestion-agent).

-### Required network configuration

-Configure the EMS server to export performance management data to a remote server. If you're using the Azure Operator Insights ingestion agent, the remote server must be an [SFTP server](set-up-ingestion-agent.md#prepare-the-sftp-server). If you're providing your own ingestion agent, the remote server needs to be accessible by your ingestion agent.

-

-1. Obtain the IP address, user, and password of the remote server.

-1. Configure the transfer of EMS statistics to a remote server by following [Copying Performance Management Statistics Files to Destination Server](https://manuals.metaswitch.com/MCC/13.1/Acuitas_Users_RevB/Content/Appendix%20Interfacing%20with%20Northbound%20Interfaces/Exported_Performance_Management_Data.htm#northbound_2817469247_308739) in the _Acuitas User's Guide_.

-> [!IMPORTANT]

-> Increase the frequency of the cron job by reducing the `timeInterval` argument from `15` (default) to `5` minutes.

-### Required ingestion configuration

-Use the information in this section to configure your ingestion method. Refer to the documentation for your chosen method to determine how to supply these values.

-| Data type | Required container name | Requirements for data |

-||||

-| `pmstats` | `pmstats` | Performance data from MCC nodes. File names must start with the dataset name. For example, `WORKFLOWPERFSTATSSLOT` data must be ingested in files whose names start with `WORKFLOWPERFSTATSSLOT`. |

-If you're using the Azure Operator Insights ingestion agent:

-|Information | Configuration setting for Azure Operator Ingestion agent | Recommended value |

-| | | |

-| [Settling time](ingestion-agent-overview.md#processing-files) | `source.sftp_pull.filtering.settling_time` | `60s` (upload files that haven't been modified in the last 60 seconds) |

-| Schedule for checking for new files | `source.sftp_pull.scheduling.cron` | `0 */5 * * * * *` (every 5 minutes) |

-> [!TIP]

-> For more information about all the configuration options for the ingestion agent, see [Configuration reference for Azure Operator Insights ingestion agent](ingestion-agent-configuration-reference.md).

-### Requirements for the Azure Operator Insights ingestion agent

-The Azure Operator Insights ingestion agent collects files from _ingestion pipelines_ that you configure on it. Ingestion pipelines include the details of the SFTP server, the files to collect from it and how to manage those files.

-You must choose how to set up your agents, pipelines, and VMs using the following rules.

- - The scale and redundancy characteristics of your deployment.

- - The number and size of the files, and how frequently the files are copied.

-As a guide, this table documents the throughput that the recommended specification on a standard D4s_v3 Azure VM can achieve.

-| File count | File size (KiB) | Time (seconds) | Throughput (Mbps) |

-||--|-|-|

-| 64 | 16,384 | 6 | 1,350 |

-| 1,024 | 1,024 | 10 | 910 |

-| 16,384 | 64 | 80 | 100 |

-| 65,536 | 16 | 300 | 25 |

-For example, if you need to collect from two file sources, you could:

-Each Linux VM running the agent must meet the following minimum specifications.

-| Resource | Requirements |

-|-||

-| OS | Red Hat Enterprise Linux 8.6 or later, or Oracle Linux 8.8 or later |

-| vCPUs | Minimum 4, recommended 8 |

-| Memory | Minimum 32 GB |

-| Disk | 30 GB |

-| Network | Connectivity to the SFTP server and to Azure |

-| Software | systemd, logrotate, and zip installed |

-| Other | SSH or alternative access to run shell commands |

-| DNS | (Preferable) Ability to resolve Microsoft hostnames. If not, you need to perform extra configuration when you set up the agent (described in [Map Microsoft hostnames to IP addresses for ingestion agents that can't resolve public hostnames](map-hostnames-ip-addresses.md).) |

-## Related content

-> [!NOTE]

-> Affirmed Networks login credentials are required to access the MCC product documentation.

-description: Learn how to add and remove user permissions to the KQL consumption URL for Azure Operator Insights.

-# Manage permissions to the KQL consumption URL

-Azure Operator Insights enables you to control access to the KQL consumption URL of each Data Product based on email addresses or distribution lists. Use the following steps to configure read-only access to the consumption URL.

-Azure Operator Insights supports a single role that gives Read access to all tables and columns on the consumption URL.

-## Add user access

-1. Sign in to the [Azure portal](https://portal.azure.com).

-1. Go to your Azure Operator Insights Data Product resource.

-1. In the left-hand menu under **Security**, select **Permissions**.

-1. Select **Add Reader** to add a new user.

-1. Type in the user's email address or distribution list and select **Add Reader(s)**.

-1. Wait for about 30 seconds, then refresh the page to view your changes.

-## Remove user access

-1. Sign in to the [Azure portal](https://portal.azure.com).

-1. Go to your Azure Operator Insights Data Product resource.

-1. In the left-hand menu under **Security**, select **Permissions**.

-1. Select the **Delete** symbol next to the user who you want to remove.

- > [!NOTE]

- > There is no confirmation dialog box, so be careful when deleting users.

-1. Wait for about 30 seconds, then refresh the page to view your changes.

-description: This article outlines how to access and use dashboards in the Azure Operator Insights Data Product.

-#CustomerIntent: As a Data Product user, I want to access dashboards so that I can view my data.

-# Use Data Product dashboards to visualize data

-This article covers accessing and using the dashboards in the Azure Operator Insights Data Product.

-## Prerequisites

-A deployed Data Product, see [Create an Azure Operator Insights Data Product](data-product-create.md).

-## Get access to the dashboards

-Access to the dashboards is controlled by role-based access control (RBAC).

-1. In the Azure portal, select the Data Product resource and open the Permissions pane. You must have the `Reader` role. If you do not, contact an owner of the resource to grant you `Reader` permissions.

-1. In the Overview pane of the Data Product, open the link to the dashboards.

-1. Select any dashboard to open it and view the visualizations.

-## Filter data

-Each dashboard is split into pages with a set of filters at the top of the page.

- You can enter multiple values in the free text fields by separating the inputs with a comma and no spaces, for example: `London,Paris`.

-Some tiles report `UNDETECTED` for any filters with an empty entry. You can't filter these undetected entries.

-## Exploring the queries

-Each tile in a dashboard runs a query against the data. To edit these queries and run them manually, you can open these queries in the query editor.

-1. Select the ellipsis in the top right corner of the tile, and select **Explore Query**.

-1. Your query opens in a new tab in the query editor. If the query is all on one line, right-click the query block and select **Format Document**.

-1. Select **Run** or press *Shift + Enter* to run the query.

-## Editing the dashboards

-Users with Edit permissions on dashboards can make changes.

-1. In the dashboard, change the state from **Viewing** to **Editing** in the top left of the screen.

-1. Select **Add** to add new tiles, or select the pencil to edit existing tiles.

-## Related content

-description: In this article, learn how to deploy an Azure Operator Insights Data Product resource.

-# Deploy an Azure Operator Insights Data Product

-In this article, you learn how to create an Azure Operator Insights Data Product instance.

-> [!NOTE]

-> Access is currently only available by request. More information is included in the application form. We appreciate your patience as we work to enable broader access to Azure Operator Insights Data Product. Apply for access by [filling out this form](https://aka.ms/AAn1mi6).

-## Prerequisites

-## Prepare your Azure portal or Azure CLI environment

-You can use the Azure portal or the Azure CLI to follow the steps in this article.

-# [Portal](#tab/azure-portal)

-Confirm that you can sign in to the [Azure portal](https://portal.azure.com) and can access the subscription.

-# [Azure CLI](#tab/azure-cli)

-You can run Azure CLI commands in one of two ways:

-### Use Azure Cloud Shell

-Azure Cloud Shell is a free Bash shell that you can run directly within the Azure portal. The Azure CLI is preinstalled and configured to use with your account. Select the **Cloud Shell** button on the menu in the upper-right section of the Azure portal:

-[](https://portal.azure.com)

-The button launches an interactive shell that you can use to run the steps outlined in this how-to article:

-[](https://portal.azure.com)

-### Install the Azure CLI locally

-You can also install and use the Azure CLI locally. If you plan to use Azure CLI locally, make sure you have installed the latest version of the Azure CLI. See [Install the Azure CLI](/cli/azure/install-azure-cli).

-To log into your local installation of the CLI, run the az sign-in command:

-```azurecli-interactive

-az login

-```

-### Change the active subscription

-Azure subscriptions have both a name and an ID. You can switch to a different subscription with [az account set](/cli/azure/account#az-account-set), specifying the desired subscription name or ID.

- ```azurecli-interactive

- az account set --subscription "<SubscriptionName>"

- ```

- ```azurecli-interactive

- az account set --subscription "<SubscriptionID>"

- ```

-> [!NOTE]

-> Replace any values shown in the form \<KeyVaultName\> with the values for your deployment.

-## Create a resource group

-A resource group is a logical container into which Azure resources are deployed and managed.

-# [Portal](#tab/azure-portal)

-If you plan to use CMK-based data encryption or Microsoft Purview, set up a resource group now:

-1. Sign in to the [Azure portal](https://portal.azure.com).

-1. Select **Resource groups**.

-1. Select **Create** and follow the prompts.

-For more information, see [Create resource groups](../azure-resource-manager/management/manage-resource-groups-portal.md#create-resource-groups).

-If you don't plan to use CMK-based date encryption or Microsoft Purview, you can set up a resource group now or when you [create the Data Product resource](#create-an-azure-operator-insights-data-product-resource).

-# [Azure CLI](#tab/azure-cli)

-Use the `az group create` command to create a resource group named \<ResourceGroup\> in the region where you want to deploy.

-```azurecli-interactive

-az group create --name "<ResourceGroup>" --location "<Region>"

-```

-## Set up resources for CMK-based data encryption or Microsoft Purview

-If you plan to use CMK-based data encryption or Microsoft Purview, you must set up an Azure Key Vault instance and a user-assigned managed identity (UAMI) first.

-### Set up a key in an Azure Key Vault

-An Azure Key Vault instance stores your Customer Managed Key (CMK) for data encryption. The Data Product uses this key to encrypt your data over and above the standard storage encryption. You need to have Subscription/Resource group owner permissions to perform this step.

-# [Portal](#tab/azure-portal)

-1. [Create an Azure Key Vault resource](/azure/key-vault/general/quick-create-portal) in the same subscription and resource group that you set up in [Create a resource group](#create-a-resource-group).

-1. Provide your user account with the Key Vault Administrator role on the Azure Key Vault resource. This is done via the **Access Control (IAM)** tab on the Azure Key Vault resource.

-1. Navigate to the object and select **Keys**. Select **Generate/Import**.

-1. Enter a name for the key and select **Create**.

-1. Select the newly created key and select the current version of the key.

-1. Copy the Key Identifier URI to your clipboard to use when creating the Data Product.

-# [Azure CLI](#tab/azure-cli)

-<!-- CLI link is [Create an Azure Key Vault resource](../key-vault/general/quick-create-cli.md) in the same subscription and resource group where you intend to deploy the Data Product resource. -->

-#### Create a key vault

-Use the Azure CLI `az keyvault create` command to create a Key Vault in the resource group from the previous step. You must provide:

-

-```azurecli-interactive

-az keyvault create --name "<KeyVaultName>" --resource-group "<ResourceGroup>" --location "<Region>"

-```

-The output of this command shows properties of the newly created key vault. Take note of:

-At this point, your Azure account is the only one authorized to perform any operations on this new vault.

-#### Assign roles for the key vault

-Provide your user account with the Key Vault Administrator role on the Azure Key Vault resource.

-```azurecli-interactive

-az role assignment create --role "Key Vault Administrator" --assignee <YourEmailAddress> --scope /subscriptions/<SubscriptionID>/resourcegroups/<ResourceGroup>/providers/Microsoft.KeyVault/vaults/<KeyVaultName>

-```

-#### Create a key

-```azurecli-interactive

-az keyvault key create --vault-name "<KeyVaultName>" -n <keyName> --protection software

-```

-From the output screen, copy the `KeyID` and store it in your clipboard for later use.

-<!-- PowerShell link is [Create an Azure Key Vault resource](../key-vault/general/quick-create-powershell.md) in the same subscription and resource group where you intend to deploy the Data Product resource. -->

-### Set up a user-assigned managed identity

-# [Portal](#tab/azure-portal)

-1. [Create a user-assigned managed identity](../active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities.md?pivots=identity-mi-methods-azp#create-a-user-assigned-managed-identity) using Microsoft Entra ID for CMK-based encryption. The Data Product also uses the user-assigned managed identity (UAMI) to interact with the Microsoft Purview account.

-1. Navigate to the Azure Key Vault resource that you created earlier and assign the UAMI with **Key Vault Administrator** role.

-# [Azure CLI](#tab/azure-cli)

-<!-- Managed identity link for the CLI: /entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azcli -->

-#### Create a user-assigned managed identity

-To create a user-assigned managed identity, your account needs the Managed Identity Contributor role assignment.

-Use the `az identity create` command to create a user-assigned managed identity. The -g parameter specifies the resource group where to create the user-assigned managed identity. The -n parameter specifies its name. Replace the \<ResourceGroup\> and \<UserAssignedIdentityName\> parameter values with your own values.

-> [!IMPORTANT]

-> When you create user-assigned managed identities, only alphanumeric characters (0-9, a-z, and A-Z) and the hyphen (-) are supported.

-```azurecli-interactive

-az identity create -g <ResourceGroup> -n <UserAssignedIdentityName>

-```

-Copy the `principalId` from the output screen and store it in your clipboard for later use.

-#### Assign the user-assigned managed identity to the key vault

-```azurecli-interactive

-az role assignment create --role "Key Vault Administrator" --assignee <principalId> --scope /subscriptions/<SubscriptionID>/resourcegroups/<ResourceGroup>/providers/Microsoft.KeyVault/vaults/<KeyVaultName>

-```

-<!-- Managed identity link for PowerShell: /entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-powershell -->

-## Create an Azure Operator Insights Data Product resource

-You create the Azure Operator Insights Data Product resource.

-# [Portal](#tab/azure-portal)

-1. Sign in to the [Azure portal](https://portal.azure.com/).

-1. In the search bar, search for Operator Insights and select **Azure Operator Insights - Data Products**.

-1. On the Azure Operator Insights - Data Products page, select **Create**.

-1. On the Basics tab of the **Create a Data Product** page:

- 1. Select your subscription.

- 1. Select the resource group you previously created for the Key Vault resource.

- 1. Under **Instance details**, complete the following fields:

- - **Name** - Enter the name for your Data Product resource. The name must start with a lowercase letter and can contain only lowercase letters and numbers.

- - **Publisher** - Select the organization that created and published the Data Product that you want to deploy.

- - **Product** - Select the name of the Data Product.

- - **Version** - Select the version.

- Select **Next: Advanced**.

- :::image type="content" source="media/data-product-selection.png" alt-text="Screenshot of the Instance details section of the Basics configuration for a Data Product in the Azure portal.":::

-

-1. In the Advanced tab of the **Create a Data Product** page:

- 1. Enable Purview if you're integrating with Microsoft Purview.

- Select the subscription for your Purview account, select your Purview account, and enter the Purview collection ID.

- 1. Enable Customer managed key if you're using CMK for data encryption.

- 1. Select the user-assigned managed identity that you set up as a prerequisite.

- 1. Carefully paste the Key Identifier URI that was created when you set up Azure Key Vault as a prerequisite.

-

-1. To add one or more owners for the Data Product, which will also appear in Microsoft Purview, select **Add owner**, enter the email address, and select **Add owners**.

-1. In the Tags tab of the **Create a Data Product** page, select or enter the name/value pair used to categorize your Data Product resource.

-1. Select **Review + create**.

-1. Select **Create**. Your Data Product instance is created in about 20-25 minutes. During this time, all the underlying components are provisioned. After this process completes, you can work with your data ingestion, explore sample dashboards and queries, and so on.

-# [Azure CLI](#tab/azure-cli)

-To create an Azure Operator Insights Data Product with the minimum required parameters, use the following command:

-```azurecli-interactive

-az network-analytics data-product create --name <DataProductName> --resource-group <ResourceGroup> --location <Region> --publisher Microsoft --product <ProductName> --major-version <ProductMajorVersion>

-```

-Use the following values for \<ProductName\> and \<ProductMajorVersion>.

-|Data Product |\<ProductName\> |\<ProductMajorVersion>|

-||||

-|Quality of Experience - Affirmed MCC GIGW |`Quality of Experience - Affirmed MCC GIGW`|`1.0`|

-|Quality of Experience - Affirmed MCC PGW or GGSN |`Quality of Experience - Affirmed MCC PGW or GGSN`|`1.0`|

-|Monitoring - Affirmed MCC|`Monitoring - Affirmed MCC`|`0` or `1`|

-To create an Azure Operator Insights DataProduct with all parameters, use the following command:

-```azurecli-interactive

-az network-analytics data-product create --name <DataProductName> --resource-group <ResourceGroup> --location <Region> --publisher Microsoft --product <ProductName> --major-version <ProductMajorVersion --owners <<xyz@email>> --customer-managed-key-encryption-enabled Enabled --key-encryption-enable Enabled --encryption-key '{"keyVaultUri":"<VaultURI>","keyName":"<KeyName>","keyVersion":"<KeyVersion>"}' --purview-account <PurviewAccount> --purview-collection <PurviewCollection> --identity '{"type":"userAssigned","userAssignedIdentities":{"/subscriptions/<SubscriptionID>/resourceGroups/<ResourceGroup>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<UserAssignedIdentityName>"}}' --tags '{"key1":"value1","key2":"value2"}'

-```

-## Deploy sample insights

-Once your Data Product instance is created, you can deploy a sample insights dashboard. This dashboard works with the sample data that came along with the Data Product instance.

-1. Navigate to your Data Product resource on the Azure portal and select the Permissions tab on the Security section.

-1. Select **Add Reader**. Type the user's email address to be added to Data Product reader role.

-> [!NOTE]

-> The reader role is required for you to have access to the insights consumption URL.

-3. Download the sample JSON template file for your Data Product's dashboard:

- * Quality of Experience - Affirmed MCC GIGW: [https://go.microsoft.com/fwlink/p/?linkid=2254536](https://go.microsoft.com/fwlink/p/?linkid=2254536)

- * Monitoring - Affirmed MCC: [https://go.microsoft.com/fwlink/p/?linkid=2254551](https://go.microsoft.com/fwlink/?linkid=2254551)

-1. Copy the consumption URL from the Data Product overview screen into the clipboard.

-1. Open a web browser, paste in the URL and select enter.

-1. When the URL loads, select on the Dashboards option on the left navigation pane.

-1. Select the **New Dashboard** drop down and select **Import dashboard from file**. Browse to select the JSON file downloaded previously, provide a name for the dashboard, and select **Create**.

-1. Select the three dots (...) at the top right corner of the consumption URL page and select **Data Sources**.

-1. Select the pencil icon next to the Data source name to edit the data source.

-1. Under the Cluster URI section, replace the URL with your Data Product consumption URL and select connect.

-1. In the Database drop-down, select your Database. Typically, the database name is the same as your Data Product instance name. Select **Apply**.

-> [!NOTE]

-> These dashboards are based on synthetic data and may not have complete or representative examples of the real-world experience.

-## Explore sample data using Kusto

-The consumption URL also allows you to write your own Kusto query to get insights from the data.

-1. On the Overview page, copy the consumption URL and paste it in a new browser tab to see the database and list of tables.

- :::image type="content" source="media/data-product-properties.png" alt-text="Screenshot of part of the Overview pane in the Azure portal, showing the consumption URL.":::

-1. Use the ADX query plane to write Kusto queries.

- * For Quality of Experience - Affirmed MCC GIGW, try the following queries:

- ```kusto

- enriched_flow_events_sample

- | summarize Application_count=count() by flowRecord_dpiStringInfo_application

- | order by Application_count desc

- | take 10

- ```

- ```kusto

- enriched_flow_events_sample

- | summarize SumDLOctets = sum(flowRecord_dataStats_downLinkOctets) by bin(eventTimeFlow, 1h)

- | render columnchart

- ```

- * For Monitoring - Affirmed MCC Data Product, try the following queries:

- ```kusto

- SYSTEMCPUSTATISTICSCORELEVEL_SAMPLE

- | where systemCpuStats_core >= 25 and systemCpuStats_core <= 36

- | summarize p90ssm_avg_1_min_cpu_util=round(percentile(ssm_avg_1_min_cpu_util, 90), 2) by resourceId

- ```

- ```kusto

- PGWCALLPERFSTATSGRID_SAMPLE

- | summarize clusterTotal=max(NumUniqueSubscribers) by bin(timestamp, 1d)

- | render linechart

- ```

-## Optionally, delete Azure resources

-If you're using this Data Product to explore Azure Operator Insights, you should delete the resources you've created to avoid unnecessary Azure costs.

-# [Portal](#tab/azure-portal)

-1. On the **Home** page of the Azure portal, select **Resource groups**.

-1. Select the resource group for your Azure Operator Insights Data Product and verify that it contains the Azure Operator Insights Data Product instance.

-1. At the top of the Overview page for your resource group, select **Delete resource group**.

-1. Enter the resource group name to confirm the deletion, and select **Delete**.

-# [Azure CLI](#tab/azure-cli)

-```azurecli-interactive

-az group delete --name "ResourceGroup"

-```

-## Next step

-Upload data to your Data Product:

-1. Read the documentation for your Data Product to determine any requirements for ingestion.

-2. Set up an ingestion solution:

- - To use the Azure Operator Insights ingestion agent, [install and configure the agent](set-up-ingestion-agent.md).

- - To use [Azure Data Factory](/azure/data-factory/), follow [Use Azure Data Factory to ingest data into an Azure Operator Insights Data Product](ingestion-with-data-factory.md).

-description: Learn about the data product factory (preview) for Azure Operator Insights, and how it can help you design and create new Data Products.

-#CustomerIntent: As a partner developing a Data Product, I want to understand what the data product factory is so that I can use it.

-# What is the Azure Operator Insights data product factory (preview)?

-Azure Operator Insights Data Products process data from operator networks, enrich it, and make it available for analysis. They can include prebuilt dashboards, and allow operators to view their data in other analysis tools. For more information, see [What is Azure Operator Insights?](overview.md).

-The Azure Operator Insights data product factory (preview) allows partners to easily design and create new Data Products for the Azure Operator Insights platform. Partners can develop pipelines to analyze network data and offer insights, while allowing the Azure Operator Insights platform to process operator-scale data.

-The data product factory is built on the Azure Operator Insights platform, which provides low-latency, transformation and analysis. You can publish Data Products from the data product factory to the Azure Marketplace for monetization.

-## Features of the data product factory (preview)

-The data product factory (preview) offers:

- - Design of data pipelines for ingesting data, transforming it and generating insights.

- - Configuration of Microsoft Purview catalogs for data governance.

- - Data quality metrics.

- - Integration into workflow and ticketing systems empowering automation based on AI-generated insights.

- - Integration into network-wide management solutions such as OSS/NMS platforms.

-## Using the data product factory (preview)

-The data product factory (preview) is a self-service environment for partners to design, create, and test new Data Products.

-Each Data Product is defined by a data product definition: a set of files defining the transformation, aggregation, summarization, and visualization of the data.

-The data product factory is delivered as a GitHub-based SDK containing:

-## Next step

-Apply for access to the data product factory SDK by filling in the [application form](https://forms.office.com/r/vMP9bjQr6n).

-description: This article outlines how to access and query the data in the Azure Operator Insights Data Product.

-#CustomerIntent: As a consumer of the Data Product, I want to query data that has been collected so that I can visualize the data and gain customized insights.

-# Query data in the Data Product

-This article outlines how to access and query your data.

-The Azure Operator Insights Data Product stores enriched and processed data, which is available for querying with the Consumption URL.

-## Prerequisites

- - To check your access, sign in to the [Azure portal](https://portal.azure.com), go to the Data Product resource and open the **Permissions** pane. You must have the `Reader` role.

- - If you don't have this role, ask an owner of the resource to give you `Reader` permissions by following [Manage permissions to the consumption URL](consumption-plane-configure-permissions.md).

-## Add the consumption URL in Azure Data Explorer

-1. Sign in to the [Azure portal](https://portal.azure.com).

-1. Go to your Azure Operator Insights Data Product resource.

-1. In the **Overview** pane, copy the Consumption URL.

-1. Open the [Azure Data Explorer web UI](https://dataexplorer.azure.com/) and select **Add** > **Connection**.

-1. Paste your Consumption URL in the connection box and select **Add**.

-For more information, see [Add a cluster connection in the Azure Data Explorer web UI](/azure/data-explorer/add-cluster-connection).

-## Perform a query

-Now that you have access to your data, confirm you can run a query.

-1. In the [Azure Data Explorer web UI](https://dataexplorer.azure.com/), expand the drop-down for the Data Product Consumption URL for which you added a connection.

-1. Double-click on the database you want to run your queries against. This database is set as the context in the banner above the query editor.

-1. In the query editor, run one of the following simple queries to check access to the data.

-```kql

-// Lists all available tables in the database.

-.show tables

-// Returns the schema of the named table. Replace $TableName with the name of table in the database.

-$TableName

-| getschema

-// Take the first entry of the table. Replace $TableName with the name of table in the database.

-$TableName

-| take 1

-```

-With access to the data, you can run queries to gain insights or you can visualize and analyze your data. These queries are written in [Kusto Query Language (KQL)](/azure/data-explorer/kusto/query/).

-Aggregated data in the Data Product is stored in [materialized views](/azure/data-explorer/kusto/management/materialized-views/materialized-view-overview). These views can be queried like tables, or by using the [materialized_view() function](/azure/data-explorer/kusto/query/materialized-view-function). Queries against materialized views are highly performant when using the `materialized_view()` function.

-## Related content

-description: Learn about the schema needed to implement the Device data type in the Quality of Experience ΓÇô Affirmed MCC Data Product for Azure Operator Insights.

-<!-- #CustomerIntent: As a Data Product user, I want to add the ability to add device reference data to further enrich the MCC Event Data Records-->

-# Device reference schema for the Quality of Experience Affirmed MCC Data Product

-You can enrich Event Data Record (EDR) data in the Quality of Experience Affirmed MCC Data Product with information about the devices involved in the session. You must provide this device data in the `device` data type. Prepare CSV files that conform to the following schema and upload the files into the input Azure Data Lake Storage. For more information about data types, including the `device` data type, see [Data types](concept-data-types.md).

-## Schema for device reference information

-| Source field name | Type | Description |

-| | | |

-| `TAC` | String | Type Allocation Code (TAC): a unique identifier assigned to mobile devices. Typically first eight digits of IMEI number. Can have leading zeros if TAC is six or seven digits. Matched against the IMEI in the session EDRs |

-| `Make` | String | The manufacturer or brand of the mobile device. |

-| `Model` | String | The specific model or version of the mobile device. |

-| `DeviceType` | String | Categorizes the mobile device based on its primary function (for example, handheld or feature phone). |

-| `PlatformType` | String | Identifies the underlying operating system or platform running on the mobile device. |

-| `IsOwnedDevice` | String | Indicates if the device model was ranged by the operator. A value of `Y`/`1` signifies it is, while `N`/`0` indicates it isn't. |

-| `Is3G` | String | Indicates whether the mobile device supports 3G. A value of `Y`/`E`/`1` signifies 3G capability, while `N`/`0` indicates its absence. |

-| `IsLTE` | String | Indicates whether the mobile device supports Long-Term Evolution (LTE) technology. A value of `Y`/`E`/`1` signifies LTE capability, while `N`/`0` indicates its absence |

-| `IsVoLTE` | String | Indicates whether the mobile device supports Voice over LTE. A value of `Y`/`E`/`1` signifies VoLTE capability, while `N`/`0` indicates its absence. |

-| `Is5G` | String | Indicates whether the mobile device supports 5G. A value of `Y`/`E`/`1` signifies 5G capability, while `N`/`0` indicates its absence. |

-description: This article documents the complete set of configuration for the Azure Operator Insights ingestion agent.

-# Configuration reference for Azure Operator Insights ingestion agent

-This reference provides the complete set of configuration for the [Azure Operator Insights ingestion agent](ingestion-agent-overview.md), listing all fields with explanatory comments.

-Configuration comprises three parts:

-This reference shows two pipelines: one with an MCC EDR source and one with an SFTP pull source.

-```yaml

-# A unique identifier for this agent instance. Reserved URL characters must be percent-encoded. It's included in the upload path to the Data Product's input storage account.

-agent_id: agent01

-# Config for secrets providers. We support reading secrets from Azure Key Vault and from the VM's local filesystem.

-# Multiple secret providers can be defined and each must be given a unique name, which is referenced later in the config.

-# A secret provider of type `key_vault` which contains details required to connect to the Azure Key Vault and allow connection to the Data Product's input storage account. This is always required.

-# A secret provider of type `file_system`, which specifies a directory on the VM where secrets are stored. For example for an SFTP pull source, for storing credentials for connecting to an SFTP server.

-secret_providers:

- - name: data_product_keyvault_mi

- key_vault:

- vault_name: contoso-dp-kv

- managed_identity:

- object_id: 22330f5b-4d7e-496d-bbdd-84749eeb009b

- - name: data_product_keyvault_sp

- key_vault:

- vault_name: contoso-dp-kv

- service_principal:

- tenant_id: ad5421f5-99e4-44a9-8a46-cc30f34e8dc7

- client_id: 98f3263d-218e-4adf-b939-eacce6a590d2

- cert_path: /path/to/local/certficate.p12

- - name: local_file_system

- # The file system provider specifies a folder in which secrets are stored.

- # Each secret must be an individual file without a file extension, where the secret name is the file name, and the file contains the secret only.

- file_system:

- # The absolute path to the secrets directory

- secrets_directory: /path/to/secrets/directory

-pipelines:

- # Pipeline IDs must be unique for a given agent instance. Any URL reserved characters must be percent-encoded.

- - id: mcc-edrs

- source:

- mcc_edrs:

- <mcc edrs source configuration>

- sink:

- <sink configuration>

- - id: contoso-logs

- source:

- sftp_pull:

- <sftp pull source configuration>

- sink:

- <sink configuration>

-```

-## Sink configuration

-All pipelines require sink config, which covers upload of files to the Data Product's input storage account.

-```yaml

-sink:

- # The container within the Data Product's input storage account. This *must* be exactly the name of the container that Azure Operator Insights expects. See the Data Product documentation for what value is required.

- container_name: example-container

- # Optional A string giving an optional base path to use in the container in the Data Product's input storage account. Reserved URL characters must be percent-encoded. See the Data Product for what value, if any, is required.

- base_path: base-path

- sas_token:

- # This must reference a secret provider configured above.

- secret_provider: data_product_keyvault_mi

- # The name of a secret in the corresponding provider.

- # This will be the name of a secret in the Key Vault.

- # This is created by the Data Product and should not be changed.

- secret_name: input-storage-sas

- # Optional. How often the sink should refresh its SAS token for the Data Product's input storage account. Defaults to 1h. Examples: 30s, 10m, 1h, 1d.

- cache_period: 1h

- # Optional. The maximum number of blobs that can be uploaded to the Data Product's input storage account in parallel. Further blobs will be queued in memory until an upload completes. Defaults to 10.

- # Note: This value is also the maximum number of concurrent SFTP reads for the SFTP pull source. Ensure your SFTP server can handle this many concurrent connections. If you set this to a value greater than 10 and are using an OpenSSH server, you may need to increase `MaxSessions` and/or `MaxStartups` in `sshd_config`.

- maximum_parallel_uploads: 10

- # Optional. The maximum size of each block that is uploaded to the Data Product's input storage account.

- # Each blob is composed of one or more blocks. Defaults to 32 MiB. Units are B, KiB, MiB, GiB, etc.

- block_size: 32 MiB

-```

-## Source configuration

-All pipelines require source config, which covers how the ingestion agent ingests files and where from. There are two supported source types: MCC EDRs and SFTP pull.

-Combining different types of source in one agent instance isn't recommended in production, but is supported for lab trials and testing.

-### MCC EDR source configuration

-```yaml

-source:

- mcc_edrs:

- # The maximum amount of data to buffer in memory before uploading. Units are B, KiB, MiB, GiB, etc.

- message_queue_capacity: 32 MiB

- # Quick check on the maximum RAM that the agent should use.

- # This is a guide to check the other tuning parameters, rather than a hard limit.

- maximum_overall_capacity: 1216 MiB

- listener:

- # The TCP port to listen on. Must match the port MCC is configured to send to. Defaults to 36001.

- port: 36001

- # EDRs greater than this size are dropped. Subsequent EDRs continue to be processed.

- # This condition likely indicates MCC sending larger than expected EDRs. MCC is not normally expected

- # to send EDRs larger than the default size. If EDRs are being dropped because of this limit,

- # investigate and confirm that the EDRs are valid, and then increase this value. Units are B, KiB, MiB, GiB, etc.

- soft_maximum_message_size: 20480 B

- # EDRs greater than this size are dropped and the connection from MCC is closed. This condition

- # likely indicates an MCC bug or MCC sending corrupt data. It prevents the agent from uploading

- # corrupt EDRs to Azure. You should not need to change this value. Units are B, KiB, MiB, GiB, etc.

- hard_maximum_message_size: 100000 B

- batching:

- # The maximum size of a single blob (file) to store in the Data Product's input storage account.

- maximum_blob_size: 128 MiB. Units are B, KiB, MiB, GiB, etc.

- # The maximum time to wait when no data is received before uploading pending batched data to the Data Product's input storage account. Examples: 30s, 10m, 1h, 1d.

- blob_rollover_period: 5m

-```

-### SFTP pull source configuration

-This configuration specifies which files are ingested from the SFTP server.

-Multiple SFTP pull sources can be defined for one agent instance, where they can reference either different SFTP servers, or different folders on the same SFTP server.

-```yaml

-source:

- sftp_pull:

- server: Information relating to the SFTP session.

- # The IP address or hostname of the SFTP server.

- host: 192.0.2.0

- # Optional. The port to connect to on the SFTP server. Defaults to 22.

- port: 22

- # The path on the VM to the 'known_hosts' file for the SFTP server. This file must be in SSH format and contain details of any public SSH keys used by the SFTP server. This is required by the agent to verify it is connecting to the correct SFTP server.

- known_hosts_file: /path/to/known_hosts

- # The name of the user on the SFTP server which the agent will use to connect.

- user: sftp-user

- # The form of authentication to the SFTP server. This can take the values 'password' or 'private_key'. The appropriate field(s) must be configured below depending on which type is specified.

- password:

- # The name of the secret provider configured above which contains the secret for the SFTP user.

- secret_provider: local_file_system

- # Only for use with password authentication. The name of the file containing the password in the secrets_directory folder

- secret_name: sftp-user-password

- # Only for use with private key authentication. The name of the file containing the SSH key in the secrets_directory folder

- key_secret_name: sftp-user-ssh-key

- # Optional. Only for use with private key authentication. The passphrase for the SSH key. This can be omitted if the key is not protected by a passphrase.

- passphrase_secret_name: sftp-user-ssh-key-passphrase

- filtering:

- # The path to a folder on the SFTP server that files will be uploaded to Azure Operator Insights from.

- base_path: /path/to/sftp/folder

- # Optional. A regular expression to specify which files in the base_path folder should be ingested. If not specified, the agent will attempt to ingest all files in the base_path folder (subject to exclude_pattern, settling_time and exclude_before_time).

- include_pattern: ".*\.csv$" # Only include files which end in ".csv"

- # Optional. A regular expression to specify any files in the base_path folder which should not be ingested. Takes priority over include_pattern, so files which match both regular expressions will not be ingested.

- # The exclude_pattern can also be used to ignore whole directories, but the pattern must still match all files under that directory. e.g. `^excluded-dir/.*$` or `^excluded-dir/` but *not* `^excluded-dir$`

- exclude_pattern: "^\.staging/|\.backup$" # Exclude all file paths that start with ".staging/" or end in ".backup"

- # A duration, such as "10s", "5m", "1h".. During an upload run, any files last modified within the settling time are not selected for upload, as they may still be being modified.

- settling_time: 1m

- # Optional. A datetime that adheres to the RFC 3339 format. Any files last modified before this datetime will be ignored.

- exclude_before_time: "2022-12-31T21:07:14-05:00"

- scheduling:

- # An expression in cron format, specifying when upload runs are scheduled for this source. All times refer to UTC. The cron schedule should include fields for: second, minute, hour, day of month, month, day of week, and year. E.g.:

- # `* /3 * * * * *` for once every 3 minutes

- # `0 30 5 * * * *` for 05:30 every day

- # `0 15 3 * * Fri,Sat *` for 03:15 every Friday and Saturday

- cron: "*/30 * * * Apr-Jul Fri,Sat,Sun 2025"

-```

-description: Understand how ingestion agents for Azure Operator Insights collect and upload data about your network to Azure.

-#CustomerIntent: As a someone deploying Azure Operator Insights, I want to understand how ingestion agents work so that I can set one up and configure it for my network.

-# Ingestion agent overview

-An _ingestion agent_ uploads data to an Azure Operator Insights Data Product. We provide an ingestion agent called the Azure Operator Insights ingestion agent that you can install on a Linux virtual machine to upload data from your network. This ingestion agent supports uploading:

-Combining different types of source in one agent instance isn't recommended in production, but is supported for lab trials and testing.

-## MCC EDR source overview

-An ingestion agent configured with an MCC EDR source is designed for use with an Affirmed Networks Mobile Content Cloud (MCC). It ingests Event Data Records (EDRs) from MCC network elements, and uploads them to Azure Operator Insights. To learn more, see [Quality of Experience - Affirmed MCC Data Product](concept-mcc-data-product.md).

-## SFTP pull source overview

-An ingestion agent configured with an SFTP pull source collects files from one or more SFTP servers, and uploads them to Azure Operator Insights.

-### File sources

-An ingestion agent collects files from _ingestion pipelines_ that you configure on it. A pipeline includes the details of the SFTP server, the files to collect from it and how to manage those files.

-For example, a single SFTP server might have logs, CSV files and text files. You could configure each type of file as a separate ingestion pipeline. For each ingestion pipeline, you can specify the directory to collect files from (optionally including or excluding specific files based on file paths), how often to collect files and other options. For full details of the available options, see [Configuration reference for Azure Operator Insights ingestion agent](ingestion-agent-configuration-reference.md).

-Ingestion pipelines have the following restrictions:

-### Processing files

-The ingestion agent uploads files to Azure Operator Insights during scheduled _upload runs_. The frequency of these runs is defined in the pipeline's configuration. Each upload run uploads files according to the pipeline's configuration:

-The ingestion agent records when it last completed an upload run for a file source. It uses this record to determine which files to upload during the next upload run, using the following process:

-1. The agent checks the last recorded time.

-1. The agent uploads any files modified since that time. It assumes that it processed older files during a previous upload run.

-1. At the end of the upload run:

- - If the agent uploaded all the files or the only errors were nonretryable errors, the agent updates the record. The new time is based on the time the upload run started, minus the settling time.

- - If the upload run had retryable errors (for example, if the connection to Azure was lost), the agent doesn't update the record. Not updating the record allows the agent to retry the upload for any files that didn't upload successfully. Retries don't duplicate any data previously uploaded.

-The ingestion agent is designed to be highly reliable and resilient to low levels of network disruption. If an unexpected error occurs, the agent restarts and provides service again as soon as it's running. After a restart, the agent carries out an immediate catch-up upload run for all configured file sources. It then returns to its configured schedule.

-## Authentication

-The ingestion agent authenticates to two separate systems, with separate credentials.

-For configuration instructions, see [Set up authentication to Azure](set-up-ingestion-agent.md#set-up-authentication-to-azure), [Prepare the VMs](set-up-ingestion-agent.md#prepare-the-vms) and [Configure the agent software](set-up-ingestion-agent.md#configure-the-agent-software).

-## Next step

-> [!div class="nextstepaction"]

-> [Install the Azure Operator Insights ingestion agent and configure it to upload data](set-up-ingestion-agent.md)

-description: Release notes for Azure Connected Machine agent versions older than six months.

-# Archive for What's new with Azure Operator Insights ingestion agent

-The primary [What's new in Azure Operator Insights ingestion agent?](ingestion-agent-release-notes.md) article contains updates for the last six months, while this article contains all the older information.

-The Azure Operator Insights ingestion agent receives improvements on an ongoing basis. This article provides you with information about:

-## Related content

-description: This article has release notes for Azure Operator Insights ingestion agent. For many of the summarized issues, there are links to more details.

-# What's new with Azure Operator Insights ingestion agent

-The Azure Operator Insights ingestion agent receives improvements on an ongoing basis. To stay up to date with the most recent developments, this article provides you with information about:

-This page is updated for each new release of the ingestion agent, so revisit it regularly. If you're looking for items older than six months, you can find them in [archive for What's new with Azure Operator Insights ingestion agent](ingestion-agent-release-notes-archive.md).

-## Version 2.0.0 - March 2024

-Supported distributions:

-This release has been produced in accordance with Microsoft's Secure Development Lifecycle, including processes for authorizing software changes, antimalware scanning, and scanning and mitigating security bugs and vulnerabilities.

-### Known issues

-None

-### New features

-### Fixed

-None

-## Version 1.0.0 - February 2024

-Supported distributions:

-This release has been produced in accordance with Microsoft's Secure Development Lifecycle, including processes for authorizing software changes, antimalware scanning, and scanning and mitigating security bugs and vulnerabilities.

-### Known issues

-None

-### New features

-This is the first release of the Azure Operator Insights ingestion agent. It supports ingestion of Affirmed MCC EDRs and of arbitrary files from an SFTP server.

-### Fixed

-None

-## Related content

-description: Set up Azure Data Factory to ingest data into an Azure Operator Insights Data Product.

-#CustomerIntent: As a admin in an operator network, I want to upload data to Azure Operator Insights so that my organization can use Azure Operator Insights.

-# Use Azure Data Factory to ingest data into an Azure Operator Insights Data Product

-This article covers how to set up [Azure Data Factory](/azure/data-factory/) to write data into an Azure Operator Insights Data Product.

-For more information on Azure Data Factory, see [What is Azure Data Factory](/azure/data-factory/introduction).

-> [!WARNING]

-> Data Products do not support private links. It is not possible to set up a private link between a Data Product and Azure Data Factory.

-## Prerequisites

- - To find the key vault, search for a resource group with a name starting with `<data-product-name>-HostedResources-`; the key vault is in this resource group.

-## Create a Key Vault linked service

-To connect Azure Data Factory to another Azure service, you must create a [linked service](/azure/data-factory/concepts-linked-services?tabs=data-factory). First, create a linked service to connect Azure Data Factory to the Data Product's key vault.

-1. In the [Azure portal](https://ms.portal.azure.com/#home), find the Azure Data Factory resource.

-1. From the **Overview** pane, launch the Azure Data Factory studio.

-1. Go to the **Manage** view, then find **Connections** and select **Linked Services**.

-1. Create a new linked service using the **New** button.

- 1. Select the **Azure Key Vault** type.

- 1. Set the target to the Data Product's key vault (the key vault is in the resource group with name starting with `<data-product-name>-HostedResources-` and is named `aoi-<uid>-kv`).

- 1. Set the authentication method to **System Assigned Managed Identity**.

-1. Grant Azure Data Factory permissions on the Key Vault resource.

- 1. Go to the Data Product's key vault in the Azure portal.

- 1. In the **Access Control (IAM)** pane, add a new role assignment.

- 1. Give the Data Factory managed identity (this has the same name as the Data Factory resource) the 'Key Vault Secrets User' role.

-## Create a Blob Storage linked service

-Data Products expose a Blob Storage endpoint for ingesting data. Use the newly created Key Vault linked service to connect Azure Data Factory to the Data Product ingestion endpoint.

-1. In the [Azure portal](https://ms.portal.azure.com/#home), find the Azure Data Factory resource.

-2. From the **Overview** pane, launch the Azure Data Factory studio.

-3. Go to the **Manage** view, then find **Connections** and select **Linked Services**.

-4. Create a new linked service using the **New** button.

- 1. Select the Azure Blob Storage type.

- 1. Set the authentication type to **SAS URI**.

- 1. Choose **Azure Key Vault** as the source.

- 1. Select the Key Vault linked service that you created in [Create a key vault linked service](#create-a-key-vault-linked-service).

- 1. Set the secret name to `input-storage-sas`.

- 1. Leave the secret version as the default value ('Latest version').

-Now the Data Factory is connected to the Data Product ingestion endpoint.

-## Create Blob Storage datasets

-To use the Data Product as the sink for a [Data Factory pipeline](/azure/data-factory/concepts-pipelines-activities?tabs=data-factory), you must create a sink [dataset](/azure/data-factory/concepts-datasets-linked-services?tabs=data-factory).

-1. In the [Azure portal](https://ms.portal.azure.com/#home), find the Azure Data Factory resource.

-2. From the **Overview** pane, launch the Azure Data Factory studio.

-3. Go to the **Author** view -> Add resource -> Dataset.

-4. Create a new Azure Blob Storage dataset.

- 1. Select your output type.

- 1. Set the linked service to the Data Product ingestion linked service that you created in [Create a blob storage linked service](#create-a-blob-storage-linked-service).

- 1. Set the container name to the name of the data type that the dataset is associated with.

- - This information can be found in the **Required ingestion configuration** section of the documentation for your Data Product.

- - For example, see [Required ingestion configuration](concept-monitoring-mcc-data-product.md#required-ingestion-configuration) for the Monitoring - MCC Data Product.

- 1. Ensure the folder path includes at least one directory; files copied into the root of the container won't be correctly ingested.

- 1. Set the other fields as appropriate for your data.

-5. Follow the Azure Data Factory documentation (for example [Creating a pipeline with the UI](/azure/data-factory/concepts-pipelines-activities?tabs=data-factory#creating-a-pipeline-with-ui)) to create a pipeline with this new dataset as the sink.

-Repeat this step for all required datasets.

-> [!IMPORTANT]

-> The Data Product may use the folder prefix or the file name prefix (this can be set as part of the pipeline, for example in the [Copy Activity](/azure/data-factory/connector-azure-blob-storage?tabs=data-factory#blob-storage-as-a-sink-type)) to determine how to process an ingested file. For your Data Product's requirements for folder prefixes or file name prefixes, see the **Required ingestion configuration** section of the Data Product's documentation. For example, see [Required ingestion configuration](concept-monitoring-mcc-data-product.md#required-ingestion-configuration) for the Monitoring - MCC Data Product.

-## Create Data Pipelines

-Your Azure Data Factory is now configured to connect to your Data Product. To ingest data using this configuration, you must follow the Data Factory documentation.

-1. [Set up a connection in Azure Data Factory](/azure/data-factory/connector-overview) to the service containing the source data.

-2. [Set up pipelines in Azure Data Factory](/azure/data-factory/concepts-pipelines-activities?tabs=data-factory#creating-a-pipeline-with-ui) to copy data from the source into your Data Product, using the datasets created in [the last step](#create-blob-storage-datasets).

-## Related content

-Learn how to:

-description: This article helps you understand managed identity and how it works in Azure Operator Insights.

-# Managed identity for Azure Operator Insights

-This article helps you understand managed identity (formerly known as Managed Service Identity/MSI) and how it works in Azure Operator Insights.

-## Overview of managed identities

-Managed identities eliminate the need to manage credentials. Managed identities provide an identity for service instances to use when connecting to resources that support Microsoft Entra ID (formerly Azure Active Directory) authentication. For example, the service can use a managed identity to access resources like [Azure Key Vault](/azure/key-vault/general/overview), where data admins can securely store credentials or access storage accounts. The service uses the managed identity to obtain Microsoft Entra ID tokens.

-Microsoft Entra ID offers two types of managed identities:

-For more general information about managed identities, see [What are managed identities for Azure resources?](/entra/identity/managed-identities-azure-resources/overview).

-## User-assigned managed identities in Azure Operator Insights

-Azure Operator Insights use a user-assigned managed identity for:

-When you [create a Data Product](data-product-create.md), you set up the managed identity and associate it with the Data Product. To use the managed identity with Microsoft Purview, you must also [grant the managed identity the appropriate permissions in Microsoft Purview](purview-setup.md#access-and-set-up-your-microsoft-purview-account).

-You use Microsoft Entra ID to manage user-assigned managed identities. For more information, see [Create, list, delete, or assign a role to a user-assigned managed identity using the Azure portal](/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities).

-## System-assigned managed identities in Azure Operator Insights

-Azure Operator Insights Data Products don't support system-assigned managed identities.

-Azure Operator Insights ingestion agents on Azure VMs support system-assigned managed identities for accessing a Data Product's Key Vault. See [Use a managed identity for authentication](set-up-ingestion-agent.md#use-a-managed-identity-for-authentication).

-## Related content

-See [Store credential in Azure Key Vault](../data-factory/store-credentials-in-key-vault.md) for information about when and how to use managed identity.

-See [What are managed identities for Azure resources?](/entra/identity/managed-identities-azure-resources/overview) for more background on managed identities for Azure resources, on which managed identity in Azure Operator Insights is based.

-description: Configure the Azure Operator Insights ingestion agent to use fixed IP addresses instead of hostnames.

-#CustomerIntent: As an admin in an operator network, I want to make the ingestion agent work without DNS, so that the ingestion agent can upload data to Azure Operator Insights.

-# Map Microsoft hostnames to IP addresses for ingestion agents that can't resolve public hostnames

-The Azure Operator Insights ingestion agent needs to resolve some Microsoft hostnames. If the VMs onto which you install the agent can't use DNS to resolve these hostnames, you need to add entries on each agent VM to map the hostnames to IP addresses.

-This process assumes that you're connecting to Azure over ExpressRoute and are using Private Links and/or Service Endpoints. If you're connecting over public IP addressing, you **cannot** use this workaround. Your VMs must be able to resolve public hostnames.

-## Prerequisites

-## Create service endpoints and private links

-1. Create the following resources from a virtual network that is peered to your ingestion agents.

- - A Service Endpoint to Azure Storage.

- - A Private Link or Service Endpoint to the Key Vault created by your Data Product. The Key Vault is the same one that you found in [Grant permissions for the Data Product Key Vault](set-up-ingestion-agent.md#grant-permissions-for-the-data-product-key-vault) when you started setting up the ingestion agent.

-1. Note the IP addresses of these two connections.

-## Find URLs for your Data Product

-1. Note the ingestion storage URL for your Data Product. You can find the ingestion storage URL on your Data Product overview page in the Azure portal, in the form `<account-name>.blob.core.windows.net`.

-1. Note the URL of the Data Product Key Vault. This Key Vault is in a resource group named `<data-product-name>-HostedResources-<unique-id>`. On the Key Vault overview page, you want the 'Vault URI' field, which appears as `<vault-name>.vault.azure.net`.

-## Look up a public IP address for login.microsoft.com

-Use a DNS lookup tool to find a public IP address for `login.microsoftonline.com`. For example:

- ```

- nslookup login.microsoftonline.com

- ```

- ```

- dig login.microsoftonline.com

- ```

-You can use any of the IP addresses.

-## Configure the ingestion agent to map between the IP addresses and the hostnames

-1. Add a line to */etc/hosts* on the VM linking the two values in the following format, for each of the storage and Key Vault.

- ```

- <Storage private IP>   <ingestion URL>

- <Key Vault private IP>  <Key Vault URL>

- ````

-1. Add the public IP address for `login.microsoftonline.com` to */etc/hosts*.

- ```

- <Public IP>   login.microsoftonline.com

- ````

-## Next step

-> [!div class="nextstepaction"]

-> [Continue setting up the ingestion agent](set-up-ingestion-agent.md#install-the-agent-software)

-description: Important reference material needed when you monitor Azure Operator Insights

-<!-- VERSION 2.3

-Template for monitoring data reference article for Azure services. This article is support for the main "Monitoring Azure Operator Insights" article for the service. -->

-# Monitoring Azure Operator Insights data reference

-This article describes the data you can collect in Azure Monitor for Azure Operator Insights. See [Monitoring Azure Operator Insights](monitor-operator-insights.md) for details on how to collect and analyze this monitoring data.

-## Metrics

-Azure Operator Insights doesn't provide metrics in Azure Monitor.

-## Resource logs

-This section lists the types of resource logs you can collect for Azure Operator Insights.

-|Resource Log Type | Resource Provider / Type Namespace<br/> and link to individual logs |

-|-|--|

-| DataProducts| [Microsoft.NetworkAnalytics/DataProducts](/azure/azure-monitor/reference/supported-logs/microsoft-networkanalytics-dataproducts-logs) |

-The DataProducts logs have the following categories:

-When you configure a diagnostic setting, you can select these categories individually, or select the Audit group. The Audit group contains all the categories except the Digestion category.

-For reference, see a list of [all resource logs category types supported in Azure Monitor](/azure/azure-monitor/platform/resource-logs-schema).

-## Azure Monitor Logs tables

-This section lists all of the Azure Monitor Logs Kusto tables relevant to Azure Operator Insights and available for query by Log Analytics.

-|Log type|Table name|Details|

-|--|-|-|

-|Transformation|[AOIDigestion](/azure/azure-monitor/reference/tables/aoidigestion)| Contains `Transformation` logs (called `Digestion` in the table)|

-|Ingestion and storage |[AOIStorage](/azure/azure-monitor/reference/tables/aoistorage)| Contains `Ingestion`, `IngestionDelete` and `ReadStorage` |

-|Database queries|[AOIDatabaseQuery](/azure/azure-monitor/reference/tables/aoidatabasequery)| Contains `DatabaseQuery` |

-### Diagnostics tables

-Azure Operator Insights uses the tables listed in [Azure Monitor Logs tables](#azure-monitor-logs-tables) to store resource log information. It doesn't use the Azure Diagnostics table.

-## Activity log

-The following table lists the operations that Azure Operator Insights can record in the Activity log. This table is a subset of the possible entries you might find in the activity log.

-| Namespace | Description |

-|:|:|

-|`Microsoft.NetworkAnalytics`|Logs relating to creating, modifying and deleting Data Product resources|

-|`Microsoft.OperationalInsights/workspaces/query/AOI*`|Logs relating to querying Azure Operator Insights data in Azure Monitor|

-See [all the possible resource provider operations in the activity log](/azure/role-based-access-control/resource-provider-operations).

-For more information on the schema of Activity Log entries, see [Activity Log schema](/azure/azure-monitor/essentials/activity-log-schema).

-## Schemas

-Azure Operator Insights uses the following schemas for logs.

-|Log type|Relates to|Schema link|

-|--|-|--|

-|Transformation logs|Processing (called digestion in the schema) of data|[AOIDigestion](/azure/azure-monitor/reference/tables/aoidigestion)|

-|Storage logs|Operations on the Data Product's storage|[AOIStorage](/azure/azure-monitor/reference/tables/aoistorage)|

-|Database query logs|Queries run on the Data Product's database|[AOIDatabaseQuery](/azure/azure-monitor/reference/tables/aoidatabasequery)|

-## See Also

-description: Start here to learn how to monitor Azure Operator Insights

-<!-- VERSION 2.3 2022_05_17

-Template for the main monitoring article for Azure services. -->

-# Monitoring Azure Operator Insights

-When you have critical applications and business processes relying on Azure resources, you want to monitor those resources for their availability, performance, and operation.

-Azure Operator Insights Data Products use [Azure Monitor](/azure/azure-monitor/overview). They collect the same kinds of monitoring data as other Azure resources that are described in [Monitoring data from Azure resources](/azure/azure-monitor/essentials/monitor-azure-resource#monitoring-data-from-Azure-resources). See [Monitoring Azure Operator Insights data reference](monitor-operator-insights-data-reference.md) for detailed information on the monitoring data created by Data Products.

-> [!TIP]

-> If you're unfamiliar with the features of Azure Monitor common to all Azure services that use it, read [Monitoring Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource).

-Ingestion agents also collect monitoring data that you or Microsoft Support can use for troubleshooting.

-## Metrics for Data Products: Overview, collection and analysis

-Azure Operator Insights doesn't provide metrics in Azure Monitor.

-## Activity logs for Data Products: Overview, collection and analysis

-The [Activity log](/azure/azure-monitor/essentials/activity-log) is a type of platform log in Azure that provides insight into subscription-level events. For Azure Operator Insights, the Activity log includes activities like creating a Data Product or changing its settings.

-The Activity log is collected and stored automatically by Azure. You can:

-## Resource logs for Data Products: Overview, collection and analysis

-Resource logs provide an insight into operations that were performed within an Azure resource. This is known as the *data plane*. For Data Products, resource logs include ingestion (activity on files uploaded to Azure Operator Insights), transformation (processing the data in those files), and management of the processed data.

-Resource logs aren't collected and stored until you create a *diagnostic setting* that routes them to one or more locations. We recommend routing them to a Log Analytics workspace, which stores the logs in [Azure Monitor Logs](../azure-monitor/logs/data-platform-logs.md). Log Analytics allows you to analyze the logs of all your Azure resources together in Azure Monitor Logs and take advantage of all the features available to Azure Monitor Logs including [log queries](../azure-monitor/logs/log-query-overview.md) and [log alerts](../azure-monitor/alerts/alerts-log.md).

-For instructions on using getting started with Log Analytics and creating a diagnostic setting, see [Get started with resource logs for Data Products](#get-started-with-resource-logs-for-data-products). For more information about the data available, see [Data Product information in Azure Monitor Logs](#data-product-information-in-azure-monitor-logs).

-### Get started with resource logs for Data Products

-To start monitoring a Data Product with Azure Monitor Logs and Log Analytics:

-1. Create a Log Analytics workspace by following [Create a Log Analytics workspace](../azure-monitor/logs/quick-create-workspace.md).

-1. In the **Diagnostic setting** view of your Data Product, create a diagnostic setting that routes the logs that you want to collect to the Log Analytics workspace. To use the example query in this procedure, include **Database Query** (in addition to any other category of logs that you want to collect).

- - For instructions, see [Create diagnostic setting to collect platform logs and metrics in Azure](/azure/azure-monitor/platform/diagnostic-settings). You can use the Azure portal, CLI, or PowerShell.

- - The categories of logs for Azure Operator Insights are listed in [Azure Operator Insights monitoring data reference](monitor-operator-insights-data-reference.md#resource-logs).

-1. To use the example query in this procedure, run a query on the data in your Data Product by following [Query data in the Data Product](data-query.md). This step ensures that Azure Monitor Logs has some data for your Data Product.

-1. Return to your Data Product resource and select **Logs** from the Azure Operator Insights menu to access Log Analytics.

-1. Run the following query to view the log for the query that you ran on your Data Product, replacing _username@example.com_ with the email address you used when you ran the query. You can also adapt the sample queries in [Sample Kusto queries](#sample-kusto-queries).

- ```kusto

- AOIDatabaseQuery

- | where User has_cs "username@example.com"

- | take 100

- ```

-> [!IMPORTANT]

-> When you select **Logs** from the Azure Operator Insights menu, Log Analytics is opened with the query scope set to the current Data Product. This means that log queries will only include data from that resource. If you want to run a query that includes data from other Data Products or data from other Azure services, select **Logs** from the **Azure Monitor** menu. See [Log query scope and time range in Azure Monitor Log Analytics](/azure/azure-monitor/logs/scope) for details.

-### Data Product information in Azure Monitor Logs

-For a full list of the types of resource logs collected for Azure Operator Insights, see [Monitoring Azure Operator Insights data reference: Resource logs](monitor-operator-insights-data-reference.md#resource-logs).

-Data in Azure Monitor Logs is stored in tables where each table has its own set of unique properties. For a list of the Azure Operator Insights tables used by Azure Monitor Logs and queryable by Log Analytics, see [Monitoring Azure Operator Insights data reference: Azure Monitor Logs tables](monitor-operator-insights-data-reference.md#azure-monitor-logs-tables).

-All resource logs in Azure Monitor have the same fields followed by service-specific fields. The common schema is outlined in [Azure Monitor resource log schema](/azure/azure-monitor/essentials/resource-logs-schema) The schemas for Azure Operator Insights resource logs are found in the [Azure Operator Insights Data Reference: Schemas](monitor-operator-insights-data-reference.md#schemas).

-### Sample Kusto queries

-You can use the following example queries in a Log Analytics workspace to help you monitor your Data Products:

- ```kusto

- AOIDigestion

- | where Message startswith_cs "Failed to decode row"

- | take 100

- ```

- ```kusto

- AOIDigestion

- | where Message startswith_cs "Failed to digest file"

- | parse FilePath with Source:string "/" *

- | summarize count() by Source

- ```

- ```kusto

- AOIDatabaseQuery

- | where DatabaseName has_cs "edrdp" and User has_cs "username@example.com"

- | take 100

- ```

- ```kusto

- AOIStorage

- | where Category has_cs "Ingestion"

- | take 100

- ```

- ```kusto

- AOIStorage

- | where Category has_cs "IngestionDelete"

- | take 100

- ```

- ```kusto

- AOIStorage

- | where Category has_cs "ReadStorage"

- | take 100

- ```

-For a list of common queries for Azure Operator Insights, see the [Log Analytics queries interface](/azure/azure-monitor/logs/queries).

-## Monitoring for ingestion agents

-Azure Operator Insights also requires ingestion agents deployed in your network.

-Ingestion agents that we provide automatically collect metrics and logs for troubleshooting. Metrics and logs are stored on the VM on which you installed the agent, and aren't uploaded to Azure Monitor. For details, see [Monitor and troubleshoot ingestion agents for Azure Operator Insights](monitor-troubleshoot-ingestion-agent.md).

-## Next steps

-description: Learn how to detect, troubleshoot, and fix problems with Azure Operator Insights ingestion agents.

-#CustomerIntent: As a someone managing an agent that has already been set up, I want to monitor and troubleshoot it so that Data Products in Azure Operator Insights receive the correct data.

-# Monitor and troubleshoot Azure Operator Insights ingestion agents

-For an overview of ingestion agents, see [Ingestion agent overview](ingestion-agent-overview.md).

-If you notice problems with data collection from your ingestion agents, use the information in this section to fix common problems or create a diagnostics package. You can upload the diagnostics package to support tickets that you create in the Azure portal.

-The ingestion agent is a software package, so the diagnostics are limited to the functioning of the application. We don't provide OS or resource monitoring. You're encouraged to use standard tooling such as snmpd, Prometheus node exporter, or other tools to send OS-level data, logs, and metrics to your own monitoring systems. [Monitor virtual machines with Azure Monitor](../azure-monitor/vm/monitor-virtual-machine.md) describes tools you can use if your ingestion agents are running on Azure VMs.

-The agent writes logs and metrics to files under */var/log/az-aoi-ingestion/*. If the agent is failing to start for any reason, such as misconfiguration, the *stdout.log* file contains human-readable logs explaining the issue.

-Metrics are reported in a simple human-friendly form.

-## Prerequisites

-## Ingestion agent diagnostics

-To collect a diagnostics package, SSH to the Virtual Machine and run the command `/usr/bin/microsoft/az-aoi-ingestion-gather-diags`. This command generates a date-stamped zip file in the current directory that you can copy from the system.

-If you have configured collection of logs through the Azure Monitor agent, you can view ingestion agent logs in the portal view of your Log Analytics workspace, and may not need to collect a diagnostics package to debug your issues.

-> [!NOTE]

-> Microsoft Support might request diagnostics packages when investigating an issue. Diagnostics packages don't contain any customer data or the value of any credentials.

-## Problems common to all sources

-Problems broadly fall into four categories.

-### Agent fails to start

-Symptoms: `sudo systemctl status az-aoi-ingestion` shows that the service is in failed state.

- ```

- sudo systemctl start az-aoi-ingestion

- ```

-

-### No data appearing in Azure Operator Insights

-Symptoms: no data appears in Azure Data Explorer.

-## Problems with the MCC EDR source

-This section covers problems specific to the MCC EDR source.

-You can also use the diagnostics provided by the MCCs, or by Azure Operator Insights itself in Azure Monitor, to help identify and debug ingestion issues.

-### MCC can't connect

-Symptoms: MCC reports alarms about MSFs being unavailable.

-### No EDRs appearing in Azure Operator Insights

-Symptoms: no data appears in Azure Data Explorer.

-### Data missing or incomplete

-Symptoms: Azure Monitor shows a lower incoming EDR rate in ADX than expected.

-

-## Problems with the SFTP pull source

-This section covers problems specific to the SFTP pull source.

-You can also use the diagnostics provided by Azure Operator Insights itself in Azure Monitor to help identify and debug ingestion issues.

-### Agent can't connect to SFTP server

-Symptoms: No files are uploaded to Azure Operator Insights. The agent log file, */var/log/az-aoi-ingestion/stdout.log*, contains errors about connecting the SFTP server.

- - On the agent VM, run `ssh-keygen -l -F *<sftp-server-IP-or-hostname>*`.

- - If there's no output, then `known_hosts` doesn't contain a matching entry. Follow the instructions in [Setup the Azure Operator Insights ingestion agent](set-up-ingestion-agent.md) to add a `known_hosts` entry for the SFTP server.

-### No files are uploaded to Azure Operator Insights

-Symptoms: No data appears in Azure Data Explorer. Logs of category `Ingestion` don't appear in [Azure Operator Insights monitoring data](monitor-operator-insights-data-reference.md#resource-logs) or they contain errors. The [Number of ingested rows](concept-data-quality-monitoring.md#metrics) data quality metric for the relevant data type is zero.

-### Files are missing

-Symptoms: Data is missing from Azure Data Explorer. Logs of category `Ingestion` in [Azure Operator Insights monitoring data](monitor-operator-insights-data-reference.md#resource-logs) are lower than expected or they contain errors. The [Number of ingested rows](concept-data-quality-monitoring.md#metrics) data quality metric for the relevant data type is lower than expected.

- - The files exist on the SFTP server under the path defined in `base_path`. Ensure that there are no symbolic links in the file paths of the files to upload: the ingestion agent ignores symbolic links.

- - The "last modified" time of the files is at least `settling_time` seconds earlier than the time of the most recent upload run for this file source.

- - The "last modified" time of the files is later than `exclude_before_time` (if specified).

- - The file path relative to `base_path` matches the regular expression given by `include_pattern` (if specified).

- - The file path relative to `base_path` *doesn't* match the regular expression given by `exclude_pattern` (if specified).

-### Files are uploaded more than once

-Symptoms: Duplicate data appears in Azure Operator Insights.

-## Related content

-Learn how to:

-description: Azure Operator Insights is an Azure service for monitoring and analyzing data from multiple sources.

-# What is Azure Operator Insights?

-Azure Operator Insights is a fully managed service that enables the collection and analysis of massive quantities of network data gathered from complex multi-part or multi-vendor network functions. It delivers statistical, machine learning, and AI-based insights for operator-specific workloads to help operators understand the health of their networks and the quality of their subscribers' experiences in near real-time.

-Azure Operator Insights accelerates time to business value by eliminating the pain and time-consuming task of assembling off-the-shelf cloud components (chemistry set). This reduces load on ultra-lean operator platform and data engineering teams by making the following turnkey:

-The result is that the operator has a lower total cost of ownership but higher insights of their network over equivalent on-premises or cloud chemistry set platforms.

-## How does Azure Operator Insights work?

-Azure Operator Insights requires two separate types of resources.

- - You can use prebuilt dashboards provided by the Data Product or build your own in Azure Data Explorer. Azure Data Explorer also allows you to query your data directly, analyze it in Power BI or use it with Logic Apps. For more information, see [Data visualization in Data Products](concept-data-visualization.md).

- - Data Products provide [metrics for monitoring the quality of your data](concept-data-quality-monitoring.md).

- - Data Products are designed for specific types of source data and provide specialized processing for that source data. For more information, see [Data types](concept-data-types.md).

- Diagram of the Azure Operator Insights architecture. It shows ingestion by ingestion agents from on-premises data sources, processing in a Data Product, and analysis and use in Logic Apps and Power BI.

-For more information about the architecture of Azure Operator Insights, see [Architecture of Azure Operator Insights](architecture.md).

-We provide the following Data Products.

-|Data Product |Purpose |Supporting ingestion agent|

-||||

-|[Quality of Experience - Affirmed MCC Data Product](concept-mcc-data-product.md) | Analysis and insight from EDRs provided by Affirmed Networks Mobile Content Cloud (MCC) network elements| [Azure Operator Insights ingestion agent](ingestion-agent-overview.md) configured to use EDRs as a source|

-| [Monitoring - Affirmed MCC Data Product](concept-monitoring-mcc-data-product.md) | Analysis and insight from performance management data (performance statistics) from Affirmed Networks MCC network elements| [Azure Operator Insights ingestion agent](ingestion-agent-overview.md) configured to use SFTP as a source |

-If you prefer, you can provide your own ingestion agent to upload data to your chosen Data Product.

-Azure Operator Insights also offers the data product factory (preview) to allow partners and operators to build new Data Products. For more information, see [the overview of the Azure Operator Insights data product factory](data-product-factory.md).

-## How can I use Azure Operator Insights for end-to-end insights?

-Azure Operator Insights provides built-in support for discovering and joining Data Products together in a data mesh to achieve higher-value end-to-end insights for multi-site multi-vendor networks. Individual Data Products provide specialized data processing, enrichment, and visualizations, while using the Azure Operator Insights platform to manage operator-scale data. All Data Products share a standardized and composable architecture, and support consistent processes for operating and designing Data Products.

-## How do I get access to Azure Operator Insights?

-Access is currently limited by request. More information is included in the application form. We appreciate your patience as we work to enable broader access to Azure Operator Insights Data Product. Apply for access by [filling out this form](https://aka.ms/AAn1mi6).

-## Related content

-description: In this article, learn how to set up Microsoft Purview to explore an Azure Operator Insights Data Product.

-# Use Microsoft Purview with an Azure Operator Insights Data Product

-This article outlines how to set up Microsoft Purview to explore an Azure Operator Insights Data Product.

-Data governance is about managing data as a strategic asset, ensuring that there are controls in place around data, its content, structure, use, and safety. Microsoft Purview (formerly Azure Purview) is responsible for implementing data governance and allows you to monitor, organize, govern, and manage your entire data estate.

-When it comes to Azure Operator Insights, Microsoft Purview provides simple overviews and catalogs of all Data Product resources. To integrate Microsoft Purview into your Data Product solution, provide your Microsoft Purview account and chosen collection when creating an Azure Operator Insights Data Product in the Azure portal.

-The Microsoft Purview account and collection is populated with catalog details of your Data Product during the resource creation or resource upgrade process.

-## Prerequisites

-## Access and set up your Microsoft Purview account

-You can access your Purview account through the Azure portal by going to `https://web.purview.azure.com` and selecting your Microsoft Entra ID and account name. Or by going to `https://web.purview.azure.com/resource/<yourpurviewaccountname>`.

-To begin to catalog a Data Product in this account, [create a collection](../purview/how-to-create-and-manage-collections.md) to hold the Data Product.

-Provide the user-assigned managed identity (UAMI) for your Azure Operator Insights Data Product with necessary roles in the Microsoft Purview compliance portal. This UAMI was set up when the Data Product was created. For information on how to set up this UAMI, see [Set up a user-assigned managed identity](data-product-create.md#set-up-a-user-assigned-managed-identity). At the desired collection, assign this UAMI to the **Collection admin**, **Data source admin**, and **Data curator** roles. Alternately, you can apply the UAMI at the root collection/account level. All collections would inherit these role assignments by default.

-Assign roles to your users using effective role-based access control (RBAC). There are multiple roles that can be assigned, and assignments can be done on an account root and collection level. For more information, see how to [add roles and restrict access through collections](../purview/how-to-create-and-manage-collections.md#add-roles-and-restrict-access-through-collections).

-[Using the Microsoft Purview compliance portal](../purview/use-microsoft-purview-governance-portal.md) explains how to use the user interface and navigate the service. Microsoft Purview includes options to scan in data sources, but this option isn't required for integrating Azure Operator Insights Data Products with Microsoft Purview. When you complete this procedure, all Azure services and assets are automatically populated to your Purview catalog.

-## Connect Microsoft Purview to your Data Product

-When creating an Azure Operator Insights Data Product, select the **Advanced** tab and enable Purview.

-Select **Select Purview Account** to provide the required values to populate a Purview collection with Data Product details.

-## Understand Data Product representation in Microsoft Purview

-A Data Product is made up of many Azure Services and Data Assets, which are represented as assets of multiple types inside the Microsoft Purview compliance portals. The following asset types are represented.

-### Data Product

-An overall representation of the Azure Operator Insights Data Product.

-| **Additional fields** | **Description** |

-|--|--|

-| Description | Brief description of the Data Product |

-| Owners | A list of owners of this Data Product |

-| Azure Region | The region where the Data Product is deployed |

-| Docs | A link to documents that explain the data |

-### AOI Data Lake

-Also known as Azure Data Lake Storage.

-| **Additional fields** | **Description** |

-|--|-|

-| DFS Endpoint Address | Provides access to Parquet files in Azure Operator Insights Data Lake |

-### AOI Database

-Also known as Azure Data Explorer.

-| **Additional fields** | **Description** |

-|--|-|

-| KQL Endpoint Address | Provides access to Azure Operator Insights tables for exploration using KQL |

-### AOI Table

-Azure Data Explorer tables and materialized views.

-| **Additional fields** | **Description** |

-|--|-|

-| Description | Brief description of each table and view |

-| Schema | Contains the table columns and their details |

-### AOI Parquet Details

-Each Azure Data Explorer table is an equivalent Parquet file type.

-| **Additional fields** | **Description** |

-|--|-|

-| Path | Top-level path for the Parquet file type: container/dataset\_name |

-| Description | Identical to the equivalent AOI Table |

-| Schema | Identical to the equivalent AOI Table |

-### AOI Column

-The columns belong to Azure Operator Insights tables and the equivalent AOI Parquet Details.

-| **Additional fields** | **Description** |

-|--||

-| Type | The data type of this column |

-| Description | Brief description for this column |

-| Schema | Identical to the equivalent AOI Table |

-There are relationships between assets where necessary. For example, a Data Product can have many AOI Databases and one AOI Data Lake related to it.

-## Explore your Data Product with Microsoft Purview

-When the Data Product creation process is complete, you can see the catalog details of your Data Product in the collection. Select **Data map > Collections** from the left pane and select your collection.

-> [!NOTE]

-> The Microsoft Purview integration with Azure Operator Insights Data Products only features the Data catalog and Data map of the Microsoft Purview compliance portal.

-Select **Assets** to view the Data Product catalog and to list all assets of your Data Product.

-Select **Assets** to view the asset catalog of your Data Product. You can filter by the data source type for the asset type. For each asset, you can display properties, a list of owners (if applicable), and the related assets.

-When viewing all assets, filtering by data source type is helpful.

-### Asset properties and endpoints

-When looking at individual assets, select the **Properties** tab to display properties and related assets for that asset.

-You can use the Properties tab to find endpoints in AOI Database and AOI Tables.

-### Related assets

-Select the **Related** tab of an asset to display a visual representation of the existing relationships, summarized and grouped by the asset types.

-Select an asset type (such as aoi\_database as shown in the example) to view a list of related assets.

-### Exploring schemas

-The AOI Table and AOI Parquet Details have schemas. Select the **Schema** tab to display the details of each column.

-## Related content

-[Use the Microsoft Purview compliance portal](../purview/use-microsoft-purview-governance-portal.md)

-description: Learn how to rotate secrets for Azure Operator Insights ingestion agents.

-#CustomerIntent: As a someone managing an agent that has already been set up, I want to rotate its secrets so that Data Products in Azure Operator Insights continue to receive the correct data.

-# Rotate secrets for Azure Operator Insights ingestion agents

-The ingestion agent is a software package that is installed onto a Linux Virtual Machine (VM) owned and managed by you.

-It uses a managed identity or service principal to obtain, from the Data Product's Azure Key Vault, the credentials needed to upload data to the Data Product's input storage account.

-If you use a service principal, you must refresh its credentials before they expire. In this article, you'll rotate the service principal certificates on the ingestion agent.

-## Prerequisites

-None.

-## Rotate certificates

-1. Create a new certificate, and add it to the service principal. For instructions, refer to [Upload a trusted certificate issued by a certificate authority](/entra/identity-platform/howto-create-service-principal-portal).

-1. Obtain the new certificate and private key in the base64-encoded P12 format, as described in [Set up Ingestion Agents for Azure Operator Insights](set-up-ingestion-agent.md#prepare-certificates-for-the-service-principal).

-1. Copy the certificate to the ingestion agent VM.

-1. Save the existing certificate file and replace with the new certificate file.

-1. Restart the agent.

- ```

- sudo systemctl restart az-aoi-ingestion.service

- ```

-## Related content

-Learn how to:

-description: Set up the ingestion agent for Azure Operator Insights by installing it and configuring it to upload data to Data Products.

-#CustomerIntent: As a admin in an operator network, I want to upload data to Azure Operator Insights so that my organization can use Azure Operator Insights.

-# Install the Azure Operator Insights ingestion agent and configure it to upload data

-When you follow this article, you set up an Azure Operator Insights _ingestion agent_ on a virtual machine (VM) in your network and configure it to upload data to a Data Product. This ingestion agent supports uploading:

-For an overview of ingestion agents, see [Ingestion agent overview](ingestion-agent-overview.md).

-## Prerequisites

-From the documentation for your Data Product, obtain the:

-## VM security recommendations

-The VM used for the ingestion agent should be set up following best practice for security. We recommend the following actions:

-### Networking

-When using an Azure VM:

-

-When using an on-premises VM:

-### Disk encryption

-Ensure Azure disk encryption is enabled (this is the default when you create the VM).

-### OS version

-### Access

-Limit access to the VM to a minimal set of users. Configure audit logging on the VM - for example, using the Linux audit package - to record sign-in attempts and actions taken by logged-in users.

-We recommend that you restrict the following types of access.

-### Microsoft Defender for Cloud

-When using an Azure VM, also follow all recommendations from Microsoft Defender for Cloud. You can find these recommendations in the portal by navigating to the VM, then selecting Security.

-## Set up authentication to Azure

-The ingestion agent must be able to authenticate with the Azure Key Vault created by the Data Product to retrieve storage credentials. The method of authentication can either be:

-> [!IMPORTANT]

-> You may need a Microsoft Entra tenant administrator in your organization to perform this setup for you.

-### Use a managed identity for authentication

-If the ingestion agent is running in Azure, we recommend managed identities. For more detailed information, see the [overview of managed identities](managed-identity.md#overview-of-managed-identities).

-> [!NOTE]

-> Ingestion agents on Azure VMs support both system-assigned and user-assigned managed identities. For multiple agents, a user-assigned managed identity is simpler because you can authorise the identity to the Data Product Key Vault for all VMs running the agent.

-1. Create or obtain a user-assigned managed identity, by following the instructions in [Manage user-assigned managed identities](/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities). If you plan to use a system-assigned managed identity, don't create a user-assigned managed identity.

-1. Follow the instructions in [Configure managed identities for Azure resources on a VM using the Azure portal](/entra/identity/managed-identities-azure-resources/qs-configure-portal-windows-vm) according to the type of managed identity being used.

-1. Note the Object ID of the managed identity. The Object ID is a UUID of the form xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, where each character is a hexadecimal digit.

-You can now [grant permissions for the Data Product Key Vault](#grant-permissions-for-the-data-product-key-vault).

-### Use a service principal for authentication

-If the ingestion agent is running outside of Azure, such as an on-premises network then you **cannot use managed identities** and must instead authenticate to the Data Product Key Vault using a service principal with a certificate credential. Each agent must also have a copy of the certificate stored on the virtual machine.

-#### Create a service principal

-1. Create or obtain a Microsoft Entra ID service principal. Follow the instructions detailed in [Create a Microsoft Entra app and service principal in the portal](/entra/identity-platform/howto-create-service-principal-portal). Leave the **Redirect URI** field empty.

-1. Note the Application (client) ID, and your Microsoft Entra Directory (tenant) ID (these IDs are UUIDs of the form xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, where each character is a hexadecimal digit).

-#### Prepare certificates for the service principal

-The ingestion agent only supports certificate credentials for service principals. It's up to you whether you use the same certificate and key for each VM, or use a unique certificate and key for each. Using a certificate per VM provides better security and has a smaller impact if a key is leaked or the certificate expires. However, this method adds a higher maintainability and operational complexity.

-1. Obtain one or more certificates. We strongly recommend using trusted certificates from a certificate authority. Certificates can be generated from Azure Key Vault: see [Set and retrieve a certificate from Key Vault using Azure portal](/azure/key-vault/certificates/quick-create-portal). Doing so allows you to configure expiry alerting and gives you time to regenerate new certificates and apply them to your ingestion agents before they expire. Once a certificate expires, the agent is unable to authenticate to Azure and no longer uploads data. For details of this approach see [Renew your Azure Key Vault certificates](/azure/key-vault/certificates/overview-renew-certificate). If you choose to use Azure Key Vault then:

- - This Azure Key Vault must be a different instance to the Data Product Key Vault, either one you already control, or a new one.

- - You need the 'Key Vault Certificates Officer' role on this Azure Key Vault in order to add the certificate to the Key Vault. See [Assign Azure roles using the Azure portal](../role-based-access-control/role-assignments-portal.yml) for details of how to assign roles in Azure.

-2. Add the certificate or certificates as credentials to your service principal, following [Create a Microsoft Entra app and service principal in the portal](/entra/identity-platform/howto-create-service-principal-portal).

-3. Ensure the certificates are available in PKCS#12 (P12) format, with no passphrase protecting them.

- - If the certificate is stored in an Azure Key Vault, download the certificate in the PFX format. PFX is identical to P12.

- - On Linux, you can convert a certificate and private key using OpenSSL. When prompted for an export password, press <kbd>Enter</kbd> to supply an empty passphrase. This can then be stored in an Azure Key Vault as outlined in step 1.

- ```

- openssl pkcs12 -nodes -export -in <certificate.pem> -inkey <key.pem> -out <certificate.p12>

- ```

-> [!IMPORTANT]

-> The P12 file must not be protected with a passphrase.

-4. Validate your P12 file. This displays information about the P12 file including the certificate and private key.

- ```

- openssl pkcs12 -nodes -in <certificate.p12> -info

- ```

-5. Ensure the P12 file is base64 encoded. On Linux, you can base64 encode a P12 certificate by using the `base64` command.

- ```

- base64 -w 0 <certificate.p12> > <base64-encoded-certificate.p12>

- ```

-### Grant permissions for the Data Product Key Vault

-1. Find the Azure Key Vault that holds the storage credentials for the input storage account. This Key Vault is in a resource group named *`<data-product-name>-HostedResources-<unique-id>`*.

-1. Grant your service principal the 'Key Vault Secrets User' role on this Key Vault. You need Owner level permissions on your Azure subscription. See [Assign Azure roles using the Azure portal](../role-based-access-control/role-assignments-portal.yml) for details of how to assign roles in Azure.

-1. Note the name of the Key Vault.

-## Prepare the SFTP server

-This section is only required for the SFTP pull source.

-On the SFTP server:

-1. Ensure port 22/TCP to the VM is open.

-1. Create a new user, or determine an existing user on the SFTP server that the ingestion agent should use to connect to the SFTP server.

- - By default the ingestion agent searches every directory under the base path, so this user must be able to read all of them. Any directories that the user does not have permission to access must be excluded using the `exclude_pattern` configuration.

- > [!Note]

- > Implicitly excluding directories by not specifying them in the included pattern is not sufficient to stop the agent searching those directories. See [the configuration reference](ingestion-agent-configuration-reference.md) for more detail on excluding directories.

-1. Determine the authentication method that the ingestion agent should use to connect to the SFTP server. The agent supports:

- - Password authentication

- - SSH key authentication

-1. Configure the SFTP server to remove files after a period of time (a _retention period_). Ensure the retention period is long enough that the agent should process the files before the SFTP server deletes them. The example configuration file contains configuration for checking for new files every five minutes.

-> [!IMPORTANT]

-> Your SFTP server must remove files after a suitable retention period so that it does not run out of disk space. The ingestion agent does not remove files automatically.

->

-> A shorter retention time reduces disk usage, increases the speed of the agent and reduces the risk of duplicate uploads. However, a shorter retention period increases the risk that data is lost if data cannot be retrieved by the agent or uploaded to Azure Operator Insights.

-## Prepare the VMs

-Repeat these steps for each VM onto which you want to install the agent.

-1. Ensure you have an SSH session open to the VM, and that you have `sudo` permissions.

-1. Install systemd, logrotate, and zip on the VM, if not already present. For example:

- ```

- sudo dnf install systemd logrotate zip

- ```

-1. If you're using a service principal, copy the base64-encoded P12 certificate (created in the [Prepare certificates](#prepare-certificates-for-the-service-principal) step) to the VM, in a location accessible to the ingestion agent.

-1. Configure the agent VM based on the type of ingestion source.

- # [SFTP sources](#tab/sftp)

- 1. Verify that the VM has the following ports open. These ports must be open both in cloud network security groups and in any firewall running on the VM itself (such as firewalld or iptables).

- - Port 443/TCP outbound to Azure

- - Port 22/TCP outbound to the SFTP server

- 1. Create a directory to use for storing secrets for the agent. We call this directory the _secrets directory_. Note its path.

- 1. Create a file in the secrets directory containing password or private SSH key for the SFTP server.

- - The file must not have a file extension.

- - Choose an appropriate name for this file, and note it for later. This name is referenced in the agent configuration.

- - The file must contain only the secret value (password or SSH key), with no extra whitespace.

- 1. If you're using an SSH key that has a passphrase to authenticate, use the same method to create a separate file that contains the passphrase.

- 1. Ensure the SFTP server's public SSH key is listed on the VM's global known_hosts file located at */etc/ssh/ssh_known_hosts*.

- > [!TIP]

- > Use the Linux command `ssh-keyscan` to add a server's SSH public key to a VM's *known_hosts* file manually. For example, `ssh-keyscan -H <server-ip> | sudo tee -a /etc/ssh/ssh_known_hosts`.

- # [MCC EDR sources](#tab/edr)

- Verify that the VM has the following ports open. These ports must be open both in cloud network security groups and in any firewall running on the VM itself (such as firewalld or iptables).

- - Port 36001/TCP inbound from the MCCs

- - Port 443/TCP outbound to Azure

- You can configure the inbound rule with:

- ```

- sudo firewall-cmd --permanent --new-service=mcc-connection

- sudo firewall-cmd --permanent --service=mcc-connection --add-port=36001/tcp

- sudo firewall-cmd --add-service=mcc-connection --permanent

- sudo firewall-cmd --reload

- ```

-

-## Ensure that the VM can resolve Microsoft hostnames

-Check that the VM can resolve public hostnames to IP addresses. For example, open an SSH session and use `dig login.microsoftonline.com` to check that the VM can resolve `login.microsoftonline.com` to an IP address.

-If the VM can't use DNS to resolve public Microsoft hostnames to IP addresses, [map the required hostnames to IP addresses](map-hostnames-ip-addresses.md). Return to this procedure when you have finished the configuration.

-## Install the agent software

-The agent software package is hosted on the "Linux software repository for Microsoft products" at [https://packages.microsoft.com](https://packages.microsoft.com)

-**The name of the ingestion agent package is `az-aoi-ingestion`.**

-To download and install a package from the software repository, follow the relevant steps for your VM's Linux distribution in [

-How to install Microsoft software packages using the Linux Repository](/linux/packages#how-to-install-microsoft-software-packages-using-the-linux-repository).

- For example, if you're installing on a VM running Red Hat Enterprise Linux (RHEL) 8, follow the instructions under the [Red Hat-based Linux distributions](/linux/packages#red-hat-based-linux-distributions) heading, substituting the following parameters:

-## Configure the agent software

-The configuration you need is specific to the type of source and your Data Product. Ensure you have access to your Data Product's documentation to see the required values. For example:

-1. Connect to the VM over SSH.

-1. Change to the configuration directory.

- ```

- cd /etc/az-aoi-ingestion

- ```

-1. Make a copy of the default configuration file.

- ```

- sudo cp example_config.yaml config.yaml

- ```

-1. Set the `agent_id` field to a unique identifier for the agent instance – for example `london-sftp-1`. This name becomes searchable metadata in Operator Insights for all data ingested by this agent. Reserved URL characters must be percent-encoded.

-1. Configure the `secret_providers` section.

- # [SFTP sources](#tab/sftp)

- SFTP sources require two types of secret providers.

- - A secret provider of type `key_vault`, which contains details required to connect to the Data Product's Azure Key Vault and allow connection to the Data Product's input storage account.

- - A secret provider of type `file_system`, which specifies a directory on the VM for storing credentials for connecting to an SFTP server.

-

- 1. For the secret provider with type `key_vault` and name `data_product_keyvault`, set the following fields.

- - `vault_name` must be the name of the Key Vault for your Data Product. You identified this name in [Grant permissions for the Data Product Key Vault](#grant-permissions-for-the-data-product-key-vault).

- - Depending on the type of authentication you chose in [Set up authentication to Azure](#set-up-authentication-to-azure), set either `managed_identity` or `service_principal`.

- - For a managed identity: set `object_id` to the Object ID of the managed identity that you created in [Use a managed identity for authentication](#use-a-managed-identity-for-authentication).

- - For a service principal: set `tenant_id` to your Microsoft Entra ID tenant, `client_id` to the Application (client) ID of the service principal that you created in [Create a service principal](#create-a-service-principal), and `cert_path` to the file path of the base64-encoded P12 certificate on the VM.

- 1. For the secret provider with type `file_system` and name `local_file_system`, set the following fields.

- - `secrets_directory` to the absolute path to the secrets directory on the agent VM, which was created in the [Prepare the VMs](#prepare-the-vms) step.

-

- You can add more secret providers (for example, if you want to upload to multiple Data Products) or change the names of the default secret providers.

- # [MCC EDR sources](#tab/edr)

-

- Configure a secret provider with type `key_vault` and name `data_product_keyvault`, setting the following fields.

- 1. For the secret provider with type `key_vault` and name `data_product_keyvault`, set the following fields.

- - `vault_name` must be the name of the Key Vault for your Data Product. You identified this name in [Grant permissions for the Data Product Key Vault](#grant-permissions-for-the-data-product-key-vault).

- - Depending on the type of authentication you chose in [Set up authentication to Azure](#set-up-authentication-to-azure), set either `managed_identity` or `service_principal`.

- - For a managed identity: set `object_id` to the Object ID of the managed identity that you created in [Use a managed identity for authentication](#use-a-managed-identity-for-authentication).

- - For a service principal: set `tenant_id` to your Microsoft Entra ID tenant, `client_id` to the Application (client) ID of the service principal that you created in [Create a service principal](#create-a-service-principal), and `cert_path` to the file path of the base64-encoded P12 certificate on the VM.

- You can add more secret providers (for example, if you want to upload to multiple Data Products) or change the names of the default secret provider.

-

-1. Configure the `pipelines` section using the example configuration and your Data Product's documentation. Each `pipeline` has three configuration sections.

- - `id`. The ID identifies the pipeline and must not be the same as any other pipeline ID for this ingestion agent. Any URL reserved characters must be percent-encoded. Refer to your Data Product's documentation for any recommendations.

- - `source`. Source configuration controls which files are ingested. You can configure multiple sources.

- # [SFTP sources](#tab/sftp)

- Delete all pipelines in the example except the `contoso-logs` example, which contains `sftp_pull` source configuration.

- Update the example to meet your requirements. The following fields are required for each source.

- - `host`: the hostname or IP address of the SFTP server.

- - `filtering.base_path`: the path to a folder on the SFTP server that files will be uploaded to Azure Operator Insights from.

- - `known_hosts_file`: the path on the VM to the global known_hosts file, located at `/etc/ssh/ssh_known_hosts`. This file should contain the public SSH keys of the SFTP host server as outlined in [Prepare the VMs](#prepare-the-vms).

- - `user`: the name of the user on the SFTP server that the agent should use to connect.

- - Depending on the method of authentication you chose in [Prepare the VMs](#prepare-the-vms), set either `password` or `private_key`.

- - For password authentication, set `secret_name` to the name of the file containing the password in the `secrets_directory` folder.

- - For SSH key authentication, set `key_secret_name` to the name of the file containing the SSH key in the `secrets_directory` folder. If the private key is protected with a passphrase, set `passphrase_secret_name` to the name of the file containing the passphrase in the `secrets_directory` folder.

- - All secret files should have permissions of `600` (`rw-`), and an owner of `az-aoi-ingestion` so only the ingestion agent and privileged users can read them.

- ```

- sudo chmod 600 <secrets_directory>/*

- sudo chown az-aoi-ingestion <secrets_directory>/*

- ```

-

- For required or recommended values for other fields, refer to the documentation for your Data Product.

- > [!TIP]

- > The agent supports additional optional configuration for the following:

- > - Specifying a pattern of files in the `base_path` folder which will be uploaded (by default all files in the folder are uploaded).

- > - Specifying a pattern of files in the `base_path` folder which should not be uploaded.

- > - A time and date before which files in the `base_path` folder will not be uploaded.

- > - How often the ingestion agent uploads files (the value provided in the example configuration file corresponds to every hour).

- > - A settling time, which is a time period after a file is last modified that the agent will wait before it is uploaded (the value provided in the example configuration file is 5 minutes).

- >

- > For more information about these configuration options, see [Configuration reference for Azure Operator Insights ingestion agent](ingestion-agent-configuration-reference.md).

- # [MCC EDR sources](#tab/edr)

- Delete all pipelines in the example except `mcc_edrs`. Most of the fields in `mcc_edrs` are set to default values. You can leave them unchanged unless you need a specific value.

-

- - `sink`. Sink configuration controls uploading data to the Data Product's input storage account.

- - In the `sas_token` section, set the `secret_provider` to the appropriate `key_vault` secret provider for the Data Product, or use the default `data_product_keyvault` if you used the default name earlier. Leave `secret_name` unchanged.

- - Refer to your Data Product's documentation for information on required values for other parameters.

- > [!IMPORTANT]

- > The `container_name` field must be set exactly as specified by your Data Product's documentation.

-## Start the agent software

-1. Start the agent.

- ```

- sudo systemctl start az-aoi-ingestion

- ```

-1. Check that the agent is running.

- ```

- sudo systemctl status az-aoi-ingestion

- ```

- 1. If you see any status other than `active (running)`, look at the logs as described in [Monitor and troubleshoot ingestion agents for Azure Operator Insights](monitor-troubleshoot-ingestion-agent.md) to understand the error. It's likely that some configuration is incorrect.

- 1. Once you resolve the issue,  attempt to start the agent again.

- 1. If issues persist, raise a support ticket.

-1. Once the agent is running, ensure it starts automatically after reboot.

- ```

- sudo systemctl enable az-aoi-ingestion.service

- ```

-## [Optional] Configure log collection for access through Azure Monitor

-If you're running the ingestion agent on an Azure VM or on an on-premises VM connected by Azure Arc, you can send ingestion agent logs to Azure Monitor using the Azure Monitor Agent. Using Azure Monitor to access logs can be simpler than accessing logs directly on the VM.

-To collect ingestion agent logs, follow [the Azure Monitor documentation to install the Azure Monitor Agent and configure log collection](../azure-monitor/agents/data-collection-text-log.md).

- - The `YourOptionalColumn` section from the sample `$tableParams` JSON is unnecessary for the ingestion agent, and can be removed.

- ```

- <CustomTableName>

- | extend RawData = replace_regex(RawData, '\\x1b\\[\\d{1,4}m', '') // Remove any color tags

- | parse RawData with TimeGenerated: datetime ' ' Level ' ' Message // Parse the log lines into the TimeGenerated, Level and Message columns for easy filtering

- | order by TimeGenerated desc

- ```

- > [!NOTE]

- > This query can't be used as a data source transform, because `replace_regex` isn't available in data source transforms.

-### Sample logs

-```

-2024-04-30T17:16:00.000544Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_sftp_pull_source::sftp::source: Starting run with 'last checkpoint' timestamp: None

-2024-04-30T17:16:00.000689Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_sftp_pull_source::sftp::source: Starting Completion Handler task

-2024-04-30T17:16:00.073495Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_sftp_pull_source::sftp::sftp_file_tree_explorer: Start traversing files with base path "/"

-2024-04-30T17:16:00.086427Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_sftp_pull_source::sftp::sftp_file_tree_explorer: Finished traversing files

-2024-04-30T17:16:00.086698Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_sftp_pull_source::sftp::source: File explorer task is complete, with result Ok(())

-2024-04-30T17:16:00.086874Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_sftp_pull_source::sftp::source: Send files to sink task is complete

-2024-04-30T17:16:00.087041Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_sftp_pull_source::sftp::source: Processed all completion notifications for run

-2024-04-30T17:16:00.087221Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_sftp_pull_source::sftp::source: Run complete with no retryable errors - updating last checkpoint timestamp

-2024-04-30T17:16:00.087351Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_sftp_pull_source::sftp::source: Run lasted 0 minutes and 0 seconds with result: RunStats { successful_uploads: 0, retryable_errors: 0, non_retryable_errors: 0, blob_already_exists: 0 }

-2024-04-30T17:16:00.087421Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_sftp_pull_source::sftp::file: Closing 1 active SFTP connections

-2024-04-30T17:16:00.087966Z  INFO sftp_pull{pipeline_id="test-files"}:execute_run{start_time="2024-04-30 17:16:00.000524 UTC"}: az_ingestion_common::scheduler: Run completed successfully. Update the 'last checkpoint' time to 2024-04-30T17:15:30.000543200Z

-2024-04-30T17:16:00.088122Z  INFO sftp_pull{pipeline_id="test-files"}: az_ingestion_common::scheduler: Schedule next run at 2024-04-30T17:17:00Z

-```

-## Related content

-Learn how to:

-description: Learn how to upgrade the Azure Operator Insights ingestion agent to receive the latest new features or fixes.

-#CustomerIntent: As a someone managing an agent that has already been set up, I want to upgrade it to receive the latest enhancements or fixes.

-# Upgrade Azure Operator Insights ingestion agents

-The ingestion agent is a software package that is installed onto a Linux Virtual Machine (VM) owned and managed by you. You might need to upgrade the agent.

-This article describes how to upgrade your ingestion agent, and how to roll back an upgrade.

-## Prerequisites

-Decide which version of the ingestion agent you would like to upgrade to. If you don't specify a version when you upgrade, you'll upgrade to the most recent version.

-See [What's new with Azure Operator Insights ingestion agent](ingestion-agent-release-notes.md) for a list of recent releases and to see what's changed in each version. If you're looking for an agent version that's more than six months old, check out the [release notes archive](ingestion-agent-release-notes-archive.md).

-If you would like to verify the authenticity of the ingestion agent package before upgrading, see [How to use the GPG Repository Signing Key](/linux/packages#how-to-use-the-gpg-repository-signing-key).

-## Upgrade the agent software

-To upgrade to a new release of the agent, repeat the following steps on each VM that has the old agent.

-1. Connect to the VM over SSH.

-1. Save a copy of the existing */etc/az-aoi-ingestion/config.yaml* configuration file.

-1. Upgrade the agent using your VM's package manager. For example, for Red Hat-based Linux Distributions:

- ```

- sudo dnf upgrade az-aoi-ingestion

- ```

- Answer `y` when prompted.

- 1. Alternatively, to upgrade to a specific version of the agent, specify the version number in the command. For example, for version 2.0.0 on a RHEL8 system, use the following command:

- ```

- sudo dnf install az-aoi-ingestion-2.0.0

- ```

-1. Make any changes to the configuration file described by your support contact or the documentation for the new version. Most upgrades don't require any configuration changes.

-1. Restart the agent.

- ```

- sudo systemctl restart az-aoi-ingestion.service

- ```

-1. Once the agent is running, configure the az-aoi-ingestion service to automatically start on a reboot.

- ```

- sudo systemctl enable az-aoi-ingestion.service

- ```

-1. Verify that the agent is running and that it's copying files as described in [Monitor and troubleshoot Ingestion Agents for Azure Operator Insights](monitor-troubleshoot-ingestion-agent.md).

-## Roll back an upgrade

-If an upgrade or configuration change fails:

-1. Downgrade back to the previous version by reinstalling the previous version of the agent. For example, to downgrade to version 1.0.0 on a RHEL8 system, use the following command:

- ```

- sudo dnf downgrade az-aoi-ingestion-1.0.0

- ```

-1. Copy the backed-up configuration file from before the change to the */etc/az-aoi-ingestion/config.yaml* file.

-1. Restart the agent.

- ```

- sudo systemctl restart az-aoi-ingestion.service

- ```

-1. When the agent is running, configure the az-aoi-ingestion service to automatically start on a reboot.

- ```

- sudo systemctl enable az-aoi-ingestion.service

- ```

-## Related content

-Learn how to:

-To provide threat-based recommendations, SOC optimization looks at your ingested logs and enabled analytics rules, and compares it to the logs and detections that are required to protect, detect, and respond to specific types of attacks. This optimization type is also known as *coverage optimization*, and is based on Microsoft's security research.

- --name myResourceGroup \

- --location eastus

-Create a route table with [az network route-table create](/cli/azure/network/route-table#az-network-route-table-create). The following example creates a route table named *myRouteTablePublic*.

- --resource-group myResourceGroup \

- --name myRouteTablePublic

- --name ToPrivateSubnet \

- --resource-group myResourceGroup \

- --route-table-name myRouteTablePublic \

- --name myVirtualNetwork \

- --resource-group myResourceGroup \

- --subnet-name Public \

-Create two additional subnets with [az network vnet subnet create](/cli/azure/network/vnet/subnet).

- --vnet-name myVirtualNetwork \

- --resource-group myResourceGroup \

- --name Private \

- --vnet-name myVirtualNetwork \

- --resource-group myResourceGroup \

- --name DMZ \

-Associate the *myRouteTablePublic* route table to the *Public* subnet with [az network vnet subnet update](/cli/azure/network/vnet/subnet).

- --vnet-name myVirtualNetwork \

- --name Public \

- --resource-group myResourceGroup \

- --route-table myRouteTablePublic

-An NVA is a VM that performs a network function, such as routing, firewalling, or WAN optimization. We will create a basic NVA from a general purpose Ubuntu VM, for demonstration purposes.

-Create a VM to be used as the NVA in the *DMZ* subnet with [az vm create](/cli/azure/vm). When you create a VM, Azure creates and assigns a network interface *myVmNvaVMNic* and a public IP address to the VM, by default. The `--public-ip-address ""` parameter instructs Azure not to create and assign a public IP address to the VM, since the VM doesn't need to be connected to from the internet. If SSH keys do not already exist in a default key location, the command creates them. To use a specific set of keys, use the `--ssh-key-value` option.

- --resource-group myResourceGroup \

- --name myVmNva \

- --subnet DMZ \

- --vnet-name myVirtualNetwork \

-The VM takes a few minutes to create. Do not continue to the next step until Azure finishes creating the VM and returns output about the VM.

-For a network interface myVmNvaVMNic to be able to forward network traffic sent to it, that is not destined for its own IP address, IP forwarding must be enabled for the network interface. Enable IP forwarding for the network interface with [az network nic update](/cli/azure/network/nic).

- --name myVmNvaVMNic \

- --resource-group myResourceGroup \

-Within the VM, the operating system, or an application running within the VM, must also be able to forward network traffic. We will use the `sysctl` command to enable the Linux kernel to forward packets. To run this command without logging onto the VM, we will use the [Custom Script extension](/azure/virtual-machines/extensions/custom-script-linux) [az vm extension set](/cli/azure/vm/extension):

- --resource-group myResourceGroup \

- --vm-name myVmNva \

-The command may take up to a minute to execute. Note that this change will not persist after a VM reboot, so if the NVA VM is rebooted for any reason, the script will need to be repeated.

-Create two VMs in the virtual network so you can validate that traffic from the *Public* subnet is routed to the *Private* subnet through the NVA in a later step.

-Create a VM in the *Public* subnet with [az vm create](/cli/azure/vm). The `--no-wait` parameter enables Azure to execute the command in the background so you can continue to the next command. To streamline this article, a password is used. Keys are typically used in production deployments. If you use keys, you must also configure SSH agent forwarding. For more information, see the documentation for your SSH client. Replace `<replace-with-your-password>` in the following command with a password of your choosing.

-```azurecli-interactive

-adminPassword="<replace-with-your-password>"

- --resource-group myResourceGroup \

- --name myVmPublic \

- --vnet-name myVirtualNetwork \

- --subnet Public \

- --admin-password $adminPassword \

-Create a VM in the *Private* subnet.

- --resource-group myResourceGroup \

- --name myVmPrivate \

- --vnet-name myVirtualNetwork \

- --subnet Private \

- --admin-password $adminPassword

- "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachines/myVmPrivate",

- "location": "eastus",

- "publicIpAddress": "13.90.242.231",

- "resourceGroup": "myResourceGroup"

-Take note of the **publicIpAddress**. This address is used to access the VM from the internet in a later step.

-Using an SSH client of your choice, connect to the VMs created above. For example, the following command can be used from a command line interface such as [WSL](/windows/wsl/install) to create an SSH session with the *myVmPrivate* VM. Replace *\<publicIpAddress>* with the public IP address of your VM. In the example above, the IP address is *13.90.242.231*.

-ssh azureuser@<publicIpAddress>

-When prompted for a password, enter the password you selected in [Create virtual machines](#create-virtual-machines).

-Use the following command to install trace route on the *myVmPrivate* VM:

-Use the following command to test routing for network traffic to the *myVmPublic* VM from the *myVmPrivate* VM.

-traceroute myVmPublic

-traceroute to myVmPublic (10.0.0.4), 30 hops max, 60 byte packets

-1 10.0.0.4 (10.0.0.4) 1.404 ms 1.403 ms 1.398 ms

-You can see that traffic is routed directly from the *myVmPrivate* VM to the *myVmPublic* VM. Azure's default routes, route traffic directly between subnets.

-Use the following command to SSH to the *myVmPublic* VM from the *myVmPrivate* VM:

-ssh azureuser@myVmPublic

-Use the following command to install trace route on the *myVmPublic* VM:

-sudo apt-get install traceroute

-Use the following command to test routing for network traffic to the *myVmPrivate* VM from the *myVmPublic* VM.

-traceroute myVmPrivate

-traceroute to myVmPrivate (10.0.1.4), 30 hops max, 60 byte packets

-1 10.0.2.4 (10.0.2.4) 0.781 ms 0.780 ms 0.775 ms

-2 10.0.1.4 (10.0.0.4) 1.404 ms 1.403 ms 1.398 ms

-You can see that the first hop is 10.0.2.4, which is the NVA's private IP address. The second hop is 10.0.1.4, the private IP address of the *myVmPrivate* VM. The route added to the *myRouteTablePublic* route table and associated to the *Public* subnet caused Azure to route the traffic through the NVA, rather than directly to the *Private* subnet.

-Close the SSH sessions to both the *myVmPublic* and *myVmPrivate* VMs.

-az group delete --name myResourceGroup --yes

-In this article, you created a route table and associated it to a subnet. You created a simple NVA that routed traffic from a public subnet to a private subnet. Deploy a variety of pre-configured NVAs that perform network functions such as firewall and WAN optimization from the [Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/category/networking). To learn more about routing, see [Routing overview](virtual-networks-udr-overview.md) and [Manage a route table](manage-route-table.yml).

-While you can deploy many Azure resources within a virtual network, resources for some Azure PaaS services cannot be deployed into a virtual network. You can still restrict access to the resources of some Azure PaaS services to traffic only from a virtual network subnet though. To learn how, see [Restrict network access to PaaS resources](tutorial-restrict-network-access-to-resources-cli.md).