+ | Managed By Tenant ID | The **Tenant ID** of your Azure AD B2C tenant (also known as the directory ID). |

+1. Make sure you're using the directory that contains your Azure AD B2C tenant:

+ 1. Select the **Directories + subscriptions** icon in the portal toolbar.

+ 2. On the **Portal settings | Directories + subscriptions** page, find your Azure AD B2C directory in the **Directory name** list, and then select **Switch**.

+1. If there are existing settings for the resource, you'll see a list of settings already configured. Either select **Add diagnostic setting** to add a new setting, or select **Edit settings** to edit an existing setting. Each setting can have no more than one of each of the destination types.

+ 

+1. Select **Send to Log Analytics Workspace**, and then:

+ 1. Under **Subscription**, select your subscription.

+ 2. Under **Log Analytics Workspace**, select the name of the workspace you created earlier such as `AzureAdB2C`.

+1. From **Log Analytics workspace** window, select **Logs**

+1. From the **Log Analytics workspace** window, select **Workbooks**.

+Alerts are created by alert rules in Azure Monitor and can automatically run saved queries or custom log searches at regular intervals. You can create alerts based on specific performance metrics or when certain events occur. You can also create alerts on absence of an event, or a number of events occur within a particular time window. For example, alerts can be used to notify you when average number of sign in exceeds a certain threshold. For more information, see [Create alerts](../azure-monitor/alerts/alerts-log.md).

+1. In your browser debugger, [run the following JavaScript in the Console](/microsoft-edge/devtools-guide-chromium/console/console-javascript). The JavaScript code will present information about the sign in user.

+* Learn how to [customize and enhance the Azure AD B2C authentication experience for your web app](enable-authentication-azure-static-app-options.md).

+This article explains how to add Azure Active Directory B2C (Azure AD B2C) authentication functionality to an Azure Web App. For more information, check out the [File-based configuration in Azure App Service authentication](/azure/app-service/configure-authentication-file-based) article.

+* After successful authentication, you can show display name on the navigation bar. To view the claims that the Azure AD B2C token returns to your app, check out the [Work with user identities in Azure App Service authentication](/azure/app-service/configure-authentication-user-identities).

+* Lear how to [Work with OAuth tokens in Azure App Service authentication](/azure/app-service/configure-authentication-oauth-tokens).

+This article explains how to add Azure Active Directory B2C (Azure AD B2C) authentication functionality to an Azure Web App. For more information, check out the [configure your App Service or Azure Functions app to login using an OpenID Connect provider](/azure/app-service/configure-authentication-provider-openid-connect) article.

+ > Your client secret will be stored as an app setting to ensure secrets are stored in a secure fashion. You can update that setting later to use [Key Vault references](/azure/app-service/app-service-key-vault-references) if you wish to manage the secret in Azure Key Vault.

+* After successful authentication, you can show display name on the navigation bar. To view the claims that the Azure AD B2C token returns to your app, check out the [Work with user identities in Azure App Service authentication](/azure/app-service/configure-authentication-user-identities).

+* Lear how to [Work with OAuth tokens in Azure App Service authentication](/azure/app-service/configure-authentication-oauth-tokens).

+1. Make sure you're using the directory that contains your subscription:

+ 1. Find the directory that contains your subscription and select the **Switch** button next to it. Switching a directory reloads the portal. If the directory that contains your subscription has the **Current** label next to it, you don't need to do anything.

+ 

+ <ValidationTechnicalProfile ReferenceId="REST-ReadProfileFromCustomersDatabase" ContinueOnError="true" >

+- The eighth retry happens 120 hours after the first failure.

+You can invite a Google user to B2B collaboration in various ways. For example, you can [add them to your directory via the Azure portal](b2b-quickstart-add-guest-users-portal.md). When they redeem your invitation, their experience varies depending on whether they're already signed in to Google:

+* Application authorization - Token issued by Azure AD can include claims generated from user attributes so that applications can make authorization decision based on the claims in token.

+* Establish Azure AD footprint

+>[!NOTE]

+> The states in this diagram represent a logical progression of cloud transformation.

+* **Establish new location** - You purchase your destination and establish connectivity between the current location and the new location. This enables you to maintain your productivity and ability to operate. In this content, the activities are described in **[Establish Azure AD footprint](road-to-the-cloud-establish.md)**. The results transition you to State 2.

+* **Establish Azure AD footprint**: Initialize your new Azure AD tenant to supports the vision for your end-state deployment. Adopt a [Zero Trust](https://www.microsoft.com/security/blog/2020/04/30/zero-trust-deployment-guide-azure-active-directory/) approach and a security model that [protects your tenant from on-premises compromise](../fundamentals/protect-m365-from-on-premises-attacks.md) early in your journey.

+* **Implement cloud-first approach**: Establish a policy that mandates all new devices, apps and services should be cloud-first. New applications and services using legacy protocols (NTLM, Kerberos, LDAP etc.) should be by exception only.

+* **Transition to the cloud**: Shift the management and integration of users, apps and devices away from on-premises and over to cloud-first alternatives. Optimize user provisioning by taking advantage of [cloud-first provisioning capabilities](../governance/what-is-provisioning.md) that integrate with Azure AD.

+3. Determine what processes can be used to supply authoritative information in the absence of a system of record. For example, if there are digital identities for visitors, but the organization has no database for visitors, then it may be necessary to find an alternate way to determine when an digital identity for a visitor is no longer needed.

+4. Ensure that changes from the system of record or other processes are replicated to each of the directories or databases that require an update.

+ - Move - when an individual moves between boundaries that require additional access authorizations to be added or removed to their digital identity

+ - Leave- when an individual leaves the scope of needing access, access may need to be removed, and subsequently the identity may no longer be required by applications other than for audit or forensics purposes

+So for example, if a new employee joins your organization and that employee has never been affiliated with your organization before, that employee will require a new digital identity, represented as a user account in Azure AD. The creation of this account would fall into a "Joiner" process, which could be automated if there was a system of record such as Workday that could indicate when the new employee starts work. Later, if your organization has an employee move from say, Sales to Marketing, they would fall into a "Mover" process. This would require removing the access rights they had in the Sales organization which they no longer require, and granting them rights in the Marketing organization that they new require.

+6. Reboot for the changes to take effect.

+The following documentation provides reference information for the ADConnectivityTools PowerShell Module that is included with Azure AD Connect in `C:\Program Files\Microsoft Azure Active Directory Connect\Tools\ADConnectivityTool.psm1`.

+> - Microsoft Purview doesn't support the Global Reader role.

+az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --enable-secret-rotation

+az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --enable-secret-rotation --rotation-poll-interval 5m

+az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --disable-secret-rotation

+| counter-key | The key to use for the rate limit policy. For each key value, a single counter is used for all scopes at which the policy is configured. | Yes | N/A |

+| counter-key | The key to use for the quota policy. For each key value, a single counter is used for all scopes at which the policy is configured. | Yes | N/A |

+A backend can have NICs, virtual machine scale sets, public IP addresses, internal IP addresses, fully qualified domain names (FQDN), and multi-tenant back-ends like Azure App Service. In this example, you create two virtual machines to use as backend servers for the application gateway. You also install NGINX on the virtual machines to test the application gateway.

+ > For more information, see [**create a Form Recognizer resource**](create-a-form-recognizer-resource.md).

+* Models trained with API version 2022-06-30-preview or later will accept tabular field labels.

+* Documents analyzed with custom neural models using API version 2022-06-30-preview or later will produce tabular fields aggregated across the tables.

+As of August 01 2022, Form Recognizer custom neural model training will only be available in the following Azure regions until further notice:

+* Brazil South

+* Canada Central

+* Central India

+* Japan East

+* North Europe

+* South Central US

+* Southeast Asia

+> You can [copy a model](disaster-recovery.md) trained in one of the select regions listed above to **any other region** and use it accordingly.

+Custom neural models differ from custom template models in a few different ways. The custom template or model relies on a consistent visual template to extract the labeled data. Custom neural models support structured, semi-structured, and unstructured documents to extract fields. When you're choosing between the two model types, start with a neural model, and test to determine if it supports your functional needs.

+ > [Form Recognizer API v3.0](https://westus.dev.cognitive.microsoft.com/docs/services/form-recognizer-api-v3-0-preview-2/operations/AnalyzeDocument)

+|🆕 [prebuilt-tax.us.w2](concept-w2.md#field-extraction) | ✓ | | ✓ | | ✓ | | | ✓ |

+|🆕 [prebuilt-document](concept-general-document.md#data-extraction)| ✓ | | ✓ | ✓ | ✓ | | ✓ | |

+| [prebuilt-layout](concept-layout.md#data-extraction) | Γ£ô | | Γ£ô | Γ£ô | Γ£ô | Γ£ô | | |

+> * You can refer to the [API migration guide](v3-migration-guide.md) for detailed information about migrating from v2.1 to v3.0.

+ > Custom neural models models are only available in a few regions. If you plan on training a neural model, please select or create a resource in one of [these supported regions](../concept-custom-neural.md#supported-regions).

+> * You can refer to the [API migration guide](v3-migration-guide.md) for detailed information about migrating from v2.1 to v3.0.

+> [!TIP]

+> Create a Cognitive Services resource if you plan to access multiple cognitive services under a single endpoint/key. For Form Recognizer access only, create a Form Recognizer resource. Please note that you'll need a single-service resource if you intend to use [Azure Active Directory authentication](../../../active-directory/authentication/overview-authentication.md).

+> * You can refer to the [API migration guide](v3-migration-guide.md) for detailed information about migrating from v2.1 to v3.0.

+* If your project has a table tag, you can open the labeling panel, and populate the tag as you would label key-value fields.

+Import the **Microsoft.Identity.Client** NuGet package, which is used to acquire a token.

+Create a confidential client application property.

+```csharp

+private IConfidentialClientApplication _confidentialClientApplication;

+private IConfidentialClientApplication ConfidentialClientApplication

+{

+ get {

+ if (_confidentialClientApplication == null) {

+ _confidentialClientApplication = ConfidentialClientApplicationBuilder.Create(ClientId)

+ .WithClientSecret(ClientSecret)

+ .WithAuthority($"https://login.windows.net/{TenantId}")

+ .Build();

+ }

+ return _confidentialClientApplication;

+ }

+}

+```

+Next, use the following code to acquire an `AuthenticationResult`, using the authentication values you got when you [created the Immersive Reader resource](./how-to-create-immersive-reader.md).

+public async Task<string> GetTokenAsync()

+ const string resource = "https://cognitiveservices.azure.com/";

+ var authResult = await ConfidentialClientApplication.AcquireTokenForClient(

+ new[] { $"{resource}/.default" })

+ .ExecuteAsync()

+ .ConfigureAwait(false);

+ return authResult.AccessToken;

+* For Python 3 Hybrid jobs on Linux machines, we depend on the Python 3 version installed on the machine to run DSC OMSConfig and the Linux Hybrid Worker. Different versions should work if there are no breaking changes in method signatures or contracts between versions of Python 3.

+| connectionString | Yes | Read-write connection string for the App Configuration instance. The connection string should be stored as a secret in the GitHub repository, and only the secret name should be used in the workflow. |

+ Title: Clean up past installations

+description: Describes how to remove Azure Arc-enabled data controller and associated resources from past installations.

+# Clean up from past installations

+If you installed the data controller in the past and later deleted the data controller, there may be some cluster level objects that would still need to be deleted.

+This article describes how to delete these cluster level objects.

+## Replace values in sample script

+For some of the tasks, you'll need to replace `{namespace}` with the value for your namespace. Substitute the name of the namespace the data controller was deployed in into `{namespace}`. If unsure, get the name of the `mutatingwebhookconfiguration` using `kubectl get clusterrolebinding`.

+## Run script to remove artifacts

+Run the following commands to delete the data controller cluster level objects:

+> [!NOTE]

+> Not all of these objects will exist in your environment. The objects in your environment depend on which version of the Arc data controller was installed

+```console

+# Clean up azure arc data service artifacts

+# Custom resource definitions (CRD)

+kubectl delete crd datacontrollers.arcdata.microsoft.com

+kubectl delete crd postgresqls.arcdata.microsoft.com

+kubectl delete crd sqlmanagedinstances.sql.arcdata.microsoft.com

+kubectl delete crd sqlmanagedinstancerestoretasks.tasks.sql.arcdata.microsoft.com

+kubectl delete crd dags.sql.arcdata.microsoft.com

+kubectl delete crd exporttasks.tasks.arcdata.microsoft.com

+kubectl delete crd monitors.arcdata.microsoft.com

+kubectl delete crd activedirectoryconnectors.arcdata.microsoft.com

+# Substitute the name of the namespace the data controller was deployed in into {namespace}.

+# Cluster roles and role bindings

+kubectl delete clusterrole arcdataservices-extension

+kubectl delete clusterrole arc:cr-arc-metricsdc-reader

+kubectl delete clusterrole arc:cr-arc-dc-watch

+kubectl delete clusterrole cr-arc-webhook-job

+kubectl delete clusterrole {namespace}:cr-upgrade-worker

+kubectl delete clusterrole {namespace}:cr-deployer

+kubectl delete clusterrolebinding {namespace}:crb-arc-metricsdc-reader

+kubectl delete clusterrolebinding {namespace}:crb-arc-dc-watch

+kubectl delete clusterrolebinding crb-arc-webhook-job

+kubectl delete clusterrolebinding {namespace}:crb-upgrade-worker

+kubectl delete clusterrolebinding {namespace}:crb-deployer

+# Substitute the name of the namespace the data controller was deployed in into {namespace}. If unsure, get the name of the mutatingwebhookconfiguration using 'kubectl get clusterrolebinding'

+# API services

+# Up to May 2021 release

+kubectl delete apiservice v1alpha1.arcdata.microsoft.com

+kubectl delete apiservice v1alpha1.sql.arcdata.microsoft.com

+# June 2021 release

+kubectl delete apiservice v1beta1.arcdata.microsoft.com

+kubectl delete apiservice v1beta1.sql.arcdata.microsoft.com

+# GA/July 2021 release

+kubectl delete apiservice v1.arcdata.microsoft.com

+kubectl delete apiservice v1.sql.arcdata.microsoft.com

+# Substitute the name of the namespace the data controller was deployed in into {namespace}. If unsure, get the name of the mutatingwebhookconfiguration using 'kubectl get mutatingwebhookconfiguration'

+kubectl delete mutatingwebhookconfiguration arcdata.microsoft.com-webhook-{namespace}

+```

+## Next steps

+[Start by creating a Data Controller](create-data-controller-indirect-cli.md)

+Already created a Data Controller? [Create an Azure Arc-enabled SQL Managed Instance](create-sql-managed-instance.md)

+ Title: Create a data controller using Kubernetes tools

+description: Create a data controller using Kubernetes tools

+# Create Azure Arc-enabled data controller using Kubernetes tools

+A data controller manages Azure Arc-enabled data services for a Kubernetes cluster. This article describes how to use Kubernetes tools to create a data controller.

+Creating the data controller has the following high level steps:

+1. Create the namespace and bootstrapper service

+1. Create the data controller

+> For simplicity, the steps below assume that you are a Kubernetes cluster administrator. For production deployments or more secure environments, it is recommended to follow the security best practices of "least privilege" when deploying the data controller by granting only specific permissions to users and service accounts involved in the deployment process.

+## Prerequisites

+Review the topic [Plan an Azure Arc-enabled data services deployment](plan-azure-arc-data-services.md) for overview information.

+To create the data controller using Kubernetes tools you will need to have the Kubernetes tools installed. The examples in this article will use `kubectl`, but similar approaches could be used with other Kubernetes tools such as the Kubernetes dashboard, `oc`, or `helm` if you are familiar with those tools and Kubernetes yaml/json.

+[Install the kubectl tool](https://kubernetes.io/docs/tasks/tools/install-kubectl/)

+## Create the namespace and bootstrapper service

+The bootstrapper service handles incoming requests for creating, editing, and deleting custom resources such as a data controller.

+Save a copy of [arcdata-deployer.yaml](https://raw.githubusercontent.com/microsoft/azure_arc/main/arc_data_services/deploy/yaml/bootstrapper-unified.yaml), and replace the placeholder `{{NAMESPACE}}` in *all the places* in the file with the desired namespace name, for example: `arc`.

+> [!IMPORTANT]

+> The bootstrapper-unified.yaml template file defaults to pulling the bootstrapper container image from the Microsoft Container Registry (MCR). If your environment can't directly access the Microsoft Container Registry, you can do the following:

+- Follow the steps to [pull the container images from the Microsoft Container Registry and push them to a private container registry](offline-deployment.md).

+- [Create an image pull secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-by-providing-credentials-on-the-command-line) named `arc-private-registry` for your private container registry.

+- Change the image URL for the bootstrapper image in the bootstrap.yaml file.

+- Replace `arc-private-registry` in the bootstrap.yaml file if a different name was used for the image pull secret.

+Run the following command to create the namespace and bootstrapper service with the edited file.

+kubectl apply --namespace arc -f bootstrapper-unified.yaml

+Verify that the bootstrapper pod is running using the following command.

+kubectl get pod --namespace arc -l app=bootstrapper

+If the status is not _Running_, run the command a few times until the status is _Running_.

+## Create the data controller

+Now you are ready to create the data controller itself.

+First, create a copy of the [template file](https://raw.githubusercontent.com/microsoft/azure_arc/main/arc_data_services/deploy/yaml/data-controller.yaml) locally on your computer so that you can modify some of the settings.

+### Create the metrics and logs dashboards user names and passwords

+At the top of the file, you can specify a user name and password that is used to authenticate to the metrics and logs dashboards as an administrator. Choose a secure password and share it with only those that need to have these privileges.

+### Create certificates for logs and metrics dashboards

+Optionally, you can create SSL/TLS certificates for the logs and metrics dashboards. Follow the instructions at [Specify SSL/TLS certificates during Kubernetes native tools deployment](monitor-certificates.md).

+### Edit the data controller configuration

+Edit the data controller configuration as needed:

+- **dockerRegistry**: The secret to use to pull the images from a private container registry if required.

+The following example shows a completed data controller yaml.

+kubectl create --namespace arc -f data-controller.yaml

+kubectl get datacontroller --namespace arc

+You can also check on the creation status or logs of any particular pod by running a command like below. This is especially useful for troubleshooting any issues.

+kubectl logs <pod name> --namespace arc

+#kubectl logs control-2g7b1 --namespace arc

+- [Create a PostgreSQL Hyperscale server group using Kubernetes-native tools](./create-postgresql-hyperscale-server-group-kubernetes-native-tools.md)

+- Point-in-time restore is database level feature, not an instance level feature. You cannot restore the entire instance with Point-in-time restore.

+- You can only restore to the same Azure Arc-enabled SQL managed instance from where the backup was taken.

+## July 12, 2022

+This release is published July 12, 2022

+### Image tag

+`v1.9.0_2022-07-12`

+For complete release version information, see [Version log](version-log.md#july-12-2022).

+### Miscellaneous

+- Extended the disk metrics reported in monitoring dashboards to include more queue length stats and more counters for IOPS. All disks are in scope for data collection that start with `vd` or `sd` now.

+### Arc-enabled SQL Managed Instance

+- Added buffer cache hit ratio to `collectd` and surface it in monitoring dashboards.

+- Improvements to the formatting of the legends on some dashboards.

+- Added process level CPU and memory metrics to the monitoring dashboards for the SQL managed instance process.

+- `syncSecondaryToCommit` property is now available to be viewed and edited in Azure portal and Azure Data Studio.

+- Added ability to set the DNS name for the readableSecondaries service in Azure CLI and Azure portal.

+- Now collecting the `agent.log`, `security.log` and `sqlagentstartup.log` for Arc-enabled SQL Managed instance to ElasticSearch so they're searchable via Kibana and optionally uploading them to Azure Log Analytics.

+- There are more additional notifications when provisioning new SQL managed instances is blocked due to not exporting/uploading billing data to Azure.

+### Data controller

+- Permissions required to deploy the Arc data controller have been reduced to a least-privilege level.

+- When deployed via the Azure CLI, the Arc data controller is now installed via a K8s job that uses a helm chart to do the installation. There's no change to the user experience.

+ Title: Upgrade indirectly connected data controller for Azure Arc using Kubernetes tools

+description: Article describes how to upgrade an indirectly connected data controller for Azure Arc using Kubernetes tools

+# Upgrade an indirectly connected Azure Arc-enabled data controller using Kubernetes tools

+1. Create the service account for running upgrade.

+1. Upgrade the bootstrapper.

+1. Upgrade the data controller.

+Prior to beginning the upgrade of the data controller, you'll need:

+To upgrade the data controller using Kubernetes tools, you need to have the Kubernetes tools installed.

+You'll need to connect and authenticate to a Kubernetes cluster and have an existing Kubernetes context selected prior to beginning the upgrade of the data controller.

+### Create the service account for running upgrade

+ > [!IMPORTANT]

+ > Requires Kubernetes permissions for creating service account, role binding, cluster role, cluster role binding, and all the RBAC permissions being granted to the service account.

+Save a copy of [arcdata-deployer.yaml](https://raw.githubusercontent.com/microsoft/azure_arc/main/arc_data_services/arcdata-deployer.yaml), and replace the placeholder `{{NAMESPACE}}` in the file with the namespace of the data controller, for example: `arc`. Run the following command to create the deployer service account with the edited file.

+```console

+kubectl apply --namespace arc -f arcdata-deployer.yaml

+```

+### Upgrade the bootstrapper

+The following command creates a job for upgrading the bootstrapper and related Kubernetes objects.

+ > [!IMPORTANT]

+ > The yaml file in the following command defaults to mcr.microsoft.com/arcdata. Please save a copy of the yaml file and update it to a use a different registry/repository if necessary.

+```console

+kubectl apply --namespace arc -f https://raw.githubusercontent.com/microsoft/azure_arc/main/arc_data_services/upgrade/yaml/bootstrapper-upgrade-job.yaml

+```

+The following command patches the image tag to upgrade the data controller.

+```console

+kubectl apply --namespace arc -f https://raw.githubusercontent.com/microsoft/azure_arc/main/arc_data_services/upgrade/yaml/data-controller-upgrade.yaml

+## July 12, 2022

+|Component|Value|

+|--|--|

+|Container images tag |`v1.9.0_2022-07-12`|

+|CRD names and version|`datacontrollers.arcdata.microsoft.com`: v1beta1, v1 through v6<br/>`exporttasks.tasks.arcdata.microsoft.com`: v1beta1, v1, v2<br/>`kafkas.arcdata.microsoft.com`: v1beta1<br/>`monitors.arcdata.microsoft.com`: v1beta1, v1, v2<br/>`sqlmanagedinstances.sql.arcdata.microsoft.com`: v1beta1, v1 through v6<br/>`postgresqls.arcdata.microsoft.com`: v1beta1, v1beta2<br/>`sqlmanagedinstancerestoretasks.tasks.sql.arcdata.microsoft.com`: v1beta1, v1<br/>`failovergroups.sql.arcdata.microsoft.com`: v1beta1, v1beta2, v1<br/>`activedirectoryconnectors.arcdata.microsoft.com`: v1beta1, v1beta2<br/>|

+|Azure Resource Manager (ARM) API version|2022-03-01-preview (No change)|

+|`arcdata` Azure CLI extension version|1.4.3 ([Download](https://aka.ms/az-cli-arcdata-ext))|

+|Arc enabled Kubernetes helm chart extension version|1.2.20031002|

+|Arc Data extension for Azure Data Studio<br/>`arc`<br/>`azcli`|<br/>1.3.0 ([Download](https://azuredatastudioarcext.blob.core.windows.net/stage/arc-1.3.0.vsix))</br>1.3.0 ([Download](https://azuredatastudioarcext.blob.core.windows.net/stage/azcli-1.3.0.vsix))|

+> [!IMPORTANT]

+> Do not use an [instrumentation key](../azure-monitor/app/separate-resources.md#about-resources-and-instrumentation-keys) and a [connection string](../azure-monitor/app/sdk-connection-string.md#overview) simultaneously. Whichever was set last will take precedence.

+Azure Functions offers built-in integration with Azure Application Insights to monitor your function execution and traces written from your code. To learn more, see [Monitor Azure Functions](functions-monitoring.md). Azure Monitor also provides facilities for monitoring the health of the function app itself. To learn more, see [Monitoring with Azure Monitor](monitor-functions.md).

+ Title: Monitor executions in Azure Functions

+description: Learn how to use Azure Application Insights with Azure Functions to monitor function executions.

+# Monitor executions in Azure Functions

+[Azure Functions](functions-overview.md) offers built-in integration with [Azure Application Insights](../azure-monitor/app/app-insights-overview.md) to monitor functions executions. This article provides an overview of the monitoring capabilities provided by Azure for monitoring Azure Functions.

+You can also monitor the function app itself by using Azure Monitor. To learn more, see [Monitoring Azure Functions with Azure Monitor](monitor-functions.md).

+In addition to log-based telemetry data collected by Application Insights, you can also get data about how the function app is running from [Azure Monitor Metrics](../azure-monitor/essentials/data-platform-metrics.md). To learn more, see [Monitoring with Azure Monitor](monitor-functions.md).

+| 4.x (recommended) | PowerShell 7.2<br/>PowerShell 7 (recommended) | .NET 6 |

+ Title: Monitoring Azure Functions data reference

+description: Important reference material needed when you monitor Azure Functions

+# Monitoring Azure Functions data reference

+This reference applies to the use of Azure Monitor for monitoring function apps hosted in Azure Functions. See [Monitoring function app with Azure Monitor](monitor-functions.md) for details on using Azure Monitor to collect and analyze monitoring data from your function apps.

+See [Monitor Azure Functions](functions-monitoring.md) for details on using Application Insights to collect and analyze log data from individual functions in your function app.

+## Metrics

+This section lists all the automatically collected platform metrics collected for Azure Functions.

+### Azure Functions specific metrics

+There are two metrics specific to Functions that are of interest:

+| Metric | Description |

+| - | - |

+| **FunctionExecutionCount** | Function execution count indicates the number of times your function app has executed. This value correlates to the number of times a function runs in your app. |

+| **FunctionExecutionUnits** | Function execution units are a combination of execution time and your memory usage. Memory data isn't a metric currently available through Azure Monitor. However, if you want to optimize the memory usage of your app, can use the performance counter data collected by Application Insights. This metric isn't currently supported for Premium and Dedicated (App Service) plans running on Linux.|

+These metrics are used specifically when [estimating Consumption plan costs](functions-consumption-costs.md).

+### General App Service metrics

+Aside from Azure Functions specific metrics, the App Service platform implements more metrics, which you can use to monitor function apps. For the complete list, see [metrics available to App Service apps](../app-service/web-sites-monitor.md#understand-metrics) and [Monitoring App Service data reference](../app-service/monitor-app-service-reference.md#metrics).

+## Metric Dimensions

+For more information on what metric dimensions are, see [Multi-dimensional metrics](../azure-monitor/essentials/data-platform-metrics.md#multi-dimensional-metrics).

+Azure Functions doesn't have any metrics that contain dimensions.

+## Resource logs

+This section lists the types of resource logs you can collect for your function apps.

+| Log type | Description |

+|-|-|

+| FunctionAppLogs | Function app logs |

+For more information, see [Monitoring App Service data reference](../app-service/monitor-app-service-reference.md#resource-logs).

+For reference, see a list of [all resource logs category types supported in Azure Monitor](../azure-monitor/essentials/resource-logs-schema.md).

+## Azure Monitor Logs tables

+Azure Functions uses Kusto tables from Azure Monitor Logs. You can query the [FunctionAppLogs table](/azure/azure-monitor/reference/tables/functionapplogs) with Log Analytics. For more information, see the [Azure Monitor Log Table Reference](/azure/azure-monitor/reference/tables/tables-resourcetype#app-services).

+## Activity log

+The following table lists the operations related to Azure Functions that may be created in the Activity log.

+| Operation | Description |

+|:|:|

+|Microsoft.web/sites/functions/listkeys/action | Return the [keys for the function](functions-bindings-http-webhook-trigger.md#authorization-keys).|

+|Microsoft.Web/sites/host/listkeys/action | Return the [host keys for the function app](functions-bindings-http-webhook-trigger.md#authorization-keys).|

+|Microsoft.Web/sites/host/sync/action | [Sync triggers](functions-deployment-technologies.md#trigger-syncing) operation.|

+|Microsoft.Web/sites/start/action| Function app started. |

+|Microsoft.Web/sites/stop/action| Function app stopped.|

+|Microsoft.Web/sites/write| Change a function app setting, such as runtime version or enable remote debugging.|

+You may also find logged operations that relate to the underlying App Service behaviors. For a more complete list, see [Resource Provider Operations](/azure/role-based-access-control/resource-provider-operations#microsoftweb).

+For more information on the schema of Activity Log entries, see [Activity Log schema](../azure-monitor/essentials/activity-log-schema.md).

+## See Also

+* See [Monitoring Azure Functions](monitor-functions.md) for a description of monitoring Azure Functions.

+* See [Monitoring Azure resources with Azure Monitor](../azure-monitor/essentials/monitor-azure-resource.md) for details on monitoring Azure resources.

+ Title: Monitoring Azure Functions

+description: Start here to learn how to monitor function apps running in Azure Functions using Azure Monitor

+# Monitoring Azure Functions

+When you have critical applications and business processes relying on Azure resources, you want to monitor those resources for their availability, performance, and operation.

+This article describes the monitoring data generated by apps hosted in Azure Functions. Azure Functions uses [Azure Monitor](../azure-monitor/overview.md) to monitor the health of your function apps. If you're unfamiliar with the features of Azure Monitor common to all Azure services that use it, see [Monitoring Azure resources with Azure Monitor](../azure-monitor/essentials/monitor-azure-resource.md).

+Azure Functions uses Application Insights to collect and analyze log data from individual function executions in your function app. For more information, see [Monitor functions in Azure](functions-monitoring.md).

+## Monitoring data

+Azure Functions collects the same kinds of monitoring data as other Azure resources that are described in [Azure Monitor data collection](../azure-monitor/essentials/monitor-azure-resource.md#monitoring-data-from-azure-resources).

+See [Monitoring Azure Functions data reference](monitor-functions-reference.md) for detailed information on the metrics and logs metrics created by Azure Functions.

+## Collection and routing

+Platform metrics and the Activity log are collected and stored automatically, but can be routed to other locations by using a diagnostic setting.

+Resource Logs aren't collected and stored until you create a diagnostic setting and route them to one or more locations.

+See [Create diagnostic setting to collect platform logs and metrics in Azure](../azure-monitor/essentials/diagnostic-settings.md) for the detailed process for creating a diagnostic setting using the Azure portal, CLI, or PowerShell. When you create a diagnostic setting, you specify which categories of logs to collect. The categories for *Azure Functions* are listed in [Azure Functions monitoring data reference](monitor-functions-reference.md#resource-logs).

+The metrics and logs you can collect are discussed in the following sections.

+## Analyzing metrics

+You can analyze metrics for *Azure Functions* with metrics from other Azure services using metrics explorer by opening **Metrics** from the **Azure Monitor** menu. See [Getting started with Azure Metrics Explorer](../azure-monitor/essentials/metrics-getting-started.md) for details on using this tool.

+For a list of the platform metrics collected for Azure Functions, see [Monitoring *Azure Functions* data reference metrics](monitor-functions-reference.md#metrics).

+For reference, you can see a list of [all resource metrics supported in Azure Monitor](../azure-monitor/essentials/metrics-supported.md).

+The following examples use Monitor Metrics to help estimate the cost of running your function app on a Consumption plan. To learn more about estimating Consumption plan costs, see [Estimating Consumption plan costs](functions-consumption-costs.md).

+## Analyzing logs

+Data in Azure Monitor Logs is stored in tables where each table has its own set of unique properties.

+All resource logs in Azure Monitor have the same fields followed by service-specific fields. The common schema is outlined in [Azure Monitor resource log schema](../azure-monitor/essentials/resource-logs-schema.md).

+The [Activity log](../azure-monitor/essentials/activity-log.md) is a type of platform log in Azure that provides insight into subscription-level events. You can view it independently or route it to Azure Monitor Logs, where you can do much more complex queries using Log Analytics.

+For a list of the types of resource logs collected for Azure Functions, see [Monitoring Azure Functions data reference](monitor-functions-reference.md#resource-logs)

+For a list of the tables used by Azure Monitor Logs and queryable by Log Analytics, see [Monitoring Azure Functions data reference](monitor-functions-reference.md#azure-monitor-logs-tables)

+### Sample Kusto queries

+> [!IMPORTANT]

+> When you select **Logs** from the Azure Functions menu, Log Analytics is opened with the query scope set to the current resource. This means that log queries will only include data from that resource. If you want to run a query that includes data from other resources or data from other Azure services, select **Logs** from the **Azure Monitor** menu. See [Log query scope and time range in Azure Monitor Log Analytics](../azure-monitor/logs/scope.md) for details.

+Following are queries that you can use to help you monitor your Azure Function.

+The following sample query can help you monitor all your functions app logs:

+```Kusto

+FunctionAppLogs

+| project TimeGenerated, HostInstanceId, Message, _ResourceId

+| order by TimeGenerated desc

+```

+The following sample query can help you monitor a specific functions app's logs:

+```Kusto

+FunctionAppLogs

+| where FunctionName == "<Function name>"

+| order by TimeGenerated desc

+```

+The following sample query can help you monitor exceptions on a specific functions app's logs:

+```Kusto

+FunctionAppLogs

+| where ExceptionDetails != ""

+| where FunctionName == "<Function name>"

+| order by TimeGenerated desc

+```

+## Alerts

+Azure Monitor alerts proactively notify you when important conditions are found in your monitoring data. They allow you to identify and address issues in your system before your customers notice them. You can set alerts on [metrics](../azure-monitor/alerts/alerts-metric-overview.md), [logs](../azure-monitor/alerts/alerts-unified-log.md), and the [activity log](../azure-monitor/alerts/activity-log-alerts.md). Different types of alerts have benefits and drawbacks.

+If you're creating or running an application that run on Functions [Azure Monitor Application Insights](../azure-monitor/overview.md#application-insights) may offer other types of alerts.

+The following table lists common and recommended alert rules for Functions.

+| Alert type | Condition | Examples |

+|:|:|:|

+| Metric | Average connections| When number of connections exceed a set value|

+| Metric | HTTP 404| When HTTP 404 responses exceed a set value|

+| Metric | HTTP Server Errors| When HTTP 5xx errors exceed a set value|

+| Activity Log | Create or Update Web App | When app is created or updated|

+| Activity Log | Delete Web App | When app is deleted|

+| Activity Log | Restart Web App| When app is restarted|

+| Activity Log | Stop Web App| When app is stopped|

+## Next steps

+For more information about monitoring Azure Functions, see the following articles:

+* [Monitor Azure Functions](functions-monitoring.md) - details how-to monitor a function app.

+* [Monitoring Azure Functions data reference](monitor-functions-reference.md) - reference of the metrics, logs, and other important values created by your function app.

+* [Monitoring Azure resources with Azure Monitor](../azure-monitor/essentials/monitor-azure-resource.md) - details monitoring Azure resources.

+* [Analyze Azure Functions telemetry in Application Insights](analyze-telemetry-data.md) - details how-to view and query the data being collected from a function app.

+> Do not remove the legacy agents if being used by other [Azure solutions or services](./azure-monitor-agent-overview.md#supported-services-and-features). Use the migration helper to discover which solutions/services you use today.

+## Using AMA Migration Helper (preview)

+You can access the workbook [here](https://ms.portal.azure.com/#view/AppInsightsExtension/UsageNotebookBlade/ComponentId/Azure%20Monitor/ConfigurationId/community-Workbooks%2FAzure%20Monitor%20-%20Agents%2FAgent%20Migration%20Tracker/Type/workbook/WorkbookTemplateName/AMA%20Migration%20Helper), or find it on the [Azure portal (preview)](https://portal.azure.com/?feature.includePreviewTemplates=true) under **Monitor** > **Workbooks** > **Public Templates** > **Azure Monitor essentials** > **AMA Migration Helper**.

+|Log alerts| In log alerts, the alert is resolved at different rates based on the frequency of the alert:<ul> <li>**1 minute**: The alert condition isn't met for 10 minutes.</li> <li>**5-15 minutes**: The alert condition isn't met for three frequency periods.</li> <li>**15 minutes - 11 hours**: The alert condition isn't met for two frequency periods.</li> <li>**11 to 12 hours**: The alert condition isn't met for one frequency period.</li></ul>|

+ connectionString: "<Your Connection String>"

+- Enter any UI entry point, like the Web App **Diagnose and Solve Problems** tool, or

+## Enable web app in-guest change collection via Azure Portal

+From your resource's overview page in Azure portal, select **Diagnose and solve problems** the left menu. As you enter the Diagnose and Solve Problems tool, the **Microsoft.ChangeAnalysis** resource provider will automatically be registered.

+### Diagnose and solve problems tool for Web App

+You can view change data via the **Web App Down** and **Application Crashes** detectors. The graph summarizes:

+- The change types over time.

+- Details on those changes.

+By default, the graph displays changes from within the past 24 hours help with immediate problems.

+- Learn how [remediation access control works](../../governance/policy/how-to/remediate-resources.md#how-remediation-access-control-works).

+- Install the [Azure CLI](/cli/azure/install-azure-cli).

+ Title: Enable Profiler for Azure Service Fabric applications

+description: Profile live Azure Service Fabric apps with Application Insights

+# Enable Profiler for Azure Service Fabric applications

+Application Insights Profiler is included with Azure Diagnostics. You can install the Azure Diagnostics extension by using an Azure Resource Manager template for your Service Fabric cluster. Get a [template that installs Azure Diagnostics on a Service Fabric Cluster](https://github.com/Azure/azure-docs-json-samples/blob/master/application-insights/ServiceFabricCluster.json).

+In this article, you will:

+- Add the Application Insights Profiler property to your Azure Resource Manager template.

+- Deploy your Service Fabric cluster with the Application Insights Profiler instrumentation key.

+- Enable Application Insights on your Service Fabric application.

+- Redeploy your Service Fabric cluster to enable Profiler.

+## Prerequisites

+- Profiler supports .NET Framework, .NET Core, and .NET Core [LTS](https://dotnet.microsoft.com/platform/support/policy/dotnet-core) and newer applications.

+ - Verify you're using [.NET Framework 4.6.1](/dotnet/framework/migration-guide/how-to-determine-which-versions-are-installed) or later.

+ - Confirm that the deployed OS is `Windows Server 2012 R2` or later.

+- [An Azure Service Fabric managed cluster](../../service-fabric/quickstart-managed-cluster-portal.md).

+## Create deployment template

+1. In your Service Fabric managed cluster, navigate to where you've implemented the [Azure Resource Manager template](https://github.com/Azure/azure-docs-json-samples/blob/master/application-insights/ServiceFabricCluster.json).

+1. Locate the `WadCfg` tags in the [Azure Diagnostics](../agents/diagnostics-extension-overview.md) extension in the deployment template file.

+ ```json

+ "settings": {

+ "WadCfg": {

+ "SinksConfig": {

+ "Sink": [

+ {

+ "name": "MyApplicationInsightsProfilerSinkVMSS",

+ "ApplicationInsightsProfiler": "YOUR_APPLICATION_INSIGHTS_INSTRUMENTATION_KEY"

+ }

+ ]

+ },

+ },

+ }

+ ```

+ For information about adding the Diagnostics extension to your deployment template, see [Use monitoring and diagnostics with a Windows VM and Azure Resource Manager templates](../../virtual-machines/extensions/diagnostics-template.md).

+## Deploy your Service Fabric cluster

+After updating the `WadCfg` with your instrumentation key, deploy your Service Fabric cluster.

+

+Application Insights Profiler will be installed and enabled when the Azure Diagnostics extension is installed.

+## Enable Application Insights on your Service Fabric application

+For Profiler to collect profiles for your requests, your application must be tracking operations with Application Insights.

+- **For stateless APIs**, you can refer to instructions for [tracking requests for profiling](./profiler-trackrequests.md).

+- **For tracking custom operations in other kinds of apps**, see [track custom operations with Application Insights .NET SDK](../app/custom-operations-tracking.md).

+Redeploy your application once you've enabled Application Insights.

+## Generate traffic and view Profiler traces

+1. Launch an [availability test](../app/monitor-web-app-availability.md) to generate traffic to your application.

+1. Wait 10 to 15 minutes for traces to be sent to the Application Insights instance.

+1. View the [Profiler traces](./profiler-overview.md) via the Application Insights instance the Azure portal.

+For help with troubleshooting Profiler issues, see [Profiler troubleshooting](./profiler-troubleshooting.md).

+ Title: Azure Workbooks composite bar renderer

+description: Learn about all the Azure Workbooks composite bar renderer visualizations.

+With Azure Workbooks, data can be rendered by using the composite bar. This bar is made up of multiple bars.

+The following image shows the composite bar for database status. It shows how many servers are online, offline, and in a recovering state.

+

+The composite bar renderer is supported for grid, tile, and graph visualizations.

+## Add the composite bar renderer

+1. Switch the workbook to edit mode by selecting **Edit**.

+1. Select **Add** > **Add query**.

+1. Set **Data source** to `JSON` and set **Visualization** to `Grid`.

+1. Add the following JSON data:

+ ```json

+ [

+ {"sub":"X", "server": "A", "online": 20, "recovering": 3, "offline": 4, "total": 27},

+ {"sub":"X", "server": "B", "online": 15, "recovering": 8, "offline": 5, "total": 28},

+ {"sub":"Y", "server": "C", "online": 25, "recovering": 4, "offline": 5, "total": 34},

+ {"sub":"Y", "server": "D", "online": 18, "recovering": 6, "offline": 9, "total": 33}

+ ]

+ ```

+1. Run the query.

+1. Select **Column Settings** to open the settings pane.

+1. Under **Columns**, select `total`. For **Column renderer**, select `Composite Bar`.

+1. Under **Composite Bar Settings**, set the following settings:

+ | Column Name | Color |

+ |-|--|

+ | online | Green |

+ | recovering | Yellow |

+ | offline | Red (Bright) |

+1. For **Label**, enter `["online"] of ["total"] are healthy`.

+1. In the column settings for **online**, **offline**, and **recovering**, you can set **Column renderer** to `Hidden` (optional).

+1. Select the **Labels** tab and update the label for the total column as `Database Status` (optional).

+1. Select **Apply**.

+The composite bar settings will look like the following screenshot:

+

+The composite bar with the preceding settings:

+

+Select the column name and corresponding color to render the column in that color as part of a composite bar. You can insert, delete, and move rows up and down.

+The composite bar label is displayed at the top of the composite bar. You can use a mix of static text, columns, and parameters. If **Label** is empty, the value of the current columns is displayed as the label. In the previous example, if we left the **Label** field blank, the value of total columns would be displayed.

+Both the column name and parameter name are case sensitive. You can also make labels a link by selecting **Make this item a link** and then adding link settings.

+Aggregations are useful for Tree/Group By visualizations. The data for a column for the group row is decided by the aggregation set for that column. Three types of aggregations are applicable for composite bars: None, Sum, and Inherit.

+1. In **Tree/Group By Settings**, under **Tree type**, select **Group By**.

+1. Select the field you want to group by.

+ 

+The setting of **None** for aggregation means that no results are displayed for that column for the group rows.

+

+If aggregation is set as **Sum**, the column in the group row shows the composite bar by using the sum of the columns used to render it. The label will also use the sum of the columns referred to in it.

+In the following example, **online**, **offline**, and **recovering** all have max aggregation set to them and the aggregation for the total column is **Sum**.

+

+If aggregation is set as **Inherit**, the column in the group row shows the composite bar by using the aggregation set by users for the columns used to render it. The columns used in **Label** also use the aggregation set by the user. If the current column renderer is **Composite Bar** and is referred to in the label (like **total** in the preceding example), then **Sum** is used as the aggregation for that column.

+In the following example, **online**, **offline**, and **recovering** all have max aggregation set to them and the aggregation for total column is **Inherit**.

+

+For grid visualizations, the sorting of the rows for the column with the composite bar renderer works based on the value that's the sum of the columns used to render the composite bar computer dynamically. In the previous examples, the value used for sorting is the sum of the **online**, **recovering**, and **offline** columns for that particular row.

+## Tile visualizations

+To make a composite bar renderer for a tile visualization:

+1. Select **Add** > **Add query**.

+1. Change the data source to `JSON`. Enter the data from the [previous example](#add-the-composite-bar-renderer).

+1. Change **Visualization** to `Tiles`.

+1. Run the query.

+1. Select **Tile Settings**.

+1. Under **Tile fields**, select **Left**.

+1. Under **Field settings**, set the following settings:

+ 1. **Use column**: `server`

+ 1. **Column renderer**: `Text`

+1. Under **Tile fields**, select **Bottom**.

+1. Under **Field settings**, set the following settings:

+ 1. **Use column**: `total`

+ 1. **Column renderer**: `Composite Bar`

+ 1. Under **Composite Bar Settings**, set the following settings:

+ | Column Name | Color |

+ |-|--|

+ | online | Green |

+ | recovering | Yellow |

+ | offline | Red (Bright) |

+ 1. For **Label**, enter `["online"] of ["total"] are healthy`.

+1. Select **Apply**.

+

+The composite bar view for tiles with the preceding settings will look like this example:

+

+## Graph visualizations

+To make a composite bar renderer for a graph visualization (type Hive Clusters):

+1. Select **Add** > **Add query**.

+2. Change **Data source** to `JSON`. Enter the data from the [previous example](#add-the-composite-bar-renderer).

+1. Change **Visualization** to `Graphs`.

+1. Run the query.

+1. Select **Graph Settings**.

+1. Under **Node Format Settings**, select **Center Content**.

+1. Under **Field settings**, set the following settings:

+ 1. **Use column**: `total`

+ 1. **Column renderer**: `Composite Bar`

+ 1. Under **Composite Bar Settings**, set the following settings:

+ |Column Name | Color |

+ |-|--|

+ | online | Green |

+ | recovering | Yellow |

+ | offline | Red (Bright) |

+ 1. For **Label**, enter `["online"] of ["total"] are healthy`.

+1. Under **Layout Settings**, set the following settings:

+ 1. **Graph Type**: `Hive Clusters`

+ 1. **Node ID**: `server`

+ 1. **Group By Field**: `None`

+ 1. **Node Size**: `100`

+ 1. **Margin between hexagons**: `5`

+ 1. **Coloring Type**: `None`

+

+The composite bar view for a graph with the preceding settings will look like this example:

+

+[Get started with Azure Workbooks](workbooks-getting-started.md)

+ Title: Azure Workbooks graph visualizations

+description: Learn about all the Azure Workbooks graph visualizations.

+Azure Workbooks graph visualizations support visualizing arbitrary graphs based on data from logs to show the relationships between monitoring entities.

+The following graph shows data flowing in and out of a computer via various ports to and from external computers. It's colored by type, for example, computer vs. port vs. external IP. The edge sizes correspond to the amount of data flowing in between. The underlying data comes from KQL query targeting VM connections.

+[](./media/workbooks-graph-visualizations/graph.png#lightbox)

+## Add a graph

+1. Switch the workbook to edit mode by selecting **Edit**.

+1. Use the **Add query** link to add a log query control to the workbook.

+1. For **Query type**, select **Logs**. For **Resource type**, select, for example, **Application Insights**, and select the resources to target.

+1. Use the query editor to enter the KQL for your analysis.

+1. Set **Visualization** to **Graph**.

+1. Select **Graph Settings** to open the **Graph Settings** pane.

+1. In **Node Format Settings** at the top, set:

+ * **Top Content**

+ - **Use column**: `Name`

+ * **Column renderer**: `Text`

+ * **Center Content**

+ - **Use column**: `Calls`

+ * **Column renderer**: `Big Number`

+ * **Color palette**: `None`

+ * **Bottom Content**

+ - **Use column**: `Kind`

+ * **Column renderer**: `Text`

+1. In **Layout Settings** at the bottom, set:

+ * **Node ID**: `Id`

+ * **Source ID**: `SourceId`

+ * **Target ID**: `TargetId`

+ * **Edge Label**: `None`

+ * **Edge Size**: `Calls`

+ * **Node Size**: `None`

+ * **Coloring Type**: `Categorical`

+ * **Node Color Field**: `Kind`

+ * **Color palette**: `Pastel`

+1. Select **Save and Close** at the bottom of the pane.

+[](./media/workbooks-graph-visualizations/graph-settings.png#lightbox)

+| Setting | Description |

+| `Node ID` | Selects a column that provides the unique ID of nodes on the graph. The value of the column can be a string or a number. |

+| `Source ID` | Selects a column that provides the IDs of source nodes for edges on the graph. Values must map to a value in the `Node Id` column. |

+| `Target ID` | Selects a column that provides the IDs of target nodes for edges on the graph. Values must map to a value in the `Node Id` column. |

+| `Edge Size` | Selects a column that provides the metric on which the edge widths will be based. |

+| `Node Size` | Selects a column that provides the metric on which the node areas will be based. |

+| Coloring type | Description |

+| `Categorical` | Nodes are assigned colors based on the value or category from a column in the result set. In the preceding example, the coloring is based on the column `Kind` of the result set. Supported palettes are `Default`, `Pastel`, and `Cool tone`. |

+You can specify what content goes to the different parts of a node: top, left, center, right, and bottom. Graphs can use any renderers' workbook supports like text, big numbers, spark lines, and icons.

+## Field-based node coloring

+1. Switch the workbook to edit mode by selecting **Edit**.

+1. Use the **Add query** link to add a log query control to the workbook.

+1. For **Query type**, select **Logs**. For **Resource type**, select, for example, **Application Insights**, and select the resources to target.

+1. Use the query editor to enter the KQL for your analysis.

+1. Set **Visualization** to `Graph`.

+1. Select **Graph Settings** to open the **Graph Settings** pane.

+1. In **Node Format Settings** at the top, set:

+ * **Top Content**:

+ * **Use column**: `Name`

+ * **Column renderer**: `Text`

+ * **Center Content**:

+ * **Use column**: `Calls`

+ * **Column renderer**: `Big Number`

+ * **Color palette**: `None`

+ * **Bottom Content**:

+ * **Use column**: `Kind`

+ * **Column renderer**: `Text`

+1. In **Layout Settings** at the bottom, set:

+ * **Node ID**:`Id`

+ * **Source ID**: `SourceId`

+ * **Target ID**: `TargetId`

+ * **Edge Label**: `None`

+ * **Edge Size**: `Calls`

+ * **Node Size**: `Node`

+ * **Coloring Type**: `Field Based`

+ * **Node Color Field**: `Color`

+1. Select **Save and Close** at the bottom of the pane.

+[](./media/workbooks-graph-visualizations/graph-field-based.png#lightbox)

+* Graphs also support the composite bar renderer. To learn more, see [Composite bar renderer](workbooks-composite-bar.md).

+ Title: Azure Workbooks honeycomb visualizations

+description: Learn about Azure Workbooks honeycomb visualizations.

+# Honeycomb visualizations

+Azure Workbooks honeycomb visualizations provide high-density views of metrics or categories that can optionally be grouped as clusters. They're useful for visually identifying hotspots and drilling in further.

+The following image shows the CPU utilization of virtual machines across two subscriptions. Each cell represents a virtual machine. The color/label represents its average CPU utilization. Red cells are hot machines. The virtual machines are clustered by subscription.

+[](.\media\workbooks-honey-comb\cpu-example.png#lightbox)

+## Add a honeycomb

+1. Switch the workbook to edit mode by selecting **Edit**.

+1. Select **Add** > **Add query** to add a log query control to the workbook.

+1. For **Data source**, select **Logs**. For **Resource type**, select **Log Analytics**. For **Resource**, point to a workspace that has a virtual machine performance log.

+1. Use the query editor to enter the KQL for your analysis.

+ ```kusto

+ Perf

+ | where CounterName == 'Available MBytes'

+ | summarize CounterValue = avg(CounterValue) by Computer, _ResourceId

+ | extend ResourceGroup = extract(@'/subscriptions/.+/resourcegroups/(.+)/providers/microsoft.compute/virtualmachines/.+', 1, _ResourceId)

+ | extend ResourceGroup = iff(ResourceGroup == '', 'On-premises computers', ResourceGroup), Id = strcat(_ResourceId, '::', Computer)

+ ```

+1. Run the query.

+1. Set **Visualization** to `Graph`.

+1. Select **Graph Settings**.

+ 1. In **Node Format Settings** at the top, set:

+ 1. **Top Content**

+ - **Use column**: `Computer`

+ - **Column renderer**: `Text`

+ 1. **Center Content**

+ - **Use column**: `CounterValue`

+ - **Column renderer**: `Big Number`

+ - **Color palette**: `None`

+ - Select the **Custom number formatting** checkbox.

+ - **Units**: `Megabytes`

+ - **Maximum fractional digits**: `1`

+ 1. In **Layout Settings** at the bottom, set:

+ - **Graph Type**: `Hive Clusters`

+ - **Node ID**: `Id`

+ - **Group By Field**: `None`

+ - **Node Size**: 100

+ - **Margin between hexagons**: `5`

+ - **Coloring Type**: `Heatmap`

+ - **Node Color Field**: `CounterValue`

+ - **Color palette**: `Red to Green`

+ - **Minimum value**: `100`

+ - **Maximum value**: `2000`

+1. Select **Save and Close** at the bottom of the pane.

+[](.\media\workbooks-honey-comb\available-memory.png#lightbox)

+## Honeycomb layout settings

+| Setting | Description |

+| `Node ID` | Selects a column that provides the unique ID of nodes. The value of the column can be a string or a number. |

+| `Node Size` | Sets the size of the hexagonal cells. Use with the `Margin between hexagons` property to customize the look of the honeycomb chart. |

+| `Margin between hexagons` | Sets the gap between the hexagonal cells. Use with the `Node size` property to customize the look of the honeycomb chart. |

+| `Coloring Type` | Selects the scheme to use to color the node. |

+| `Node Color Field` | Selects a column that provides the metric on which the node areas will be based. |

+| Coloring type | Description |

+| `Categorical` | Nodes are assigned colors based on the value or category from a column in the result set. In the preceding example, the coloring is based on the column `Kind` of the result set. Supported palettes are `Default`, `Pastel`, and `Cool tone`. |

+| `Heatmap` | In this type, the cells are colored based on a metric column and a color palette. Color coding provides a simple way to highlight metrics spread across cells. |

+| `Thresholds` | In this type, cell colors are set by threshold rules, for example, _CPU > 90% => Red, 60% > CPU > 90% => Yellow, CPU < 60% => Green_. |

+You can specify what content goes to the different parts of a node: top, left, center, right, and bottom. You're free to use any of the renderers supported by workbooks like text, big numbers, spark lines, and icons.

+- Learn how to create a [composite bar renderer in workbooks](workbooks-composite-bar.md).

+ Title: Azure Workbooks map visualizations

+description: Learn about Azure Workbooks map visualizations.

+Azure Workbooks map visualizations aid in pinpointing issues in specific regions and showing high-level aggregated views of the monitoring data. Maps aggregate all the data mapped to each location, country, or region.

+The following screenshot shows the total transactions and end-to-end latency for different storage accounts. Here the size is determined by the total number of transactions. The color metrics below the map show the end-to-end latency.

+At first glance, the number of transactions in the **West US** region is small compared to the **East US** region. But the end-to-end latency for the **West US** region is higher than the **East US** region. This information provides initial insight that something is amiss for **West US**.

+

+## Add a map

+A map can be visualized if the underlying data or metrics have:

+- Latitude/longitude information.

+- Azure resource information.

+- Azure location information.

+- Country/region, name, or country/region code.

+### Use an Azure location

+1. Switch the workbook to edit mode by selecting **Edit**.

+1. Select **Add** > **Add query**.

+1. Change **Data Source** to **Azure Resource Graph**. Then select any subscription that has a storage account.

+1. Enter the following query for your analysis and then select **Run Query**.

+1. Set **Size** to `Large`.

+1. Set **Visualization** to `Map`.

+1. All the settings will be autopopulated. For custom settings, select **Map Settings** to open the settings pane.

+1. The following screenshot of the map visualization shows storage accounts for each Azure region for the selected subscription.

+

+### Use an Azure resource

+1. Switch the workbook to edit mode by selecting **Edit**.

+1. Select **Add** > **Add Metric**.

+1. Use a subscription that has storage accounts.

+1. Change **Resource Type** to `storage account`. In **Resource**, select multiple storage accounts.

+1. Select **Add Metric** and add a transaction metric:

+ 1. **Namespace**: `Account`

+ 1. **Metric**: `Transactions`

+ 1. **Aggregation**: `Sum`

+ 

+1. Select **Add Metric** and add the **Success E2E Latency** metric.

+ 1. **Namespace**: `Account`

+ 1. **Metric**: `Success E2E Latency`

+ 1. **Aggregation**: `Average`

+ 

+1. Set **Size** to `Large`.

+1. Set **Visualization** to `Map`.

+1. In **Map Settings**, set:

+ 1. **Location info using**: `Azure Resource`

+ 1. **Azure resource field**: `Name`

+ 1. **Size by**: `microsoft.storage/storageaccounts-Transaction-Transactions`

+ 1. **Aggregation for location**: `Sum of values`

+ 1. **Coloring type**: `Heatmap`

+ 1. **Color by**: `microsoft.storage/storageaccounts-Transaction-SuccessE2ELatency`

+ 1. **Aggregation for color**: `Sum of values`

+ 1. **Color palette**: `Green to Red`

+ 1. **Minimum value**: `0`

+ 1. **Metric value**: `microsoft.storage/storageaccounts-Transaction-SuccessE2ELatency`

+ 1. **Aggregate other metrics by**: `Sum of values`

+ 1. Select the **Custom formatting** checkbox.

+ 1. **Unit**: `Milliseconds`

+ 1. **Style**: `Decimal`

+ 1. **Maximum fractional digits**: `2`

+### Use country/region

+1. Switch the workbook to edit mode by selecting **Edit**.

+1. Select **Add** > **Add query**.

+1. Change **Data source** to `Log`.

+1. Select **Resource type** as `Application Insights`. Then select any Application Insights resource that has `pageViews` data.

+1. Use the query editor to enter the KQL for your analysis and select **Run Query**.

+1. Set **Size** to `Large`.

+1. Set **Visualization** to `Map`.

+1. All the settings will be autopopulated. For custom settings, select **Map Settings**.

+### Use latitude/location

+1. Switch the workbook to edit mode by selecting **Edit**.

+1. Select **Add** > **Add query**.

+1. Change **Data source** to `JSON`.

+1. Enter the JSON data in the query editor and select **Run Query**.

+1. Set **Size** values to `Large`.

+1. Set **Visualization** to `Map`.

+1. In **Map Settings** under **Metric Settings**, set **Metric Label** to `displayName`. Then select **Save and Close**.

+The following map visualization shows users for each latitude and longitude location with the selected label for metrics.

+

+Map settings include layout, color, and metrics.

+| Setting | Description |

+| `Location info using` | Select a way to get the location of items shown on the map. <br> **Latitude/Longitude**: Select this option if there are columns with latitude and longitude information. Each row with latitude and longitude data will be shown as a distinct item on the map. <br> **Azure location**: Select this option if there's a column that has Azure location (eastus, westeurope, centralindia) information. Specify that column and it will fetch the corresponding latitude and longitude for each Azure location. It will group the same location rows together based on the aggregation specified to show the locations on the map. <br> **Azure resource**: Select this option if there's a column that has Azure resource information such as an Azure Storage account and an Azure Cosmos DB account. Specify that column and it will fetch the corresponding latitude and longitude for each Azure resource. It will group the same location (Azure location) rows together based on the aggregation specified to show the locations on the map. <br> **Country/Region**: Select this option if there's a column that has country/region name/code (US, United States, IN, India, CN, China) information. Specify that column and it will fetch the corresponding latitude and longitude for each country/region/code. It will group rows together with the same Country-Region Code/Country-Region Name to show the locations on the map. Country Name and Country Code won't be grouped together as a single entity on the map.

+| `Latitude/Longitude` | These two options will be visible if the `Location Info` field value is **Latitude/Longitude**. Select the column that has latitude in the `Latitude` field and longitude in the `Longitude` field, respectively. |

+| `Azure location field` | This option will be visible if the `Location Info` field value is **Azure location**. Select the column that has the Azure location information. |

+| `Azure resource field` | This option will be visible if the `Location Info` field value is **Azure resource**. Select the column that has the Azure resource information. |

+| `Country/Region field` | This option will be visible if the `Location Info` field value is **Country/Region**. Select the column that has the country/region information. |

+| `Size by` | This option controls the size of the items shown on the map. Size depends on the value in the column specified by the user. Currently, the radius of the circle is directly proportional to the square root of the column's value. If **None** is selected, all the circles will show the default region size.|

+| `Aggregation for location` | This field specifies how to aggregate the `Size by` columns that have the same Azure Location/Azure Resource/Country-Region. |

+| `Minimum region size` | This field specifies the minimum radius of the item shown on the map. It's used when there's a significant difference between the `Size by` column's values, which causes smaller items to be hardly visible on the map. |

+| `Maximum region size` | This field specifies the maximum radius of the item shown on the map. It's used when the `Size by` column's values are extremely large and they're covering a huge area of the map.|

+| `Default region size` | This field specifies the default radius of the item shown on the map. The default radius is used when the `Size by` column is **None** or the value is 0.|

+| `Opacity of items on Map` | This field specifies the transparency of items shown on the map. Opacity of 1 means no transparency. Opacity of 0 means that items won't be visible on the map. If there are too many items on the map, opacity can be set to a low value so that all the overlapping items are visible.|

+| Coloring type | Description |

+| `Thresholds` | In this type, cell colors are set by threshold rules, for example, _CPU > 90% => Red, 60% > CPU > 90% => Yellow, CPU < 60% => Green_. <ul><li> `Color by`: The value of this column will be used by Thresholds/Heatmap logic.</li> <li>`Aggregation for color`: This field specifies how to aggregate the `Color by` columns that have the same Azure Location/Azure Resource/Country-Region. </li> <ul> |

+| `Heatmap` | In this type, the cells are colored based on the color palette and `Color by` field. This type will also have the same `Color by` and `Aggregation for color` options as thresholds. |

+| Setting | Description |

+| `Metric Label` | This option will be visible if the `Location Info` field value is **Latitude/Longitude**. With this feature, you can pick the label to show for the metrics shown below the map. |

+| `Metric Value` | This field specifies a metric value to be shown below the map. |

+| `Aggregate 'Others' metrics by` | This field specifies the aggregation used for an "Others" group if it's shown. |

+| `Custom formatting` | Use this field to set units, style, and formatting options for number values. This setting is the same as a [grid's custom formatting](../visualize/workbooks-grid-visualizations.md#custom-formatting).|

+Learn how to create [honeycomb visualizations in workbooks](../visualize/workbooks-honey-comb.md).

+description: Learn how to use Azure Workbooks templates.

+# Azure Workbooks templates

+Azure Workbooks templates are curated reports designed for flexible reuse by multiple users and teams. When you open a template, a transient workbook is created and populated with the content of the template. Workbooks are visible in green. Workbook templates are visible in purple.

+You can adjust the template-based workbook parameters and perform analysis without fear of breaking the future reporting experience for colleagues. If you open a template, make some adjustments, and save it, the template is saved as a workbook. This workbook appears in green. The original template is left untouched.

+The design and architecture of templates is also different from saved workbooks. Saving a workbook creates an associated Azure Resource Manager resource. But the transient workbook that's created when you open a template doesn't have a unique resource associated with it. The resources associated with a workbook affect who has access to that workbook. Learn more about [Azure Workbooks access control](workbooks-overview.md#access-control).

+ :::image type="content" source="./media/workbooks-overview/failure-analysis.png" alt-text="Screenshot that shows the Application Failure Analysis template." border="false" lightbox="./media/workbooks-overview/failure-analysis.png":::

+When you open the template, a temporary workbook is created that you can interact with. By default, the workbook opens in read mode. Read mode displays only the information for the intended analysis experience that was created by the original template author.

+You can adjust the subscription, targeted apps, and the time range of the data you want to display. After you make those selections, the grid of HTTP Requests is also interactive. Selecting an individual row changes the data rendered in the two charts at the bottom of the report.

+## Edit a template

+To understand how this workbook template is put together, switch to edit mode by selecting **Edit**.

+ :::image type="content" source="./media/workbooks-overview/edit.png" alt-text="Screenshot that shows the Edit button." border="false" :::

+**Edit** buttons on the right correspond with each individual aspect of your workbook.

+ :::image type="content" source="./media/workbooks-overview/edit-mode.png" alt-text="Screenshot that shows Edit buttons." border="false" lightbox="./media/workbooks-overview/edit-mode.png":::

+If you select the **Edit** button immediately under the grid of requested data, you can see that this part of the workbook consists of a Kusto query against data from an Application Insights resource.

+ :::image type="content" source="./media/workbooks-overview/kusto.png" alt-text="Screenshot that shows the underlying Kusto query." border="false" lightbox="./media/workbooks-overview/kusto.png":::

+Select the other **Edit** buttons on the right to see some of the core components that make up workbooks, like:

+- Markdown-based [text boxes](../visualize/workbooks-text-visualizations.md).

+- [Parameter selection](../visualize/workbooks-parameters.md) UI elements.

+- Other [chart/visualization types](workbooks-visualizations.md).

+Exploring the prebuilt templates in edit mode, modifying them to fit your needs, and saving your own custom workbook is a good way to start to learn about what's possible with Azure Workbooks.

+| Microsoft.EventGrid/domains | [listKeys](/rest/api/eventgrid/controlplane-version2022-06-15/domains/list-shared-access-keys) |

+| Microsoft.EventGrid/topics | [listKeys](/rest/api/eventgrid/controlplane-version2022-06-15/topics/list-shared-access-keys) |

+The resourceGroup and subscription properties are only allowed on modules. These properties are not allowed on individual resources. Use modules if you want to deploy an extension resource with the scope set to a resource in a different resource group.

+The following example shows how to apply a lock on a storage account that resides in a different resource group.

+* **main.bicep:**

+ ```bicep

+ param resourceGroup2Name string

+ param storageAccountName string

+ module applyStoreLock './storageLock.bicep' = {

+ name: 'addStorageLock'

+ scope: resourceGroup(resourceGroup2Name)

+ params: {

+ storageAccountName: storageAccountName

+ }

+ }

+ ```

+* **storageLock.bicep:**

+ ```bicep

+ param storageAccountName string

+ resource storage 'Microsoft.Storage/storageAccounts@2021-09-01' existing = {

+ name: storageAccountName

+ }

+ resource storeLock 'Microsoft.Authorization/locks@2017-04-01' = {

+ scope: storage

+ name: 'storeLock'

+ properties: {

+ level: 'CanNotDelete'

+ notes: 'Storage account should not be deleted.'

+ }

+ }

+ ```

+Don't use variables for the API version.

+ * [How to connect and sign on to an Azure virtual machine running Windows](../../virtual-machines/windows/connect-rdp.md)

+ * [Setting up WinRM access for Virtual Machines in Azure Resource Manager](../../virtual-machines/windows/connect-winrm.md)

+* For recommendations about how to build templates that work in all Azure cloud environments, see [Develop ARM templates for cloud consistency](./template-cloud-consistency.md).

+The resourceGroup and subscription properties are only allowed on nested or linked deployments. These properties are not allowed on individual resources. Use nested or linked deployments if you want to deploy an extension resource with the scope set to a resource in a different resource group.

+| Microsoft.EventGrid/domains | [listKeys](/rest/api/eventgrid/controlplane-version2022-06-15/domains/list-shared-access-keys) |

+| Microsoft.EventGrid/topics | [listKeys](/rest/api/eventgrid/controlplane-version2022-06-15/topics/list-shared-access-keys) |

+ Title: Deploy Azure Video Indexer by using an ARM template

+description: Learn how to create an Azure Video Indexer account by using an Azure Resource Manager (ARM) template.

+# Tutorial: Deploy Azure Video Indexer by using an ARM template

+In this tutorial, you'll create an Azure Video Indexer account by using the Azure Resource Manager template (ARM template, which is in preview). The resource will be deployed to your subscription and will create the Azure Video Indexer resource based on parameters defined in the *avam.template* file.

+> This sample is *not* for connecting an existing Azure Video Indexer classic account to a Resource Manager-based Azure Video Indexer account.

+>

+> For full documentation on the Azure Video Indexer API, visit the [developer portal](https://aka.ms/avam-dev-portal). For the latest API version for *Microsoft.VideoIndexer*, see the [template reference](/azure/templates/microsoft.videoindexer/accounts?tabs=bicep).

+You need an Azure Media Services account. You can create one for free through [Create a Media Services account](/azure/media-services/latest/account-create-how-to).

+### Option 1: Select the button for deploying to Azure, and fill in the missing parameters

+### Option 2: Deploy by using a PowerShell script

+1. Open the [template file](https://github.com/Azure-Samples/media-services-video-indexer/blob/master/ARM-Quick-Start/avam.template.json) and inspect its contents.

+2. Fill in the required parameters.

+3. Run the following PowerShell commands:

+ * Create a new resource group on the same location as your Azure Video Indexer account by using the [New-AzResourceGroup](/powershell/module/az.resources/new-azresourcegroup) cmdlet.

+ ```powershell

+ New-AzResourceGroup -Name myResourceGroup -Location eastus

+ ```

+ * Deploy the template to the resource group by using the [New-AzResourceGroupDeployment](/powershell/module/az.resources/new-azresourcegroupdeployment) cmdlet.

+ ```powershell

+ New-AzResourceGroupDeployment -ResourceGroupName myResourceGroup -TemplateFile ./avam.template.json

+ ```

+> If you want to work with Bicep format, see [Deploy by using Bicep](./deploy-with-bicep.md).

+* Description: The name of the new Azure Video Indexer account.

+* Required: true

+* Description: The Azure location where the Azure Video Indexer account should be created.

+> You need to deploy your Azure Video Indexer account in the same location (region) as the associated Azure Media Services resource.

+* Description: The resource ID of the Azure Media Services resource.

+* Description: The resource ID of the managed identity that's used to grant access between Azure Media Services resource and the Azure Video Indexer account.

+* Description: The array of objects that represents custom user tags on the Azure Video Indexer account.

+* Required: false

+* [Azure Video Indexer documentation](./index.yml)

+* [Azure Video Indexer developer portal](https://api-portal.videoindexer.ai/)

+After you complete this tutorial, head to other Azure Video Indexer samples described in [README.md](https://github.com/Azure-Samples/media-services-video-indexer/blob/master/README.md).

+* [Deploy resources with ARM templates](../azure-resource-manager/templates/deploy-powershell.md)

+* [Deploy resources with Bicep and the Azure CLI](../azure-resource-manager/bicep/deploy-cli.md)

+Connect a [classic paid Azure Video Indexer account to a Resource Manager-based account](connect-classic-account-to-arm.md).

+ Title: Auto scale Azure Web PubSub service

+description: Learn how to autoscale Azure WebPubSub service.

+# Automatically scale units of an Azure WebPubSub service

+> [!IMPORTANT]

+> Autoscaling is only available in Azure WebPubSub service Premium tier.

+Azure WebPubSub service Premium tier supports an *autoscale* feature, which is an implementation of [Azure Monitor autoscale](../azure-monitor/autoscale/autoscale-overview.md). Autoscale allows you to automatically scale the unit count for your WebPubSub service to match the actual load on the service. Autoscale can help you optimize performance and cost for your application.

+Azure WebPubSub adds its own [service metrics](concept-metrics.md). However, most of the user interface is shared and common to other [Azure services that support autoscaling](../azure-monitor/autoscale/autoscale-overview.md#supported-services-for-autoscale). If you're new to the subject of Azure Monitor Metrics, review [Azure Monitor Metrics aggregation and display explained](../azure-monitor/essentials/metrics-aggregation-explained.md) before digging into WebPubSub service Metrics.

+## Understanding autoscale in WebPubSub service

+Autoscale allows you to set conditions that will dynamically change the units allocated to WebPubSub service while the service is running. Autoscale conditions are based on metrics, such as **Server Load**. Autoscale can also be configured to run on a schedule, such as every day between certain hours.

+For example, you can implement the following scaling scenarios using autoscale.

+- Increase units when the **Connection Quota Utilization** above 70%.

+- Decrease units when the **Server Load** is below 20%.

+- Create a schedule to add more units during peak hours and reduce units during off hours.

+Multiple factors affect the performance of WebPubSub service. No one metric provides a complete view of system performance. For example, if you're sending a large number of messages you might need to scale out even though the connection quota is relatively low. The combination of both **Connection Quota Utilization** and **Server Load** gives an indication of overall system load. The following guidelines apply.

+- Scale out if the connection count is over 80-90%. Scaling out before your connection count is exhausted ensures that you'll have sufficient buffer to accept new connections before scale-out takes effect.

+- Scale out if the **Server Load** is over 80-90%. Scaling early ensures that the service has enough capacity to maintain performance during the scale-out operation.

+The autoscale operation usually takes effect 3-5 minutes after it's triggered. It's important not to change the units too often. A good rule of thumb is to allow 30 minutes from the previous autoscale before performing another autoscale operation. In some cases, you might need to experiment to find the optimal autoscale interval.

+## Custom autoscale settings

+Open the autoscale settings page:

+1. Go to the [Azure portal](https://portal.azure.com).

+1. Open the **WebPubSub** service page.

+1. From the menu on the left, under **Settings** choose **Scale out**.

+1. Select the **Configure** tab. If you have a Premium tier WebPubSub instance, you'll see two options for **Choose how to scale your resource**:

+ - **Manual scale**, which lets you manually change the number of units.

+ - **Custom autoscale**, which lets you create autoscale conditions based on metrics and/or a time schedule.

+1. Choose **Custom autoscale**. Use this page to manage the autoscale conditions for your Azure WebPubSub service.

+### Default scale condition

+When you open custom autoscale settings for the first time, you'll see the **Default** scale condition already created for you. This scale condition is executed when none of the other scale conditions match the criteria set for them. You can't delete the **Default** condition, but you can rename it, change the rules, and change the action taken by autoscale.

+You can't set the default condition to autoscale on a specific days or date range. The default condition only supports scaling to a unit range. To scale according to a schedule, you'll need to add a new scale condition.

+Autoscale doesn't take effect until you save the default condition for the first time after selecting **Custom autoscale**.

+## Add or change a scale condition

+There are two options for how to scale your Azure WebPubSub resource:

+- **Scale based on a metric** - Scale within unit limits based on a dynamic metric. One or more scale rules are defined to set the criteria used to evaluate the metric.

+- **Scale to specific units** - Scale to a specific number of units based on a date range or recurring schedule.

+### Scale based on a metric

+The following procedure shows you how to add a condition to increase units (scale out) when the Connection Quota Utilization is greater than 70% and decrease units (scale in) when the Connection Quota Utilization is less than 20%. Increments or decrements are done between available units.

+1. On the **Scale out** page, select **Custom autoscale** for the **Choose how to scale your resource** option.

+1. Select **Scale based on a metric** for **Scale mode**.

+1. Select **+ Add a rule**.

+ :::image type="content" source="./media/howto-scale-autoscale/default-autoscale.png" alt-text="Screenshot of custom rule based on a metric.":::

+1. On the **Scale rule** page, follow these steps:

+ 1. Select a metric from the **Metric name** drop-down list. In this example, it's **Connection Quota Utilization**.

+ 1. Select an operator and threshold values. In this example, they're **Greater than** and **70** for **Metric threshold to trigger scale action**.

+ 1. Select an **operation** in the **Action** section. In this example, it's set to **Increase**.

+ 1. Then, select **Add**

+ :::image type="content" source="./media/howto-scale-autoscale/default-scale-out.png" alt-text="Screenshot of default autoscale rule screen.":::

+1. Select **+ Add a rule** again, and follow these steps on the **Scale rule** page:

+ 1. Select a metric from the **Metric name** drop-down list. In this example, it's **Connection Quota Utilization**.

+ 1. Select an operator and threshold values. In this example, they're **Less than** and **20** for **Metric threshold to trigger scale action**.

+ 1. Select an **operation** in the **Action** section. In this example, it's set to **Decrease**.

+ 1. Then, select **Add**

+ :::image type="content" source="./media/howto-scale-autoscale/default-scale-in.png" alt-text="Screenshot Connection Quota Utilization scale rule.":::

+1. Set the **minimum**, **maximum**, and **default** number of units.

+1. Select **Save** on the toolbar to save the autoscale setting.

+### Scale to specific units

+Follow these steps to configure the rule to scale to a specific unit range.

+1. On the **Scale out** page, select **Custom autoscale** for the **Choose how to scale your resource** option.

+1. Select **Scale to a specific units** for **Scale mode**.

+1. For **Units**, select the number of default units.

+ :::image type="content" source="./media/howto-scale-autoscale/default-specific-units.png" alt-text="Screenshot of scale rule criteria.":::

+## Add more conditions

+The previous section showed you how to add a default condition for the autoscale setting. This section shows you how to add more conditions to the autoscale setting.

+1. On the **Scale out** page, select **Custom autoscale** for the **Choose how to scale your resource** option.

+1. Select **Add a scale condition** under the **Default** block.

+ :::image type="content" source="./media/howto-scale-autoscale/additional-add-condition.png" alt-text="Screenshot of custom scale rule screen.":::

+1. Confirm that the **Scale based on a metric** option is selected.

+1. Select **+ Add a rule** to add a rule to increase units when the **Connection Quota Utilization** goes above 70%. Follow steps from the [default condition](#default-scale-condition) section.

+1. Set the **minimum** and **maximum** and **default** number of units.

+1. You can also set a **schedule** on a custom condition (but not on the default condition). You can either specify start and end dates for the condition (or) select specific days (Monday, Tuesday, and so on.) of a week.

+ 1. If you select **Specify start/end dates**, select the **Timezone**, **Start date and time** and **End date and time** (as shown in the following image) for the condition to be in effect.

+ 1. If you select **Repeat specific days**, select the days of the week, timezone, start time, and end time when the condition should apply.

+## Next steps

+For more information about managing autoscale from the Azure CLI, see [**az monitor autoscale**](/cli/azure/monitor/autoscale?view=azure-cli-latest&preserve-view=true).

+For Docker support on a custom image, install [Docker Community Edition (CE)](https://www.docker.com/community-edition) or [Docker Enterprise Edition (EE)](https://docker-docs.netlify.app/ee/).

+# Quickstart: Detect Personally Identifiable Information (PII)

+> Authoring functionality is available via the REST API and [Authoring SDK (preview)](/dotnet/api/overview/azure/ai.language.questionanswering-readme-pre). This article provides examples of using the REST API with cURL. For full documentation of all parameters and functionality available consult the [REST API reference content](/rest/api/cognitiveservices/questionanswering/question-answering-projects).

+# Quickstart: Using Text Analytics for health client library and REST API

+To delete a deployment, you can use the [Azure CLI](/cli/azure/cognitiveservices/account/deployment?view=azure-cli-latest&preserve-view=true#az-cognitiveservices-account-deployment-delete), Azure OpenAI Studio or [REST APIs](../reference.md#delete-a-deployment). here's an example of how to delete your deployment with the Azure CLI:

+- Azure CLI. [Installation Guide](/cli/azure/install-azure-cli)

+ scopes: [

+ "https://auth.msft.communication.azure.com/Teams.ManageCalls",

+ "https://auth.msft.communication.azure.com/Teams.ManageChats"

+ ],

+ scopes: [

+ "https://auth.msft.communication.azure.com/Teams.ManageCalls",

+ "https://auth.msft.communication.azure.com/Teams.ManageChats"

+ ],

+1. Authenticate Alice using Azure Active Directory: Alice is authenticated using a standard OAuth flow with *Microsoft Authentication Library (MSAL)*. If authentication is successful, the client application receives an Azure AD access token, with a value of 'A1' and an Object ID of an Azure AD user with a value of 'A2'. Tokens are outlined later in this article. Authentication from the developer perspective is explored in this [quickstart](../../quickstarts/manage-teams-identity.md).

+1. Get an access token for Alice: The customized Teams application performs control plane logic, using artifacts 'A1', 'A2' and 'A3'. This produces Azure Communication Services access token 'D' and gives Alice access. This access token can also be used for data plane actions in Azure Communication Services, like Calling.

+- Artifact A1

+ - Permissions: _`https://auth.msft.communication.azure.com/Teams.ManageCalls`_, _`https://auth.msft.communication.azure.com/Teams.ManageChats`_

+- Artifact A2

+ - Type: Object ID of an Azure AD user

+ - Azure AD application ID: Fabrikam's _`Azure AD application ID`_

+- Artifact A3

+ - Type: Azure AD application ID

+ - Azure AD application ID: Fabrikam's _`Azure AD application ID`_

+1. Authenticate Alice using the Fabrikam application: Alice is authenticated through Fabrikam's customized Teams application. A standard OAuth flow with Microsoft Authentication Library (MSAL) is used. If authentication is successful, the client application, the Contoso app in this case, receives an Azure AD access token with a value of 'A1' and an Object ID of an Azure AD user with a value of 'A2'. Token details are outlined below. Authentication from the developer perspective is explored in this [quickstart](../../quickstarts/manage-teams-identity.md).

+1. Get an access token for Alice: The Contoso application performs control plane logic, using artifacts 'A1', 'A2' and 'A3'. This generates Azure Communication Services access token 'D' for Alice within the Contoso application. This access token can be used for data plane actions in Azure Communication Services, like Calling.

+- Artifact A1

+ - Permission: _`https://auth.msft.communication.azure.com/Teams.ManageCalls`_, _`https://auth.msft.communication.azure.com/Teams.ManageChats`_

+- Artifact A2

+ - Type: Object ID of an Azure AD user

+ - Azure AD application ID: Fabrikam's _`Azure AD application ID`_

+- Artifact A3

+ - Type: Azure AD application ID

+ - Azure AD application ID: Contoso's _`Azure AD application ID`_

+| _`https://auth.msft.communication.azure.com/Teams.ManageChats`_ | Manage chats in Teams | Create, read, update, and delete 1:1 or group chat threads on behalf of the signed-in user. Read, send, update, and delete messages in chat threads on behalf of the signed-in user. | No | No |

+When troubleshooting issues with the Call Automation SDK, like call recording and call management problems, you'll need to collect the Server Call ID. This ID can be collected using the ```getServerCallId``` method.

+## Verification of Teams license eligibility to use Azure Communication Services support for Teams users

+There are two ways to verify your Teams License eligibility to use Azure Communication Services support for Teams users:

+* **Verification via Teams web client**

+* **Checking your current Teams license via Microsoft Graph API**

+#### Verification via Teams web client

+To verify your Teams License eligibility via Teams web client, follow the steps listed below:

+1. Open your browser and navigate to [Teams web client](https://teams.microsoft.com/).

+1. Sign in with credentials that have a valid Teams license.

+1. If the authentication is successful and you remain in the https://teams.microsoft.com/ domain, then your Teams License is eligible. If authentication fails or you're redirected to the https://www.teams.live.com domain, then your Teams License isn't eligible to use Azure Communication Services support for Teams users.

+#### Checking your current Teams license via Microsoft Graph API

+You can find your current Teams license using [licenseDetails](https://docs.microsoft.com/graph/api/resources/licensedetails) Microsoft Graph API that returns licenses assigned to a user. Follow the steps below to use the Graph Explorer tool to view licenses assigned to a user:

+1. Open your browser and navigate to [Graph Explorer](https://developer.microsoft.com/graph/graph-explorer)

+1. Sign in to Graph Explorer using the credentials.

+ :::image type="content" source="./media/troubleshooting/graph-explorer-sign-in.png" alt-text="Screenshot of how to sign in to Graph Explorer.":::

+1. In the query box, enter the following API and click **Run Query** :

+ <!-- { "blockType": "request" } -->

+ ```http

+ https://graph.microsoft.com/v1.0/me/licenseDetails

+ ```

+ :::image type="content" source="./media/troubleshooting/graph-explorer-query-box.png" alt-text="Screenshot of how to enter API in Graph Explorer.":::

+ Or you can query for a particular user by providing the user ID using the following API:

+ <!-- { "blockType": "request" } -->

+ ```http

+ https://graph.microsoft.com/v1.0/users/{id}/licenseDetails

+ ```

+1. The **Response preview** pane displays output as follows:

+ Note that the response object shown here might be shortened for readability.

+ <!-- {

+ "blockType": "response",

+ "truncated": true,

+ "isCollection": true

+ } -->

+ ```http

+ {

+ "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('071cc716-8147-4397-a5ba-b2105951cc0b')/assignedLicenses",

+ "value": [

+ {

+ "skuId": "b05e124f-c7cc-45a0-a6aa-8cf78c946968",

+ "servicePlans":[

+ {

+ "servicePlanId":"57ff2da0-773e-42df-b2af-ffb7a2317929",

+ "servicePlanName":"TEAMS1",

+ "provisioningStatus":"Success",

+ "appliesTo":"User"

+ }

+ ]

+ }

+ ]

+ }

+ ```

+1. Find license detail where property `servicePlanName` has one of the values in the [Eligible Teams Licenses table](../quickstarts/eligible-teams-licenses.md#eligible-teams-licenses)

+| 429 | Too many requests | Ensure that your client-side application handles this scenario in a user-friendly manner. If the error persists, please file a support request. |

+| 4000 | Message is rejected due to fraud detection | Ensure you aren't exceeding the maximum number of messages allowed for your number|

+| 4003 | Message failed to deliver due to unsupported destination| Check if the destination you're trying to send to is supported |

+| 4004 | Message failed to deliver since Destination/To number doesn't exist| Ensure the To number you're sending to is valid |

+| 4006 | The Destination/To number isn't reachable| Try resending the message at a later time |

+| 4008 | You've exceeded the maximum number of messages allowed for your profile| Ensure you aren't exceeding the maximum number of messages allowed for your number or use queues to batch the messages |

+| 5000 | Message failed to deliver. Please reach out Microsoft support team for more details| File a support request through the Azure portal |

+| 5002 | Message Delivery Timeout| Try resending the message |

+| 9999 | Message failed to deliver due to unknown error/failure| Try resending the message |

+ Title: Teams license requirements to use Azure Communication Services support for Teams users

+description: This article describes Teams License requirements and how users can find their current Teams license.

+# Teams License requirements to use Azure Communication Services support for Teams users

+To use Azure Communication Services support for Teams users, you need an Azure Active Directory instance with users that have a valid Teams license. Furthermore, license must be assigned to the administrators or relevant users. This article describes the Teams license requirements to use Azure Communication Services support for Teams users.

+### Eligible Teams licenses

+Ensure that your Azure Active Directory users have at least one of the following eligible Teams licenses:

+| Service Plan (friendly names) | Service Plan ID |

+|: |: |

+| TEAMS1 | 57ff2da0-773e-42df-b2af-ffb7a2317929 |

+| TEAMS_FREE | 4fa4026d-ce74-4962-a151-8e96d57ea8e4 |

+| TEAMS_GOV | 304767db-7d23-49e8-a945-4a7eb65f9f28 |

+| TEAMS_GCCHIGH | 495922d5-f138-498b-8967-4acdcdfb2a74 |

+| TEAMS_AR_GCCHIGH | 9953b155-8aef-4c56-92f3-72b0487fce41 |

+| TEAMS_DOD | ec0dd2de-a877-4059-a9b8-5838b5629b2a |

+| TEAMS_AR_DOD | fd500458-c24c-478e-856c-a6067a8376cd |

+For more information, see [Azure AD Product names and service plan identifiers](../../active-directory/enterprise-users/licensing-service-plan-reference.md).

+### How to find current Teams license

+You can find your current Teams license using [licenseDetails](https://docs.microsoft.com/graph/api/resources/licensedetails) Microsoft Graph API that returns licenses assigned to a user.

+For more information on verification for eligibility, see [Verification of Teams license eligibility](../concepts/troubleshooting-info.md#verification-of-teams-license-eligibility-to-use-azure-communication-services-support-for-teams-users).

+## Next steps

+The following articles might be of interest to you:

+- Try [quickstart for authentication of Teams users](./manage-teams-identity.md).

+- Try [quickstart for calling to a Teams user](./voice-video-calling/get-started-with-voice-video-calling-custom-teams-client.md).

+- Learn more about [Custom Teams endpoint](../concepts/teams-endpoint.md)

+- Learn more about [Teams interoperability](../concepts/teams-interop.md)

+- An Azure Active Directory instance with users that have a Teams license. For more information, see [Teams License requirements](./eligible-teams-licenses.md).

+1. The Contoso Administrator adds API permissions to `Teams.ManageCalls` and `Teams.ManageChats` from Communication Services.

+1. The Fabrikam Administrator grants Communication Services `Teams.ManageCalls` and `Teams.ManageChats` permissions to the Contoso application. This step is required if only Fabrikam Administrator can grant access to the application with the `Teams.ManageCalls` and `Teams.ManageChats` permissions.

+Users must be authenticated against Azure AD applications with the Azure Communication Service Teams.ManageCalls and Teams.ManageChats permissions. If you don't have an existing application that you want to use for this quickstart, you can create a new application registration.

+The application must declare Teams.ManageCalls and Teams.ManageChats permissions to have access to Teams calling capabilities in the Tenant. Teams user would be requesting an Azure AD user token with this permission for token exchange.

+1. Navigate to your Azure AD app in the Azure portal and select **API permissions**

+1. Select the permissions **Teams.ManageCalls** and **Teams.ManageCalls**, then select **Add permissions**

+

+Azure Active Directory tenant can be configured, to require Azure AD administrator consent for the Teams.ManageCalls and Teams.ManageChats permissions of the application. In such a case, the Azure AD Administrator must grant permissions to the Contoso application for Communication Services Teams.ManageCalls and Teams.ManageChats. The Fabrikam Azure AD Administrator provides consent via a unique URL.

+The following roles can provide consent on behalf of a company:

+- Global admin

+- Application admin

+- Cloud application admin

+If you want to check roles in Azure portal, see [List Azure role assignments](../../role-based-access-control/role-assignments-list-portal.md).

+You can see that the status of the Communication Services Teams.ManageCalls and Teams.ManageChats permissions are *Granted for {Directory_name}*.

+1. The Contoso developer configures the Microsoft Authentication Library (MSAL) to authenticate the user for the application that was created earlier by the Administrator for Communication Services Teams.ManageCalls and Teams.ManageChats permissions.

+1. The Contoso *client application* uses the MSAL to authenticate the user against the Fabrikam Azure AD tenant for the Contoso application with Communication Services Teams.ManageCalls and Teams.ManageChats permissions.

+vm_series='DCASv5'

+az vm list-skus \

+ --size dc \

+ --query "[?family=='standard${vm_series}Family'].{name:name,locations:locationInfo[0].location,AZ_a:locationInfo[0].zones[0],AZ_b:locationInfo[0].zones[1],AZ_c:locationInfo[0].zones[2]}" \

+ --all \

+vm_series='DCASv5'

+az vm list-skus \

+ --size dc \

+ --query "[?family=='standard${vm_series}Family']"

+### [.NET SDK V3](#tab/dotnetv3)

+### [Java SDK V4](#tab/javav4)

+```java

+static class Family {

+ public String id;

+ public String firstName;

+ public String lastName;

+ public String _partitionKey;

+ public Family(String id, String firstName, String lastName, String _partitionKey) {

+ this.id = id;

+ this.firstName = firstName;

+ this.lastName = lastName;

+ this._partitionKey = _partitionKey;

+ }

+}

+...

+CosmosDatabase cosmosDatabase = cosmosClient.getDatabase("testdb");

+CosmosContainer cosmosContainer = cosmosDatabase.getContainer("testcontainer");

+// Create single item

+Family family = new Family("id-1", "John", "Doe", "Doe");

+cosmosContainer.createItem(family, new PartitionKey(family._partitionKey), new CosmosItemRequestOptions());

+// Create items through bulk operations

+family = new Family("id-2", "Jane", "Doe", "Doe");

+CosmosItemOperation createItemOperation = CosmosBulkOperations.getCreateItemOperation(family,

+ new PartitionKey(family._partitionKey));

+cosmosContainer.executeBulkOperations(Collections.singletonList(createItemOperation));

+```

+For the complete sample, see the [Java samples][2] GitHub repository.

+

+## Migrate the documents

+While the container definition is enhanced with a partition key property, the documents within the container arenΓÇÖt auto migrated. Which means the system partition key property `/_partitionKey` path is not automatically added to the existing documents. You need to repartition the existing documents by reading the documents that were created without a partition key and rewrite them back with `_partitionKey` property in the documents.

+## Access documents that don't have a partition key

+Applications can access the existing documents that donΓÇÖt have a partition key by using the special system property called "PartitionKey.None", this is the value of the non-migrated documents. You can use this property in all the CRUD and query operations. The following example shows a sample to read a single Document from the NonePartitionKey.

+```java

+CosmosItemResponse<JsonNode> cosmosItemResponse =

+ cosmosContainer.readItem("itemId", PartitionKey.NONE, JsonNode.class);

+```

+For the complete sample on how to repartition the documents, see the [Java samples][2] GitHub repository.

+[1]: https://github.com/Azure/azure-cosmos-dotnet-v3/tree/master/Microsoft.Azure.Cosmos.Samples/Usage/NonPartitionContainerMigration

+[2]: https://github.com/Azure-Samples/azure-cosmos-java-sql-api-samples/tree/main/src/main/java/com/azure/cosmos/examples/nonpartitioncontainercrud

+ Title: Quickstart - Azure Cosmos DB Table API for .NET

+description: Learn how to build a .NET app to manage Azure Cosmos DB Table API resources in this quickstart.

+ms.devlang: dotnet

+# Quickstart: Azure Cosmos DB Table API for .NET

+This quickstart shows how to get started with the Azure Cosmos DB Table API from a .NET application. The Cosmos DB Table API is a schemaless data store allowing applications to store structured NoSQL data in the cloud. You'll learn how to create tables, rows, and perform basic tasks within your Cosmos DB resource using the [Azure.Data.Tables Package (NuGet)](https://www.nuget.org/packages/Azure.Data.Tables/).

+> [!NOTE]

+> The [example code snippets](https://github.com/Azure-Samples/cosmos-db-table-api-dotnet-samples) are available on GitHub as a .NET project.

+[Table API reference documentation](/azure/storage/tables) | [Azure.Data.Tables Package (NuGet)](https://www.nuget.org/packages/Azure.Data.Tables/)

+## Prerequisites

+* An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free).

+* [.NET 6.0](https://dotnet.microsoft.com/en-us/download)

+* [Azure Command-Line Interface (CLI)](/cli/azure/) or [Azure PowerShell](/powershell/azure/)

+### Prerequisite check

+* In a terminal or command window, run ``dotnet --list-sdks`` to check that .NET 6.x is one of the available versions.

+* Run ``az --version`` (Azure CLI) or ``Get-Module -ListAvailable AzureRM`` (Azure PowerShell) to check that you have the appropriate Azure command-line tools installed.

+## Setting up

+This section walks you through how to create an Azure Cosmos account and set up a project that uses the Table API NuGet packages.

+### Create an Azure Cosmos DB account

+This quickstart will create a single Azure Cosmos DB account using the Table API.

+#### [Azure CLI](#tab/azure-cli)

+#### [PowerShell](#tab/azure-powershell)

+#### [Portal](#tab/azure-portal)

+### Get Table API connection string

+#### [Azure CLI](#tab/azure-cli)

+#### [PowerShell](#tab/azure-powershell)

+#### [Portal](#tab/azure-portal)

+### Create a new .NET app

+Create a new .NET application in an empty folder using your preferred terminal. Use the [``dotnet new console``](/dotnet/core/tools/dotnet-newt) to create a new console app.

+```console

+dotnet new console -output <app-name>

+### Install the NuGet package

+Add the [Azure.Data.Tables](https://www.nuget.org/packages/Azure.Data.Tables) NuGet package to the new .NET project. Use the [``dotnet add package``](/dotnet/core/tools/dotnet-add-package) command specifying the name of the NuGet package.

+```console

+### Configure environment variables

+## Code examples

+* [Authenticate the client](#authenticate-the-client)

+* [Create a table](#create-a-table)

+* [Create an item](#create-an-item)

+* [Get an item](#get-an-item)

+* [Query items](#query-items)

+The sample code described in this article creates a table named ``adventureworks``. Each table row contains the details of a product such as name, category, quantity, and a sale indicator. Each product also contains a unique identifier.

+You'll use the following Table API classes to interact with these resources:

+- [``TableServiceClient``](/dotnet/api/azure.data.tables.tableserviceclient) - This class provides methods to perform service level operations with Azure Cosmos DB Table API.

+- [``TableClient``](/dotnet/api/azure.data.tables.tableclient) - This class allows you to interact with tables hosted in the Azure Cosmos DB table API.

+- [``TableEntity``](/dotnet/api/azure.data.tables.tableentity) - This class is a reference to a row in a table that allows you to manage properties and column data.

+### Authenticate the client

+From the project directory, open the *Program.cs* file. In your editor, add a using directive for ``Azure.Data.Tables``.

+Define a new instance of the ``TableServiceClient`` class using the constructor, and [``Environment.GetEnvironmentVariable``](/dotnet/api/system.environment.getenvironmentvariables) to read the connection string you set earlier.

+### Create a table

+Retrieve an instance of the `TableClient` using the `TableServiceClient` class. Use the [``TableClient.CreateIfNotExistsAsync``](/dotnet/api/azure.data.tables.tableclient.createifnotexistsasync) method on the `TableClient` to create a new table if it doesn't already exist. This method will return a reference to the existing or newly created table.

+### Create an item

+The easiest way to create a new item in a table is to create a class that implements the [``ITableEntity``](/dotnet/api/azure.data.tables.itableentity) interface. You can then add your own properties to the class to populate columns of data in that table row.

+Create an item in the collection using the `Product` class by calling [``TableClient.AddEntityAsync<T>``](/dotnet/api/azure.data.tables.tableclient.addentityasync).

+### Get an item

+You can retrieve a specific item from a table using the [``TableEntity.GetEntityAsync<T>``](/dotnet/api/azure.data.tables.tableclient.getentity) method. Provide the `partitionKey` and `rowKey` as parameters to identify the correct row to perform a quick *point read* of that item.

+### Query items

+After you insert an item, you can also run a query to get all items that match a specific filter by using the `TableClient.Query<T>` method. This example filters products by category using [Linq](/dotnet/standard/linq) syntax, which is a benefit of using strongly typed `ITableEntity` models like the `Product` class.

+> [!NOTE]

+> You can also query items using [OData](/rest/api/storageservices/querying-tables-and-entities) syntax. You can see an example of this approach in the [Query Data](/azure/cosmos-db/table/tutorial-query-table) tutorial.

+## Run the code

+This app creates an Azure Cosmos Table API table. The example then creates an item and then reads the exact same item back. Finally, the example creates a second item and then performs a query that should return multiple items. With each step, the example outputs metadata to the console about the steps it has performed.

+To run the app, use a terminal to navigate to the application directory and run the application.

+```dotnetcli

+dotnet run

+The output of the app should be similar to this example:

+```output

+Single product name:

+Yamba Surfboard

+Multiple products:

+Yamba Surfboard

+Sand Surfboard

+## Clean up resources

+When you no longer need the Azure Cosmos DB Table API account, you can delete the corresponding resource group.

+### [Azure CLI](#tab/azure-cli)

+Use the [``az group delete``](/cli/azure/group#az-group-delete) command to delete the resource group.

+```azurecli-interactive

+az group delete --name $resourceGroupName

+### [PowerShell](#tab/azure-powershell)

+Use the [``Remove-AzResourceGroup``](/powershell/module/az.resources/remove-azresourcegroup) cmdlet to delete the resource group.

+```azurepowershell-interactive

+$parameters = @{

+ Name = $RESOURCE_GROUP_NAME

+Remove-AzResourceGroup @parameters

+### [Portal](#tab/azure-portal)

+1. Navigate to the resource group you previously created in the Azure portal.

+ > [!TIP]

+ > In this quickstart, we recommended the name ``msdocs-cosmos-quickstart-rg``.

+1. Select **Delete resource group**.

+ :::image type="content" source="media/dotnet-quickstart/delete-resource-group-option.png" alt-text="Screenshot of the Delete resource group option in the navigation bar for a resource group.":::

+1. On the **Are you sure you want to delete** dialog, enter the name of the resource group, and then select **Delete**.

+ :::image type="content" source="media/dotnet-quickstart/delete-confirmation.png" alt-text="Screenshot of the delete confirmation page for a resource group.":::

+In this quickstart, you learned how to create an Azure Cosmos DB Table API account, create a table, and manage entries using the .NET SDK. You can now dive deeper into the SDK to learn how to perform more advanced data queries and management tasks in your Azure Cosmos DB Table API resources.

+> [Get started with Azure Cosmos DB Table API and .NET](/azure/cosmos-db/table/how-to-dotnet-get-started)

+1. [Create a new Azure Cosmos account](create-table-dotnet.md#create-an-azure-cosmos-db-account), or select an existing account.

+SQL databases are deployed as part of a SQL server instance, but usage is tracked at the database level. Additionally, you might also have charges on the parent server, like for Microsoft Defender for Cloud. To get the total cost for your SQL deployment in classic cost analysis, you need to manually sum up the cost of the server and each individual database. As an example, you can see the **aepool** elastic pool at the top of the list below and the **treyanalyticsengine** server lower down on the first page. What you don't see is another database even lower in the list. You can imagine how troubling this situation would be when you need the total cost of a large server instance with many databases.

+Some resources have related dependencies that aren't explicit children or nested under the logical parent in Azure Resource Manager. Examples include disks used by a virtual machine or web apps assigned to an App Service plan. Unfortunately, Cost Management isn't aware of these relationships and can't group them automatically. This experimental feature uses tags to summarize the total cost of your related resources together. You'll see a single row with the parent resource. When you expand the parent resource, you'll see each linked resource listed individually with their respective cost.

+As an example, let's say you have an Azure Virtual Desktop host pool configured with two VMs. Tagging the VMs and corresponding network/disk resources groups them under the host pool, giving you the total cost of the session host VMs in your host pool deployment. This example gets even more interesting if you want to also include the cost of any cloud solutions made available via your host pool.

+6. Specify a tag key of "cm-resource-parent" (make sure it's typed correctly) and paste the resource ID from step 3.

+<a name="cav3forecast"></a>

+## Forecast in the cost analysis preview

+Show the forecast for the current period at the top of the cost analysis preview.

+Charts can be enabled from the [Try preview](https://aka.ms/costmgmt/trypreview) page in the Azure portal. Use the **How would you rate the cost analysis preview?** Option at the bottom of the page to share feedback about the preview.

+<a name="productscolumn"></a>

+## Product column in the cost analysis preview

+Every service tracks different usage attributes of the resources you've deployed. Each of these usage attributes is tracked via a "meter" in your cost data. Meters are grouped into categories and include other metadata to help you understand the charges. WeΓÇÖre testing new columns in the Resources and Services views in the cost analysis preview for Microsoft Customer Agreement. You may see a single Product column instead of the Service, Tier, and Meter columns.

+You can also enable this preview from the [Try preview](https://aka.ms/costmgmt/trypreview) page in the Azure portal. Note this preview is only applicable for Microsoft Customer Agreement accounts.

+<a name="recommendationinsights"></a>

+## Cost savings insights in the cost analysis preview

+Cost insights surface important details about your subscriptions, like potential anomalies or top cost contributors. To support your cost optimization goals, cost insights now include the total cost savings available from Azure Advisor for your subscription.

+You can enable cost savings insights for subscriptions from the [Try preview](https://aka.ms/costmgmt/trypreview) page in the Azure portal.

+<a name="resourceessentials"></a>

+## View cost for your resources

+Cost analysis is available from every management group, subscription, resource group, and billing scope in the Azure portal and the Microsoft 365 admin center. To make cost data more readily accessible for resource owners, you can now find a **View cost** link at the top-right of every resource overview screen, in **Essentials**. Clicking the link will open classic cost analysis with a resource filter applied.

+The view cost link is enabled by default in the [Azure preview portal](https://preview.portal.azure.com).

+<a name="whatsnew"></a>

+## What's new in Cost Management

+Learn about new and updated features or other announcements directly from within the Cost Management experience in the Azure portal. You can also follow along using the [Cost Management updates on the Azure blog](https://aka.ms/costmgmt/blog).

+What's new can be enabled from the [Try preview](https://aka.ms/costmgmt/trypreview) page in the Azure portal.

+If you manage many subscriptions, resource groups, or management groups and need to switch between them often, you might want to include the **Change scope from menu** option.

+| **Reservation** | Break down costs by reservation. | Any usage or purchases that aren't associated with a reservation will show as **No reservation** or **No values**. Group by **Publisher type** to identify other Azure, AWS, or Marketplace purchases. |

+<tr><td>Assert error handling</td><td>Assert error handling are now supported in data flows for data quality and data validation<br><a href="data-flow-assert.md">Learn more</a></td></tr>

+<tr><td>SAP Change Data Capture (CDC) capabilities in the new SAP ODP connector (Public Preview)</td><td>SAP Change Data Capture (CDC) capabilities are now supported in the new SAP ODP connector.<br><a href="https://techcommunity.microsoft.com/t5/azure-data-factory-blog/announcing-the-public-preview-of-the-sap-cdc-solution-in-azure/ba-p/3420904">Learn more</a></td></tr>

+<tr><td><b>Orchestration</b></td><td>ΓÇÿturnOffAsync' property is available in Web activity</td><td>Web activity supports an async request-reply pattern that invokes HTTP GET on the Location field in the response header of an HTTP 202 Response. It helps web activity automatically poll the monitoring end-point till the job runs. ΓÇÿturnOffAsync' property is supported to disable this behavior in cases where polling isn't needed<br><a href="control-flow-web-activity.md#type-properties">Learn more</a></td></tr>

+<tr><td><b>Monitoring</b></td><td> Rerun pipeline with new parameters</td><td>You can now rerun pipelines with new parameter values in Azure Data Factory.<br><a href="monitor-visually.md#rerun-pipelines-and-activities">Learn more</a></td></tr>

+<tr><td><b>Continuous integration and continuous delivery (CI/CD)</b></td><td>Cross tenant Azure DevOps support</td><td>Configure a repository using Azure DevOps Git not in the same tenant as the Azure Data Factory。<br><a href="cross-tenant-connections-to-azure-devops.md">Learn more</a></td></tr>

+Azure Dedicated HSM is a specialized service that addresses unique requirements for a specific type of large-scale organization. As a result, it's expected that the bulk of Azure customers will not fit the profile of use for this service. Many will find the Azure Key Vault or Azure Managed HSM service to be more appropriate and cost effective. For an comparison of offerings, see [Azure key management services](../security/fundamentals/key-management.md#azure-key-management-services)

+To help you decide if Azure Dedicated HSM is a fit for your requirements, we've identified the following criteria.

+> [!NOTE]

+> Customers must have a assigned Microsoft Account Manager and meet the monetary requirement of five million ($5M) USD or greater in overall committed Azure revenue annually to qualify for onboarding and use of Azure Dedicated HSM.

+ Title: Microsoft Defender for Storage - the benefits and features

+You can [enable Microsoft Defender for Storage](../storage/common/azure-defender-storage-configure.md#set-up-microsoft-defender-for-cloud) at either the subscription level (recommended) or the resource level.

+Defender for Storage continually analyzes the telemetry stream generated by the [Azure Blob Storage](https://azure.microsoft.com/services/storage/blobs/) and Azure Files services. When potentially malicious activities are detected, security alerts are generated. These alerts are displayed in Microsoft Defender for Cloud, together with the details of the suspicious activity along with the relevant investigation steps, remediation actions, and security recommendations.

+Analyzed telemetry of Azure Blob Storage includes operation types such as `Get Blob`, `Put Blob`, `Get Container ACL`, `List Blobs`, and `Get Blob Properties`. Examples of analyzed Azure Files operation types include `Get File`, C`reate File`, `List Files`, `Get File Properties`, and `Put Range`.

+You can check out [the full list of Microsoft Defender for Storage alerts](alerts-reference.md#alerts-azurestorage).

+- [How do I estimate charges at the account level?](#how-do-i-estimate-charges-at-the-account-level)

+- [Can I exclude a specific Azure Storage account from a protected subscription?](#can-i-exclude-a-specific-azure-storage-account-from-a-protected-subscription)

+- [How do I configure automatic responses for security alerts?](#how-do-i-configure-automatic-responses-for-security-alerts)

+**Estimated date for change:** August 2022

+ Title: OT monitoring appliance reference overview - Microsoft Defender for IoT

+description: Provides an overview of all appliances available for use with Microsoft Defender for IoT OT sensors and on-premises management consoles.

+# OT monitoring appliance reference

+This article provides an overview of the OT monitoring appliances supported with Microsoft Defender for IoT.

+Each article provides details about the appliance and any extra software installation procedures required. For more information, see [Install OT system software](../how-to-install-software.md) and [Update Defender for IoT OT monitoring software](../update-ot-software.md).

+## Corporate environments

+The following OT monitoring appliances are available for corporate deployments:

+- [HPE ProLiant DL360](hpe-proliant-dl360.md)

+## Large enterprises

+The following OT monitoring appliances are available for large enterprise deployments:

+- [HPE ProLiant DL20/DL20 Plus (4SFF)](hpe-proliant-dl20-plus-enterprise.md)

+## Production line

+The following OT monitoring appliances are available for production line deployments:

+- [HPE ProLiant DL20/DL20 Plus (NHP 2LFF) for SMB deployments](hpe-proliant-dl20-plus-smb.md)

+- [Dell Edge 5200 (Rugged)](dell-edge-5200.md)

+- [YS-techsystems YS-FIT2 (Rugged)](ys-techsystems-ys-fit2.md)

+## Next steps

+For more information, see:

+- [Which appliances do I need?](../ot-appliance-sizing.md)

+- [Pre-configured physical appliances for OT monitoring](../ot-pre-configured-appliances.md)

+- [OT monitoring with virtual appliances](../ot-virtual-appliances.md)

+The following image shows how data can stream into Defender for IoT from network sensors, Microsoft Defender for Endpoint, and partner sources to provide a unified view of IoT/OT security. Defender for IoT in the Azure portal provides asset inventories, vulnerability assessments, and continuous threat monitoring.

+- The sensors are purpose-built for IoT and OT networks. They connect to a SPAN port or network TAP and can provide visibility into IoT and OT risks within minutes of connecting to the network.

+- The sensors use IoT and OT-aware analytics engines and Layer-6 Deep Packet Inspection (DPI) to detect IoT and OT threats, such as fileless malware, based on anomalous or unauthorized activity.

+- You must manually upload any threat intelligence packages.

+For example, for OT networks, the **policy violation detection** engine alerts users of any deviation from baseline behavior, such as unauthorized use of specific function codes, access to specific objects, or changes to device configuration. The policy violation engine models industry control system (ICS) networks as deterministic sequences of states and transitions - using a patented technique called Industrial Finite State Modeling (IFSM). The policy violation detection engine creates a baseline for industrial control system (ICS) networks. Since many detection algorithms were built for IT, rather than OT, networks, an extra baseline for ICS networks helps to shorten the systems learning curve for new detections.

+- **Protocol violation detection engine**: Identifies the use of packet structures and field values that violate ICS protocol specifications, for example: Modbus exception, and initiation of an obsolete function code alerts.

+- **Industrial malware detection engine**: Identifies behaviors that indicate the presence of known malware, such as Conficker, Black Energy, Havex, WannaCry, NotPetya, and Triton.

+- **Anomaly detection engine**: Detects unusual machine-to-machine (M2M) communications and behaviors. By modeling ICS networks as deterministic sequences of states and transitions, the platform requires a shorter learning period than generic mathematical approaches or analytics originally developed for IT rather than OT. It also detects anomalies faster, with minimal false positives. Anomaly detection engine alerts include Excessive SMB sign-in attempts, and PLC Scan Detected alerts.

+- **Operational incident detection**: Detects operational issues such as intermittent connectivity that can indicate early signs of equipment failure. For example, the device might be disconnected (unresponsive), and Siemens S7 stop PLC command was sent alerts.

+- An Azure account. If you don't already have an Azure account, you can [create your free Azure account today](https://azure.microsoft.com/free/).

+## Add a Defender for IoT plan to an Azure subscription

+This procedure describes how to add a Defender for IoT plan to an Azure subscription.

+ :::image type="content" source="media/how-to-manage-subscriptions/onboard-plan.png" alt-text="Screenshot of adding a plan to your subscription." lightbox="media/how-to-manage-subscriptions/onboard-plan.png":::

+1. **Review & purchase**. Review the listed charges for your selections and **accept the terms and conditions**.

+Your plan will be shown under the associated subscription in the **Plans and pricing** grid.

+ Title: Install OT network monitoring software - Microsoft Defender for IoT

+description: Learn how to install agentless monitoring software for an OT sensor and an on-premises management console for Microsoft Defender for IoT. Use this article if you're reinstalling software on a preconfigured appliance, or if you've chosen to install software on your own appliances.

+# Install OT agentless monitoring software

+This article describes how to install agentless monitoring software for OT sensors and on-premises management consoles. You might need the procedures in this article if you're reinstalling software on a preconfigured appliance, or if you've chosen to install software on your own appliances.

+You can obtain the latest versions of our OT sensor and on-premises management console software from the Azure portal. On the Defender for IoT > **Getting started** page, select the **Sensor**, **On-premises management console**, or **Updates** tab and locate the software you need.

+1. Select the monitor interface. For example:

+1. If one of the monitoring ports is for ERSPAN, select it. For example:

+1. Select the interface to be used as the management interface. For example:

+1. Enter the sensor's IP address. For example:

+1. Enter the path of the mounted logs folder. We recommend using the default path. For example:

+1. Enter the Subnet Mask IP address. For example:

+1. Enter the default gateway IP address.

+1. Enter the DNS Server IP address.

+1. Enter the sensor hostname. For example:

+You can enhance security to your on-premises management console by adding a secondary NIC dedicated for attached sensors within an IP address range. When you use a secondary NIC, the first is dedicated for end-users, and the secondary supports the configuration of a gateway for routed networks.

+System health validations are supported via the sensor or on-premises management console UI or CLI, and are available for both the **Support** and **CyberX** users.

+## Configure tunneling access for sensors through the on-premises management console

+Enhance system security by preventing direct user access to the sensor.

+Instead of direct access, use proxy tunneling to let users access the sensor from the on-premises management console with a single firewall rule. This technique narrows the possibility of unauthorized access to the network environment beyond the sensor. The user's experience when signing in to the sensor remains the same.

+When tunneling access is configured, users use the following URL syntax to access their sensor consoles: `https://<on-premises management console address>/<sensor address>/<page URL>`

+For example, the following image shows a sample architecture where users access the sensor consoles via the on-premises management console.

+The interface between the IT firewall, on-premises management console, and the OT firewall is done using a reverse proxy with URL rewrites. The interface between the OT firewall and the sensors is done using reverse SSH tunnels.

+**To enable tunneling access for sensors**:

+OT network sensors use agentless, patented technology to discover, learn, and continuously monitor network devices for a deep visibility into OT/ICS/IoT risks. Sensors carry out data collection, analysis, and alerting on-site, making them ideal for locations with low bandwidth or high latency.

+Before performing the procedures in this article, make sure you understand your own network architecture and how you'll connect to Defender for IoT. For more information, see:

+Perform the steps in this section before deploying Defender for IoT on your network.

+- TLS/SSL certificates (optional but recommended).

+- Make sure that you have terminal software (like PuTTY) or a supported browser. Supported browsers include the latest versions of Microsoft Edge, Chrome, Firefox, or Safari (Mac only).

+- Make sure the required firewall rules are open on the workstation. Verify that your organizational security policy allows access as required. For more information, see [Networking requirements](#networking-requirements).

+After you've installed the Defender for IoT sensor or on-premises management console software, a local, self-signed certificate is generated, and used to access the sensor web application.

+The first time they sign in to Defender for IoT, administrator users are prompted to provide an SSL/TLS certificate. Optional certificate validation is enabled by default.

+| TLS/SSL | TCP | In/Out | 443 | Give the sensor access to the on-premises management console. | The connection between the sensor, and the on-premises management console | Sensor | On-premises management console |

+| Tunneling | TCP | In | 9000 </br></br> In addition to port 443 </br></br> Allows access from the sensor, or end user, to the on-premises management console </br></br> Port 22 from the sensor to the on-premises management console | Monitoring | Tunneling | Endpoint, Sensor | On-premises management console |

+1. Verify that the computer you're trying to connect is on the same network as the appliance.

+5. After you restart, connect with user support, and use the **network list** command to verify that the parameters were changed.

+Reference articles for OT monitoring appliances also include installation procedures in case you need to install software on your own appliances, or reinstall software on preconfigured appliances.

+For more information, see [Purchase sensors or download software for sensors](onboard-sensors.md#purchase-sensors-or-download-software-for-sensors).

+## Advantages of preconfigured appliances

+Pre-configured physical appliances have been validated for Defender for IoT OT system monitoring, and have the following advantages over installing your own software:

+- **Performance** over the total assets monitored

+- **Compatibility** with new Defender for IoT releases, with validations for upgrades and driver support

+- **Stability**, validated physical appliances undergo traffic monitoring and packet loss tests

+- **In-lab experience**, Microsoft support teams train using validated physical appliances and have a working knowledge of the hardware

+- **Availability**, components are selected to offer long-term worldwide availability

+You can [order](mailto:hardware.sales@arrow.com) any of the following preconfigured appliances for monitoring your OT networks:

+|**C5600** | [HPE ProLiant DL360](appliance-catalog/hpe-proliant-dl360.md) | **Max bandwidth**: 3Gbp/s <br>**Max devices**: 12,000 <br> 32 Cores/32G RAM/5.6TB | **Mounting**: 1U <br>**Ports**: 15x RJ45 or 8x SFP (OPT) |

+|**E1800, E1000, E500** | [HPE ProLiant DL20/DL20 Plus](appliance-catalog/hpe-proliant-dl20-plus-enterprise.md) <br> (4SFF) | **Max bandwidth**: 1 Gbp/s<br>**Max devices**: 10,000 <br> 8 Cores/32G RAM/1.8TB | **Mounting**: 1U <br>**Ports**: 8x RJ45 or 6x SFP (OPT) |

+|**L500** | [HPE ProLiant DL20/DL20 Plus](appliance-catalog/hpe-proliant-dl20-plus-smb.md) <br> (NHP 2LFF) | **Max bandwidth**: 200Mbp/s<br>**Max devices**: 1,000 <br> 4 Cores/8G RAM/500GB | **Mounting**: 1U<br>**Ports**: 4x RJ45 |

+|**L100** | [YS-Techsystems YS-FIT2](appliance-catalog/ys-techsystems-ys-fit2.md) <br>(Rugged MIL-STD-810G) | **Max bandwidth**: 10Mbp/s <br>**Max devices**: 100 <br> 4 Cores/8G RAM/128GB | **Mounting**: DIN/VESA<br>**Ports**: 2x RJ45 |

+|**L64** | [Dell Edge 5200](appliance-catalog/dell-edge-5200.md) <br> (Rugged MIL-STD-810G) | **Max bandwidth**: 60Mbp/s<br>**Max devices**: 1,000 <br> 8 Cores/32G RAM/100GB | **Mounting**: Wall Mount<br>**Ports**: 3x RJ45 |

+|**E1800, E1000, E500** | [HPE ProLiant DL20/DL20 Plus](appliance-catalog/hpe-proliant-dl20-plus-enterprise.md) <br> (4SFF) | 300 | **Mounting**: 1U <br>**Ports**: 8x RJ45 or 6x SFP (OPT) |

+Continue understanding system requirements for physical or virtual appliances.

+For more information, see [Which appliances do I need?](ot-appliance-sizing.md) and [OT monitoring with virtual appliances](ot-virtual-appliances.md).

+Then, use any of the following procedures to continue:

+Our OT monitoring appliance reference articles also include extra installation procedures in case you need to install software on your own appliances, or reinstall software on preconfigured appliances. For more information, see [OT monitoring appliance reference](appliance-catalog/appliance-catalog-overview.md).

+Microsoft Defender for IoT is a unified security solution for identifying IoT and OT devices, vulnerabilities, and threats. With Defender for IoT, you can manage them through a central interface. This set of documentation describes how end-user organizations can secure their entire IoT/OT environment, including protecting existing devices or building security into new IoT innovations.

+Traditional network security monitoring tools may lack understanding of networks containing specialized protocols, devices, and relevant machine-to-machine (M2M) behaviors. Agentless monitoring in Defender for IoT provides visibility and security into those networks.

+ - Identify unpatched devices, open ports, unauthorized applications, unauthorized connections, changes to device configurations, PLC code, firmware, and more.

+ - Run searches in historical traffic across all relevant dimensions and protocols. Access full-fidelity PCAPs to drill down further.

+ - Detect advanced threats that you may have missed by static IOCs, such as zero-day malware, fileless malware, and living-off-the-land tactics.

+- **Respond to threats** by integrating with Microsoft services, such as Microsoft Sentinel, non-Microsoft systems, and APIs. Use advanced integrations for security information and event management (SIEM), security operations and response (SOAR), extended detection and response (XDR) services, and more.

+- **Cloud**: Extend your journey to the cloud by delivering your data to Azure. There you can visualize data from a central location. That data can be shared with other Microsoft services for end-to-end security monitoring and response.

+- **On-premises**: For example, in air-gapped environments, you might want to keep all of your data fully on-premises. Use the data provided by each sensor and the central visualizations provided by an on-premises management console to ensure security on your network.

+- **Hybrid**: You may have hybrid network requirements where you can deliver some data to the cloud and other data must remain on-premises. In this case, set up your system in a flexible and scalable configuration that fits your needs.

+For example, in an environment running MODBUS, you can generate an alert when the sensor detects a write command to a memory register on a specific IP address and Ethernet destination. Or you might want to generate an alert when any access is performed to a specific IP address. Alerts are triggered when Horizon alert rule conditions are met.

+Microsoft Defender for IoT can protect IoT and OT devices, whether they're connected to IT, OT, or dedicated IoT networks.

+Use Microsoft Defender for IoT's sensors as extra data sources. They provide visibility in areas of your organization's network where Microsoft Defender for Endpoint isn't deployed, and when employees are accessing information remotely. Microsoft Defender for IoT's sensors provide visibility into both the IoT-to-IoT and the IoT-to-internet communications. Integrating Defender for IoT and Defender for Endpoint synchronizes any enterprise IoT devices discovered on the network by either service.

+This tutorial describes how to get started with your Enterprise IoT monitoring deployment with Microsoft Defender for IoT.

+Defender for IoT supports the entire breadth of IoT devices in your environment, including everything from corporate printers and cameras, to purpose-built, proprietary, and unique devices.

+Integrate with [Microsoft Defender for Endpoint](/microsoft-365/security/defender-endpoint/enable-microsoft-defender-for-iot-integration) for analytics features that include alerts, vulnerabilities, and recommendations for your enterprise devices.

+> * Prepare your networking requirements

+> * Set up an Enterprise IoT sensor in the Azure portal

+> * Install the sensor software

+- A Defender for IoT plan added to your Azure subscription.

+ You can add a plan from Defender for IoT in the Azure portal, or from Defender for Endpoint. If you already have a subscription that has Defender for IoT onboarded for OT environments, youΓÇÖll need to edit the plan to add Enterprise IoT.

+ For more information, see [Quickstart: Get started with Defender for IoT](getting-started.md), [Edit a plan](how-to-manage-subscriptions.md#edit-a-plan), or the [Defender for Endpoint documentation](/microsoft-365/security/defender-endpoint/enable-microsoft-defender-for-iot-integration).

+- Required Azure permissions, as listed in [Quickstart: Getting Started with Defender for IoT](getting-started.md#permissions).

+Before you deploy your Enterprise IoT sensor, you'll need to configure your server or VM, and connect a Network Interface Card (NIC) to a switch monitoring (SPAN) port.

+**To set up a server or VM**:

+ | **Minimum** | To support up to 1 Gbps: <br><br>- 4 CPUs, each with 2.4 GHz or more<br>- 16-GB RAM of DDR4 or better<br>- 250 GB HDD |

+ | **Recommended** | To support up to 15 Gbps: <br><br>- 8 CPUs, each with 2.4 GHz or more<br>- 32-GB RAM of DDR4 or better<br>- 500 GB HDD |

+ - Two network adapters

+ - Ubuntu 18.04 operating system

+1. Connect a NIC to a switch as follows:

+ - **Physical device** - Connect a monitoring network interface (NIC) to a switch monitoring (SPAN) port.

+ - **VM** - Connect a vNIC to a vSwitch in promiscuous mode.

+ For a VM, run the following command to enable the network adapter in promiscuous mode.

+ ```bash

+ ifconfig <monitoring port> up promisc

+ ```

+1. Validate incoming traffic to the monitoring port. Run:

+## Prepare networking requirements

+This procedure describes how to prepare networking requirements on your server or VM to work with Defender for IoT with Enterprise IoT networks.

+**To prepare your networking requirements**:

+ - **HTTPS** - 443 TCP

+ - **DNS** - 53 TCP

+1. Make sure that your server or VM can access the cloud using HTTP on port 443 to the following Microsoft domains:

+ - **EventHub**: `*.servicebus.windows.net`

+ - **Storage**: `*.blob.core.windows.net`

+ - **Download Center**: `download.microsoft.com`

+ - **IoT Hub**: `*.azure-devices.net`

+### (Optional) Download Azure public IP ranges

+You can also download and add the [Azure public IP ranges](https://www.microsoft.com/download/details.aspx?id=56519) so your firewall will allow the Azure domains that are specified above, along with their region.

+> The Azure public IP ranges are updated weekly. New ranges appearing in the file will not be used in Azure for at least one week. To use this option, download the new json file every week and perform the necessary changes at your site to correctly identify services running in Azure.

+You'll need a Defender for IoT network sensor to discover and continuously monitor Enterprise IoT devices. Defender for IoT sensors use the Enterprise IoT network and endpoint sensors to gain comprehensive visibility.

+**Prerequisites**: Make sure that you've completed [Set up a server or Virtual Machine (VM)](#set-up-a-server-or-virtual-machine-vm) and [Prepare networking requirements](#prepare-networking-requirements), including verifying that you have the listed required resources.

+ :::image type="content" source="media/tutorial-get-started-eiot/onboard-sensor.png" alt-text="Screenshot of the Getting started page for Enterprise IoT security.":::

+1. Copy the command to a safe location, and continue by [installing the sensor](#install-the-sensor) software.

+ The installation wizard appears when the command process completes:

+ - In the **What is the name of the monitored interface?** screen, use the SPACEBAR to select the interfaces you want to monitor with your sensor, and then select OK.

+ - In the **Set up proxy server** screen, select whether to set up a proxy server for your sensor (**Yes** / **No**).

+ (Optional) If you're setting up a proxy server, define the following values, selecting **Ok** after each option:

+ - Proxy server host

+ - Proxy server port

+ - Proxy server username

+ - Proxy server password

+The installation process completes.

+## Validate your setup

+Wait 1 minute after your sensor installation has completed before starting to validate your sensor setup.

+**To validate the sensor setup**:

+1. To process your system sanity, run:

+1. In the results that display, ensure that the following containers are up:

+ - `compose_statistics-collector_1`

+ - `compose_cloud-communication_1`

+ - `compose_horizon_1`

+ - `compose_attributes-collector_1`

+ - `compose_properties_1`

+1. Check your port validation to see which interface is defined to handle port mirroring. Run:

+ For example:

+ :::image type="content" source="media/tutorial-get-started-eiot/defined-interface.png" alt-text="Screenshot of a result showing an interface defined to handle port monitoring.":::

+1. Wait 5 minutes and then check your traffic D2C sanity. Run:

+ Check your results to ensure that packets are being sent to the Event Hubs.

+- View your devices and network information in the Defender for IoT **Device inventory** page on the Azure portal.

+ To view your device inventory, go to **Defender for IoT** > **Device inventory**.

+- You can also view your sensors from the **Sites and sensors** page. Enterprise IoT sensors are all automatically added to the same site, named **Enterprise network**.

+Once youΓÇÖve onboarded a plan and set up your sensor, your device data integrates automatically with Microsoft Defender for Endpoint. Discovered devices appear in both the Defender for IoT and Defender for Endpoint portals. Use this integration to extend security analytics capabilities for your Enterprise IoT devices and providing complete coverage.

+## Remove an Enterprise IoT network sensor (optional)

+Remove a sensor if it's no longer in use with Defender for IoT.

+**To remove a sensor**, run the following command on the sensor server or VM:

+ Title: Tutorial - Get started with Microsoft Defender for IoT for OT security

+- VMware, ESXi 5.5 or later, installed and operational on your sensor.

+Before you can start using your Defender for IoT sensor, you'll need to onboard your new virtual sensor to your Azure subscription, and download the virtual sensor's activation file to activate the sensor.

+ :::image type="content" source="media/tutorial-onboarding/onboard-a-sensor.png" alt-text="Screenshot of the Getting started page for OT network sensors.":::

+After your OT sensor is connected, continue with any of the following to start analyzing your data:

+ * You should also configure [Cross-Origin Resource Sharing (CORS)](/rest/api/storageservices/cross-origin-resource-sharing--cors--support-for-the-azure-storage-services) for your storage account, so that 3D Scenes Studio will be able to access your storage container. For complete CORS setting information, see [Use 3D Scenes Studio (preview)](how-to-use-3d-scenes-studio.md#prerequisites).

+Or, learn how to use the studio's full feature set in [Use 3D Scenes Studio](how-to-use-3d-scenes-studio.md).

+ 1. Select **Add event type definitions** to declare the kind of events that are sent to the channel and to its associated partner topic. Event types are shown to customers when creating event subscriptions on the partner topic and are used to select the specific event types to send to an event handler destination.

+ :::image type="content" source="./media/onboard-partner/event-type-definition-1.png" alt-text="Screenshot that shows the Event Types Definitions section with Add event types definitions option selected.":::

+ :::image type="content" source="./media/onboard-partner/event-type-definition-2.png" alt-text="Screenshot that shows the definition of a sample event type.":::

+

+ :::image type="content" source="./media/onboard-partner/event-type-definition-3.png" alt-text="Screenshot that shows a list with the event type definition that was added.":::

+## Manage a channel

+If you created a channel you may be interested to update the configuration once the resource has been created.

+1. Go to the **Configuration** on the channel. You may update message for partner topic activation, expiration time if not activated, and event type definitions.

+ :::image type="content" source="./media/onboard-partner/channel-configuration.png" alt-text="Screenshot that shows the Configuration page of a channel.":::

+> [!IMPORTANT]

+> Don't forget to save changes before leaving the configuration page.

+- You want your service to react to customer events that originate in Microsoft Azure.

+- You want your customer to react to Microsoft Azure service events using their applications hosted by your platform. You use your platform's event routing capabilities to deliver events to the right customer solution.

+ - It's the resource type that allows you to create partner resources on a customer's Azure subscription. When you create a channel of type `partner topic`, a partner topic is created on a customer's Azure subscription. A partner topic is a customer's resource to which events are routed when a partner system publishes events. Similarly, when a channel of type `partner destination` is created, a partner destination is created on a customer's Azure subscription. Partner destinations are resources that represent a partner system endpoint to where events are delivered. A channel is the kind of resource, along with partner topics and partner destinations that enable bi-directional event integration.

+ You may want to declare the event types that are routed to the channel and to its associated partner topic. Event types are shown to customers when creating event subscriptions on the partner topic and are used to select the specific event types to send to an event handler destination. [Learn more](onboard-partner.md#create-a-channel).

+ >[!IMPORTANT]

+ >Event types can be managed on the channel and once the values are updated, changes are reflected immediately on the associated partner topic.

+## Response headers

+The following response headers will be stripped if the origin response is cacheable. For example, Cache control header has max-age value.

+- Set-Cookie

+ [how remediation access control works](./how-to/remediate-resources.md#how-remediation-access-control-works).

+ [remediation - configure the policy definition](../how-to/remediate-resources.md#configure-the-policy-definition).

+ [remediation - configure the policy definition](../how-to/remediate-resources.md#configure-the-policy-definition).

+Resources that are non-compliant to policies with **deployIfNotExists** or **modify** effects can be put into a

+compliant state through **Remediation**. Remediation is accomplished through **remediation tasks** that deploy the **deployIfNotExists** template or the **modify** operations of the assigned policy on your existing resources and subscriptions, whether that assignment is on a management group,

+subscription, resource group, or individual resource. This article shows the steps needed to

+## How remediation access control works

+When Azure Policy starts a template deployment when evaluating **deployIfNotExists** policies or modifies a resource when evaluating **modify** policies, it does so using

+If the managed identity is missing roles, an error is displayed in the portal

+ > [!NOTE]

+ > Changing a policy definition does not automatically update the assignment or the associated managed identity.

+Remediation security can be configured through the following steps:

+- [Configure the policy definition](#configure-the-policy-definition)

+- [Configure the managed identity](#configure-the-managed-identity)

+- [Grant permissions to the managed identity through defined roles](#grant-permissions-to-the-managed-identity-through-defined-roles)

+- [Create a remediation task](#create-a-remediation-task)

+## Configure the policy definition

+As a prerequisite, the policy definition must define the roles that **deployIfNotExists** and **modify** need to successfully deploy the content of the included template. No action is required for a built-in policy definition because these roles are prepopulated. For a custom policy definition, under the **details**

+property, add a **roleDefinitionIds** property. This property is an array of strings that match

+## Configure the managed identity

+Each Azure Policy assignment can be associated with only one managed identity. However, the managed identity can be assigned multiple roles. Configuration occurs in two steps: first create either a system-assigned or user-assigned managed identity, then grant it the necessary roles.

+ > When creating a managed identity through the portal, roles will be granted automatically to the managed identity. If **roleDefinitionIds** are later edited in the policy definition, the new permissions must be manually granted, even in the portal.

+### Create the managed identity

+# [Portal](#tab/azure-portal)

+When creating an assignment using the portal, Azure Policy can generate a system-assigned managed identity and grant it the roles defined in the policy definition's **roleDefinitionIds**. Alternatively, you can specify a user-assigned managed identity that receives the same role assignment.

+# [PowerShell](#tab/azure-powershell)

+To create an identity during the assignment of the policy, **Location** must be defined and **Identity** used.

+The following example gets the definition of the built-in policy **Deploy SQL DB transparent data encryption** sets the target resource group, and then creates the assignment using a **system assigned** managed identity.

+# [Azure CLI](#tab/azure-cli)

+To create an identity during the assignment of the policy, use [az policy assignment create](/cli/azure/policy/assignment?view=azure-cli-latest#az-policy-assignment-create&preserve-view=true) commands with the parameters **--location**, **--mi-system-assigned**, **--mi-user-assigned**, and **--identity-scope** depending on whether the managed identity should be system-assigned or user-assigned.

+To add a system-assigned identity or a user-assigned identity to a policy assignment, follow example [az policy assignment identity assign](/cli/azure/policy/assignment/identity?view=azure-cli-latest#az-policy-assignment-identity-assign&preserve-view=true) commands.

+### Grant permissions to the managed identity through defined roles

+> [!IMPORTANT]

+>

+> If the managed identity does not have the permissions needed to execute the required remediation task, it will be granted permissions *automatically* only through the portal. You may skip this step if creating a managed identity through the portal.

+>

+> For all other methods, the assignment's managed identity must be manually granted access through the addition of roles, or else the remediation deployment will fail.

+>

+> Example scenarios that require manual permissions:

+> - If the assignment is created through SDK

+> - If a resource modified by **deployIfNotExists** or **modify** is outside the scope of the policy

+> assignment

+> - If the template accesses properties on resources outside the scope of the policy assignment

+>

+# [Portal](#tab/azure-portal)

+There are two ways to grant an assignment's managed identity the defined roles using the portal: by

+1. Select the appropriate role that matches a **roleDefinitionId** from the policy definition.

+# [PowerShell](#tab/azure-powershell)

+The new managed identity must complete replication through Azure Active Directory before it can be

+granted the needed roles. Once replication is complete, the following example iterates the policy

+definition in `$policyDef` for the **roleDefinitionIds** and uses

+[New-AzRoleAssignment](/powershell/module/az.resources/new-azroleassignment) to

+grant the new managed identity the roles.

+```azurepowershell-interactive

+# Use the $policyDef to get to the roleDefinitionIds array

+$roleDefinitionIds = $policyDef.Properties.policyRule.then.details.roleDefinitionIds

+if ($roleDefinitionIds.Count -gt 0)

+{

+ $roleDefinitionIds | ForEach-Object {

+ $roleDefId = $_.Split("/") | Select-Object -Last 1

+ New-AzRoleAssignment -Scope $resourceGroup.ResourceId -ObjectId $assignment.Identity.PrincipalId -RoleDefinitionId $roleDefId

+ }

+}

+```

+# [Azure CLI](#tab/azure-cli)

+The new managed identity must complete replication through Azure Active Directory before it can be granted the needed roles. Once replication is complete, the roles specified in the policy definition's **roleDefinitionIds** should be granted to the managed identity.

+Access the roles specified in the policy definition using the [az policy definition show](/cli/azure/policy/definition?view=azure-cli-latest#az-policy-definition-show&preserve-view=true) command, then iterate over each **roleDefinitionId** to create the role assignment using the [az role assignment create](/cli/azure/policy/definition?view=azure-cli-latest#az-policy-definition-show&preserve-view=true) command.

+## Create a remediation task

+# [Portal](#tab/azure-portal)

+Launch the Azure Policy service in the Azure portal by selecting **All services**, then searching for and selecting **Policy**.

+### Step 1: Initiate remediation task creation

+There are three ways to create a remediation task through the portal.

+#### Option 1: Create a remediation task from the Remediation page

+1. All **deployIfNotExists** and **modify** policy assignments are

+ shown on the **Policies to remediate** tab. Select one with resources

+ that are non-compliant to open the **New remediation task** page.

+

+1. Follow steps to [specify remediation task details](#step-2-specify-remediation-task-details).

+#### Option 2: Create a remediation task from a non-compliant policy assignment

+1. Select **Compliance** on the left side of the Azure Policy page.

+1. Select a non-compliant policy or initiative assignment containing **deployIfNotExists** or **modify** effects.

+1. Select the **Create Remediation Task** button at the top of the page to open the **New remediation task** page.

+1. Follow steps to [specify remediation task details](#step-2-specify-remediation-task-details).

+#### Option 3: Create a remediation task during policy assignment

+If the policy or initiative definition to assign has a **deployIfNotExists** or a **Modify** effect,

+the **Remediation** tab of the wizard offers a _Create a remediation task_ option, which creates a remediation task at the same time as the policy assignment.

+ > This is the most streamlined approach for creating a remediation task and is supported for policies assigned on a _subscription_. For policies assigned on a _management group_, remediation tasks should be created using [Option 1](#option-1-create-a-remediation-task-from-the-remediation-page) or [Option 2](#option-2-create-a-remediation-task-from-a-non-compliant-policy-assignment) after evaluation has determined resource compliance.

+1. From the assignment wizard in the portal, navigate to the **Remediation** tab. Select the check box for **Create a remediation task**.

+1. If the remediation task is initiated from an initiative assignment, select the policy to remediate from the drop-down.

+1. Configure the [managed identity](#configure-the-managed-identity) and fill out the rest of the wizard. The remediation task will be created when the assignment is created.

+### Step 2: Specify remediation task details

+This step is only applicable when using [Option 1](#option-1-create-a-remediation-task-from-the-remediation-page) or [Option 2](#option-2-create-a-remediation-task-from-a-non-compliant-policy-assignment) to initiate remediation task creation.

+1. If the remediation task is initiated from an initiative assignment, select the policy to remediate from the drop-down. One **deployIfNotExists** or **modify** policy can be remediated through a single Remediation task at a time.

+1. Optionally modify remediation settings on the **New remediation task** page:

+ - **Resource Count** - Determines how many non-compliant resources to remediate in a given remediation task. The default value is 500 (the previous limit). The maximum number is 50,000 resources.

+### Step 3: Track remediation task progress

+1. Navigate to the **Remediation tasks** tab on the **Remediation** page. Click on a remediation task to view details about the filtering used, the current status, and a list of resources being remediated.

+1. From the **Remediation task** details page, right-click on a resource to view either the remediation

+ task's deployment or the resource. At the end of the row, select **Related events** to see

+Resources deployed through a **remediation task** are added to the **Deployed Resources** tab on the policy assignment details page.

+# [PowerShell](#tab/azure-powershell)

+You may also choose to adjust remediation settings through these optional parameters:

+- `-FailureThreshold` - Used to specify whether the remediation task should fail if the percentage of failures exceeds the given threshold. Provided as a number between 0 to 100. By default, the failure threshold is 100%.

+- `-ParallelDeploymentCount` - Determines how many non-compliant resources to remediate in a given remediation task. The default value is 500 (the previous limit). The maximum number is 50,000 resources.

+- `-ResourceCount` - Determines how many resources to remediate at the same time. The allowed values are 1 to 30 resources at a time. The default value is 10.

+For more remediation cmdlets and examples, see the [Az.PolicyInsights](/powershell/module/az.policyinsights/#policy_insights)

+# [Azure CLI](#tab/azure-cli)

+To create a **remediation task** with Azure CLI, use the `az policy remediation` commands. Replace

+`{subscriptionId}` with your subscription ID and `{myAssignmentId}` with your **deployIfNotExists**

+or **modify** policy assignment ID.

+```azurecli-interactive

+# Login first with az login if not using Cloud Shell

+# Create a remediation for a specific assignment

+az policy remediation create --name myRemediation --policy-assignment '/subscriptions/{subscriptionId}/providers/Microsoft.Authorization/policyAssignments/{myAssignmentId}'

+```

+For more remediation commands and examples, see the [az policy

+remediation](/cli/azure/policy/remediation) commands.

+> [Configure policy definitions for remediation](./how-to/remediate-resources.md#configure-the-policy-definition).

+ [how remediation access control works](../how-to/remediate-resources.md#how-remediation-access-control-works).

+ :::image type="content" source="../media/create-and-manage/initiative-definition-2.png" alt-text="Screenshot of the selected policy definitions with their reference ID and group on the initiative definition page.":::

+ the initiative but with different parameters, this configuration adds or replaces an 'Env' tag

+ [how remediation access control works](../how-to/remediate-resources.md#how-remediation-access-control-works).

+work. We need to grant Trent a space for an exception. Create a new resource group,

+description: How to connect devices through an IoT Edge transparent gateway to an IoT Central application. The article shows how to use both the IoT Edge 1.1 and 1.2 runtimes.

+IoT Edge supports the [*transparent* and *translation* gateway patterns](../../iot-edge/iot-edge-as-gateway.md). This article summarizes how to implement the transparent gateway pattern. In this pattern, the gateway passes messages from the downstream device through to the IoT Hub endpoint in your IoT Central application. The gateway doesn't manipulate the messages as they pass through. In IoT Central, each downstream device appears as child to the gateway device:

+This article shows how to implement the scenario by using either the IoT Edge 1.1 runtime or the IoT Edge 1.2 runtime.

+# [IoT Edge 1.1](#tab/edge1-1)

+To complete the steps in this article, you need:

+- An active Azure subscription. If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.

+- An [IoT Central application created](howto-create-iot-central-application.md) from the **Custom application** template. To learn more, see [Create an IoT Central application](howto-create-iot-central-application.md).

+To follow the steps in this article, download the following files to your computer:

+- [Thermostat device model (thermostat-1.json)](https://raw.githubusercontent.com/Azure/iot-plugandplay-models/main/dtmi/com/example/thermostat-1.json) - this file is the device model for the downstream devices.

+- [Transparent gateway manifest (EdgeTransparentGatewayManifest.json)](https://raw.githubusercontent.com/Azure-Samples/iot-central-docs-samples/master/transparent-gateway-1-1/EdgeTransparentGatewayManifest.json) - this file is the IoT Edge deployment manifest for the gateway device.

+# [IoT Edge 1.2](#tab/edge1-2)

+- [Transparent gateway manifest (EdgeTransparentGatewayManifest.json)](https://raw.githubusercontent.com/Azure-Samples/iot-central-docs-samples/master/transparent-gateway-1-2/EdgeTransparentGatewayManifest.json) - this file is the IoT Edge deployment manifest for the gateway device.

+> To learn how to deploy the IoT Edge 1.1 or 1.2 runtime to a physical device, see [Create an IoT Edge device](../../iot-edge/how-to-create-iot-edge-device.md) in the IoT Edge documentation.

+# [IoT Edge 1.1](#tab/edge1-1)

+To try out the transparent gateway scenario, select the following button to deploy two Linux virtual machines. One virtual machine has the IoT Edge 1.1 runtime installed and is the transparent IoT Edge gateway. The other virtual machine is a downstream device where you run code to send simulated thermostat telemetry:

+[](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure-Samples%2Fiot-central-docs-samples%2Fmaster%2Ftransparent-gateway-1-1%2FDeployGatewayVMs.json)

+When the two virtual machines are deployed and running, verify the IoT Edge gateway device is running on the `edgegateway` virtual machine:

+1. Go to the **Devices** page in your IoT Central application. If the IoT Edge gateway device is connected to IoT Central, its status is **Provisioned**.

+1. Open the IoT Edge gateway device and verify the status of the modules on the **Modules** page. If the IoT Edge runtime started successfully, the status of the **$edgeAgent** and **$edgeHub** modules is **Running**:

+ :::image type="content" source="media/how-to-connect-iot-edge-transparent-gateway/iot-edge-runtime-1-1.png" alt-text="Screenshot showing the $edgeAgent and $edgeHub version 1.1 modules running on the IoT Edge gateway." lightbox="media/how-to-connect-iot-edge-transparent-gateway/iot-edge-runtime-1-1.png":::

+ > [!TIP]

+ > You may have to wait for several minutes while the virtual machine starts up and the device is provisioned in your IoT Central application.

+# [IoT Edge 1.2](#tab/edge1-2)

+To try out the transparent gateway scenario, select the following button to deploy two Linux virtual machines. One virtual machine has the IoT Edge 1.2 runtime installed and is the transparent IoT Edge gateway. The other virtual machine is a downstream device where you run code to send simulated thermostat telemetry:

+[](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure-Samples%2Fiot-central-docs-samples%2Fmaster%2Ftransparent-gateway-1-2%2FDeployGatewayVMs.json)

+ :::image type="content" source="media/how-to-connect-iot-edge-transparent-gateway/iot-edge-runtime-1-2.png" alt-text="Screenshot showing the $edgeAgent and $edgeHub version 1.2 modules running on the IoT Edge gateway." lightbox="media/how-to-connect-iot-edge-transparent-gateway/iot-edge-runtime-1-2.png":::

+# [IoT Edge 1.1](#tab/edge1-1)

+

+ - *~/certs/certs/iot-edge-device-mycacert-full-chain.cert.pem* - A device CA certificate that's referenced from the IoT Edge configuration file. In a gateway scenario, this CA certificate is how the IoT Edge device verifies its identity to downstream devices.

+ The example shown above assumes you're signed in as **AzureUser** and created a device CA certificate called "mycacert".

+If the runtime doesn't start, check the changes you made in the IoT Edge configuration file and see [Troubleshoot your IoT Edge device](../../iot-edge/troubleshoot.md).

+Your transparent gateway is now configured and ready to start forwarding telemetry from downstream devices.

+# [IoT Edge 1.2](#tab/edge1-2)

+1. Use SSH to connect to and sign in on your gateway device virtual machine.

+1. Run the following commands to clone the IoT Edge repository and generate your demo certificates:

+ ```bash

+ # Clone the repo

+ cd ~

+ git clone https://github.com/Azure/iotedge.git

+

+ # Generate the demo certificates

+ mkdir certs

+ cd certs

+ cp ~/iotedge/tools/CACertificates/*.cnf .

+ cp ~/iotedge/tools/CACertificates/certGen.sh .

+ ./certGen.sh create_root_and_intermediate

+ ./certGen.sh create_edge_device_ca_certificate "mycacert"

+ ```

+ After you run the previous commands, the following files are ready to use in the next steps:

+ - *~/certs/certs/azure-iot-test-only.root.ca.cert.pem* - The root CA certificate used to make all the other demo certificates for testing an IoT Edge scenario.

+ - *~/certs/certs/iot-edge-device-mycacert-full-chain.cert.pem* - A device CA certificate that's referenced from the IoT Edge configuration file. In a gateway scenario, this CA certificate is how the IoT Edge device verifies its identity to downstream devices.

+ - *~/certs/private/iot-edge-device-mycacert.key.pem* - The private key associated with the device CA certificate.

+ To learn more about these demo certificates, see [Create demo certificates to test IoT Edge device features](../../iot-edge/how-to-create-test-certificates.md).

+1. Open the *config.toml* file in a text editor. For example:

+ ```bash

+ sudo nano /etc/aziot/config.toml

+ ```

+1. Locate the `Certificate settings` settings. Add the certificate settings as follows:

+ ```text

+ trust_bundle_cert = "file:///home/AzureUser/certs/certs/azure-iot-test-only.root.ca.cert.pem"

+ [edge_ca]

+ cert = "file:///home/AzureUser/certs/certs/iot-edge-device-ca-mycacert-full-chain.cert.pem"

+ pk = "file:///home/AzureUser/certs/private/iot-edge-device-ca-mycacert.key.pem"

+ ```

+ The example shown above assumes you're signed in as **AzureUser** and created a device CA certificate called "mycacert".

+1. Save the changes and restart the IoT Edge runtime:

+ ```bash

+ sudo iotedge config apply

+ ```

+If the IoT Edge runtime starts successfully after your changes, the status of the **$edgeAgent** and **$edgeHub** modules changes to **Running** on the **Modules** page for your gateway device in IoT Central.

+If the runtime doesn't start, check the changes you made in the IoT Edge configuration file and see [Troubleshoot your IoT Edge device](../../iot-edge/troubleshoot.md).

+ wget https://raw.githubusercontent.com/Azure-Samples/iot-central-docs-samples/master/transparent-gateway-1-1/provision_device.py

+The `leafdevice` virtual machine needs a copy of the root CA certificate you created on the `edgegateway` virtual machine. Copy the */home/AzureUser/certs/certs/azure-iot-test-only.root.ca.cert.pem* file from the `edgegateway` virtual machine to your home directory on the `leafdevice` virtual machine. You can use the **scp** command to copy files between Linux virtual machines. For example, from the `leafdevice` machine:

+```bash

+scp AzureUser@edgegateway:/home/AzureUser/certs/certs/azure-iot-test-only.root.ca.cert.pem .

+```

+ wget https://raw.githubusercontent.com/Azure-Samples/iot-central-docs-samples/master/transparent-gateway-1-1/simple_thermostat.py

+> [!NOTE]

+> Only users in a role that have the necessary permissions can view, create, edit, and delete queries. To learn more, see [Manage users and roles in your IoT Central application](howto-manage-users-roles.md).

+PUT https://{your app subdomain}.azureiotcentral.com/api/organizations/{organizationId}?api-version=2022-05-31

+GET https://{your app subdomain}.azureiotcentral.com/api/organizations/{organizationId}?api-version=2022-05-31

+PATCH https://{your app subdomain}.azureiotcentral.com/api/organizations/{organizationId}?api-version=2022-05-31

+## Use organizations

+### Manage roles

+The REST API lets you list the roles defined in your IoT Central application. Use the following request to retrieve a list of application role and organization role IDs from your application. To learn more see, [How to manage IoT Central organizations](howto-create-organizations.md):

+```http

+GET https://{your app subdomain}.azureiotcentral.com/api/roles?api-version=2022-05-31

+```

+The response to this request looks like the following example that includes the application role and organization role IDs.

+```json

+{

+ "value": [

+ {

+ "id": "ca310b8d-2f4a-44e0-a36e-957c202cd8d4",

+ "displayName": "Administrator"

+ },

+ {

+ "id": "ae2c9854-393b-4f97-8c42-479d70ce626e",

+ "displayName": "Operator"

+ },

+ {

+ "id": "344138e9-8de4-4497-8c54-5237e96d6aaf",

+ "displayName": "Builder"

+ },

+ {

+ "id": "c495eb57-eb18-489e-9802-62c474e5645c",

+ "displayName": "Org Admin"

+ },

+ {

+ "id": "b4935647-30e4-4ed3-9074-dcac66c2f8ef",

+ "displayName": "Org Operator"

+ },

+ {

+ "id": "84cc62c1-dabe-49d3-b16e-8b291232b285",

+ "displayName": "Org Viewer"

+ }

+ ]

+}

+```

+### Create an API token to a node in an organization hierarchy

+Use the following request to create Create an API token to a node in an organization hierarchy in your application:

+```http

+PUT https://{your app subdomain}.azureiotcentral.com/api/apiTokens/{tokenId}?api-version=2022-05-31

+```

+* tokenId - Unique ID of the token

+The following example shows a request body that creates an API token for an organization in a IoT Central application.

+```json

+{

+ "roles": [

+ {

+ "role": "84cc62c1-dabe-49d3-b16e-8b291232b285",

+ "organization": "seattle"

+ }

+ ]

+}

+```

+The request body has some required fields:

+|Name|Description|

+|-|--|

+|role |ID of one of the organization roles.|

+|organization| ID of the organization|

+The response to this request looks like the following example:

+```json

+{

+ "id": "token1",

+ "roles": [

+ {

+ "role": "84cc62c1-dabe-49d3-b16e-8b291232b285",

+ "organization": "seattle"

+ }

+ ],

+ "expiry": "2023-07-07T17:05:08.407Z",

+ "token": "SharedAccessSignature sr=8a0617**********************4c0d71c&sig=3RyX69G4%2FBZZnG0LXOjQv*************e8s%3D&skn=token1&se=1688749508407"

+}

+```

+### Associate a user with a node in an organization hierarchy

+Use the following request to create and associate a user with a node in an organization hierarchy in your application. The ID and email must be unique in the application:

+```http

+PUT https://{your app subdomain}.azureiotcentral.com/api/users/user-001?api-version=2022-05-31

+```

+In the following request body, the `role` is the ID of one of the organization roles and `organization` is the ID of the organization

+```json

+{

+ "id": "user-001",

+ "type": "email",

+ "roles": [

+ {

+ "role": "84cc62c1-dabe-49d3-b16e-8b291232b285",

+ "organization": "seattle"

+ }

+ ],

+ "email": "user5@contoso.com"

+}

+```

+The response to this request looks like the following example. The role value identifies which role the user is associated with:

+```json

+{

+ "id": "user-001",

+ "type": "email",

+ "roles": [

+ {

+ "role": "84cc62c1-dabe-49d3-b16e-8b291232b285",

+ "organization": "seattle"

+ }

+ ],

+ "email": "user5@contoso.com"

+}

+```

+### Add and associate a device to an organization

+Use the following request to associate a new device with an organization

+```http

+PUT https://{your app subdomain}.azureiotcentral.com/api/devices/{deviceId}?api-version=2022-05-31

+```

+The following example shows a request body that adds a device for a device template. You can get the `template` details from the device templates page in IoT Central application UI.

+```json

+{

+ "displayName": "CheckoutThermostat",

+ "template": "dtmi:contoso:Thermostat;1",

+ "simulated": true,

+ "enabled": true,

+ "organizations": [

+ "seattle"

+ ]

+}

+```

+The request body has some required fields:

+* `@displayName`: Display name of the device.

+* `@enabled`: declares that this object is an interface.

+* `@etag`: ETag used to prevent conflict in device updates.

+* `simulated`: Whether the device is simulated.

+* `template` : The device template definition for the device.

+* `organizations` : List of organization IDs that the device is a part of. Currently, you can only associate a device with a single organization.

+The response to this request looks like the following example:

+```json

+{

+ "id": "thermostat1",

+ "etag": "eyJoZWFkZXIiOiJcIjI0MDAwYTdkLTAwMDAtMDMwMC0wMDAwLTYxYjgxZDIwMDAwMFwiIiwiZGF0YSI6IlwiMzMwMDQ1M2EtMDAwMC0wMzAwLTAwMDAtNjFiODFkMjAwMDAwXCIifQ",

+ "displayName": "CheckoutThermostat",

+ "simulated": true,

+ "provisioned": false,

+ "template": "dtmi:contoso:Thermostat;1",

+ "enabled": true,

+ "organizations": [

+ "seattle"

+ ]

+}

+```

+### Add and associate a device group to an organization

+Use the following request to create and associate a new device group with an organization.

+```http

+PUT https://{your app subdomain}.azureiotcentral.com/api/deviceGroups/{deviceGroupId}?api-version=2022-05-31

+```

+When you create a device group, you define a `filter` that selects the devices to add to the group. A `filter` identifies a device template and any properties to match. The following example creates device group that contains all devices associated with the "dtmi:modelDefinition:dtdlv2" template where the `provisioned` property is true.

+```json

+{

+ "displayName": "Device group 1",

+ "description": "Custom device group.",

+ "filter": "SELECT * FROM devices WHERE $template = \"dtmi:modelDefinition:dtdlv2\" AND $provisioned = true",

+ "organizations": [

+ "seattle"

+ ]

+}

+```

+The request body has some required fields:

+* `@displayName`: Display name of the device group.

+* `@filter`: Query defining which devices should be in this group.

+* `description`: Short summary of device group.

+* `organizations` : List of organization IDs that the device is a part of. Currently, you can only associate a device with a single organization.

+The response to this request looks like the following example:

+```json

+{

+ "id": "group1",

+ "displayName": "Device group 1",

+ "description": "Custom device group.",

+ "filter": "SELECT * FROM devices WHERE $template = \"dtmi:modelDefinition:dtdlv2\" AND $provisioned = true",

+ "organizations": [

+ "seattle"

+ ]

+}

+```

+sudo apt-get autoremove iotedge

+sudo apt-get autoremove --purge aziot-edge

+Leave out the `--purge` flag if you plan to reinstall IoT Edge and use the same configuration information in the future. The `--purge` flags deletes all the files associated with IoT Edge, including your configuration files.

+sudo apt-get autoremove --purge moby-engine

+sudo apt-get autoremove iotedge

+sudo apt-get autoremove --purge aziot-edge

+Leave out the `--purge` flag if you plan to reinstall IoT Edge and use the same configuration information in the future. The `--purge` flags deletes all the files associated with IoT Edge, including your configuration files.

+sudo apt-get autoremove --purge moby-engine

+| [Debian 10 ¹](https://www.debian.org/releases/buster/) |  |  |  |

+| [Ubuntu 18.04 ²](https://wiki.ubuntu.com/BionicBeaver/ReleaseNotes) | |  | |

+| [Ubuntu 20.04 ²](https://wiki.ubuntu.com/FocalFossa/ReleaseNotes) | |  | |

+¹ With the release of 1.3, there are new system calls that cause crashes in Debian 10. To see the workaround, view the [Known issue: Debian 10 (Buster) on ARMv7](https://github.com/Azure/azure-iotedge/releases) section of the 1.3 release notes for details.

+² Installation packages are made available on the [Azure IoT Edge releases](https://github.com/Azure/azure-iotedge/releases). See the installation steps in [Offline or specific version installation](how-to-provision-single-device-linux-symmetric.md#offline-or-specific-version-installation-optional).

+# Device Update for IoT Hub configuration file

+The Device Update agent gets its configuration information from the `du-config.json` file on the device. The agent reads these values and reports them to the Device Update service:

+* connectionData

+When installing Debian agent on an IoT Device with a Linux OS, modify the `/etc/adu/du-config.json` file to update values. For a Yocto build system, in the partition or disk called `adu`, create a json file called `/adu/du-config.json`.

+| Name | Description |

+| SchemaVersion | The schema version that maps the current configuration file format version. |

+| aduShellTrustedUsers | The list of users that can launch the **adu-shell** program. Note, adu-shell is a broker program that does various update actions as 'root'. The Device Update default content update handlers invoke adu-shell to do tasks that require super user privilege. Examples of tasks that require this privilege are `apt-get install` or executing a privileged script. |

+| aduc_manufacturer | Reported by the **AzureDeviceUpdateCore:4.ClientMetadata:4** interface to classify the device for targeting the update deployment. |

+| aduc_model | Reported by the **AzureDeviceUpdateCore:4.ClientMetadata:4** interface to classify the device for targeting the update deployment. |

+| connectionType | Accepted values are `string` or `AIS`. Use `string` when connecting the device to IoT Hub manually for testing purposes. For production scenarios, use `AIS` when using the IoT Identity Service to connect the device to IoT Hub. For more information, see [understand IoT Identity Service configurations](https://azure.github.io/iot-identity-service/configuration.html). |

+| connectionData |If connectionType = "string", add your IoT device's device or module connection string here. If connectionType = "AIS", set the connectionData to empty string (`"connectionData": ""`). |

+| manufacturer | Reported by the Device Update agent as part of the **DeviceInformation** interface. |

+| model | Reported by the Device Update agent as part of the **DeviceInformation** interface. |

+```json

+# Device Update for IoT Hub error codes

+The Device Update for IoT Hub Core PnP interface reports `ResultCode` and `ExtendedResultCode`, which can be used to diagnose failures. For more information about the Device Update Core PnP interface, see [Device Update and Plug and Play](device-update-plug-and-play.md).

+For more information about parsing codes, see [Device Update Agent result codes and extended result codes](https://github.com/Azure/iot-hub-device-update/tree/main/docs/agent-reference/device-update-agent-extended-result-codes.md) or [implement a custom Content Handler](https://github.com/Azure/iot-hub-device-update/tree/main/src/content_handlers).

+The following table lists error codes pertaining to the content service component of the Device Update service. The content service component is responsible for importing update content. More troubleshooting information is also available for [importing proxy updates](device-update-proxy-update-troubleshooting.md).

+| Error code | String error | Next steps |

+|--|--|--|

+| UpdateAlreadyExists | Update with the same identity already exists. | Make sure you're importing an update that hasnΓÇÖt already been imported into this instance of Device Update for IoT Hub. |

+| DuplicateContentImport | Identical content imported simultaneously multiple times. | Make sure you're importing an update that hasnΓÇÖt already been imported into this instance of Device Update for IoT Hub. |

+| CannotProcessImportManifest | Error processing import manifest. | Refer to [import concepts](import-concepts.md) and [import update](create-update.md) documentation for proper import manifest formatting. |

+| CannotDownload | Cannot download import manifest. | Check to make sure the URL for the import manifest file is still valid. |

+| CannotParse | Cannot parse import manifest. | Check your import manifest for accuracy against the schema defined in the [import update](create-update.md) documentation. |

+| UnsupportedVersion | Import manifest schema version is not supported. | Make sure your import manifest is using the latest schema defined in the [import update](create-update.md) documentation. |

+| Error importing update due to exceeded limit. | Cannot import additional update provider. | You've reached a [limit](device-update-limits.md) on the number of different __providers__ allowed in your instance of Device Update for IoT Hub. Delete some updates from your instance and try again. |

+| Error importing update due to exceeded limit. | Cannot import additional update name for the specified provider. | You've reached a [limit](device-update-limits.md) on the number of different __names__ allowed under one provider in your instance of Device Update for IoT Hub. Delete some updates from your instance and try again. |

+| Error importing update due to exceeded limit. | Cannot import additional update version for the specified provider and name. | You've reached a [limit](device-update-limits.md) on the number of different __versions__ allowed under one provider and name in your instance of Device Update for IoT Hub. Delete some updates with that name from your instance and try again. |

+| Error importing update due to exceeded limit. | Cannot import additional update provider with the specified compatibility.<br><br>_or_<br><br>Cannot import additional update name with the specified compatibility.<br><br>_or_<br><br>Cannot import additional update version with the specified compatibility. | When defining [compatibility properties](import-schema.md#compatibility-object) in an import manifest, keep in mind that Device Update for IoT Hub supports a single provider and name combination for a given set of compatibility properties. If you try to use the same compatibility properties with more than one provider/name combination, you'll see these errors. To resolve this issue, make sure that all updates for a given device (as defined by compatibility properties) use the same provider and name. |

+| CannotProcessUpdateFile | Error processing source file. | |

+| ContentFileCannotDownload | Cannot download source file. | Check to make sure the URL for the update file(s) is still valid. |

+| SourceFileMalwareDetected | A known malware signature was detected in a file being imported. | Content imported into Device Update for IoT Hub is scanned for malware by several different mechanisms. If a known malware signature is identified, the import fails and a unique error message is returned. The error message contains the description of the malware signature, and a file hash for each file where the signature was detected. You can use the file hash to find the exact file being flagged, and use the description of the malware signature to check that file for malware. <br><br>Once you have removed the malware from any files being imported, you can start the import process again. |

+| SourceFilePendingMalwareAnalysis | A signature was detected in a file being imported that may indicate malware is present. | Content imported into Device Update for IoT Hub is scanned for malware by several different mechanisms. The import fails if a scan signature has characteristics of malware, even if there is not an exact match to known malware. When this occurs, a unique error message is returned. The error message contains the description of the suspected malware signature, and a file hash for each file where the signature was detected. You can use the file hash to find the exact file being flagged, and use the description of the malware signature to check that file for malware.<br><br>Once you've removed the malware from any files being imported, you can start the import process again. If you're certain your files are free of malware and continue to see this error, use the [Contact Microsoft Support](troubleshoot-device-update.md#contact) process. |

+## Next steps

+[Troubleshoot issues with Device Update](.\troubleshoot-device-update.md)

+In order to deploy an update to devices from Device Update for IoT Hub, you first have to import that update into the Device Update service. This article provides an overview of some important concepts to understand when it comes to importing updates.

+An import manifest is a JSON file that defines important information about the update that you're importing. You submit both your import manifest and associated update file or files (such as a firmware update package) as part of the import process. The metadata that is defined in the import manifest is used to ingest the update. Some of the metadata is also used at deployment time - for example, to validate if an update was installed correctly.

+For example:

+The import manifest contains several items that represent important Device Update for IoT Hub concepts. These items are outlined in this section. For information about the full import schema, see [Import manifest JSON schema](./import-schema.md).

+### Update identity

+The *update identity* or *updateId* is the unique identifier for an update in Device Update for IoT Hub. It's composed of three parts:

+- **Version**: a version number distinguishing this update from others that have the same provider and name.

+For example:

+```json

+{

+ "updateId": {

+ "provider": "Contoso",

+ "name": "Toaster",

+ "version": "1.0"

+ }

+}

+```

+> UpdateId is used by the Device Update service only, and may be different from the identities of actual software components on the device.

+*Compatibility* defines the criteria of a device that can install the update. It contains device properties that are a set of arbitrary key value pairs that are reported from a device. Only devices with matching properties will be eligible for deployment. An update may be compatible with multiple device classes by having more than one set of device properties.

+Here's an example of an update that can only be deployed to a device that reports *Contoso* and *Toaster* as its device manufacturer and model.

+The *Instructions* part contains the necessary information or *steps* for device agent to install the update. The simplest update contains a single inline step. That step executes the included payload file using a *handler* registered with the device agent:

+An update may contain *reference* steps that instruct the device agent to install another update with its own import manifest altogether, establishing a parent and child update relationship. For example, an update for a toaster may contain two child updates:

+> An update may contain any combination of inline and reference steps.

+The *Files* part contains the metadata of update payload files like their names, sizes, and hash. Device Update for IoT Hub uses this metadata for integrity validation during the import process. The same information is then forwarded to the device agent to repeat the integrity validation prior to installation.

+> An update that contains only reference steps won't have any update payload file in the parent update.

+> An import manifest JSON filename must end with `.importmanifest.json` when imported through Azure portal.

+Certain limits are enforced for each Device Update for IoT Hub instance. If you haven't already reviewed them, see [Device Update limits](./device-update-limits.md).

+- To learn more about the import process, see [Prepare an update to import](./create-update.md).

+- Review the [Import manifest schema](./import-schema.md).

+# Importing updates into Device Update for IoT Hub: schema and other information

+If you want to import an update into Device Update for IoT Hub, be sure you've reviewed the [concepts](import-concepts.md) and [how-to guide](import-update.md) first. If you're interested in the details of import manifest schema, or information about API permissions, see below.

+The import manifest JSON schema is hosted at [SchemaStore.org](https://json.schemastore.org/azure-deviceupdate-import-manifest-4.0.json).

+## Schema

+|Property|Type|Description|Required|

+|**description**|`string`|Optional update description.<br><br>Maximum length: 512 characters|No|

+|**createdDateTime**|`string`|Date & time import manifest was created in ISO 8601 format.<br><br>Example: `"2020-10-02T22:18:04.9446744Z"`|Yes|

+Additional properties aren't allowed.

+## updateId object

+The *updateID* object is a unique identifier for each update.

+|Property|Type|Description|Required|

+|**provider**|`string`|Entity who is creating or directly responsible for the update. It can be a company name.<br><br>Pattern: `^[a-zA-Z0-9.-]+$`<br>Maximum length: 64 characters|Yes|

+|**name**|`string`|Identifier for a class of update. It can be a device class or model name.<br><br>Pattern: `^[a-zA-Z0-9.-]+$`<br>Maximum length: 64 characters|Yes|

+|**version**|`string`|Two- to four-part dot-separated numerical version numbers. Each part must be a number between 0 and 2147483647 and leading zeroes will be dropped.<br><br>Pattern: `^\d+(?:\.\d+)+$`<br>Examples: `"1.0"`, `"2021.11.8"`|Yes|

+Additional properties aren't allowed.

+For example:

+```json

+{

+ "updateId": {

+ "provider": "Contoso",

+ "name": "Toaster",

+ "version": "1.0"

+ }

+}

+```

+## compatibility object

+The *compatibility* object describes the properties of a device that this update is compatible with.

+The same exact set of compatibility properties can't be used with more than one update provider and name combination.

+For example:

+```json

+{

+ "compatibility": [

+ {

+ "deviceManufacturer": "Contoso",

+ "deviceModel": "Toaster"

+ }

+ ]

+}

+```

+## instructions object

+The *instructions* object provides the update installation instructions. The instructions object contains a list of steps to be performed. Steps can either be code to execute or a pointer to another update.

+|Property|Type|Description|Required|

+|**steps**|`array[1-10]`|Each element in the array must be either an [inlineStep object](#inlinestep-object) or a [referenceStep object](#referencestep-object).|Yes|

+Additional properties aren't allowed.

+For example:

+```json

+{

+ "instructions": {

+ "steps": [

+ {

+ "type": "inline",

+ ...

+ },

+ {

+ "type": "reference",

+ ...

+ }

+ ]

+ }

+}

+```

+## inlineStep object

+An *inline* step object is an installation instruction step that performs code execution.

+|Property|Type|Description|Required|

+|||||

+|**type**|`string`|Instruction step type that performs code execution. Must be `inline`.<br><br>Defaults to `inline` if no value is provided.|No|

+|**description**|`string`|Optional instruction step description.<br><br>Maximum length: 64 characters|No|

+|**handler**|`string`|Identity of the handler on the device that can execute this step.<br><br>Pattern: `^\S+/\S+:\d{1,5}$`<br>Minimum length: 5 characters<br>Maximum length: 32 characters<br>Examples: `microsoft/script:1`, `microsoft/swupdate:1`, `microsoft/apt:1` |Yes|

+|**files**|`string` `[1-10]`| Names of update files defined as [file objects](#file-object) that the agent will pass to the handler. Each element in the array must have length between 1 and 255 characters. |Yes|

+Additional properties aren't allowed.

+For example:

+```json

+{

+ "steps": [

+ {

+ "description": "pre-install script",

+ "handler": "microsoft/script:1",

+ "handlerProperties": {

+ "arguments": "--pre-install"

+ },

+ "files": [

+ "configure.sh"

+ ]

+ }

+ ]

+}

+```

+## referenceStep object

+A *reference* step object is an installation instruction step that installs another update.

+|Property|Type|Description|Required|

+|**type**|`referenceStepType`|Instruction step type that installs another update. Must be `reference`.|Yes|

+|**description**|`stepDescription`|Optional instruction step description.<br><br>Maximum length: 64 characters |No|

+Additional properties aren't allowed.

+For example:

+```json

+{

+ "steps": [

+ {

+ "type": "reference",

+ "updateId": {

+ "provider": "Contoso",

+ "name": "Toaster.HeatingElement",

+ "version": "1.0"

+ }

+ }

+ ]

+}

+```

+## file object

+A *file* object is an update payload file, for example, binary, firmware, script, etc. Each file object must be unique within an update.

+|Property|Type|Description|Required|

+|**filename**|`string`|Update payload file name.<br><br>Maximum length: 255 characters|Yes|

+|**sizeInBytes**|`number`|File size in number of bytes.<br><br>Maximum size: 2147483648 bytes|Yes|

+Additional properties aren't allowed.

+For example:

+```json

+{

+ "files": [

+ {

+ "filename": "configure.sh",

+ "sizeInBytes": 7558,

+ "hashes": {...}

+ }

+ ]

+}

+```

+## fileHashes object

+Base64-encoded file hashes with the algorithm name as key. At least the SHA-256 algorithm must be specified, and other algorithms may be specified if supported by the agent. For an example of how to calculate the hash correctly, see the Get-AduFileHashes function in [AduUpdate.psm1 script](https://github.com/Azure/iot-hub-device-update/blob/main/tools/AduCmdlets/AduUpdate.psm1).

+|Property|Type|Description|Required|

+For example:

+```json

+{

+ "hashes": {

+ "sha256": "/CD7Sn6fiknWa3NgcFjGlJ+ccA81s1QAXX4oo5GHiFA="

+ }

+}

+```

+### Device State Events

+Device connection state events are available for devices connecting using either the MQTT or AMQP protocol, or using either of these protocols over WebSockets. Requests made only with HTTPS won't trigger device connection state notifications.

+* For devices connecting using Java, Node, or Python [Azure IoT SDKs](iot-hub-devguide-sdks.md) with the [MQTT protocol](iot-hub-mqtt-support.md) will have connection states sent automatically.

+* For devices connecting using the Java, Node, or Python [Azure IoT SDKs](iot-hub-devguide-sdks.md) with the [AMQP protocol](iot-hub-amqp-support.md), a cloud-to-device link should be created to reduce any delay in accurate connection states.

+* For devices connecting using the .NET [Azure IoT SDK](iot-hub-devguide-sdks.md) with the [MQTT](iot-hub-mqtt-support.md) or [AMQP](iot-hub-amqp-support.md) protocol wonΓÇÖt send a device connected event until an initial device-to-cloud or cloud-to-device message is sent/received.

+* Outside of the Azure IoT SDKs, in MQTT these operations equate to SUBSCRIBE or PUBLISH operations on the appropriate messaging [topics](iot-hub-mqtt-support.md). Over AMQP these equate to attaching or transferring a message on the [appropriate link paths](iot-hub-amqp-support.md).

+### Device State Interval

+

+ Title: From artifacts to models in MLflow

+description: Learn about how MLflow uses the concept of models instead of artifacts to represent your trained models and enable a streamlined path to deployment.

+# From artifacts to models in MLflow

+The following article explains the differences between an artifact and a model in MLflow and how to transition from one to the other. It also explains how Azure Machine Learning uses the MLflow model's concept to enabled streamlined deployment workflows.

+## What's the difference between an artifact and a model?

+If you are not familiar with MLflow, you may not be aware of the difference between logging artifacts or files vs. logging MLflow models. There are some fundamental differences between the two:

+### Artifacts

+Any file generated (and captured) from an experiment's run or job is an artifact. It may represent a model serialized as a Pickle file, the weights of a PyTorch or TensorFlow model, or even a text file containing the coefficients of a linear regression. Other artifacts can have nothing to do with the model itself, but they can contain configuration to run the model, pre-processing information, sample data, etc. As you can see, an artifact can come in any format.

+You can log artifacts in MLflow in a similar way you log a file with Azure ML SDK v1:

+```python

+filename = 'model.pkl'

+with open(filename, 'wb') as f:

+ pickle.dump(model, f)

+mlflow.log_artifact(filename)

+```

+### Models

+A model in MLflow is also an artifact, as it matches the definition we introduced above. However, we make stronger assumptions about this type of artifacts. Such assumptions allow us to create a clear contract between the saved artifacts and what they mean. When you log your models as artifacts (simple files), you need to know what the model builder meant for each of them in order to know how to load the model for inference. When you log your models as a Model entity, you should be able to tell what it is based on the contract mentioned.

+Logging models has the following advantages:

+> [!div class="checklist"]

+> * You don't need to provide an scoring script nor an environment for deployment.

+> * Swagger is enabled in endpoints automatically and the __Test__ feature can be used in Azure ML studio.

+> * Models can be used as pipelines inputs directly.

+> * You can use the Responsable AI dashbord.

+## The MLModel format

+MLflow adopts the MLModel format as a way to create a contract between the artifacts and what they represent. The MLModel format stores assets in a folder. Among them, there is a particular file named MLModel. This file is the single source of truth about how a model can be loaded and used.

+The following example shows how the `MLmodel` file for a computer version model trained with `fastai` may look like:

+__MLmodel__

+```yaml

+artifact_path: classifier

+flavors:

+ fastai:

+ data: model.fastai

+ fastai_version: 2.4.1

+ python_function:

+ data: model.fastai

+ env: conda.yaml

+ loader_module: mlflow.fastai

+ python_version: 3.8.12

+model_uuid: e694c68eba484299976b06ab9058f636

+run_id: e13da8ac-b1e6-45d4-a9b2-6a0a5cfac537

+signature:

+ inputs: '[{"type": "tensor",

+ "tensor-spec":

+ {"dtype": "uint8", "shape": [-1, 300, 300, 3]}

+ }]'

+ outputs: '[{"type": "tensor",

+ "tensor-spec":

+ {"dtype": "float32", "shape": [-1,2]}

+ }]'

+```

+### The model's flavors

+Considering the variety of machine learning frameworks available to use, MLflow introduced the concept of flavor as a way to provide a unique contract to work across all of them. A flavor indicates what to expect for a given model created with a specific framework. For instance, TensorFlow has its own flavor, which specifies how a TensorFlow model should be persisted and loaded. Because each model flavor indicates how they want to persist and load models, the MLModel format doesn't enforce a single serialization mechanism that all the models need to support. Such decision allows each flavor to use the methods that provide the best performance or best support according to their best practices - without compromising compatibility with the MLModel standard.

+The following is an example of the `flavors` section for an `fastai` model.

+```yaml

+flavors:

+ fastai:

+ data: model.fastai

+ fastai_version: 2.4.1

+ python_function:

+ data: model.fastai

+ env: conda.yaml

+ loader_module: mlflow.fastai

+ python_version: 3.8.12

+```

+### Signatures

+[Model signatures in MLflow](https://www.mlflow.org/docs/latest/models.html#model-signature) are an important part of the model specification, as they serve as a data contract between the model and the server running our models. They are also important for parsing and enforcing model's input's types at deployment time. [MLflow enforces types when data is submitted to your model if a signature is available](https://www.mlflow.org/docs/latest/models.html#signature-enforcement).

+Signatures are indicated when the model gets logged and persisted in the `MLmodel` file, in the `signature` section. **Autolog's** feature in MLflow automatically infers signatures in a best effort way. However, it may be required to [log the models manually if the signatures inferred are not the ones you need](https://www.mlflow.org/docs/latest/models.html#how-to-log-models-with-signatures).

+There are two types of signatures:

+* **Column-based signature** corresponding to signatures that operate to tabular data. Models with this signature can expect to receive `pandas.DataFrame` objects as inputs.

+* **Tensor-based signature:** corresponding to signatures that operate with n-dimensional arrays or tensors. Models with this signature can expect to receive a `numpy.ndarray` as inputs (or a dictionary of `numpy.ndarray` in the case of named-tensors).

+The following example corresponds to a computer vision model trained with `fastai`. This model receives a batch of images represented as tensors of shape `(300, 300, 3)` with the RGB representation of them (unsigned integers). It outputs batches of predictions (probabilities) for two classes.

+__MLmodel__

+```yaml

+signature:

+ inputs: '[{"type": "tensor",

+ "tensor-spec":

+ {"dtype": "uint8", "shape": [-1, 300, 300, 3]}

+ }]'

+ outputs: '[{"type": "tensor",

+ "tensor-spec":

+ {"dtype": "float32", "shape": [-1,2]}

+ }]'

+```

+> [!TIP]

+> Azure Machine Learning generates Swagger endpoints for MLflow models with a signature available. This makes easier to test deployed endpoints using the Azure ML studio.

+### Model's environment

+Requirements for the model to run are specified in the `conda.yaml` file. Dependencies can be automatically detected by MLflow or they can be manually indicated when you call `mlflow.<flavor>.log_model()` method. The latter can be needed in cases that the libraries included in your environment are not the ones you intended to use.

+The following is an example of an environment used for a model created with `fastai` framework:

+__conda.yaml__

+```yaml

+channels:

+- conda-forge

+dependencies:

+- python=3.8.5

+- pip

+- pip:

+ - mlflow

+ - astunparse==1.6.3

+ - cffi==1.15.0

+ - configparser==3.7.4

+ - defusedxml==0.7.1

+ - fastai==2.4.1

+ - google-api-core==2.7.1

+ - ipython==8.2.0

+ - psutil==5.9.0

+name: mlflow-env

+```

+> [!NOTE]

+> MLflow environments and Azure Machine Learning environments are different concepts. While the former opperates at the level of the model, the latter operates at the level of the workspace (for registered environments) or jobs/deployments (for annonymous environments). When you deploy MLflow models in Azure Machine Learning, the model's environment is built and used for deployment. Alternatively, you can override this behaviour with the [Azure ML CLI v2](concept-v2.md) and deploy MLflow models using a specific Azure Machine Learning environments.

+### Model's predict function

+All MLflow models contain a `predict` function. This function is the one that is called when a model is deployed using a no-code-deployment experience. What the `predict` function returns (classes, probabilities, a forecast, etc.) depend on the framework (i.e. flavor) used for training. Read the documentation of each flavor to know what they return.

+In same cases, you may need to customize this function to change the way inference is executed. On those cases, you will need to [log models with a different behavior in the predict method](how-to-log-mlflow-models.md#logging-models-with-a-different-behavior-in-the-predict-method) or [log a custom model's flavor](how-to-log-mlflow-models.md#logging-custom-models).

+## Start logging models

+We recommend starting taking advantage of MLflow models in Azure Machine Learning. There are different ways to start using the model's concept with MLflow. Read [How to log MLFlow models](how-to-log-mlflow-models.md) to a comprehensive guide.

+Azure Machine Learning uses MLflow Tracking for metric logging and artifact storage for your experiments, whether you created the experiment via the Azure Machine Learning Python SDK, Azure Machine Learning CLI or the Azure Machine Learning studio. Learn more at [Log & view metrics and log files with MLflow](how-to-log-view-metrics.md).

+MLflow provides support for a variety of [machine learning frameworks](https://mlflow.org/docs/latest/models.html#built-in-model-flavors) (scikit-learn, Keras, Pytorch, and more); however, it might not cover every use case. For example, you may want to create an MLflow model with a framework that MLflow does not natively support or you may want to change the way your model does pre-processing or post-processing when running jobs. To know more about MLflow models read [From artifacts to models in MLflow](concept-mlflow-models.md).

+> [!IMPORTANT]

+> If you are used to deploying models using scoring scripts and custom environments and you are looking to know how to achieve the same functionality using MLflow models, we recommend reading [Using MLflow models for no-code deployment](how-to-log-mlflow-models.md).

+ Title: Logging MLflow models

+description: Learn how to start logging MLflow models instead of artifacts using MLflow SDK in Azure Machine Learning.

+# Logging MLflow models

+The following article explains how to start logging your trained models (or artifacts) as MLflow models. It explores the different methods to customize the way MLflow packages your models and hence how it runs them.

+## Why logging models instead of artifacts?

+If you are not familiar with MLflow, you may not be aware of the difference between logging artifacts or files vs. logging MLflow models. We recommend reading the article [From artifacts to models in MLflow](concept-mlflow-models.md) for an introduction to the topic.

+A model in MLflow is also an artifact, but with a specific structure that serves as a contract between the person that created the model and the person that intends to use it. Such contract helps build the bridge about the artifacts themselves and what they mean.

+Logging models has the following advantages:

+> [!div class="checklist"]

+> * You don't need to provide a scoring script nor an environment for deployment.

+> * Swagger is enabled in endpoints automatically and the __Test__ feature can be used in Azure ML studio.

+> * Models can be used as pipelines inputs directly.

+> * You can use the Responsable AI dashbord.

+There are different ways to start using the model's concept in Azure Machine Learning with MLflow, as explained in the following sections:

+## Logging models using autolog

+One of the simplest ways to start using this approach is by using MLflow autolog functionality. Autolog allows MLflow to instruct the framework associated to with the framework you are using to log all the metrics, parameters, artifacts and models that the framework considers relevant. By default, most models will be log if autolog is enabled. Some flavors may decide not to do that in specific situations. For instance, the flavor PySpark won't log models if they exceed a certain size.

+You can turn on autologging by using either `mlflow.autolog()` or `mlflow.<flavor>.autolog()`. The following example uses `autolog()` for logging a classifier model trained with XGBoost:

+```python

+import mlflow

+from xgboost import XGBClassifier

+from sklearn.metrics import accuracy_score

+with mlflow.start_run():

+ mlflow.autolog()

+ model = XGBClassifier(use_label_encoder=False, eval_metric="logloss")

+ model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)

+ y_pred = model.predict(X_test)

+ accuracy = accuracy_score(y_test, y_pred)

+```

+> [!TIP]

+> If you are using Machine Learning pipelines, like for instance [Scikit-Learn pipelines](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html), use the `autolog` functionality of that flavor for logging models. Models are automatically logged when the `fit()` method is called on the pipeline object. The notebook [Training and tracking an XGBoost classifier with MLflow](https://github.com/Azure/azureml-examples/blob/main/notebooks/using-mlflow/train-with-mlflow/xgboost_classification_mlflow.ipynb) demostrates how to log a model with preprocessing using pipelines.

+## Logging models with a custom signature, environment or samples

+You can log models manually using the method `mlflow.<flavor>.log_model` in MLflow. Such workflow has the advantages of retaining control of different aspects of how the model is logged.

+Use this method when:

+> [!div class="checklist"]

+> * You want to indicate pip packages or a conda environment different from the ones that are automatically detected.

+> * You want to include input examples.

+> * You want to include specific artifacts into the package that will be needed.

+> * Your signature is not correctly inferred by `autolog`. This is specifically important when you deal with inputs that are tensors where the signature needs specific shapes.

+> * Somehow the default behavior of autolog doesn't fill your purpose.

+The following example code logs a model for an XGBoost classifier:

+```python

+import mlflow

+from xgboost import XGBClassifier

+from sklearn.metrics import accuracy_score

+from mlflow.models import infer_signature

+from mlflow.utils.environment import _mlflow_conda_env

+with mlflow.start_run():

+ mlflow.autolog(log_models=False)

+ model = XGBClassifier(use_label_encoder=False, eval_metric="logloss")

+ model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)

+ y_pred = model.predict(X_test)

+ accuracy = accuracy_score(y_test, y_pred)

+

+ # Signature

+ signature = infer_signature(X_test, y_test)

+

+ # Conda environment

+ custom_env =_mlflow_conda_env(

+ additional_conda_deps=None,

+ additional_pip_deps=["xgboost==1.5.2"],

+ additional_conda_channels=None,

+ )

+

+ # Sample

+ input_example = X_train.sample(n=1)

+ # Log the model manually

+ mlflow.xgboost.log_model(model,

+ artifact_path="classifier",

+ conda_env=custom_env,

+ signature=signature,

+ input_example=input_example)

+```

+> [!NOTE]

+> * `log_models=False` is configured in `autolog`. This prevents MLflow to automatically log the model, as it is done manually later.

+> * `infer_signature` is a convenient method to try to infer the signature directly from inputs and outpus.

+> * `mlflow.utils.environment._mlflow_conda_env` is a private method in MLflow SDK and it may change in the future. This example uses it just for sake of simplicity, but it must be used with caution or generate the YAML definition manually as a Python dictionary.

+## Logging models with a different behavior in the predict method

+When you log a model using either `mlflow.autolog` or using `mlflow.<flavor>.log_model`, the flavor used for the model decides how inference should be executed and what gets returned by the model. MLflow doesn't enforce any specific behavior in how the `predict` generate results. There are scenarios where you probably want to do some pre-processing or post-processing before and after your model is executed.

+A solution to this scenario is to implement machine learning pipelines that moves from inputs to outputs directly. Although this is possible (and sometimes encourageable for performance considerations), it may be challenging to achieve. For those cases, you probably want to [customize how your model does inference using a custom models](#logging-custom-models) as explained in the following section.

+## Logging custom models

+MLflow provides support for a variety of [machine learning frameworks](https://mlflow.org/docs/latest/models.html#built-in-model-flavors) including FastAI, MXNet Gluon, PyTorch, TensorFlow, XGBoost, CatBoost, h2o, Keras, LightGBM, MLeap, ONNX, Prophet, spaCy, Spark MLLib, Scikit-Learn, and statsmodels. However, they may be times where you need to change how a flavor works, log a model not natively supported by MLflow or even log a model that uses multiple elements from different frameworks. For those cases, you may need to create a custom model flavor.

+For this type of models, MLflow introduces a flavor called `pyfunc` (standing from Python function). Basically this flavor allows you to log any object you want as a model, as long as it satisfies two conditions:

+* You implement the method `predict` (at least).

+* The Python object inherits from `mlflow.pyfunc.PythonModel`.

+> [!TIP]

+> Serializable models that implements the Scikit-learn API can use the Scikit-learn flavor to log the model, regardless of whether the model was built with Scikit-learn. If your model can be persisted in Pickle format and the object has methods `predict()` and `predict_proba()` (at least), then you can use `mlflow.sklearn.log_model()` to log it inside a MLflow run.

+# [Using a model wrapper](#tab/wrapper)

+The simplest way of creating your custom model's flavor is by creating a wrapper around your existing model object. MLflow will serialize it and package it for you. Python objects are serializable when the object can be stored in the file system as a file (generally in Pickle format). During runtime, the object can be materialized from such file and all the values, properties and methods available when it was saved will be restored.

+Use this method when:

+> [!div class="checklist"]

+> * Your model can be serialized in Pickle format.

+> * You want to retain the models state as it was just after training.

+> * You want to customize the way the `predict` function works.

+The following sample wraps a model created with XGBoost to make it behaves in a different way to the default implementation of the XGBoost flavor (it returns the probabilities instead of the classes):

+```python

+from mlflow.pyfunc import PythonModel, PythonModelContext

+class ModelWrapper(PythonModel):

+ def __init__(self, model):

+ self._model = model

+ def predict(self, context: PythonModelContext, data):

+ # You don't have to keep the semantic meaning of `predict`. You can use here model.recommend(), model.forecast(), etc

+ return self._model.predict_proba(data)

+ # You can even add extra functions if you need to. Since the model is serialized,

+ # all of them will be available when you load your model back.

+ def predict_batch(self, data):

+ pass

+```

+Then, a custom model can be logged in the run like this:

+```python

+mport mlflow

+from xgboost import XGBClassifier

+from sklearn.metrics import accuracy_score

+from mlflow.models import infer_signature

+with mlflow.start_run():

+ mlflow.xgboost.autolog(log_models=False)

+ model = XGBClassifier(use_label_encoder=False, eval_metric="logloss")

+ model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)

+ y_probs = model.predict_proba(X_test)

+ accuracy = accuracy_score(y_test, y_probs.argmax(axis=1))

+ mlflow.log_metric("accuracy", accuracy)

+ signature = infer_signature(X_test, y_probs)

+ mlflow.pyfunc.log_model("classifier",

+ python_model=ModelWrapper(model),

+ signature=signature)

+```

+> [!TIP]

+> Note how the `infer_signature` method now uses `y_probs` to infer the signature. Our target column has the target class, but our model now returns the two probabilities for each class.

+# [Using artifacts](#tab/artifacts)

+Wrapping your model may be simple, but sometimes your model needs multiple pieces to be loaded or it can't just be serialized simply as a Pickle file. In those cases, the `PythonModel` supports indicating an arbitrary list of **artifacts**. Each artifact will be packaged along with your model.

+Use this method when:

+> [!div class="checklist"]

+> * Your model can't be serialized in Pickle format or there is a better format available for that.

+> * Your model have one or many artifacts that need to be referenced in order to load the model.

+> * You may want to persist some inference configuration properties (i.e. number of items to recommend).

+> * You want to customize the way the model is loaded and how the `predict` function works.

+To log a custom model using artifacts, you can do something as follows:

+```python

+encoder_path = 'encoder.pkl'

+joblib.dump(encoder, encoder_path)

+model_path = 'xgb.model'

+model.save_model(model_path)

+mlflow.pyfunc.log_model("classifier",

+ python_model=ModelWrapper(),

+ artifacts={

+ 'encoder': encoder_path,

+ 'model': model_path

+ },

+ signature=signature)

+```

+> [!NOTE]

+> * The model was saved using the save method of the framework used (it's not saved as a pickle).

+> * `ModelWrapper()` is the model wrapper, but the model is not passed as a parameter to the constructor.

+> A new parameter is indicated, `artifacts`, that is a dictionary with keys as the name of the artifact and values as the path is the local file system where the artifact is stored.

+The corresponding model wrapper then would look as follows:

+```python

+from mlflow.pyfunc import PythonModel, PythonModelContext

+class ModelWrapper(PythonModel):

+ def load_context(self, context: PythonModelContext):

+ import pickle

+ from xgboost import XGBClassifier

+ from sklearn.preprocessing import OrdinalEncoder

+

+ self._encoder = pickle.loads(context.artifacts["encoder"])

+ self._model = XGBClassifier(use_label_encoder=False, eval_metric="logloss")

+ model.load_model(context.artifacts["model"])

+ def predict(self, context: PythonModelContext, data):

+ return self._model.predict_proba(data)

+```

+The complete training routine would look as follows:

+```python

+import mlflow

+from xgboost import XGBClassifier

+from sklearn.preprocessing import OrdinalEncoder

+from sklearn.metrics import accuracy_score

+from mlflow.models import infer_signature

+with mlflow.start_run():

+ mlflow.xgboost.autolog(log_models=False)

+

+ encoder = OrdinalEncoder(handle_unknown='ignore')

+ X_train['thal'] = enc.fit_transform(X_train['thal'])

+ X_test['thal'] = enc.transform(X_test['thal'])

+

+ model = XGBClassifier(use_label_encoder=False, eval_metric="logloss")

+ model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)

+ y_probs = model.predict_proba(X_test)

+ accuracy = accuracy_score(y_test, y_probs.argmax(axis=1))

+ mlflow.log_metric("accuracy", accuracy)

+ encoder_path = 'encoder.pkl'

+ joblib.dump(encoder, encoder_path)

+ model_path = "xgb.model"

+ model.save_model(model_path)

+ signature = infer_signature(X, y_probs)

+ mlflow.pyfunc.log_model("classifier",

+ python_model=ModelWrapper(),

+ artifacts={

+ 'encoder': encoder_path,

+ 'model': model_path

+ },

+ signature=signature)

+```

+# [Using a model loader](#tab/loader)

+Sometimes your model logic is complex and there are several source code files being used to make your model work. This would be the case when you have a Python library for your model for instance. In this scenario, you want to package the library all along with your model so it can move as a single piece.

+Use this method when:

+> [!div class="checklist"]

+> * Your model can't be serialized in Pickle format or there is a better format available for that.

+> * Your model can be stored in a folder where all the requiered artifacts are placed.

+> * Your model's logic is complex and it requires multiple source files. Potentially, there is a library that supports your model.

+> * You want to customize the way the model is loaded and how the `predict` function works.

+MLflow supports this kind of models too by allowing you to specify any arbitrary source code to package along with the model as long as it has a *loader module*. Loader modules can be specified in the `log_model()` instruction using the argument `loader_module` which indicates the Python namespace where the loader is implemented. The argument `code_path` is also required, where you indicate the source files where the `loader_module` is defined. You are required to implement in this namespace a function called `_load_pyfunc(data_path: str)` that received the path of the artifacts and returns an object with a method predict (at least).

+```python

+model_path = 'xgb.model'

+model.save_model(model_path)

+mlflow.pyfunc.log_model("classifier",

+ data_path=model_path,

+ code_path=['loader_module.py'],

+ loader_module='loader_module'

+ signature=signature)

+```

+> [!NOTE]

+> * The model was saved using the save method of the framework used (it's not saved as a pickle).

+> * A new parameter, `data_path`, was added pointing to the folder where the model's artifacts are located. This can be a folder or a file. Whatever is on that folder or file, it will be packaged with the model.

+> * A new parameter, `code_path`, was added pointing to the location where the source code is placed. This can be a path or a single file. Whatever is on that folder or file, it will be packaged with the model.

+The corresponding `loader_module.py` implementation would be:

+__loader_module.py__

+```python

+class MyModel():

+ def __init__(self, model):

+ self._model = model

+ def predict(self, data):

+ return self._model.predict_proba(data)

+def _load_pyfunc(data_path: str):

+ import os

+ model = XGBClassifier(use_label_encoder=False, eval_metric='logloss')

+ model.load_model(os.path.abspath(data_path))

+ return MyModel(model)

+```

+> [!NOTE]

+> * The class `MyModel` doesn't inherits from `PythonModel` as we did before, but it has a `predict` function.

+> * The model's source code is on a file. This can be any source code you want. If your project has a folder src, it is a great candidate.

+> * We added a function `_load_pyfunc` which returns an instance of the model's class.

+The complete training code would look as follows:

+```python

+import mlflow

+from xgboost import XGBClassifier

+from sklearn.metrics import accuracy_score

+from mlflow.models import infer_signature

+with mlflow.start_run():

+ mlflow.xgboost.autolog(log_models=False)

+ model = XGBClassifier(use_label_encoder=False, eval_metric="logloss")

+ model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)

+ y_probs = model.predict_proba(X_test)

+ accuracy = accuracy_score(y_test, y_probs.argmax(axis=1))

+ mlflow.log_metric("accuracy", accuracy)

+ model_path = "xgb.model"

+ model.save_model(model_path)

+ signature = infer_signature(X_test, y_probs)

+ mlflow.pyfunc.log_model("classifier",

+ data_path=model_path,

+ code_path=["loader_module.py"],

+ loader_module="loader_module",

+ signature=signature)

+```

+## Next steps

+* [Deploy MLflow models](how-to-deploy-mlflow-models.md)

+MLflow introduces the concept of "models" as a way to package all the artifacts required for a given model to function. Models in MLflow are always a folder with an arbitrary number of files, depending on the framework used to generate the model. Logging models has the advantage of tracking all the elements of the model as a single entity that can be __registered__ and then __deployed__. On top of that, MLflow models enjoy the benefit of [no-code deployment](how-to-deploy-mlflow-models.md) and can be used with the [Responsible AI dashboard](how-to-responsible-ai-dashboard.md) in studio. Read the article [From artifacts to models in MLflow](concept-mlflow-models.md) for more information.

+To save the model from a training run, use the `log_model()` API for the framework you're working with. For example, [mlflow.sklearn.log_model()](https://mlflow.org/docs/latest/python_api/mlflow.sklearn.html#mlflow.sklearn.log_model). For more details about how to log MLflow models see [Logging MLflow models](how-to-log-mlflow-models.md) For migrating existing models to MLflow, see [Convert custom models to MLflow](how-to-convert-custom-model-to-mlflow.md).

+| Registering models from runs outputs/artifacts in a different tracking server/workspace | **&check;** | | | |

+### Prerequisites

+* Install the `azureml-mlflow` package.

+* If you are running outside an Azure ML compute, configure the MLflow tracking URI or MLflow's registry URI to point to the workspace you are working on. For more information about how to Set up tracking environment, see [Track runs using MLflow with Azure Machine Learning](how-to-use-mlflow-cli-runs.md#set-up-tracking-environment) for more details.

+> [!NOTE]

+> Models can only be registered to the registry in the same workspace where the run was tracked. Cross-workspace operations are not supported by the moment in Azure Machine Learning.

+> The method `save_model()` works in the same way as `log_model()`. While `log_model()` saves the model inside on an active run, `save_model()` uses the local file system for saving the model.

+> [!WARNING]

+> Stage names are case sensitive.

+ * Configure MLflow for tracking in Azure Machine Learning, as explained in the next section.

+### Set up tracking environment

+To configure MLflow for working with Azure Machine Learning, you need to point your MLflow environment to the Azure Machine Learning MLflow Tracking URI.

+> [!NOTE]

+> When running on Azure Compute (Azure Notebooks, Jupyter Notebooks hosted on Azure Compute Instances or Compute Clusters) you don't have to configure the tracking URI. It's automatically configured for you.

+

+# [Using the Azure ML SDK v2](#tab/azuremlsdk)

+You can get the Azure ML MLflow tracking URI using the [Azure Machine Learning SDK v2 for Python](concept-v2.md). Ensure you have the library `azure-ai-ml` installed in the cluster you are using. The following sample gets the unique MLFLow tracking URI associated with your workspace. Then the method [`set_tracking_uri()`](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.set_tracking_uri) points the MLflow tracking URI to that URI.

+1. Using the workspace configuration file:

+ ```Python

+ from azure.ai.ml import MLClient

+ from azure.identity import DefaultAzureCredential

+ import mlflow

+ ml_client = MLClient.from_config(credential=DefaultAzureCredential()

+ azureml_mlflow_uri = ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri

+ mlflow.set_tracking_uri(azureml_mlflow_uri)

+ ```

+ > [!TIP]

+ > You can download the workspace configuration file by:

+ > 1. Navigate to [Azure ML studio](https://ml.azure.com)

+ > 2. Click on the uper-right corner of the page -> Download config file.

+ > 3. Save the file `config.json` in the same directory where you are working on.

+1. Using the subscription ID, resource group name and workspace name:

+ ```Python

+ from azure.ai.ml import MLClient

+ from azure.identity import DefaultAzureCredential

+ import mlflow

+ #Enter details of your AzureML workspace

+ subscription_id = '<SUBSCRIPTION_ID>'

+ resource_group = '<RESOURCE_GROUP>'

+ workspace_name = '<AZUREML_WORKSPACE_NAME>'

+ ml_client = MLClient(credential=DefaultAzureCredential(),

+ subscription_id=subscription_id,

+ resource_group_name=resource_group)

+ azureml_mlflow_uri = ml_client.workspaces.get(workspace_name).mlflow_tracking_uri

+ mlflow.set_tracking_uri(azureml_mlflow_uri)

+ ```

+ > [!IMPORTANT]

+ > `DefaultAzureCredential` will try to pull the credentials from the available context. If you want to specify credentials in a different way, for instance using the web browser in an interactive way, you can use `InteractiveBrowserCredential` or any other method available in `azure.identity` package.

+# [Using an environment variable](#tab/environ)

+Another option is to set one of the MLflow environment variables [MLFLOW_TRACKING_URI](https://mlflow.org/docs/latest/tracking.html#logging-to-a-tracking-server) directly in your terminal.

+```Azure CLI

+export MLFLOW_TRACKING_URI=$(az ml workspace show --query mlflow_tracking_uri | sed 's/"//g')

+>[!IMPORTANT]

+> Make sure you are logged in to your Azure account on your local machine, otherwise the tracking URI returns an empty string. If you are using any Azure ML compute the tracking environment and experiment name is already configured.

+# [Building the MLflow tracking URI](#tab/build)

+The Azure Machine Learning Tracking URI can be constructed using the subscription ID, region of where the resource is deployed, resource group name and workspace name. The following code sample shows how:

+```python

+region = ""

+subscription_id = ""

+resource_group = ""

+workspace_name = ""

+azureml_mlflow_uri = f"azureml://{region}.api.azureml.ms/mlflow/v1.0/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.MachineLearningServices/workspaces/{workspace_name}"

+mlflow.set_tracking_uri(azureml_mlflow_uri)

+> [!NOTE]

+> You can also get this URL by:

+> 1. Navigate to [Azure ML studio](https://ml.azure.com)

+> 2. Click on the uper-right corner of the page -> View all properties in Azure Portal -> MLflow tracking URI.

+> 3. Copy the URI and use it with the method `mlflow.set_tracking_uri`.

+## Train MLflow Projects on local compute

+This example shows how to submit MLflow projects locally with Azure Machine Learning.

+ # [Using the Azure ML SDK v2](#tab/azuremlsdk)

+ [!INCLUDE [sdk v1](../../includes/machine-learning-sdk-v2.md)]

+ You can get the Azure ML MLflow tracking URI using the [Azure Machine Learning SDK v2 for Python](concept-v2.md). Ensure you have the library `azure-ai-ml` installed in the cluster you are using. The following sample gets the unique MLFLow tracking URI associated with your workspace. Then the method [`set_tracking_uri()`](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.set_tracking_uri) points the MLflow tracking URI to that URI.

+ a. Using the workspace configuration file:

+ ```Python

+ from azure.identity import DefaultAzureCredential

+ import mlflow

+ ml_client = MLClient.from_config(credential=DefaultAzureCredential()

+ azureml_mlflow_uri = ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri

+ mlflow.set_tracking_uri(azureml_mlflow_uri)

+ ```

+ > [!TIP]

+ > You can download the workspace configuration file by:

+ > 1. Navigate to [Azure ML studio](https://ml.azure.com)

+ > 2. Click on the uper-right corner of the page -> Download config file.

+ > 3. Save the file `config.json` in the same directory where you are working on.

+ b. Using the subscription ID, resource group name and workspace name:

+ ```Python

+ from azure.ai.ml import MLClient

+ from azure.identity import DefaultAzureCredential

+ import mlflow

+ #Enter details of your AzureML workspace

+ subscription_id = '<SUBSCRIPTION_ID>'

+ resource_group = '<RESOURCE_GROUP>'

+ workspace_name = '<AZUREML_WORKSPACE_NAME>'

+ ml_client = MLClient(credential=DefaultAzureCredential(),

+ resource_group_name=resource_group)

+ azureml_mlflow_uri = ml_client.workspaces.get(workspace_name).mlflow_tracking_uri

+ > [!IMPORTANT]

+ > `DefaultAzureCredential` will try to pull the credentials from the available context. If you want to specify credentials in a different way, for instance using the web browser in an interactive way, you can use `InteractiveBrowserCredential` or any other method available in `azure.identity` package.

+ # [Building the MLflow tracking URI](#tab/build)

+ region = ""

+ resource_group = ""

+ workspace_name = ""

+ azureml_mlflow_uri = f"azureml://{region}.api.azureml.ms/mlflow/v1.0/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.MachineLearningServices/workspaces/{workspace_name}"

+ > 1. Navigate to [Azure ML studio](https://ml.azure.com)

+ # [Using the Azure ML SDK v2](#tab/azuremlsdk)

+ [!INCLUDE [sdk v1](../../includes/machine-learning-sdk-v2.md)]

+ You can get the Azure ML MLflow tracking URI using the [Azure Machine Learning SDK v2 for Python](concept-v2.md). Ensure you have the library `azure-ai-ml` installed in the cluster you are using. The following sample gets the unique MLFLow tracking URI associated with your workspace. Then the method [`set_tracking_uri()`](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.set_tracking_uri) points the MLflow tracking URI to that URI.

+ a. Using the workspace configuration file:

+ ```Python

+ from azure.identity import DefaultAzureCredential

+ import mlflow

+ ml_client = MLClient.from_config(credential=DefaultAzureCredential()

+ azureml_mlflow_uri = ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri

+ mlflow.set_tracking_uri(azureml_mlflow_uri)

+ ```

+ > [!TIP]

+ > You can download the workspace configuration file by:

+ > 1. Navigate to [Azure ML studio](https://ml.azure.com)

+ > 2. Click on the uper-right corner of the page -> Download config file.

+ > 3. Save the file `config.json` in the same directory where you are working on.

+ b. Using the subscription ID, resource group name and workspace name:

+ ```Python

+ from azure.ai.ml import MLClient

+ from azure.identity import DefaultAzureCredential

+ import mlflow

+ #Enter details of your AzureML workspace

+ subscription_id = '<SUBSCRIPTION_ID>'

+ resource_group = '<RESOURCE_GROUP>'

+ workspace_name = '<AZUREML_WORKSPACE_NAME>'

+ ml_client = MLClient(credential=DefaultAzureCredential(),

+ resource_group_name=resource_group)

+ azureml_mlflow_uri = ml_client.workspaces.get(workspace_name).mlflow_tracking_uri

+ > [!IMPORTANT]

+ > `DefaultAzureCredential` will try to pull the credentials from the available context. If you want to specify credentials in a different way, for instance using the web browser in an interactive way, you can use `InteractiveBrowserCredential` or any other method available in `azure.identity` package.

+ # [Building the MLflow tracking URI](#tab/build)

+ region = ""

+ resource_group = ""

+ workspace_name = ""

+ azureml_mlflow_uri = f"azureml://{region}.api.azureml.ms/mlflow/v1.0/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.MachineLearningServices/workspaces/{workspace_name}"

+ > 1. Navigate to [Azure ML studio](https://ml.azure.com)

+

+1. Using the workspace configuration file:

+ ```Python

+ from azure.ai.ml import MLClient

+ from azure.identity import DefaultAzureCredential

+ import mlflow

+ ml_client = MLClient.from_config(credential=DefaultAzureCredential()

+ azureml_mlflow_uri = ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri

+ mlflow.set_tracking_uri(azureml_mlflow_uri)

+ ```

+ > [!TIP]

+ > You can download the workspace configuration file by:

+ > 1. Navigate to [Azure ML studio](https://ml.azure.com)

+ > 2. Click on the uper-right corner of the page -> Download config file.

+ > 3. Save the file `config.json` in the same directory where you are working on.

+1. Using the subscription ID, resource group name and workspace name:

+ ```Python

+ from azure.ai.ml import MLClient

+ from azure.identity import DefaultAzureCredential

+ import mlflow

+ #Enter details of your AzureML workspace

+ subscription_id = '<SUBSCRIPTION_ID>'

+ resource_group = '<RESOURCE_GROUP>'

+ workspace_name = '<AZUREML_WORKSPACE_NAME>'

+ ml_client = MLClient(credential=DefaultAzureCredential(),

+ subscription_id=subscription_id,

+ resource_group_name=resource_group)

+ azureml_mlflow_uri = ml_client.workspaces.get(workspace_name).mlflow_tracking_uri

+ mlflow.set_tracking_uri(azureml_mlflow_uri)

+ ```

+ > [!IMPORTANT]

+ > `DefaultAzureCredential` will try to pull the credentials from the available context. If you want to specify credentials in a different way, for instance using the web browser in an interactive way, you can use `InteractiveBrowserCredential` or any other method available in `azure.identity` package.

+> You can also get this URL by:

+> 1. Navigate to [Azure ML studio](https://ml.azure.com)

+> 2. Click on the uper-right corner of the page -> View all properties in Azure Portal -> MLflow tracking URI.

+> 3. Copy the URI and use it with the method `mlflow.set_tracking_uri`.

+ > Then run `nodetool repair --full` on all the nodes in your existing cluster's data center. You should run this **only after all of the prior steps have been taken**. This should ensure that all historical data is replicated to your new data centers in Azure Managed Instance for Apache Cassandra. If you have a very large amount of data in your existing cluster, it may be necessary to run the repairs at the keyspace or even table level - see [here](https://cassandra.apache.org/doc/latest/cassandra/operating/repair.html) for more details on running repairs in Cassandra. Prior to changing the replication settings, you should also make sure that any application code that connects to your existing Cassandra cluster is using LOCAL_QUORUM. You should leave it at this setting during the migration (it can be switched back afterwards if required). After the migration is completed, you can enable automatic repair again, and point your application code to the new Cassandra Managed Instance data center's seed nodes (and revert the quorum settings if preferred).

+ >

+ > Finally, to decommission your old data center:

+ >

+ > - Run `ALTER KEYSPACE` for each keyspace, removing the old data center.

+ > - We recommend running `nodetool repair` for each keyspace as well, before the below step.

+ > - Run [nodetool decommision](https://cassandra.apache.org/doc/latest/cassandra/operating/topo_changes.html#removing-nodes) for each on premise data center node.

+ - [Map existing custom DNS name](/azure/app-service/app-service-web-tutorial-custom-domain).

+ - [Secure a custom DNS with a TLS/SSL binding](/azure/app-service/configure-ssl-bindings).

+- [Review best practices](/azure/app-service/deploy-best-practices) for deploying to Azure App service.

+ - [Map existing custom DNS name](/azure/app-service/app-service-web-tutorial-custom-domain).

+ - [Secure a custom DNS with a TLS/SSL binding](/azure/app-service/configure-ssl-bindings).

+- [Review best practices](/azure/app-service/deploy-best-practices) for deploying to Azure App service.

+- [Map existing custom DNS name](/azure/app-service/app-service-web-tutorial-custom-domain).

+- [Secure a custom DNS with a TLS/SSL binding](/azure/app-service/configure-ssl-bindings).

+- [Review best practices](/azure/app-service/deploy-best-practices) for deploying to Azure App service.

+Intelligent tuning operates around three main parameters for the given time: `checkpoint_completion_target`, `max_wal_size`, and `bgwriter_delay`.

+Private endpoints created through Azure Cognitive Search APIs are referred to as "shared private links" or "managed private endpoints". The concept of a shared private link is that an Azure PaaS resource already has a private endpoint through [Azure Private Link service](https://azure.microsoft.com/services/private-link/), and Azure Cognitive Search is sharing access. Although access is shared, a shared private link creates its own private connection that's used exclusively by Azure Cognitive Search. The shared private link is the mechanism by which Azure Cognitive Search makes the connection to resources in a private network.

+> [Import data wizard](search-import-data-portal.md) or invoking indexer-based indexing from the portal over a shared private link is not supported.

+Azure Cognitive Search won't store data outside of your specified region without your authorization. Specifically, the following features write to an Azure Storage resource: [enrichment cache](cognitive-search-incremental-indexing-conceptual.md), [debug session](cognitive-search-debug-session.md), [knowledge store](knowledge-store-concept-intro.md). The storage account is one that you provide, and it could be in any region.

+If both the storage account and the search service are in the same region, network traffic between search and storage uses a private IP address and occurs over the Microsoft backbone network. Because private IP addresses are used, you can't configure IP firewalls or a private endpoint for network security. Instead, use the [trusted service exception](search-indexer-howto-access-trusted-service-exception.md) as an alternative when both services are in the same region.

+Azure Cognitive Search won't store data outside of your specified region without your authorization. Specifically, the following features write to an Azure Storage resource: [enrichment cache](cognitive-search-incremental-indexing-conceptual.md), [debug session](cognitive-search-debug-session.md), [knowledge store](knowledge-store-concept-intro.md). The storage account is one that you provide, and it could be in any region.

+If both the storage account and the search service are in the same region, network traffic between search and storage uses a private IP address and occurs over the Microsoft backbone network. Because private IP addresses are used, you can't configure IP firewalls or a private endpoint for network security. Instead, use the [trusted service exception](search-indexer-howto-access-trusted-service-exception.md) as an alternative when both services are in the same region.

+Microsoft Sentinel's [Microsoft 365 Defender](/microsoft-365/security/mtp/microsoft-threat-protection) connector with incident integration allows you to stream all Microsoft 365 Defender incidents and alerts into Microsoft Sentinel, and keeps the incidents synchronized between both portals. Microsoft 365 Defender incidents include all their alerts, entities, and other relevant information, and they group together, and are enriched by, alerts from Microsoft 365 Defender's component services **Microsoft Defender for Endpoint**, **Microsoft Defender for Identity**, **Microsoft Defender for Office 365**, and **Microsoft Defender for Cloud Apps**, as well as alerts from other services such as **Microsoft Purview Data Loss Prevention (DLP)**.

+- Get started [detecting threats with Microsoft Sentinel](./detect-threats-built-in.md).

+az webapp connection create postgres-flexible --client-type django

+[Rollup 62](https://support.microsoft.com/topic/update-rollup-62-for-azure-site-recovery-e7aff36f-b6ad-4705-901c-f662c00c402b) | 9.49.6395.1 | 5.1.7418.0 | 9.49.6395.1 | 5.1.7418.0 | 2.0.9248.0

+## Updates (July 2022)

+### Update Rollup 62

+[Update rollup 62](https://support.microsoft.com/topic/update-rollup-62-for-azure-site-recovery-e7aff36f-b6ad-4705-901c-f662c00c402b) provides the following updates:

+**Update** | **Details**

+ |

+**Providers and agents** | Updates to Site Recovery agents and providers as detailed in the rollup KB article.

+**Issue fixes/improvements** | A number of fixes and improvement as detailed in the rollup KB article.

+**Azure VM disaster recovery** | Added support for RHEL 8.6 and CentOS 8.6 Linux distros.

+**VMware VM/physical disaster recovery to Azure** | Added support for RHEL 8.6 and CentOS 8.6 Linux distros.<br/><br/> Added support for configuring proxy bypass rules for VMware and Hyper-V replications, using private endpoints.<br/><br/> Added fixes related to various security issues present in the classic experience.

+**Hyper-V disaster recovery to Azure** | Added support for configuring proxy bypass rules for VMware and Hyper-V replications, using private endpoints.

+Linux Red Hat Enterprise | 5.2 to 5.11</b><br/> 6.1 to 6.10</b> </br> 7.0, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, [7.7](https://support.microsoft.com/help/4528026/update-rollup-41-for-azure-site-recovery), [7.8](https://support.microsoft.com/help/4564347/), [7.9 Beta version](https://support.microsoft.com/help/4578241/), [7.9](https://support.microsoft.com/help/4590304/) </br> [8.0](https://support.microsoft.com/help/4531426/update-rollup-42-for-azure-site-recovery), 8.1, [8.2](https://support.microsoft.com/help/4570609), [8.3](https://support.microsoft.com/help/4597409/), [8.4](https://support.microsoft.com/topic/883a93a7-57df-4b26-a1c4-847efb34a9e8) (4.18.0-305.30.1.el8_4.x86_64 or higher), [8.5](https://support.microsoft.com/topic/883a93a7-57df-4b26-a1c4-847efb34a9e8) (4.18.0-348.5.1.el8_5.x86_64 or higher) <br/> Few older kernels on servers running Red Hat Enterprise Linux 5.2-5.11 & 6.1-6.10 do not have [Linux Integration Services (LIS) components](https://www.microsoft.com/download/details.aspx?id=55106) pre-installed. If in-built LIS components are missing, ensure to install the [components](https://www.microsoft.com/download/details.aspx?id=55106) before enabling replication for the machines to boot in Azure. <br/><br/>Note: RHEl 5 is not supported for modernized VMware protection experience (in Preview).

+Linux: CentOS | 5.2 to 5.11</b><br/> 6.1 to 6.10</b><br/> </br> 7.0, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, [7.7](https://support.microsoft.com/help/4528026/update-rollup-41-for-azure-site-recovery), [7.8](https://support.microsoft.com/help/4564347/), [7.9](https://support.microsoft.com/help/4578241/) </br> [8.0](https://support.microsoft.com/help/4531426/update-rollup-42-for-azure-site-recovery), 8.1, [8.2](https://support.microsoft.com/help/4570609), [8.3](https://support.microsoft.com/help/4597409/), 8.4, 8.5 <br/><br/> Few older kernels on servers running CentOS 5.2-5.11 & 6.1-6.10 do not have [Linux Integration Services (LIS) components](https://www.microsoft.com/download/details.aspx?id=55106) pre-installed. If in-built LIS components are missing, ensure to install the [components](https://www.microsoft.com/download/details.aspx?id=55106) before enabling replication for the machines to boot in Azure. <br/><br/>Note: CentOS 5 is not supported for modernized VMware protection experience (in Preview).

+- Version 2 uses [Galois/Counter Mode (GCM)](https://en.wikipedia.org/wiki/Galois/Counter_Mode) mode with AES.

+- Version 1 uses [Cipher Block Chaining (CBC)](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher-block_chaining_.28CBC.29) mode with AES.

+> Using version 1 of client-side encryption is no longer recommended due to a security vulnerability in the client library's implementation of CBC mode. For more information about this security vulnerability, see [Azure Storage updating client-side encryption in SDK to address security vulnerability](https://aka.ms/azstorageclientencryptionblog). If you are currently using version 1, we recommend that you update your application to use version 2 and migrate your data. See the following section, [Mitigate the security vulnerability in your applications](#mitigate-the-security-vulnerability-in-your-applications), for further guidance.

+### Mitigate the security vulnerability in your applications

+| Application is using client-side encryption a version of the client library that supports only client-side encryption v1. | Update your application to use a version of the client library that supports client-side encryption v2. See [SDK support matrix for client-side encryption](#sdk-support-matrix-for-client-side-encryption) for a list of supported versions. [Learn more...](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/AzureStorageNetMigrationV12.md)<br/><br/>Update your code to use client-side encryption v2. [Learn more...](#example-encrypting-and-decrypting-a-blob-with-client-side-encryption-v2)<br/><br/>Download any encrypted data to decrypt it, then reencrypt it with client-side encryption v2. [Learn more...](#reencrypt-previously-encrypted-data-with-client-side-encryption-v2) |

+| Application is using client-side encryption with a version of the client library that supports client-side encryption v2. | Update your code to use client-side encryption v2. [Learn more...](#example-encrypting-and-decrypting-a-blob-with-client-side-encryption-v2)<br/><br/>Download any encrypted data to decrypt it, then reencrypt it with client-side encryption v2. [Learn more...](#reencrypt-previously-encrypted-data-with-client-side-encryption-v2) |

+### SDK support matrix for client-side encryption

+The following table shows which versions of the client libraries for .NET, Java, and Python support which versions of client-side encryption:

+| | .NET | Java | Python |

+|--|--|--|--|

+| **Client-side encryption v2 and v1** | [Versions 12.13.0 and later](https://www.nuget.org/packages/Azure.Storage.Blobs) | [Versions 12.18.0 and later](https://search.maven.org/artifact/com.azure/azure-storage-blob) | [Versions 12.13.0 and later](https://pypi.org/project/azure-storage-blob) |

+| **Client-side encryption v1 only** | Versions 12.12.0 and earlier | Versions 12.17.0 and earlier | Versions 12.12.0 and earlier |

+If your application is using client-side encryption with an earlier version of the .NET, Java, or Python client library, you must first upgrade your code to a version that supports client-side encryption v2. Next, you must decrypt and re-encrypt your data with client-side encryption v2. If necessary, you can use a version of the client library that supports client-side encryption v2 side-by-side with an earlier version of the client library while you are migrating your code. For code examples, see [Example: Encrypting and decrypting a blob with client-side encryption v2](#example-encrypting-and-decrypting-a-blob-with-client-side-encryption-v2).

+To use client-side encryption from your Java code, reference the [Blob Storage client library](/jav).

+- To learn more about .NET Core, see [Get started with .NET in 10 minutes](https://dotnet.microsoft.com/learn/dotnet/hello-world-tutorial/intro).

+- Version 2 uses [Galois/Counter Mode (GCM)](https://en.wikipedia.org/wiki/Galois/Counter_Mode) mode with AES.

+- Version 1 uses [Cipher Block Chaining (CBC)](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher-block_chaining_.28CBC.29) mode with AES.

+> Using version 1 of client-side encryption is no longer recommended due to a security vulnerability in the client library's implementation of CBC mode. For more information about this security vulnerability, see [Azure Storage updating client-side encryption in SDK to address security vulnerability](https://aka.ms/azstorageclientencryptionblog). If you are currently using version 1, we recommend that you update your application to use version 2 and migrate your data.

+>

+> The Azure Table Storage SDK supports only version 1 of client-side encryption. Using client-side encryption with Table Storage is not recommended.

+| Blob Storage client libraries for .NET (version 12.13.0 and above), Java (version 12.18.0 and above), and Python (version 12.13.0 and above) | 2.0<br/><br/>1.0 (for backward compatibility only) | Update your code to use client-side encryption v2.<br/><br/>Download any encrypted data to decrypt it, then reencrypt it with client-side encryption v2. | [Client-side encryption for blobs](../blobs/client-side-encryption.md) |

+| Blob Storage client library for .NET (version 12.12.0 and below), Java (version 12.17.0 and below), and Python (version 12.12.0 and below) | 1.0 (not recommended) | Update your application to use a version of the Blob Storage SDK that supports client-side encryption v2. See [SDK support matrix for client-side encryption](../blobs/client-side-encryption.md#sdk-support-matrix-for-client-side-encryption) for details.<br/><br/>Update your code to use client-side encryption v2.<br/><br/>Download any encrypted data to decrypt it, then reencrypt it with client-side encryption v2. | [Client-side encryption for blobs](../blobs/client-side-encryption.md) |

+| Queue Storage client library for .NET (version 12.11.0 and above) and Python (version 12.4 and above) | 2.0<br/><br/>1.0 (for backward compatibility only) | Update your code to use client-side encryption v2. | [Client-side encryption for queues](../queues/client-side-encryption.md) |

+| Queue Storage client library for .NET (version 12.10.0 and below) and Python (version 12.3.0 and below) | 1.0 (not recommended) | Update your application to use a version of the Queue Storage SDK version that supports client-side encryption v2. See [SDK support matrix for client-side encryption](../queues/client-side-encryption.md#sdk-support-matrix-for-client-side-encryption)<br/><br/>Update your code to use client-side encryption v2. | [Client-side encryption for queues](../queues/client-side-encryption.md) |

+- Version 2 uses [Galois/Counter Mode (GCM)](https://en.wikipedia.org/wiki/Galois/Counter_Mode) mode with AES.

+- Version 1 uses [Cipher Block Chaining (CBC)](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher-block_chaining_.28CBC.29) mode with AES.

+> Using version 1 of client-side encryption is no longer recommended due to a security vulnerability in the client library's implementation of CBC mode. For more information about this security vulnerability, see [Azure Storage updating client-side encryption in SDK to address security vulnerability](https://aka.ms/azstorageclientencryptionblog). If you are currently using version 1, we recommend that you update your application to use version 2 and migrate your data. See the following section, [Mitigate the security vulnerability in your applications](#mitigate-the-security-vulnerability-in-your-applications), for further guidance.

+### Mitigate the security vulnerability in your applications

+| Application is using client-side encryption a version of the client library that supports only client-side encryption v1. | Update your application to use a version of the client library that supports client-side encryption v2. See [SDK support matrix for client-side encryption](#sdk-support-matrix-for-client-side-encryption) for a list of supported versions. <br/><br/>Update your code to use client-side encryption v2. |

+| Application is using client-side encryption with a version of the client library that supports client-side encryption v2. | Update your code to use client-side encryption v2. |

+### SDK support matrix for client-side encryption

+The following table shows which versions of the client libraries for .NET and Python support which versions of client-side encryption:

+| | .NET | Python |

+|--|--|--|--|

+| **Client-side encryption v2 and v1** | [Versions 12.11.0 and later](https://www.nuget.org/packages/Azure.Storage.Queues) | [Versions 12.4.0 and later](https://pypi.org/project/azure-storage-queue) |

+| **Client-side encryption v1 only** | Versions 12.10.0 and earlier | Versions 12.3.0 and earlier |

+If your application is using client-side encryption with an earlier version of the .NET or Python client library, you must first upgrade your code to a version that supports client-side encryption v2. Next, you must decrypt and re-encrypt your data with client-side encryption v2. If necessary, you can use a version of the client library that supports client-side encryption v2 side-by-side with an earlier version of the client library while you are migrating your code.

+- Connect to a container in Azure Data Lake Storage (ADLS) Gen2 that is linked to your Azure Synapse Analytics workspace.

+- Read the data from a PySpark Notebook using `spark.read.load`.

+- Convert the data to a Pandas dataframe using `.toPandas()`.

+- Synapse Analytics workspace with ADLS Gen2 configured as the default storage - You need to be the **Storage Blob Data Contributor** of the ADLS Gen2 filesystem that you work with. For details on how to create a workspace, see [Creating a Synapse workspace](get-started-create-workspace.md).

+1. In the Azure portal, create a container in the same ADLS Gen2 used by Synapse Studio. You can skip this step if you want to use the default linked storage account in your Azure Synapse Analytics workspace.

+1. In Synapse Studio, select **Data**, select the **Linked** tab, and select the container under **Azure Data Lake Storage Gen2**.

+1. Select the uploaded file, select **Properties**, and copy the **ABFSS Path** value.

+1. In the left pane, select **Develop**.

+1. Select **+** and select "Notebook" to create a new notebook.

+1. In **Attach to**, select your Apache Spark Pool. If you don't have one, select **Create Apache Spark pool**.

+After a few minutes, the text displayed should look similar to the following.

+- [How to use file mount/unmount API in Synapse](spark/synapse-file-mount-api.md)

+- [Azure Architecture Center: Explore data in Azure Blob storage with the pandas Python package](/azure/architecture/data-science-process/explore-data-blob)

+- [Tutorial: Use Pandas to read/write Azure Data Lake Storage Gen2 data in serverless Apache Spark pool in Synapse Analytics](spark/tutorial-use-pandas-spark-pool.md)

+ Title: Introduction to file mount in Azure Synapse Analytics

+description: "Tutorial: How to use file mount/unmount API in Azure Synapse Analytics"

+Synapse studio team built two new mount/unmount APIs in the Microsoft Spark Utilities (MSSparkUtils) package, you can use mount to attach remote storage (Azure Blob Storage, Azure Data Lake Storage (ADLS) Gen2) to all working nodes (driver node and worker nodes). Once in place, you can access data in storage as if they were one the local file system with local file API. For more information, see [Introduction to Microsoft Spark Utilities](microsoft-spark-utilities.md).

+The document will show you how to use mount/unmount API in your workspace, mainly includes below sections:

+> + Azure Fileshare mount is temporarily disabled, you can use ADLS Gen2/blob mount following the [How to mount Gen2/blob Storage](#How-to-mount-Gen2/blob-Storage).

+> + Azure Data Lake storage Gen1 storage is not supported, you can migrate to ADLS Gen2 following the [Migration gudiance](../../storage/blobs/data-lake-storage-migrate-gen1-to-gen2-azure-portal.md) before using mount APIs.

+<a id="How-to-mount-Gen2/blob-Storage"></a>

+## How to mount storage

+Here we will illustrate how to mount Azure Data Lake Storage (ADLS) Gen2 or Azure Blob Storage step by step as an example, mounting blob storage works similarly.

+Assuming you have one ADLS Gen2 account named `storegen2` and the account has one container name `mycontainer`, and you want to mount the `mycontainer` to `/test` of your Spark pool.

+

+To mount container `mycontainer`, `mssparkutils` need to check whether you have the permission to access the container at first, currently we support three authentication methods to trigger mount operation, **LinkedService**, **accountKey**, and **sastoken**.

+Trigger mount via linked Service is our recommend way, there won't have any security leak issue by this way, since `mssparkutils` doesn't store any secret/auth value itself, and it will always fetch auth value from linked service to request blob data from the remote storage.

+You can create linked service for ADLS Gen2 or blob storage. Currently, two authentication methods are supported when created linked service, one is using account key, another is using managed identity.

+After you create linked service successfully, you can easily mount the container to your Spark pool with below Python code.

+In addition to mount with linked service, `mssparkutils` also support explicitly passing account key or [SAS (shared access signature)](/samples/azure-samples/storage-dotnet-sas-getting-started/storage-dotnet-sas-getting-started/) token as parameter to mount the target.

+For security reasons, it is recommended to store Account key or SAS token in Azure Key Vaults (as the below example figure shows), then retrieving them with `mssparkutil.credentials.getSecret` API. For more information, see [Manage storage account keys with Key Vault and the Azure CLI (legacy)](../../key-vault/secrets/overview-storage-keys.md).

+> For security reasons, do not store credentials in code.

+Assuming you have a ADLS Gen2 storage account named `storegen2` and the account has one file share named `myfileshare`, and you want to mount the `myfileshare` to `/test` of your spark pool.

+Mount azure file share only supports the account key authentication method, below is the code sample to mount **myfileshare** to `/test` and we reuse the Azure Key Value settings of `MountKV` here:

+In the above example, we pre-defined the schema format of source URL for the file share to: `https://<filesharename>@<accountname>.file.core.windows.net`, and we stored the account key in AKV, and retrieving them with `mssparkutil.credentials.getSecret` API instead of explicitly passing it to the mount API.

+So, for example if you mount `mycontainer` to `/test` folder, the created local mount point is `/synfs/{jobid}/test`, that means if you want to access mount point via local fs APIs after a successful mount, the local path used should be `/synfs/{jobid}/test`

+The main purpose of the mount operation is to let customer access the data stored in remote storage account using local file system API, you can also access data using `mssparkutils fs` API with mounted path as a parameter. The path format used here is a little different.

+Assuming you mounted to the ADLS Gen2 container `mycontainer` to `/test` using mount API.

+While when you want to access the data with `mssparkutils fs` API, the path format is like:

+You can see the `synfs` is used as schema in this case instead of a part of the mounted path.

+Below are three examples to show how to access file with mount point path using `mssparkutils fs`, while **49** is a Spark job ID we got from calling `mssparkutils.env.getJobId()`.

+You can also use Spark read API with mounted path as parameter to access the data after mount as well, the path format here is same with the format of using `mssparkutils fs` API:

+Below are two code examples, one is for a mounted ADLS Gen2 storage, another for a mounted blob storage.

+<a id="read-file-from-a-mounted-gen2-storage-account"></a>

+### Read file from a mounted ADLS Gen2 storage account

+The below example assumes an ADLS Gen2 storage was already mounted then read file using mount path.

+# Assume a ADLS Gen2 storage was already mounted then read file using mount path

+Notice that if you mounted a blob storage account then want to access it using `mssparkutils` or Spark API, you need to explicitly configure the sas token via spark configuration at first before try to mount container using mount API.

+1. Update Spark configuration as below code example if you want to access it using `mssparkutils` or Spark API after trigger mount, you can bypass this step if you only want to access it using local file api after mount:

+2. Create link service `myblobstorageaccount` and mount blob storage account with link service:

+3. Mount the blob storage container and then read file using mount path through the local file API:

+4. Read data from mounted blob storage through Spark read API:

+Unmount with your mount point, `/test` in our example:

+## Next steps

+- [Get Started with Azure Synapse Analytics](../get-started.md)

+- [Monitor your Synapse Workspace](../get-started-monitor.md)

+- [Introduction to Microsoft Spark Utilities](microsoft-spark-utilities.md)

+> Virtual machine scale set encryption is supported with API version `2017-03-30` onwards. If you are using templates to enable scale set encryption, update the API version for virtual machine scale sets and the ADE extension inside the template. See this [sample template](https://github.com/Azure/azure-quickstart-templates/blob/master/quickstarts/microsoft.compute/encrypt-running-vmss-windows/azuredeploy.json) for more information.

+| Canonical | UbuntuServer | 18.04-LTS |

+| Canonical | UbuntuServer | 18.04-LTS-Gen2 |

+| Canonical | UbuntuServer | 20.04-LTS |

+| Canonical | UbuntuServer | 20.04-LTS-Gen2 |

+| MicrosoftCblMariner | Cbl-Mariner | cbl-mariner-1 |

+| MicrosoftCblMariner | Cbl-Mariner | 1-Gen2 |

+| MicrosoftCblMariner | Cbl-Mariner | cbl-mariner-2

+| MicrosoftCblMariner | Cbl-Mariner | cbl-mariner-2-Gen2

+| MicrosoftWindowsServer | WindowsServer | 2012-R2-Datacenter |

+| MicrosoftWindowsServer | WindowsServer | 2016-Datacenter |

+| MicrosoftWindowsServer | WindowsServer | 2016-Datacenter-gs |

+| MicrosoftWindowsServer | WindowsServer | 2016-Datacenter-with-Containers |

+| MicrosoftWindowsServer | WindowsServer | 2019-Datacenter-Core |

+| MicrosoftWindowsServer | WindowsServer | 2019-Datacenter-Core-with-Containers |

+| MicrosoftWindowsServer | WindowsServer | 2019-Datacenter-with-Containers |

+| MicrosoftWindowsServer | WindowsServer | 2012-R2-Datacenter |

+| [Capacity Reservation](../virtual-machines/capacity-reservation-overview.md) | Yes | Yes | Yes |

+ $ipconf = New-AzVmssIPConfig -ApplicationGatewayBackendAddressPoolsId /subscriptions/{subscriptionId}/resourceGroups/myResourceGroup/providers/Microsoft.Network/applicationGateways/{applicationGatewayName}/backendAddressPools/{applicationGatewayBackendAddressPoolName} -SubnetId $vmss.VirtualMachineProfile.NetworkProfile.NetworkInterfaceConfigurations[0].IpConfigurations[0].Subnet.Id -Name $vmss.VirtualMachineProfile.NetworkProfile.NetworkInterfaceConfigurations[0].IpConfigurations[0].Name

+**Applies to:** :heavy_check_mark: Linux VMs :heavy_check_mark: Windows VMs :heavy_check_mark: Uniform scale set :heavy_check_mark: Flexible scale sets

+**Applies to:** :heavy_check_mark: Linux VMs :heavy_check_mark: Windows VMs :heavy_check_mark: Uniform scale set :heavy_check_mark: Flexible scale sets

+**Applies to:** :heavy_check_mark: Linux VMs :heavy_check_mark: Windows VMs :heavy_check_mark: Uniform scale set :heavy_check_mark: Flexible scale sets

+**Applies to:** :heavy_check_mark: Linux VMs :heavy_check_mark: Windows VMs :heavy_check_mark: Uniform scale set :heavy_check_mark: Flexible scale sets

+Get started reserving Compute capacity. Check out our other related Capacity Reservation articles:

+- [Create a capacity reservation](capacity-reservation-create.md)

+- [Overallocating capacity reservation](capacity-reservation-overallocate.md)

+- [Modify a capacity reservation](capacity-reservation-modify.md)

+- [Associate a VM](capacity-reservation-associate-vm.md)

+- [Remove a VM](capacity-reservation-remove-vm.md)

+- [Associate a VM scale set - Flexible](capacity-reservation-associate-virtual-machine-scale-set-flex.md)

+- [Associate a VM scale set - Uniform](capacity-reservation-associate-virtual-machine-scale-set.md)

+- [Remove a VM scale set](capacity-reservation-remove-virtual-machine-scale-set.md)

+**Applies to:** :heavy_check_mark: Uniform scale set :heavy_check_mark: Flexible scale sets

+Windows Guest Agent has a feature to automatically collect some logs. This feature is controlled by the CollectGuestLogs.exe process.

+* Windows 11

+| [OS disk](../managed-disks-overview.md) |Yes |The VM needs a disk to store the OS in most cases. |

+- See [Monitoring Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource) for details on monitoring Azure resources.

+- You can't specify the set of IP addresses for the prefix (though you can specify which IP you want from the prefix). Azure gives the IP addresses for the prefix, based on the size that you specify. Additionally, all public IP addresses created from the prefix must exist in the same Azure region and subscription as the prefix. Addresses must be assigned to resources in the same region and subscription.

+2. In **Create virtual network**, enter or select values for the following settings on the *Basics* tab:

+ | **Setting** | **Description** |

+ | | |

+ | **Project details** | |

+ | **Subscription** | Select a [subscription](../azure-glossary-cloud-terminology.md?toc=%2fazure%2fvirtual-network%2ftoc.json#subscription). You cannot use the same virtual network in more than one Azure subscription. However, you can connect a virtual network in one subscription to virtual networks in other subscriptions with [virtual network peering](virtual-network-peering-overview.md). Any Azure resource that you connect to the virtual network must be in the same subscription as the virtual network.|

+ |**Resource group**|Select an existing [resource group](../azure-resource-manager/management/overview.md?toc=%2fazure%2fvirtual-network%2ftoc.json#resource-groups) or create a new one. An Azure resource that you connect to the virtual network can be in the same resource group as the virtual network or in a different resource group.|

+ | **Instance details** |

+ | **Name** |The name must be unique in the [resource group](../azure-glossary-cloud-terminology.md?toc=%2fazure%2fvirtual-network%2ftoc.json#resource-group) that you select to create the virtual network in. You cannot change the name after the virtual network is created. You can create multiple virtual networks over time. For naming suggestions, see [Naming conventions](/azure/cloud-adoption-framework/ready/azure-best-practices/naming-and-tagging#naming-and-tagging-resources). Following a naming convention can help make it easier to manage multiple virtual networks.|

+ | **Region** | Select an Azure [region](https://azure.microsoft.com/regions/). A virtual network can be in only one Azure region. However, you can connect a virtual network in one region to a virtual network in another region by using a VPN gateway. Any Azure resource that you connect to the virtual network must be in the same region as the virtual network.|

+1. Select **IP Addresses** tab or **Next: IP Addresses >**, and enter the following IP address information:

+ - **Add IPv6 address space**

+| **AzureConnectors** | This tag represents the IP addresses used for managed connectors that make inbound webhook callbacks to the Azure Logic Apps service and outbound calls to their respective services, for example, Azure Storage or Azure Event Hubs. | Both | Yes | Yes |

+| **AzureDevOps** | Azure DevOps. | Inbound | No | Yes |

+| **AzureLoadTestingInstanceManagement** | This service tag is used for inbound connectivity from Azure Load Testing service to the load generation instances injected into your virtual network in the private load testing scenario. <br/><br/>**Note:** This tag is intended to be used in Azure Firewall, NSG, UDR and all other gateways for inbound connectivity. | No | Yes |

+| **PowerPlatformPlex** | This tag represents the IP addresses used by the infrastructure to host Power Platform extension execution on behalf of the customer. | Inbound | Yes | Yes |

+| **ServiceFabric** | Azure Service Fabric.<br/><br/>**Note**: This tag represents the Service Fabric service endpoint for control plane per region. This enables customers to perform management operations for their Service Fabric clusters from their VNET endpoint. (For example, https:// westus.servicefabric.azure.com). | Both | No | No |

+### <a name="forced-tunneling"></a>Direct all traffic to the VPN tunnel (forced tunneling)

+You can include 0/0 if you're using the Azure VPN Client version 2.1900:39.0 or higher. Modify the downloaded profile xml file and add the **\<includeroutes>\<route>\<destination>\<mask> \</destination>\</mask>\</route>\</includeroutes>** tags. Make sure to update the version number to **2**.

+For more information about configuring forced tunneling, including additional configuration options, see [How to configure forced tunneling](how-to-forced-tunnel.md).

+## Create a Virtual WAN hub

+## Set up Point-to-site VPN

+## Advertise default route to clients

+## Download the Point-to-site VPN profile

+## Configure forced-tunneling for Azure VPN clients (OpenVPN)

+### Windows clients

+### MacOS clients