+ <!-- See also https://learn.microsoft.com/azure/active-directory-b2c/active-directory-b2c-reference-policies#linking-user-flows -->

+ * To learn more about B2C tokens, visit https://learn.microsoft.com/azure/active-directory-b2c/tokens-overview

+ // Handling SameSite cookie according to https://learn.microsoft.com/aspnet/core/security/samesite?view=aspnetcore-3.1

+ // Handling SameSite cookie according to https://learn.microsoft.com/aspnet/core/security/samesite?view=aspnetcore-3.1

+## Prerequisites

+ var termsOfUseUrl = 'https://learn.microsoft.com/legal/termsofuse';

+# https://learn.microsoft.com/azure/active-directory-b2c/microsoft-graph-get-started

+You can automate other administration tasks, for example, [manage Azure AD B2C user accounts with Microsoft Graph](microsoft-graph-operations.md).

+Azure AD DS creates additional resources to function properly, such as public IP addresses, virtual network interfaces, and a load balancer. If any of these resources are modified, the managed domain is in an unsupported state and can't be managed. For more information about these resources, see [Network resources used by Azure AD DS](network-considerations.md#network-resources-used-by-azure-ad-ds).

+This alert is generated when one of these required resources is modified and can't automatically be recovered by Azure AD DS. To resolve the alert, [open an Azure support request][azure-support] to fix the instance.

+FIDO2 security keys offer an alternative. FIDO2 security keys can replace weak credentials with strong hardware-backed public/private-key credentials which can't be reused, replayed, or shared across services. Security keys support shared device scenarios, allowing you to carry your credential with you and safely authenticate to an Azure Active Directory joined Windows 10 device thatΓÇÖs part of your organization.

+1. First, your authenticator needs to have a FIDO2 certification. We won't be able to work with providers who don't have a FIDO2 certification. To learn more about the certification, please visit this website: [https://fidoalliance.org/certification/](https://fidoalliance.org/certification/)

+ - You'll complete and send all passed results to Microsoft Engineering team

+| Provider | Biometric | USB | NFC | BLE | FIPS Certified | Contact |

+||:--:|::|::|::|:--:|--|

+| AuthenTrend | ![y] | ![y]| ![y]| ![y]| ![n] | https://authentrend.com/about-us/#pg-35-3 |

+| Ciright | ![n] | ![n]| ![y]| ![n]| ![n] | https://www.cyberonecard.com/ |

+| Crayonic | ![y] | ![n]| ![y]| ![y]| ![n] | https://www.crayonic.com/keyvault |

+| Ensurity | ![y] | ![y]| ![n]| ![n]| ![n] | https://www.ensurity.com/contact |

+| Excelsecu | ![y] | ![y]| ![y]| ![y]| ![n] | https://www.excelsecu.com/productdetail/esecufido2secu.html |

+| Feitian | ![y] | ![y]| ![y]| ![y]| ![y] | https://shop.ftsafe.us/pages/microsoft |

+| Fortinet | ![n] | ![y]| ![n]| ![n]| ![n] | https://www.fortinet.com/ |

+| Giesecke + Devrient (G+D) | ![y] | ![y]| ![y]| ![y]| ![n] | https://www.gi-de.com/en/identities/enterprise-security/hardware-based-authentication |

+| GoTrustID Inc. | ![n] | ![y]| ![y]| ![y]| ![n] | https://www.gotrustid.com/idem-key |

+| HID | ![n] | ![y]| ![y]| ![n]| ![n] | https://www.hidglobal.com/contact-us |

+| Hypersecu | ![n] | ![y]| ![n]| ![n]| ![n] | https://www.hypersecu.com/hyperfido |

+| IDmelon Technologies Inc. | ![y] | ![y]| ![y]| ![y]| ![n] | https://www.idmelon.com/#idmelon |

+| Kensington | ![y] | ![y]| ![n]| ![n]| ![n] | https://www.kensington.com/solutions/product-category/why-biometrics/ |

+| KONA I | ![y] | ![n]| ![y]| ![y]| ![n] | https://konai.com/business/security/fido |

+| NeoWave | ![n] | ![y]| ![y]| ![n]| ![n] | https://neowave.fr/en/products/fido-range/ |

+| Nymi | ![y] | ![n]| ![y]| ![n]| ![n] | https://www.nymi.com/nymi-band |

+| Octatco | ![y] | ![y]| ![n]| ![n]| ![n] | https://octatco.com/ |

+| OneSpan Inc. | ![n] | ![y]| ![n]| ![y]| ![n] | https://www.onespan.com/products/fido |

+| Swissbit | ![n] | ![y]| ![y]| ![n]| ![n] | https://www.swissbit.com/en/products/ishield-fido2/ |

+| Thales Group | ![n] | ![y]| ![y]| ![n]| ![y] | https://cpl.thalesgroup.com/access-management/authenticators/fido-devices |

+| Thetis | ![y] | ![y]| ![y]| ![y]| ![n] | https://thetis.io/collections/fido2 |

+| Token2 Switzerland | ![y] | ![y]| ![y]| ![n]| ![n] | https://www.token2.swiss/shop/product/token2-t2f2-alu-fido2-u2f-and-totp-security-key |

+| TrustKey Solutions | ![y] | ![y]| ![n]| ![n]| ![n] | https://www.trustkeysolutions.com/security-keys/ |

+| VinCSS | ![n] | ![y]| ![n]| ![n]| ![n] | https://passwordless.vincss.net |

+| Yubico | ![y] | ![y]| ![y]| ![n]| ![y] | https://www.yubico.com/solutions/passwordless/ |

+<!--Image references-->

+[y]: ./media/fido2-compatibility/yes.png

+[n]: ./media/fido2-compatibility/no.png

+ # See https://learn.microsoft.com/azure/active-directory/azuread-dev/azure-ad-endpoint-comparison#scopes-not-resources

+ # See https://learn.microsoft.com/azure/active-directory/develop/v2-permissions-and-consent#the-default-scope

+// https://learn.microsoft.com/azure/active-directory-b2c/active-directory-b2c-reference-tokens

+// Add a token cache, see https://learn.microsoft.com/azure/active-directory/develop/msal-net-token-cache-serialization?tabs=desktop

+ // See https://learn.microsoft.com/dotnet/standard/threading/cancellation-in-managed-threads

+> To be notified of updates to this page, add this URL to your RSS feed reader:<br/>`https://learn.microsoft.com/api/search/rss?search=%22Azure+Active+Directory+breaking+changes+reference%22&locale=en-us`

+ // See https://learn.microsoft.com/dotnet/standard/threading/cancellation-in-managed-threads

+// Add a token cache, see https://learn.microsoft.com/azure/active-directory/develop/msal-net-token-cache-serialization?tabs=desktop

+ * https://learn.microsoft.com/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow

+The Microsoft identity platform supports authentication for various modern app architectures, all of them based on industry-standard protocols [OAuth 2.0 or OpenID Connect](active-directory-v2-protocols.md). This article describes the types of apps that you can build by using Microsoft identity platform, regardless of your preferred language or platform. The information is designed to help you understand high-level scenarios before you start working with the code in the [application scenarios](authentication-flows-app-scenarios.md#application-scenarios).

+Many modern apps have a single-page app front end written primarily in JavaScript, often with a framework like Angular, React, or Vue. The Microsoft identity platform supports these apps by using the [OpenID Connect](v2-protocols-oidc.md) protocol for authentication and one of two types of authorization grants defined by OAuth 2.0. The supported grant types are either the [OAuth 2.0 implicit grant flow](v2-oauth2-implicit-grant-flow.md) or the more recent [OAuth 2.0 authorization code + PKCE flow](v2-oauth2-auth-code-flow.md) (see below).

+For most of the history of OAuth 2.0, the [implicit flow](v2-oauth2-implicit-grant-flow.md) was the recommended way to build single-page apps. With the removal of [third-party cookies](reference-third-party-cookies-spas.md) and [greater attention](https://tools.ietf.org/html/draft-ietf-oauth-security-topics-14) paid to security concerns around the implicit flow, the authorization code flow for single-page apps should now be implemented to ensure compatibility of your app in Safari and other privacy-conscious browsers. The continued use of the implicit flow is not recommended.

+Further details of different types of tokens used in the Microsoft identity platform are available in the [access token](access-tokens.md) reference and [id_token](id-tokens.md) reference.

+To see this scenario in action, try the code samples in [Sign in users from a Web app](scenario-web-app-sign-user-overview.md).

+In addition to simple sign-in, a web server app might need to access another web service, such as a Representational State Transfer ([REST](https://docs.microsoft.com/rest/api/azure/)) API. In this case, the web server app engages in a combined OpenID Connect and OAuth 2.0 flow, by using the [OAuth 2.0 authorization code flow](v2-oauth2-auth-code-flow.md). For more information about this scenario, refer to our code [sample](https://github.com/Azure-Samples/active-directory-aspnetcore-webapp-openidconnect-v2/blob/master/2-WebApp-graph-user/2-1-Call-MSGraph/README.md).

+Learn more about [how to manage inactive user accounts in Azure AD](https://learn.microsoft.com/azure/active-directory/reports-monitoring/howto-manage-inactive-user-accounts).

+2. Create a review to remove inactive external guests. Admins define inactive as period of days. They disable and later delete guests that donΓÇÖt sign in to the tenant within that time frame. By default, this doesn't affect recently created users. [Learn more about how to identify inactive accounts](https://learn.microsoft.com/azure/active-directory/reports-monitoring/howto-manage-inactive-user-accounts#how-to-detect-inactive-user-accounts).

+1. Create a [dynamic group](https://learn.microsoft.com/azure/active-directory/enterprise-users/groups-create-rule) for the guest users you want to review. For example,

+2. To [create an Access Review](https://learn.microsoft.com/azure/active-directory/governance/create-access-review)

+1. Create a [dynamic group](https://learn.microsoft.com/azure/active-directory/enterprise-users/groups-create-rule) for the guest users you want to review. For example,

+2. To [create an access review](https://learn.microsoft.com/azure/active-directory/governance/create-access-review) for the dynamic group, navigate to **Azure Active Directory > Identity Governance > Access Reviews**.

+As a developer or IT administrator, you can use [API connectors](self-service-sign-up-add-api-connector.md#create-an-api-connector) to integrate your [self-service sign-up user flows](self-service-sign-up-overview.md) with web APIs to customize the sign-up experience and integrate with external systems. For example, with API connectors, you can:

+- Learn how to [add a custom approval system to self-service sign-up](self-service-sign-up-add-approvals.md)

+ // Initialize the Microsoft Graph authentication provider. For more info, visit: https://learn.microsoft.com/graph/sdks/choose-authentication-providers?tabs=Javascript#using--for-server-side-applications

+ // Execute the MS Graph command. For more information, visit: https://learn.microsoft.com/graph/api/invitation-post

+* [Secure your application by using OpenID Connect and Azure AD](/training/modules/secure-app-with-oidc-and-azure-ad/)

+| [End-user-initiated](multi-tenant-user-management-scenarios.md#end-user-initiated-scenario) | Resource tenant admins delegate the ability to invite guest users to the tenant, an app, or a resource to users within the resource tenant. Users from the home tenant are invited or sign up individually. | <li>Users need improvised access to resources. <li>No automatic synchronization of user attributes is necessary.<li>Unified GAL is not needed. |

+Organization | <ul><li>Read all company information<li>Read all domains<li>Read configuration of certificate-based authentication<li>Read all partner contracts</li></ul> | <ul><li>Read company display name<li>Read all domains<li>Read configuration of certificate-based authentication</li></ul> | <ul><li>Read company display name<li>Read all domains</li></ul>

+[Azure AD SAML Toolkit](../saas-apps/saml-toolkit-tutorial.md), [Otsuka Shokai (大塚商会)](../saas-apps/otsuka-shokai-tutorial.md), [ANAQUA](../saas-apps/anaqua-tutorial.md), [Azure VPN Client](https://portal.azure.com/), [ExpenseIn](../saas-apps/expensein-tutorial.md), [Helper Helper](../saas-apps/helper-helper-tutorial.md), [Costpoint](../saas-apps/costpoint-tutorial.md), [GlobalOne](../saas-apps/globalone-tutorial.md), [Mercedes-Benz In-Car Office](https://me.secure.mercedes-benz.com/), [Skore](https://app.justskore.it/), [Oracle Cloud Infrastructure Console](../saas-apps/oracle-cloud-tutorial.md), [CyberArk SAML Authentication](../saas-apps/cyberark-saml-authentication-tutorial.md), [Scrible Edu](https://www.scrible.com/sign-in/#/create-account), [PandaDoc](../saas-apps/pandadoc-tutorial.md), [Perceptyx](https://apexdata.azurewebsites.net/learn.microsoft.com/azure/active-directory/saas-apps/perceptyx-tutorial), Proptimise OS, [Vtiger CRM (SAML)](../saas-apps/vtiger-crm-saml-tutorial.md), Oracle Access Manager for Oracle Retail Merchandising, Oracle Access Manager for Oracle E-Business Suite, Oracle IDCS for E-Business Suite, Oracle IDCS for PeopleSoft, Oracle IDCS for JD Edwards

+>Get notified about when to revisit this page for updates by copying and pasting this URL: `https://learn.microsoft.com/api/search/rss?search=%22Release+notes+-+Azure+Active+Directory%22&locale=en-us` into your  feed reader.

+Tasks are specific actions that run automatically when a workflow is triggered. An Execution condition defines the 'Scope' of "who" and the 'Trigger' of "when" a workflow will be performed. For example, sending a manager an email 7 days before the value in the NewEmployeeHireDate attribute of new employees can be described as a workflow. It consists of:

+ - **Employee retirement/terminations/off-boarding** - That users who are no longer tied to the company for various reasons (termination, separation, leave of absence or retirement), have their access revoked in a timely manner.

+- Easily **troubleshoot** workflow scenarios with the Workflow history and Audit logs.

+- **Reduce** or remove manual tasks that were done in the past with automated lifecycle workflows.

+- **Apply** logic apps to extend workflows for more complex scenarios using your existing Logic apps.

+- [Create a Lifecycle workflow](create-lifecycle-workflow.md)

+* The Active Directory schema version and forest functional level must be Windows Server 2003 or later. The domain controllers can run any version as long as the schema version and forest-level requirements are met. You might require [a paid support program](/lifecycle/policies/fixed#extended-support) if you require support for domain controllers running Windows Server 2016 or older.

+- Azure AD Connect must be installed on a domain-joined Windows Server 2019 or later - note that Windows Server 2022 is not yet supported. You can deploy Azure AD Connect on Windows Server 2016 but since WS2016 is in extended support, you may require [a paid support program](/lifecycle/policies/fixed#extended-support) if you require support for this configuration.

+ - The servers where AD FS or Web Application Proxy are installed must be Windows Server 2012 R2 or later. Windows remote management must be enabled on these servers for remote installation. You may require [a paid support program](/lifecycle/policies/fixed#extended-support) if you require support for Windows Server 2016 and older.

+- Disable Soft Matching on your tenant. Soft Matching is a great feature to help transfering source of autority for existing cloud only objects to Azure AD Connect, but it comes with certain security risks. If you do not require it, you should [disable Soft Matching](how-to-connect-syncservice-features.md#blocksoftmatch)

+Your organization may use [MAM app protection policies](https://learn.microsoft.com/mem/intune/apps/app-protection-policy) to protect corporate data in apps on end users' devices.

+IT admins should [issue a selective wipe](https://learn.microsoft.com/mem/intune/apps/apps-selective-wipe) to impacted users following UPN changes. This will force impacted end users to reauthenticate and reenroll with their new UPNs.

+Understand the SP initiated flow by following the steps mentioned in [Datawiza and Azure AD authentication architecture](https://learn.microsoft.com/azure/active-directory/manage-apps/datawiza-with-azure-ad#datawiza-with-azure-ad-authentication-architecture).

+ - See, [Quickstart: Create a new tenant in Azure Active Directory.](https://learn.microsoft.com/azure/active-directory/fundamentals/active-directory-access-create-new-tenant)

+ - See, [Azure AD Connect sync: Understand and customize synchronization](https://learn.microsoft.com/azure/active-directory/hybrid/how-to-connect-sync-whatis).

+ - See, [Azure AD built-in roles, all roles](https://learn.microsoft.com/azure/active-directory/roles/permissions-reference#all-roles).

+portal](https://learn.microsoft.com/azure/active-directory/authentication/tutorial-enable-azure-mfa).

+- [Configure Datawiza and Azure AD for secure hybrid access](https://learn.microsoft.com/azure/active-directory/manage-apps/datawiza-with-azure-ad)

+- [Configure Datawiza with Azure AD B2C](https://learn.microsoft.com/azure/active-directory-b2c/partner-datawiza)

+- Consider completing the single sign-on training in [Enable single sign-on for applications by using Azure Active Directory](/training/modules/enable-single-sign-on).

+To review the admin consent requests programmatically, use the [appConsentRequest resource type](/graph/api/resources/appconsentrequest) and [userConsentRequest resource type](/graph/api/resources/userconsentrequest) and their associated methods in Microsoft Graph. You cannot approve or deny consent requests using Microsoft Graph.

+Users often are unable to consent to the permissions an application is requesting. Configure the admin consent workflow to allow users to provide a justification and request an administrator's review and approval of an application. For training on how to configure admin consent workflow in your Azure AD tenant, see [Configure admin consent workflow](/training/modules/configure-admin-consent-workflow).

+Consider implementing SSO in your application. You can manually configure most applications for SSO. The most popular options in Azure AD are [SAML-based SSO and OpenID Connect-based SSO](../develop/active-directory-v2-protocols.md). Before you start, make sure that you understand the requirements for SSO and how to [plan for deployment](plan-sso-deployment.md). For training related to configuring SAML-based SSO for an enterprise application in your Azure AD tenant, see [Enable single sign-on for an application by using Azure Active Directory](/training/modules/enable-single-sign-on).

+> [!VIDEO https://learn.microsoft.com/Shows/On-NET/Using-Azure-Managed-identities/player?format=ny]

+For more information about this cmdlet, see [Get-MgAuditLogSignIn](https://learn.microsoft.com/powershell/module/microsoft.graph.reports/get-mgauditlogsignin?view=graph-powershell-1.0).

+* [Sign-ins report overview](concept-sign-ins.md)

+| IdP Entity ID (`idp-entity-id`) | Azure AD Identifier |

+| IdP Single Sign-On URL (`idp-single-sign-on-url`) | Azure Login URL |

+ set idp-entity-id <Azure AD Identifier>

+ set idp-single-sign-on-url <Azure Login URL>

+ | FirstName | user.givenname |

+ | LastName | user.surname |

+1. On the **Set up single sign-on with SAML** page, in the **SAML Signing Certificate** section, find **Certificate (PEM)** and select **Download** to download the certificate and save it on your computer.

+ 

+1. Log in to the MURAL Identity website as an administrator.

+1. Click your **name** in the bottom left corner of the dashboard and select **Company dashboard** from the list of options.

+1. Click **SSO** in the left sidebar and perform the below steps.

+ 

+a. Download the **MURAL's metadata**.

+b. In the **Sign in URL** textbox, paste the **Login URL** value, which you have copied from the Azure portal.

+c. In the **Sign in certificate**, upload the **Certificate (PEM)**, which you have downloaded from the Azure portal.

+d. Select **HTTP-POST** as the Request binding type and select **SHA256** as the Sign in algorithm type.

+e. In the **Claim mapping** section, fill the following fields.

+* Email address: `http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress`

+* First name: `http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname`

+* Last name: `http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname`

+f. Click **Test single sign-on** to test the configuration and **Save** it.

+> [!NOTE]

+> For more information on how to configure the SSO at MURAL, please follow [this](https://support.mural.co/articles/6224385-mural-s-azure-ad-integration) support page.

+* Go to MURAL Identity Sign on URL directly and initiate the login flow from there.

+You can also use Microsoft My Apps to test the application in any mode. When you click the MURAL Identity tile in the My Apps, if configured in SP mode you would be redirected to the application sign-on page for initiating the login flow and if configured in IDP mode, you should be automatically signed in to the MURAL Identity for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+ Title: 'Tutorial: Azure AD SSO integration with RocketReach SSO'

+description: Learn how to configure single sign-on between Azure Active Directory and RocketReach SSO.

+# Tutorial: Azure AD SSO integration with RocketReach SSO

+In this tutorial, you'll learn how to integrate RocketReach SSO with Azure Active Directory (Azure AD). When you integrate RocketReach SSO with Azure AD, you can:

+* Control in Azure AD who has access to RocketReach SSO.

+* Enable your users to be automatically signed-in to RocketReach SSO with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+## Prerequisites

+To get started, you need the following items:

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* RocketReach SSO single sign-on (SSO) enabled subscription.

+* Along with Cloud Application Administrator, Application Administrator can also add or manage applications in Azure AD.

+For more information, see [Azure built-in roles](../roles/permissions-reference.md).

+## Scenario description

+In this tutorial, you configure and test Azure AD SSO in a test environment.

+* RocketReach SSO supports **SP** and **IDP** initiated SSO.

+* RocketReach SSO supports **Just In Time** user provisioning.

+## Add RocketReach SSO from the gallery

+To configure the integration of RocketReach SSO into Azure AD, you need to add RocketReach SSO from the gallery to your list of managed SaaS apps.

+1. Sign in to the Azure portal using either a work or school account, or a personal Microsoft account.

+1. On the left navigation pane, select the **Azure Active Directory** service.

+1. Navigate to **Enterprise Applications** and then select **All Applications**.

+1. To add new application, select **New application**.

+1. In the **Add from the gallery** section, type **RocketReach SSO** in the search box.

+1. Select **RocketReach SSO** from results panel and then add the app. Wait a few seconds while the app is added to your tenant.

+Alternatively, you can also use the [Enterprise App Configuration Wizard](https://portal.office.com/AdminPortal/home?Q=Docs#/azureadappintegration). In this wizard, you can add an application to your tenant, add users/groups to the app, assign roles, as well as walk through the SSO configuration as well. You can learn more about O365 wizards [here](/microsoft-365/admin/misc/azure-ad-setup-guides?view=o365-worldwide&preserve-view=true).

+## Configure and test Azure AD SSO for RocketReach SSO

+Configure and test Azure AD SSO with RocketReach SSO using a test user called **B.Simon**. For SSO to work, you need to establish a link relationship between an Azure AD user and the related user at RocketReach SSO.

+To configure and test Azure AD SSO with RocketReach SSO, perform the following steps:

+1. **[Configure Azure AD SSO](#configure-azure-ad-sso)** - to enable your users to use this feature.

+ 1. **[Create an Azure AD test user](#create-an-azure-ad-test-user)** - to test Azure AD single sign-on with B.Simon.

+ 1. **[Assign the Azure AD test user](#assign-the-azure-ad-test-user)** - to enable B.Simon to use Azure AD single sign-on.

+1. **[Configure RocketReach SSO](#configure-rocketreach-sso)** - to configure the single sign-on settings on application side.

+ 1. **[Create RocketReach SSO test user](#create-rocketreach-sso-test-user)** - to have a counterpart of B.Simon in RocketReach SSO that is linked to the Azure AD representation of user.

+1. **[Test SSO](#test-sso)** - to verify whether the configuration works.

+## Configure Azure AD SSO

+Follow these steps to enable Azure AD SSO in the Azure portal.

+1. In the Azure portal, on the **RocketReach SSO** application integration page, find the **Manage** section and select **single sign-on**.

+1. On the **Select a single sign-on method** page, select **SAML**.

+1. On the **Set up single sign-on with SAML** page, click the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+1. On the **Basic SAML Configuration** section, the user does not have to perform any step as the app is already pre-integrated with Azure.

+1. Click **Set additional URLs** and perform the following step if you wish to configure the application in **SP** initiated mode:

+ In the **Sign-on URL** text box, type the URL:

+ `https://rocketreach.co/login/sso`

+1. On the **Set up single sign-on with SAML** page, in the **SAML Signing Certificate** section, find **Certificate (Base64)** and select **Download** to download the certificate and save it on your computer.

+ 

+1. On the **Set up RocketReach SSO** section, copy the appropriate URL(s) based on your requirement.

+ 

+### Create an Azure AD test user

+In this section, you'll create a test user in the Azure portal called B.Simon.

+1. From the left pane in the Azure portal, select **Azure Active Directory**, select **Users**, and then select **All users**.

+1. Select **New user** at the top of the screen.

+1. In the **User** properties, follow these steps:

+ 1. In the **Name** field, enter `B.Simon`.

+ 1. In the **User name** field, enter the username@companydomain.extension. For example, `B.Simon@contoso.com`.

+ 1. Select the **Show password** check box, and then write down the value that's displayed in the **Password** box.

+ 1. Click **Create**.

+### Assign the Azure AD test user

+In this section, you'll enable B.Simon to use Azure single sign-on by granting access to RocketReach SSO.

+1. In the Azure portal, select **Enterprise Applications**, and then select **All applications**.

+1. In the applications list, select **RocketReach SSO**.

+1. In the app's overview page, find the **Manage** section and select **Users and groups**.

+1. Select **Add user**, then select **Users and groups** in the **Add Assignment** dialog.

+1. In the **Users and groups** dialog, select **B.Simon** from the Users list, then click the **Select** button at the bottom of the screen.

+1. If you are expecting a role to be assigned to the users, you can select it from the **Select a role** dropdown. If no role has been set up for this app, you see "Default Access" role selected.

+1. In the **Add Assignment** dialog, click the **Assign** button.

+## Configure RocketReach SSO

+To configure single sign-on on **RocketReach SSO** side, you need to send the downloaded **Certificate (Base64)** and appropriate copied URLs from Azure portal to [RocketReach SSO support team](mailto:support@rocketreach.co). They set this setting to have the SAML SSO connection set properly on both sides.

+### Create RocketReach SSO test user

+In this section, a user called B.Simon is created in RocketReach SSO. RocketReach SSO supports just-in-time user provisioning, which is enabled by default. There is no action item for you in this section. If a user doesn't already exist in RocketReach SSO, a new one is created after authentication.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+#### SP initiated:

+* Click on **Test this application** in Azure portal. This will redirect to RocketReach SSO Sign-on URL where you can initiate the login flow.

+* Go to RocketReach SSO Sign-on URL directly and initiate the login flow from there.

+#### IDP initiated:

+* Click on **Test this application** in Azure portal and you should be automatically signed in to the RocketReach SSO for which you set up the SSO.

+You can also use Microsoft My Apps to test the application in any mode. When you click the RocketReach SSO tile in the My Apps, if configured in SP mode you would be redirected to the application sign-on page for initiating the login flow and if configured in IDP mode, you should be automatically signed in to the RocketReach SSO for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Next steps

+Once you configure RocketReach SSO you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+## Choose a shortname for your Workspace in Sketch

+Follow these steps to choose a shortname and gather information to continue the setup process in Azure AD.

+>[!Note]

+> Before starting this process, make sure SSO is available in your Workspace, check there is an SSO tab in your Workspace Admin panel.

+> If you don't see the SSO tab, please reach out to customer support.

+1. [Sign in to your Workspace](https://www.sketch.com/signin/) as an Admin.

+1. Head to the **People & Settings** section in the sidebar.

+1. Click on the **Single Sign-On** tab.

+1. Click **Choose** a short name.

+1. Enter a unique name, it should have less than 16 characters and can only include letters, numbers or hyphens. You can edit this name later on.

+1. Click **Submit**.

+1. Click on the first tab **Set Up Identity Provider**. In this tab, youΓÇÖll find the unique Workspace values youΓÇÖll need to set up the integration with Azure AD.

+ 1. **EntityID:** In Azure AD, this is the `Identifier` field.

+ 1. **ACS URL:** In Azure AD, this is the `Reply URL` field.

+Make sure to keep these values at hand! YouΓÇÖll need them in the next step. Click Copy next to each value to copy it to your clipboard.

+ a. In the **Identifier** textbox, use the `EntityID` field from the previous step. It looks like:

+ b. In the **Reply URL** textbox, use the `ACS URL` field from the previous step. It looks like:

+1. Click **Set additional URLs** and perform the following step:

+ 

+Follow these steps to finish the configuration in Sketch.

+1. In your Workspace, head to the **Set up Sketch** tab in the **Single Sign-On** window.

+1. Upload the XML file you downloaded previously in the **Import XML Metadata file** section.

+1. Log out.

+1. Click **Sign in with SSO**.

+1. Use the shortname you configured previously to proceed.

+Once you configure Sketch you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+When onboarding users you can remove the need for error prone manual onboarding steps by using Verified ID with A10TIX account onboarding. Verified IDs can be used to digitally onboard employees, students, citizens, or others to securely access resources and services. For example, rather than an employee needing to go to a central office to activate an employee badge, they can use a Verified ID to verify their identity to activate a badge that is delivered to them remotely. Rather than a citizen receiving a code they must redeem to access governmental services, they can use a Verified ID to prove their identity and gain access. Learn more about [account onboarding](https://learn.microsoft.com/azure/active-directory/verifiable-credentials/plan-verification-solution#account-onboarding).

+Verifiable Credentials can be used to onboard employees, students, citizens, or others to access services. For example, rather than an employee needing to go to a central office to activate an employee badge, they can use a verifiable credential to verify their identity to activate a badge that is delivered to them remotely. Rather than a citizen receiving a code they must redeem to access governmental services, they can use a VC to prove their identity and gain access. Learn more about [account onboarding](https://learn.microsoft.com/azure/active-directory/verifiable-credentials/plan-verification-solution#account-onboarding).

+ Title: Sovereign cloud feature variations

+description: List of feature variations and usage limitations for Advisor in sovereign clouds.

+# Azure Advisor in sovereign clouds

+Azure sovereign clouds enable you to build and digitally transform workloads in the cloud while meeting your security, compliance, and policy requirements.

+## Azure Government (United States)

+The following Azure Advisor recommendation **features aren't currently available** in Azure Government:

+### Cost

+- (Preview) Consider App Service stamp fee reserved capacity to save over your on-demand costs.

+- (Preview) Consider Azure Data Explorer reserved capacity to save over your pay-as-you-go costs.

+- (Preview) Consider Azure Synapse Analytics (formerly SQL DW) reserved capacity to save over your pay-as-you-go costs.

+- (Preview) Consider Blob storage reserved capacity to save on Blob v2 and Data Lake Storage Gen2 costs.

+- (Preview) Consider Blob storage reserved instance to save on Blob v2 and Data Lake Storage Gen2 costs.

+- (Preview) Consider Cache for Redis reserved capacity to save over your pay-as-you-go costs.

+- (Preview) Consider Cosmos DB reserved capacity to save over your pay-as-you-go costs.

+- (Preview) Consider Database for MariaDB reserved capacity to save over your pay-as-you-go costs.

+- (Preview) Consider Database for MySQL reserved capacity to save over your pay-as-you-go costs.

+- (Preview) Consider Database for PostgreSQL reserved capacity to save over your pay-as-you-go costs.

+- (Preview) Consider SQL DB reserved capacity to save over your pay-as-you-go costs.

+- (Preview) Consider SQL PaaS DB reserved capacity to save over your pay-as-you-go costs.

+- Consider App Service stamp fee reserved instance to save over your on-demand costs.

+- Consider Azure Synapse Analytics (formerly SQL DW) reserved instance to save over your pay-as-you-go costs.

+- Consider Cache for Redis reserved instance to save over your pay-as-you-go costs.

+- Consider Cosmos DB reserved instance to save over your pay-as-you-go costs.

+- Consider Database for MariaDB reserved instance to save over your pay-as-you-go costs.

+- Consider Database for MySQL reserved instance to save over your pay-as-you-go costs.

+- Consider Database for PostgreSQL reserved instance to save over your pay-as-you-go costs.

+- Consider SQL PaaS DB reserved instance to save over your pay-as-you-go costs.

+### Operational

+- Add Azure Monitor to your virtual machine (VM) labeled as production.

+- Delete and recreate your pool using a VM size that will soon be retired.

+- Enable Traffic Analytics to view insights into traffic patterns across Azure resources.

+- Enforce 'Add or replace a tag on resources' using Azure Policy.

+- Enforce 'Allowed locations' using Azure Policy.

+- Enforce 'Allowed virtual machine SKUs' using Azure Policy.

+- Enforce 'Audit VMs that don't use managed disks' using Azure Policy.

+- Enforce 'Inherit a tag from the resource group' using Azure Policy.

+- Update Azure Spring Cloud API Version.

+- Update your outdated Azure Spring Cloud SDK to the latest version.

+- Upgrade to the latest version of the Immersive Reader SDK.

+### Performance

+- Accelerated Networking may require stopping and starting the VM.

+- Arista Networks vEOS Router may experience high CPU utilization, reduced throughput and high latency.

+- Barracuda Networks NextGen Firewall may experience high CPU utilization, reduced throughput and high latency.

+- Cisco Cloud Services Router 1000V may experience high CPU utilization, reduced throughput and high latency.

+- Consider increasing the size of your NVA to address persistent high CPU.

+- Distribute data in server group to distribute workload among nodes.

+- More than 75% of your queries are full scan queries.

+- NetApp Cloud Volumes ONTAP may experience high CPU utilization, reduced throughput and high latency.

+- Palo Alto Networks VM-Series Firewall may experience high CPU utilization, reduced throughput and high latency.

+- Reads happen on most recent data.

+- Rebalance data in Hyperscale (Citus) server group to distribute workload among worker nodes more evenly.

+- Update Attestation API Version.

+- Update Key Vault SDK Version.

+- Update to the latest version of your Arista VEOS product for Accelerated Networking support.

+- Update to the latest version of your Barracuda NG Firewall product for Accelerated Networking support.

+- Update to the latest version of your Check Point product for Accelerated Networking support.

+- Update to the latest version of your Cisco Cloud Services Router 1000V product for Accelerated Networking support.

+- Update to the latest version of your F5 BigIp product for Accelerated Networking support.

+- Update to the latest version of your NetApp product for Accelerated Networking support.

+- Update to the latest version of your Palo Alto Firewall product for Accelerated Networking support.

+- Upgrade your ExpressRoute circuit bandwidth to accommodate your bandwidth needs.

+- Use SSD Disks for your production workloads.

+- vSAN capacity utilization has crossed critical threshold.

+### Reliability

+- Avoid hostname override to ensure site integrity.

+- Check Point Virtual Machine may lose Network Connectivity.

+- Drop and recreate your HDInsight clusters to apply critical updates.

+- Upgrade device client SDK to a supported version for IotHub.

+- Upgrade to the latest version of the Azure Connected Machine agent.

+## Right size calculations

+The calculation for recommending that you should right-size or shut down underutilized virtual machines in Azure Government is as follows:

+- Advisor monitors your virtual machine usage for seven days and identifies low-utilization virtual machines.

+- Virtual machines are considered low utilization if their CPU utilization is 5% or less and their network utilization is less than 2%, or if the current workload can be accommodated by a smaller virtual machine size.

+If you want to be more aggressive at identifying underutilized virtual machines, you can adjust the CPU utilization rule on a per subscription basis.

+## Next steps

+For more information about Advisor recommendations, see:

+- [Introduction to Azure Advisor](./advisor-overview.md)

+- [Reliability recommendations](./advisor-high-availability-recommendations.md)

+- [Performance recommendations](./advisor-reference-performance-recommendations.md)

+- [Cost recommendations](./advisor-reference-cost-recommendations.md)

+- [Operational excellence recommendations](./advisor-reference-operational-excellence-recommendations.md)

+> For more information on how to use resource tags to organize and govern your Azure resources, please see the [Cloud Adoption FrameworkΓÇÖs guidance](/azure/cloud-adoption-framework/ready/azure-best-practices/resource-tagging) and [Build a cloud governance strategy on Azure](/training/modules/build-cloud-governance-strategy-azure/).

+[principles-sse]: /training/modules/sustainable-software-engineering-overview/

+> [!IMPORTANT]

+> Deployment Center for Azure Kubernetes Service will be retired on March 31, 2023. [Learn more](/azure/aks/deployment-center-launcher#retirement)

+## Retirement

+Deployment Center for Azure Kubernetes will be retired on March 31, 2023 in favor of [Automated deployments](/azure/aks/automated-deployments). We encourage you to switch for enjoy similar capabilities.

+#### Migration Steps

+There is no migration required as AKS Deployment center experience does not store any information itself, it just helps users with their Day 0 getting started experience on Azure. Moving forward, the recommended way for users to get started on CI/CD for AKS will be using [Automated deployments](/azure/aks/automated-deployments) feature.

+For existing pipelines, users will still be able to perform all operations from GitHub Actions or Azure DevOps after the retirement of this experience. Only the ability to create and view pipelines from Azure portal will be removed. See [GitHub Actions](https://docs.github.com/en/actions) or [Azure DevOps](/azure/devops/pipelines/get-started/pipelines-get-started) to learn how to get started.

+For new application deployments to AKS, instead of using Deployment center users can get the same capabilities by using Automated deployments.

+#### FAQΓÇ»

+1. Where can I manage my CD pipeline after this experience is deprecated?ΓÇ»

+Post retirement, you will not be able to view or create CD pipelines from Azure portalΓÇÖs AKS blade. However, as with the current experience, you can go to GitHub Actions or Azure DevOps portal and view or update the configured pipelines there.

+2. Will I lose my earlier configured pipelines?

+No. All the created pipelines will still be available and functional in GitHub or Azure DevOps. Only the experience of creating and viewing pipelines from Azure portal will be retired.

+3. How can I still configure CD pipelines directly through Azure portal?

+You can use Automated deployments available in the AKS blade in Azure portal.

+AKS nodes run the "chrony" service which pulls time from the localhost. Containers running on pods get the time from the AKS nodes. Applications launched inside a container use time from the container of the pod.

+> [Learn how to create multiple pipelines on GitHub Actions with AKS](/training/modules/aks-deployment-pipeline-github-actions)

+[actions/checkout]: https://github.com/actions/checkout

+Create a username to use as administrator credentials for the Windows Server nodes on your cluster. The following commands prompt you for a username and set it to *WINDOWS_USERNAME* for use in a later command (remember that the commands in this article are entered into a BASH shell).

+## Add a Windows Server 2022 node pool

+// https://learn.microsoft.com/azure/aks/troubleshooting#what-naming-restrictions-are-enforced-for-aks-resources-and-parameters

+AKS uses Windows Server 2019 and Windows Server 2022 as the host OS version and only supports process isolation. Container images built by using other Windows Server versions are not supported. For more information, see [Windows container version compatibility][windows-container-compat].

+Group-managed service account (gMSA) support is generally available for Windows on AKS. See [Enable Group Managed Service Accounts (GMSA) for your Windows Server nodes on your Azure Kubernetes Service (AKS) cluster](use-group-managed-service-accounts.md)

+- Take the [Cost Management](/training/paths/control-spending-manage-bills?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) guided learning course.

+[Migrate an on-premises web application to Azure App Service](/training/modules/migrate-app-service-migration-assistant/)

+[At-scale migration of .NET web apps](/training/modules/migrate-app-service-migration-assistant/)

+[At-scale assessment of .NET web apps](/training/modules/migrate-app-service-migration-assistant/)

+- [https://learn.microsoft.com/azure/cosmos-db/plan-manage-costs](../cosmos-db/plan-manage-costs.md)

+- [https://learn.microsoft.com/azure/storage/common/storage-plan-manage-costs](../storage/common/storage-plan-manage-costs.md)

+- [https://learn.microsoft.com/azure/machine-learning/concept-plan-manage-cost](../machine-learning/concept-plan-manage-cost.md)

+- Take the [Cost Management](/training/paths/control-spending-manage-bills?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) guided learning course.

+1. To upgrade to Debian 11 (Bullseye), create an app setting on your slot named `WEBSITE_LINUX_OS_VERSION` with a value of `DEBIAN|BULLSEYE`.

+ az webapp config appsettings set -g MyResourceGroup -n MyUniqueApp --settings WEBSITE_LINUX_OS_VERSION="DEBIAN|BULLSEYE"

+1. [Swap your production and staging slots](deploy-staging-slots.md#swap-two-slots). This will apply the `WEBSITE_LINUX_OS_VERSION=DEBIAN|BULLSEYE` app setting to production.

+To deploy applications to Azure App Service, developers can use the [Maven Plugin for App Service](/training/modules/publish-web-app-with-maven-plugin-for-azure-app-service/), [VSCode Extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azureappservice), or the Azure CLI to deploy apps. Use the following command to deploy our app to the App Service:

+> [Java in App Service Linux dev guide](configure-language-java.md?pivots=platform-linux)

+ <!-- If above is not run then it takes a whole day for references to update? https://learn.microsoft.com/azure/app-service/app-service-key-vault-references#rotation -->

+This article assumes you're already familiar with [Node.js development](/training/paths/build-javascript-applications-nodejs/) and have Node and MongoDB installed locally. You'll also need an Azure account with an active subscription. If you don't have an Azure account, you [can create one for free](https://azure.microsoft.com/free/nodejs/).

+> [Configure Node.js app in App Service](./configure-language-nodejs.md)

+* Knowledge of Python with Flask development or [Python with Django development](/training/paths/django-create-data-driven-websites/)

+ // Requires DI configuration to access app settings. See https://learn.microsoft.com/azure/app-service/configure-language-dotnetcore#access-environment-variables

+> [!Note]

+>Access logs with clientIP value 127.0.0.1 originate from an internal security process running on the application gateway instances. You can safely ignore these log entries.

+### Virtual network permission

+Since application gateway resources are deployed within a virtual network resource, Application Gateway performs a check to verify the permission on the provided virtual network resource. This is verified during both create and manage operations.

+You should check your [Azure role-based access control](../role-based-access-control/role-assignments-list-portal.md) to verify that users or Service Principals who operate application gateways have at least **Microsoft.Network/virtualNetworks/subnets/join/action** or some higher permission such as the built-in [Network contributor](../role-based-access-control/built-in-roles.md) role on the virtual network. Visit [Add, change, or delete a virtual network subnet](../virtual-network/virtual-network-manage-subnet.md) to know more on subnet permissions.

+If a [built-in](../role-based-access-control/built-in-roles.md) role doesn't provide the right permission, you can [create and assign a custom role](../role-based-access-control/custom-roles-portal.md) for this purpose.

+<!-- See https://learn.microsoft.com/azure/storage/common/monitor-storage-reference#metrics-dimensions for an example. Part is copied below. -->

+<!-- OPTION 1 - Minimum - Link to relevant bookmarks in https://learn.microsoft.com/azure/azure-monitor/reference/tables/tables-resourcetype where your service tables are listed. These files are auto generated from the REST API. If this article is missing tables that you and the PM know are available, both of you contact azmondocs@microsoft.com.

+- See [Monitoring Azure resources with Azure Monitor](../azure-monitor/essentials/monitor-azure-resource.md) for details on monitoring Azure resources.

+<!-- OPTIONAL: Add specific examples of configuration for this service. For example, CLI and PowerShell commands for creating diagnostic setting. Ideally, customers should set up a policy to automatically turn on collection for services. Azure monitor has Resource Manager template examples you can point to. See https://learn.microsoft.com/azure/azure-monitor/samples/resource-manager-diagnostic-settings. Contact azmondocs@microsoft.com if you have questions. -->

+- See [Monitoring Azure resources with Azure Monitor](../azure-monitor/essentials/monitor-azure-resource.md) for details on monitoring Azure resources.

+- [Learn module: Introduction to Azure Application Gateway](/training/modules/intro-to-azure-application-gateway)

+- [Learn module: Introduction to Azure Application Gateway](/training/modules/intro-to-azure-application-gateway)

+# https://learn.microsoft.com/azure/private-link/disable-private-endpoint-network-policy

+# https://learn.microsoft.com/azure/private-link/disable-private-endpoint-network-policy

+# https://learn.microsoft.com/azure/private-link/disable-private-endpoint-network-policy

+# https://learn.microsoft.com/azure/private-link/disable-private-endpoint-network-policy

+> [!VIDEO https://learn.microsoft.com/Shows/Docs-Azure/Azure-Form-Recognizer/player]

+> [!VIDEO https://learn.microsoft.com/Shows/Docs-Azure/Azure-Form-Recognizer/player]

+ // Learn more about chunk usage and supported MIME types https://learn.microsoft.com/azure/cognitive-services/immersive-reader/reference#chunk

+ // Learn more about options https://learn.microsoft.com/azure/cognitive-services/immersive-reader/reference#options

+* View code samples on [GitHub](https://github.com/microsoft/immersive-reader-sdk/tree/master/js/samples/advanced-csharp)

+ Your Automation account can now use the system-assigned identity, that is registered with Azure Active Directory (Azure AD) and is represented by an object ID.

+# https://learn.microsoft.com/azure/azure-arc/data/upload-logs

+> [Pre-release testing](preview-testing.md)

+Azure Arc-enabled data services provide you the option to connect to Azure in two different *connectivity modes*:

+|**Self-service provisioning**|Supported<br/>Use Azure Data Studio, the appropriate CLI, or Kubernetes native tools like Helm, `kubectl`, or `oc`, or use Azure Arc-enabled Kubernetes GitOps provisioning.|Supported<br/>In addition to the indirectly connected mode creation options, you can also create through the Azure portal, Azure Resource Manager APIs, the Azure CLI, or ARM templates.

+- [Helm chart (direct connected mode)](#helm-chart-direct-connected-mode)

+- [Azure Arc data processing service](#azure-arc-data-processing-service)

+### Helm chart (direct connected mode)

+The Helm chart used to provision the Azure Arc data controller bootstrapper and cluster level objects, such as custom resource definitions, cluster roles, and cluster role bindings, is pulled from an Azure Container Registry.

+To use proxy, verify that the agents meet the network requirements. See [Meet network requirements](../kubernetes/quickstart-connect-cluster.md#meet-network-requirements).

+### Azure Arc data processing service

+Points to the data processing service endpoint in connection

+#### Connection target

+- `san-af-eastus-prod.azurewebsites.net`

+- `san-af-eastus2-prod.azurewebsites.net`

+- `san-af-australiaeast-prod.azurewebsites.net`

+- `san-af-centralus-prod.azurewebsites.net`

+- `san-af-westus2-prod.azurewebsites.net`

+- `san-af-westeurope-prod.azurewebsites.net`

+- `san-af-southeastasia-prod.azurewebsites.net`

+- `san-af-koreacentral-prod.azurewebsites.net`

+- `san-af-northeurope-prod.azurewebsites.net`

+- `san-af-westeurope-prod.azurewebsites.net`

+- `san-af-uksouth-prod.azurewebsites.net`

+- `san-af-francecentral-prod.azurewebsites.net`

+#### Protocol

+HTTPS

+#### Can use proxy

+Yes

+To use proxy, verify that the agents meet the network requirements. See [Meet network requirements](../kubernetes/quickstart-connect-cluster.md#meet-network-requirements).

+#### Authentication

+None

+> [!VIDEO https://learn.microsoft.com/Shows/Inside-Azure-for-IT/Azure-Arcenabled-data-services-in-disconnected-mode/player?format=ny]

+> [!VIDEO https://learn.microsoft.com/Shows/Inside-Azure-for-IT/Azure-Arcenabled-data-services-in-connected-mode/player?format=ny]

+> [!VIDEO https://learn.microsoft.com/Shows/Inside-Azure-for-IT/Choose-the-right-data-solution-for-your-hybrid-environment/player?format=ny]

+> [!VIDEO https://learn.microsoft.com/Shows/Azure-Friday/Durable-Functions-in-Azure-Functions/player]

+> [!VIDEO https://learn.microsoft.com/Events/dotnetConf/2018/S204/player]

+| <ul><li>[Using Visual Studio](functions-create-your-first-function-visual-studio.md)</li><li>[Using Visual Studio Code](create-first-function-vs-code-csharp.md)</li><li>[Using command line tools](create-first-function-cli-csharp.md)</li></ul> | <ul><li>[Hosting options](functions-scale.md)</li><li>[Performance&nbsp;considerations](functions-best-practices.md)</li><li>[Visual Studio development](functions-develop-vs.md)</li><li>[Dependency injection](functions-dotnet-dependency-injection.md)</li></ul> | <ul><li>[Create serverless applications](/training/paths/create-serverless-applications/)</li><li>[C# samples](/samples/browse/?products=azure-functions&languages=csharp)</li></ul> |

+| **Explore an interactive tutorial**| <li>[Choose the best Azure serverless technology for your business scenario](/training/modules/serverless-fundamentals/)<li>[Well-Architected Framework - Performance efficiency](/training/modules/azure-well-architected-performance-efficiency/)<li>[Execute an Azure Function with triggers](/training/modules/execute-azure-function-with-triggers/) <br><br>See a [full listing of interactive tutorials](/training/browse/?expanded=azure&products=azure-functions).|

+| **Explore an interactive tutorial**| <li>[Choose the best Azure serverless technology for your business scenario](/training/modules/serverless-fundamentals/)<li>[Well-Architected Framework - Performance efficiency](/training/modules/azure-well-architected-performance-efficiency/)<li>[Develop an App using the Maven Plugin for Azure Functions](/training/modules/develop-azure-functions-app-with-maven-plugin/) <br><br>See a [full listing of interactive tutorials](/training/browse/?expanded=azure&products=azure-functions).|

+| **Explore an interactive tutorial** | <li>[Choose the best Azure serverless technology for your business scenario](/training/modules/serverless-fundamentals/)<li>[Well-Architected Framework - Performance efficiency](/training/modules/azure-well-architected-performance-efficiency/)<li>[Build Serverless APIs with Azure Functions](/training/modules/build-api-azure-functions/)<li>[Create serverless logic with Azure Functions](/training/modules/create-serverless-logic-with-azure-functions/)<li>[Refactor Node.js and Express APIs to Serverless APIs with Azure Functions](/training/modules/shift-nodejs-express-apis-serverless/) <br><br>See a [full listing of interactive tutorials](/training/browse/?expanded=azure&products=azure-functions).|

+| **Explore an interactive tutorial** | <li>[Choose the best Azure serverless technology for your business scenario](/training/modules/serverless-fundamentals/)<li>[Well-Architected Framework - Performance efficiency](/training/modules/azure-well-architected-performance-efficiency/)<li>[Build Serverless APIs with Azure Functions](/training/modules/build-api-azure-functions/)<li>[Create serverless logic with Azure Functions](/training/modules/create-serverless-logic-with-azure-functions/)<li>[Execute an Azure Function with triggers](/training/modules/execute-azure-function-with-triggers/) <br><br>See a [full listing of interactive tutorials](/training/browse/?expanded=azure&products=azure-functions).|

+| **Explore an interactive tutorial** | <li>[Choose the best Azure serverless technology for your business scenario](/training/modules/serverless-fundamentals/)<li>[Well-Architected Framework - Performance efficiency](/training/modules/azure-well-architected-performance-efficiency/)<li>[Build Serverless APIs with Azure Functions](/training/modules/build-api-azure-functions/)<li>[Create serverless logic with Azure Functions](/training/modules/create-serverless-logic-with-azure-functions/) <br><br>See a [full listing of interactive tutorials](/training/browse/?expanded=azure&products=azure-functions).|

+# https://learn.microsoft.com/windows/win32/winrm/installation-and-configuration-for-windows-remote-management.

+1. Navigate to the Kudu endpoint for the function app, which is located at `https://<FUNCTION_APP>.scm.azurewebsites.net`, where `<FUNCTION_APP>` is the name of your app.

+Azure Functions lets you develop functions using C# in one of the following ways:

+| Type | Execution process | Code extension | Development environment | Reference |

+| | - | | | |

+| C# script | in-process | .csx | [Portal](functions-create-function-app-portal.md)<br/>[Core Tools](functions-run-local.md) | This article |

+| C# class library | in-process | .cs | [Visual Studio](functions-develop-vs.md)<br/>[Visual Studio Code](functions-develop-vs-code.md)<br />[Core Tools](functions-run-local.md)s | [In-process C# class library functions](functions-dotnet-class-library.md) |

+| C# class library (isolated process)| out-of-process | .cs | [Visual Studio](functions-develop-vs.md)<br/>[Visual Studio Code](functions-develop-vs-code.md)<br />[Core Tools](functions-run-local.md) | [.NET isolated process functions](dotnet-isolated-process-guide.md) |

+Data flows into your C# function via method arguments. Argument names are specified in a `function.json` file, and there are predefined names for accessing things like the function logger and cancellation tokens.

+A function app's *.csx* files are compiled when an instance is initialized. This compilation step means things like cold start may take longer for C# script functions compared to C# class libraries. This compilation step is also why C# script functions are editable in the Azure portal, while C# class libraries aren't.

+The folder structure for a C# script project looks like the following example:

+The binding extensions required in [version 2.x and later versions](functions-versions.md) of the Functions runtime are defined in the `extensions.csproj` file, with the actual library files in the `bin` folder. When developing locally, you must [register binding extensions](./functions-bindings-register.md#extension-bundles). When you develop functions in the Azure portal, this registration is done for you.

+The following assemblies may be referenced by simple-name, by runtime version:

+# [v2.x+](#tab/functionsv2)

+* `Newtonsoft.Json`

+* `Microsoft.WindowsAzure.Storage`^*

+^*Removed in version 4.x of the runtime.

+# [v1.x](#tab/functionsv1)

+In code, assemblies are referenced like the following example:

+```csharp

+#r "AssemblyName"

+```

+By default, Core Tools reads the function.json files and adds the required packages to an *extensions.csproj* C# class library project file in the root of the function app's file system (wwwroot). Because Core Tools uses dotnet.exe, you can use it to add any NuGet package reference to this extensions file. During installation, Core Tools builds the extensions.csproj to install the required libraries. Here's an example *extensions.csproj* file that adds a reference to *Microsoft.ProjectOxford.Face* version *1.1.0*:

+Version 1.x of the Functions runtime uses a *project.json* file to define dependencies. Here's an example *project.json* file:

+If you're working on your project only in the portal, you'll need to manually create the extensions.csproj file or a Nuget.Config file directly in the site. To learn more, see [Manually install extensions](functions-how-to-use-azure-function-app-settings.md#manually-install-extensions).

+supported by that binding type. `T` can't be an `out` parameter type (such as `out JObject`). For example, the

+### Multiple attributes example

+The following table lists the .NET attributes for each binding type and the packages in which they're defined.

+| <ul><li>[Node.js function using Visual Studio Code](./create-first-function-vs-code-node.md)</li><li>[Node.js function with terminal/command prompt](./create-first-function-cli-node.md)</li><li>[Node.js function using the Azure portal](functions-create-function-app-portal.md)</li></ul> | <ul><li>[Developer guide](functions-reference.md)</li><li>[Hosting options](functions-scale.md)</li><li>[TypeScript functions](#typescript)</li><li>[Performance&nbsp; considerations](functions-best-practices.md)</li></ul> | <ul><li>[Create serverless applications](/training/paths/create-serverless-applications/)</li><li>[Refactor Node.js and Express APIs to Serverless APIs](/training/modules/shift-nodejs-express-apis-serverless/)</li></ul> |

+Learn how to [develop, test, and publish Azure functions by using Azure Functions core tools](/training/modules/develop-test-deploy-azure-functions-with-core-tools/). Azure Functions Core Tools is [open source and hosted on GitHub](https://github.com/azure/azure-functions-cli). To file a bug or feature request, [open a GitHub issue](https://github.com/azure/azure-functions-cli/issues).

+Connection strings and other credentials stored in application settings gives all of the functions in the function app the same set of permissions in the associated resource. Consider minimizing the number of functions with access to specific credentials by moving functions that don't use those credentials to a separate function app. You can always use techniques such as [function chaining](/training/modules/chain-azure-functions-data-using-bindings/) to pass data between functions in different function apps.

+Connection strings and other credentials stored in application settings gives all of the functions in the function app the same set of permissions in the associated resource. Consider minimizing the number of functions with access to specific credentials by moving functions that don't use those credentials to a separate function app. You can always use techniques such as [function chaining](/training/modules/chain-azure-functions-data-using-bindings/) to pass data between functions in different function apps.

+> Learn more through the interactive tutorial [Refactor Node.js and Express APIs to Serverless APIs with Azure Functions](/training/modules/shift-nodejs-express-apis-serverless/).

+- Learn more with the interactive tutorial [Refactor Node.js and Express APIs to Serverless APIs with Azure Functions](/training/modules/shift-nodejs-express-apis-serverless/)

+For feature variations and limitations, including API endpoints, see [Speech service in sovereign clouds](../cognitive-services/speech-service/sovereign-clouds.md).

+For feature variations and limitations, including API endpoints, see [Translator in sovereign clouds](../cognitive-services/translator/sovereign-clouds.md).

+For feature variations and limitations, see [Azure Advisor in sovereign clouds](../advisor/advisor-sovereign-clouds.md).

+ Write-Output "Script requires Azure PowerShell modules to be present. Obtain Azure PowerShell from https://learn.microsoft.com//powershell/azure/install-az-ps" -Verbose

+[Artificial intelligence](/training/modules/azure-artificial-intelligence/1-introduction-to-azure-artificial-intelligence) (AI) holds tremendous potential for governments. [Machine learning](/training/modules/azure-artificial-intelligence/3-machine-learning) (ML) is a data science technique that allows computers to learn to use existing data, without being explicitly programmed, to forecast future behaviors, outcomes, and trends. Moreover, [ML technologies](/azure/architecture/data-guide/technology-choices/data-science-and-machine-learning) can discover patterns, anomalies, and predictions that can help governments in their missions. As technical barriers continue to fall, decision-makers face the opportunity to develop and explore transformative AI applications. There are five main vectors that can make it easier, faster, and cheaper to adopt ML:

+The exponential growth of unstructured data gathering in recent years has created many analytical problems for government agencies. This problem intensifies when data sets come from diverse sources such as text, audio, video, imaging, and so on. [Knowledge mining](/training/modules/azure-artificial-intelligence/2-knowledge-mining) is the process of discovering useful knowledge from a collection of diverse data sources. This widely used data mining technique is a process that includes data preparation and selection, data cleansing, incorporation of prior knowledge on data sets, and interpretation of accurate solutions from the observed results. This process has proven to be useful for large volumes of data in different government agencies.

+> [!VIDEO https://learn.microsoft.com/Shows/Azure-Friday/Enable-government-missions-in-the-cloud-with-Azure-Government/player]

+> [!VIDEO https://learn.microsoft.com/Shows/Internet-of-Things-Show/Azure-Maps/player?format=ny]

+>[!VIDEO https://learn.microsoft.com/Shows/Internet-of-Things-Show/Clustering-point-data-in-Azure-Maps/player?format=ny]

+>[!VIDEO https://learn.microsoft.com/Shows/Internet-of-Things-Show/Clustering-point-data-in-Azure-Maps/player?format=ny]

+>[!VIDEO https://learn.microsoft.com/Shows/Internet-of-Things-Show/Data-Driven-Styling-with-Azure-Maps/player?format=ny]

+>[!VIDEO https://learn.microsoft.com/Shows/Internet-of-Things-Show/Data-Driven-Styling-with-Azure-Maps/player?format=ny]

+>[!VIDEO https://learn.microsoft.com/Shows/Internet-of-Things-Show/Azure-Maps-Weather-services-for-developers/player?format=ny]

+> [!VIDEO https://learn.microsoft.com/Shows/Internet-of-Things-Show/Easily-integrate-spatial-data-into-the-Azure-Maps/player?format=ny]

+> [Accessibility in Action Digital Badge learning path](https://ready.azurewebsites.net/learning/track/2940)

+> [!VIDEO https://learn.microsoft.com/Shows/Internet-of-Things-Show/Heat-Maps-and-Image-Overlays-in-Azure-Maps/player?format=ny]

+>[!VIDEO https://learn.microsoft.com/Shows/Internet-of-Things-Show/Heat-Maps-and-Image-Overlays-in-Azure-Maps/player?format=ny]

+| Geolocation | Γ£ô |

+Use [Profile Now](../profiler/profiler-settings.md) for the on-demand profiling option. [Profile Now](../profiler/profiler-settings.md) will immediately profile all agents attached to the Application Insights instance.

+@description('See documentation on tags: https://learn.microsoft.com/azure/azure-resource-manager/management/tag-resources.')

+ "description": "See documentation on tags: https://learn.microsoft.com/azure/azure-resource-manager/management/tag-resources."

+@description('See documentation on tags: https://learn.microsoft.com/azure/azure-resource-manager/management/tag-resources.')

+ "description": "See documentation on tags: https://learn.microsoft.com/azure/azure-resource-manager/management/tag-resources."

+This article describes how to enable Application Insights for an [ASP.NET Core](/aspnet/core) application deployed as an Azure Web App. This implementation uses an SDK-based approach. An [auto-instrumentation approach](./codeless-overview.md) is also available.

+For a sample application, we'll use an [ASP.NET Core MVC application](/aspnet/core/tutorials/first-mvc-app) that targets `net6.0`. However, you can apply these instructions to all ASP.NET Core applications. If you're using the [Worker Service](/aspnet/core/fundamentals/host/hosted-services#worker-service-template), use the instructions from [here](./worker-service.md).

+The [Application Insights SDK for ASP.NET Core](https://nuget.org/packages/Microsoft.ApplicationInsights.AspNetCore) can monitor your applications no matter where or how they run. If your application is running and has network connectivity to Azure, Application Insights can collect telemetry from it. Application Insights monitoring is supported everywhere .NET Core is supported. The following scenarios are supported:

+* **Web server**: Internet Information Server (IIS) or Kestrel

+To complete this tutorial, you need:

+* The following Visual Studio workloads:

+ * ASP.NET and web development

+ * Data storage and processing

+ * Azure development

+Please follow the [guidance to deploy the sample application from its GitHub repository.](https://github.com/gitopsbook/sample-app-deployment).

+In order to provide globally unique names to resources, a six-character suffix is assigned to some resources. Please make note of this suffix for use later on in this article.

+1. In the [Azure portal](https://portal.azure.com), select the **application-insights-azure-cafe** resource group.

+ :::image type="content" source="media/tutorial-asp-net-core/create-resource-menu.png" alt-text="Screenshot of the application-insights-azure-cafe resource group in the Azure portal with the + Create button highlighted on the toolbar menu." lightbox="media/tutorial-asp-net-core/create-resource-menu.png":::

+3. On the **Create a resource** screen, search for and select **Application Insights** in the marketplace search textbox.

+ <!-- The long description for search-application-insights.png: Screenshot of the Create a resource screen in the Azure portal. The screenshot shows a search for Application Insights highlighted and Application Insights displaying in the search results, which is also highlighted. -->

+ :::image type="content" source="media/tutorial-asp-net-core/search-application-insights.png" alt-text="Screenshot of the Create a resource screen in the Azure portal." lightbox="media/tutorial-asp-net-core/search-application-insights.png":::

+ :::image type="content" source="media/tutorial-asp-net-core/create-application-insights-overview.png" alt-text="Screenshot of the Application Insights overview screen in the Azure portal with the Create button highlighted." lightbox="media/tutorial-asp-net-core/create-application-insights-overview.png":::

+5. On the Application Insights screen, **Basics** tab, complete the form by using the following table, then select the **Review + create** button. Fields not specified in the table below may retain their default values.

+ | Log Analytics Workspace | Select **azure-cafe-log-analytics-workspace**. Alternatively, you can create a new log analytics workspace. |

+ :::image type="content" source="media/tutorial-asp-net-core/application-insights-basics-tab.png" alt-text="Screenshot of the Basics tab of the Application Insights screen in the Azure portal with a form populated with the preceding values." lightbox="media/tutorial-asp-net-core/application-insights-basics-tab.png":::

+ :::image type="content" source="media/tutorial-asp-net-core/application-insights-validation-passed.png" alt-text="Screenshot of the Application Insights screen in the Azure portal. The message stating validation has passed and Create button are both highlighted." lightbox="media/tutorial-asp-net-core/application-insights-validation-passed.png":::

+7. Once the resource is deployed, return to the `application-insights-azure-cafe` resource group, and select the Application Insights resource you deployed.

+ :::image type="content" source="media/tutorial-asp-net-core/application-insights-resource-group.png" alt-text="Screenshot of the application-insights-azure-cafe resource group in the Azure portal with the Application Insights resource highlighted." lightbox="media/tutorial-asp-net-core/application-insights-resource-group.png":::

+8. On the Overview screen of the Application Insights resource, select the **Copy to clipboard** button to copy the connection string value. You will use the connection string value in the next section of this article.

+ <!-- The long description for application-insights-connection-string-overview.png: Screenshot of the Application Insights Overview screen in the Azure portal. The screenshot shows the connection string value highlighted and the Copy to clipboard button selected and highlighted. -->

+ :::image type="content" source="media/tutorial-asp-net-core/application-insights-connection-string-overview.png" alt-text="Screenshot of the Application Insights Overview screen in the Azure portal." lightbox="media/tutorial-asp-net-core/application-insights-connection-string-overview.png":::

+1. Return to the `application-insights-azure-cafe` resource group and open the **azure-cafe-web-{SUFFIX}** App Service resource.

+ :::image type="content" source="media/tutorial-asp-net-core/web-app-service-resource-group.png" alt-text="Screenshot of the application-insights-azure-cafe resource group in the Azure portal with the azure-cafe-web-{SUFFIX} resource highlighted." lightbox="media/tutorial-asp-net-core/web-app-service-resource-group.png":::

+2. From the left menu, under the Settings section, select **Configuration**. Then, on the **Application settings** tab, select **+ New application setting** beneath the Application settings header.

+ <!-- The long description for app-service-app-setting-button.png: Screenshot of the App Service resource screen in the Azure portal. The screenshot shows Configuration in the left menu under the Settings section selected and highlighted, the Application settings tab selected and highlighted, and the + New application setting toolbar button highlighted. -->

+ :::image type="content" source="media/tutorial-asp-net-core/app-service-app-setting-button.png" alt-text="Screenshot of the App Service resource screen in the Azure portal." lightbox="media/tutorial-asp-net-core/app-service-app-setting-button.png":::

+ | Value | Paste the Application Insights connection string value you copied in the preceding section. |

+ :::image type="content" source="media/tutorial-asp-net-core/add-edit-app-setting.png" alt-text="Screenshot of the Add/Edit application setting blade in the Azure portal with the preceding values populated in the Name and Value fields." lightbox="media/tutorial-asp-net-core/add-edit-app-setting.png":::

+ :::image type="content" source="media/tutorial-asp-net-core/save-app-service-configuration.png" alt-text="Screenshot of the App Service Configuration screen in the Azure portal with the Save button highlighted on the toolbar menu." lightbox="media/tutorial-asp-net-core/save-app-service-configuration.png":::

+1. In Visual Studio, open `1 - Starter Application\src\AzureCafe.sln`.

+2. In the Visual Studio Solution Explorer panel, right-click on the AzureCafe project file and select **Manage NuGet Packages**.

+ :::image type="content" source="media/tutorial-asp-net-core/manage-nuget-packages-menu.png" alt-text="Screenshot of the Visual Studio Solution Explorer with the Azure Cafe project selected and the Manage NuGet Packages context menu item highlighted." lightbox="media/tutorial-asp-net-core/manage-nuget-packages-menu.png":::

+3. Select the **Browse** tab and then search for and select **Microsoft.ApplicationInsights.AspNetCore**. Select **Install**, and accept the license terms. It is recommended you use the latest stable version. For the full release notes for the SDK, see the [open-source GitHub repo](https://github.com/Microsoft/ApplicationInsights-dotnet/releases).

+ <!-- The long description for asp-net-core-install-nuget-package.png: Screenshot that shows the NuGet Package Manager user interface in Visual Studio with the Browse tab selected. Microsoft.ApplicationInsights.AspNetCore is entered in the search box, and the Microsoft.ApplicationInsights.AspNetCore package is selected from a list of results. In the right pane, the latest stable version of the Microsoft.ApplicationInsights.AspNetCore package is selected from a drop down list and the Install button is highlighted. -->

+ :::image type="content" source="media/tutorial-asp-net-core/asp-net-core-install-nuget-package.png" alt-text="Screenshot of the NuGet Package Manager user interface in Visual Studio." lightbox="media/tutorial-asp-net-core/asp-net-core-install-nuget-package.png":::

+ Keep Visual Studio open for the next section of the article.

+1. From the Visual Studio Solution Explorer, open the **Program.cs** file.

+ :::image type="content" source="media/tutorial-asp-net-core/solution-explorer-programcs.png" alt-text="Screenshot of the Visual Studio Solution Explorer with the Program.cs file highlighted." lightbox="media/tutorial-asp-net-core/solution-explorer-programcs.png":::

+2. Insert the following code prior to the `builder.Services.AddControllersWithViews()` statement. This code automatically reads the Application Insights connection string value from configuration. The `AddApplicationInsightsTelemetry` method registers the `ApplicationInsightsLoggerProvider` with the built-in dependency injection container that will then be used to fulfill [ILogger](/dotnet/api/microsoft.extensions.logging.ilogger) and [ILogger\<TCategoryName\>](/dotnet/api/microsoft.extensions.logging.iloggerprovider) implementation requests.

+ :::image type="content" source="media/tutorial-asp-net-core/enable-server-side-telemetry.png" alt-text="Screenshot of a code window in Visual Studio with the preceding code snippet highlighted." lightbox="media/tutorial-asp-net-core/enable-server-side-telemetry.png":::

+ > Learn more about the [configuration options in ASP.NET Core](/aspnet/core/fundamentals/configuration).

+The preceding steps are enough to help you start collecting server-side telemetry. The sample application has client-side components. Follow the next steps to start collecting [usage telemetry](./usage-overview.md).

+1. In Visual Studio Solution Explorer, open `\Views\_ViewImports.cshtml`.

+2. Add the following code at the end of the existing file.

+ :::image type="content" source="media/tutorial-asp-net-core/view-imports-injection.png" alt-text="Screenshot of the _ViewImports.cshtml file in Visual Studio with the preceding line of code highlighted." lightbox="media/tutorial-asp-net-core/view-imports-injection.png":::

+3. To properly enable client-side monitoring for your application, in Visual Studio Solution Explorer, open `\Views\Shared\_Layout.cshtml` and insert the following code immediately before the closing `<\head>` tag. This JavaScript snippet must be inserted in the `<head>` section of each page of your application that you want to monitor.

+ :::image type="content" source="media/tutorial-asp-net-core/layout-head-code.png" alt-text="Screenshot of the _Layout.cshtml file in Visual Studio with the preceding line of code highlighted within the head section of the file." lightbox="media/tutorial-asp-net-core/layout-head-code.png":::

+ > An alternative to using `FullScript` is `ScriptBody`. Use `ScriptBody` if you need to control the `<script>` tag to set a Content Security Policy:

+When investigating causes for performance degradation, it is important to include insights into database calls. You enable monitoring by configuring the [dependency module](./asp-net-dependencies.md). Dependency monitoring, including SQL, is enabled by default.

+Follow these steps to capture the full SQL query text.

+1. From the Visual Studio Solution Explorer, open the **Program.cs** file.

+3. To enable SQL command text instrumentation, insert the following code immediately after the `builder.Services.AddApplicationInsightsTelemetry()` code.

+ :::image type="content" source="media/tutorial-asp-net-core/enable-sql-command-text-instrumentation.png" alt-text="Screenshot of a code window in Visual Studio with the preceding code highlighted." lightbox="media/tutorial-asp-net-core/enable-sql-command-text-instrumentation.png":::

+After you deploy the web application code, telemetry will flow to Application Insights. The Application Insights SDK automatically collects incoming web requests to your application.

+1. From the Visual Studio Solution Explorer, right-click on the **AzureCafe** project and select **Publish** from the context menu.

+ :::image type="content" source="media/tutorial-asp-net-core/web-project-publish-context-menu.png" alt-text="Screenshot of the Visual Studio Solution Explorer with the Azure Cafe project selected and the Publish context menu item highlighted." lightbox="media/tutorial-asp-net-core/web-project-publish-context-menu.png":::

+ :::image type="content" source="media/tutorial-asp-net-core/publish-profile.png" alt-text="Screenshot of the AzureCafe publish profile with the Publish button highlighted." lightbox="media/tutorial-asp-net-core/publish-profile.png":::

+ When the Azure Cafe web application is successfully published, a new browser window opens to the Azure Cafe web application.

+ :::image type="content" source="media/tutorial-asp-net-core/azure-cafe-index.png" alt-text="Screenshot of the Azure Cafe web application." lightbox="media/tutorial-asp-net-core/azure-cafe-index.png":::

+3. To generate some telemetry, follow these steps in the web application to add a review.

+ 1. To view a cafe's menu and reviews, select **Details** next to a cafe.

+ :::image type="content" source="media/tutorial-asp-net-core/cafe-details-button.png" alt-text="Screenshot of a portion of the Azure Cafe list in the Azure Cafe web application with the Details button highlighted." lightbox="media/tutorial-asp-net-core/cafe-details-button.png":::

+ 2. To view and add reviews, on the Cafe screen, select the **Reviews** tab. Select the **Add review** button to add a review.

+ :::image type="content" source="media/tutorial-asp-net-core/cafe-add-review-button.png" alt-text="Screenshot of the Cafe details screen in the Azure Cafe web application with the Add review button highlighted." lightbox="media/tutorial-asp-net-core/cafe-add-review-button.png":::

+ 3. On the Create a review dialog, enter a name, rating, comments, and upload a photo for the review. When finished, select **Add review**.

+ :::image type="content" source="media/tutorial-asp-net-core/create-a-review-dialog.png" alt-text="Screenshot of the Create a review dialog in the Azure Cafe web application." lightbox="media/tutorial-asp-net-core/create-a-review-dialog.png":::

+ 4. If you need to generate additional telemetry, add additional reviews.

+You can use [Live Metrics](./live-stream.md) to quickly verify if Application Insights monitoring is configured correctly. Live Metrics shows CPU usage of the running process in near real time. It can also show other telemetry such as Requests, Dependencies, and Traces. Note that it might take a few minutes for the telemetry to appear in the portal and analytics.

+### Viewing the application map

+Application Insights introspects the incoming telemetry data and is able to generate a visual map of the system integrations it detects.

+2. Open the resource group for the sample application, which is `application-insights-azure-cafe`.

+4. From the left menu, beneath the **Investigate** heading, select **Application map**. Observe the generated Application map.

+ :::image type="content" source="media/tutorial-asp-net-core/application-map.png" alt-text="Screenshot of the Application Insights application map in the Azure portal." lightbox="media/tutorial-asp-net-core/application-map.png":::

+2. On the left menu, beneath the **Investigate** header, select **Performance**.

+3. The **Operations** tab contains details of the HTTP calls received by the application. To toggle between Server and Browser (client-side) views of the data, use the Server/Browser toggle.

+ <!-- The long description for server-performance.png: Screenshot of the Application Insights Performance screen in the Azure portal. The screenshot shows the Server/Browser toggle and HTTP calls received by the application highlighted. -->

+ :::image type="content" source="media/tutorial-asp-net-core/server-performance.png" alt-text="Screenshot of the Performance screen in the Azure portal." lightbox="media/tutorial-asp-net-core/server-performance.png":::

+

+ <!-- The long description for select-operation-performance.png: Screenshot of the Application Insights Performance screen in the Azure portal. The screenshot shows a POST operation and a sample operation from the suggested list selected and highlighted and the Drill into samples button is highlighted. -->

+ :::image type="content" source="media/tutorial-asp-net-core/select-operation-performance.png" alt-text="Screenshot of the Application Insights Performance screen in the Azure portal with operations and sample operations listed." lightbox="media/tutorial-asp-net-core/select-operation-performance.png":::

+ The end-to-end transaction displays for the selected request. In this case, a review was created, including an image, so it includes calls to Azure Storage and the Language Service (for sentiment analysis). It also includes database calls into SQL Azure to persist the review. In this example, the first selected Event displays information relative to the HTTP POST call.

+ :::image type="content" source="media/tutorial-asp-net-core/e2e-http-call.png" alt-text="Screenshot of the end-to-end transaction in the Azure portal with the HTTP Post call selected." lightbox="media/tutorial-asp-net-core/e2e-http-call.png":::

+5. Select a SQL item to review the SQL command text issued to the database.

+ :::image type="content" source="media/tutorial-asp-net-core/e2e-db-call.png" alt-text="Screenshot of the end-to-end transaction in the Azure portal with SQL command details." lightbox="media/tutorial-asp-net-core/e2e-db-call.png":::

+6. Optionally, select the Dependency (outgoing) requests to Azure Storage or the Language Service.

+7. Return to the **Performance** screen and select the **Dependencies** tab to investigate calls into external resources. Notice the Operations table includes calls into Sentiment Analysis, Blob Storage, and Azure SQL.

+ :::image type="content" source="media/tutorial-asp-net-core/performance-dependencies.png" alt-text="Screenshot of the Application Insights Performance screen in the Azure portal with the Dependencies tab selected and the Operations table highlighted." lightbox="media/tutorial-asp-net-core/performance-dependencies.png":::

+Application Insights is one type of [logging provider](/dotnet/core/extensions/logging-providers) available to ASP.NET Core applications that becomes available to applications when the [Application Insights for ASP.NET Core](#install-the-application-insights-nuget-package) NuGet package is installed and [server-side telemetry collection is enabled](#enable-application-insights-server-side-telemetry).

+As a reminder, the following code in **Program.cs** registers the `ApplicationInsightsLoggerProvider` with the built-in dependency injection container.

+With the `ApplicationInsightsLoggerProvider` registered as the logging provider, the app is ready to log into Application Insights by using either constructor injection with <xref:Microsoft.Extensions.Logging.ILogger> or the generic-type alternative <xref:Microsoft.Extensions.Logging.ILogger%601>.

+> By default, the logging provider is configured to automatically capture log events with a severity of <xref:Microsoft.Extensions.Logging.LogLevel.Warning?displayProperty=nameWithType> or greater.

+Consider the following example controller. It demonstrates the injection of ILogger, which is resolved with the `ApplicationInsightsLoggerProvider` that is registered with the dependency injection container. Observe in the **Get** method that an Informational, Warning, and Error message are recorded.

+ :::image type="content" source="media/tutorial-asp-net-core/values-api-url.png" alt-text="Screenshot of a browser window with /api/Values appended to the URL in the address bar." lightbox="media/tutorial-asp-net-core/values-api-url.png":::

+2. In the [Azure portal](https://portal.azure.com), wait a few moments and then select the **azure-cafe-insights-{SUFFIX}** Application Insights resource.

+ :::image type="content" source="media/tutorial-asp-net-core/application-insights-resource-group.png" alt-text="Screenshot of the application-insights-azure-cafe resource group in the Azure portal with the Application Insights resource highlighted." lightbox="media/tutorial-asp-net-core/application-insights-resource-group.png":::

+3. From the left menu of the Application Insights resource, under the **Monitoring** section, select **Logs**.

+

+4. In the **Tables** pane, under the **Application Insights** tree, double-click on the **traces** table.

+5. Modify the query to retrieve traces for the **Values** controller as follows, then select **Run** to filter the results.

+ The results display the logging messages present in the controller. A log severity of 2 indicates a warning level, and a log severity of 3 indicates an Error level.

+6. Alternatively, you can also write the query to retrieve results based on the category of the log. By default, the category is the fully qualified name of the class where the ILogger is injected. In this case, the category name is **ValuesController** (if there is a namespace associated with the class, the name will be prefixed with the namespace). Re-write and run the following query to retrieve results based on category.

+`ILogger` implementations have a built-in mechanism to apply [log filtering](/dotnet/core/extensions/logging#how-filtering-rules-are-applied). This filtering lets you control the logs that are sent to each registered provider, including the Application Insights provider. You can use the filtering either in configuration (using an *appsettings.json* file) or in code. For more information about log levels and guidance on how to use them appropriately, see the [Log Level](/aspnet/core/fundamentals/logging#log-level) documentation.

+The `ApplicationInsightsLoggerProvider` is aliased as **ApplicationInsights** in configuration. The following section of an *appsettings.json* file sets the default log level for all providers to <xref:Microsoft.Extensions.Logging.LogLevel.Warning?displayProperty=nameWithType>. The configuration for the ApplicationInsights provider, specifically for categories that start with "ValuesController," overrides this default value with <xref:Microsoft.Extensions.Logging.LogLevel.Error?displayProperty=nameWithType> and higher.

+Deploying the sample application with the preceding code in *appsettings.json* will yield only the error trace being sent to Application Insights when interacting with the **ValuesController**. This is because the **LogLevel** for the **ValuesController** category is set to **Error**. Therefore, the **Warning** trace is suppressed.

+To disable logging by using configuration, set all LogLevel values to "None".

+Similarly, within the code, set the default level for the `ApplicationInsightsLoggerProvider` and any subsequent log levels to **None**.

+* All autoscale failures are logged to the Activity Log. You can then configure an [activity log alert](../alerts/activity-log-alerts.md) so that you can be notified via email, SMS, or webhooks whenever there's an autoscale failure.

+* Similarly, all successful scale actions are posted to the Activity Log. You can then configure an activity log alert so that you can be notified via email, SMS, or webhooks whenever there's a successful autoscale action. You can also configure email or webhook notifications to get notified for successful scale actions via the notifications tab on the autoscale setting.

+If you use only one part of the combination, autoscale will only take action in a single direction (scale out, or in) until it reaches the maximum, or minimum instance counts, as defined in the profile. This isn't optimal, ideally you want your resource to scale up at times of high usage to ensure availability. Similarly, at times of low usage you want your resource to scale down, so you can realize cost savings.

+When you use a scale-in and scale-out rule, ideally use the same metric to control both. Otherwise, itΓÇÖs possible that the scale-in and scale-out conditions could be met at the same time resulting in some level of flapping. For example, the following rule combination isn't* recommended because there's no scale-in rule for memory usage:

+2. Messages keep coming and when you review the storage queue, the total count reads 50. You might assume that autoscale should start a scale-out action. However, note that it's still 50/2 = 25 messages per instance. So, scale-out doesn't occur. For the first scale-out to happen, the total message count in the storage queue should be 100.

+4. A third storage queue instance is added due to a scale-out action. The next scale-out action won't happen until the total message count in the queue reaches 150 because 150/3 = 50.

+The default instance count is important because autoscale scales your service to that count when metrics aren't available. Therefore, select a default instance count that's safe for your workloads.

+* Metrics aren't available for autoscale service to make a scale decision.

+* Autoscale detects flapping and aborts the scale attempt. You'll see a log type of `Flapping` in this situation. If you see this, consider whether your thresholds are too narrow.

+* Autoscale detects flapping but is still able to successfully scale. You'll see a log type of `FlappingOccurred` in this situation. If you see this, the autoscale engine has attempted to scale (for example, from 4 instances to 2), but has determined that this would cause flapping. Instead, the autoscale engine has scaled to a different number of instances (for example, using 3 instances instead of 2), which no longer causes flapping, so it has scaled to this number of instances.

+The [PCI Security Standards Council](https://www.pcisecuritystandards.org/) has set a deadline of [June 30th, 2018](https://www.pcisecuritystandards.org/pdfs/PCI_SSC_Migrating_from_SSL_and_Early_TLS_Resource_Guide.pdf) to disable older versions of TLS/SSL and upgrade to more secure protocols. Once Azure drops legacy support, if your agents can't communicate over at least TLS 1.2 you wouldn't be able to send data to Azure Monitor Logs.

+|AvailableStorage|No|(deprecated) Available Storage|Bytes|Total|"Available Storage" will be removed from Azure Monitor at the end of September 2023. Cosmos DB collection storage size is now unlimited. The only restriction is that the storage size for each logical partition key is 20GB. You can enable PartitionKeyStatistics in Diagnostic Log to know the storage consumption for top partition keys. For more info about Cosmos DB storage quota, please check this doc https://learn.microsoft.com/azure/cosmos-db/concepts-limits. After deprecation, the remaining alert rules still defined on the deprecated metric will be automatically disabled post the deprecation date.|CollectionName, DatabaseName, Region|

+// You can see your per-node costs in your Azure usage and charge data. For more information, see https://learn.microsoft.com/azure/cost-management-billing/understand/download-azure-daily-usage.

+For more information, see [this guide on how to create your first Power BI model and report](/training/modules/build-your-first-power-bi-report/).

+|[Dependency analysis in Azure Migrate Discovery and assessment - Azure Migrate](https://learn.microsoft.com/azure/migrate/concepts-dependency-visualization)|Revamped the guidance for migrating from Log Analytics Agent to Azure Monitor Agent.|

+| [VM insights guest health (preview)](vm/vminsights-health-overview.md) | Added deprecation statement |

+ [19/Nov/2020:18:41:10 +13:00] DEBUG: [PID:0020257:StorageANF:659] [1] Innerexception: Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://learn.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials

+* Unlock your cloud skills with more [Learn modules]](/training/azure/).

+learn.microsoft.com (Azure documentation)

+It provides a short introduction to the pipeline task you need for deploying a Bicep file. If you want more detailed steps on setting up the pipeline and project, see [Deploy Azure resources by using Bicep and Azure Pipelines](/training/paths/bicep-azure-pipelines/).

+If you would rather learn about Bicep best practices through step-by-step guidance, see [Structure your Bicep code for collaboration](/training/modules/structure-bicep-code-collaboration/).

+This article describes the Bicep functions for working with arrays. The lambda functions for working with arrays can be found [here](./bicep-functions-lambda.md).

+## flatten

+`flatten(arrayToFlatten)`

+Takes an array of arrays, and returns an array of sub-array elements, in the original order. Sub-arrays are only flattened once, not recursively.

+Namespace: [sys](bicep-functions.md#namespaces-for-functions).

+### Parameters

+| Parameter | Required | Type | Description |

+|: |: |: |: |

+| arrayToFlattern |Yes |array |The array of sub-arrays to flatten.|

+### Return value

+Array

+### Example

+The following example shows how to use the flatten function.

+```bicep

+param arrayToTest array = [

+ ['one', 'two']

+ ['three']

+ ['four', 'five']

+]

+output arrayOutput array = flatten(arrayToTest)

+```

+The output from the preceding example with the default values is:

+| Name | Type | Value |

+| - | - | -- |

+| arrayOutput | array | ['one', 'two', 'three', 'four', 'five'] |

+ Title: Bicep functions - lambda

+description: Describes the lambda functions to use in a Bicep file.

+# Lambda functions for Bicep

+This article describes the lambda functions to use in Bicep. Lambda expressions (or lambda functions) are essentially blocks of code that can be passed as an argument. In Bicep, lambda expression is in this format:

+```bicep

+<lambda variable> => <expression>

+```

+> [!NOTE]

+> The lambda functions are only supported in Bicep CLI version 0.10.61 or newer.

+## Limitations

+Bicep lambda function has these limitations:

+- Lambda expression can only be specified directly as function arguments in these functions: [`filter()`](#filter), [`map()`](#map), [`reduce()`](#reduce), and [`sort()`](#sort).

+- Using lambda variables (the temporary variables used in the lambda expressions) inside resource or module array access isn't currently supported.

+- Using lambda variables inside the [`listKeys`](./bicep-functions-resource.md#list) function isn't currently supported.

+- Using lambda variables inside the [reference](./bicep-functions-resource.md#reference) function isn't currently supported.

+## filter

+`filter(inputArray, lambda expression)`

+Filters an array with a custom filtering function.

+Namespace: [sys](bicep-functions.md#namespaces-for-functions).

+### Parameters

+| Parameter | Required | Type | Description |

+|: |: |: |: |

+| inputArray |Yes |array |The array to filter.|

+| lambda expression |Yes |expression |The lambda expression applied to each input array element. If false, the item will be filtered out of the output array.|

+### Return value

+An array.

+### Examples

+The following examples show how to use the filter function.

+```bicep

+var dogs = [

+ {

+ name: 'Evie'

+ age: 5

+ interests: ['Ball', 'Frisbee']

+ }

+ {

+ name: 'Casper'

+ age: 3

+ interests: ['Other dogs']

+ }

+ {

+ name: 'Indy'

+ age: 2

+ interests: ['Butter']

+ }

+ {

+ name: 'Kira'

+ age: 8

+ interests: ['Rubs']

+ }

+]

+output oldDogs array = filter(dogs, dog => dog.age >=5)

+```

+The output from the preceding example shows the dogs that are five or older:

+| Name | Type | Value |

+| - | - | -- |

+| oldDogs | Array | [{"name":"Evie","age":5,"interests":["Ball","Frisbee"]},{"name":"Kira","age":8,"interests":["Rubs"]}] |

+```bicep

+var itemForLoop = [for item in range(0, 10): item]

+output filteredLoop array = filter(itemForLoop, i => i > 5)

+output isEven array = filter(range(0, 10), i => 0 == i % 2)

+```

+The output from the preceding example:

+| Name | Type | Value |

+| - | - | -- |

+| filteredLoop | Array | [6, 7, 8, 9] |

+| isEven | Array | [0, 2, 4, 6, 8] |

+**filterdLoop** shows the numbers in an array that are greater than 5; and **isEven** shows the even numbers in the array.

+## map

+`map(inputArray, lambda expression)`

+Applies a custom mapping function to each element of an array.

+Namespace: [sys](bicep-functions.md#namespaces-for-functions).

+### Parameters

+| Parameter | Required | Type | Description |

+|: |: |: |: |

+| inputArray |Yes |array |The array to map.|

+| lambda expression |Yes |expression |The lambda expression applied to each input array element, in order to generate the output array.|

+### Return value

+An array.

+### Example

+The following example shows how to use the map function.

+```bicep

+var dogs = [

+ {

+ name: 'Evie'

+ age: 5

+ interests: ['Ball', 'Frisbee']

+ }

+ {

+ name: 'Casper'

+ age: 3

+ interests: ['Other dogs']

+ }

+ {

+ name: 'Indy'

+ age: 2

+ interests: ['Butter']

+ }

+ {

+ name: 'Kira'

+ age: 8

+ interests: ['Rubs']

+ }

+]

+output dogNames array = map(dogs, dog => dog.name)

+output sayHi array = map(dogs, dog => 'Hello ${dog.name}!')

+output mapObject array = map(range(0, length(dogs)), i => {

+ i: i

+ dog: dogs[i].name

+ greeting: 'Ahoy, ${dogs[i].name}!'

+})

+```

+The output from the preceding example is:

+| Name | Type | Value |

+| - | - | -- |

+| dogNames | Array | ["Evie","Casper","Indy","Kira"] |

+| sayHi | Array | ["Hello Evie!","Hello Casper!","Hello Indy!","Hello Kira!"] |

+| mapObject | Array | [{"i":0,"dog":"Evie","greeting":"Ahoy, Evie!"},{"i":1,"dog":"Casper","greeting":"Ahoy, Casper!"},{"i":2,"dog":"Indy","greeting":"Ahoy, Indy!"},{"i":3,"dog":"Kira","greeting":"Ahoy, Kira!"}] |

+**dogNames** shows the dog names from the array of objects; **sayHi** concatenates "Hello" and each of the dog names; and **mapObject** creates another array of objects.

+## reduce

+`reduce(inputArray, initialValue, lambda expression)`

+Reduces an array with a custom reduce function.

+Namespace: [sys](bicep-functions.md#namespaces-for-functions).

+### Parameters

+| Parameter | Required | Type | Description |

+|: |: |: |: |

+| inputArray |Yes |array |The array to reduce.|

+| initialValue |No |any |Initial value.|

+| lambda expression |Yes |expression |The lambda expression used to aggregate the current value and the next value.|

+### Return value

+Any.

+### Example

+The following examples show how to use the reduce function.

+```bicep

+var dogs = [

+ {

+ name: 'Evie'

+ age: 5

+ interests: ['Ball', 'Frisbee']

+ }

+ {

+ name: 'Casper'

+ age: 3

+ interests: ['Other dogs']

+ }

+ {

+ name: 'Indy'

+ age: 2

+ interests: ['Butter']

+ }

+ {

+ name: 'Kira'

+ age: 8

+ interests: ['Rubs']

+ }

+]

+var ages = map(dogs, dog => dog.age)

+output totalAge int = reduce(ages, 0, (cur, prev) => cur + prev)

+output totalAgeAdd1 int = reduce(ages, 1, (cur, prev) => cur + prev)

+```

+The output from the preceding example is:

+| Name | Type | Value |

+| - | - | -- |

+| totalAge | int | 18 |

+| totalAgeAdd1 | int | 19 |

+**totalAge** sums the ages of the dogs; **totalAgeAdd1** has an initial value of 1, and adds all the dog ages to the initial values.

+```bicep

+output reduceObjectUnion object = reduce([

+ { foo: 123 }

+ { bar: 456 }

+ { baz: 789 }

+], {}, (cur, next) => union(cur, next))

+```

+The output from the preceding example is:

+| Name | Type | Value |

+| - | - | -- |

+| reduceObjectUnion | object | {"foo":123,"bar":456,"baz":789} |

+The [union](./bicep-functions-object.md#union) function returns a single object with all elements from the parameters. The function call unionizes the key value pairs of the objects into a new object.

+## sort

+`sort(inputArray, lambda expression)`

+Sorts an array with a custom sort function.

+Namespace: [sys](bicep-functions.md#namespaces-for-functions).

+### Parameters

+| Parameter | Required | Type | Description |

+|: |: |: |: |

+| inputArray |Yes |array |The array to sort.|

+| lambda expression |Yes |expression |The lambda expression used to compare two array elements for ordering. If true, the second element will be ordered after the first in the output array.|

+### Return value

+An array.

+### Example

+The following example shows how to use the sort function.

+```bicep

+var dogs = [

+ {

+ name: 'Evie'

+ age: 5

+ interests: ['Ball', 'Frisbee']

+ }

+ {

+ name: 'Casper'

+ age: 3

+ interests: ['Other dogs']

+ }

+ {

+ name: 'Indy'

+ age: 2

+ interests: ['Butter']

+ }

+ {

+ name: 'Kira'

+ age: 8

+ interests: ['Rubs']

+ }

+]

+output dogsByAge array = sort(dogs, (a, b) => a.age < b.age)

+```

+The output from the preceding example sorts the dog objects from the youngest to the oldest:

+| Name | Type | Value |

+| - | - | -- |

+| dogsByAge | Array | [{"name":"Indy","age":2,"interests":["Butter"]},{"name":"Casper","age":3,"interests":["Other dogs"]},{"name":"Evie","age":5,"interests":["Ball","Frisbee"]},{"name":"Kira","age":8,"interests":["Rubs"]}] |

+## Next steps

+- See [Bicep functions - arrays](./bicep-functions-array.md) for additional array related Bicep functions.

+* [flatten](./bicep-functions-array.md#flatten)

+## Lambda functions

+The following functions are available for working with lambda expressions. All of these functions are in the `sys` namespace.

+* [filter](bicep-functions-lambda.md#filter)

+* [map](bicep-functions-lambda.md#map)

+* [reduce](bicep-functions-lambda.md#reduce)

+* [sort](bicep-functions-lambda.md#sort)

+If you would rather learn about about child resources through step-by-step guidance, see [Deploy child and extension resources by using Bicep](/training/modules/child-extension-bicep-templates).

+If you would rather learn about conditions through step-by-step guidance, see [Build flexible Bicep templates by using conditions and loops](/training/modules/build-flexible-bicep-templates-conditions-loops/).

+* Review the Learn module [Build flexible Bicep templates by using conditions and loops](/training/modules/build-flexible-bicep-templates-conditions-loops/).

+It provides a short introduction to GitHub actions and Bicep files. If you want more detailed steps on setting up the GitHub actions and project, see [Deploy Azure resources by using Bicep and GitHub Actions](/training/paths/bicep-github-actions).

+If you would rather learn about deployment scopes through step-by-step guidance, see [Deploy resources to subscriptions, management groups, and tenants by using Bicep](/training/modules/deploy-resources-scopes-bicep/).

+If you would rather learn about deployment scopes through step-by-step guidance, see [Deploy resources to subscriptions, management groups, and tenants by using Bicep](/training/modules/deploy-resources-scopes-bicep/).

+If you would rather learn about deployment scopes through step-by-step guidance, see [Deploy resources to subscriptions, management groups, and tenants by using Bicep](/training/modules/deploy-resources-scopes-bicep/).

+If you would rather learn about the what-if operation through step-by-step guidance, see [Preview Azure deployment changes by using what-if](/training/modules/arm-template-whatif/).

+* For a Learn module that demonstrates using what-if, see [Preview changes and validate Azure resources by using what-if and the ARM template test toolkit](/training/modules/arm-template-test/).

+If you would rather learn about the ARM template test toolkit through step-by-step guidance, see [Extend ARM templates by using deployment scripts](/training/modules/extend-resource-manager-template-deployment-scripts).

+> [Extend ARM templates by using deployment scripts](/training/modules/extend-resource-manager-template-deployment-scripts)

+- For a Learn module that covers passing a secure value from a key vault, see [Manage complex cloud deployments by using advanced ARM template features](/training/modules/manage-deployments-advanced-arm-template-features/).

+[<img src="media/learn-bicep/build-first-bicep-template.svg" width="101" height="120" alt="The badge for the Build your first Bicep template module." role="presentation"></img>](/training/modules/build-first-bicep-template/)

+[Build your first Bicep template](/training/modules/build-first-bicep-template/)

+ [<img src="media/learn-bicep/fundamentals-bicep.svg" width="101" height="120" alt="The trophy for the Fundamentals of Bicep learning path." role="presentation"></img>](/training/paths/fundamentals-bicep/)

+ [Part 1: Fundamentals of Bicep](/training/paths/fundamentals-bicep/)

+ [<img src="media/learn-bicep/intermediate-bicep.svg" width="101" height="120" alt="The trophy for the Intermediate Bicep learning path." role="presentation"></img>](/training/paths/intermediate-bicep/)

+ [Part 2: Intermediate Bicep](/training/paths/intermediate-bicep/)

+ [<img src="media/learn-bicep/advanced-bicep.svg" width="101" height="120" alt="The trophy for the Advanced Bicep learning path." role="presentation"></img>](/training/paths/advanced-bicep/)

+ [Part 3: Advanced Bicep](/training/paths/advanced-bicep/)

+ [<img src="media/learn-bicep/bicep-azure-pipelines.svg" width="101" height="120" alt="The trophy for the Deploy Azure resources using Bicep and Azure Pipelines learning path." role="presentation"></img>](/training/paths/bicep-azure-pipelines/)

+ [Option 1: Deploy Azure resources by using Bicep and Azure Pipelines](/training/paths/bicep-azure-pipelines/)

+ [<img src="media/learn-bicep/bicep-github-actions.svg" width="101" height="120" alt="The trophy for the Deploy Azure resources using Bicep and GitHub Actions learning path." role="presentation"></img>](/training/paths/bicep-github-actions/)

+ [Option 2: Deploy Azure resources by using Bicep and GitHub Actions](/training/paths/bicep-github-actions/)

+If you would rather learn about loops through step-by-step guidance, see [Build flexible Bicep templates by using conditions and loops](/training/modules/build-flexible-bicep-templates-conditions-loops/).

+In this article we summarize this recommended workflow. For detailed guidance, see [Migrate Azure resources and JSON ARM templates to use Bicep](/training/modules/migrate-azure-resources-bicep/).

+If you would rather learn about modules through step-by-step guidance, see [Create composable Bicep files by using modules](/training/modules/create-composable-bicep-files-using-modules/).

+- For a tutorial, see [Deploy Azure resources by using Bicep templates](/training/modules/deploy-azure-resources-by-using-bicep-templates/).

+If you would rather learn about parameters through step-by-step guidance, see [Build reusable Bicep templates by using parameters](/training/modules/build-reusable-bicep-templates-parameters).

+If you would rather learn about parameters through step-by-step guidance, see [Share Bicep modules by using private registries](/training/modules/share-bicep-modules-using-private-registries).

+> [Learn modules for Bicep](learn-bicep.md)

+If you would rather learn about extension resources through step-by-step guidance, see [Deploy child and extension resources by using Bicep](/training/modules/child-extension-bicep-templates).

+To learn more about template specs, and for hands-on guidance, see [Publish libraries of reusable infrastructure code by using template specs](/training/modules/arm-template-specs).

+To learn more about template specs, and for hands-on guidance, see [Publish libraries of reusable infrastructure code by using template specs](/training/modules/arm-template-specs).

+ Title: Create a custom resource provider

+description: Describes how to create a custom resource provider and deploy custom resources.

+# Quickstart: Create a custom resource provider and deploy custom resources

+In this quickstart, you create a custom resource provider and deploy custom resources for that resource provider. For more information about custom providers, see [Azure Custom Resource Providers Overview](overview.md).

+- After the **ARMClient** is installed, you can display usage information from a PowerShell command prompt by typing: `armclient.exe`. Or, go to the [ARMClient wiki](https://github.com/projectkudu/ARMClient/wiki).

+To set up the custom resource provider, deploy an [example template](https://github.com/Azure/azure-docs-json-samples/blob/master/custom-providers/customprovider.json) to your Azure subscription.

+The template deploys the following resources to your subscription:

+- Function app with the operations for the resources and actions.

+- Storage account for storing users that are created through the custom provider.

+- Custom resource provider that defines the custom resource types and actions. It uses the function app endpoint for sending requests.

+- Custom resource from the custom resource provider.

+To deploy the custom resource provider, use Azure CLI, PowerShell, or the Azure portal.

+To deploy the template from the Azure portal, select the **Deploy to Azure** button.

+## View custom resource provider and resource

+In the portal, the custom resource provider is a hidden resource type. To confirm that the resource provider was deployed, go to the resource group and select **Show hidden types**.

+To see the custom resource that you deployed, use the `GET` operation on your resource type. The resource type `Microsoft.CustomProviders/resourceProviders/users` shown in the JSON response includes the resource that was created by the template.

+ "id": "/subscriptions/<sub-id>/resourceGroups/<rg-name>/providers/Microsoft.CustomProviders/resourceProviders/<provider-name>/users/ana",

+ "name": "ana",

+ "FullName": "Ana Bowman",

+ "Location": "Moon",

+ "FullName": "Ana Bowman",

+ "Location": "Moon"

+ "id": "/subscriptions/<sub-id>/resourceGroups/<rg-name>/providers/Microsoft.CustomProviders/resourceProviders/<provider-name>/users/ana",

+ "name": "ana",

+Your custom resource provider also has an action named `ping`. The code that processes the request is implemented in the function app. The `ping` action replies with a greeting.

+To send a `ping` request, use the `POST` operation on your action.

+## Use PUT to create resource

+In this quickstart, the template used the resource type `Microsoft.CustomProviders/resourceProviders/users` to deploy a resource. You can also use a `PUT` operation to create a resource. For example, if a resource isn't deployed with the template, the `PUT` operation will create a resource.

+In this example, because the template already deployed a resource, the `PUT` operation creates a new resource.

+You can rerun the `GET` operation from the [view custom resource provider and resource](#view-custom-resource-provider-and-resource) section to show the two resources that were created. This example shows output from the Azure CLI command.

+```json

+{

+ "value": [

+ {

+ "id": "/subscriptions/<sub-id>/resourceGroups/<rg-name>/providers/Microsoft.CustomProviders/resourceProviders/<provider-name>/users/ana",

+ "name": "ana",

+ "properties": {

+ "FullName": "Ana Bowman",

+ "Location": "Moon",

+ "provisioningState": "Succeeded"

+ },

+ "type": "Microsoft.CustomProviders/resourceProviders/users"

+ },

+ {

+ "id": "/subscriptions/<sub-id>/resourceGroups/<rg-name>/providers/Microsoft.CustomProviders/resourceProviders/<provider-name>/users/testuser",

+ "name": "testuser",

+ "properties": {

+ "FullName": "Test User",

+ "Location": "Earth",

+ "provisioningState": "Succeeded"

+ },

+ "type": "Microsoft.CustomProviders/resourceProviders/users"

+ }

+ ]

+}

+```

+## Clean up resources

+If you're finished with the resources created in this article, you can delete the resource group. When you delete a resource group, all the resources in that resource group are deleted.

+# [Azure CLI](#tab/azure-cli)

+```azurecli-interactive

+az group delete --resource-group $rgName

+```

+# [PowerShell](#tab/azure-powershell)

+```azurepowershell-interactive

+Remove-AzResourceGroup -Name $rgName

+```

+description: This tutorial describes how to create a function app in Azure Functions that works with Azure Custom Providers.

+1. Go to the **Integrate** tab for the `HttpTrigger`.

+1. Install the `Microsoft.Azure.WebJobs.Extensions.Storage` extension if it isn't already installed.

+1. Go to the **Integrate** tab for the `HttpTrigger`.

+> If your C# project file is missing from the project directory, you can add it manually, or it will appear after the `Microsoft.Azure.WebJobs.Extensions.Storage` extension is installed on the function app.

+* For a Learn module that covers conditional deployment, see [Manage complex cloud deployments by using advanced ARM template features](/training/modules/manage-deployments-advanced-arm-template-features/).

+ "type": "Microsoft.DataFactory/factories",

+ "name": "exampleDataFactory",

+ ...

+ "resources": [

+ {

+ "type": "datasets",

+ "name": "exampleDataSet",

+ "dependsOn": [

+ "exampleDataFactory"

+ ],

+ ...

+ }

+ ]

+}

+- For a Learn module that covers resource copy, see [Manage complex cloud deployments by using advanced ARM template features](/training/modules/manage-deployments-advanced-arm-template-features/).

+> [Learn module: Automate the deployment of ARM templates by using GitHub Actions](/training/modules/deploy-templates-command-line-github-actions/)

+To learn more about what-if, and for hands-on guidance, see [Preview Azure deployment changes by using what-if](/training/modules/arm-template-whatif).

+- For a Learn module that covers using what if, see [Preview changes and validate Azure resources by using what-if and the ARM template test toolkit](/training/modules/arm-template-test/).

+To learn more about the ARM template test toolkit, and for hands-on guidance, see [Extend ARM templates by using deployment scripts](/training/modules/extend-resource-manager-template-deployment-scripts).

+> [Learn module: Extend ARM templates by using deployment scripts](/training/modules/extend-resource-manager-template-deployment-scripts/)

+- For a Learn module that covers passing a secure value from a key vault, see [Manage complex cloud deployments by using advanced ARM template features](/training/modules/manage-deployments-advanced-arm-template-features/).

+> [!VIDEO https://learn.microsoft.com/Shows/Azure-Enablement/How-and-why-to-learn-about-ARM-templates/player]

+* To learn about ARM templates through a guided set of Learn modules, see [Deploy and manage resources in Azure by using ARM templates](/training/paths/deploy-manage-resource-manager-templates/).

+ ...

+}

+* For a Learn module that covers resource dependencies, see [Manage complex cloud deployments by using advanced ARM template features](/training/modules/manage-deployments-advanced-arm-template-features/).

+This article is intended for users who have some familiarity with ARM templates. It provides detailed information about the structure of the template. For a step-by-step tutorial that guides you through the process of creating a template, see [Tutorial: Create and deploy your first ARM template](template-tutorial-create-first-template.md). To learn about ARM templates through a guided set of Learn modules, see [Deploy and manage resources in Azure by using ARM templates](/training/paths/deploy-manage-resource-manager-templates/).

+To learn more about template specs, and for hands-on guidance, see [Publish libraries of reusable infrastructure code by using template specs](/training/modules/arm-template-specs).

+- For a Learn module that covers using the test toolkit, see [Preview changes and validate Azure resources by using what-if and the ARM template test toolkit](/training/modules/arm-template-test/).

+If you want to learn about the benefits of using templates and why you should automate deployments with templates, see [ARM template overview](overview.md). To learn about ARM templates through a guided set of [Learn modules](/training), see [Deploy and manage resources in Azure by using JSON ARM templates](/training/paths/deploy-manage-resource-manager-templates).

+For a Learn module that covers resource copy, see [Manage complex cloud deployments by using advanced ARM template features](/training/modules/manage-deployments-advanced-arm-template-features/).

+For a Learn module that covers resource dependencies, see [Manage complex cloud deployments by using advanced ARM template features](/training/modules/manage-deployments-advanced-arm-template-features/).

+For a Learn module that covers deployment scripts, see [Extend ARM templates by using deployment scripts](/training/modules/extend-resource-manager-template-deployment-scripts/).

+For a Learn module that covers conditions, see [Manage complex cloud deployments by using advanced ARM template features](/training/modules/manage-deployments-advanced-arm-template-features/).

+For a Learn module that uses a secure value from a key vault, see [Manage complex cloud deployments by using advanced ARM template features](/training/modules/manage-deployments-advanced-arm-template-features/).

+To learn more about the ARM template test toolkit, and for hands-on guidance, see [Validate Azure resources by using the ARM Template Test Toolkit](/training/modules/arm-template-test).

+- For a Learn module that covers using the test toolkit, see [Validate Azure resources by using the ARM Template Test Toolkit](/training/modules/arm-template-test/).

+> Learn to use SignalR and Azure Functions together in the interactive tutorial [Enable automatic updates in a web application using Azure Functions and SignalR Service](/training/modules/automatic-update-of-a-webapp-using-azure-functions-and-signalr).

+* [Enable automatic updates in a web application using Azure Functions and SignalR Service](/training/modules/automatic-update-of-a-webapp-using-azure-functions-and-signalr)

+* [Azure SignalR Service Serverless Quickstart - JavaScript](signalr-quickstart-azure-functions-javascript.md)

+ // See: https://learn.microsoft.com/dotnet/framework/performance/thread-pool-etw-events#threadpoolworkerthreadadjustmentadjustment

+> [!VIDEO https://learn.microsoft.com/shows/Data-Exposed/What-is-Azure-SQL-Edge/player]

+> [!VIDEO https://learn.microsoft.com/shows/Data-Exposed/Azure-SQL-Edge-Demo-Renewable-Energy/player]

+5. Follow the instructions in README.md to set up the demo environment and execute the demo.

+>Get notified about when to revisit this page for updates by copying and pasting this URL: `https://learn.microsoft.com/api/search/rss?search=%22Azure+Video+Analyzer+on+IoT+Edge+release+notes%22&locale=en-us` into your RSS feed reader.

+<!-- OPTION 1 - Minimum - Link to relevant bookmarks in https://learn.microsoft.com/azure/azure-monitor/platform/metrics-supported, which is auto generated from underlying systems. Not all metrics are published depending on whether your product group wants them to be. If the metric is published, but descriptions are wrong of missing, contact your PM and tell them to update them in the Azure Monitor "shoebox" manifest. If this article is missing metrics that you and the PM know are available, both of you contact azmondocs@microsoft.com.

+<!-- See https://learn.microsoft.com/azure/storage/common/monitor-storage-reference#metrics-dimensions for an example. Part is copied below. -->

+<!-- OPTION 1 - Minimum - Link to relevant bookmarks in https://learn.microsoft.com/azure/azure-monitor/platform/resource-logs-categories, which is auto generated from the REST API. Not all resource log types metrics are published depending on whether your product group wants them to be. If the resource log is published, but category display names are wrong or missing, contact your PM and tell them to update them in the Azure Monitor "shoebox" manifest. If this article is missing resource logs that you and the PM know are available, both of you contact azmondocs@microsoft.com.

+<!-- OPTION 1 - Minimum - Link to relevant bookmarks in https://learn.microsoft.com/azure/azure-monitor/reference/tables/tables-resourcetype where your service tables are listed. These files are auto generated from the REST API. If this article is missing tables that you and the PM know are available, both of you contact azmondocs@microsoft.com.

+- See [Monitoring Azure resources with Azure Monitor](../azure-monitor/essentials/monitor-azure-resource.md) for details on monitoring Azure resources.

+<!-- OPTIONAL: Add specific examples of configuration for this service. For example, CLI and PowerShell commands for creating diagnostic setting. Ideally, customers should set up a policy to automatically turn on collection for services. Azure monitor has Resource Manager template examples you can point to. See https://learn.microsoft.com/azure/azure-monitor/samples/resource-manager-diagnostic-settings. Contact azmondocs@microsoft.com if you have questions. -->

+>Get notified about when to revisit this page for updates by copying and pasting this URL: `https://learn.microsoft.com/api/search/rss?search=%22Azure+Media+Services+Video+Indexer+release+notes%22&locale=en-us` into your RSS feed reader.

+- [Create logic apps connector for ARM-based accounts](logic-apps-connector-arm-accounts.md).

+### Expanded the supported languages in LID and MLID through the API

+We expand the list of the languages to be supported in LID (language identification) and MLID (multi language Identification) using APIs.

+For more information, see [supported languages](language-support.md).

+> [!VIDEO https://learn.microsoft.com/shows/IT-Ops-Talk/Configure-backups-at-scale-using-Azure-Policy/player]

+ > [!VIDEO https://learn.microsoft.com/shows/IT-Ops-Talk/Automatically-retry-failed-backup-jobs-using-Azure-Resource-Graph-and-Azure-Automation-Runbooks/player]

+MARSAgentInstaller.exe /q # Please note the commandline install options available here: https://learn.microsoft.com/azure/backup/backup-client-automation#installation-options

+# See: https://learn.microsoft.com/rest/api/backup/securitypins/get

+# See: https://learn.microsoft.com/powershell/module/azurerm.keyvault/Add-AzureKeyVaultKey?view=azurermps-6.13.0

+[Learn more](../backup-client-automation.md) about how to use PowerShell to deploy and manage on-premises backups using MARS agent.

+## Azure TLS certificate changes

+Azure TLS/SSL endpoints now contain updated certificates chaining up to new root CAs. Ensure that the following changes include the updated root CAs. [Learn more](../security/fundamentals/tls-certificate-changes.md#what-changed) about the possible impacts on your applications.

+Earlier, most of the TLS certificates, used by Azure services, chained up to the following Root CA:

+Common name of CA | Thumbprint (SHA1)

+ |

+[Baltimore CyberTrust Root](https://cacerts.digicert.com/BaltimoreCyberTrustRoot.crt) | d4de20d05e66fc53fe1a50882c78db2852cae474

+Now, TLS certificates, used by Azure services, helps to chain up to one of the following Root CAs:

+Common name of CA | Thumbprint (SHA1)

+ |

+[DigiCert Global Root G2](https://cacerts.digicert.com/DigiCertGlobalRootG2.crt) | df3c24f9bfd666761b268073fe06d1cc8d4f82a4

+[DgiCert Global Root CA](https://cacerts.digicert.com/DigiCertGlobalRootG2.crt) | a8985d3a65e5e5c4b2d7d66d40c6dd2fb19c5436

+[Baltimore CyberTrust Root](https://cacerts.digicert.com/BaltimoreCyberTrustRoot.crt)| d4de20d05e66fc53fe1a50882c78db2852cae474

+[D-TRUST Root Class 3 CA 2 2009](https://www.d-trust.net/cgi-bin/D-TRUST_Root_Class_3_CA_2_2009.crt) | 58e8abb0361533fb80f79b1b6d29d3ff8d5f00f0

+[Microsoft RSA Root Certificate Authority 2017](https://www.microsoft.com/pkiops/certs/Microsoft%20RSA%20Root%20Certificate%20Authority%202017.crt) | 73a5e64a3bff8316ff0edccc618a906e4eae4d74

+[Microsoft ECC Root Certificate Authority 2017](https://www.microsoft.com/pkiops/certs/Microsoft%20ECC%20Root%20Certificate%20Authority%202017.crt) | 999a64c37ff47d9fab95f14769891460eec4c3c5

+- The backup components connections failures with error 10054 (An existing connection was forcibly closed by the remote host).

+* [Learn module: Introduction to Azure Bastion](/training/modules/intro-to-azure-bastion/).

+// https://learn.microsoft.com/azure/active-directory/develop/quickstart-register-app

+* Support Batch private pool using a new private endpoint (sub-resource: **nodeManagement**) for Azure Batch account.

+Batch pool without public IP addresses (classic) will retire on **31 March 2023** and will be updated to simplified compute node communication pools without public IPs. For existing pools that use the previous preview version of Batch pool without public IP addresses (classic), it's only possible to migrate pools created in a virtual network. To migrate the pool, follow the opt-in process for simplified compute node communication:

+ Similar to Batch pools without public IP addresses (classic). As there's no public IP address for the Batch pool, users will need to connect their pool nodes from within the virtual network. You can create a jump box VM in the virtual network or use other remote connectivity solutions like [Azure Bastion](../bastion/bastion-overview.md).

+ After **31 March 2023**, we'll stop supporting Batch pool without public IP addresses. The functionality of the existing pool in that configuration may break, such as scale-out operations, or may be actively scaled down to zero at any point in time after that date.

+For more information, see [Simplified compute node communication](./simplified-compute-node-communication.md).

+To make the statistical data available for customers, the Batch service allocates batch pools and schedule jobs with an in-house MapReduce implementation to perform background periodic roll-up of statistics. The aggregation is performed for all accounts/pools/jobs in each region, no matter if customer needs or queries the stats for their account/pool/job. The operating cost includes 11 VMs allocated in each region to execute MapReduce aggregation jobs. For busy regions, we had to increase the pool size further to accommodate the extra aggregation load.

+The purpose of the API is designed and maintained to serve the customer in troubleshooting. However, not many customers use it in real life, and the customers are interested in extracting the details for not more than a month. Now more advanced ways of log/job/pool data can be collected and used on a need basis using Azure portal logs, Alerts, Log export, and other methods. Therefore, we're retiring the Job/Pool Lifetime.

+ Azure portal has various options to enable the logs, namely system logs, diagnostic logs. See [Monitor Batch Solutions](./monitoring-overview.md) for more information.

+ Azure portal log feature allows every customer to extract the output and error logs to their workspace. See [Monitor with Application Insights](./monitor-application-insights.md) for more information.

+For more information, see [Azure Monitor Logs](../azure-monitor/logs/data-platform-logs.md).

+Low priority VMs are a deprecated feature, and it will never become Generally Available (GA). Spot VMs are the official preemptible offering from the Compute platform, and are generally available. Therefore, we'll retire Low Priority VMs on **30 September 2025**. After that, we'll stop supporting Low priority VMs. The existing Low priority pools may no longer work or be provisioned.

+* Stop/Deallocate (default) ΓÇô when evicted, the VM is deallocated, but you keep (and pay for) underlying disks. This is the ideal for cases where the state is stored on disks.

+Customers in User Subscription mode can include Spot VMs using the following the steps below:

+ 

+ See the quick start [link](./batch-account-create-portal.md) on creating a new Batch account/pool/task.

+ See [Spot VMs](../virtual-machines/spot-vms.md) for more information on using Spot VMs. Yes, you can see historical pricing and eviction rates per size in a region in the portal.

+- [Learn module: Introduction to Azure Content Delivery Network (CDN)](/training/modules/intro-to-azure-content-delivery-network).

+## September 2022 Guest OS

+>[!NOTE]

+>The September Guest OS is currently being rolled out to Cloud Service VMs that are configured for automatic updates. When the rollout is complete, this version will be made available for manual updates through the Azure portal and configuration files. The following patches are included in the September Guest OS. This list is subject to change.

+| Product Category | Parent KB Article | Vulnerability Description | Guest OS | Date First Introduced |

+| | | | | |

+| Rel 22-09 | [5017315] | Latest Cumulative Update(LCU) | 6.48 | Sep 13, 2022 |

+| Rel 22-09 | [5016618] | IE Cumulative Updates | 2.128, 3.115, 4.108 | Aug 9, 2022 |

+| Rel 22-09 | [5017316] | Latest Cumulative Update(LCU) | 7.16 | Sep 13, 2022 |

+| Rel 22-09 | [5017305] | Latest Cumulative Update(LCU) | 5.72 | Sep 13, 2022 |

+| Rel 22-09 | [5013641] | .NET Framework 3.5 and 4.7.2 Cumulative Update | 6.48 | May 10, 2022 |

+| Rel 22-09 | [5017397] | Servicing Stack Update | 2.128 | Sep 13, 2022 |

+| Rel 22-09 | [5017361] | September '22 Rollup | 2.128 | Sep 13, 2022 |

+| Rel 22-09 | [5013637] | .NET Framework 3.5 Security and Quality Rollup LKG | 2.128 | Sep 13, 2022 |

+| Rel 22-09 | [5013644] | .NET Framework 4.6.2 Security and Quality Rollup LKG | 2.128 | May 10, 2022 |

+| Rel 22-09 | [5016263] | Servicing Stack Update | 3.115 | July 12, 2022 |

+| Rel 22-09 | [5017370] | September '22 Rollup | 3.115 | Sep 13, 2022 |

+| Rel 22-09 | [5013635] | .NET Framework 3.5 Security and Quality Rollup LKG | 3.115 | Sep 13, 2022 |

+| Rel 22-09 | [5013642] | .NET Framework 4.6.2 Security and Quality Rollup LKG | 3.115 | May 10, 2022 |

+| Rel 22-09 | [5017398] | Servicing Stack Update | 4.108 | Sep 13, 2022 |

+| Rel 22-09 | [5017367] | Monthly Rollup | 4.108 | Sep 13, 2022 |

+| Rel 22-09 | [5013638] | .NET Framework 3.5 Security and Quality Rollup LKG | 4.108 | Jun 14, 2022 |

+| Rel 22-09 | [5013643] | .NET Framework 4.6.2 Security and Quality Rollup LKG | 4.108 | May 10, 2022 |

+| Rel 22-09 | [4578013] | OOB Standalone Security Update | 4.108 | Aug 19, 2020 |

+| Rel 22-09 | [5017396] | Servicing Stack Update | 5.72 | Sep 13, 2022 |

+| Rel 22-09 | [4494175] | Microcode | 5.72 | Sep 1, 2020 |

+| Rel 22-09 | 5015896 | Servicing Stack Update | 6.48 | Sep 1, 2020 |

+| Rel 22-09 | [5013626] | .NET Framework 4.8 Security and Quality Rollup LKG | 6.48 | May 10, 2022 |

+[5017315]: https://support.microsoft.com/kb/5017315

+[5016618]: https://support.microsoft.com/kb/5016618

+[5017316]: https://support.microsoft.com/kb/5017316

+[5017305]: https://support.microsoft.com/kb/5017305

+[5013641]: https://support.microsoft.com/kb/5013641

+[5017397]: https://support.microsoft.com/kb/5017397

+[5017361]: https://support.microsoft.com/kb/5017361

+[5013637]: https://support.microsoft.com/kb/5013637

+[5013644]: https://support.microsoft.com/kb/5013644

+[5016263]: https://support.microsoft.com/kb/5016263

+[5017370]: https://support.microsoft.com/kb/5017370

+[5013635]: https://support.microsoft.com/kb/5013635

+[5013642]: https://support.microsoft.com/kb/5013642

+[5017398]: https://support.microsoft.com/kb/5017398

+[5017367]: https://support.microsoft.com/kb/5017367

+[5013638]: https://support.microsoft.com/kb/5013638

+[5013643]: https://support.microsoft.com/kb/5013643

+[4578013]: https://support.microsoft.com/kb/4578013

+[5017396]: https://support.microsoft.com/kb/5017396

+[4494175]: https://support.microsoft.com/kb/4494175

+[5015896]: https://support.microsoft.com/kb/5015896

+[5013626]: https://support.microsoft.com/kb/5013626

+- **Code snippets**: In Microsoft [technical documentation](/) and [training resources](/training), select the **Try It** button that appears with Azure CLI and Azure PowerShell code snippets:

+- **Details**: Administrators may wish to disable access to Cloud Shell for their users. Cloud Shell utilizes access to the `ux.console.azure.com` domain, which can be denied, stopping any access to Cloud Shell's entrypoints including `portal.azure.com`, `shell.azure.com`, Visual Studio Code Azure Account extension, and `learn.microsoft.com`. In the US Government cloud, the entrypoint is `ux.console.azure.us`; there is no corresponding `shell.azure.us`.

+What is Microsoft Cognitive Services? | Microsoft Docs || https://learn.microsoft.com/azure/cognitive-services/Welcome

+* [Detect and analyze faces with the Face service](/training/modules/detect-analyze-faces/)

+* [Analyze images with the Computer Vision service](/training/modules/analyze-images-computer-vision/)

+* [Read Text in Images and Documents with the Computer Vision Service](/training/modules/read-text-images-documents-with-computer-vision-service/)

+* [Introduction to Content Moderator](/training/modules/intro-to-content-moderator/)

+* [Classify and moderate text with Azure Content Moderator](/training/modules/classify-and-moderate-text-with-azure-content-moderator/)

+* [Classify images with the Custom Vision service](/training/modules/classify-images-custom-vision/)

+* [Classify endangered bird species with Custom Vision](/training/modules/cv-classify-bird-species/)

+|*URL for public image|``|`How can I create a bot with `||

+> [!VIDEO https://learn.microsoft.com/Shows/AI-Show/Introducing-QnA-managed-Now-in-Public-Preview/player]

+ | SubscriptionKey | The key used for both applications |

+ | region | Your Speech resource region. For example: `westus2` |

+ | subscriptionkey | Your Speech resource key. |

+ | SubscriptionKey | The key used for both applications |

+ // This code creates the `DialogServiceConnector` with your resource information.

+ // create a DialogServiceConfig by providing a Custom Commands application id and Speech resource key

+ // The RecoLanguage property is optional (default en-US); note that only en-US is supported in Preview

+ const string speechSubscriptionKey = "YourSpeechSubscriptionKey"; // Your Speech resource key

+ const string region = "YourServiceRegion"; // The Speech resource region.

+1. Replace the strings `YourApplicationId`, `YourSpeechSubscriptionKey`, and `YourServiceRegion` with your own values for your app, speech key, and [region](regions.md)

+A Speech resource is required before you can use Custom Neural Voice. Follow these instructions to create a Speech resource in Azure. If you don't have an Azure account, you can sign up for a new one.

+Once you've created an Azure account and a Speech resource, you'll need to sign in to Speech Studio and connect your subscription.

+1. Get your Speech resource key from the Azure portal.

+>- You can create up to 50 endpoints with a standard (S0) Speech resource, each with its own custom neural voice.

+>- To use your custom neural voice, you must specify the voice model name, use the custom URI directly in an HTTP request, and use the same Speech resource to pass through the authentication of the text-to-speech service.

+* The **Endpoint key** shows the Speech resource key the endpoint is associated with. Use the endpoint key as the value of your `Ocp-Apim-Subscription-Key` request header.

+For information about endpoint ID, region, and Speech resource key parameters, see [request parameters](#request-parameters).

+Ocp-Apim-Subscription-Key: YourResourceKey

+Host: <YourResourceRegion>.customvoice.api.speech.microsoft.com

+curl -v -X GET "https://<YourResourceRegion>.customvoice.api.speech.microsoft.com/api/texttospeech/v3.0/endpoints/<YourEndpointId>" -H "Ocp-Apim-Subscription-Key: <YourResourceKey >"

+For information about endpoint ID, region, and Speech resource key parameters, see [request parameters](#request-parameters).

+Ocp-Apim-Subscription-Key: YourResourceKey

+Host: <YourResourceRegion>.customvoice.api.speech.microsoft.com

+curl -v -X POST "https://<YourResourceRegion>.customvoice.api.speech.microsoft.com/api/texttospeech/v3.0/endpoints/<YourEndpointId>/suspend" -H "Ocp-Apim-Subscription-Key: <YourResourceKey >" -H "content-type: application/json" -H "content-length: 0"

+For information about endpoint ID, region, and Speech resource key parameters, see [request parameters](#request-parameters).

+Ocp-Apim-Subscription-Key: YourResourceKey

+Host: <YourResourceRegion>.customvoice.api.speech.microsoft.com

+curl -v -X POST "https://<YourResourceRegion>.customvoice.api.speech.microsoft.com/api/texttospeech/v3.0/endpoints/<YourEndpointId>/resume" -H "Ocp-Apim-Subscription-Key: <YourResourceKey >" -H "content-type: application/json" -H "content-length: 0"

+You use these request parameters with calls to the REST API. See [application settings](#application-settings) for information about where to get your region, endpoint ID, and Speech resource key in Speech Studio.

+| `YourResourceRegion` | Path | `True` | string | The Azure region the endpoint is associated with. |

+| `Ocp-Apim-Subscription-Key` | Header | `True` | string | The Speech resource key the endpoint is associated with. |

+| 401 | Unauthorized | The request isn't authorized. Check to make sure your Speech resource key or [token](rest-speech-to-text-short.md#authentication) is valid and in the correct region. |

+| 429 | Too Many Requests | You've exceeded the quota or rate of requests allowed for your Speech resource. |

+var speechConfig = SpeechConfig.FromSubscription(YourResourceKey, YourResourceRegion);

+auto speechConfig = SpeechConfig::FromSubscription(YourResourceKey, YourResourceRegion);

+SpeechConfig speechConfig = SpeechConfig.fromSubscription(YourResourceKey, YourResourceRegion);

+1. Replace the placeholders in this method with your LUIS resource key, region, and app ID as follows.

+ | `YourLanguageUnderstandingSubscriptionKey` | Your LUIS resource key. Again, you must get this item from your Azure dashboard. You can find it on your app's **Azure Resources** page (under **Manage**) in the [LUIS portal](https://www.luis.ai/home). |

+ | `YourLanguageUnderstandingServiceRegion` | The short identifier for the region your LUIS resource is in, such as `westus` for West US. See [Regions](regions.md). |

+First, you need to create a speech configuration from your LUIS prediction key and region. You can use speech configurations to create recognizers for the various capabilities of the Speech SDK. The speech configuration has multiple ways to specify the resource you want to use; here, we use `FromSubscription`, which takes the resource key and region.

+> Use the key and region of your LUIS resource, not a Speech resource.

+Next, create an intent recognizer using `new IntentRecognizer(config)`. Since the configuration already knows which resource to use, you don't need to specify the key again when creating the recognizer.

+- **Speech resource:** A resource for Cognitive Speech Services for speech-to-text and text-to-speech conversions. Create a Speech resource on the [Azure portal](https://portal.azure.com). For more information, see [Create a new Azure Cognitive Services resource](~/articles/cognitive-services/cognitive-services-apis-create-account.md?tabs=speech#create-a-new-azure-cognitive-services-resource).

+With your Speech resource key and echo bot's bot ID, you're ready to try out the [UWP Voice Assistant sample](windows-voice-assistants-faq.yml#the-uwp-voice-assistant-sample). Follow the instructions in the readme to run the app and enter your credentials.

+* Replace `<your_key>` with your Speech resource key. This information is available in the **Overview** tab for your resource in the [Azure portal](https://aka.ms/azureportal).

+* Replace `<your_key>` with your Speech resource key. This information is available in the **Overview** tab for your resource in the [Azure portal](https://aka.ms/azureportal).

+| Create | 400 | The voice synthesis is not enabled in this region. | Change the speech resource key with a supported region. |

+| | 400 | Only the **Standard** speech resource for this region is valid. | Change the speech resource key to the "Standard" pricing tier. |

+At this time, Custom Commands supports speech resources created in regions that have [voice assistant capabilities](./regions.md#voice-assistants).

+ The default view is your list of Speech resources.

+ > If you don't see the select resource page, you can navigate there by choosing "Resource" from the settings menu on the top bar.

+1. Select your Speech resource, and then select **Go to Studio**.

+ The default view is a list of the Custom Commands applications you have under your selected resource.

+The Speech service is [available in various regions](./regions.md). Speech resource keys are tied to a single region. When you acquire a key, you select a specific region, where your data, model and deployments reside.

+If you use the default endpoints, you should configure your client code to monitor for errors. If errors persist, be prepared to redirect to another region where you have a Speech resource.

+ - [Regional key and the region code](./rest-speech-to-text.md)

+Replace `<REGION_IDENTIFIER>` with the identifier that matches the [region](regions.md) of your Speech resource.

+| `Ocp-Apim-Subscription-Key` | Your resource key for the Speech service. | Either this header or `Authorization` is required. |

+Ocp-Apim-Subscription-Key: YOUR_RESOURCE_KEY

+| 401 | Unauthorized | A resource key or an authorization token is invalid in the specified region, or an endpoint is invalid. |

+| 403 | Forbidden | A resource key or authorization token is missing. |

+request.Headers["Ocp-Apim-Subscription-Key"] = "YOUR_RESOURCE_KEY";

+The text-to-speech REST API supports neural text-to-speech voices, which support specific languages and dialects that are identified by locale. Each available endpoint is associated with a region. A Speech resource key for the endpoint or region that you plan to use is required. Here are links to more information:

+| `Ocp-Apim-Subscription-Key` | Your Speech resource key. | Either this header or `Authorization` is required. |

+Ocp-Apim-Subscription-Key: YOUR_RESOURCE_KEY

+| 401 | Unauthorized | The request is not authorized. Make sure your resource key or token is valid and in the correct region. |

+| 429 | Too many requests | You have exceeded the quota or rate of requests allowed for your resource. |

+These regions are supported for text-to-speech through the REST API. Be sure to select the endpoint that matches your Speech resource region.

+| 401 | Unauthorized | The request is not authorized. Make sure your Speech resource key or token is valid and in the correct region. |

+| 429 | Too many requests | You have exceeded the quota or rate of requests allowed for your resource. |

+ - Speech translation

+| [Speech-to-text REST API](rest-speech-to-text.md) | `https://<REGION_IDENTIFIER>.api.cognitive.microsoft.us/<URL_PATH>` |

+| [Speech-to-text REST API](rest-speech-to-text.md) | `https://<REGION_IDENTIFIER>.api.cognitive.azure.cn/<URL_PATH>` |

+var config = SpeechConfig.FromHost("azCnHost", subscriptionKey);

+auto config = SpeechConfig::FromHost("azCnHost", subscriptionKey);

+SpeechConfig config = SpeechConfig.fromHost("azCnHost", subscriptionKey);

+speech_config = speechsdk.SpeechConfig(host="azCnHost", subscription=subscriptionKey)

+SPXSpeechConfiguration *speechConfig = [[SPXSpeechConfiguration alloc] initWithHost:"azCnHost" subscription:subscriptionKey];

+> See [Adjust an application to use a Speech resource without private endpoints](#adjust-an-application-to-use-a-speech-resource-without-private-endpoints) later in this article.

+> When you're using the Speech-to-text REST API for short audio and Text-to-speech REST API in private endpoint scenarios, use a resource key passed through the `Ocp-Apim-Subscription-Key` header. (See details for [Speech-to-text REST API for short audio](rest-speech-to-text-short.md#request-headers) and [Text-to-speech REST API](rest-text-to-speech.md#request-headers))

+ var config = SpeechConfig.FromSubscription(speechKey, azureRegion);

+ - `speechKey` contains the key of the private-endpoint-enabled Speech resource.

+ var config = SpeechConfig.FromEndpoint(endPoint, speechKey);

+ auto config = SpeechConfig::FromEndpoint(endPoint, speechKey);

+ SpeechConfig config = SpeechConfig.fromEndpoint(endPoint, speechKey);

+ speech_config = speechsdk.SpeechConfig(endpoint=endPoint, subscription=speechKey)

+ SPXSpeechConfiguration *speechConfig = [[SPXSpeechConfiguration alloc] initWithEndpoint:endPoint subscription:speechKey];

+> When you're using the Speech-to-text REST API for short audio and Text-to-speech REST API in custom domain scenarios, use a Speech resource key passed through the `Ocp-Apim-Subscription-Key` header. (See details for [Speech-to-text REST API for short audio](rest-speech-to-text-short.md#request-headers) and [Text-to-speech REST API](rest-text-to-speech.md#request-headers))

+var config = SpeechConfig.FromSubscription(speechKey, azureRegion);

+## Create a resource configuration

+To get started, you need a Speech resource key and region identifier (for example, `eastus`, `westus`). Create a Speech resource on the [Azure portal](https://portal.azure.com). For more information, see [Create a new Azure Cognitive Services resource](~/articles/cognitive-services/cognitive-services-apis-create-account.md?tabs=speech#create-a-new-azure-cognitive-services-resource).

+To configure your resource key and region identifier, run the following commands:

+spx config @key --set SPEECH-KEY

+spx config @region --set SPEECH-REGION

+To get started, you need a Speech resource key and region identifier (for example, `eastus`, `westus`). Create a Speech resource on the [Azure portal](https://portal.azure.com). For more information, see [Create a new Azure Cognitive Services resource](~/articles/cognitive-services/cognitive-services-apis-create-account.md?tabs=speech#create-a-new-azure-cognitive-services-resource).

+To configure your Speech resource key and region identifier, run the following commands in PowerShell:

+spx --% config @key --set SPEECH-KEY

+spx --% config @region --set SPEECH-REGION

+Also, there might be a problem with your Speech resource key or authorization token. For more information, see the next section.

+* If you're using a resource key for authentication, you might see the error because:

+ - The key is missing or invalid

+ - You have exceeded your resource's usage quota

+### Validate your resource key

+You can verify that you have a valid resource key by running one of the following commands.

+> Replace `YOUR_RESOURCE_KEY` and `YOUR_REGION` with your own resource key and associated region.

+ 'Ocp-Apim-Subscription-Key' = 'YOUR_RESOURCE_KEY'

+ curl -v -X POST "https://YOUR_REGION.api.cognitive.microsoft.com/sts/v1.0/issueToken" -H "Ocp-Apim-Subscription-Key: YOUR_RESOURCE_KEY" -H "Content-type: application/x-www-form-urlencoded" -H "Content-Length: 0"

+If you entered a valid resource key, the command returns an authorization token, otherwise an error is returned.

+|Error (AuthenticationFailure) : WebSocket Upgrade failed with an authentication error (401). Check for correct resource key (or authorization token) and region name| On the **Settings** page of the app, make sure that you entered the key and its region correctly. |

+|Error (ConnectionFailure) : Connection was closed by the remote host. Error code: 1011. Error details: Response status code does not indicate success: 500 (InternalServerError)| Your bot specified a neural voice in the [speak](https://github.com/microsoft/botframework-sdk/blob/master/specs/botframework-activity/botframework-activity.md#speak) field of its output activity, but the Azure region associated with your resource key doesn't support neural voices. See [neural voices](./regions.md#speech-service) and [standard voices](how-to-migrate-to-prebuilt-neural-voice.md).|

+- [DialogServiceConfig](/dotnet/api/microsoft.cognitiveservices.speech.dialog.dialogserviceconfig): For configuration settings like resource key and its region.

+ "sourceUrl": "https://myblob.blob.core.windows.net/source"

+ > If you're new to Visual Studio, try the [Introduction to Visual Studio](/training/modules/go-get-started/) Learn module.

+> If you're new to Go, try the [Get started with Go](/training/modules/go-get-started/) Learn module.

+ > If you're new to Node.js, try the [Introduction to Node.js](/training/modules/intro-to-nodejs/) Learn module.

+ > If you're new to Python, try the [Introduction to Python](/training/paths/beginner-python/) Learn module.

+ > If you're new to Visual Studio, try the [Introduction to Visual Studio](/training/modules/go-get-started/) Learn module.

+> If you're new to Go, try the [Get started with Go](/training/modules/go-get-started/) Learn module.

+ > If you're new to Node.js, try the [Introduction to Node.js](/training/modules/intro-to-nodejs/) Learn module.

+ > If you're new to Python, try the [Introduction to Python](/training/paths/beginner-python/) Learn module.

+- Take the [Cost Management](/training/paths/control-spending-manage-bills?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) guided learning course.

+string imageURL = "https://learn.microsoft.com/azure/cognitive-services/bing-visual-search/media/ms_srleaders.jpg";

+> [What is the Bing Visual Search API?](./overview.md)

+For more information, see <a href="/cpp/c-runtime-library/reference/getenv-s-wgetenv-s" target="_blank">`getenv_s`</a> and <a href="/cpp/c-runtime-library/reference/getenv-wgetenv" target="_blank">`getenv`</a>.

+#include <iostream>

+std::string getEnvironmentVariable(const char* name);

+ auto value = getEnvironmentVariable("ENVIRONMENT_VARIABLE_KEY");

+}

+std::string getEnvironmentVariable(const char* name)

+{

+#if defined(_MSC_VER)

+ size_t requiredSize = 0;

+ (void)getenv_s(&requiredSize, nullptr, 0, name);

+ if (requiredSize == 0)

+ {

+ return "";

+ }

+ auto buffer = std::make_unique<char[]>(requiredSize);

+ (void)getenv_s(&requiredSize, buffer.get(), requiredSize, name);

+ return buffer.get();

+#else

+ auto value = getenv(name);

+ return value ? value : "";

+#endif

+* [Use Flask to translate text, analyze sentiment, and synthesize speech](/training/modules/python-flask-build-ai-web-app/)

+ "sourceUri": "https://learn.microsoft.com/azure/cognitive-services/qnamaker/overview/overview",

+ "answer": "The latest question answering docs are on https://learn.microsoft.com",

+> [!VIDEO https://learn.microsoft.com/Shows/AI-Show/Introducing-Text-Analytics-for-Health/player]

+* [Conversational language understanding](./conversational-language-understanding/overview.md) is available in the following regions:

+ * Central India

+ * Switzerland North

+ * West US 2

+## Inference Explainability

+Personalizer can help you to understand which features are the most and least influential when determining the best action. When enabled, inference explainability includes feature scores from the underlying model into the Rank API response, so your application receives this information at the time of inference.

+Feature scores empower you to better understand the relationship between features and the decisions made by Personalizer. They can be used to provide insight to your end-users into why a particular recommendation was made, or to analyze whether your model is exhibiting bias toward or against certain contextual settings, users, and actions.

+Setting the service configuration flag IsInferenceExplainabilityEnabled in your service configuration enables Personalizer to include feature values and weights in the Rank API response. To update your current service configuration, use the [Service Configuration ΓÇô Update API](https://docs.microsoft.com/rest/api/personalizer/1.1preview1/service-configuration/update?tabs=HTTP). In the JSON request body, include your current service configuration and add the additional entry: ΓÇ£IsInferenceExplainabilityEnabledΓÇ¥: true. If you donΓÇÖt know your current service configuration, you can obtain it from the [Service Configuration ΓÇô Get API](https://docs.microsoft.com/rest/api/personalizer/1.1preview1/service-configuration/get?tabs=HTTP)

+```JSON

+{

+ "rewardWaitTime": "PT10M",

+ "defaultReward": 0,

+ "rewardAggregation": "earliest",

+ "explorationPercentage": 0.2,

+ "modelExportFrequency": "PT5M",

+ "logMirrorEnabled": true,

+ "logMirrorSasUri": "https://testblob.blob.core.windows.net/container?se=2020-08-13T00%3A00Z&sp=rwl&spr=https&sv=2018-11-09&sr=c&sig=signature",

+ "logRetentionDays": 7,

+ "lastConfigurationEditDate": "0001-01-01T00:00:00Z",

+ "learningMode": "Online",

+ "isAutoOptimizationEnabled": true,

+ "autoOptimizationFrequency": "P7D",

+ "autoOptimizationStartDate": "2019-01-19T00:00:00Z",

+"isInferenceExplainabilityEnabled": true

+}

+```

+### How to interpret feature scores?

+Enabling inference explainability will add a collection to the JSON response from the Rank API called *inferenceExplanation*. This contains a list of feature names and values that were submitted in the Rank request, along with feature scores learned by PersonalizerΓÇÖs underlying model. The feature scores provide you with insight on how influential each feature was in the model choosing the action.

+```JSON

+{

+ "ranking": [

+ {

+ "id": "EntertainmentArticle",

+ "probability": 0.8

+ },

+ {

+ "id": "SportsArticle",

+ "probability": 0

+ },

+ {

+ "id": "NewsArticle",

+ "probability": 0.2

+ }

+ ],

+ "eventId": "75269AD0-BFEE-4598-8196-C57383D38E10",

+ "rewardActionId": "EntertainmentArticle",

+ "inferenceExplanation": [

+ {

+ "idΓÇ¥: "EntertainmentArticle",

+ "features": [

+ {

+ "name": "user.profileType",

+ "score": 3.0

+ },

+ {

+ "name": "user.latLong",

+ "score": -4.3

+ },

+ {

+ "name": "user.profileType^user.latLong",

+ "score" : 12.1

+ },

+ ]

+ ]

+}

+```

+Recall that Personalizer will either return the _best action_ as determined by the model or an _exploratory action_ chosen by the exploration policy. The best action is the one that the model has determined has the highest probability of maximizing the average reward, whereas exploratory actions are chosen among the set of all possible actions provided in the Rank API call. Actions taken during exploration do not leverage the feature scores in determining which action to take, therefore **feature scores for exploratory actions should not be used to gain an understanding of why the action was taken.** [You can learn more about exploration here](https://docs.microsoft.com/azure/cognitive-services/personalizer/concepts-exploration).

+For the best actions returned by Personalizer, the feature scores can provide general insight where:

+* Larger positive scores provide more support for the model choosing the best action.

+* Larger negative scores provide more support for the model not choosing the best action.

+* Scores close to zero have a small effect on the decision to choose the best action.

+### Important considerations for Inference Explainability

+* **Increased latency.** Enabling _Inference Explainability_ will significantly increase the latency of Rank API calls due to processing of the feature information. Run experiments and measure the latency in your scenario to see if it satisfies your applicationΓÇÖs latency requirements. Future versions of Inference Explainability will mitigate this issue.

+* **Correlated Features.** Features that are highly correlated with each other can reduce the utility of feature scores. For example, suppose Feature A is highly correlated with Feature B. It may be that Feature AΓÇÖs score is a large positive value while Feature BΓÇÖs score is a large negative value. In this case, the two features may effectively cancel each other out and have little to no impact on the model. While Personalizer is very robust to highly correlated features, when using _Inference Explainability_, ensure that features sent to Personalizer are not highly correlated

+The explore / best action traffic allocation is made randomly following the percentage set for exploration, and the default algorithm for exploration is epsilon-greedy.

+[Offline evaluation](concepts-offline-evaluation.md)

+- **Transparency & Explainability:** Consider enabling and using Personalizer's [inference explainability](https://learn.microsoft.com/azure/cognitive-services/personalizer/concepts-features?branch=main#inference-explainability) capability to better understand which features play a significant role in Personalizer's decision choice in each Rank call. This capability empowers you to provide your users with transparency regarding how their data played a role in producing the recommended best action. For example, you can give your users a button labeled "Why These Suggestions?" that shows which top features played a role in producing the Personalizer results. This information can also be used to better understand what data attributes about your users, contexts, and actions are working in favor of Personalizer's choice of best action, which are working against it, and which may have little or no effect. This capability can also provide insights about your user segments and help you identify and address potential biases.

+- Take the [Cost Management](/training/paths/control-spending-manage-bills?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) guided learning course.

+An access token is valid for a period of time between 1 and 24 hours. After it expires, the access token is invalidated and can't be used to access any primitive.

+To generate a token with a custom validity, specify the desired validity period when generating the token. If no custom validity is specified, the token will be valid for 24 hours.

+We recommend using short lifetime tokens for one-off meetings and longer lifetime tokens for agents using the application for longer periods of time.

+| | Reply to specific chat message | ❌ |

+>[!VIDEO https://www.youtube.com/embed/FF1LS516Bjw]

+// For more detailed tutorials on storage queues, see: https://learn.microsoft.com/azure/storage/queues/storage-tutorial-queues

+- [Introduction to Communication Services](/training/modules/intro-azure-communication-services/)

+- [Send an SMS message from a C# console application with Azure Communication Services](/training/modules/communication-service-send-sms-console-app/)

+- [Create a voice calling web app with Azure Communication Services](/training/modules/communication-services-voice-calling-web-app)

+ // The https://learn.microsoft.com/azure/developer/javascript/how-to/with-web-app/azure-function-file-upload

+ * [Learn: Create and manage Dataverse environments](/training/modules/create-manage-environments/)

+https://learn.microsoft.com/azure/azure-functions/functions-networking-options

+- [Deploy with an internal environment](vnet-custom-internal.md)

+az containerapp revision show `

+```azurecli

+* To learn more, see the [Azure Cosmos DB SQL API](/training/modules/intro-to-azure-cosmos-db-core-api/) training module.

+* Check out the training module on how to [Design hybrid transactional and analytical processing using Azure Synapse Analytics](/training/modules/design-hybrid-transactional-analytical-processing-using-azure-synapse-analytics/)

+This API stores data in document format. It offers the best end-to-end experience as we have full control over the interface, service, and the SDK client libraries. Any new feature that is rolled out to Azure Cosmos DB is first available on SQL API accounts. Azure Cosmos DB SQL API accounts provide support for querying items using the Structured Query Language (SQL) syntax, one of the most familiar and popular query languages to query JSON objects. To learn more, see the [Azure Cosmos DB SQL API](/training/modules/intro-to-azure-cosmos-db-core-api/) training module and [getting started with SQL queries](sql-query-getting-started.md) article.

+You can also checkout the training module on how to [configure Azure Synapse Link for Azure Cosmos DB.](/training/modules/configure-azure-synapse-link-with-azure-cosmos-db/)

+* Checkout the training module on how to [configure Azure Synapse Link for Azure Cosmos DB.](/training/modules/configure-azure-synapse-link-with-azure-cosmos-db/)

+// https://learn.microsoft.com/azure/cosmos-db/mongodb-custom-commands#update-collection

+ * If you know typical request rates for your current database workload, read about [estimating request units using Azure Cosmos DB capacity planner](estimate-ru-capacity-planner.md)

+* See the training module on how to [Model and partition your data in Azure Cosmos DB.](/training/modules/model-partition-data-azure-cosmos-db/)

+* Take the [Cost Management](/training/paths/control-spending-manage-bills?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) guided learning course.

+* A continuous running [Azure WebJob](/training/modules/run-web-app-background-task-with-webjobs/).

+> [!VIDEO https://learn.microsoft.com/Shows/Docs-Azure/Quickstart-Use-Nodejs-to-connect-and-query-data-from-Azure-Cosmos-DB-SQL-API-account/player]

+> [Build an app that queries and adds data to Azure Cosmos DB SQL API](/training/modules/build-dotnet-app-cosmos-db-sql-api/)

+Sign into the [Azure portal](https://portal.azure.com) and navigate to your Azure Cosmos DB account. Check that the three records from the ΓÇ£hotelsΓÇ¥ topic are created in your account.

+1. Sign into the [Azure portal](https://portal.azure.com/learn.learn.microsoft.com) and navigate to your Azure Cosmos DB account.

+* Kafka Connect for Azure Cosmos DB [sink connector](kafka-connector-sink.md)

+ #https://learn.microsoft.com/azure/cosmos-db/spark-connector

+* See the training module on how to [Model and partition your data in Azure Cosmos DB.](/training/modules/model-partition-data-azure-cosmos-db/)

+// https://learn.microsoft.com/dotnet/api/microsoft.azure.documents.resource.id#remarks

+* Check out the training module on how to [Design hybrid transactional and analytical processing using Azure Synapse Analytics](/training/modules/design-hybrid-transactional-analytical-processing-using-azure-synapse-analytics/)

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+ // https://learn.microsoft.com/rest/api/storageservices/Table-Service-Error-Codes

+| EnrollmentReader | Enrollment readers can view data at the enrollment, department, and account scopes. The data contains charges for all of the subscriptions under the scopes, including across tenants. Can view the Azure Prepayment (previously called monetary commitment) balance associated with the enrollment. | 24f8edb6-1668-4659-b5e2-40bb5f3a7d7e |

+If you want to a delete credit card, see [Delete an Azure billing payment method](delete-azure-payment-method.md).

+> Azure doesn't support virtual or prepaid cards.

+ 'billingplan': 'Monthly',

+Make your purchase using the returned `reservationOrderId` that you got from the preceding [Get the reservation order and price](#get-the-reservation-order-and-price) section.

+ 'billingplan': 'Monthly',

+ 'renew': true

+- See [SKUs for SAP HANA on Azure (Large Instances)](../../virtual-machines/workloads/sap/hana-available-skus.md) for the available SKU list and regions.

+- **Azure Backup Storage reserved capacity** - A capacity reservation lowers storage costs of backup data in a Recovery Services Vault.

+2020-07-06T09:50:50.8775530Z ##[error]Check out the troubleshooting guide to see if your issue is addressed: https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment#troubleshooting

+* Use updated [default parameterization template.](/azure/data-factory/continuous-integration-delivery-resource-manager-custom-parameters#default-parameterization-template) as one time migration to new method of including global parameters. This template references to all values in global parameter list. You also have to update the deployment task in the **release pipeline** if you are already overriding the template parameters there.

+* Update the template parameter names in CI/CD pipeline if you are already overriding the template parameters (for global parameters).

+ Title: Transform data from an SAP ODP source with the SAP CDC connector in Azure Data Factory or Azure Synapse Analytics

+description: Learn how to transform data from an SAP ODP source to supported sink data stores by using mapping data flows in Azure Data Factory or Azure Synapse Analytics.

+# Transform data from an SAP ODP source using the SAP CDC connector in Azure Data Factory or Azure Synapse Analytics

+This article outlines how to use mapping data flow to transform data from an SAP ODP source using the SAP CDC connector. To learn more, read the introductory article for [Azure Data Factory](introduction.md) or [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md). For an introduction to transforming data with Azure Data Factory and Azure Synapse analytics, read [mapping data flow](concepts-data-flow-overview.md).

+>[!TIP]

+>To learn the overall support on SAP data integration scenario, see [SAP data integration using Azure Data Factory whitepaper](https://github.com/Azure/Azure-DataFactory/blob/master/whitepaper/SAP%20Data%20Integration%20using%20Azure%20Data%20Factory.pdf) with detailed introduction on each SAP connector, comparsion and guidance.

+## Supported capabilities

+This SAP CDC connector is supported for the following capabilities:

+| Supported capabilities|IR |

+|| --|

+|[Mapping data flow](concepts-data-flow-overview.md) (source/-)|&#9313;|

+<small>*&#9312; Azure integration runtime &#9313; Self-hosted integration runtime*</small>

+This SAP CDC connector leverages the SAP ODP framework to extract data from SAP source systems. For an introduction to the architecture of the solution, read [Introduction and architecture to SAP change data capture (CDC)](sap-change-data-capture-introduction-architecture.md) in our [SAP knowledge center](industry-sap-overview.md).

+The SAP ODP framework is contained in most SAP NetWeaver based systems, including SAP ECC, SAP S/4HANA, SAP BW, SAP BW/4HANA, SAP LT Replication Server (SLT), except very old ones. For prerequisites and minimum required releases, see [Prerequisites and configuration](sap-change-data-capture-prerequisites-configuration.md#sap-system-requirements).

+The SAP CDC connector supports basic authentication or Secure Network Communications (SNC), if SNC is configured.

+## Prerequisites

+To use this SAP CDC connector, you need to:

+- Set up a self-hosted integration runtime (version 3.17 or later). For more information, see [Create and configure a self-hosted integration runtime](create-self-hosted-integration-runtime.md).

+- Download the 64-bit [SAP Connector for Microsoft .NET 3.0](https://support.sap.com/en/product/connectors/msnet.html) from SAP's website, and install it on the self-hosted integration runtime machine. During installation, make sure you select the **Install Assemblies to GAC** option in the **Optional setup steps** window.

+ :::image type="content" source="./media/connector-sap-business-warehouse-open-hub/install-sap-dotnet-connector.png" alt-text="Screenshot showing installation of SAP Connector for .NET.":::

+- The SAP user who's being used in the SAP table connector must have the permissions described in [User Configuration](sap-change-data-capture-prerequisites-configuration.md#set-up-the-sap-user):

+## Get started

+## Create a linked service for the SAP CDC connector using UI

+Follow the steps described in [Prepare the SAP CDC linked service](sap-change-data-capture-prepare-linked-service-source-dataset.md#set-up-a-linked-service) to create a linked service for the SAP CDC connector in the Azure portal UI.

+## Dataset properties

+To prepare an SAP CDC dataset, follow [Prepare the SAP CDC source dataset](sap-change-data-capture-prepare-linked-service-source-dataset.md#set-up-the-source-dataset)

+## Transform data with the SAP CDC connector

+SAP CDC datasets can be used as source in mapping data flow. Since the raw SAP ODP change feed is difficult to interpret and to correctly update to a sink, mapping data flow takes care of this by evaluating technical attributes provided by the ODP framework (e.g., ODQ_CHANGEMODE) automatically. This allows users to concentrate on the required transformation logic without having to bother with the internals of the SAP ODP change feed, the right order of changes, etc.

+### Mapping data flow properties

+To create a mapping data flow using the SAP CDC connector as a source, complete the following steps:

+1. In ADF Studio, go to the **Data flows** section of the **Author** hub, select the **…** button to drop down the **Data flow actions** menu, and select the **New data flow** item. Turn on debug mode by using the **Data flow debug** button in the top bar of data flow canvas.

+ :::image type="content" source="media/sap-change-data-capture-solution/sap-change-data-capture-mapping-data-flow-data-flow-debug.png" alt-text="Screenshot of the data flow debug button in mapping data flow.":::

+1. In the mapping data flow editor, select **Add Source**.

+ :::image type="content" source="media/sap-change-data-capture-solution/sap-change-data-capture-mapping-data-flow-add-source.png" alt-text="Screenshot of add source in mapping data flow.":::

+1. On the tab **Source settings** select a prepared SAP CDC dataset or select the **New** button to create a new one. Alternatively, you can also select **Inline** in the **Source type** property and continue without defining an explicit dataset.

+ :::image type="content" source="media/sap-change-data-capture-solution/sap-change-data-capture-mapping-data-flow-select-dataset.png" alt-text="Screenshot of the select dataset option in source settings of mapping data flow source.":::

+1. On the tab **Source options** select the option **Full on every run** if you want to load full snapshots on every execution of your mapping data flow, or **Full on the first run, then incremental** if you want to subscribe to a change feed from the SAP source system. In this case, the first run of your pipeline will do a delta initialization, which means it will return a current full data snapshot and create an ODP delta subscription in the source system so that with subsequent runs, the SAP source system will return incremental changes since the previous run only. In case of incremental loads it is required to specify the keys of the ODP source object in the **Key columns** property.

+ :::image type="content" source="media/sap-change-data-capture-solution/sap-change-data-capture-mapping-data-flow-run-mode.png" alt-text="Screenshot of the run mode property in source options of mapping data flow source.":::

+ :::image type="content" source="media/sap-change-data-capture-solution/sap-change-data-capture-mapping-data-flow-key-columns.png" alt-text="Screenshot of the key columns selection in source options of mapping data flow source.":::

+1. For details on the tabs **Projection**, **Optimize** and **Inspect**, please follow [mapping data flow](concepts-data-flow-overview.md).

+- **Message**: `Fail to read data form Azure Blob Storage for Azure Blob connector needs MD5 algorithm which can't co-work with FIPS mode. Please change diawp.exe.config in self-hosted integration runtime install directory to disable FIPS policy following https://learn.microsoft.com/dotnet/framework/configure-apps/file-schema/runtime/enforcefipspolicy-element.`

+# Virtual network injection method: Standard or Express. For comparison, see https://learn.microsoft.com/azure/data-factory/azure-ssis-integration-runtime-virtual-network-configuration.

+# For the basic pricing tier, specify "Basic," not "B." For standard, premium, and elastic pool tiers, specify "S0," "S1," "S2," "S3," etc. See https://learn.microsoft.com/azure/sql-database/sql-database-resource-limits-database-server.

+# Virtual network injection method: Standard or Express. For comparison, see https://learn.microsoft.com/azure/data-factory/azure-ssis-integration-runtime-virtual-network-configuration.

+# For the basic pricing tier, specify "Basic," not "B." For standard, premium, and elastic pool tiers, specify "S0," "S1," "S2," "S3," etc. See https://learn.microsoft.com/azure/sql-database/sql-database-resource-limits-database-server.

+> [!VIDEO https://learn.microsoft.com/Shows/Azure-Friday/Hybrid-data-movement-across-multiple-Azure-Data-Factories/player]

+**Column name pattern:** Select how to format the column name of each pivot column. The outputted column name will be a combination of the pivot key value, column prefix and optional prefix, suffix, middle characters.

+> [!VIDEO https://learn.microsoft.com/Shows/Azure-Friday/Iterative-development-and-debugging-with-Azure-Data-Factory/player]

+# Virtual network injection method: Standard or Express. For comparison, see https://learn.microsoft.com/azure/data-factory/azure-ssis-integration-runtime-virtual-network-configuration.

+> [!VIDEO https://learn.microsoft.com/shows/azure-friday/Monitor-your-Azure-Data-Factory-pipelines-proactively-with-alerts/player]

+> [!VIDEO https://learn.microsoft.com/shows/azure-friday/Parameterize-connections-to-your-data-stores-in-Azure-Data-Factory/player]

+- Take the [Cost Management](/training/paths/control-spending-manage-bills?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) guided learning course.

+>[!VIDEO https://learn.microsoft.com/Shows/Azure-Friday/Visually-build-pipelines-for-Azure-Data-Factory-v2/Player]

+ Title: Debug SAP CDC connector (preview) by sending logs

+description: Learn how to debug issues with the Azure Data Factory SAP CDC (change data capture) connector (preview) by sending self-hosted integration runtime logs to Microsoft.

+# Debug the SAP CDC connector by sending self-hosted integration runtime logs

+If you want Microsoft to debug Azure Data Factory issues with your SAP CDC connector (preview), send us your self-hosted integration runtime logs, and then contact us.

+[SAP CDC (Change Data Capture) Connector](connector-sap-change-data-capture.md)

+ Title: Overview and architecture of the SAP CDC capabilities (preview)

+description: Learn about the SAP change data capture (CDC) capabilities (preview) in Azure Data Factory and understand its architecture.

+# Overview and architecture of the SAP CDC capabilities (preview)

+Learn about the SAP change data capture (CDC) capabilities (preview) in Azure Data Factory and understand the architecture.

+## SAP CDC capabilities

+Microsoft customers indicate that they need a connector that can extract only the delta between two sets of data. In data, a *delta* is any change in a dataset that's the result of an update, insert, or deletion in the dataset. A delta extraction connector uses the [SAP change data capture (CDC) feature](https://help.sap.com/docs/SAP_DATA_SERVICES/ec06fadc50b64b6184f835e4f0e1f52f/1752bddf523c45f18ce305ac3bcd7e08.html?q=change%20data%20capture) that exists in most SAP systems to determine the delta in a dataset. The SAP CDC capabilities in Data Factory use the SAP Operational Data Provisioning (ODP) framework to replicate the delta in an SAP source dataset.

+This article provides a high-level architecture of the SAP CDC capabilities in Azure Data Factory. Get more information about the SAP CDC capabilities:

+## How to use the SAP CDC capabilities

+At the core of the SAP CDC capabilities is the new SAP CDC connector (preview). It can connect to all SAP systems that support ODP. This includes SAP ECC, SAP S/4HANA, SAP BW, and SAP BW/4HANA. The solution works either directly at the application layer or indirectly via an SAP Landscape Transformation Replication Server (SLT) as a proxy. It doesn't rely on watermarking to extract SAP data either fully or incrementally. The data the SAP CDC connector extracts includes not only physical tables but also logical objects that are created by using the tables. An example of a table-based object is an SAP Advanced Business Application Programming (ABAP) Core Data Services (CDS) view.

+Use the SAP CDC connector with Data Factory features like mapping data flow activities, and tumbling window triggers for a low-latency SAP CDC replication solution in a self-managed pipeline.

+## The SAP CDC architecture

+The Azure side includes the Data Factory mapping data flow that can transform and load the SAP data into any data sink supported by mapping data flows. This includes storage destinations like Azure Data Lake Storage Gen2 or databases like Azure SQL Database or Azure Synapse Analytics. The Data Factory data flow activity also can load the results in Data Lake Storage Gen2 in delta format. You can use the Delta Lake Time Travel feature to produce snapshots of SAP data for a specific period. You can run your pipeline and mapping data flows frequently by using a Data Factory tumbling window trigger to replicate SAP data in Azure with low latency and without using watermarking.

+To get started, create a Data Factory SAP CDC linked service, an SAP CDC source dataset, and a pipeline with a mapping data flow activity in which you use the SAP CDC source dataset. To extract the data from SAP, a self-hosted integration runtime is required that you install on an on-premises computer or on a virtual machine (VM). An on-premises computer has a line of sight to your SAP source systems and to your SLT server. The Data Factory data flow activity runs on a serverless Azure Databricks or Apache Spark cluster, or on an Azure integration runtime.

+The SAP CDC connector uses the SAP ODP framework to extract various data source types, including:

+- SAP extractors, originally built to extract data from SAP ECC and load it into SAP BW

+- ABAP CDS views, the new data extraction standard for SAP S/4HANA

+- InfoProviders and InfoObjects datasets in SAP BW and SAP BW/4HANA

+- SAP application tables, when you use an SAP LT replication server (SLT) as a proxy

+In this process, the SAP data sources are *providers*. The providers run on SAP systems to produce either full or incremental data in an operational delta queue (ODQ). The Data Factory mapping data flow source is a *subscriber* of the ODQ.

+ Title: Manage your SAP CDC (preview) ETL process

+# Manage your SAP CDC (preview) ETL process

+After you create a pipeline in Azure Data Factory using the SAP CDC connector (preview), it's important to manage the solution.

+1. In **Subscriber**, enter the value for the **Subscriber name** property of your SAP CDC (preview) linked service. In the **Request Selection** dropdown, select **All** to show all data extractions that use the linked service.

+ You can see all registered subscriber processes in the operational delta queue (ODQ). Subscriber processes represent data extractions from Azure Data Factory copy activities that use your SAP CDC linked service. For each ODQ subscription, you can look at details to see all full and delta extractions. For each extraction, you can see individual data packages that were consumed.

+The SAP CDC connector in Data Factory reads delta changes from the SAP ODP framework. The deltas are recorded in ODQ tables.

+Here are current limitations of the SAP CDC connector in Data Factory:

+ Title: Set up a linked service and dataset for the SAP CDC connector (preview)

+description: Learn how to set up a linked service and source dataset to use with the SAP CDC (change data capture) connector (preview) in Azure Data Factory.

+# Set up a linked service and source dataset for the SAP CDC connector (preview)

+Learn how to set up the linked service and source dataset for the SAP CDC connector (preview) in Azure Data Factory.

+To set up an SAP CDC (preview) linked service:

+1. In **New linked service**, search for **SAP**. Select **SAP CDC (Preview)**, and then select **Continue**.

+ :::image type="content" source="media/sap-change-data-capture-solution/sap-cdc-linked-service-selection.png" alt-text="Screenshot of the linked service source selection, with SAP CDC (Preview) selected.":::

+ 1. In **Subscriber name**, enter a unique name to register and identify this Data Factory connection as a subscriber that consumes data packages that are produced in the Operational Delta Queue (ODQ) by your SAP system. For example, you might name it `<your data factory -name>_<your linked service name>`. Make sure to only use upper case letters.

+ Make sure you assign a unique subscriber name to every linked service connecting to the same SAP system. This will make monitoring and trouble shooting on SAP side much easier.

+ :::image type="content" source="media/sap-change-data-capture-solution/sap-cdc-linked-service-configuration.png" alt-text="Screenshot of the SAP CDC linked service configuration.":::

+## Set up the source dataset

+1. In Azure Data Factory Studio, go to the Author hub of your data factory. In **Factory Resources**, under **Datasets** > **Dataset Actions**, select **New dataset**.

+1. In **New dataset**, search for **SAP**. Select **SAP CDC (Preview)**, and then select **Continue**.

+ :::image type="content" source="media/sap-change-data-capture-solution/sap-cdc-source-dataset-selection.png" alt-text="Screenshot of the SAP CDC (Preview) dataset type in the New dataset dialog.":::

+1. In **Set properties**, enter a name for the SAP CDC linked service data source. In **Linked service**, select the dropdown and select **New**.

+1. Select your SAP CDC linked service for the new source dataset and set the rest of the properties for the linked service:

+ 1. In **ODP context**, select the context of the ODP data extraction. Here are some examples:

+ 1. For **ODP name**, under the selected data extraction context, select the name of the data source object to extract. If you connect to your SAP source system by using SLT as a proxy, the **Preview data** feature currently isn't supported.

+ :::image type="content" source="media/sap-change-data-capture-solution/sap-cdc-source-dataset-configuration.png" alt-text="Screenshot of the SAP CDC (Preview) dataset configuration page.":::

+1. Select **OK** to create your new SAP CDC source dataset.

+## Set up a mapping data flow using the SAP CDC dataset as a source

+To set up a mapping data flow using the SAP CDC dataset as a source, follow [Transform data with the SAP CDC connector](connector-sap-change-data-capture.md#transform-data-with-the-sap-cdc-connector)

+ Title: Prerequisites and setup for the SAP CDC connector (preview)

+description: Learn about the prerequisites and setup for the SAP CDC connector (preview) in Azure Data Factory.

+# Prerequisites and setup for the SAP CDC connector (preview)

+Learn about the prerequisites for the SAP CDC connector (preview) in Azure Data Factory and how to set up the solution in Azure Data Factory Studio.

+To preview the SAP CDC capabilities in Azure Data Factory, be able to complete these prerequisites:

+- Be familiar with Data Factory concepts like integration runtimes, linked services, datasets, activities, data flows, pipelines, and triggers.

+- Set up an SAP CDC (preview) linked service.

+- Set up the Data Factory copy activity with an SAP CDC (preview) source dataset.

+The ODP framework is part of many SAP systems, including SAP ECC and SAP S/4HANA. It is also contained in SAP BW and SAP BW/4HANA. To ensure that your SAP releases have ODP, see the following SAP documentation or support notes. Even though the guidance primarily refers to SAP BW and SAP Data Services, the information also applies to Data Factory.

+ Title: Set up a self-hosted integration runtime for the SAP CDC connector (preview)

+# Set up a self-hosted integration runtime for the SAP CDC connector (preview)

+Learn how to create and set up a self-hosted integration runtime for the SAP CDC connector (preview) in Azure Data Factory.

+To prepare a self-hosted integration runtime to use with the SAP CDC connector (preview), complete the steps that are described in the following sections.

+> [!VIDEO https://learn.microsoft.com/Shows/Azure-Friday/Execute-Jars-and-Python-scripts-on-Azure-Databricks-using-Data-Factory/player]

+> [!VIDEO https://learn.microsoft.com/Shows/Azure-Friday/Execute-Jars-and-Python-scripts-on-Azure-Databricks-using-Data-Factory/player]

+> [!VIDEO https://learn.microsoft.com/Shows/Azure-Friday/How-to-execute-Azure-Machine-Learning-service-pipelines-in-Azure-Data-Factory/player]

+> [!VIDEO https://learn.microsoft.com/Shows/Azure-Friday/ingest-prepare-and-transform-using-azure-databricks-and-data-factory/player]

+> [!VIDEO https://learn.microsoft.com/Shows/Azure-Friday/Create-dependent-pipelines-in-your-Azure-Data-Factory/player]

+# For the basic pricing tier, specify "Basic", not "B" - For standard/premium/elastic pool tiers, specify "S0", "S1", "S2", "S3", etc., see https://learn.microsoft.com/azure/sql-database/sql-database-resource-limits-database-server

+# For the basic pricing tier, specify "Basic", not "B" - For standard/premium/elastic pool tiers, specify "S0", "S1", "S2", "S3", etc., see https://learn.microsoft.com/azure/sql-database/sql-database-resource-limits-database-server

+>[Customize your Azure-SSIS IR](./how-to-configure-azure-ssis-ir-custom-setup.md)

+ <!-- Cannot get the link to render properly for version at https://learn.microsoft.com/azure/iot-edge/troubleshoot?view=iotedge-2020-11 -->

+1. Deallocate the VM by stopping the VM in the portal. To stop the VM, go to **Virtual machines** > **Overview**, and select the VM. Then, on the VM properties page, select **Stop**.<!--Follow-up (formatting): Create an include file for stopping a VM. Use it here and in prerequisites for "Use the Azure portal to manage network interfaces on the VMs" (https://learn.microsoft.com/azure/databox-online/azure-stack-edge-gpu-manage-virtual-machine-network-interfaces-portal#prerequisites).-->

+* [Learn module: Introduction to Azure DDoS Protection](/training/modules/introduction-azure-ddos-protection/)

+- [Legacy Assessments APIs deprecation](#legacy-assessments-apis-deprecation)

+### Legacy Assessments APIs deprecation

+The following APIs are deprecated:

+- Security Tasks

+- Security Statuses

+- Security Summaries

+These three APIs exposed old formats of assessments and are replaced by the [Assessments APIs](/rest/api/defenderforcloud/assessments) and [SubAssessments APIs](/rest/api/defenderforcloud/sub-assessments). All data that is exposed by these legacy APIs are also available in the new APIs.

+- [The Learn module on how to use workflow automation to automate a security response](/training/modules/resolve-threats-with-azure-security-center/)

+ Title: How to manage network connections

+description: This article describes how to create, delete, attach and remove Microsoft Dev Box network connections.

+<!-- Intent: As a dev infrastructure manager, I want to be able to manage network connections so that I can enable dev boxes to connect to my existing networks and deploy them in the desired region. -->

+# Manage network connections

+Network connections allow dev boxes to connect to existing virtual networks, and determine the region into which dev boxes are deployed.

+When planning network connectivity for your dev boxes, you must:

+- Ensure you have sufficient permissions to create and configure network connections.

+- Ensure you have at least one virtual network (VNet) and subnet available for your dev boxes.

+- Identify the region or location closest to your dev boxes users. Deploying dev boxes into a region close to the users provides them with a better experience.

+- Determine whether dev boxes should connect to your existing networks using an Azure Active Directory (Azure AD) join, or a Hybrid Azure AD join.

+## Permissions

+To manage a network connection, you need the following permissions:

+|Action|Permission required|

+|--|--|

+|Create and configure VNet and subnet|Network Contributor permissions on an existing virtual network (owner or contributor) or permission to create a new virtual network and subnet.|

+|Create or delete network connection|Owner or Contributor permissions on an Azure Subscription or a specific resource group.|

+|Add or remove network connection |Write permission on the dev center.|

+## Create a virtual network and subnet

+To create a network connection, you need an existing VNet and subnet. If you don't have a VNet and subnet available, use the following steps to create them:

+1. Sign in to the [Azure portal](https://portal.azure.com).

+

+1. In the search box, enter *Virtual Network*, and then select **Virtual Network** from the search results.

+1. On the Virtual Network page, select **Create**.

+1. On the Create virtual network page, enter or select this information on the **Basics** tab:

+ | Setting | Value |

+ | - | -- |

+ | Subscription | Select your subscription. |

+ | Resource group | Select an existing resource group, or to create a new one: </br> Select **Create new**. </br> Enter *rg-name*. </br> Select **OK**. |

+ | Name | Enter *VNet-name*. |

+ | Region | Select the region for the VNet and dev boxes. |

+ :::image type="content" source="./media/how-to-manage-network-connection/example-basics-tab.png" alt-text="Screenshot of creating a virtual network in Azure portal." border="true":::

+ > [!Important]

+ > The region you select for the VNet is the where the dev boxes will be deployed.

+1. On the **IP Addresses** tab, accept the default settings.

+1. On the **Security** tab, accept the default settings.

+1. On the **Review + create** tab review the settings.

+1. Select **Create**.

+

+## Allow access to Dev Box endpoints from your network

+Network ingress and egress can be controlled using a firewall, network security groups, and even Microsoft Defender.

+If your organization routes egress traffic through a firewall, you need to open certain ports to allow the Dev Box service to function. For more information, see [Network requirements](/windows-365/enterprise/requirements-network).

+## Plan a network connection

+The following steps show you how to create and configure a network connection in Microsoft Dev Box.

+### Types of Azure Active Directory Join

+The Dev Box service requires a configured and working Azure AD join or Hybrid AD join, which defines how dev boxes join your domain and access resources.

+If your organization uses Azure AD, you can use an Azure AD join, sometimes called a native Azure AD join. Dev box users sign into Azure AD joined dev boxes using their Azure AD account and access resources based on the permissions assigned to that account. Azure AD join enables access to cloud-based and on-premises apps and resources.

+If your organization has an on-premises Active Directory implementation, you can still benefit from some of the functionality provided by Azure AD by using hybrid Azure AD joined dev boxes. These dev boxes are joined to your on-premises Active Directory and registered with Azure Active Directory. Hybrid Azure AD joined dev boxes require network line of sight to your on-premises domain controllers periodically. Without this connection, devices become unusable.

+You can learn more about each type of join and how to plan for them here:

+- [Plan your hybrid Azure Active Directory join deployment](/azure/active-directory/devices/hybrid-azuread-join-plan)

+- [Plan your Azure Active Directory join deployment](/azure/active-directory/devices/azureadjoin-plan)

+### Create a network connection

+1. Sign in to the [Azure portal](https://portal.azure.com).

+1. In the search box, type *Network connections* and then select **Network connections** from the list.

+1. On the **Network Connections** page, select **+Create**.

+ :::image type="content" source="./media/how-to-manage-network-connection/network-connections-empty.png" alt-text="Screenshot showing the Network Connections page with Create highlighted.":::

+1. Follow the steps on the appropriate tab to create your network connection.

+ #### [**Azure AD join**](#tab/AzureADJoin/)

+ On the **Create a network connection** page, on the **Basics** tab, enter the following values:

+ |Name|Value|

+ |-|-|

+ |**Domain join type**|Select **Azure active directory join**.|

+ |**Subscription**|Select the subscription in which you want to create the network connection.|

+ |**Resource group**|Select an existing resource group or select **Create new**, and enter a name for the resource group.|

+ |**Name**|Enter a descriptive name for your network connection.|

+ |**Virtual network**|Select the virtual network you want the network connection to use.|

+ |**Subnet**|Select the subnet you want the network connection to use.|

+ :::image type="content" source="./media/how-to-manage-network-connection/create-native-network-connection-full-blank.png" alt-text="Screenshot showing the create network connection basics tab with Azure Active Directory join highlighted.":::

+ #### [**Hybrid Azure AD join**](#tab/HybridAzureADJoin/)

+ On the **Create a network connection** page, on the **Basics** tab, enter the following values:

+ |Name|Value|

+ |-|-|

+ |**Domain join type**|Select **Hybrid Azure active directory join**.|

+ |**Subscription**|Select the subscription in which you want to create the network connection.|

+ |**Resource group**|Select an existing resource group or select **Create new**, and enter a name for the resource group.|

+ |**Name**|Enter a descriptive name for your network connection.|

+ |**Virtual network**|Select the virtual network you want the network connection to use.|

+ |**Subnet**|Select the subnet you want the network connection to use.|

+ |**AD DNS domain name**| The DNS name of the Active Directory domain that you want to use for connecting and provisioning Cloud PCs. For example, corp.contoso.com. |

+ |**Organizational unit**| An organizational unit (OU) is a container within an Active Directory domain, which can hold users, groups, and computers. |

+ |**AD username UPN**| The username, in user principal name (UPN) format, that you want to use for connecting the Cloud PCs to your Active Directory domain. For example, svcDomainJoin@corp.contoso.com. This service account must have permission to join computers to the domain and, if set, the target OU. |

+ |**AD domain password**| The password for the user specified above. |

+ :::image type="content" source="./media/how-to-manage-network-connection/create-hybrid-network-connection-full-blank.png" alt-text="Screenshot showing the create network connection basics tab with Hybrid Azure Active Directory join highlighted.":::

+

+Use the following steps to finish creating your network connection, for both Azure AD join and Hybrid Azure AD join:

+ 1. Select **Review + Create**.

+ 1. On the **Review** tab, select **Create**.

+ 1. When the deployment is complete, select **Go to resource**. You'll see the Network Connection overview page.

+

+## Attach network connection to dev center

+You need to attach a network connection to a dev center before it can be used in projects to create dev box pools.

+1. In the [Azure portal](https://portal.azure.com), in the search box, type *Dev centers* and then select **Dev centers** from the list.

+1. Select the dev center you created and select **Networking**.

+

+1. Select **+ Add**.

+

+1. In the **Add network connection** pane, select the network connection you created earlier, and then select **Add**.

+

+ :::image type="content" source="./media/how-to-manage-network-connection/add-network-connection.png" alt-text="Screenshot showing the Add network connection pane.":::

+After creation, several health checks are run on the network. You can view the status of the checks on the resource overview page. Network connections that pass all the health checks can be added to a dev center and used in the creation of dev box pools. The dev boxes within the dev box pools will be created and domain joined in the location of the VNet assigned to the network connection.

+To resolve any errors, refer to the [Troubleshoot Azure network connections](/windows-365/enterprise/troubleshoot-azure-network-connection).

+## Remove a network connection from a dev center

+You can remove a network connection from a dev center if you no longer want it to be used to connect to network resources. Network connections can't be removed if they are in use by one or more dev box pools.

+1. In the [Azure portal](https://portal.azure.com), in the search box, type *Dev centers* and then select **Dev centers** from the list.

+1. Select the dev center you created and select **Networking**.

+

+1. Select the network connection you want to remove and then select **Remove**.

+ :::image type="content" source="./media/how-to-manage-network-connection/remove-network-connection.png" alt-text="Screenshot showing the network connection page with Remove highlighted.":::

+1. Read the warning message, and then select **Ok**.

+The network connection will no longer be available for use in the dev center.

+## Next steps

+<!-- [Manage a dev center](./how-to-manage-dev-center.md) -->

+- [Quickstart: Configure a Microsoft Dev Box Project](./quickstart-configure-dev-box-project.md)

+> [!VIDEO https://learn.microsoft.com/Events/Build/2018/BRK3308/player]

+ Title: Retirement of DevOps Starter for Azure | Microsoft Docs

+description: Retirement of Azure Devops Starter and Migration

+documentationcenter: ''

+editor:

+ms.assetid:

+ na

+# Retirement of DevOps Starter

+Azure DevOps Starter will be retired March 31, 2023. The corresponding REST APIs for [Microsoft.DevOps](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/devops/resource-manager/Microsoft.DevOps/) and [Microsoft.VisualStudio/accounts/projects](/rest/api/visualstudio/projects) resources will be retired as well.

+Customers are encouraged to use [Azure Developer CLI](/azure/developer/azure-developer-cli/overview?tabs=nodejs) instead.

+## Azure Developer CLI

+The replacement [Azure Developer CLI (azd)](/azure/developer/azure-developer-cli/overview?tabs=nodejs) is a developer command-line tool for building cloud apps. It provides commands that map to key stages in your workflow: code, build, deploy, monitor, repeat. You can use Azure CLI to create, provision, and deploy a new application in a single step.

+## Comparison between Azure DevOps and Azure Developer CLI

+| DevOps Starter | Azure Developer CLI |

+| | - |

+| Deploy to Azure with few clicks | A single step to deploy to Azure |

+| Configures code, deployment, monitoring | Configures code, deployment, monitoring |

+| Provides sample application to get started | Provides sample applications to get started |

+| Allows userΓÇÖs repo to be deployed | Allows userΓÇÖs repo to be deployed |

+| UI-based experience in Azure portal | CLI-based experience |

+## Migration:

+There is no migration required because DevOps Starter does not store any information, it just helps users with their Day 0 getting started experience on Azure. Moving forward the recommended way for users to get started on Azure will be [Azure Developer CLI](/azure/developer/azure-developer-cli/overview?tabs=nodejs).

+1. For choosing language, framework and target service, choose an appropriate [template](https://github.com/search?q=org:azure-samples%20topic:azd-templates) from azd repo and run the command `azd up --template \<template-name\>`

+2. For provisioning Azure service resources, run the command `azd provision`

+3. For creating CI/CD pipelines, run the command `azd pipeline config`

+4. For application insights monitoring, run the command `azd monitor`

+For existing application deployments, **DevOps starter does not store any information itself** and users can use following to get same information:

+1. Azure resource details in Azure portal ΓÇô In Azure portal, visit the resource page for which you had configured DevOps starter.

+2. To see pipeline and deployment information, go to the corresponding GitHub Actions workflow or Azure pipeline to view runs and deployments.

+3. To see monitoring details in Application insights, go to application insights for your Azure resource and look at the monitoring charts.

+## FAQ

+### What is the difference between DevOps starter and Azure developer CLI?

+Both are tools, which enable quick setting up of application deployment to Azure and configure CI/CD pipeline for the same. They enable users to quickly get started with Azure.

+Azure Developer CLI provides more developer-friendly commands in contrast to the UI wizard for DevOps Starter. This also means better clarity with config-as-code.

+### Will I lose my application or the Azure resources if I am not able to access DevOps starter?

+No. Application code, deployments, and Azure resources that host the application will still be available. DevOps Starter does not store any of these resources.

+### Will I lose the CI/CD pipeline that I created using DevOps Starter?

+No. You can still manage CI/CD pipelines in GitHub Actions or Azure Pipelines.

+- [SRE in Context](/training/modules/intro-to-site-reliability-engineering/3-sre-in-context)

+- [Key SRE Principles and Practices: virtuous cycles](/training/modules/intro-to-site-reliability-engineering/4-key-principles-1-virtuous-cycles)

+- [Key SRE Principles and Practices: The human side of SRE](/training/modules/intro-to-site-reliability-engineering/5-key-principles-2-human-side-of-sre)

+- [Getting Started with SRE](/training/modules/intro-to-site-reliability-engineering/6-getting-started)

+* [Learn module: Introduction to Azure DNS](/training/modules/intro-to-azure-dns).

+$forwardingrule = New-AzDnsForwardingRulesetForwardingRule -ResourceGroupName myresourcegroup -DnsForwardingRulesetName myruleset -Name "AzurePrivate" -DomainName "azure.contoso.com" -ForwardingRuleState "Enabled" -TargetDnsServer $targetDNS3

+* [Learn module: Introduction to Azure DNS](/training/modules/intro-to-azure-dns).

+* Conditional forwarding is supported using [Azure DNS Private Resolver](dns-private-resolver-overview.md). To enable resolution between Azure and on-premises networks, see [Name resolution for VMs and role instances](../virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances.md).

+* [Learn module: Introduction to Azure DNS](/training/modules/intro-to-azure-dns).

+* [Learn module: Introduction to Azure DNS](/training/modules/intro-to-azure-dns).

+* [Learn module: Introduction to Azure DNS](/training/modules/intro-to-azure-dns).

+ 1. Install Pre requisites Az PowerShell modules (https://learn.microsoft.com/powershell/azure/install-az-ps?view=azps-5.7.0)

+For more information on the Azure PowerShell module, see [Azure PowerShell documentation](/powershell/azure/).

+* [Learn module: Introduction to Azure DNS](/training/modules/intro-to-azure-dns).

+- [Microsoft Learn training](/training/)

+- [Microsoft Learn training](/training/)

+- [Microsoft Learn training](/training/)

+Students and faculty alike can get access to all the software download benefits through the Education Hub. The Education Hub is built within the Azure portal and it provides your students easy access to the entire catalog of software, as well as access to the entire [Microsoft Learn training](/training/) catalog.

+ Title: Microsoft Energy Data Services Preview csv parser ingestion workflow concept #Required; page title is displayed in search results. Include the brand.

+description: Learn how to use CSV parser ingestion. #Required; article description that is displayed in search results.

+# CSV parser ingestion concepts

+One of the simplest generic data formats that are supported by the Microsoft Energy Data Services Preview ingestion process is the "comma separated values" format, which is called a CSV format. The CSV format is processed through a CSV Parser DAG definition.

+CSV Parser DAG implements an ELT approach to data loading, that is, data is loaded after it's extracted. Customers can use CSV Parser DAG to load data that doesn't match the [OSDU&trade;](https://osduforum.org) canonical schema. Customers need to create and register a custom schema using the schema service matching the format of the CSV file.

+## What does CSV ingestion do?

+* **Schema validation** ΓÇô ensure CSV file conforms to the schema.

+* **Type conversion** ΓÇô ensure that the type of a field is as defined and converts it to the defined type if otherwise.

+* **ID generation** ΓÇô used to upload into storage service. It helps in scenarios where the ingestion failed half-way as ID generation logic is idempotent, one avoids duplicate data on platform.

+* **Reference handling** ΓÇô enable customers to refer to actual data on the platform and access it.

+* **Persistence** ΓÇô It persists each row after validations by calling storage service API. Once persisted, the data is available for consumption through search and storage service APIs.

+## CSV Parser ingestion functionality

+The CSV parser ingestion currently supports the following functionality as a one-step DAG:

+- CSV file is parsed as per the schema (one row in CSV = 1 record ingested into the data platform)

+- CSV file contents match the contents of the provided schema.

+ - **Success**: validate the schema vs. the header of the CSV file and the values of the first nrows. Use the schema for all downstream tasks to build the metadata.

+ - **Fail**: log the error(s) in the schema validation, proceed with ingestion if errors are non-breaking

+- Convert all characters to UTF8, and gracefully handle/replace characters that can't be converted to UTF8.

+- Unique data identity for an object in the Data Platform - CSV Ingestion generates Unique Identifier (ID) for each record by combining source, entity type and a base64 encoded string formed by concatenating natural key(s) in the data. In case the schema used for CSV Ingestion doesn't contain any natural keys storage service will generate random IDs for every record

+- Typecast to JSON-supported data types:

+ - **Number** - Typecast integers, doubles, floats, etc. as described in the schema to "number". Some common spatial formats, such as Degrees/Minutes/Seconds (DMS) or Easting/Northing should be typecast to "String." Special Handling of these string formats will be handled in the Spatial Data Handling Task.

+ - **Date** - Typecast dates as described in the schema to a date, doing a date format conversion toISO8601TZ format (for fully qualified dates). Some date fragments (such as years) can't be easily converted to this format and should be typecast to a number instead, or if textual date representations, for example, "July" should be typecast to string.

+ - **Others** - All other encountered attributes should be typecast as string.

+- Stores a batch of records in the context of a particular ingestion job. Fragments/outputs from the previous steps are collected into a batch, and formatted in a way that is compatible with the Storage Service with the appropriate additional information, such as the ACL's, Legal tags, etc.

+- Support frame of reference handling:

+ - **Unit** - converting declared frame of reference information into the appropriate persistable reference as per the Unit Service. This information is stored in the meta[] block.

+ - **CRS** - the CRS Frame of Reference (FoR) information should be included in the schema of the data, including the source CRS (either geographic or projected), and if projected, the CRS info and persistable reference (if provided in schema) information is stored in the meta[] block.

+- Creates relationships as declared in the source schema.

+- Supports publishing status of ingested/failed records on GSM article

+## CSV parser ingestion components

+* **File service** ΓÇô Facilitates management of files on data platform. Uploading, Secure discovery and downloading of files are capabilities provided by file service.

+* **Schema service** ΓÇô Facilitates management of Schemas on data platform. Creating, fetching and searching for schemas are capabilities provided by schema service.

+* **Storage Service** ΓÇô JSON object store that facilitates storage of metadata information for domain entities. Also raises storage events when records are saved using storage service.

+* **Unit Service** ΓÇô Facilitates management and conversion of Units

+* **Workflow service** ΓÇô Facilitates management of workflows on data platform. Wrapper over the workflow engine and abstract many technical nuances of the workflow engine from consumers.

+* **Airflow engine** ΓÇô Heart of the ingestion framework. Actual Workflow orchestrator.

+* **DAGs** ΓÇô Based on Direct Acyclic Graph concept, are workflows that are authored, orchestrated, managed and monitored by the workflow engine.

+## CSV ingestion components diagram

+## CSV ingestion sequence diagram

+## CSV parser ingestion workflow

+### Prerequisites

+* To trigger APIs, the user must have the below access and a valid authorization token

+ * Access to

+ * Access to Workflow service.

+ * Following is list of service level groups that you need access to register and execute DAG using workflow service.

+ * "service.workflow.creator"

+ * "service.workflow.viewer"

+ * "service.workflow.admin"

+### Steps to execute a DAG using Workflow Service

+* **Create schema** ΓÇô Definition of the kind of records that will be created as outcome of ingestion workflow. The schema is uploaded through the schema service. The schema needs to be registered using schema service.

+* **Uploading the file** ΓÇô Use file Service to upload a file. The file service provides a signed URL, which enables the customers to upload the data without credential requirements.

+* **Create Metadata record for the file** ΓÇô Use file service to create meta data. The meta data enables discovery of file and secure downloads. It also provides a mechanism to provide information associated with the file that is needed during the processing of the file.

+* The file ID created is provided to the CSV parser, which takes care of downloading the file, ingesting the file, and ingesting the records with the help of workflow service. The customers also need to register the workflow, the CSV parser DAG is already deployed in the Airflow.

+* **Trigger the Workflow service** ΓÇô To trigger the workflow, the customer needs to provide the file ID, the kind of the file and data partition ID. Once the workflow is triggered, the customer gets a run ID.

+Workflow service provides API to monitor the status of each workflow run. Once the csv parser run is completed, data is ingested into OSDU&trade; Data Platform, and can be searched through search service

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+Advance to the CSV parser tutorial and learn how to perform a CSV parser ingestion

+> [!div class="nextstepaction"]

+> [Tutorial: Sample steps to perform a CSV parser ingestion](tutorial-csv-ingestion.md)

+ Title: Domain data management services concepts #Required; page title is displayed in search results. Include the brand.

+description: Learn how to use Domain Data Management Services #Required; article description that is displayed in search results.

+# Domain data management service concepts

+**Domain Data Management Service (DDMS)** ΓÇô is a platform component that extends [OSDU&trade;](https://osduforum.org) core data platform with domain specific model and optimizations. DDMS is a mechanism of a platform extension that:

+* delivers optimized handling of data for each (non-overlapping) "domain."

+* single vertical discipline or business area, for example, Petrophysics, Geophysics, Seismic

+* a functional aspect of one or more vertical disciplines or business areas, for example, Earth Model

+* delivers high performance capabilities not supported by OSDU&trade; generic normal APIs.

+* can help achieve the extension of OSDU&trade; scope to new business areas.

+* may be developed in a distributed manner with separate resources/sponsors.

+OSDU&trade; Technical Standard defines the following types of OSDU&trade; application types:

+| Application Type | Description |

+| | -- |

+| OSDU&trade;&trade; Embedded Applications | An application developed and managed within the OSDU&trade; Open-Source community that is built on and deployed as part of the OSDU&trade; Data Platform distribution. |

+| ISV Extension Applications | An application, developed and managed in the marketplace that is NOT part of THE OSDU&trade; Data Platform distributions, and when selected is deployed within the OSDU&trade; Data Platform as add-ons |

+| ISV third Party Applications | An application, developed and managed in the marketplace that integrates with the OSDU&trade; Data Platform, and runs outside the OSDU&trade; Data Platform |

+| Characteristics | Embedded | Extension | Third Party |

+| -- | - | | |

+| Developed, managed, and deployed by | The OSDU&trade; Data Platform | ISV | ISV |

+| Software License | Apache 2 | ISV | ISV |

+| Mandatory as part of an OSDU&trade; distribution | Yes | No | No |

+| Replaceable | Yes, with preservation of behavior | Yes | Yes |

+| Architecture Compliance | The OSDU&trade; Standard | The OSDU&trade; Standard | ISV |

+| Examples | OS CRS <br /> Wellbore DDMS | ESRI CRS <br /> Petrel DS | Petrel |

+## Who did we build this for?

+**IT Developers** build systems to connect data to domain applications (internal and external ΓÇô for example, Petrel) which enables data managers to deliver projects to geoscientists. The DDMS suite on Microsoft Energy Data Services helps automate these workflows and eliminates time spent managing updates.

+**Geoscientists** use domain applications for key Exploration and Production workflows such as Seismic interpretation and Well tie analysis. While these users won't directly interact with the DDMS, their expectations for data performance and accessibility will drive requirements for the DDMS in the Foundation Tier. Azure will enable geoscientists to stream cross domain data instantly in OSDU&trade; compatible applications (for example, Petrel) connected to Microsoft Energy Data Services.

+**Data managers** spend a significant number of time fulfilling requests for data retrieval and delivery. The Seismic, Wellbore, and Petrel Data Services enable them to discover and manage data in one place while tracking version changes as derivatives are created.

+## Platform landscape

+Microsoft Energy Data Services is an OSDU&trade; compatible product, meaning that its landscape and release model are dependent on OSDU&trade;.

+Currently, OSDU&trade; certification and release process are not fully defined yet and this topic should be defined as a part of the Microsoft Energy Data Services Foundation Architecture.

+OSDU&trade; R3 M8 is the base for the scope of the Microsoft Energy Data Services Foundation Private Preview ΓÇô as a latest stable, tested version of the platform.

+## Learn more: OSDU&trade; DDMS community principles

+[OSDU&trade; community DDMS Overview](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU&trade;-(C)/Design-and-Implementation/Domain-&-Data-Management-Services#ddms-requirements) provides an extensive overview of DDMS motivation and community requirements from a user, technical, and business perspective. These principles are extended to Microsoft Energy Data Services.

+## DDMS requirements

+A DDMS meets the following requirements, further classified into capability, architectural, operational and openness/extensibility requirements:

+|**#** | **Description** | **Business rationale** | **Principle** |

+|||||

+| 1 | Data can be ingested with low friction | Need to seamlessly integrate with systems of record, to start with the industry standards | Capability |

+| 2 | New data is available in workflows with minimal latency | Deliver new data in context of the end-user workflow ΓÇô seamlessly and fast. | Capability |

+| 3 | Domain data and services are highly usable | The business anticipates a large set of use-cases where domain data is used in various workflows. Need to make the consumption simple and efficient | Capability |

+| 4 | Scalable performance for E&P workflows | E&P data has specific access requirements, way beyond standard cloud storage. Scalable E&P data requires E&P workflow experience and insights | Capability |

+| 5 | Data is available for visual analytics and discovery (Viz/BI) | Deliver minimum set of visualization capabilities on the data | Capability |

+| 6 | One source of truth for data | Drive towards reduction of duplication | Capability |

+| 7 | Data is secured, and access governed | Securely stored and managed | Architectural |

+| 8 | All data is preserved and immutable | Ability to associate data to milestones and have data/workflow traceable across the ecosystem | Architectural |

+| 9 | Data is globally identifiable | No risk of overwriting or creating non-unique relationships between data and activities | Architectural |

+| 10 | Data lineage is tracked | Required for auditability, re-creation of the workflow, and learning from work previously done | Architectural |

+| 11 | Data is discoverable | Possible to find and consume back ingested data | Architectural |

+| 12 | Provisioning | Efficient provisioning of the DDMS and auto integration with the Data Ecosystem | Operational |

+| 13 | Business Continuity | Deliver on industry expectation for business continuity (RPO, RTO, SLA) | Operational |

+| 14 | Cost | Cost efficient delivery of data | Operational |

+| 15 | Auditability | Deliver required forensics to support cyber security incident investigations | Operational |

+| 16 | Accessibility | Deliver technology | Operational |

+| 17 | Domain-Centric Data APIs | | Openness and Extensibility |

+| 18 | Workflow composability and customizations | | Openness and Extensibility |

+| 19 | Data-Centric Extensibility | | Openness and Extensibility |

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+Advance to the seismic ddms sdutil tutorial to learn how to use sdutil to load seismic data into seismic store.

+> [!div class="nextstepaction"]

+> [Tutorial: Seismic store sdutil](tutorial-seismic-ddms-sdutil.md)

+ Title: Microsoft Energy Data Services Preview entitlement concepts #Required; page title is displayed in search results. Include the brand.

+description: This article describes the various concepts regarding the entitlement services in Microsoft Energy Data Services Preview #Required; article description that is displayed in search results.

+# Entitlement service

+Access management is a critical function for any service or resource. Entitlement service helps you manage who has access to your Microsoft Energy Data Service instance, what they can do with it, and what services they have access to.

+## Groups

+The entitlements service of Microsoft Energy Data Services allows you to create groups, and an entitlement group defines permissions on services/data sources for your Microsoft Energy Data Services instance. Users added by you to that group obtain the associated permissions.

+The main motivation for entitlements service is data authorization, but the functionality enables three use cases:

+- **Data groups** used for data authorization (for example, data.welldb.viewers, data.welldb.owners)

+- **Service groups** used for service authorization (for example, service.storage.user, service.storage.admin)

+- **User groups** used for hierarchical grouping of user and service identities (for example, users.datalake.viewers, users.datalake.editors)

+## Users

+For each group, you can either add a user as an OWNER or a MEMBER. The only difference being if you're an OWNER of a group, then you can manage the members of that group.

+> [!NOTE]

+> Do not delete the OWNER of a group unless there is another OWNER to manage the users.

+## Group naming

+All group identifiers (emails) will be of form {groupType}.{serviceName|resourceName}.{permission}@{partition}.{domain}.com. A group naming convention has been adopted such that the group's name should start with the word "data." for data groups; "service." for service groups; and "users." for user groups. An exception is when a data partition is provisioned. When a data partition is created, so is a corresponding group: users (for example, for data partition `opendes`, the group `users@opendes.dataservices.energy` is created).

+## Permissions/roles

+The OSDU&trade; Data Ecosystem user groups provide an abstraction from permission and user management and--without a user creating their own groups--the following user groups exist by default:

+- **users.datalake.viewers**: viewer level authorization for OSDU Data Ecosystem services.

+- **users.datalake.editors**: editor level authorization for OSDU Data Ecosystem services and authorization to create the data using OSDU&trade; Data Ecosystem storage service.

+- **users.datalake.admins**: admin level authorization for OSDU Data Ecosystem services.

+A full list of all API endpoints for entitlements can be found in [OSDU entitlement service](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/release/0.15/docs/tutorial/Entitlements-Service.md#entitlement-service-api). We have provided few illustrations below. Depending on the resources you have, you need to use the entitlements service in different ways than what is shown below. [Entitlement permissions](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/release/0.15/docs/tutorial/Entitlements-Service.md#permissions) on the endpoints and the corresponding minimum level of permissions required.

+> [!NOTE]

+> The OSDU documentation refers to V1 endpoints, but the scripts noted in this documentation refers to V2 endpoints, which work and have been successfully validated

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+<!-- Add a context sentence for the following links -->

+> [!div class="nextstepaction"]

+> [How to manage users](how-to-manage-users.md)

+ Title: Microsoft Energy Data Services Preview - index and search workflow concepts #Required; page title is displayed in search results. Include the brand.

+description: Learn how to use indexing and search workflows #Required; article description that is displayed in search results.

+#Customer intent: As a developer, I want to understand indexing and search workflows so that I could search for ingested data in the platform.

+# Microsoft Energy Data Services Preview indexing and search workflows

+All data and associated metadata ingested into the platform are indexed to enable search. The metadata is accessible to ensure awareness even when the data isn't available.

+## Indexer Service

+The `Indexer Service` provides a mechanism for indexing documents that contain structured and unstructured data.

+> [!NOTE]

+> This service is not a public service and only meant to be called internally by other core platform services.

+

+### Indexing workflow

+The below diagram illustrates the Indexing workflow:

+When a customer loads data into the platform, the associated metadata is ingested using the `Storage service`. The `Storage service` provides a set of APIs to manage the entire metadata lifecycle such as ingestion (persistence), modification, deletion, versioning, retrieval, and data schema management. Each storage metadata record created by the `Storage service` contains a *kind* parameter that refers to an underlying *schema*. This schema determines the attributes that will be indexed by the `Indexer service`.

+

+When the `Storage service` creates a metadata record, it raises a *recordChangedMessages* event that is collected in the Azure Service Bus (message queue). The `Indexer queue` service pulls the message from the Azure Service Bus, performs basic validation and sends it over to the `Indexer service`. If there are any failures in sending the messages to the `Indexer service`, the `Indexer queue` service retries sending the message up to a maximum allowed configurable retry count. If the retry attempts fail, a negative acknowledgment is sent to the Azure Service Bus, which then archives the message.

+When the *recordChangedMessages* event is received by the `Indexer Service`, it fetches the required schemas from the schema cache or through the `Schema service` APIs. The `Indexer Service` then creates a new index within Elasticsearch (if not already present), and then sends a bulk query to create or update the records as needed. If the response from Elasticsearch is a failure response of type *service unavailable* or *request timed out*, then the `Indexer Service` creates *recordChangedMessages* for these failed record IDs and puts the message in the Azure Service Bus. These messages will again be pulled by the `Indexer Queue` service and will follow the same flow as before.

+

+For more information, see [Indexer service OSDU&trade; documentation](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/release/0.15/docs/tutorial/IndexerService.md) provides information on indexer service

+## Search workflow

+

+`Search service` provides a mechanism for discovering indexed metadata documents. The Search API supports full-text search on string fields, range queries on date, numeric, or string field, etc. along with geo-spatial searches.

+For a detailed tutorial on `Search service`, refer [Search service OSDU&trade; documentation](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/release/0.15/docs/tutorial/SearchService.md)

+

+## Reindex workflow

+Reindex API allows users to reindex a kind without reingesting the records via storage API. For detailed information, refer to

+[Reindex OSDU&trade; documentation](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/release/0.15/docs/tutorial/IndexerService.md#reindex)

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+<!-- Add a context sentence for the following links -->

+> [!div class="nextstepaction"]

+> [Domain data management service concepts](concepts-ddms.md)

+ Title: Microsoft Energy Data Services Preview manifest ingestion concepts #Required; page title is displayed in search results. Include the brand.

+description: This article describes manifest ingestion concepts #Required; article description that is displayed in search results.

+# Manifest-based ingestion concepts

+Manifest-based file ingestion provides end-users and systems a robust mechanism for loading metadata in Microsoft Energy Data Services Preview instance. A manifest is a JSON document that has a pre-determined structure for capturing entities that conform to the [OSDU&trade;](https://osduforum.org/) Well-known Schema (WKS) definitions.

+Manifest-based file ingestion doesn't understand the contents of the file or doesn't parse the file. It just creates a metadata record for the file and makes it searchable. It doesn't infer or does anything on top of the file.

+## Understanding the manifest

+The manifest schema has containers for the following entities

+* **ReferenceData** (*zero or more*) - A set of permissible values to be used by other (master or transaction) data fields. Examples include *Unit of Measure (feet)*, *Currency*, etc.

+* **MasterData** (*zero or more*) - A single source of basic business data used across multiple systems, applications, and/or process. Examples include *Wells* and *Wellbores*

+* **WorkProduct (WP)** (*one - must be present if loading WorkProductComponents*) - A session boundary or collection (project, study) encompasses a set of entities that need to be processed together. As an example, you can take the ingestion of one or more log collections.

+* **WorkProductComponents (WPC)** (*zero or more - must be present if loading datasets*) - A typed, smallest, independently usable unit of business data content transferred as part of a Work Product (a collection of things ingested together). Each Work Product Component (WPC) typically uses reference data, belongs to some master data, and maintains a reference to datasets. Example: *Well Logs, Faults, Documents*

+* **Datasets** (*zero or more - must be present if loading WorkProduct and WorkProductComponent records*) - Each Work Product Component (WPC) consists of one or more data containers known as datasets.

+## Manifest-based file ingestion workflow steps

+1. A manifest is submitted to the Workflow Service using the manifest ingestion workflow name (for example, "Osdu_ingest")

+2. Once the request is validated and the user authorization is complete, the workflow service will load and initiate the manifest ingestion workflow.

+3. The first step is to check the syntax of the manifest.

+ 1. Retrieve the **kind** property of the manifest

+ 2. Retrieve the **schema definition** from the Schema service for the manifest kind

+ 3. Validate that the manifest is syntactically correct according to the manifest schema definitions.

+ 4. For each Reference data, Master data, Work Product, Work Product Component, and Dataset, do the following activities:

+ 1. Retrieve the **kind** property.

+ 2. Retrieve the **schema definition** from the Schema service for the kind

+ 3. Validate that the entity is syntactically correct according to the schema definition and submits the manifest to the Workflow Service

+ 4. Validate that mandatory attributes exist in the manifest

+ 5. Validate that all property values follow the patterns defined in the schemas

+ 6. Validate that no extra properties are present in the manifest

+ 5. Any entity that doesn't pass the syntax check is rejected

+4. The content is checked for a series of validation rules

+ 1. Validation of referential integrity between Work Product Components and Datasets

+ 1. There are no orphan Datasets defined in the WP (each Dataset belongs to a WPC)

+ 2. Each Dataset defined in the WPC is described in the WP Dataset block

+ 3. Each WPC is linked to at least

+ 2. Validation that referenced parent data exists

+ 3. Validation that Dataset file paths aren't empty

+5. Process the contents into storage

+ 1. Write each valid entity into the data platform via the Storage API

+ 2. Capture the ID generated to update surrogate-keys where surrogate-keys are used

+6. Workflow exits

+## Manifest ingestion components

+* **Workflow Service** is a wrapper service on top of the Airflow workflow engine, which orchestrates the ingestion workflow. Airflow is the chosen workflow engine by the [OSDU&trade;](https://osduforum.org/) community to orchestrate and run ingestion workflows. Airflow isn't directly exposed to clients, instead its features are accessed through the workflow service.

+* **File Service** is used to upload files, file collections, and other types of source data to the data platform.

+* **Storage Service** is used to save the manifest records into the data platform.

+* **Airflow engine** is the workflow engine that executes DAGs (Directed Acyclic Graphs).

+* **Schema Service** stores schemas used in the data platform. Schemas are being referenced during the Manifest-based file ingestion.

+* **Entitlements Service** manages access groups. This service is used during the ingestion for verification of ingestion permissions. This service is also used during the metadata record retrieval for validation of "read" writes.

+* **Search Service** is used to perform referential integrity check during the manifest ingestion process.

+## Manifest ingestion workflow sequence

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+Advance to the manifest ingestion tutorial and learn how to perform a manifest-based file ingestion

+> [!div class="nextstepaction"]

+> [Tutorial: Sample steps to perform a manifest-based file ingestion](tutorial-manifest-ingestion.md)

+ Title: How to manage partitions

+description: This is a how-to article on managing data partitions using the Microsoft Energy Data Services Preview instance UI.

+# How to manage data partitions?

+The article describes how you can add data partitions to an existing Microsoft Energy Data Services (MEDS) instance. The concept of "data partitions" in MEDS is picked from [OSDU&trade;](https://osduforum.org/) where single deployment can contain multiple partitions.

+Each partition provides the highest level of data isolation within a single deployment. All access rights are governed at a partition level. Data is separated in a way that allows for the partition's life cycle and deployment to be handled independently. (See [Partition Service](https://community.opengroup.org/osdu/platform/home/-/issues/31) in OSDU&trade;)

+> [!NOTE]

+> You can create maximum five data partitions in one MEDS instance. Currently, in line with the data partition capabilities that are available in OSDU&trade;, you can only create data partitions but can't delete or rename data existing data partitions.

+## Create a data partition

+1. **Open the "Data Partitions" menu-item from left-panel of MEDS overview page.**

+ [](media/how-to-add-more-data-partitions/dynamic-data-partitions-discovery-meds-overview-page.png#lightbox)

+2. **Select "Create"**

+ The page shows a table of all data partitions in your MEDS instance with the status of the data partition next to it. Clicking "Create" option on the top opens a right-pane for next steps.

+ [](media/how-to-add-more-data-partitions/start-create-data-partition.png#lightbox)

+3. **Choose a name for your data partition**

+ Each data partition name needs to be - "1-10 characters long and be a combination of lowercase letters, numbers and hyphens only" The data partition name will be prepended with the name of the MEDS instance. Choose a name for your data partition and hit create. Soon as you hit create, the deployment of the underlying data partition resources such as Cosmos DB and Storage Accounts is started.

+ >[!NOTE]

+ >It generally takes 15-20 minutes to create a data partition.

+ [](media/how-to-add-more-data-partitions/create-data-partition-name-validation.png#lightbox)

+ If the deployment is successful, the status changes to "created successfully" with or without clicking "Refresh" on top.

+ [](media/how-to-add-more-data-partitions/create-progress.png#lightbox)

+## Delete a failed data partition

+The data-partition deployment triggered in the previous process might fail in some cases due to issues - quota limits reached, ARM template deployment transient issues, data seeding failures, and failure in connecting to underlying AKS clusters.

+The status of such data partitions shows as "Creation Failed". You can delete these deployments using the "delete" button that shows next to all failed data partition deployments. This deletion will clean up any records created in the backend. You can retry creating the data partitions later.

+[](media/how-to-add-more-data-partitions/delete-failed-instances.png#lightbox)

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+You can start loading data in your new data partitions.

+> [!div class="nextstepaction"]

+> [Load data using manifest ingestion](tutorial-manifest-ingestion.md)

+ Title: Microsoft Energy Data Services Preview - How to convert a segy to ovds file #Required; page title is displayed in search results. Include the brand.

+description: This article explains how to convert a SGY file to oVDS file format #Required; article description that is displayed in search results.

+# How to convert a SEG-Y file to oVDS?

+Seismic data stored in the industry standard SEG-Y format can be converted to Open VDS (oVDS) format for use in applications via the Seismic DMS.

+[OSDU&trade; SEG-Y to oVDS conversation](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/tree/release/0.15)

+## Prerequisites

+### Postman

+* Download and install [Postman](https://www.postman.com/) desktop app.

+* Import the [oVDS Conversions.postman_collection](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M9/Azure-M9/Services/DDMS/oVDS_Conversions.postman_collection.json) into Postman. All curl commands used below are added to this collection. Update your Environment file accordingly

+* Microsoft Energy Data Services Preview instance is created already

+* Clone the **sdutil** repo as shown below:

+ ```markdown

+ git clone https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil.git

+ git checkout azure/stable

+ ```

+## Step by step guide

+1. Check if VDS is registered with the workflow service or not:

+ ```markdown

+ curl --location --request GET '<url>/api/workflow/v1/workflow/'

+ --header 'Data-Partition-Id: <datapartition>'

+ --header 'Content-Type: application/json'

+ --header 'Authorization: Bearer {{TOKEN}}

+ ```

+ You should see VDS converter DAG in the list. IF NOT in the response list then REPORT the issue to Azure Team

+2. Open **sdutil** and edit the `config.yaml` at the root

+ Update `config` to:

+ ```yaml

+ seistore:

+ service: '{"azure": {"azureEnv":{"url": "<url>/seistore-svc/api/v3", "appkey": ""}}}'

+ url: '<url>/seistore-svc/api/v3'

+ cloud_provider: azure

+ env: glab

+ auth-mode: JWT Token

+ ssl_verify: false

+ auth_provider:

+ azure: '{

+ "provider": "azure",

+ "authorize_url": "https://login.microsoftonline.com/", "oauth_token_host_end": "/oauth2/v2.0/token",

+ "scope_end":"/.default openid profile offline_access",

+ "redirect_uri":"http://localhost:8080",

+ "login_grant_type": "refresh_token",

+ "refresh_token": "<RefreshToken acquired earlier>"

+ }'

+ azure:

+ empty: none

+ ```

+ > [!NOTE]

+ > See [Generate a refresh token](how-to-generate-refresh-token.md) on how to generate a refresh token. If you continue to follow other "how-to" documentation, you'll use this refresh token again. Once you've generated the token, store it in a place where you'll be able to access it in the future.

+3. Run **sdutil** to see if it's working fine. Follow the directions in [Setup and Usage for Azure env](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/tree/azure/stable#setup-and-usage-for-azure-env). Understand that depending on your OS and Python version, you may have to run `python3` command as opposed to `python`.

+ > [!NOTE]

+ > when running `python sdutil config init`, you don't need to enter anything when prompted with `Insert the azure (azureGlabEnv) application key:`.

+4. Upload the seismic file

+ ```markdown

+ python sdutil cp \source.segy sd://<datapartition>/<subproject>/destination.segy

+ ```

+5. Fetch the idtoken from sdutil for the uploaded file.

+ ```markdown

+ python sdutil auth idtoken

+ ```

+6. Trigger the DAG through `POSTMAN` or using the call below:

+ ```bash

+ curl --location --request POST '<url>/api/workflow/v1/workflow/<dag-name>/workflowRun' \

+ --header 'data-partition-id: <datapartition>' \

+ --header 'Content-Type: application/json' \

+ --header 'Authorization: Bearer {{TOKEN}}' \

+ --data-raw '{

+ "executionContext": {

+ "vds_url": "sd://<datapartition>/<subproject>",

+ "persistent_id": "<filename>",

+ "id_token": "<token>",

+ "segy_url": "sd://<datapartition>/<subproject>/<filename>.segy"

+ }

+ }'

+ ```

+7. Let the DAG run to complete state. You can check the status using the workflow status call

+8. Verify the converted files are present on the specified location in DAG Trigger or not

+ ```markdown

+ python sdutil ls sd://<datapartition>/<subproject>/

+ ```

+9. If you would like to download and inspect your VDS files, don't use the `cp` command as it will not work. The VDS conversion results in multiple files, therefore the `cp` command won't be able to download all of them in one command. Use either the [SEGYExport](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/tools/SEGYExport/README.html) or [VDSCopy](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/tools/VDSCopy/README.html) tool instead. These tools use a series of REST calls accessing a [naming scheme](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/connection.html) to retrieve information about all the resulting VDS files.

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+<!-- Add a context sentence for the following links -->

+> [!div class="nextstepaction"]

+> [How to convert a segy to zgy file](/how-to-convert-segy-to-zgy.md)

+ Title: Microsoft Energy Data Service - How to convert segy to zgy file #Required; page title is displayed in search results. Include the brand.

+description: This article describes how to convert a SEG-Y file to a ZGY file #Required; article description that is displayed in search results.

+# How to convert a SEG-Y file to ZGY?

+Seismic data stored in industry standard SEG-Y format can be converted to ZGY for use in applications such as Petrel via the Seismic DMS. See here for [ZGY Conversion FAQ's](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion#faq) and more background can be found in the OSDU&trade; community here: [SEG-Y to ZGY conversation](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/tree/azure/m10-master)

+## Prerequisites

+### Postman

+* Download and install [Postman](https://www.postman.com/) desktop app.

+* Import the [oZGY Conversions.postman_collection](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M9/Azure-M9/Services/DDMS/oZGY%20Conversions.postman_collection.json) into Postman. All curl commands used below are added to this collection. Update your Environment file accordingly

+* Microsoft Energy Data Services Preview instance is created already

+* Clone the **sdutil** repo as shown below:

+ ```markdown

+ git clone https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil.git

+ git checkout azure/stable

+ ```

+* The [jq command](https://stedolan.github.io/jq/download/), using your favorite tool on your favorite OS.

+

+## Step by Step guide

+1. The user needs to be part of the `users.datalake.admins` group and user needs to generate a valid refresh token. See [How to generate a refresh token](how-to-generate-refresh-token.md) for further instructions. If you continue to follow other "how-to" documentation, you'll use this refresh token again. Once you've generated the token, store it in a place where you'll be able to access it in the future. If it isn't present, add the group for the member ID. In this case, use the app ID you have been using for everything as the `user-email`.

+ > [!NOTE]

+ > `data-partition-id` should be in the format `<instance-name>-<data-partition-name>` in both the header and the url, and will be for any following command that requires `data-partition-id`.

+ ```bash

+ curl --location --request POST "<url>/api/entitlements/v2/groups/users.datalake.admins@<data-partition>.<domain>.com/members" \

+ --header 'Content-Type: application/json' \

+ --header 'data-partition-id: <data-partition>' \

+ --header 'Authorization: Bearer {{TOKEN}}' \

+ --data-raw '{

+ "email" : "<user-email>",

+ "role" : "MEMBER"

+ }

+ ```

+ You can also add the user to this group by using the entitlements API and assigning the required group ID. In order to check the entitlements groups for a user, perform the command [Get entitlements groups for a given user](how-to-manage-users.md#get-entitlements-groups-for-a-given-user). In order to get all the groups available, do the following command:

+ ```bash

+ curl --location --request GET "<url>/api/entitlements/v2/groups/" \

+ --header 'data-partition-id: <data-partition>' \

+ --header 'Authorization: Bearer {{TOKEN}}'

+ ```

+2. Check if ZGY is registered with the workflow service or not:

+ ```bash

+ curl --location --request GET '<url>/api/workflow/v1/workflow/' \

+ --header 'Data-Partition-Id: <data-partition>' \

+ --header 'Content-Type: application/json' \

+ --header 'Authorization: Bearer {{TOKEN}}'

+ ```

+ You should see ZGY converter DAG in the list. IF NOT in the response list then REPORT the issue to Azure Team

+3. Register Data partition to Seismic:

+ ```bash

+ curl --location --request POST '<url>/seistore-svc/api/v3/tenant/<data-partition>' \

+ --header 'Authorization: Bearer {{TOKEN}}' \

+ --header 'Content-Type: application/json' \

+ --data-raw '{

+ "esd": "{{data-partition}}.{{domain}}.com",

+ "gcpid": "{{data-partition}}",

+ "default_acl": "users.datalake.admins@{{data-partition}}.{{domain}}.com"}'

+ ```

+4. Create Legal tag

+ ```bash

+ curl --location --request POST '<url>/api/legal/v1/legaltags' \

+ --header 'Content-Type: application/json' \

+ --header 'data-partition-id: <data-partition>' \

+ --header 'Authorization: Bearer {{TOKEN}}' \

+ --data-raw '{

+ "name": "<tag-name>",

+ "description": "Legal Tag added for Seismic",

+ "properties": {

+ "contractId": "123456",

+ "countryOfOrigin": [

+ "US",

+ "CA"

+ ],

+ "dataType": "Public Domain Data",

+ "exportClassification": "EAR99",

+ "originator": "Schlumberger",

+ "personalData": "No Personal Data",

+ "securityClassification": "Private",

+ "expirationDate": "2025-12-25"

+ }

+ }'

+ ```

+5. Create Subproject. Use your previously created entitlements groups that you would like to add as ACLs (Access Control List) admins and viewers. If you haven't yet created entitlements groups, follow the directions as outlined in [How to manage users?](how-to-manage-users.md). If you would like to see what groups you have, use [Get entitlements groups for a given user](how-to-manage-users.md#get-entitlements-groups-for-a-given-user). Data access isolation achieved with this dedicated ACL (access control list) per object within a given data partition. You may have many subprojects within a data partition, so this command allows you to provide access to a specific subproject without providing access to an entire data partition. Data partition entitlements don't necessarily translate to the subprojects within it, so it's important to be explicit about the ACLs for each subproject, regardless of what data partition it is in.

+ > [!NOTE]

+ > Later in this tutorial, you'll need at least one `owner` and at least one `viewer`. These user groups will look like `data.default.owners` and `data.default.viewers`. Make sure to include one of each in your list of `acls` in the request below.

+ ```bash

+ curl --location --request POST '<url>/seistore-svc/api/v3/subproject/tenant/<data-partition>/subproject/<subproject>' \

+ --header 'Authorization: Bearer {{TOKEN}}' \

+ --header 'Content-Type: text/plain' \

+ --data-raw '{

+ "admin": "test@email",

+ "storage_class": "MULTI_REGIONAL",

+ "storage_location": "US",

+ "acls": {

+ "admins": [

+ "<user-group>@<data-partition>.<domain>.com",

+ "<user-group>@<data-partition>.<domain>.com"

+ ],

+ "owners": [

+ "<user-group>@<data-partition>.<domain>.com"

+ ],

+ "viewers": [

+ "<user-group>@<data-partition>.<domain>.com"

+ ]

+ }

+ }'

+ ```

+ The following request is an example of the create subproject request:

+ ```bash

+ curl --location --request POST 'https://<instance>.energy.azure.com/seistore-svc/api/v3/subproject/tenant/<instance>-<data-partition-name>/subproject/subproject1' \

+ --header 'Authorization: Bearer eyJ...' \

+ --header 'Content-Type: text/plain' \

+ --data-raw '{

+ "admin": "test@email",

+ "storage_class": "MULTI_REGIONAL",

+ "storage_location": "US",

+ "acls": {

+ "admins": [

+ "service.seistore.p4d.tenant01.subproject01.admin@slb.p4d.cloud.slb-ds.com",

+ "service.seistore.p4d.tenant01.subproject01.editor@slb.p4d.cloud.slb-ds.com"

+ ],

+ "owners": [

+ "data.default.owners@slb.p4d.cloud.slb-ds.com"

+ ],

+ "viewers": [

+ "service.seistore.p4d.tenant01.subproject01.viewer@slb.p4d.cloud.slb-ds.com"

+ ]

+ }

+ }'

+ ```

+6. Patch Subproject with the legal tag you created above:

+ ```bash

+ curl --location --request PATCH '<url>/seistore-svc/api/v3/subproject/tenant/<data-partition>/subproject/<subproject-name>' \

+ --header 'ltag: <Tag-name-above>' \

+ --header 'recursive: true' \

+ --header 'Authorization: Bearer {{TOKEN}}' \

+ --header 'Content-Type: text/plain' \

+ --data-raw '{

+ "admin": "test@email",

+ "storage_class": "MULTI_REGIONAL",

+ "storage_location": "US",

+ "acls": {

+ "admins": [

+ "<user-group>@<data-partition>.<domain>.com",

+ "<user-group>@<data-partition>.<domain>.com"

+ ],

+ "viewers": [

+ "<user-group>@<data-partition>.<domain>.com"

+ ]

+ }

+ }'

+ ```

+ > [!NOTE]

+ > Recall that the format of the legal tag will be prefixed with the Microsoft Energy Data Services instance name and data partition name, so it looks like `<instancename>`-`<datapartitionname>`-`<legaltagname>`.

+7. Open the [sdutil](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/tree/azure/stable) codebase and edit the `config.yaml` at the root. Update this config to:

+ ```yaml

+ seistore:

+ service: '{"azure": {"azureEnv":{"url": "<url>/seistore-svc/api/v3", "appkey": ""}}}'

+ url: '<url>/seistore-svc/api/v3'

+ cloud_provider: azure

+ env: glab

+ auth-mode: JWT Token

+ ssl_verify: false

+ auth_provider:

+ azure: '{

+ "provider": "azure",

+ "authorize_url": "https://login.microsoftonline.com/", "oauth_token_host_end": "/oauth2/v2.0/token",

+ "scope_end":"/.default openid profile offline_access",

+ "redirect_uri":"http://localhost:8080",

+ "login_grant_type": "refresh_token",

+ "refresh_token": "<RefreshToken acquired earlier>"

+ }'

+ azure:

+ empty: none

+ ```

+ > [!NOTE]

+ > See [How to generate a refresh token](how-to-generate-refresh-token.md). Once you've generated the token, store it in a place where you'll be able to access it in the future.

+8. Run the following commands using **sdutil** to see its working fine. Follow the directions in [Setup and Usage for Azure env](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/tree/azure/stable#setup-and-usage-for-azure-env). Understand that depending on your OS and Python version, you may have to run `python3` command as opposed to `python`. If you run into errors with these commands, refer to the [SDUTIL tutorial](/tutorials/tutorial-seismic-ddms-sdutil.md).

+ > [!NOTE]

+ > when running `python sdutil config init`, you don't need to enter anything when prompted with `Insert the azure (azureGlabEnv) application key:`.

+ ```bash

+ python sdutil config init

+ python sdutil auth login

+ python sdutil ls sd://<data-partition>/<subproject>/

+ ```

+9. Upload your seismic file to your Seismic Store. Here's an example with a SEGY-format file called `source.segy`:

+ ```bash

+ python sdutil cp source.segy sd://<data-partition>/<subproject>/destination.segy

+ ```

+ If you would like to use a test file we supply instead, download [this file](https://community.opengroup.org/osdu/platform/testing/-/tree/master/Postman%20Collection/40_CICD_OpenVDS) to your local machine then run the following command:

+ ```bash

+ python sdutil cp ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.segy sd://<data-partition>/<subproject>/destination.segy

+ ```

+ The sample records were meant to be similar to real-world data so a significant part of their content isn't directly related to conversion. This file is large and will take up about 1 GB of space.

+10. Create the manifest file (otherwise known as the records file)

+ ZGY conversion uses a manifest file that you'll upload to your storage account in order to run the conversion. This manifest file is created by using multiple JSON files and running a script. The JSON files for this process are stored [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/tree/master/doc/sample-records/volve). For more information on Volve, where the dataset definitions come from, visit [their website](https://www.equinor.com/en/what-we-do/digitalisation-in-our-dna/volve-field-data-village-download.html). Complete the following steps in order to create the manifest file:

+ * Clone the [repo](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/tree/master/) and navigate to the folder doc/sample-records/volve

+ * Edit the values in the `prepare-records.sh` bash script:

+ * `DATA_PARTITION_ID=<your-partition-id>`

+ * `ACL_OWNER=data.default.owners@<your-partition-id>.<your-tenant>.com`

+ * `ACL_VIEWER=data.default.viewers@<your-partition-id>.<your-tenant>.com`

+ * `LEGAL_TAG=<legal-tag-created-above>`

+ > [!NOTE]

+ > Recall that the format of the legal tag will be prefixed with the Microsoft Energy Data Services instance name and data partition name, so it looks like `<instancename>`-`<datapartitionname>`-`<legaltagname>`.

+ * The output will be a JSON array with all objects and will be saved in the `all_records.json` file.

+ * Save the `filecollection_segy_id` and the `work_product_id` values in that JSON file to use in the conversion step. That way the converter knows where to look for this contents of your `all_records.json`.

+11. Insert the contents of your `all_records.json` file in storage for work-product, seismic trace data, seismic grid, and file collection (that is, copy and paste the contents of that file to the `--data-raw` field in the following command):

+ ```bash

+ curl --location --request PUT '<url>/api/storage/v2/records' \

+ --header 'Content-Type: application/json' \

+ --header 'data-partition-id: <data-partition>' \

+ --header 'Authorization: Bearer {{TOKEN}}' \

+ --data-raw '[

+ {

+ ...

+ "kind": "osdu:wks:work-product--WorkProduct:1.0.0",

+ ...

+ },

+ {

+ ...

+ "kind": "osdu:wks:work-product-component--SeismicTraceData:1.0.0"

+ ...

+ },

+ {

+ ...

+ "kind": "osdu:wks:work-product-component--SeismicBinGrid:1.0.0",

+ ...

+ },

+ {

+ ...

+ "kind": "osdu:wks:dataset--FileCollection.SEGY:1.0.0",

+ ...

+ }

+ ]

+ '

+ ```

+12. Trigger the ZGY Conversion DAG to convert your data using the values you had saved above. Your call will look like this:

+ ```bash

+ curl --location --request POST '<url>/api/workflow/v1/workflow/<dag-name>/workflowRun' \

+ --header 'data-partition-id: <data-partition>' \

+ --header 'Content-Type: application/json' \

+ --header 'Authorization: Bearer {{TOKEN}}' \

+ --data-raw '{

+ "executionContext": {

+ "data_partition_id": <data-partition>,

+ "sd_svc_api_key": "test-sd-svc",

+ "storage_svc_api_key": "test-storage-svc",

+ "filecollection_segy_id": "<data-partition>:dataset--FileCollection.SEGY:<guid>",

+ "work_product_id": "<data-partition>:work-product--WorkProduct:<guid>"

+ }

+ }'

+ ```

+13. Let the DAG run to the `succeeded` state. You can check the status using the workflow status call. You'll get run ID in the response of the above call

+ ```bash

+ curl --location --request GET '<url>/api/workflow/v1/workflow/<dag-name>/workflowRun/<run-id>' \

+ --header 'Data-Partition-Id: <data-partition>' \

+ --header 'Content-Type: application/json' \

+ --header 'Authorization: Bearer {{TOKEN}}'

+ ```

+14. You can see if the converted file is present using the following command:

+ ```bash

+ python sdutil ls sd://<data-partition>/<subproject>

+ ```

+15. You can download and inspect the file using the [sdutil](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/tree/azure/stable) `cp` command:

+ ```bash

+ python sdutil cp sd://<data-partition>/<subproject>/<filename.zgy> <local/destination/path>

+ ```

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+<!-- Add a context sentence for the following links -->

+> [!div class="nextstepaction"]

+> [How to convert segy to ovds](/how-to-convert-segy-to-ovds.md)

+ Title: How to generate a refresh token for Microsoft Energy Data Service #Required; page title is displayed in search results. Include the brand.

+description: This article describes how to generate a refresh token #Required; article description that is displayed in search results.

+# OAuth 2.0 authorization

+The following are the basic steps to use the OAuth 2.0 authorization code grant flow to get a refresh token from the Microsoft identity platform endpoint:

+ 1. Register your app with Azure AD.

+ 2. Get authorization.

+ 3. Get a refresh token.

+

+## 1. Register your app with Azure AD

+To use the Microsoft Energy Data Services Preview platform endpoint, you must register your app using the [Azure app registration portal](https://go.microsoft.com/fwlink/?linkid=2083908). You can use either a Microsoft account or a work or school account to register an app.

+To configure an app to use the OAuth 2.0 authorization code grant flow, save the following values when registering the app:

+- The `Directory (tenant) ID` that will be used in place of `{Tenant ID}`

+- The `application (client) ID` assigned by the app registration portal, which will be used instead of `client_id`.

+- A `client (application) secret`, either a password or a public/private key pair (certificate). The client secret isn't required for native apps. This secret will be used instead of `{AppReg Secret}` later.

+- A `redirect URI (or reply URL)` for your app to receive responses from Azure AD.

+

+> [!NOTE]

+> If there's no redirect URIs specified, add a platform, select "Web", then add `http://localhost:8080`, and select save.

+For steps on how to configure an app in the Azure portal, see [Register your app](/azure/active-directory/develop/quickstart-register-app#register-an-application).

+## 2. Get authorization

+The first step to getting an access token for many OpenID Connect (OIDC) and OAuth 2.0 flows is to redirect the user to the Microsoft identity platform /authorize endpoint. Azure AD will sign the user in and request their consent for the permissions your app requests. In the authorization code grant flow, after consent is obtained, Azure AD will return an `authorization_code` to your app that it can redeem at the Microsoft identity platform /token endpoint for an access token.

+### Authorization request

+The authorization code flow begins with the client directing the user to the `/authorize` endpoint. This step is the interactive part of the flow, where the user takes action.

+The following shows an example of an authorization request:

+```bash

+ https://login.microsoftonline.com/{Tenant ID}/oauth2/v2.0/authorize?client_id={AppReg ID}

+ &response_type=code

+ &redirect_uri=http%3a%2f%2flocalhost%3a8080

+ &response_mode=query

+ &scope={AppReg ID}%2f.default&state=12345&sso_reload=true

+```

+| Parameter | Required? | Description |

+| | | |

+|`{Tenant ID}`|Required|Name of your Azure AD tenant|

+| client_id |Required |The application ID assigned to your app in the [Azure portal](https://portal.azure.com). |

+| response_type |Required |The response type, which must include `code` for the authorization code flow. You can receive an ID token if you include it in the response type, such as `code+id_token`, and in this case, the scope needs to include `openid`.|

+| redirect_uri |Required |The redirect URI of your app, where authentication responses are sent and received by your app. It must exactly match one of the redirect URIs that you registered in the portal, except that it must be URL-encoded. |

+| scope |Required |A space-separated list of scopes. The `openid` scope indicates a permission to sign in the user and get data about the user in the form of ID tokens. The `offline_access` scope is optional for web applications. It indicates that your application will need a *refresh token* for extended access to resources. The client-id indicates the token issued are intended for use by Azure AD B2C registered client. The `https://{tenant-name}/{app-id-uri}/{scope}` indicates a permission to protected resources, such as a web API. |

+| response_mode |Recommended |The method that you use to send the resulting authorization code back to your app. It can be `query`, `form_post`, or `fragment`. |

+| state |Recommended |A value included in the request that can be a string of any content that you want to use. Usually, a randomly generated unique value is used, to prevent cross-site request forgery attacks. The state also is used to encode information about the user's state in the app before the authentication request occurred. For example, the page the user was on, or the user flow that was being executed. |

+### Authorization response

+In the response, you'll get an `authorization code` in the URL bar.

+```bash

+http://localhost:8080/?code=0.BRoAv4j5cvGGr0...au78f&state=12345&session....

+```

+The browser will redirect to `http://localhost:8080/?code={authorization code}&state=...` upon successful authentication.

+> [!NOTE]

+> The browser may say that the site can't be reached, but it should still have the authorization code in the URL bar.

+|Parameter| Description|

+| | |

+|code|The authorization_code that the app requested. The app can use the authorization code to request an access token for the target resource. Authorization_codes are short lived, typically they expire after about 10 minutes.|

+|state|If a state parameter is included in the request, the same value should appear in the response. The app should verify that the state values in the request and response are identical. This check helps to detect [Cross-Site Request Forgery (CSRF) attacks](https://tools.ietf.org/html/rfc6749#section-10.12) against the client.|

+|session_state|A unique value that identifies the current user session. This value is a GUID, but should be treated as an opaque value that is passed without examination.|

+Copy the code between `code=` and `&state`.

+> [!WARNING]

+> Running the URL in Postman won't work as it requires extra configuration for token retrieval.

+## 3. Get a refresh token

+Your app uses the authorization code received in the previous step to request an access token by sending a POST request to the `/token` endpoint.

+### Sample request

+```bash

+ curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d 'client_id={AppReg ID}

+ &scope={AppReg ID}%2f.default+openid+profile+offline_access

+ &code={authorization code}

+ &redirect_uri=http%3A%2F%2Flocalhost%3a8080

+ &grant_type=authorization_code

+ &client_secret={AppReg Secret}' 'https://login.microsoftonline.com/{Tenant ID}/oauth2/v2.0/token'

+```

+|Parameter |Required |Description |

+||||

+|tenant | Required | The {Tenant ID} value in the path of the request can be used to control who can sign into the application.|

+|client_id | Required | The application ID assigned to your app upon registration |

+|scope | Required | A space-separated list of scopes. The scopes that your app requests in this leg must be equivalent to or a subset of the scopes that it requested in the first (authorization) leg. If the scopes specified in this request span multiple resource server, then the v2.0 endpoint will return a token for the resource specified in the first scope. |

+|code |Required |The authorization_code that you acquired in the first leg of the flow. |

+|redirect_uri | Required |The same redirect_uri value that was used to acquire the authorization_code. |

+|grant_type | Required | Must be authorization_code for the authorization code flow. |

+|client_secret | Required | The client secret that you created in the app registration portal for your app. It shouldn't be used in a native app, because client_secrets can't be reliably stored on devices. It's required for web apps and web APIs, which have the ability to store the client_secret securely on the server side.|

+### Sample response

+```bash

+{

+ "token_type": "Bearer",

+ "scope": "User.Read profile openid email",

+ "expires_in": 4557,

+ "access_token": "eyJ0eXAiOiJKV1QiLCJub25jZSI6IkJuUXdJd0ZFc...",

+ "refresh_token": "0.ARoAv4j5cvGGr0GRqy180BHbR8lB8cvIWGtHpawGN..."

+}

+```

+|Parameter | Description |

+|||

+|token_type |Indicates the token type value. The only type that Azure AD supports is Bearer. |

+|scope |A space separated list of the Microsoft Graph permissions that the access_token is valid for. |

+|expires_in |How long the access token is valid (in seconds). |

+|access_token |The requested access token. Your app can use this token to call Microsoft Graph. |

+|refresh_token |An OAuth 2.0 refresh token. Your app can use this token to acquire extra access tokens after the current access token expires. Refresh tokens are long-lived, and can be used to retain access to resources for extended periods of time.|

+For more information, see [Generate refresh tokens](/graph/auth-v2-user#2-get-authorization).

+## Alternative options

+If you're struggling with getting a proper authorization token, follow the steps in [OSDU&trade; auth app](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/release/0.15/tools/rest/osduauth) to locally run a static webpage that generates the refresh token for you. Once it's running, fill in the correct values in the UI of the static webpage (they may be filled in with the wrong values to start). Use the UI to generate a refresh token.

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+<!-- Add a context sentence for the following links -->

+> [!div class="nextstepaction"]

+> [How to convert segy to ovds](how-to-convert-segy-to-zgy.md)

+ Title: Integrate airflow logs with Azure Monitor - Microsoft Energy Data Services Preview

+description: This is a how-to article on how to start collecting Airflow Task logs in Azure Monitor, archiving them to a storage account, and querying them in Log Analytics workspace.

+# Integrate airflow logs with Azure Monitor

+This article describes how you can start collecting Airflow Logs for your Microsoft Energy Data Services instances into Azure Monitor. This integration feature helps you debug Airflow DAG run failures.

+## Prerequisites

+* An existing **Log Analytics Workspace**.

+ This workspace will be used to query the Airflow logs using the Kusto Query Language (KQL) query editor in the Log Analytics Workspace. Useful Resource: [Create a log analytics workspace in Azure portal](../azure-monitor/logs/quick-create-workspace.md).

+* An existing **storage account**:

+ It will be used to store JSON dumps of Airflow logs. The storage account doesnΓÇÖt have to be in the same subscription as your Log Analytics workspace.

+## Enabling diagnostic settings to collect logs in a storage account

+Every Microsoft Energy Data Services instance comes inbuilt with an Azure Data Factory-managed Airflow instance. We collect Airflow logs for internal troubleshooting and debugging purposes. Airflow logs can be integrated with Azure Monitor in the following ways:

+* Storage account

+* Log Analytics workspace

+To access logs via any of the above two options, you need to create a Diagnostic Setting. Each Diagnostic Setting has three basic parts:

+| Title | Description |

+|-|-|

+| Name | This is the name of the diagnostic log. Ensure a unique name is set for each log. |

+| Categories | Category of logs to send to each of the destinations. The set of categories will vary for each Azure service. Visit: [Supported Resource Log Categories](../azure-monitor/essentials/resource-logs-categories.md) |

+| Destinations | One or more destinations to send the logs. All Azure services share the same set of possible destinations. Each diagnostic setting can define one or more destinations but no more than one destination of a particular type. It should be a storage account, an Event Hubs namespace or an event hub. |

+Follow the following steps to set up Diagnostic Settings:

+1. Open Microsoft Energy Data Services' "**Overview**" page

+1. Select "**Diagnostic Settings**" from the left panel

+ [](media/how-to-integrate-airflow-logs-with-azure-monitor/azure-monitor-diagnostic-settings-overview-page.png#lightbox)

+1. Select "**Add diagnostic setting**"

+1. Select "**Airflow Task Logs**" under Logs

+1. Select "**Archive to a storage account**"

+ [](media/how-to-integrate-airflow-logs-with-azure-monitor/creating-diagnostic-setting-destination-storage-account.png#lightbox)

+6. Verify the subscription and the storage account to which you want to archive the logs.

+## Navigate storage account to download Airflow logs

+After a diagnostic setting is created for archiving Airflow task logs into a storage account, you can navigate to the storage account **overview** page. You can then use the "Storage Browser" on the left panel to find the right JSON file that you want to investigate. Browsing through different directories is intuitive as you move from a year to a month to a day.

+1. Navigate through **Containers**, available on the left panel.

+ [](media/how-to-integrate-airflow-logs-with-azure-monitor/storage-account-containers-page-showing-collected-logs-explorer.png#lightbox)

+2. Open the information pane on the right. It contains a "download" button to save the log file locally.

+1. Downloaded logs can be analyzed in any editor.

+## Enabling diagnostic settings to integrate logs with Log Analytics Workspace

+You can integrate Airflow logs with Log Analytics Workspace by using **Diagnostic Settings** under the left panel of your Microsoft Energy Data Services instance overview page.

+[](media/how-to-integrate-airflow-logs-with-azure-monitor/creating-diagnostic-setting-choosing-destination-retention.png#lightbox)

+## Working with the integrated Airflow Logs in Log Analytics Workspace

+Data is retrieved from a Log Analytics Workspace using a query written in Kusto Query Language (KQL). A set of precreated queries is available for many Azure services (not available for Airflow at the moment) so that you don't require knowledge of KQL to get started.

+[](media/how-to-integrate-airflow-logs-with-azure-monitor/azure-monitor-log-analytics-page-viewing-collected-logs.png#lightbox)

+1. Select Logs from your resource's menu. Log Analytics opens with the **Queries** window that includes prebuilt queries for your resource type.

+2. Browse through the available queries. Identify the one to run and select Run. The query is added to the query window and the results are returned.

+## Next steps

+Now that you're collecting resource logs, create a log query alert to be proactively notified when interesting data is identified in your log data.

+> [!div class="nextstepaction"]

+> [Create a log query alert for an Azure resource](../azure-monitor/alerts/tutorial-log-alert.md)

+ Title: Integrate elastic logs with Azure Monitor - Microsoft Energy Data Services Preview

+description: This is a how-to article on how to start collecting ElasticSearch logs in Azure Monitor, archiving them to a storage account, and querying them in Log Analytics workspace.

+# Integrate elastic logs with Azure Monitor

+This article describes how you can start collecting Elasticsearch logs for your Microsoft Energy Data Services instances in Azure Monitor. This integration feature is developed to help you debug Elasticsearch related issues inside Azure Monitor.

+## Prerequisites

+- You need to have a Log Analytics workspace. It will be used to query the Elasticsearch logs dataset using the Kusto Query Language (KQL) query editor in the Log Analytics workspace. Useful Resource: [Create a log Analytics workspace in Azure portal](../azure-monitor/logs/quick-create-workspace.md)

+- You need to have a storage account. It will be used to store JSON dumps of Elasticsearch & Elasticsearch Operator logs. The storage account doesnΓÇÖt have to be in the same subscription as your Log Analytics workspace.

+## Enabling Diagnostic Settings to collect logs in a storage account & a Log Analytics workspace

+Every Microsoft Energy Data Services instance comes inbuilt with a managed Elasticsearch service. We collect Elasticsearch logs for internal troubleshooting and debugging purposes. You can get access to these logs by integrating Elasticsearch logs with Azure Monitor.

+Each diagnostic setting has three basic parts:

+| Title | Description |

+|-|-|

+| Name | This is the name of the diagnostic log. Ensure a unique name is set for each log. |

+| Categories | Category of logs to send to each of the destinations. The set of categories will vary for each Azure service. Visit: [Supported Resource Log Categories](../azure-monitor/essentials/resource-logs-categories.md) |

+| Destinations | One or more destinations to send the logs. All Azure services share the same set of possible destinations. Each diagnostic setting can define one or more destinations but no more than one destination of a particular type. It should be a storage account, an Event Hubs namespace or an event hub. |

+We support two destinations for your Elasticsearch logs from Microsoft Energy Data Services instance:

+* Storage account

+* Log Analytics workspace

+## Steps to enable diagnostic setting to collect Elasticsearch logs

+1. Open *Microsoft Energy Data Services* overview page

+1. Select *Diagnostic Settings* from the left panel

+

+ [](media/how-to-integrate-elastic-logs-with-azure-monitor/diagnostic-setting-overview-page.png#lightbox)

+1. Select *Add diagnostic setting*.

+1. Select *Elasticsearch logs* and *Elasticsearch Operator logs* under Log categories

+

+1. Select *Send to a Log Analytics workspace*

+1. Choose Subscription and the Log Analytics workspace Name. You would have created it already as a prerequisite.

+

+ [](media/how-to-integrate-elastic-logs-with-azure-monitor/diagnostic-setting-log-analytics-workspace.png#lightbox)

+1. Select *Archive to storage account*

+1. Choose Subscription and storage account Name. You would have created it already as a prerequisite.

+ [](media/how-to-integrate-elastic-logs-with-azure-monitor/diagnostic-setting-archive-storage-account.png#lightbox)

+1. Select *Save*.

+Go back to the Diagnostic Settings page. You would now see a new diagnostic setting created along with the names of the destination storage account and Log Analytics workspace you chose for this setting.

+

+[](media/how-to-integrate-elastic-logs-with-azure-monitor/diagnostic-setting-created-page.png#lightbox)

+## View Elasticsearch logs in Log Analytics workspace or download them as JSON files using storage account

+### How to view & query logs in Log Analytics workspace

+The editor in Log Analytics workspace support Kusto (KQL) queries through which you can easily perform complicated queries to extract interesting logs data from the Elasticsearch service running in your Microsoft Energy Data Services instance.

+

+

+* Run queries and see Elasticsearch logs in the Log Analytics workspace.

+ [](media/how-to-integrate-elastic-logs-with-azure-monitor/view-elasticsearch-logs.png#lightbox)

+* Run queries and see Elasticsearch Operator logs in the Log Analytics workspace.

+ [](media/how-to-integrate-elastic-logs-with-azure-monitor/view-elasticsearch-operator-logs.png#lightbox)

+### How to download logs as JSON files from storage account

+* The *Containers* menu option in the left panel of your storage account's overview page allows you to browse through the various directories that neatly store your log files.

+

+ [](media/how-to-integrate-elastic-logs-with-azure-monitor/storage-account-containers-page.png#lightbox)

+

+* Logs are organized into different folders. Drill down by month, date and time.

+ [](media/how-to-integrate-elastic-logs-with-azure-monitor/storage-account-log-file.png#lightbox)

+

+* Select any JSON file in your containers to view other options.

+ [](media/how-to-integrate-elastic-logs-with-azure-monitor/storage-account-download-log-file-json.png#lightbox)

+* Select *Download* option to download the JSON file. Open it in a code editor of your choice.

+

+ [](media/how-to-integrate-elastic-logs-with-azure-monitor/logs-downloaded-opened-editor.png#lightbox)

+

+## Next steps

+After collecting resource logs as explained in this article, there are more capabilities you can explore.

+* Create a log query alert to be proactively notified when interesting data is identified in your log data.

+ [Create a log query alert for an Azure resource](../azure-monitor/alerts/tutorial-log-alert.md)

+* Start collecting logs from other sources such as Airflow in your Microsoft Energy Data Services instance.

+ [How to Integrate Airflow logs with Azure Monitor](how-to-integrate-airflow-logs-with-azure-monitor.md)

+ Title: How to manage legal tags in Microsoft Energy Data Services Preview #Required; page title is displayed in search results. Include the brand.

+description: This article describes how to manage legal tags in Microsoft Energy Data Services Preview #Required; article description that is displayed in search results.

+# How to manage legal tags?

+A Legal tag is the entity that represents the legal status of data in the Microsoft Energy Data Services Preview instance. Legal tag is a collection of properties that governs how data can be ingested and consumed. A legal tag is necessarily required for data to be [ingested](concepts-csv-parser-ingestion.md) into your Microsoft Energy Data Services Preview instance. It's also required for the [consumption](concepts-index-and-search.md) of the data from your Microsoft Energy Data Services Preview instance. Legal tags are defined at a data partition level individually.

+While in Microsoft Energy Data Services Preview instance, [entitlement service](concepts-entitlements.md) defines access to data for a given user(s), legal tag defines the overall access to the data across users. A user may have access to manage the data within a data partition however, they may not be able to do so, until certain legal requirements are fulfilled.

+## Create a legal tag

+Run the below curl command in Azure Cloud Bash to create a legal tag for a given data partition of your Microsoft Energy Data Services Preview instance.

+```bash

+ curl --location --request POST 'https://<URI>/api/legal/v1/legaltags' \

+ --header 'data-partition-id: <data-partition-id>' \

+ --header 'Authorization: Bearer <access_token>' \

+ --header 'Content-Type: application/json' \

+ --data-raw '{

+ "name": "<legal-tag-name>",

+ "description": "<legal-tag-description>",

+ "properties": {

+ "contractId": "<contract-id>",

+ "countryOfOrigin": ["<country-of-origin>"],

+ "dataType": "<data-type>",

+ "expirationDate": "<expiration-ID>",

+ "exportClassification": "<export-classification>",

+ "originator": "<originator>",

+ "personalData": "<personal-data>",

+ "securityClassification": "Public"

+ }

+ }'

+```

+### Sample request

+```bash

+ curl --location --request POST 'https://<instance>.energy.azure.com/api/legal/v1/legaltags' \

+ --header 'data-partition-id: <instance>-<data-partition-name>' \

+ --header 'Authorization: Bearer <access_token>' \

+ --header 'Content-Type: application/json' \

+ --data-raw '{

+ "name": "<instance>-<data-partition-name>-legal-tag",

+ "description": "Microsoft Energy Data Services Preview Legal Tag",

+ "properties": {

+ "contractId": "A1234",

+ "countryOfOrigin": ["US"],

+ "dataType": "Public Domain Data",

+ "expirationDate": "2099-01-25",

+ "exportClassification": "EAR99",

+ "originator": "MyCompany",

+ "personalData": "No Personal Data",

+ "securityClassification": "Public"

+ }

+ }'

+```

+### Sample response

+```JSON

+ {

+ "name": "<instance>-<data-partition-name>-legal-tag",

+ "description": "Microsoft Energy Data Services Preview Legal Tag",

+ "properties": {

+ "countryOfOrigin": [

+ "US"

+ ],

+ "contractId": "A1234",

+ "expirationDate": "2099-01-25",

+ "originator": "MyCompany",

+ "dataType": "Public Domain Data",

+ "securityClassification": "Public",

+ "personalData": "No Personal Data",

+ "exportClassification": "EAR99"

+ }

+}

+```

+The country of origin should follow [ISO Alpha2 format](https://www.nationsonline.org/oneworld/country_code_list.htm).

+> [!NOTE]

+> Create Legal Tag api, internally appends data-partition-id to legal tag name if it isn't already present. For instance, if request has name as: ```legal-tag```, then the create legal tag name would be ```<instancename>-<data-partition-id>-legal-tag```

+```bash

+ curl --location --request POST 'https://<instance>.energy.azure.com/api/legal/v1/legaltags' \

+ --header 'data-partition-id: <instance>-<data-partition-name>' \

+ --header 'Authorization: Bearer <access_token>' \

+ --header 'Content-Type: application/json' \

+ --data-raw '{

+ "name": "legal-tag",

+ "description": "Microsoft Energy Data Services Preview Legal Tag",

+ "properties": {

+ "contractId": "A1234",

+ "countryOfOrigin": ["US"],

+ "dataType": "Public Domain Data",

+ "expirationDate": "2099-01-25",

+ "exportClassification": "EAR99",

+ "originator": "MyCompany",

+ "personalData": "No Personal Data",

+ "securityClassification": "Public"

+ }

+ }'

+```

+The sample response will have data-partition-id appended to the legal tag name and sample response will be:

+```JSON

+ {

+ "name": "<instance>-<data-partition-name>-legal-tag",

+ "description": "Microsoft Energy Data Services Preview Legal Tag",

+ "properties": {

+ "countryOfOrigin": [

+ "US"

+ ],

+ "contractId": "A1234",

+ "expirationDate": "2099-01-25",

+ "originator": "MyCompany",

+ "dataType": "Public Domain Data",

+ "securityClassification": "Public",

+ "personalData": "No Personal Data",

+ "exportClassification": "EAR99"

+ }

+}

+```

+## Get a legal tag

+Run the below curl command in Azure Cloud Bash to get the legal tag associated with a data partition of your Microsoft Energy Data Services Preview instance.

+

+```bash

+ curl --location --request GET 'https://<URI>/api/legal/v1/legaltags/<legal-tag-name>' \

+ --header 'data-partition-id: <data-partition-id>' \

+ --header 'Authorization: Bearer <access_token>'

+```

+### Sample request

+```bash

+ curl --location --request GET 'https://<instance>.energy.azure.com/api/legal/v1/legaltags/<instance>-<data-partition-name>-legal-tag' \

+ --header 'data-partition-id: <instance>-<data-partition-name>' \

+ --header 'Authorization: Bearer <access_token>'

+```

+### Sample response

+```JSON

+ {

+ "name": "<instance>-<data-partition-name>-legal-tag",

+ "description": "Microsoft Energy Data Services Preview Legal Tag",

+ "properties": {

+ "countryOfOrigin": [

+ "US"

+ ],

+ "contractId": "A1234",

+ "expirationDate": "2099-01-25",

+ "originator": "MyCompany",

+ "dataType": "Public Domain Data",

+ "securityClassification": "Public",

+ "personalData": "No Personal Data",

+ "exportClassification": "EAR99"

+ }

+ }

+```

+## Next steps

+<!-- Add a context sentence for the following links -->

+> [!div class="nextstepaction"]

+> [How to add more data partitions](how-to-add-more-data-partitions.md)

+ Title: How to manage users in Microsoft Energy Data Services Preview #Required; page title is displayed in search results. Include the brand.

+description: This article describes how to manage users in Microsoft Energy Data Services Preview #Required; article description that is displayed in search results.

+# How to manage users?

+This article describes how to manage users in Microsoft Energy Data Services Preview. It uses the [entitlements API](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/tree/master/) and acts as a group-based authorization system for data partitions within Microsoft Energy Data Service instance. For more information about Microsoft Energy Data Services Preview entitlements, see [entitlement services](concepts-entitlements.md).

+## Prerequisites

+Create a Microsoft Energy Data Services Preview instance using guide at [How to create Microsoft Energy Data Services Preview instance](quickstart-create-microsoft-energy-data-services-instance.md).

+Keep the following values handy. These values will be used to:

+* Generate the access token, which you'll need to make valid calls to the Entitlements API of your Microsoft Energy Data Services Preview instance

+* Pass as parameters for different user management requests to the Entitlements API.

+#### Find `tenant-id`

+Navigate to the Azure Active Directory account for your organization. One way to do so is by searching for "Azure Active Directory" in the Azure portal's search bar. Once there, locate `tenant-id` under the basic information section in the *Overview* tab. Copy the `tenant-id` and paste in an editor to be used later.

+#### Find `client-id`

+Often called `app-id`, it's the same value that you used to register your application during the provisioning of your [Microsoft Energy Data Services Preview instance](quickstart-create-microsoft-energy-data-services-instance.md). You'll find the `client-id` in the *Essentials* pane of Microsoft Energy Data Services Preview *Overview* page. Copy the `client-id` and paste in an editor to be used later.

+> [!NOTE]

+> The 'client-id' that is passed as values in the entitlement API calls needs to be the same which was used for provisioning of your Microsoft Energy Data Services Preview instance.

+#### Find `client-secret`

+Sometimes called an application password, a `client-secret` is a string value your app can use in place of a certificate to identity itself. Navigate to *App Registrations*. Once there, open 'Certificates & secrets' under the *Manage* section.Create a `client-secret` for the `client-id` that you used to create your Microsoft Energy Data Services Preview instance, you can add one now by clicking on *New Client Secret*. Record the secret's `value` for use in your client application code.

+> [!NOTE]

+> Don't forget to record the secret's value for use in your client application code. This secret value is never displayed again after you leave this page at the time of creation of 'client secret'.

+#### Find the `url`for your Microsoft Energy Data Services Preview instance

+Navigate to your Microsoft Energy Data Services Preview *Overview* page on Azure portal. Copy the URI from the essentials pane.

+#### Find the `data-partition-id` for your group

+You have two ways to get the list of data-partitions in your Microsoft Energy Data Services Preview instance.

+- By navigating *Data Partitions* menu-item under the Advanced section of your Microsoft Energy Data Services Preview UI.

+- By clicking on the *view* below the *data partitions* field in the essentials pane of your Microsoft Energy Data Services Preview *Overview* page.

+## Generate access token

+You need to generate access token to use entitlements API. Run the below curl command in Azure Cloud Bash after replacing the placeholder values with the corresponding values found earlier in the pre-requisites step.

+

+**Request format**

+```bash

+curl --location --request POST 'https://login.microsoftonline.com/<tenant-id>/oauth2/token' \

+--header 'Content-Type: application/x-www-form-urlencoded' \

+--data-urlencode 'grant_type=client_credentials' \

+--data-urlencode 'scope=<client-id>.default' \

+--data-urlencode 'client_id=<client-id>' \

+--data-urlencode 'client_secret=<client-secret>' \

+--data-urlencode 'resource=<client-id>'

+```

+**Sample response**

+```JSON

+ {

+ "token_type": "Bearer",

+ "expires_in": 86399,

+ "ext_expires_in": 86399,

+ "access_token": abcdefgh123456............."

+ }

+```

+Copy the `access_token` value from the response. You'll need it to pass as one of the headers in all calls to the Entitlements API of your Microsoft Energy Data Services Preview instance.

+## User management activities

+You can manage user's access to your Microsoft Energy Data Services instance or data partitions. As a prerequisite for the same, you need to find the 'object-id' (OID) of the user(s) first.

+You'll need to input `object-id` (OID) of the users as parameters in the calls to the Entitlements API of your Microsoft Energy Data Services Preview Instance. `object-id`(OID) is the Azure Active Directory User Object ID.

+### Get the list of all available groups

+Run the below curl command in Azure Cloud Bash to get all the groups that are available for your Microsoft Energy Data Services instance and its data partitions.

+```bash

+ curl --location --request GET "https://<URI>/api/entitlements/v2/groups/" \

+ --header 'data-partition-id: <data-partition>' \

+ --header 'Authorization: Bearer <access_token>'

+```

+### Add user(s) to a users group

+Run the below curl command in Azure Cloud Bash to add user(s) to the "Users" group using Entitlement service.

+```bash

+ curl --location --request POST 'https://<URI>/api/entitlements/v2/groups/users@<data-partition-id>.dataservices.energy/members' \

+ --header 'data-partition-id: <data-partition-id>' \

+ --header 'Authorization: Bearer <access_token>' \

+ --header 'Content-Type: application/json' \

+ --data-raw '{

+ "email": "<Object_ID>",

+ "role": "MEMBER"

+ }'

+```

+> [!NOTE]

+> The value to be sent for the param "email" is the Object ID of the user and not the user's email

+**Sample request**

+```bash

+ curl --location --request POST 'https://<instance>.energy.azure.com/api/entitlements/v2/groups/users@<instance>-<data-partition-name>.dataservices.energy/members' \

+ --header 'data-partition-id: <instance>-<data-partition-name>' \

+ --header 'Authorization: Bearer <access_token>' \

+ --header 'Content-Type: application/json' \

+ --data-raw '{

+ "email": "90e0d063-2f8e-4244-860a-XXXXXXXXXX",

+ "role": "MEMBER"

+ }'

+```

+**Sample Response**

+```JSON

+ {

+ "email": "90e0d063-2f8e-4244-860a-XXXXXXXXXX",

+ "role": "MEMBER"

+ }

+```

+### Add user(s) to an entitlements group

+Run the below curl command in Azure Cloud Bash to add user(s) to an entitlement group using Entitlement service.

+```bash

+ curl --location --request POST 'https://<URI>/api/entitlements/v2/groups/service.search.user@<data-partition-id>.dataservices.energy/members' \

+ --header 'data-partition-id: <data-partition-id>' \

+ --header 'Authorization: Bearer <access_token>' \

+ --header 'Content-Type: application/json' \

+ --data-raw '{

+ "email": "<Object_ID>",

+ "role": "MEMBER"

+ }'

+```

+> [!NOTE]

+> The value to be sent for the param "email" is the Object ID of the user and not the user's email

+**Sample request**

+```bash

+ curl --location --request POST 'https://<instance>.energy.azure.com/api/entitlements/v2/groups/service.search.user@<instance>-<data-partition-name>.dataservices.energy/members' \

+ --header 'data-partition-id: <instance>-<data-partition-name>' \

+ --header 'Authorization: Bearer <access_token>' \

+ --header 'Content-Type: application/json' \

+ --data-raw '{

+ "email": "90e0d063-2f8e-4244-860a-XXXXXXXXXX",

+ "role": "MEMBER"

+ }'

+```

+**Sample response**

+```JSON

+ {

+ "email": "90e0d063-2f8e-4244-860a-XXXXXXXXXX",

+ "role": "MEMBER"

+ }

+```

+### Get entitlements groups for a given user

+Run the below curl command in Azure Cloud Bash to get all the groups associated with the user.

+```bash

+ curl --location --request GET 'https://<URI>/api/entitlements/v2/members/<OBJECT_ID>/groups?type=none' \

+ --header 'data-partition-id: <data-partition-id>' \

+ --header 'Authorization: Bearer <access_token>'

+```

+**Sample request**

+```bash

+ curl --location --request GET 'https://<instance>.energy.azure.com/api/entitlements/v2/members/90e0d063-2f8e-4244-860a-XXXXXXXXXX/groups?type=none' \

+ --header 'data-partition-id: <instance>-<data-partition-name>' \

+ --header 'Authorization: Bearer <access_token>'

+```

+**Sample response**

+```JSON

+ {

+ "desId": "90e0d063-2f8e-4244-860a-XXXXXXXXXX",

+ "memberEmail": "90e0d063-2f8e-4244-860a-XXXXXXXXXX",

+ "groups": [

+ {

+ "name": "users",

+ "description": "Datalake users",

+ "email": "users@<instance>-<data-partition-name>.dataservices.energy"

+ },

+ {

+ "name": "service.search.user",

+ "description": "Datalake Search users",

+ "email": "service.search.user@<instance>-<data-partition-name>.dataservices.energy"

+ }

+ ]

+ }

+```

+### Delete entitlement groups of a given user

+Run the below curl command in Azure Cloud Bash to delete a given user to your Microsoft Energy Data Services instance data partition.

+> [!NOTE]

+> As stated above, **DO NOT** delete the OWNER of a group unless you have another OWNER that can manage users in that group.

+```bash

+ curl --location --request DELETE 'https://<URI>/api/entitlements/v2/members/<OBJECT_ID>' \

+ --header 'data-partition-id: <data-partition-id>' \

+ --header 'Authorization: Bearer <access_token>'

+```

+**Sample request**

+```bash

+ curl --location --request DELETE 'https://<instance>.energy.azure.com/api/entitlements/v2/members/90e0d063-2f8e-4244-860a-XXXXXXXXXX' \

+ --header 'data-partition-id: <instance>-<data-partition-name>' \

+ --header 'Authorization: Bearer <access_token>'

+```

+**Sample response**

+No output for a successful response

+## Next steps

+<!-- Add a context sentence for the following links -->

+Create a legal tag for your Microsoft Energy Data Services Preview instance's data partition.

+> [!div class="nextstepaction"]

+> [How to manage legal tags](how-to-manage-legal-tags.md)

+Begin your journey by ingesting data into your Microsoft Energy Data Services Preview instance.

+> [!div class="nextstepaction"]

+> [Tutorial on CSV parser ingestion](tutorial-csv-ingestion.md)

+> [!div class="nextstepaction"]

+> [Tutorial on manifest ingestion](tutorial-manifest-ingestion.md)

+ Title: Overview of domain data management services - Microsoft Energy Data Services Preview #Required; page title is displayed in search results. Include the brand.

+description: This article provides an overview of Domain data management services #Required; article description that is displayed in search results.

+# Domain data management services (DDMS)

+The energy industry works with data of an extraordinary magnitude, which has significant ramifications for storage and compute requirements. Geoscientists stream terabytes of seismic, well log, and other data types at full resolution. Immediate responsiveness of data is essential for all stages of petroleum exploration--particularly for geologic interpretation and analysis. 

+## Overview

+Domain data management services (DDMS) store, access, and retrieve metadata and bulk data from applications connected to the data platform. Developers, therefore, use DDMS to deliver seamless and secure consumption of data in the applications they build on Microsoft Energy Data Services Preview. The Microsoft Energy Data Services Preview suite of DDMS adheres to [Open Subsurface Data Universe](https://osduforum.org/) (OSDU&trade;) standards and provides enhancements in performance, geo-availability, and access controls. DDMS service is optimized for each data type and can be extended to accommodate new data types. The DDMS service preserves raw data and offers multi format support and conversion for consuming applications such as Petrel while tracking lineage. Data within the DDMS service is discoverable and governed by entitlement and legal tags.

+### OSDU&trade; definition

+

+- Highly optimized storage & access for bulk data, with highly opinionated APIs delivering the data required to enable domain workflows

+- Governed schemas that incorporate domain-specific perspective and type-safe accessors for registered entity types

+### Aspirational components for any DDMS

+ - Direct connection to OSDU&trade; core

+ - Connection to adjacent or proximal databases (blob storage, Cosmos, external) and client applications

+ - Configure infrastructure provisioning to enable optimal performance for data streaming and access

+### Additional components for most DDMS (may include but not be limited to)

+ - File format converter--for example, for Seismic DDMS: SGY to ZGY, etc.

+ - Hierarchy of data organization and chunking - Tenant, project, and data

+## Use cases and value add

+### Frictionless Exploration and Production(E&P)

+The Microsoft Energy Data Services Preview DDMS service enables energy companies to access their data in a manner that is fast, portable, testable and extendible. As a result, they'll achieve unparalleled streaming performance and use the standards and output from OSDU&trade;. The Azure DDMS service will onboard the OSDU&trade; DDMS and Schlumberger proprietary DMS. Microsoft also continues to contribute to the OSDU&trade; community DDMS to ensure compatibility and architectural alignment.

+### Seamless connection between applications and data

+Customers can deploy applications on top of Microsoft Energy Data Services Preview that have been developed as per the OSDU&trade; standard. They're able to connect applications to Core Services and DDMS without spending extensive cycles on deployment. Customers can also easily connect DELFI to Microsoft Energy Data Services Preview, eliminating the cycles associated with Petrel deployments and connection to data management systems. By connecting applications to DDMS service, Geoscientists can execute integrated E&P workflows with unparalleled performance on Azure and use OSDU&trade; core services. For example, a geophysicist can pick well ties on a seismic volume in Petrel and stream data from the seismic DMS.

+## Types of DMS

+Below are the OSDU&trade; DMS the service supports -

+### OSDU&trade; - Seismic DMS

+Seismic data is a fundamental data type for oil and gas exploration. Seismic data provides a geophysical representation of the subsurface that can be applied for prospect identification and drilling decisions. Typical seismic datasets represent a multi-kilometer survey and are therefore massive in size.

+Due to this extraordinary data size, geoscientists working on-premises struggle to use seismic data in domain applications. They suffer from crashes as the seismic dataset exceeds their workstation's RAM, which leads to significant non-productive time. To achieve performance needed for domain workflows, geoscientists must chunk a seismic dataset and view each chunk in isolation. As a result, users suffer from the time spent wrangling seismic data and the opportunity cost of missing the significant picture view of the subsurface and target reservoirs.

+The seismic DMS is part of the OSDU&trade; platform and enables users to connect seismic data to cloud storage to applications. It allows secure access to metadata associated with seismic data to efficiently retrieve and handle large blocks of data for OpenVDS, ZGY, and other seismic data formats. The DMS therefore enables users to stream huge amounts of data in OSDU&trade; compliant applications in real time. Enabling the seismic DMS on Microsoft Energy Data Services Preview opens a pathway for Azure customers to bring their seismic data to the cloud and take advantage of Azure storage and high performance computing.

+## OSDU&trade; - Wellbore DMS

+Well Logs are measurements taken while drilling, which tells energy companies information about the subsurface. Ultimately, they reveal whether hydrocarbons are present (or if the well is dry). Logs contain many attributes that inform geoscientists about the type of rock, its quality, and whether it contains oil, water, gas, or a mix. Energy companies use these attributes to determine the quality of a reservoir ΓÇô how much oil or gas is present, its quality, and ultimately, economic viability. Maintaining Well Log data and ensuring easy access to historical logs is critical to energy companies. The Wellbore DMS facilitates access to this data in any OSDU&trade; compliant application. The Wellbore DMS was contributed by Schlumberger to OSDU&trade;.

+Well Log data can come in different formats. It's most often indexed by depth or time and the increment of these measurements can vary. Well Logs typically contain multiple attributes for each vertical measurement. Well Logs can therefore be small or for more modern Well Logs that use high frequency data, greater than 1 Gb. Well Log data is smaller than seismic; however, users will want to look at upwards of hundreds of wells at a time. This scenario is common in mature areas that have been heavily drilled such as the Permian Basin in West Texas.

+Geoscientists therefore want to access numerous well logs in a single session. They often are looking at all historical drilling programs in an area. As a result, they'll look at Well Log data that was collected using a wide variety of instruments and technology. This data will vary widely in format, quality, and sampling. The Wellbore DMS resolves this data through the OSDU&trade; schemas to deliver the data to the consuming applications.

+Here are the services that the Wellbore DMS offers -

+- **Objects and Consumption** - The Wellbore DMS can consume Wellbore, log set, log, marker, trajectory, and dip objects. This covers most well related exploration workflows

+- **Lifecycle** ΓÇô The Wellbore DMS supports the dataset through creation and writing to storage, versioning, lineage, auditing, and deletion

+- **Ingestion** - connection to file, interpretation software, system of records, and acquisition systems

+- **Contextualization** (Contextualized Access)

+## OSDU&trade; - Well Delivery DMS

+The Well Delivery DMS stores critical drilling domain information related to the planning and execution of a well. Throughout a drilling program, engineers and domain experts need to access a wide variety of data types including activities, trajectories, risks, subsurface information, equipment used, fluid and cementing, rig utilization, and reports. Integrating this collection of data types together are the cornerstone to drilling insights. At the same time, until now, there was no industry wide standardization or enforced format. The common standards the Well Delivery DMS enables is critical to the Drilling Value Chain as it connects a diverse group of personas including operations, oil companies, service companies, logistics companies, etc.

+## Next steps

+Learn more about DDMS concepts below.

+> [!div class="nextstepaction"]

+> [DDMS Concepts](concepts-ddms.md)

+ Title: What is Microsoft Energy Data Services Preview? #Required; page title is displayed in search results. Include the brand.

+description: This article provides an overview of Microsoft Energy Data Services Preview #Required; article description that is displayed in search results.

+# What is Microsoft Energy Data Services Preview?

+Microsoft Energy Data Services Preview is a secure, reliable, hyperscale, fully managed cloud-based data platform solution for the energy industry. It is an enterprise-grade data platform that brings together the capabilities of OSDU&trade; Data Platform, Microsoft's secure and trusted Azure cloud platform, and Schlumberger's extensive domain expertise. It allows customers to free data from silos, provides strong data management, storage, and federation strategy. Microsoft Energy Data Services ensures compatibility with evolving community standards like OSDU&trade; and enables value addition through interoperability with both first-party and third-party solutions.

+## Principles

+Microsoft Energy Data Services conforms to the following principles:

+### Fully managed OSDU&trade; platform

+Microsoft Energy Data Services Preview is a first-party PaaS (Platform as a Service) offering where Microsoft manages the deployment, monitoring, management, scale, security, updates, and upgrades of the service so that the customers can focus on the value from the platform. Microsoft offers seamless upgrades to the latest OSDU&trade; milestone versions after testing and validation.

+Furthermore, Microsoft Energy Data Services Preview provides security capabilities like encryption for data-in-transit and data-at-rest. The authentication and authorization are provided by Azure Active Directory. Microsoft also assumes the responsibility of providing regular security patches and updates.

+Microsoft Energy Data Services Preview also supports multiple data partitions for every platform instance. More data partitions can also be created after creating an instance, as needed.

+As an Azure-based service, it also provides elasticity with auto-scaling to handle dynamically varying workload requirements. The service provides out-of-the-box compatibility and built-in integration with industry-leading applications from Schlumberger, including Petrel to provide quick time to value.

+Microsoft will provide support for the platform to enable our customers' use cases.

+### Accelerated innovation with openness in mind

+Microsoft Energy Data Services Preview is compatible with the OSDU&trade; Technical Standard enables seamless integration of existing applications that have been developed in alignment with the emerging requirements of the OSDU&trade; Standard.

+The platform's openness and integration with Microsoft Azure Marketplace brings industry-leading applications, solutions, and integration services offered by our extensive partner ecosystem to our customers.

+### Extensibility with the Microsoft ecosystem

+Most of our customers rely on ubiquitous tools and applications from Microsoft. The Microsoft Energy Data Services Preview platform is piloting how it can seamlessly work with deeply used Microsoft apps like SharePoint for data ingestion, Synapse for data transformations and pipelines, Power BI for data visualization, and other possibilities. A Power BI connector has already been released in the community, and partners are leveraging these tools and connectors to enhance their integrations with Microsoft apps and services.

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+Follow the quickstart guide to quickly deploy Microsoft Energy Data Service in your Azure subscription

+> [!div class="nextstepaction"]

+> [Quickstart: Create Microsoft Energy Data Services Preview instance](quickstart-create-microsoft-energy-data-services-instance.md)

+ Title: Create a Microsoft Energy Data Services Preview instance #Required; page title is displayed in search results. Include the brand.

+description: Quickly create a Microsoft Energy Data Services Preview instance #Required; article description that is displayed in search results.

+# Quickstart: Create a Microsoft Energy Data Services Preview instance

+Get started by creating a Microsoft Energy Data Services Preview instance on Azure portal on a web browser. You first register an Azure application on Active Directory and then use the application ID to create a Microsoft Energy Data Services instance in your chosen Azure Subscription and region.

+The setup of Microsoft Energy Data Services Preview instance can be triggered using a simple interface on Azure portal and takes about 50 minutes to complete.

+Microsoft Energy Data Services Preview is a managed "Platform as a service (PaaS)" offering from Microsoft that builds on top of the [OSDU&trade;](https://osduforum.org/) Data Platform. Microsoft Energy Data Services Preview lets you ingest, transform, and export subsurface data by letting you connect your consuming in-house or third-party applications.

+## Prerequisites

+| Prerequisite | Details |

+| | - |

+Active Azure Subscription | You'll need the Azure subscription ID in which you want to install Microsoft Energy Data Services. You need to have appropriate permissions to create Azure resources in this subscription.

+Application ID | You'll need an [application ID](../active-directory/develop/application-model.md) (often referred to as "App ID" or a "client ID"). This application ID will be used for authentication to Azure Active Directory and will be associated with your Microsoft Energy Data Services instance. You can [create an application ID](../active-directory/develop/quickstart-register-app.md) by navigating to Active directory and selecting *App registrations* > *New registration*.

+Client Secret | Sometimes called an application password, a client secret is a string value that your app can use in place of a certificate to identity itself. You can [create a client secret](../active-directory/develop/quickstart-register-app.md#add-a-client-secret) by selecting *Certificates & secrets* > *Client secrets* > *New client secret*. Record the secret's value for use in your client application code. This secret value is never displayed again after you leave this page.

+## Create a Microsoft Energy Data Services Preview instance

+1. Save your **Application (client) ID** and **client secret** from Azure Active Directory to refer to them later in this quickstart.

+1. Sign in to [Microsoft Azure Marketplace](https://portal.azure.com/?microsoft_azure_marketplace_ItemHideKey=Microsoft_Azure_OpenEnergyPlatformHidden)

+ > [!IMPORTANT]

+ > *Microsoft Energy Data Services* is accessible on the Azure Marketplace only if you use the above Azure portal link.

+1. If you have access to multiple tenants, use the *Directories + subscriptions* filter in the top menu to switch to the tenant in which you want to install Microsoft Energy Data Services.

+1. Use the search bar in the Azure Marketplace (not the global Azure search bar on top of the screen) to search for *Microsoft Energy Data Services*.

+ [](media/quickstart-create-microsoft-energy-data-services-instance/search-meds-on-azure-marketplace.png#lightbox)

+1. In the search page, select *Create* on the card titled "Microsoft Energy Data Services (Preview)".

+1. A new window appears. Complete the *Basics* tab by choosing the *subscription*, *resource group*, and the *region* in which you want to create your instance of Microsoft Energy Data Services. Enter the *App ID* that you created during the prerequisite steps.

+ [](media/quickstart-create-microsoft-energy-data-services-instance/input-basic-details.png#lightbox)

+

+ Some naming conventions to guide you at this step:

+ | Field | Name Validation |

+ | -- | |

+ Instance name | Only alphanumeric characters are allowed, and the value must be 1-15 characters long. The name is **not** case-sensitive. One resource group can't have two instances with the same name.

+ Application ID | Enter the valid Application ID that you generated and saved in the last section.

+ Data Partition name | Name should be 1-10 char long consisting of lowercase alphanumeric characters and hyphens. It should start with an alphanumeric character and not contain consecutive hyphens. The data partition names that you chose are automatically prefixed with your Microsoft Energy Data Services instance name. This compound name will be used to refer to your data partition in application and API calls.

+ > [!NOTE]

+ > Microsoft Energy Data Services instance and data partition names, once created, cannot be changed later.

+1. Select **Next: Tags** and enter any tags that you would want to specify. If nothing, this field can be left blank.

+ > [!TIP]

+ > Tags are metadata elements attached to resources. They're key-value pairs that help you identify resources based on settings that are relevant to your organization. If you want to track the deployment environment for your resources, add a key named `Environment`. To identify the resources deployed to production, give them a value of `Production`. [Learn more](../azure-resource-manager/management/tag-resources.md?tabs=json).

+ [](media/quickstart-create-microsoft-energy-data-services-instance/input-tags.png#lightbox)

+1. Select Next: **Review + Create**.

+1. Once the basic validation tests pass (validation takes a few seconds), review the Terms and Basic Details.

+ [](media/quickstart-create-microsoft-energy-data-services-instance/validation-check-after-entering-details.png#lightbox)

+1. This step is optional. You can download an Azure Resource Manager (ARM) template and use it for automated deployments of Microsoft Energy Data Services in future. Select *Download a template for automation* located on the bottom-right of the screen.

+ [](media/quickstart-create-microsoft-energy-data-services-instance/download-template-automation.png#lightbox)

+ [](media/quickstart-create-microsoft-energy-data-services-instance/automate-deploy-resource-using-azure-resource-manager.png#lightbox)

+1. Select **Create** to start the deployment.

+1. Wait while the deployment happens in the background. Review the details of the instance created.

+ [](media/quickstart-create-microsoft-energy-data-services-instance/deployment-complete.png#lightbox)

+ [](media/quickstart-create-microsoft-energy-data-services-instance/overview-energy-data-services.png#lightbox)

+

+## Delete a Microsoft Energy Data Services Preview instance

+Deleting a Microsoft Energy Data instance also deletes any data that you've ingested. This action is permanent and the ingested data can't be recovered. To delete a Microsoft Energy Data Services instance, complete the following steps:

+1. Sign in to the Azure portal and delete the *resource group* in which these components are installed.

+2. This step is optional. Go to Azure Active Directory and delete the *app registration* that you linked to your Microsoft Energy Data Services instance.

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+After provisioning a Microsoft Energy Data Services instance, you can learn about user management on this instance.

+> [!div class="nextstepaction"]

+> [How to manage users](how-to-manage-users.md)

+ Title: Release notes for Microsoft Energy Data Services Preview #Required; page title is displayed in search results. Include the brand.

+description: This topic provides release notes of Microsoft Energy Data Services Preview releases, improvements, bug fixes, and known issues. #Required; article description that is displayed in search results.

+# Release Notes for Microsoft Energy Data Services Preview

+Microsoft Energy Data Services is updated on an ongoing basis. To stay up to date with the most recent developments, this article provides you with information about:

+- The latest releases

+- Known issues

+- Bug fixes

+- Deprecated functionality

+- Plans for changes

+## Microsoft Energy Data Services Preview Release

+### Key Announcement

+Microsoft Energy Data Services is now available in public preview.

+Microsoft Energy Data Services is developed in alignment with the emerging requirements of the OSDUΓäó Technical Standard, Version 1.0. and is currently aligned with Mercury Release(R3), [Milestone-12](https://community.opengroup.org/osdu/governance/project-management-committee/-/wikis/M12-Release-Notes).

+### Partition & User Management

+- New data partitions can be [created dynamically](how-to-add-more-data-partitions.md) as needed post provising of the platform (up to five). Earlier, data partitions could only be created when provisioning a new instance.

+- The domain name for entitlement groups for [user management](how-to-manage-users.md) has been changed to "dataservices.energy".

+### Data Ingestion

+- Enabled support for user context in ingestion ([ADR: Issue 52](https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52))

+ - User identity is preserved and passed on to all ingestion workflow related services using the newly introduced _x-on-behalf-of_ header. A user needs to have appropriate service level entitlements on all dependent services involved in the ingestion workflow and only users with appropriate data level entitlements can modify data.

+- Workflow service payload is restricted to a maximum of 2 MB. If it exceeds, the service will throw an HTTP 413 error. This restriction is placed to prevent workflow requests from overwhelming the server.

+- Microsoft Energy Data Services uses Azure Data Factory (ADF) to run large scale ingestion workloads.

+### Search

+- Improved security as Elasticsearch images are now pulled from Microsoft's internal Azure Container Registry instead of public repositories.

+- Improved security by enabling encryption in transit for Elasticsearch, Registration, and Notification services.

+### Monitoring

+- Diagnostic settings can be exported from [Airflow](how-to-integrate-airflow-logs-with-azure-monitor.md) and [Elasticsearch](how-to-integrate-elastic-logs-with-azure-monitor.md) to Azure Monitor.

+### Region Availability

+- Currently, Microsoft Energy Data Services is being offered in the following regions - South Central US, East US, West Europe, and North Europe.

+

+ Title: Microsoft Energy Data Services - Steps to perform a CSV parser ingestion #Required; page title is displayed in search results. Include the brand.

+description: This tutorial shows you how to perform CSV parser ingestion #Required; article description that is displayed in search results.

+#Customer intent: As a customer, I want to learn how to use CSV parser ingestion so that I can load CSV data into the Microsoft Energy Data Services Preview instance.

+# Tutorial: Sample steps to perform a CSV parser ingestion

+CSV Parser ingestion provides the capability to ingest CSV files into the Microsoft Energy Data Services Preview instance.

+In this tutorial, you'll learn how to:

+> [!div class="checklist"]

+> * Ingest a sample wellbore data CSV file into the Microsoft Energy Data Services Preview instance using Postman

+> * Search for storage metadata records created during the CSV Ingestion using Postman

+## Prerequisites

+### Get Microsoft Energy Data Services Preview instance details

+* Microsoft Energy Data Services Preview instance is created already. If not, follow the steps outlined in [Quickstart: Create a Microsoft Energy Data Services Preview instance](quickstart-create-microsoft-energy-data-services-instance.md)

+* For this tutorial, you will need the following parameters:

+ | Parameter | Value to use | Example | Where to find these values? |

+ | | |-- |-- |

+ | CLIENT_ID | Application (client) ID | 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx | App ID or Client_ID used when registering the application with the Microsoft Identity Platform. See [Register an application](../active-directory/develop/quickstart-register-app.md#register-an-application) |

+ | CLIENT_SECRET | Client secrets | _fl****************** | Sometimes called an *application password*, a client secret is a string value your app can use in place of a certificate to identity itself. See [Add a client secret](../active-directory/develop/quickstart-register-app.md#add-a-client-secret)|

+ | TENANT_ID | Directory (tenant) ID | 72f988bf-86f1-41af-91ab-xxxxxxxxxxxx | Hover over your account name in the Azure portal to get the directory or tenant ID. Alternately, search and select *Azure Active Directory > Properties > Tenant ID* in the Azure portal. |

+ | SCOPE | Application (client) ID | 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx | Same as App ID or Client_ID mentioned above |

+ | refresh_token | Refresh Token value | 0.ATcA01-XWHdJ0ES-qDevC6r........... | Follow the [How to Generate a Refresh Token](how-to-generate-refresh-token.md) to create a refresh token and save it. This refresh token is required later to generate a user token. |

+ | DNS | URI | `<instance>`.energy.Azure.com | Overview page of Microsoft Energy Data Services instance|

+ | data-partition-id | Data Partition(s) | `<instance>`-`<data-partition-name>` | Overview page of Microsoft Energy Data Services instance|

+* Follow the [Manage users](how-to-manage-users.md) guide to add appropriate entitlements for the user running this tutorial

+### Set up and execute Postman requests

+* Download and install [Postman](https://www.postman.com/) desktop app

+* Import the following files into Postman:

+ * [CSV Workflow Postman collection](https://raw.githubusercontent.com/microsoft/meds-samples/main/postman/IngestionWorkflows.postman_collection.json)

+ * [CSV Workflow Postman Environment](https://raw.githubusercontent.com/microsoft/meds-samples/main/postman/IngestionWorkflowEnvironment.postman_environment.json)

+ > [!NOTE]

+ > To import the Postman collection and environment variables, follow the steps outlined in [Importing data into Postman](https://learning.postman.com/docs/getting-started/importing-and-exporting-data/#importing-data-into-postman)

+

+* Update the **CURRENT_VALUE** of the Postman environment with the information obtained in [Microsoft Energy Data Services Preview instance details](#get-microsoft-energy-data-services-preview-instance-details)

+* The Postman collection for CSV parser ingestion contains a total of 10 requests, which have to be executed in a sequential manner.

+* Make sure to choose the **Ingestion Workflow Environment** before triggering the Postman collection.

+ :::image type="content" source="media/tutorial-csv-ingestion/tutorial-postman-choose-environment.png" alt-text="Screenshot of the postman environment." lightbox="media/tutorial-csv-ingestion/tutorial-postman-choose-environment.png":::

+* Each request can be triggered by clicking the **Send** Button.

+* On every request Postman will validate the actual API response code against the expected response code; if there's any mismatch the test section will indicate failures.

+#### Successful Postman request

+ :::image type="content" source="media/tutorial-csv-ingestion/tutorial-postman-test-success.png" alt-text="Screenshot of a successful postman call." lightbox="media/tutorial-csv-ingestion/tutorial-postman-test-success.png":::

+#### Failed Postman request

+ :::image type="content" source="media/tutorial-csv-ingestion/tutorial-postman-test-failure.png" alt-text="Screenshot of a failure postman call." lightbox="media/tutorial-csv-ingestion/tutorial-postman-test-failure.png":::

+## Ingest a sample wellbore data CSV file into the Microsoft Energy Data Services Preview instance using Postman

+ 1. **Get a user token** - Generate the User token, which will be used to authenticate further API calls.

+ 2. **Create a schema** - Generate a schema that adheres to the columns present in the CSV file

+ 3. **Get schema details** - Get the schema created in the previous step and validate it

+ 4. **Create a legal tag** - Create a legal tag that will be added to the CSV data for data compliance purpose

+ 5. **Get a signed url for uploading a CSV file** - Get the signed URL path to which the CSV file will be uploaded

+ 6. **Upload a CSV file** - Download the [Wellbore.csv](https://github.com/microsoft/meds-samples/blob/main/test-data/wellbore.csv) to your local machine, and select this file in Postman by clicking the **Select File** option as shown in the Screenshot below.

+ :::image type="content" source="media/tutorial-csv-ingestion/tutorial-select-csv-file.png" alt-text="Screenshot of uploading a CSV file." lightbox="media/tutorial-csv-ingestion/tutorial-select-csv-file.png":::

+ 7. **Upload CSV file metadata** - Upload the file metadata information such as file location & other relevant fields

+ 8. **Trigger a CSV parser ingestion workflow** - Triggers the CSV Parser ingestion workflow DAG.

+ 9. **Get CSV parser ingestion workflow status** - Gets the status of CSV Parser Dag Run.

+## Search for storage metadata records created during the CSV Ingestion using Postman

+ 1. **Search for ingested CSV records** - Search for the CSV records created earlier.

+ :::image type="content" source="media/tutorial-csv-ingestion/tutorial-search-success.png" alt-text="Screenshot of searching ingested CSV records." lightbox="media/tutorial-csv-ingestion/tutorial-search-success.png":::

+## Next steps

+Advance to the next tutorial to learn how to do Manifest ingestion

+> [!div class="nextstepaction"]

+> [Tutorial: Sample steps to perform a manifest-based file ingestion](tutorial-manifest-ingestion.md)

+ Title: Microsoft Energy Data Services - Steps to perform a manifest-based file ingestion #Required; page title is displayed in search results. Include the brand.

+description: This tutorial shows you how to perform Manifest ingestion #Required; article description that is displayed in search results.

+#Customer intent: As a customer, I want to learn how to use manifest ingestion so that I can load manifest information into the Microsoft Energy Data Services Preview instance.

+# Tutorial: Sample steps to perform a manifest-based file ingestion

+Manifest ingestion provides the capability to ingest manifests into Microsoft Energy Data Services Preview instance

+In this tutorial, you will learn how to:

+> [!div class="checklist"]

+> * Ingest sample manifests into the Microsoft Energy Data Services Preview instance using Postman

+> * Search for storage metadata records created during the manifest ingestion using Postman

+## Prerequisites

+Before beginning this tutorial, the following prerequisites must be completed:

+### Get Microsoft Energy Data Services Preview instance details

+* Microsoft Energy Data Services Preview instance is created already. If not, follow the steps outlined in [Quickstart: Create a Microsoft Energy Data Services Preview instance](quickstart-create-microsoft-energy-data-services-instance.md)

+* For this tutorial, you will need the following parameters:

+ | Parameter | Value to use | Example | Where to find these values? |

+ | | |-- |-- |

+ | CLIENT_ID | Application (client) ID | 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx | App ID or Client_ID used when registering the application with the Microsoft Identity Platform. See [Register an application](../active-directory/develop/quickstart-register-app.md#register-an-application) |

+ | CLIENT_SECRET | Client secrets | _fl****************** | Sometimes called an *application password*, a client secret is a string value your app can use in place of a certificate to identity itself. See [Add a client secret](../active-directory/develop/quickstart-register-app.md#add-a-client-secret)|

+ | TENANT_ID | Directory (tenant) ID | 72f988bf-86f1-41af-91ab-xxxxxxxxxxxx | Hover over your account name in the Azure portal to get the directory or tenant ID. Alternately, search and select *Azure Active Directory > Properties > Tenant ID* in the Azure portal. |

+ | SCOPE | Application (client) ID | 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx | Same as App ID or Client_ID mentioned above |

+ | refresh_token | Refresh Token value | 0.ATcA01-XWHdJ0ES-qDevC6r........... | Follow the [How to Generate a Refresh Token](how-to-generate-refresh-token.md) to create a refresh token and save it. This refresh token is required later to generate a user token. |

+ | DNS | URI | `<instance>`.energy.Azure.com | Overview page of Microsoft Energy Data Services instance|

+ | data-partition-id | Data Partition(s) | `<instance>`-`<data-partition-name>` | Overview page of Microsoft Energy Data Services instance|

+* Follow the [Manage users](how-to-manage-users.md) guide to add appropriate entitlements for the user running this tutorial

+### Set up Postman and execute requests

+* Download and install [Postman](https://www.postman.com/) desktop app

+* Import the following files into Postman:

+ * [Manifest ingestion postman collection](https://raw.githubusercontent.com/microsoft/meds-samples/main/postman/IngestionWorkflows.postman_collection.json)

+ * [Manifest Ingestion postman environment](https://raw.githubusercontent.com/microsoft/meds-samples/main/postman/IngestionWorkflowEnvironment.postman_environment.json)

+ > [!NOTE]

+ > To import the Postman collection and environment variables, follow the steps outlined in [Importing data into Postman](https://learning.postman.com/docs/getting-started/importing-and-exporting-data/#importing-data-into-postman)

+* Update the **CURRENT_VALUE** of the postman environment with the information obtained in [Get Microsoft Energy Data Services Preview instance details](#get-microsoft-energy-data-services-preview-instance-details)

+* The Postman collection for manifest ingestion contains multiple requests, which will have to be executed in a sequential manner.

+* Make sure to choose the **Ingestion Workflow Environment** before triggering the Postman collection.

+ :::image type="content" source="media/tutorial-manifest-ingestion/tutorial-postman-choose-environment.png" alt-text="Screenshot of the Postman environment." lightbox="media/tutorial-manifest-ingestion/tutorial-postman-choose-environment.png":::

+* Each request can be triggered by clicking the **Send** Button.

+* On every request, Postman will validate the actual API response code against the expected response code; if there is any mismatch the test section will indicate failures.

+#### Successful Postman request

+#### Failed Postman request

+## Ingest sample manifests into the Microsoft Energy Data Services Preview instance using Postman

+ 1. **Get a user token** - Generate the User token, which will be used to authenticate further API calls.

+ 2. **Create a legal tag** - Create a legal tag that will be added to the Manifest data for data compliance purpose

+ 3. **Get a signed url for uploading a file** - Get the signed URL path to which the Manifest file will be uploaded

+ 4. **Upload a file** - Download the sample [Wellbore.csv](https://github.com/microsoft/meds-samples/blob/main/test-data/wellbore.csv) to your local machine (it could be any filetype - CSV, LAS, JSON, etc.), and select this file in Postman by clicking the **Select File** option as shown in the Screenshot below.

+ :::image type="content" source="media/tutorial-manifest-ingestion/tutorial-select-manifest-file.png" alt-text="Screenshot of a select file option." lightbox="media/tutorial-manifest-ingestion/tutorial-select-manifest-file.png":::

+ 5. **Upload file metadata** - Upload the file metadata information such as file location & other relevant fields

+ 6. **Get the file metadata** - Call to validate if the metadata got created successfully

+ 7. **Ingest Master, Reference and Work Product Component (WPC) data** - Ingest the Master, Reference and Work Product Component manifest metadata.

+ 8. **Get manifest ingestion workflow status** - The workflow will start and will be in the **running** state. Keep querying until it changes state to **finished** (typically 20-30 seconds)

+

+## Search for storage metadata records created during the manifest ingestion using Postman

+ - **Search Work Products** - Call Search service to retrieve the Work Product metadata records

+ - **Search Work Product Components** - Call Search service to retrieve the Work Product Component metadata records

+ - **Search for Dataset** - Call Search service to retrieve the Dataset metadata records

+ - **Search for Master data** - Call Search service to retrieve the Master metadata records

+ - **Search for Reference data** - Call Search service to retrieve the Reference metadata records

+## Next steps

+Advance to the next tutorial to learn about sdutil

+> [!div class="nextstepaction"]

+> [Tutorial: Seismic store sdutil](tutorial-seismic-ddms-sdutil.md)

+ Title: Microsoft Energy Data Services Preview - Seismic store sdutil tutorial #Required; page title is displayed in search results. Include the brand.

+description: Information on setting up and using sdutil, a command-line interface (CLI) tool that allows users to easily interact with seismic store. #Required; article description that is displayed in search results.

+#Customer intent: As a developer, I want to learn how to use sdutil so that I can load data into the seismic store.

+# Tutorial: Seismic store sdutil

+A command line python utility designed to easily interact with seismic store.

+Seismic store is a cloud-based solution designed to store and manage datasets of any size in the cloud by enabling a secure way to access them through a scoped authorization mechanism. Seismic Store overcomes the object size limitations imposed by a cloud provider, by managing generic datasets as multi-independent objects and, therefore, provides a generic, reliable and a better performed solution to handle data on a cloud storage.

+The **sdutil** is an intuitive command line utility tool to interact with seismic store and perform some basic operations like upload or download datasets to or from seismic store, manage users, list folders content and more.

+## Prerequisites

+Install the following prerequisites based on your OS:

+Windows

+- [64-bit Python 3.8.3](https://www.python.org/ftp/python/3.8.3/python-3.8.3-amd64.exe)

+- [Microsoft C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)

+- [Linux Subsystem Ubuntu](https://learn.microsoft.com/windows/wsl/install)

+Linux

+- [64-bit Python 3.8.3](https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz)

+Unix

+- [64-bit Python 3.8.3](https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz)

+- Apple Xcode C++ Build Tools

+Other requirements are addressed in the INSTALLATION section below.

+## Installation

+Follow the directions in the sdutil documentation for [running sdutil in Azure environments](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/tree/azure/stable#setup-and-usage-for-azure-env).

+The utility requires other modules noted in [requirements.txt](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/blob/azure/stable/requirements.txt). You could either install the modules as is or install them in virtualenv to keep your host clean from package conflicts. if you don't want to install them in a virtual environment jump directly to the step 3.

+```bash

+ # check if virtualenv is already installed

+ virtualenv --version

+ # if not install it via pip

+ pip install virtualenv

+ # create a virtual environment for sdutil

+ virtualenv sdutilenv

+ # activate the virtual environemnt

+ Windows: sdutilenv/Scripts/activate

+ Linux: source sdutilenv/bin/activate

+```

+Install required dependencies:

+```bash

+ # run it from the extracted sdutil folder

+ pip install -r requirements.txt

+```

+## Usage

+### Configuration

+1. Replace/edit `config.yaml` in `sdlib/config.yaml` by this [config-azure.yaml](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/blob/azure/stable/docs/config-azure.yaml)

+2. Update three values in `config.yaml`:

+ ```yaml

+ - service: '{"Azure": {"azureGlabEnv":{"url": "<base-url-for-microsoft-energy-data-services-instance>/seistore-svc/api/v3", "appkey": ""}}}'

+ - url: '<base-url-for-microsoft-energy-data-services-instance>/seistore-svc/api/v3'

+ - "refresh_token": "<refresh-token-for-your-env>"

+ ```

+

+ > [!NOTE]

+ > Follow the directions in [How to Generate a Refresh Token](how-to-generate-refresh-token.md) to obtain a token if not already present.

+3. Export or set below environment variables

+ ```bash

+ export AZURE_TENANT_ID=check-env-provisioning-team-as-specific-to-cluster

+ export AZURE_CLIENT_ID=check-env-provisioning-team-as-specific-to-cluster

+ export AZURE_CLIENT_SECRET=check-env-provisioning-team-as-specific-to-cluster

+ ```

+### Running the Tool

+Run the utility from the extracted utility folder by typing:

+```bash

+ python sdutil

+```

+If no arguments are specified, this menu will be displayed:

+```code

+ Seismic Store Utility

+ > python sdutil [command]

+ available commands:

+ * auth : authentication utilities

+ * unlock : remove a lock on a seismic store dataset

+ * version : print the sdutil version

+ * rm : delete a subproject or a space separated list of datasets

+ * mv : move a dataset in seismic store

+ * config : manage the utility configuration

+ * mk : create a subproject resource

+ * cp : copy data to(upload)/from(download)/in(copy) seismic store

+ * stat : print information like size, creation date, legal tag(admin) for a space separated list of tenants, subprojects or datasets

+ * patch : patch a seismic store subproject or dataset

+ * app : application authorization utilities

+ * ls : list subprojects and datasets

+ * user : user authorization utilities

+```

+At first usage time, the utility required to be initialized by invoking the sdutil config init command.

+```bash

+ python sdutil config init

+```

+Before start using the utility and perform any operation, you must sign-in the system. When you run the following sign-in command, sdutil will open a sign-in page in a web browser.

+```bash

+ python sdutil auth login

+```

+Once you've successfully logged in, your credentials will be valid for a week. You don't need to sign in again unless the credentials expired (after one week), in this case the system will require you to sign in again.

+> [!NOTE]

+> If you aren't getting the "sign-in Successful!" message, make sure your three environment variables are set and you've followed all steps in the "Configuration" section above.

+## Seistore Resources

+Before you start using the system, it's important to understand how resources are addressed in seismic store. There are three different types of resources managed by seismic store:

+- **Tenant Project:** the main project. Tenant is the first section of the seismic store path

+- **Subproject:** the working subproject, directly linked under the main tenant project. Subproject is the second section of the seismic store path.

+- **Dataset:** the seismic store dataset entity. Dataset is the third and last section of the seismic store path. The Dataset resource can be specified by using the form `path/dataset_name` where `path` is optional and have the same meaning of a directory in a generic file-system and `dataset_name` is the name of the dataset entity.

+The seismic store uri is a string used to uniquely address a resource in the system and can be obtained by appending the prefix `sd://` before the required resource path:

+```code

+ sd://<tenant>/<subproject>/<path>*/<dataset>

+```

+For example, if we have a dataset `results.segy` stored in the directory structure `qadata/ustest` in the `carbon` subproject under the `gtc` tenant project, then the corresponding sdpath will be:

+```code

+ sd://gtc/carbon/qadata/ustest/results.segy

+```

+Every resource can be addressed by using the corresponding sdpath section

+```code

+ Tenant: sd://gtc

+ Subproject: sd://gtc/carbon

+ Dataset: sd://gtc/carbon/qadata/ustest/results.segy

+```

+## Subprojects

+A subproject in Seismic Store is a working unit where datasets can be saved. The system can handles multiple subprojects under a tenant project.

+A subproject resource can be created by a **Tenant Admin Only** with the following sdutil command:

+```code

+ > python sdutil mk *sdpath *admin@email *legaltag (options)

+ create a new subproject resource in the seismic store. user can interactively

+ set the storage class for the subproject. only tenant admins are allowed to create subprojects.

+ *sdpath : the seismic store subproject path. sd://<tenant>/<subproject>

+ *admin@email : the email of the user to be set as the subproject admin

+ *legaltag : the default legal tag for the created subproject

+ (options) | --idtoken=<token> pass the credential token to use, rather than generating a new one

+```

+## Users Management

+To be able to use seismic store, a user must be registered/added to at least a subproject resource with a role that defines their access level. Seismic Store support three different roles scoped at subproject level:

+- **admin**: read/write access + users management.

+- **viewer**: read/list access

+A user can be registered by a **Subproject Admin Only** with the following sdutil command:

+```code

+ > python sdutil user [ *add | *list | *remove | *roles ] (options)

+ *add $ python sdutil user add [user@email] [sdpath] [role]*

+ add a user to a subproject resource

+ [user@email] : email of the user to add

+ [sdpath] : seismic store subproject path, sd://<tenant>/<subproject>

+ [role] : user role [admin|viewer]

+```

+## Usage Examples

+An example of how to use sdutil to manage datasets with seismic store. For this example, we'll use sd://gtc/carbon as subproject resource

+```bash

+ # create a new file

+ echo "My Test Data" > data1.txt

+ # upload the created file to seismic store

+ ./sdutil cp data1.txt sd://gtc/carbon/test/mydata/data.txt

+ # list the content of the seismic store subproject

+ ./sdutil ls sd://gtc/carbon/test/mydata/ (display: data.txt)

+ ./sdutil ls sd://gtc (display: carbon)

+ ./sdutil ls sd://gtc/carbon (display: test/)

+ ./sdutil ls sd://gtc/carbon/test (display: data/)

+ # download the file from seismic store:

+ ./sdutil cp sd://gtc/carbon/test/mydata/data.txt data2.txt

+ # check if file orginal file match the one downloaded from sesimic store:

+ diff data1.txt data2.txt

+```

+## Utility Testing

+The test folder contains a set of integral/unit and regressions/e2e tests written for [pytest](https://docs.pytest.org/en/latest/). These tests should be executed to validate the utility functionalities.

+Requirements

+ ```bash

+ # install required dependencies:

+ pip install -r test/e2e/requirements.txt

+ ```

+Integral/Unit tests

+ ```bash

+ # run integral/unit test

+ ./devops/scripts/run_unit_tests.sh

+ # test execution paramaters

+ --mnt-volume = sdapi root dir (default=".")

+ ```

+Regression tests

+ ```bash

+ # run integral/unit test

+ ./devops/scripts/run_regression_tests.sh --cloud-provider= --service-url= --service-key= --idtoken= --tenant= --subproject=

+ # test execution paramaters

+ --mnt-volume = sdapi root dir (default=".")

+ --disable-ssl-verify (to disable ssl verification)

+ ```

+## FAQ

+**How can I generate a new utility command?**

+run the command generation script (`./command_gen.py`) to automatically generate the base infrastructure for integrate new command in the sdutil utility. A folder with the command infrastructure will be created in sdlib/cmd/new_command_name

+```bash

+ ./scripts/command_gen.py new_command_name

+```

+How can I delete all files in a directory?

+```bash

+ ./sdutil ls -lr sd://tenant/subproject/your/folder/here | xargs -r ./sdutil rm --idtoken=x.xxx.x

+```

+**How can I generate the utility changelog?**

+run the changelog script (`./changelog-generator.sh`) to automatically generate the utility changelog

+```bash

+ ./scripts/changelog-generator.sh

+```

+## Setup and usage for Microsoft Energy Data Services

+Below steps are for windows subsystem linux - ubuntu 20.04

+Microsoft Energy Data Services instance is using OSDU&trade; M12 Version of sdutil

+- Download the source code from community [sdutil](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/tree/azure/stable/) Azure Stable branch.

+- In case python virtual env isn't installed, use below commands else skip to next section

+ ```bash

+ sudo apt-get update

+ sudo apt-get install python3-venv

+ ```

+- create new venv and install package

+ ```bash

+ #create new virtual env with name : sdutilenv

+ python3 -m venv sdutilenv

+ #activate the virtual end

+ source sdutilenv/bin/Activate

+ #install python package for sdutil

+ pip install -r requirements.txt

+ ```

+- replace/edit config.yaml in sdlib/config.yaml by this [config-azure.yaml](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/blob/azure/stable/docs/config-azure.yaml)

+- You need to Update three values in `config.yaml`:

+ ```yaml

+ - service: '{"Azure": {"azureGlabEnv":{"url": "<base-url-for-microsoft-energy-data-services-instance>/seistore-svc/api/v3", "appkey": ""}}}'

+ - url: '<base-url-for-microsoft-energy-data-services-instance>/seistore-svc/api/v3'

+ - "refresh_token": "<refresh-token-for-your-env>"

+ ```

+ > [!NOTE]

+ > Follow the directions in [How to Generate a Refresh Token](how-to-generate-refresh-token.md) to obtain a token if not already present.

+- Export or set below environment variables

+ ```bash

+ export AZURE_TENANT_ID=check-env-provisioning-team-as-specific-to-cluster

+ export AZURE_CLIENT_ID=check-env-provisioning-team-as-specific-to-cluster

+ export AZURE_CLIENT_SECRET=check-env-provisioning-team-as-specific-to-cluster

+ ```

+- Run below commands to sign-in, list, upload and download

+ ```bash

+ python sdutil config init

+ python sdutil auth login

+ ```

+- SAMPLE

+ ```code

+ (sdutilenv) > python sdutil config init

+ [one] Azure

+ Select the cloud provider: **enter 1**

+ Insert the Azure (azureGlabEnv) application key: **just press enter--no need to provide a key**

+ sdutil successfully configured to use Azure (azureGlabEnv)

+ Should display sign-in success message. Credentials expiry set to 1 hour.

+ ```

+- list files

+ ```bash

+ python sdutil ls sd://<tenant> e.g. sd://<datapartition>

+ python sdutil ls sd://<tenant>/<subproject> e.g. sd://<datapartition>/test

+ ```

+- upload file

+ ```bash

+ python sdutil cp local-dir/file-name-at-source.txt sd://<datapartition>/test/file-name-at-destination.txt

+ ```

+- download file

+ ```bash

+ python sdutil cp sd://<datapartition>/test/file-name-at-ddms.txt local-dir/file-name-at-destination.txt

+ ```

+ > [!NOTE]

+ > Don't use `cp` command to download VDS files. The VDS conversion results in multiple files, therefore the `cp` command won't be able to download all of them in one command. Use either the [SEGYExport](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/tools/SEGYExport/README.html) or [VDSCopy](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/tools/VDSCopy/README.html) tool instead. These tools use a series of REST calls accessing a [naming scheme](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/connection.html) to retrieve information about all the resulting VDS files.

+OSDU&trade; is a trademark of The Open Group.

+## Next steps

+> [!div class="nextstepaction"]

+> [Tutorial: Steps to interact with Well Delivery DDMS](tutorial-well-delivery-ddms.md)

+ Title: Tutorial - Sample steps to interact with Seismic DDMS in Microsoft Energy Data Services #Required; page title is displayed in search results. Include the brand.

+description: This tutorial shows you how to interact with Seismic DDMS Microsoft Energy Data Services #Required; article description that is displayed in search results.

+# Tutorial: Sample steps to interact with Seismic ddms

+Seismic DDMS provides the capability to operate on seismic data in the Microsoft Energy Data Services instance.

+In this tutorial, you will learn how to:

+> [!div class="checklist"]

+> * Register data partition to seismic

+> * Utilize seismic DDMS Api's to store and retrieve seismic data

+## Prerequisites

+### Microsoft Energy Data Services instance details

+* Once the [Microsoft Energy Data Services instance](./quickstart-create-microsoft-energy-data-services-instance.md) is created, note down the following details:

+

+ | Parameter | Value to use | Example |

+ | | |-- |

+ | CLIENT_ID | Application (client) ID | 3dbbb..... |

+ | CLIENT_SECRET | Client secrets | _fl****************** |

+ | TENANT_ID | Directory (tenant) ID | 72f988bf-86f1-41af-91ab-2d7cd011db47 |

+ | SCOPE | Application (client) ID | 3dbbb..... |

+ | base_uri | URI | instancename.energy.azure.com |

+ | data-partition-id | Data Partition(s) | instancename-datapartitionid |

+### Postman setup

+1. Download and install [Postman](https://www.postman.com/) desktop app

+2. Import the following files into Postman:

+ * To import the Postman collection and environment variables, follow the steps outlined in [Importing data into Postman](https://learning.postman.com/docs/getting-started/importing-and-exporting-data/#importing-data-into-postman)

+ * [Smoke test Postman collection](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/raw/master/source/ddms-smoke-tests/Azure%20DDMS%20OSDU%20Smoke%20Tests.postman_collection.json)

+ * [Smoke Test Environment](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/raw/master/source/ddms-smoke-tests/%5BShip%5D%20osdu-glab.msft-osdu-test.org.postman_environment.json)

+

+3. Update the **CURRENT_VALUE** of the Postman Environment with the information obtained in [Microsoft Energy Data Services instance details](#microsoft-energy-data-services-instance-details)

+## Register data partition to seismic

+ * Script to register

+ ```sh

+ curl --location --request POST '[url]/seistore-svc/api/v3/tenant/{{datapartition}}' \

+ --header 'Authorization: Bearer {{TOKEN}}' \

+ --header 'Content-Type: application/json' \

+ --data-raw '{

+ "esd": "{{datapartition}}.{{domain}}",

+ "gcpid": "{{datapartition}}",

+ "default_acl": "users.datalake.admins@{{datapartition}}.{{domain}}.com"

+ }'

+ ```

+## Utilize seismic ddms API's to store and retrieve seismic data

+In order to use the Seismic DMS, follow the steps in the Seismic DDMS SDUTIL tutorial.

+## Next steps

+<!-- Add a context sentence for the following links -->

+> [!div class="nextstepaction"]

+> [Seismic DDMS SDUTIL tutorial](./tutorial-seismic-ddms-sdutil.md)

+ Title: Microsoft Energy Data Services Preview - Steps to interact with Well Delivery DDMS #Required; page title is displayed in search results. Include the brand.

+description: This tutorial shows you how to interact with Well Delivery DDMS #Required; article description that is displayed in search results.

+# Tutorial: Sample steps to interact with Well Delivery ddms

+Well Delivery DDMS provides the capability to manage well related data in the Microsoft Energy Data Services Preview instance.

+In this tutorial, you'll learn how to:

+> [!div class="checklist"]

+> * Utilize Well Delivery DDMS API's to store and retrieve well data

+## Prerequisites

+### Get Microsoft Energy Data Services Preview instance details

+* Once the [Microsoft Energy Data Services Preview instance](quickstart-create-microsoft-energy-data-services-instance.md) is created, note down the following details:

+```Table

+ | Parameter | Value to use | Example |

+ | | |-- |

+ | CLIENT_ID | Application (client) ID | 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx |

+ | CLIENT_SECRET | Client secrets | _fl****************** |

+ | TENANT_ID | Directory (tenant) ID | 72f988bf-86f1-41af-91ab-xxxxxxxxxxxx |

+ | SCOPE | Application (client) ID | 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx |

+ | base_uri | URI | <instance>.energy.azure.com |

+ | data-partition-id | Data Partition(s) | <instance>-<data-partition-name> |

+```

+### How to set up Postman

+* Download and install [Postman](https://www.postman.com/) desktop app.

+* Import the following files into Postman:

+ * [Well Delivery DDMS Postman collection](https://raw.githubusercontent.com/microsoft/meds-samples/main/postman/WelldeliveryDDMS.postman_collection.json)

+ * [Well Delivery DDMS Postman Environment](https://raw.githubusercontent.com/microsoft/meds-samples/main/postman/WelldeliveryDDMSEnviroment.postman_environment.json)

+

+* Update the **CURRENT_VALUE** of the Postman Environment with the information obtained in [Microsoft Energy Data Services Preview instance details](#get-microsoft-energy-data-services-preview-instance-details).

+### How to execute Postman requests

+* The Postman collection for Well Delivery DDMS contains requests that allows interaction with wells, wellbore, well planning, wellbore planning, well activity program and well trajectory data.

+* Make sure to choose the **Well Delivery DDMS Environment** before triggering the Postman collection.

+* Each request can be triggered by clicking the **Send** Button.

+* On every request Postman will validate the actual API response code against the expected response code; if there's any mismatch the Test Section will indicate failures.

+### Generate a token

+1. **Get a Token** - Import the CURL command in Postman to generate the bearer token. Update the bearerToken in well delivery ddms environment. Use Bearer Token as Authorization type in other API calls.

+ ```bash

+ curl --location --request POST 'https://login.microsoftonline.com/{{TENANT_ID}}/oauth2/v2.0/token' \

+ --header 'Content-Type: application/x-www-form-urlencoded' \

+ --data-urlencode 'grant_type=client_credentials' \

+ --data-urlencode 'client_id={{CLIENT_ID}}' \

+ --data-urlencode 'client_secret={{CLIENT_SECRET}}' \

+ --data-urlencode 'scope={{SCOPE}}'

+ ```

+ :::image type="content" source="media/tutorial-well-delivery/screenshot-of-the-well-delivery-generate-token.png" alt-text="Screenshot of the well delivery generate token." lightbox="media/tutorial-well-delivery/screenshot-of-the-well-delivery-generate-token.png":::

+## Store and retrieve well data with Well Delivery ddms APIs

+1. **Create a Legal Tag** - Create a legal tag that will be added automatically to the environment for data compliance purpose.

+1. **Create Well** - Creates the well record.

+ :::image type="content" source="media/tutorial-well-delivery/screenshot-of-the-well-delivery-create-well.png" alt-text="Screenshot of the well delivery - create well." lightbox="media/tutorial-well-delivery/screenshot-of-the-well-delivery-create-well.png":::

+1. **Create Wellbore** - Creates the wellbore record.

+ :::image type="content" source="media/tutorial-well-delivery/screenshot-of-the-well-delivery-create-well-bore.png" alt-text="Screenshot of the well delivery - create wellbore." lightbox="media/tutorial-well-delivery/screenshot-of-the-well-delivery-create-well-bore.png":::

+1. **Get Well Version** - Returns the well record based on given WellId.

+ :::image type="content" source="media/tutorial-well-delivery/screenshot-of-the-well-delivery-get-well.png" alt-text="Screenshot of the well delivery - get well." lightbox="media/tutorial-well-delivery/screenshot-of-the-well-delivery-get-well.png":::

+1. **Get Wellbore Version** - Returns the wellbore record based on given WellboreId.

+ :::image type="content" source="media/tutorial-well-delivery/screenshot-of-the-well-delivery-get-well-bore.png" alt-text="Screenshot of the well delivery - get wellbore." lightbox="media/tutorial-well-delivery/screenshot-of-the-well-delivery-get-well-bore.png":::

+1. **Create ActivityPlan** - Create the ActivityPlan.

+ :::image type="content" source="media/tutorial-well-delivery/screenshot-of-the-well-delivery-create-activity-plan.png" alt-text="Screenshot of the well delivery - create activity plan." lightbox="media/tutorial-well-delivery/screenshot-of-the-well-delivery-create-activity-plan.png":::

+1. **Get ActivityPlan by Well Id** - Returns the Activity Plan object from a wellId generated in Step 1.

+ :::image type="content" source="media/tutorial-well-delivery/screenshot-of-the-well-delivery-activity-plans-by-well.png" alt-text="Screenshot of the well delivery - get activity plan by well." lightbox="media/tutorial-well-delivery/screenshot-of-the-well-delivery-activity-plans-by-well.png":::

+1. **Delete wellbore record** - Deletes the specified wellbore record.

+ :::image type="content" source="media/tutorial-well-delivery/screenshot-of-the-well-delivery-delete-well-bore.png" alt-text="Screenshot of the well delivery - delete wellbore." lightbox="media/tutorial-well-delivery/screenshot-of-the-well-delivery-delete-well-bore.png":::

+1. **Delete well record** - Deletes the specified well record.

+ :::image type="content" source="media/tutorial-well-delivery/screenshot-of-the-well-delivery-delete-well.png" alt-text="Screenshot of the well delivery - delete well." lightbox="media/tutorial-well-delivery/screenshot-of-the-well-delivery-delete-well.png":::

+Completion of the above steps indicates successful creation and retrieval of well and wellbore records. Similar steps could be followed for well planning, wellbore planning, well activity program and wellbore trajectory data.

+## See also

+Advance to the next tutorial to learn how to use sdutil to load seismic data into seismic store

+> [!div class="nextstepaction"]

+> [Tutorial: Sample steps to interact with Wellbore ddms](tutorial-wellbore-ddms.md)

+ Title: Tutorial - Sample steps to interact with Wellbore DDMS #Required; page title is displayed in search results. Include the brand.

+description: This tutorial shows you how to interact with Wellbore DDMS in Microsoft Energy Data Services #Required; article description that is displayed in search results.

+# Tutorial: Sample steps to interact with Wellbore ddms

+Wellbore ddms provides the capability to operate on well data in the Microsoft Energy Data Services instance.

+In this tutorial, you'll learn how to:

+> [!div class="checklist"]

+> * Utilize Wellbore ddms APIs to store and retrieve Wellbore and well log data

+## Prerequisites

+### Microsoft Energy Data Services instance details

+* Once the [Microsoft Energy Data Services Preview instance](quickstart-create-microsoft-energy-data-services-instance.md) is created, note down the following details:

+```Table

+ | Parameter | Value to use | Example |

+ | | |-- |

+ | CLIENT_ID | Application (client) ID | 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx |

+ | CLIENT_SECRET | Client secrets | _fl****************** |

+ | TENANT_ID | Directory (tenant) ID | 72f988bf-86f1-41af-91ab-xxxxxxxxxxxx |

+ | SCOPE | Application (client) ID | 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx |

+ | base_uri | URI | <instance>.energy.azure.com |

+ | data-partition-id | Data Partition(s) | <instance>-<data-partition-name> |

+```

+### Postman setup

+* Download and install [Postman](https://www.postman.com/) desktop app

+* Import the following files into Postman:

+ * [Wellbore ddms Postman collection](https://raw.githubusercontent.com/microsoft/meds-samples/main/postman/WellboreDDMS.postman_collection.json)

+ * [Wellbore ddms Postman Environment](https://raw.githubusercontent.com/microsoft/meds-samples/main/postman/WellboreDDMSEnvironment.postman_environment.json)

+

+* Update the **CURRENT_VALUE** of the Postman Environment with the information obtained in [Microsoft Energy Data Services instance details](#microsoft-energy-data-services-instance-details)

+### Executing Postman Requests

+* The Postman collection for Wellbore ddms contains requests that allows interaction with wells, wellbore, well log and well trajectory data.

+* Make sure to choose the **Wellbore DDMS Environment** before triggering the Postman collection.

+ :::image type="content" source="media/tutorial-wellbore-ddms/tutorial-postman-choose-wellbore-environment.png" alt-text="Choose environment." lightbox="media/tutorial-wellbore-ddms/tutorial-postman-choose-wellbore-environment.png":::

+* Each request can be triggered by clicking the **Send** Button.

+* On every request Postman will validate the actual API response code against the expected response code; if there's any mismatch the Test Section will indicate failures.

+**Successful Postman Call**

+**Failed Postman Call**

+### Utilize Wellbore ddms APIs to store and retrieve wellbore and well log data

+1. **Get an SPN Token** - Generate the Service Principal Bearer token, which will be used to authenticate further API calls.

+2. **Create a Legal Tag** - Create a legal tag that will be added automatically to the environment for data compliance purpose.

+3. **Create Well** - Creates the wellbore record in Microsoft Energy Data Services instance.

+ :::image type="content" source="media/tutorial-wellbore-ddms/tutorial-create-well.png" alt-text="Screenshot of creating a Well." lightbox="media/tutorial-wellbore-ddms/tutorial-create-well.png":::

+4. **Get Wells** - Returns the well data created in the last step.

+ :::image type="content" source="media/tutorial-wellbore-ddms/tutorial-get-wells.png" alt-text="Screenshot of getting all wells." lightbox="media/tutorial-wellbore-ddms/tutorial-get-wells.png":::

+1. **Get Well Versions** - Returns the versions of each ingested well record.

+ :::image type="content" source="media/tutorial-wellbore-ddms/tutorial-get-well-versions.png" alt-text="Screenshot of getting all Well versions." lightbox="media/tutorial-wellbore-ddms/tutorial-get-well-versions.png":::

+1. **Get specific Well Version** - Returns the details of specified version of specified record.

+ :::image type="content" source="media/tutorial-wellbore-ddms/tutorial-get-specific-well-version.png" alt-text="Screenshot of getting a specific well version." lightbox="media/tutorial-wellbore-ddms/tutorial-get-specific-well-version.png":::

+1. **Delete well record** - Deletes the specified record.

+ :::image type="content" source="media/tutorial-wellbore-ddms/tutorial-delete-well.png" alt-text="Screenshot of delete well record." lightbox="media/tutorial-wellbore-ddms/tutorial-delete-well.png":::

+***Successful completion of above steps indicates success ingestion and retrieval of well records***

+## Next steps

+Advance to the next tutorial to learn about sdutil

+> [!div class="nextstepaction"]

+> [Tutorial: Seismic store sdutil](tutorial-seismic-ddms-sdutil.md)

+>Get notified about when to revisit this page for updates by copying and pasting this URL: `https://learn.microsoft.com/api/search/rss?search=%22Release+notes+-+Azure+Event+Grid%22&locale=en-us` into your  feed reader.

+> For Apache Kafka clients, an **event hub** maps to a **Kafka topic**. For more mappings between Azure Event Hubs and Apache Kafka, see [Kafka and Event Hubs conceptual mapping](event-hubs-for-kafka-ecosystem-overview.md#kafka-and-event-hubs-conceptual-mapping)

+### Kafka and Event Hubs conceptual mapping

+| Topic | An event hub |

+> [!NOTE]

+> The above configuration properties are for the Java programming language. For **samples** that show how to use OAuth with Event Hubs for Kafka using different programming languages, see [samples on GitHub](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth).

+The goal of Event Hubs for Apache Kafka is to provide access to Azure Event Hubs capabilities to applications that are locked into the Apache Kafka API and would otherwise have to be backed by an Apache Kafka cluster.

+The payload of any Event Hubs event is a byte stream and the content can be compressed with an algorithm of your choosing. The Apache Avro encoding format supports compression natively.

+For a **tutorial** with step-by-step instructions to create an event hub and access it using SAS or OAuth, see [Quickstart: Data streaming with Event Hubs using the Kafka protocol](event-hubs-quickstart-kafka-enabled-event-hubs.md).

+Also, see the [OAuth samples on GitHub](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth).

+> - Debezium will auto-create a topic per table and a bunch of metadata topics. Kafka **topic** corresponds to an Event Hubs instance (event hub). For Apache Kafka to Azure Event Hubs mappings, see [Kafka and Event Hubs conceptual mapping](event-hubs-for-kafka-ecosystem-overview.md#kafka-and-event-hubs-conceptual-mapping).

+An ExpressRoute circuit represents a logical connection between your on-premises infrastructure and Microsoft cloud services through a connectivity provider. You can have multiple ExpressRoute circuits. Each circuit can be in the same or different regions, and can be connected to your premises through different connectivity providers.

+ExpressRoute circuits don't map to any physical entities. A circuit is uniquely identified by a standard GUID called as a service key (s-key). The service key is the only piece of information exchanged between Microsoft, the connectivity provider, and you. The s-key isn't a secret for security purposes. There's a 1:1 mapping between an ExpressRoute circuit and the s-key.

+New ExpressRoute circuits can include two independent peerings: Private peering and Microsoft peering. Whereas existing ExpressRoute circuits may have three peerings: Azure Public, Azure Private and Microsoft. Each peering is a pair of independent BGP sessions, each of them configured redundantly for high availability. There's a 1:N (1 <= N <= 3) mapping between an ExpressRoute circuit and routing domains. An ExpressRoute circuit can have any one, two, or all three peerings enabled per ExpressRoute circuit.

+### Circuit SKU upgrade and downgrade

+#### Allowed workflow

+* Upgrade from Standard to Premium SKU.

+* Upgrade from Local to Standard or Premium SKU.

+ * Can only be done using Azure CLI or Azure PowerShell.

+ * Billing type must be **unlimited**.

+* Changing from *MeteredData* to *UnlimitedData*.

+#### Unsupported workflow

+* Downgrade from Premium to Standard SKU.

+* Changing from *UnlimitedData* to *MeteredData*.

+For more information on services supported, costs, and configuration details, see the [FAQ page](expressroute-faqs.md). For information on the list of connectivity providers offering Microsoft peering support, see the [ExpressRoute locations](expressroute-locations.md) page.

+Each peering requires separate BGP sessions (one pair for each peering type). The BGP session pairs provide a highly available link. If you're connecting through layer 2 connectivity providers, you're responsible for configuring and managing routing. You can learn more by reviewing the [workflows](expressroute-workflows.md) for setting up ExpressRoute.

+ExpressRoute circuits can be monitored for availability, connectivity to VNets and bandwidth utilization using [ExpressRoute Network Insights](expressroute-network-insights.md).

+Connection Monitor for Expressroute monitors the health of Azure private peering and Microsoft peering. For more information on configuration, see [Configure Connection Monitor for ExpressRoute](how-to-configure-connection-monitor.md).

+* [Learn module: Introduction to Azure ExpressRoute](/training/modules/intro-to-azure-expressroute).

+- Take the [Cost Management](/training/paths/control-spending-manage-bills?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) guided learning course.

+- [Learn module: Introduction to Azure Firewall Manager](/training/modules/intro-to-azure-firewall-manager/).

+- [Learn module: Introduction to Azure Firewall](/training/modules/introduction-azure-firewall/)

+* [Learn module: Introduction to Azure Front Door](/training/modules/intro-to-azure-front-door/).

+ :::image type="content" source="./media/apache-spark-jupyter-spark-sql/jupyter-spark-kernel-status.png " alt-text="Screenshot shows a Jupyter window with a PySpark indicator." border="true":::

+Replace the line above with the following line once https://learn.microsoft.com/azure/templates/microsoft.healthcareapis/services goes live:

+> [!NOTE]

+> The Connection State of your peering connections must be Active before registering any prefixes.

+**Q.** When will my BGP peer come up?

+**A.** After the LAG comes up our automated process configures BGP. Note, BFD must be configured on the non-MSFT peer to start route exchange.

+**Q.** When will peering IP addresses be allocated and displayed in the Azure portal?

+**A.** Our automated process allocates addresses and sends the information via email after the port is configured on our side.

+**Q.** Are there any AS path constraints?

+**A.** Yes, for registered prefixes smaller than /24, advertised AS path must be less than four. Path of four or longer will cause the advertisement to be rejected by policy.

+**Q.** How is progress communicated outside of the portal status?

+**A.** Automated emails are sent at varying milestones

+To learn more, see [Manage an IoT Central application with the REST API](/training/modules/manage-iot-central-apps-with-rest-api/).

+To learn more, see [Tutorial: Add an Azure IoT Edge device to your Azure IoT Central application](/training/modules/connect-iot-edge-device-to-iot-central/).

+Now that you've learned how to manage jobs with the REST API, a suggested next step is to learn how to [Manage IoT Central applications with the REST API](/training/modules/manage-iot-central-apps-with-rest-api/).

+- You can also add device templates to an IoT Central application using the [REST API](/training/modules/manage-iot-central-apps-with-rest-api/) or the [CLI](howto-manage-iot-central-from-cli.md).

+To get started with the data plane APIs, see [Explore the IoT Central APIs](/training/modules/manage-iot-central-apps-with-rest-api/).

+Now that you have an overview of Azure IoT Central and are familiar with the capabilities of the IoT Central REST API, the suggested next step is to complete the [Explore the IoT Central APIs](/training/modules/manage-iot-central-apps-with-rest-api/) Learn module.

+To learn more, see [Add an Azure IoT Edge device to your Azure IoT Central application](/training/modules/connect-iot-edge-device-to-iot-central/).

+As an application platform, IoT Central lets you transform your IoT data into the business insights that drive actionable outcomes. [Rules](./tutorial-create-telemetry-rules.md), [data export](./howto-export-to-blob-storage.md), and the [public REST API](/training/modules/manage-iot-central-apps-with-rest-api/) are examples of how you can integrate IoT Central with line-of-business applications:

+> [Add an Azure IoT Edge device to your Azure IoT Central application](/training/modules/connect-iot-edge-device-to-iot-central/)

+This article helped you set up your virtual machine and IoT Edge device to be GPU-accelerated. To run an application with a similar setup, try the learning path for [NVIDIA DeepStream development with Microsoft Azure](/training/paths/nvidia-deepstream-development-with-microsoft-azure/?WT.mc_id=iot-47680-cxa). The Learn tutorial shows you how to develop optimized Intelligent Video Applications that can consume multiple video, image, and audio sources.

+- Download and unzip the [IoT C# SDK](https://github.com/Azure/azure-iot-sdk-csharp/archive/main.zip).

+1. Open a command window and go to the folder where you unzipped the IoT C# SDK. Find the folder with the arm-read-write.csproj file. You create the environment variables in this command window. Log into the [Azure portal](https://portal.azure.com) to get the keys. Select **Resource Groups** then select the resource group used for this quickstart.

+ > These messages are encoded in UTF-32 and base64. If you read the message back, you have to decode it from base64 and utf-32 in order to read it as ASCII. If you're interested, you can use the method ReadOneRowFromFile in the Routing Tutorial to read one for from one of these message files and decode it into ASCII. ReadOneRowFromFile is in the IoT C# SDK repository that you unzipped for this quickstart. Here is the path from the top of that folder: *./iothub/device/samples/getting started/RoutingTutorial/SimulatedDevice/Program.cs.* Set the boolean `readTheFile` to true, and hardcode the path to the file on disk, and it will open and translate the first row in the file.

+# https://learn.microsoft.com/azure/iot-hub/iot-hub-devguide-security#sas-token-structure

+# Use the generate_sas_token implementation that's available here: https://learn.microsoft.com/azure/iot-hub/iot-hub-devguide-security#sas-token-structure

+# https://learn.microsoft.com/azure/iot-hub/iot-hub-devguide-security#sas-token-structure

+The cloning article has a working sample associated with it, which is located in the IoT C# samples on this page: [Azure IoT hub service samples for C#](https://github.com/Azure/azure-iot-sdk-csharp/tree/main/iothub/service/samples/how%20to%20guides), with the project being ImportExportDevicesSample. You can download the sample and try it out; there are instructions in the [How to Clone an IoT Hub](iot-hub-how-to-clone.md) article.

+* **FileUploadSample**. This device app uploads a file to storage using a SAS URI provided by your IoT hub. You'll run this app from the Azure IoT C# SDK repository that you download in the prerequisites.

+* Download the Azure IoT C# SDK from [Download sample](https://github.com/Azure/azure-iot-sdk-csharp/archive/main.zip) and extract the ZIP archive.

+In this article, you use a sample from the Azure IoT C# SDK repository you downloaded earlier as the device app. You can open the files below using Visual Studio, Visual Studio Code, or a text editor of your choice.

+The sample is located at **azure-iot-sdk-csharp/iothub/device/samples/getting started/FileUploadSample** in the folder where you extracted the Azure IoT C# SDK.

+1. Next, run the device app to upload the file to Azure storage. Open a new command prompt and change folders to the **azure-iot-sdk-csharp\iothub\device\samples\getting started\FileUploadSample** under the folder where you expanded the Azure IoT C# SDK. Run the following commands. Replace the `{Your device connection string}` placeholder value in the second command with the device connection string you saw when you registered a device in the IoT Hub.

+| .NET | [ReadD2cMessages .NET](https://github.com/Azure/azure-iot-sdk-csharp/tree/main/iothub/device/samples/Getting%20Started/ReadD2cMessages) |